Dynamic SLAM method and system based on lightweight YOLOv8 and adaptive key frame strategy

Abstract

The invention relates to a dynamic SLAM method and system based on lightweight YOLOv8 and an adaptive key frame strategy, and belongs to the technical field of computer vision. According to the method, a dynamic region in an image frame is detected through an improved lightweight network, and extracted dynamic feature points are removed in combination with an epipolar geometry constrained motion consistency detection method; extracting line features from the static region based on an improved LSD algorithm, and compensating the loss of feature points; designing a key frame selection strategy of adaptive weight to delete redundant key frames, and constructing a joint optimization objective function by combining static point features and static line features to perform pose estimation and construction of a point cloud map; and finally, performing closed-loop matching by using visual and semantic features between the key frames, verifying a closed loop, establishing a pose constraint, and performing global graph optimization. According to the method, the influence of a dynamic object in a scene on the mapping quality of the SLAM algorithm can be effectively reduced, and the robustness of the algorithm in a dynamic environment is enhanced.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a dynamic SLAM method and system based on lightweight YOLOv8 and adaptive keyframe strategy. Background Technology

[0002] With the continuous maturation of artificial intelligence technology and the expansion of its application scope, intelligent robots have been widely used in industries, services, agriculture, and other fields. The "Development Plan for Building a World-Class Intelligent Connected New Energy Vehicle Industry Cluster in Chongqing" clearly outlines the development priorities of the robotics industry in its key technology innovation roadmap: breaking through key technologies for intelligent robots and developing intelligent robots to meet the challenges of the new round of technological revolution and industrial transformation.

[0003] In the current development of industrial robot technology, Simultaneous Localization and Mapping (SLAM) technology plays a crucial role. Early SLAM systems mainly relied on LiDAR sensors. While these systems could operate well in specific environments, they lacked rich semantic information compared to cameras. With the development of camera technology, visual SLAM technology emerged, which reduced costs and improved system scalability by leveraging cameras.

[0004] SLAM systems include direct methods and feature-point-based methods. Direct methods estimate pose by minimizing the pixel grayscale error between consecutive frames, relying heavily on the assumption of consistent illumination. However, in practical applications, they are easily affected by changes in illumination, leading to error accumulation. Feature-point-based methods, on the other hand, are based on geometric error constraints and achieve motion calculation by optimizing the reprojection error of feature matching points. Some methods use improved feature points to achieve scale and rotation invariance. However, traditional SLAM algorithms have lower accuracy and significantly degraded performance in dynamic scenes.

[0005] In dynamic scenes, the primary goal of visual SLAM algorithms is to effectively distinguish between dynamic targets and outliers, which can be achieved by combining them with deep learning. Algorithms that utilize deep learning to remove dynamic feature points, and combining specific algorithms with ORB SLAM, can effectively detect and segment dynamic targets. However, this method relies on target detection and semantic segmentation for localization and tracking, and is dependent on annotation, which is particularly problematic in real-time applications. There are networks that tightly integrate target detection and geometric constraint methods, reducing the impact of dynamic targets by adding lightweight segmentation networks to the SLAM system; however, lightweight networks have limitations in recognizing dynamic targets in complex scenes. Deep learning-based visual SLAM systems have been proposed, which can improve the accuracy of visual SLAM systems in dynamic environments, but require significant computational resources, limiting their application on resource-constrained devices. Furthermore, some deep learning-based visual SLAM systems offer significantly improved accuracy in dynamic environments compared to traditional methods, but still face hardware performance dependencies and limitations in the accuracy of deep learning networks. Finally, there are visual SLAM systems that combine semantic segmentation and geometric information, but these may completely exclude the entire region containing dynamic objects when distinguishing them, resulting in the exclusion of static feature points within the dynamic region. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a dynamic SLAM method and system based on lightweight YOLOv8 and an adaptive keyframe strategy. By detecting objects in the environment through object detection and region segmentation threads, dynamic regions, potential dynamic regions, and static regions in the image are identified, feature points in the dynamic regions are removed, and then line features are extracted from the static regions based on image descriptors to compensate for the features. This can effectively improve the robustness of the SLAM algorithm and solve the problems of decreased positioning accuracy and difficulty in pose estimation caused by dense moving objects in dynamic scenes.

[0007] To achieve the above objectives, the present invention provides the following technical solution: On the one hand, a dynamic SLAM method based on lightweight YOLOv8 and an adaptive keyframe strategy is disclosed, which includes the following steps: S1. Acquire consecutive raw image frames with known timestamps through the camera to provide spatiotemporally synchronized visual input for feature extraction, matching, and pose estimation; S2. Set up the LiteC2f module in the neck network of the YOLOv8 model to build an improved lightweight YOLOv8s-seg model, and use the lightweight model to extract the mask of the dynamic target region within the current image frame. S3. Extract ORB feature points from the non-dynamic mask region, filter the static feature subset based on the static mask and remove abnormal feature points, and estimate the camera pose by minimizing the reprojection error of the static feature points. S4. Based on the improved LSD algorithm and LBD descriptor, extract line features in the static region, and jointly optimize the static feature point reprojection error and line reprojection distance to perform pose estimation. S5. Keyframes are selected through an adaptive weighted keyframe strategy, new map points are generated, BA optimization is performed on the local map point cloud, and redundant keyframes and noise points are deleted. S6. Utilize visual and semantic features between keyframes for loop closure matching. After verifying loop closure, establish pose constraints and perform global graph optimization. During the optimization process, update the semantic attributes of map points and lines to achieve relocalization and semantic mapping. Figure 1 Consistent maintenance.

[0008] Further, in step S1, the original image is read from the camera, distortion correction is performed using camera calibration parameters, and the image is converted to grayscale format; then, a multi-layer image pyramid is constructed to adapt to different scale requirements, and uniformly distributed ORB feature points and descriptors are extracted on each layer; the processed image data is sent to the tracking module for subsequent pose estimation and map construction.

[0009] Furthermore, in step S2, the depthwise separable convolution module DSConv is used in the LiteC2f module to reduce computational complexity, resulting in the output of the improved lightweight YOLOv8s-seg model:

[0010] In the formula, This represents the final output of the improved model. This represents the balanced convolution method. This represents the standard convolution operation. This indicates a channel shuffling operation. This represents depthwise separable convolution. Represents the original feature map. This represents the original input feature map. The first one obtained after channel splitting Each branch feature map The total number of branches after the input feature map is split; Then, the lightweight YOLOv8s-seg model is used to process the input image frames. Perform semantic segmentation to obtain each pixel Semantic category labels and instance masks Define the set of potential dynamic categories as The static region mask is then represented as:

[0011] In the formula, Represents pixels Semantic category labels.

[0012] Furthermore, in step S3, a set of feature points is obtained through ORB feature extraction. Regions belonging to the potential dynamic category are marked as dynamic regions, and then a subset of static features is selected based on the static mask:

[0013] Based on static feature subsets Camera pose is estimated by minimizing the reprojection error of static feature points. :

[0014] In the formula, For camera projection model, For the corresponding 3D map points, For robust kernel functions; For rotation matrix, It is a translation vector. It is a special Euclidean group.

[0015] Furthermore, in step S4, the process of extracting line features within the static region based on the improved LSD algorithm and LBD descriptor, and then performing camera pose estimation, includes: First, calculate the gradient magnitude of the input image:

[0016] In the formula, and These represent the brightness gradients of the image in the horizontal and vertical directions, respectively. Then, in each local window Calculate the mean and standard deviation of the gradient within the inner range:

[0017]

[0018] In the formula, Represents a local window Any pixel in the array; This represents the mean of the gradient. This represents the standard deviation of each gradient; Therefore, the edge threshold is adaptively determined:

[0019] In the formula, Indicates adaptive weights; if Then the pixels Consider them as candidate edge points; Subsequently, directional clustering is performed on the candidate edge point set; for each group of points to be connected... Its gradient direction is , amplitude Define the gradient-weighted average direction:

[0020] And calculate the directional consistency score:

[0021] Define the consistency score threshold as ,when Furthermore, the orthogonality error from the point to the fitted line is below the threshold. When the point set is accepted as a line segment; Next, only samples with a static region percentage exceeding a threshold are retained. The candidate line, i.e. ;in, This represents the first uniform sampling of a candidate line segment. The pixel coordinates of each point This represents the total number of sampling points; Perform directional clustering on the candidate edge set. For each group of points to be connected:

[0022] in, and These represent the two endpoints of the line segment; Then, a subset of static line segments is selected based on the static mask:

[0023] For each static line segment Use LBD to calculate its local descriptor. And utilize static point feature set With static line feature set Construct a joint optimization objective function, and solve the camera pose again by minimizing the reprojection error between points and lines:

[0024] in, For the observed image line segments, The reprojection distance of the line is represented; the final camera pose is jointly optimized by fusing the static point reprojection error and the line reprojection distance, and used for subsequent localization and mapping.

[0025] Furthermore, in step S5, it is first determined whether a keyframe needs to be inserted. The determination method is as follows: First, determine whether there is new disparity information based on the pose changes between the current frame and the reference keyframe:

[0026] If the current frame and reference keyframe The camera translation vector exceeds the translation threshold. Or two postures rotating and The angle difference in the Lie algebraic form exceeds the angle threshold. This indicates a significant change in perspective, requiring new keyframes to cover the new area; Then calculate the number of validly matched static point features in the current frame. Sum of line characteristics The observation richness of the current frame is evaluated using an adaptive weighting method:

[0027] in, , These correspond to the number of static point features that were successfully matched in the current frame. and static line characteristic number The weight, These are the number of static point features and the number of static line features in the reference keyframe, respectively. Indicates the match retention rate, when Below the set threshold If the condition is met, then determine that a keyframe should be inserted; and determine the keyframe to be inserted as follows: .

[0028] Furthermore, in step S5, based on the selected keyframes... The process of generating new map points is as follows: First, from the keyframes Extract static point and line features and compare them with adjacent keyframes. Pairing based on matching features in the dataset; Then, using the pose of the two cameras... and Based on the matched static point pairs Perform triangulation and calculate the coordinates of the three-dimensional point:

[0029] This allows for the extraction of 3D points from a static background. And add it to the local map; For points falling on line features or matches generated from corresponding line segments in two frames, triangulation is performed using line endpoint constraints to obtain three-dimensional line segments. All newly generated map points and line segments will be stored together with the observation relationships of the corresponding keyframes for subsequent local BA optimization and map building.

[0030] Furthermore, in step S5, the process of performing BA optimization on the local map point cloud is as follows: Add keyframes to the new local map Select a local sub-map from the shared keyframes, including the current keyframe, adjacent keyframes, and their shared map points and lines; Then, pose for the co-view keyframes With map points line segments The goal of local BA optimization is to minimize the reprojection error of all static points and lines.

[0031] After optimization, maintain the map as follows: Remove map point and line features that have fewer observations than a preset value or whose reprojection error is greater than a preset value; Redundancy detection and deletion are performed on keyframes with highly overlapping co-view relationships and redundant observation information to maintain map sparsity.

[0032] Furthermore, in step S6, during the ongoing local mapping process, the current keyframe is continuously monitored. Is it related to historical keyframes? Forming loops: Similarity calculation based on the bag-of-words model measures the visual similarity between two frames. Each keyframe is composed of its set of feature descriptors. Represented as a bag-of-words vector Where M is the size of the visual dictionary, and the similarity score between two frames is defined as:

[0033] in , These are the bag-of-words weights of the current keyframe and the candidate keyframes; if similarity Exceeding the set threshold If so, it is determined to be a potential loop candidate; Perform geometric consistency verification between candidate keyframes: calculate the similarity transformation between cameras by matching static point and line features between the two frames. It is obtained by minimizing the reprojection error estimate:

[0034] in, For pixel observations in the current frame, The map points are candidate keyframes; if the optimized reprojection error is below the threshold and the number of matches is sufficient, then the loop closure is confirmed. After successful detection, a new loop closure constraint edge is added to the global pose graph, and this error term is included in the global graph optimization objective.

[0035] Then, in the subsequent global BA, loop closure constraints are used to eliminate cumulative drift and achieve a globally consistent map structure: global optimization is performed on the fused map, adjusting the poses of all keyframes. With map points line segments As an optimization variable, minimizing the global reprojection error function completes the global optimization:

[0036] After global optimization, all local sub-maps are spatially consistent and drift-free. After optimization, the poses of keyframes, the 3D positions of points and lines, and the co-view relationships in the global map are updated to obtain a globally consistent map structure without cumulative errors, thus achieving closed-loop correction and long-term stable mapping.

[0037] On the other hand, a system for executing the aforementioned dynamic SLAM method based on lightweight YOLOv8 and adaptive keyframe strategy is also disclosed. This system includes a semantic module, a tracking module, a local mapping module, a loop closure detection module, and a global optimization module. The semantic module processes the input image frames through an improved lightweight YOLOv8s-seg network. Perform semantic segmentation to obtain the semantic category label and instance mask for each pixel; The tracking module uses a lightweight network to detect dynamic regions within an image frame, combines an epipolar geometry-constrained motion consistency detection method to remove extracted dynamic feature points, and extracts line features from static regions based on an improved LSD algorithm to compensate for the loss of feature points. The local mapping module uses an adaptive weight-based keyframe selection strategy to remove redundant keyframes and combines static point features and static line features to construct a joint optimization objective function for pose estimation and point cloud map construction. The loop closure detection module uses visual and semantic features between keyframes to perform loop closure matching. After verifying the loop closure, the global optimization module establishes pose constraints and performs global graph optimization.

[0038] The beneficial effects of this invention are as follows: (1) This invention addresses the problem that adding a segmentation network to the algorithm would lead to excessive computational demands on the system, and proposes a lightweight segmentation network. Multiple Bottlenecks in the multi-scale feature fusion module C2f of YOLOv8 are replaced with DSConv to achieve a lightweight design. DSConv mainly reduces computational complexity through depthwise separable convolutions, thereby reducing the computational complexity of the algorithm.

[0039] (2) In view of the problem that the number of static feature points in the image after dynamic feature point removal may be insufficient or the feature points are unevenly distributed, this invention proposes an LSD algorithm that introduces a local adaptive gradient threshold and a gradient weighting direction consistency criterion to extract line features from static regions, thereby improving detection accuracy and reducing computation.

[0040] (3) In the local mapping module, this invention proposes an adaptive weighted keyframe strategy and combines it with the point and line features of static regions for pose estimation and point cloud map construction. Then, it combines loop closure detection and global BA modules to complete map optimization and updating. By measuring the degree of preservation of static geometric information between the current frame and the reference keyframe, it determines whether new keyframes need to be inserted to supplement new point and line constraints and maintain the integrity and robustness of the map.

[0041] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0042] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is an overall architecture diagram of a dynamic SLAM system based on lightweight YOLOv8 and adaptive keyframe strategy according to an embodiment of the present invention. Figure 2 This is a flowchart of a dynamic SLAM method based on lightweight YOLOv8 and adaptive keyframe strategy according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the lightweight improved YOLOv8 network neck structure according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the keyframe and semantic frame strategy for adaptive weighting in an embodiment of the present invention. Detailed Implementation

[0043] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0044] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0045] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0046] Please see Figures 1-4 This paper presents a dynamic SLAM method and system based on lightweight YOLOv8 and an adaptive keyframe strategy.

[0047] Example This embodiment first provides a dynamic SLAM system based on lightweight YOLOv8 and an adaptive keyframe strategy, such as... Figure 1 As shown, the system includes a semantic module, a tracking module, a local mapping module, a loop closure detection module, and a global optimization module. The semantic module uses an improved lightweight YOLOv8s-seg network to process the input image frames. Semantic segmentation is performed to obtain semantic category labels and instance masks for each pixel. The tracking module uses a lightweight network to detect dynamic regions within image frames and combines an epipolar geometry-constrained motion consistency detection method to remove extracted dynamic feature points. It also extracts line features from static regions based on an improved LSD algorithm to compensate for feature point loss. The local mapping module uses an adaptive weight-based keyframe selection strategy to remove redundant keyframes and combines static point features and static line features to construct a joint optimization objective function for pose estimation and point cloud map construction. The loop closure detection module uses visual and semantic features between keyframes to perform loop closure matching. After verifying loop closure, the global optimization module establishes pose constraints and performs global graph optimization.

[0048] This embodiment also provides a detailed implementation of a dynamic SLAM method based on lightweight YOLOv8 and an adaptive keyframe strategy, such as... Figure 2 The flowchart shown is a dynamic SLAM method based on lightweight YOLOv8 and an adaptive keyframe strategy, which includes the following detailed steps: S1. Acquire consecutive raw image frames with known timestamps through the camera to provide spatiotemporally synchronized visual input for feature extraction, matching, and pose estimation; S2. Improve the neck structure of the YOLOv8 network and extract the mask of the dynamic target region within the current image frame through the improved lightweight segmentation network; S3. Extract ORB feature points from the non-dynamic mask region, filter the static feature subset based on the static mask and remove abnormal feature points, and estimate the camera pose by minimizing the reprojection error of the static feature points. S4. Based on the improved LSD algorithm and LBD descriptor, extract line features in the static region, and jointly optimize the static feature point reprojection error and line reprojection distance to perform pose estimation. S5. Key frames are selected using an adaptive weight-based key frame strategy, new map points are generated, BA optimization is performed on the local map point cloud, and redundant key frames and noise points are deleted. S6. Utilize visual and semantic features between keyframes for loop closure matching. After verifying loop closure, establish pose constraints and perform global graph optimization. During the optimization process, update the semantic attributes of map points and lines to achieve relocalization and semantic mapping. Figure 1 Consistent maintenance.

[0049] In step S1 of this embodiment, the system reads the original image (monocular, binocular, or RGB-D) from the camera, first performs distortion correction using camera calibration parameters, and converts the image to grayscale format; then, it constructs a multi-layer image pyramid to adapt to different scale requirements, and extracts uniformly distributed ORB feature points and descriptors on each layer. These processed image data are sent to the tracking module for subsequent pose estimation and map construction.

[0050] In step S2 of this embodiment, as Figure 3 As shown, firstly, multiple Bottlenecks in the multi-scale feature fusion module C2f of the YOLOv8s-seg model are replaced with DSConv, and then the C2f module in the neck structure is replaced with the lightweight LiteC2f module. DSConv primarily reduces computational complexity through depthwise separable convolutions.

[0051] In the formula, This represents the final output of the improved model. This represents the balanced convolution method. This represents the standard convolution operation. This indicates a channel shuffling operation. This represents depthwise separable convolution. Represents the original feature map. This represents the original input feature map. The first one obtained after channel splitting Each branch feature map The total number of branches after the input feature map is split; The semantic module processes the input image frames through the improved network. Perform semantic segmentation to obtain each pixel semantic category tags With instance mask Define the set of potential dynamic categories as follows: Then a static region mask can be constructed:

[0052] In step S3 of this embodiment, the tracking module marks regions belonging to the potential dynamic category as dynamic regions, and obtains a set of feature points after ORB feature extraction. A subset of static features is selected based on the static mask:

[0053] After ORB feature extraction, feature points falling into dynamic regions are removed, retaining only features on the static background for matching and map construction. The tracking thread relies solely on static points during feature matching and pose estimation to avoid errors introduced by dynamic objects. During the tracking phase, camera pose... This is obtained by minimizing the reprojection error of static feature points:

[0054] in, For camera projection model, For the corresponding 3D map points, For robust kernel functions; For rotation matrix, It is a translation vector. It is a special Euclidean group.

[0055] In step S4 of this embodiment, a static mask generated by semantic segmentation is used. Only non-dynamic regions such as the background are retained. Then, under the constraint of this mask, an improved LSD algorithm is applied to the input image to extract line features. Unlike the original LSD, the improved version introduces a local adaptive gradient threshold and a gradient weighting direction consistency criterion to improve detection accuracy and reduce computational cost. First, the gradient magnitude is calculated for the input image:

[0056] In the formula, and These represent the brightness gradients of the image in the horizontal and vertical directions, respectively. Then, in each local window Calculate the mean and standard deviation of the gradient within the inner range:

[0057]

[0058] In the formula, Represents a local window Any pixel in the array; This represents the mean of the gradient. This represents the standard deviation of each gradient; Therefore, the edge threshold is adaptively determined:

[0059] In the formula, Indicates adaptive weights; if Then the pixels Consider them as candidate edge points; Subsequently, directional clustering is performed on the candidate edge point set; for each group of points to be connected... Its gradient direction is , amplitude Define the gradient-weighted average direction:

[0060] And calculate the directional consistency score:

[0061] Define the consistency score threshold as ,when Furthermore, the orthogonality error from the point to the fitted line is below the threshold. When the point set is accepted as a line segment; Next, only samples with a static region percentage exceeding a threshold are retained. The candidate line, i.e. ;in, This represents the first uniform sampling of a candidate line segment. The pixel coordinates of each point This represents the total number of sampling points; Then, directional clustering is performed on the candidate edge point set. For each group of points to be connected:

[0062] in, and These represent the two endpoints of the line segment. The system filters out a subset of static line segments based on the static mask:

[0063] Only candidate lines whose endpoints and midpoints are mostly located within the static region are retained. Then, for each static line segment... Use LBD to calculate its local descriptor. This enables cross-frame line feature matching. During the pose estimation stage, the system simultaneously utilizes the static point feature set. With static line feature set Construct a joint optimization objective function. Camera pose. Solve by minimizing the reprojection error between points and lines:

[0064] in, For the observed image line segments, This represents the reprojection distance of the line. Ultimately, the camera pose is obtained by fusing the static point reprojection error and the line reprojection distance for joint optimization, achieving stable and accurate positioning and mapping in dynamic environments.

[0065] In step S5 of this embodiment, keyframe filtering no longer relies solely on the number of matching point features and pose changes, but comprehensively considers the joint constraint information of static points and lines. For example... Figure 4 As shown, the tracking module first determines whether there is new disparity information based on the pose changes between the current frame and the reference keyframe:

[0066] If the angle difference between the camera translation vector of the current frame t and the reference keyframe k, or the angle difference between the two pose rotations in the form of Lie algebras, exceeds the threshold, it indicates that the viewpoint has changed significantly and a new keyframe is needed to cover the new area.

[0067] Then calculate the number of validly matched static point features in the current frame. Sum of line characteristics The observation richness of the current frame is evaluated using an adaptive weighting method:

[0068] The system determines whether a new keyframe needs to be inserted by measuring the degree of preservation of static geometric information between the current frame and the reference keyframe. The system also counts the number of successfully matched static point features in the current frame. Sum of line characteristics And assign weights to them respectively. , This reflects their contribution to the geometric constraints. The weighted sum of the two is then compared with the total static features in the reference keyframe. In comparison, the matching retention rate was obtained. .when Below the set threshold If this occurs, it indicates that the available static features under the current viewpoint have significantly decreased, or that new uncovered areas have appeared. Based on this, the system determines that a keyframe should be inserted to supplement new point and line constraints and maintain the integrity and robustness of the map.

[0069] In step S5 of this embodiment, when the keyframe Once identified, the system will extract the static point and line features from them and compare them with their adjacent keyframes. Pairing is performed using matching features. For matched static point pairs... Using the pose of two cameras , Perform triangulation to calculate the coordinates of a 3D point:

[0070] This method obtains three-dimensional points in a static background. This is then added to the local map. For points falling on line features or matches generated from corresponding line segments in two frames, triangulation is performed using line endpoint constraints to obtain 3D line segments. All newly generated map points and line segments will be stored along with their corresponding keyframe observation relationships for subsequent local BA optimization and map building.

[0071] In step S5 of this embodiment, when a new keyframe... After adding a local map, the system selects a local sub-map from its shared keyframes, including the current keyframe, adjacent keyframes, and their shared map points and lines. Then, it calculates the poses of these keyframes. With map points line segments The goal of local BA optimization is to minimize the reprojection error of all static points and lines.

[0072] Through Business Analytics (BA) optimization, both camera pose and map structure are jointly adjusted to minimize overall error. After optimization, the system enters the map maintenance phase. (1) Remove map points and line features that have been observed too few times or have too large reprojection errors to prevent noise points or dynamic residues from affecting the accuracy of positioning.

[0073] (2) Redundancy detection and deletion are performed on keyframes with highly overlapping co-view relationships and redundant observation information to maintain map sparsity. This ensures mapping accuracy while reducing the computational burden of subsequent optimization.

[0074] In step S6 of this embodiment, while the system's local mapping is ongoing, the loop closure detection module continuously monitors the current keyframe. Is it related to historical keyframes? This creates a loop. First, the system measures the visual similarity between two frames using a bag-of-words model. Each keyframe consists of its set of feature descriptors. Represented as a bag-of-words vector Where M is the size of the visual dictionary. The similarity score between two frames is defined as:

[0075] in , These are the bag-of-words weights for the current keyframe and the candidate keyframes, respectively. If similarity... Exceeding the set threshold If a frame is identified as a potential loop closure candidate, it is then identified. Subsequently, the system performs geometric consistency verification between the candidate keyframes. Similarity transformation between cameras is calculated by matching static point and line features between the two frames. This transformation is obtained by minimizing the reprojection error estimate:

[0076] in For pixel observations in the current frame, These are map points for candidate keyframes. If the optimized reprojection error is below the threshold and the number of matches is sufficient, the loop closure is confirmed. After successful detection, the system adds a new loop closure constraint edge to the global pose graph:

[0077] This error term is then incorporated into the global map optimization objective. Next, in the subsequent global map optimization (BA), loop closure constraints are used to eliminate accumulated drift, achieving a globally consistent map structure. The global BA optimization module performs global optimization on the fused map. This stage involves adjusting the poses of all keyframes. With map points line segments As an optimization variable, minimize the global reprojection error function:

[0078] This ensures that all local sub-maps are spatially consistent and drift-free. After optimization, the system updates the poses of keyframes, the 3D positions of points and lines, and their co-view relationships in the global map, thereby obtaining a globally consistent map structure without cumulative errors, achieving closed-loop correction and long-term stable mapping.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dynamic SLAM method based on lightweight YOLOv8 and adaptive key frame strategy, characterized in that: The method includes the following steps: S1. Acquire consecutive raw image frames with known timestamps through the camera to provide spatiotemporally synchronized visual input for feature extraction, matching, and pose estimation; S2. Set up the LiteC2f module in the neck network of the YOLOv8 model to build an improved lightweight YOLOv8s-seg model, and use the lightweight model to extract the mask of the dynamic target region within the current image frame. S3. Extract ORB feature points from the non-dynamic mask region, filter the static feature subset based on the static mask and remove abnormal feature points, and estimate the camera pose by minimizing the reprojection error of the static feature points. S4. Based on the improved LSD algorithm and LBD descriptor, extract line features in the static region, and jointly optimize the static feature point reprojection error and line reprojection distance to perform pose estimation. S5. Keyframes are selected through an adaptive weighted keyframe strategy, new map points are generated, BA optimization is performed on the local map point cloud, and redundant keyframes and noise points are deleted. S6. Use visual and semantic features between keyframes to perform loop closure matching, verify the loop closure, establish pose constraints and optimize the global graph; update the semantic attributes of map points and lines during the optimization process to achieve relocalization and semantic map consistency maintenance.

2. The dynamic SLAM method based on lightweight YOLOv8 and adaptive key frame strategy according to claim 1, characterized in that: The system reads the original image from the camera, performs distortion correction using camera calibration parameters, and converts the image to grayscale format. Then, it constructs a multi-layer image pyramid to accommodate different scale requirements and extracts uniformly distributed ORB feature points and descriptors on each layer. The processed image data is then fed into the tracking module for subsequent pose estimation and map construction.

3. The dynamic SLAM method based on lightweight YOLOv8 and adaptive key frame strategy according to claim 1, characterized in that: In step S2, the depthwise separable convolution module DSConv is used in the LiteC2f module to reduce computational complexity, resulting in the output of the improved lightweight YOLOv8s-seg model: In the formula, This represents the final output of the improved model. This represents the balanced convolution method. This represents the standard convolution operation. This indicates a channel shuffling operation. This represents depthwise separable convolution. Represents the original feature map. This represents the original input feature map. The first one obtained after channel splitting Each branch feature map The total number of branches after the input feature map is split; Then, the lightweight YOLOv8s-seg model is used to process the input image frames. Perform semantic segmentation to obtain each pixel Semantic category labels and instance masks Define the potential dynamic class set as The static region mask is then represented as: In the formula, Represents pixels Semantic category labels.

4. The dynamic SLAM method based on lightweight YOLOv8 and adaptive keyframe strategy according to claim 3, characterized in that: In step S3, a set of feature points is obtained through ORB feature extraction. Regions belonging to the potential dynamic category are marked as dynamic regions, and then a subset of static features is selected based on the static mask: Based on static feature subsets Camera pose is estimated by minimizing the reprojection error of static feature points. : wherein is a camera projection model, is a corresponding three-dimensional map point, is a robust kernel function; is a rotation matrix, is a translation vector, is a special Euclidean group.

5. The dynamic SLAM method based on lightweight YOLOv8 and adaptive key frame strategy according to claim 4, characterized in that: In step S4, the process of extracting line features from the static region based on the improved LSD algorithm and LBD descriptor, and then performing camera pose estimation, includes: First, calculate the gradient magnitude of the input image: wherein and respectively represent the luminance gradient of the image in the horizontal and vertical directions; Then, the mean and standard deviation of the gradient is calculated within each local window ​ wherein represents any one pixel point in the local window represents any one pixel point in the local window represents the mean of the gradients, represents the standard deviation of each gradient; Therefore, the edge threshold is adaptively determined: wherein represents an adaptive weight; if then the pixel is considered as a candidate edge point; Subsequently, directional clustering is performed on the candidate edge point set; for each group of points to be connected... Its gradient direction is , amplitude Define the gradient-weighted average direction: And calculate the directional consistency score: Define the consistency score threshold as ,when Furthermore, the orthogonality error from the point to the fitted line is below the threshold. When the point set is accepted as a line segment; Next, only samples with a static region percentage exceeding a threshold are retained. The candidate line, i.e. ;in, This represents the first uniform sampling of a candidate line segment. The pixel coordinates of each point This represents the total number of sampling points; Perform directional clustering on the candidate edge set. For each group of points to be connected: wherein and respectively denote the two end points of a line segment; Then, a subset of static line segments is selected based on the static mask: For each static line segment Use LBD to calculate its local descriptor. And utilize static point feature set With static line feature set Construct a joint optimization objective function, and solve the camera pose again by minimizing the reprojection error between points and lines: in, For the observed image line segments, The reprojection distance of the line is represented; the final camera pose is jointly optimized by fusing the static point reprojection error and the line reprojection distance, and used for subsequent localization and mapping.

6. The dynamic SLAM method based on lightweight YOLOv8 and adaptive key frame strategy according to claim 5, characterized in that: In step S5, it is first determined whether a keyframe needs to be inserted. The determination method is as follows: First, determine whether there is new disparity information based on the pose changes between the current frame and the reference keyframe: If the current frame and reference keyframe The camera translation vector exceeds the translation threshold. Or two postures rotating and The angle difference in the Lie algebraic form exceeds the angle threshold. This indicates a significant change in perspective, requiring new keyframes to cover the new area; Then the number of valid matched static point features in the current frame is calculated and the number of matched line features The observation richness of the current frame is evaluated in an adaptive weighted manner: in, , These correspond to the number of static point features that were successfully matched in the current frame. and static line characteristic number The weight, These are the number of static point features and the number of static line features in the reference keyframe, respectively. Indicates the match retention rate, when Below the set threshold If the condition is met, then determine that a keyframe should be inserted; and determine the keyframe to be inserted as follows: .

7. The dynamic SLAM method based on lightweight YOLOv8 and adaptive keyframe strategy according to claim 6, characterized in that: In step S5, based on the selected keyframes The process of generating new map points is as follows: First, static point-line features are extracted from key frames and paired with matching features in adjacent key frames ​ Then, using the pose of the two cameras... and Based on the matched static point pairs Perform triangulation and calculate the coordinates of the three-dimensional point: to extract three-dimensional points in the static background and add them to the local map; For the points falling on the line features or the matches generated by the corresponding line segments of two frames, triangulation is performed by the line end point constraint to obtain the three-dimensional line segments ; all the newly generated map points and line segments are stored together with the observation relationship of the corresponding key frames, which are used for subsequent local BA optimization and map construction.

8. The dynamic SLAM method based on lightweight YOLOv8 and adaptive keyframe strategy according to claim 7, characterized in that: In step S5, the process of performing BA optimization on the local map point cloud is as follows: Add keyframes to the new local map A local sub-map is taken from the shared keyframes, including the current keyframe, adjacent keyframes, and the map points and lines they share; where shared keyframes refer to all keyframes that observe the same set of map points; Then, the pose of the co-view keyframes With map points line segments The goal of local BA optimization is to minimize the reprojection error of all static points and lines. After optimization, maintain the map as follows: Remove map point and line features that have fewer observations than a preset value or whose reprojection error is greater than a preset value; Redundancy detection and deletion are performed on keyframes with highly overlapping co-view relationships and redundant observation information to maintain map sparsity.

9. A dynamic SLAM method based on lightweight YOLOv8 and an adaptive keyframe strategy according to claim 8, characterized in that: In step S6, during the ongoing local mapping process, the current keyframe is continuously monitored. Is it related to historical keyframes? Forming loops: Similarity calculation based on the bag-of-words model measures the visual similarity between two frames. Each keyframe is composed of its set of feature descriptors. Represented as a bag-of-words vector Where M is the size of the visual dictionary, and the similarity score between two frames is defined as: in , These are the bag-of-words weights of the current keyframe and the candidate keyframes; if similarity Exceeding the set threshold If so, it is determined to be a potential loop candidate; Perform geometric consistency verification between candidate keyframes: calculate the similarity transformation between cameras by matching static point and line features between the two frames. It is obtained by minimizing the reprojection error estimate: in, For pixel observations in the current frame, The map points are candidate keyframes; if the optimized reprojection error is below the threshold and the number of matches is sufficient, then the loop closure is confirmed. After successful detection, a new loop closure constraint edge is added to the global pose graph, and this error term is included in the global graph optimization objective. Then, in the subsequent global BA, loop closure constraints are used to eliminate cumulative drift and achieve a globally consistent map structure: global optimization is performed on the fused map, adjusting the poses of all keyframes. With map points line segments As an optimization variable, minimizing the global reprojection error function completes the global optimization: After global optimization, all local sub-maps are spatially consistent and drift-free. After optimization, the poses of keyframes, the 3D positions of points and lines, and the co-view relationships in the global map are updated to obtain a globally consistent map structure without cumulative errors, thus achieving closed-loop correction and long-term stable mapping.

10. A system for executing the dynamic SLAM method based on lightweight YOLOv8 and adaptive keyframe strategy as described in any one of claims 1-9, characterized in that: The system includes a semantic module, a tracking module, a local mapping module, a loop closure detection module, and a global optimization module. The semantic module processes the input image frames through an improved lightweight YOLOv8s-seg network. Perform semantic segmentation to obtain the semantic category label and instance mask for each pixel; The tracking module uses a lightweight network to detect dynamic regions within an image frame, combines an epipolar geometry-constrained motion consistency detection method to remove extracted dynamic feature points, and extracts line features from static regions based on an improved LSD algorithm to compensate for the loss of feature points. The local mapping module uses an adaptive weight-based keyframe selection strategy to remove redundant keyframes and combines static point features and static line features to construct a joint optimization objective function for pose estimation and point cloud map construction. The loop closure detection module uses visual and semantic features between keyframes to perform loop closure matching. After verifying the loop closure, the global optimization module establishes pose constraints and performs global graph optimization.

Images