YOLOv8-based point-line fusion visual SLAM (Simultaneous Localization and Mapping) method in indoor dynamic scene

Abstract

The invention relates to the technical field of visual SLAM (Simultaneous Localization and Mapping), in particular to a YOLOv8-based point-line fusion visual SLAM method in an indoor dynamic scene, which comprises the following steps of: respectively transmitting an RGB (Red, Green, Blue) image and a depth image acquired by an RGB-D (Red, Green and Blue) camera to a front-end tracking thread and a dynamic object detection thread, extracting ORB (Object Request) feature point and line features, and obtaining a visual SLAM result; meanwhile, the dynamic object detection thread adopts YOLOv8-Seg to complete instance segmentation of the dynamic object, a dynamic mask is generated, ORB point features and line features falling in the dynamic mask are judged as dynamic features, and then the dynamic features are removed; then, multi-view geometric constraint is adopted to further filter and segment missing dynamic feature points outside the mask; after the dynamic point-line features are removed, key frames are finally output and transmitted to a mapping thread, and positioning and mapping are carried out. According to the invention, the positioning precision of the visual SLAM system can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of visual SLAM technology, and in particular to a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene. Background Art

[0002] In today's era of rapid technological development, Simultaneous Localization and Mapping (SLAM) technology plays a vital role in many fields such as robot navigation, augmented reality, and unmanned driving.

[0003] The visual SLAM algorithms in the existing technology are all based on static environments or objects moving at extremely slow speeds, and work well in static and rigid scenes; however, in crowded indoor shopping mall environments, traditional visual SLAM methods may produce large positioning errors due to the movement of many people, and cannot accurately build maps. Therefore, in dynamic environments, the visual SLAM system is easily affected by moving objects, resulting in incorrect data matching.

[0004] With the rise of deep learning technology, some researchers have combined deep learning with SLAM problems in dynamic environments, and relying on semantic information to process dynamic objects has become a new option.

[0005] However, for indoor dynamic environments, only deep learning methods are used to reduce the impact of dynamic objects on positioning accuracy, and the rich line segments in indoor dynamic environments are not utilized. Compared with point features, line features have better continuity and stability, and can better deal with problems such as motion blur and occlusion in the environment.

[0006] In order to effectively overcome the shortcomings of the existing technology and improve the positioning accuracy of the visual SLAM algorithm and the robustness of the SLAM system in indoor dynamic environments, a point-line fusion visual SLAM method based on YOLOv8 in indoor dynamic scenes was proposed. Summary of the invention

[0007] The purpose of the present invention is to provide a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene. The RGB image and depth image obtained by the RGB-D camera are respectively transmitted to the front-end tracking thread and the dynamic object detection thread, and the ORB feature points and line features are extracted. At the same time, the dynamic object detection thread uses YOLOv8-Seg to complete the instance segmentation of the dynamic object, and generates a dynamic mask. The ORB point features and line features falling in the dynamic mask are determined as dynamic features and then removed; then the multi-view geometric constraints are used to further filter the dynamic feature points missing outside the segmentation mask; after removing the dynamic point and line features, the key frame is finally output and passed to the mapping thread for positioning and mapping. The present invention can effectively improve the positioning accuracy of the visual SLAM system.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A point-line fusion visual SLAM method based on YOLOv8 in indoor dynamic scenes, comprising:

[0010] Acquire RGB images and depth images of indoor scenes through RGB-D cameras;

[0011] Extracting first point features of the RGB image and the depth image;

[0012] The first feature point is an ORB feature point, and the first feature point is extracted by using an Oriented Fast algorithm;

[0013] Extracting the first line features of the RGB image and the second line features of the depth image according to the EDLines algorithm, superimposing the second line features on the first line features and the RGB image to perform line feature detection, obtaining third line features, and generating a point-line RGB image including the third line features and the first point features;

[0014] Furthermore, when the EDLines algorithm extracts the first line feature and the second line feature, it is necessary to extract multiple point features for line feature drawing. The point features include position information, scale information and direction information, and are represented by the Plücker representation method and the standard orthogonal method:

[0015] L=(U,W); U∈SO(3); W∈SO(2);

[0016] SO(n)={R∈R n×n |RR T =E,det(R)=1};

[0017]

[0018] Wherein, L is a three-dimensional space line established by the coordinate system established by the RGB-D camera, U and W are mapping equations from the Plücker coordinate system to the orthogonal coordinate system, SO(n) is a special orthogonal group containing a rotation matrix R of size n*n, R T represents the transpose of the rotation matrix R, E is the identity matrix, det is the determinant calculation, and is the location information, is the scale information, n is the normal vector of the coordinate system established by the RGB-D camera, v is the linear direction vector of the coordinate system established by the RGB-D camera, sin is the sine function, cos is the cosine function, θ is the direction information, is the point feature after being expressed by the standard orthogonal method;

[0019] The three main parameters of the EDLines algorithm are as follows: the gradient threshold α is set to 32, the anchor threshold β is set to 16, and the scanning interval γ is set to 4; the EDLines algorithm also includes: the line spacing is set to 15 pixels, if the distance between the endpoints of two lines is less than 10 pixels, the lines are merged; if the extension angle of two lines is less than 1°, the lines are merged;

[0020] The YOLOv8n-seg network is trained using the MS COCO dataset as a training sample, and the YOLOv8n-seg network with the highest training accuracy is extracted as a dynamic instance segmentation network; the RGB image is input into the YOLOv8n-seg network to generate a first mask map, and the first mask map is further processed in combination with prior semantics to generate a second mask map;

[0021] Furthermore, the YOLOv8n-seg network extracts the RGB image for segmentation, performs independent object target detection on the segmented image blocks to generate segmentation mask information; the segmentation mask is 116 parameters including 4 coordinate parameters, 80 category confidence parameters and 32 instance segmentation parameters; the 4 coordinate parameters are and θ;

[0022] The dynamic object information of the segmented image block is marked according to the segmentation mask information to generate a second mask map; the marking process is:

[0023] Distinguish the category of the dynamic object information in the segmented RGB image according to the prior semantics and the segmentation mask information, and use different colors to cover the pixels of the dynamic object in the segmented RGB image in combination with the corresponding segmentation mask information for the dynamic object information of different categories;

[0024] Inputting the point-line RGB image, the depth image and the second mask image into the SLAM system, removing the first point features and the third line features of the point-line RGB image that are judged to contain dynamic feature points in combination with the first mask image, and further filtering the first point features and the third line features of the point-line RGB image that are not marked as the dynamic feature points in the first mask image through multi-view geometry constraints and depth information of the depth image;

[0025] Further, obtain the adjacent frame I n and I n+1 , I n and I n+1 The first point feature and the third line feature are included;

[0026] According to I n and I n+1 Establish multi-view geometry, obtain I through the multi-view geometry n The first point feature on the line and the point on the third line feature in I n+1 The prediction on is the dynamic feature point and obtain The corresponding coordinates and depth information after mapping on the multi-view geometric figure; the mapping formula is:

[0027]

[0028] Among them, d2 is the expected observation The depth information of for The normalized coordinates of the expected observations, d1 is The depth information of R is the depth information of the multi-view geometry Mapping The mapping weight of x is The normalized coordinates of , t is the multi-view geometry Mapping Bias of

[0029] Get the multi-view geometry in I n+1 The first point feature on the line and the point on the third line feature and extracting the normalized coordinate x' and the depth information depth;

[0030] If |d2-depth| or When it is greater than the threshold, Considered as the dynamic feature points for filtering;

[0031] The threshold of |d2-depth| is 0.1;

[0032] Output key frames based on static feature points obtained from the first point feature and the third line feature of the point-line RGB image; and perform local mapping thread, local closed-loop detection thread and global BA optimization through the key frames.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention extracts line features in RGB images through the EDLines algorithm, and characterizes the extracted line features according to the Plücker representation assisted by the standard orthogonal representation, which makes up for the shortcomings of the labeled orthogonal representation that cannot represent coordinate changes and reprojection, so as to facilitate the subsequent screening and deletion of dynamic feature points; based on the pixel distance and angular range between the extracted line features, the line features with a closer distance are fused to make the extracted line features more obvious, and at the same time can reduce the false detection rate of dynamic feature points in the subsequent line features, and can more accurately obtain the line features of the indoor dynamic environment, so as to facilitate the high-precision positioning of the subsequent visual SLAM algorithm in the indoor dynamic environment. The inclusion of a large number of line features also helps to improve the robustness of the visual SLAM algorithm in high-precision positioning in different dynamic environments.

[0035] 2. The present invention segments RGB images by training the YOLOv8n-seg algorithm with the highest accuracy, identifies the object targets in the segmented image blocks, generates corresponding segmentation mask information, and obtains dynamic target object information; objects that have been determined as dynamic feature points are screened and marked based on semantic information and segmentation mask information; compared with conventional deep learning algorithms, the present invention retains a large number of line features while accurately reducing dynamic objects; the static feature information based on these line features can effectively improve the positioning accuracy and robustness of subsequent visual SLAM algorithms in indoor dynamic scenes.

[0036] 3. In order to further improve the positioning accuracy of the visual SLAM algorithm in indoor scenes and reduce the interference caused by dynamic objects, the present invention further screens the dynamic feature points in the point features and line features missed by the YOLOv8n-seg algorithm by establishing multi-perspective geometric figures, so as to further reduce the positioning interference of the visual SLAM algorithm. At the same time, the static feature points of a large number of line features can make the visual SLAM algorithm more accurately positioned. At the same time, a large number of line features can also ensure high-precision positioning calculation in different indoor dynamic scenes, thereby improving the robustness of the visual SLAM algorithm in different indoor scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0038] Figure 2 The RGB image obtained by the RGB-D camera of the present invention;

[0039] Figure 3 A depth map obtained by the RGB-D camera of the present invention;

[0040] Figure 4 It is the algorithm flow chart and effect diagram of YOLO8v8n-seg of the present invention;

[0041] Figure 5 This is a schematic diagram of the multi-view geometric constraints of RGB-D of the present invention;

[0042] Figure 6 It is a trajectory error comparison diagram of the visual SLAM algorithm of the present invention and the ORB-SLAM3 algorithm. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] At present, visual SLAM algorithms play an important role in various fields that require positioning and map construction. The difficulty of current visual SLAM algorithms lies in their poor adaptability to indoor scenes containing a large number of dynamic targets. The current conventional method is based on screening based on dynamic feature points, while indoor scenes contain line features with a large amount of information. These line features can effectively overcome the problems of motion blur and occlusion in indoor scenes. At the same time, the large amount of information contained in line features can also significantly improve the positioning accuracy of visual SLAM algorithms in indoor scenes. To this end, the present invention provides a point-line fusion visual SLAM method based on YOLOv8 in indoor dynamic scenes, and the technical solution is as follows:

[0045] Reference Figure 1 As shown, the RGB image and depth image of the indoor scene are obtained by using an RGB-D camera;

[0046] Extracting first point features from the RGB image and the depth image; extracting first line features from the RGB image and second line features from the depth image according to the EDLines algorithm, superimposing the second line features on the first line features and the RGB image to perform line feature detection, obtaining third line features, and generating a point-line RGB image including the third line features and the first point features;

[0047] The YOLOv8n-seg network is trained using the MS COCO dataset as a training sample, and the YOLOv8n-seg network with the highest training accuracy is extracted as a dynamic instance segmentation network; the RGB image is input into the YOLOv8n-seg network to generate a first mask map, and the first mask map is further processed in combination with prior semantics to generate a second mask map;

[0048] Inputting the point-line RGB image, the depth image and the second mask image into the SLAM system, removing the first point features and the third line features of the point-line RGB image that are judged to contain dynamic feature points in combination with the second mask image, and further filtering the first point features and the third line features of the point-line RGB image that are not marked as the dynamic feature points in the first mask image through multi-view geometry constraints and depth information of the depth image;

[0049] Output key frames based on static feature points obtained from the first point feature and the third line feature of the point-line RGB image; and perform local mapping thread, local closed-loop detection thread and global BA optimization through the key frames.

[0050] The present invention extracts point features from the RGB image and depth image acquired by the camera, and extracts line features in the image through EDLines, detects the line features, and screens out the line features that are misidentified; this method can improve the accuracy of line feature extraction in the image to ensure that more accurate line features are provided for subsequent work; retains the line features that contain a large amount of auxiliary information in the image, which is convenient for the positioning of the subsequent visual SLAM algorithm;

[0051] Based on the highly accurate YOLOv8n-seg after training, by screening and segmenting RGB images, combining semantic information and segmentation mask information to screen dynamic targets, this method can more accurately identify and screen dynamic targets while retaining the previously identified line features to the greatest extent. This method can effectively retain the static feature information of the line features in the image, providing a high-precision data feature foundation for the subsequent recognition of the visual SLAM algorithm;

[0052] In order to ensure the accuracy of dynamic target screening; the present invention establishes multi-view geometry and further screens the dynamic feature points of point features and line features to ensure that the dynamic feature points are reduced to the greatest extent and a larger proportion of static features are retained, which helps to minimize the impact of dynamic feature points on the visual SLAM algorithm. At the same time, more static feature points can retain more feature information, ensuring the positioning accuracy of the visual SLAM algorithm and its robustness in different dynamic indoor scenes.

[0053] Embodiment 1

[0054] For the purpose of specific explanation, the following embodiments are described in detail:

[0055] The RGB image and depth image of the indoor scene are obtained by using the RGB-D camera; the RGB image is referenced Figure 2 As shown, the depth image reference Figure 3 As shown;

[0056] Extract the first point features of RGB image and depth image;

[0057] The first feature point is an ORB feature point, which is extracted by the Oriented Fast algorithm. In the visual SLAM algorithm, since the line feature may ignore some dynamic targets, the positioning accuracy of the visual SLAM algorithm is reduced. Therefore, the point features in the RGB image are extracted while the line features are extracted to ensure that all dynamic feature points are extracted, which is convenient for the screening of dynamic targets, reduces the impact on the visual SLAM algorithm, and improves the positioning accuracy.

[0058] The first line features of the RGB image and the second line features of the depth image are extracted according to the EDLines algorithm, the second line features are superimposed on the first line features and the RGB image to perform line feature detection, the third line features are obtained, and a point-line RGB image including the third line features and the first point features is generated;

[0059] Furthermore, when the EDLines algorithm extracts the first line feature and the second line feature, it needs to extract multiple point features for line feature drawing. Point features include position information, scale information, and direction information. The first step of the edge drawing ED (Edge Drawing) algorithm is to calculate the image gradient. Calculate the gradient size and horizontal line angle at the pixel coordinate (x, y);

[0060]

[0061] Among them, g x (x,y) is the gradient change value on the x-axis, g y (x, y) is the gradient change value on the y-axis, x is the horizontal coordinate, y is the vertical coordinate, I() represents the pixel value of the corresponding point; g(x, y) is the gradient change value, θ(x, y) is the horizontal line angle, the range of θ is 360 degrees, and arctan is the inverse tangent function;

[0062] The position information is represented as (x, y), the scale information is represented as m, and the direction information is represented as θ;

[0063] In three-dimensional space, a line has four degrees of freedom. Common representations include Plücker representation and standard orthogonal representation;

[0064] The Plücker representation consists of 6 parameters, which are used for initialization, transformation and projection. A line in three-dimensional space can be represented as:

[0065] L=(n T ,v T ) T ;

[0066] Where v∈R 3 ,n∈R 3 ; T represents matrix transpose;

[0067] Since the standard orthogonal representation cannot represent coordinate transformation and reprojection well, these two line feature representation methods will be used; represented by Plücker representation and standard orthogonal method:

[0068] L=(U,W); U∈SO(3); W∈SO(2);

[0069] SO(n)={R∈R n×n |RR T =E,det(R)=1};

[0070]

[0071]

[0072] Where L is the three-dimensional space line established by the RGB-D camera coordinate system, U and W are the mapping equations from the Plücker coordinate system to the orthogonal coordinate system, SO(n) is a special orthogonal group containing the rotation matrix R of size n*n, R T represents the transpose of the rotation matrix R, R n*n represents the rotation matrix R of size n*n, E is the unit matrix, det is the determinant calculation, and is the location information, is the scale information, n is the normal vector of the coordinate system established by the RGB-D camera, v is the linear direction vector of the coordinate system established by the RGB-D camera, sin is the sine function, cos is the cosine function, θ is the direction information, is the point feature after being expressed by the standard orthogonal method;

[0073] After the calculation is completed, and Corresponding to x and y, Corresponding to m;

[0074] The EDLines algorithm is mainly divided into four steps for edge detection:

[0075] (1) Use a Gaussian filter to input a grayscale image to suppress noise and obtain a smooth image;

[0076] (2) Calculate the magnitude and direction of the gradient using the Sobel operator. If the gradient is greater than the gradient threshold, it is an edge pixel;

[0077] (3) Calculate a set of pixels with a very high probability of being edge pixels, called anchor points, which will generate an iterative anchor point. The anchor point corresponds to the pixel where the gradient calculation produces the maximum value, that is, the peak of the gradient map;

[0078] (4) Starting from the anchor point, connect the next anchor point to obtain the edge segment;

[0079] The method of line segment detection is as follows: assuming that the length of the line segment is n, the gradient angle of each pixel along the line segment and the number of pixels consistent with the direction of the line segment are calculated as k. For each line segment, an NFA value is calculated. If the NFA value is less than or equal to 1, the line segment is considered valid, otherwise it is discarded. The calculation formula of NFA is:

[0080]

[0081] Among them, N 4 is the number of potential line segments. Since the RGB image and the depth image are superimposed, the image sizes are N*N, so there are N after superposition. 4 points, p is the effective probability of the line segment;

[0082] Since line features contain a lot of information, whether it is a dynamic indoor scene or a static indoor scene, they are helpful for the positioning of the visual SLAM algorithm. Therefore, the line features in the RGB image and the depth image are extracted through the EDLines algorithm. In order to ensure the accuracy of the line features in the RGB image, the line features of the depth image are combined with the original RGB image to screen the line features of the RGB image to reduce the false detection rate of line feature extraction, which is helpful for the high-precision positioning of the subsequent visual SLAM algorithm. At the same time, high-accuracy line features can also improve the accuracy of the visual SLAM algorithm.

[0083] The three main parameters of the EDLines algorithm are the gradient threshold α set to 32, the anchor threshold β set to 16, and the scanning interval γ set to 4. Whether it is the gradient threshold or the anchor threshold, selecting larger data can effectively improve the accuracy of line features, thereby extracting lines that are more suitable for tracking by the visual SLAM algorithm, which helps to improve the accuracy and robustness of the subsequent visual SLAM algorithm.

[0084] The EDLines algorithm also includes setting the line spacing to 15 pixels. If the distance between the endpoints of two lines is less than 10 pixels, the lines are fused. In the actual extraction process, some line features that are too close may be included. These line features are too close, which makes it difficult for the visual SLAM algorithm to track them. It is also difficult to remove line features that may contain dynamic feature points. Therefore, by fusing the lines, the line distribution can be made more uniform, which improves the positioning accuracy of the subsequent visual SLAM algorithm.

[0085] If the extension angle of two lines is less than 1°, the lines are merged; this method can make the line distribution more uniform while ensuring that the line segments are not too dense, which helps the subsequent visual SLAM algorithm to achieve high-precision positioning in different scenes and improve robustness;

[0086] The MS COCO dataset is used as a training sample to train the YOLOv8n-seg network. MS COCO was released by Microsoft Research in 2014 and is one of the important benchmark datasets in the field of computer vision. The MS COCO dataset contains more than 330,000 images, covering 80 common object categories, such as people, animals, tables, chairs, computers, etc. It includes common dynamic objects in indoor environments;

[0087] YOLOv8n-seg combines object detection and semantic segmentation to achieve the location and pixel-level classification of objects in images; refer to Figure 4 As shown in the figure, its network structure mainly consists of three modules: Input, Backbone, and Head. In the Input module, the input image data is received and preprocessed, providing the correct format of input for subsequent network operations. The Backbone module is used to extract the features of the input image. It consists of convolutional layers, pooling layers, and residual connections, and can extract high-level features from the image. The training process is as follows:

[0088] (1) First, perform image classification training, using the default number of training rounds, batch size, and learning rate, i.e., 100 training rounds, 16 batch size, and 0.01 learning rate. Use the training dataset to train the first 20 convolutional layers, 1 pooling layer, and 1 fully connected layer of the YOLOv8n network;

[0089] (2) Use the pre-trained network parameters to initialize the first 20 convolutional layers of the YOLOv8n network, and then perform overall model training to improve image resolution;

[0090] (3) After the training is completed, the detection effect is tested. If it reaches a satisfactory level, it can be used to complete the semantic segmentation task of the SLAM system.

[0091] Extract the YOLOv8n-seg network with the highest training accuracy as the dynamic instance segmentation network; input the RGB image into the YOLOv8n-seg network to generate a first mask map, and then process the first mask map in combination with prior semantics to generate a second mask map;

[0092] Furthermore, the YOLOv8n-seg network extracts RGB images for segmentation, and performs independent object detection on the segmented image blocks to generate segmentation mask information; the segmentation mask is 116 parameters including 4 coordinate parameters, 80 category confidence parameters and 32 instance segmentation parameters; the 4 coordinate parameters are and θ;

[0093] This method can improve the accuracy of dynamic target recognition, locate dynamic target information more accurately by segmenting mask information, screen out dynamic feature points to the greatest extent, and retain point features and line features containing static feature information to the greatest extent, which is helpful for the high-precision positioning of subsequent visual SLAM algorithms, and improve the robustness of indoor scenes in different occasions through a large number of line features;

[0094] The dynamic object information of the segmented image block is marked according to the segmentation mask information to generate a second mask map; the marking is:

[0095] According to the prior semantics and segmentation mask information, the dynamic object information category in the segmented RGB image is distinguished. Different categories of dynamic object information are combined with the corresponding segmentation mask information, and different colors are used to cover the pixels of the dynamic objects in the segmented RGB image.

[0096] By marking the RGB images with different colors, it is possible to ensure that the dynamic feature targets in the RGB images are removed to the greatest extent during the algorithm implementation process, improve the accuracy of line feature tracking of the subsequent visual SLAM algorithm, and ensure the high positioning accuracy and robustness of the visual SLAM algorithm.

[0097] The point-line RGB image, the depth image and the second mask image are input into the SLAM system, and the first point features and the third line features of the point-line RGB image that are judged to contain dynamic feature points are eliminated in combination with the first mask image, and the first point features and the third line features that are not marked as dynamic feature points in the first mask image in the point-line RGB image are further filtered through multi-view geometric constraints and depth information of the depth image;

[0098] Further, obtain the adjacent frame I n and I n+1 , I n and I n+1 It includes the first point feature and the third line feature;

[0099] According to I n and In+1 Create multi-view geometry, and obtain I through multi-view geometry n The first point feature on the line and the point on the third line feature in I n+1 The prediction on is the dynamic feature point and obtain The corresponding coordinates and depth information after mapping on the multi-view geometry; the mapping formula is:

[0100]

[0101] Among them, d2 is the expected observation The depth information, for The normalized coordinates of the expected observations, d1 is The depth information of R is the depth information of multi-view geometry. Mapping The mapping weight of x is The normalized coordinates of , t is the multi-view geometry Mapping Bias of

[0102] Get Multiview Geometry in I n+1 The first point feature on the line and the point on the third line feature And extract the normalized coordinate x' and depth information depth;

[0103] If |d2-depth| or When it is greater than the threshold, Consider it as a dynamic feature point for filtering;

[0104] Reference Figure 5 As shown, O and O' are represented by and The optical center, e and e' are and The outermost point, l and l' are and The extreme outer line, P is and The intersection point after mapping, P' is and The intersection point after mapping;

[0105] Specifically, |d2-depth| and The judgment is based on the depth data and coordinate information of P and P' to determine the threshold; the depth data of P and P' is the height data of the distance plane;

[0106] In order to ensure that the dynamic feature points in the point features and line features are removed to the greatest extent during the RGB image screening process and to avoid the occurrence of missed dynamic feature points, the screened RGB images are further screened through multi-view geometry to further remove dynamic feature points and retain the point features and line features of static features to the greatest extent, thereby improving the positioning accuracy of the visual SLAM algorithm and ensuring the robustness of the algorithm in different scenarios;

[0107] The threshold of |d2-depth| is 4, and the threshold of |d2-depth| is 0.1; when these thresholds are large, dynamic feature points are difficult to remove, and when these thresholds are small, static feature points are filtered out. Therefore, based on a large number of experiments and expert experience, this embodiment sets the threshold to this data to maximize the screening of dynamic feature points, retain static feature points, and improve the positioning accuracy and robustness of the SLAM algorithm;

[0108] The key frames are output based on the static feature points obtained from the first point feature and the third line feature of the point-line RGB image; the local mapping thread, the local closed-loop detection thread and the global BA optimization are performed through the key frames.

[0109] Embodiment 2

[0110] In order to evaluate the performance of the algorithm, this embodiment uses the RGB-D data set of the Technical University of Munich for experiments. The data set contains high-dynamic scenes and low-dynamic scenes, so that the Kinect RGB-D camera records information at a rate of 30Hz. In order to quantitatively evaluate the trajectory accuracy of the visual SLAM system, this embodiment uses ATE (Absolute Trajectory Error) and to evaluate the accuracy of SLAM. ATE represents the difference between the true value and the calculated value of the camera pose, reflecting the accuracy of the algorithm and the consistency of the global trajectory. The experiment is mainly conducted on six sequences in the above data set, two sitting sequences and four walking sequences. The two sitting sequences are low-dynamic scenes, and the four walking sequences are high-dynamic scenes. For comparison, this study selected three indicators: calculating the root mean square error (RMSE), the mean error (Mean), and the median error (Median). RMSE can reflect the difference between the true value and the observed value, and can fully reflect the performance of the SLAM system.

[0111] In the experiment of this embodiment, the software environment is Ubuntu 18.04, and the platform used is a self-assembled desktop computer, configured with Intel(R) Core(TM) i5-12490F CPU@3.00GHz, 16G running memory, NVIDIA Ge-Force RTX3060Ti GPU, and 8GB video memory. The SLAM system is written in C++, compiled by cmake project, and the instance segmentation network is written in Python, using C++ to call Python interface to implement its functions.

[0112] Table 1. Comparison of absolute trajectory errors

[0113]

[0114] According to the data in Table 1, the performance of this algorithm is better than ORB-SLAM3 in sequences with moving objects and large internal motion amplitudes in the image. In the six selected image sequences, ATE is reduced by an average of 75.99%, and in the four high-dynamic scenes, ATE is reduced by an average of 94.61%. For low-dynamic scenes, PLYO-SLAM also has a 43.26% improvement. Compared with the ORB-SLAM3 algorithm, the improved algorithm uses YOLOv8n-seg to detect and remove feature point lines in dynamic environments. After removing the influence of these moving objects on the SLAM system's attitude estimation, the accuracy of camera attitude estimation in high-dynamic scenes is greatly improved, and the robustness is enhanced. In low-dynamic scenes, the improved algorithm is slightly improved compared to ORB-SLAM3. Analysis of image sequences shows that in low-dynamic scenes, the movement of people in the scene is less, and the impact on the SLAM system is less.

[0115] Figure 6 The trajectory error comparison diagram of this embodiment and ORB-SLAM3 is shown. It can be seen that the estimated trajectory of this embodiment is very close to the actual trajectory.

[0116] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A point-line fusion visual SLAM method based on YOLOv8 in indoor dynamic scenes. It is characterized in that include: Acquire RGB images and depth images of indoor scenes through RGB-D cameras; Extracting first point features of the RGB image and the depth image; Extracting the first line features of the RGB image and the second line features of the depth image according to the EDLines algorithm, superimposing the second line features on the first line features and the RGB image to perform line feature detection, obtaining third line features, and generating a point-line RGB image including the third line features and the first point features; The YOLOv8n-seg network is trained using the MS COCO dataset as a training sample, and the YOLOv8n-seg network with the highest training accuracy is extracted as a dynamic instance segmentation network; the RGB image is input into the YOLOv8n-seg network to generate a first mask map, and the first mask map is further processed in combination with prior semantics to generate a second mask map; Inputting the point-line RGB image, the depth image and the second mask image into the SLAM system, removing the first point features and the third line features of the point-line RGB image that are judged to contain dynamic feature points in combination with the second mask image, and further filtering the first point features and the third line features of the point-line RGB image that are not marked as the dynamic feature points in the first mask image through multi-view geometry constraints and depth information of the depth image; Output key frames based on static feature points obtained from the first point feature and the third line feature of the point-line RGB image; and perform local mapping thread, local closed-loop detection thread and global BA optimization through the key frames.

2. According to claim 1, a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene, It is characterized in that The first point feature is extracted by using an Oriented Fast algorithm, where the first point feature is an ORB feature point.

3. According to claim 1, a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene, It is characterized in that When the EDLines algorithm extracts the first line feature and the second line feature, it is necessary to extract multiple point features for line feature drawing. The point features include position information, scale information and direction information, and are represented by the Plücker representation method and the standard orthogonal method: L=(U,W); U∈SO(3); W∈SO(2); SO(n)6{R∈R n×n |RR T E,it(R)m1}. Wherein, L is a three-dimensional space line established by the coordinate system established by the RGB-D camera, U and W are mapping equations from the Plücker coordinate system to the orthogonal coordinate system, SO(n) is a special orthogonal group containing a rotation matrix R of size n*n, R T represents the transpose of the rotation matrix R, E is the identity matrix, det is the determinant calculation, and is the location information, is the scale information, n is the normal vector of the coordinate system established by the RGB-D camera, v is the linear direction vector of the coordinate system established by the RGB-D camera, sin is the sine function, cos is the cosine function, θ is the direction information, is the point feature after being expressed by the standard orthogonal method.

4. According to claim 1, a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene, It is characterized in that The three main parameters of the EDLines algorithm are the gradient threshold α set to 32, the anchor point threshold β set to 16, and the scanning interval γ set to 4.

5. According to claim 4, a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene, It is characterized in that The EDLines algorithm also includes setting the line spacing to 15 pixels, and merging the lines if the distance between the endpoints of two lines is less than 10 pixels.

6. According to claim 4, a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene, It is characterized in that The EDLines algorithm also includes fusing the lines if the extension angle of two lines is less than 1°.

7. According to claim 1, a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene, It is characterized in that The RGB image is input into the YOLOv8n-seg network to generate a first mask map, and the first mask map is processed again in combination with prior semantics to generate a second mask map, including: The YOLOv8n-seg network extracts the RGB image for segmentation, and performs independent dynamic object target detection on the segmented image blocks to generate segmentation mask information; the segmentation mask is 116 parameters including 4 coordinate parameters, 80 category confidence parameters and 32 instance segmentation parameters; the 4 coordinate parameters are and θ; The dynamic object information of the segmented image block is marked according to the segmentation mask information to generate a second mask map.

8. According to claim 7, a point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene, It is characterized in that The segmentation mask information generates a second mask image by marking the dynamic object information of the segmented image block with different colors, and the marking process is as follows: The dynamic object information category in the segmented RGB image is distinguished according to the prior semantics and the segmentation mask information, and different colors are used to cover the pixels of the dynamic object in the segmented RGB image in combination with the corresponding segmentation mask information for the dynamic object information of different categories.

9. The point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene according to claim 1, It is characterized in that Further filtering the point and line RGB image by multi-view geometric constraints and the depth information of the depth image, the first point features and the third line features not marked as the dynamic feature points in the first mask image include: Get adjacent frame I n and I n+1 , I n and I n+1 The first point feature and the third line feature are included; According to I n and I n+1 Establish multi-view geometry, obtain I through the multi-view geometry n The first point feature on the line and the point x on the third line feature In in I n+1 The prediction on is the dynamic feature point and get The corresponding coordinates and depth information after mapping on the multi-view geometric figure; the mapping formula is: Among them, d 2 Expected observation The depth information of for The normalized coordinates of the expected observation, d 1 for The depth information of R is the depth information of the multi-view geometry Mapping The mapping weight of x is The normalized coordinates of , t is the multi-view geometry Mapping Bias of Get the multi-view geometry in I n+1 The first point feature on the line and the point on the third line feature and extracting the normalized coordinate x' and the depth information depth; If |d 2 -depth|or When it is greater than the threshold, The dynamic feature points are regarded as the dynamic feature points for filtering.

10. The point-line fusion visual SLAM method based on YOLOv8 in an indoor dynamic scene according to claim 9, It is characterized in that The threshold is 4, |d 2 The threshold for -depth| is 0.1.

Images