Dynamic environment VSLAM method based on semantic segmentation network and depth change analysis

Abstract

The present invention relates to a dynamic environment VSLAM method based on semantic segmentation network and depth change analysis. The method is based on ORB-SLAM3, introduces a semantic segmentation network and a dynamic target detection module, first processes the input RGB image and depth image, and divides the targets in the scene into dynamic targets, static targets and semi-static targets; then, the feature points on the target are screened, the feature points of the dynamic target are eliminated, and the feature points of the static target are retained. For the semi-static target, a time series depth change analysis strategy is used to build a deep time series model and extract dynamic features, and the dynamic probability score is calculated to eliminate the dynamic feature points. The screened feature points are entered into the SLAM process to realize tracking, local mapping, closed-loop detection, map fusion and global optimization, thereby obtaining a trajectory map corresponding to the RGB image. The method can effectively solve the problem of dynamic target interference and improve the robustness and accuracy of the VSLAM system in complex dynamic environments.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and robot navigation, and more specifically, to a method for simultaneous visual localization and mapping (VSLAM) in dynamic environments, particularly a method combining a lightweight semantic segmentation network and deep change analysis technology to improve the robustness and positioning accuracy of VSLAM in dynamic environments. Background Art

[0002] Visual Simultaneous Localization and Mapping (VSLAM) technology has become a hot topic of research in fields such as autonomous robot navigation, unmanned vehicles, and augmented reality (AR), due to its ability to achieve real-time positioning and mapping in unknown environments. However, traditional VSLAM systems mostly assume a static environment, an assumption that is difficult to maintain in practical applications. Moving objects in dynamic environments can interfere with the extraction and matching of feature points, thereby affecting the system's pose estimation and map accuracy, and even causing system failure. Achieving high robustness and high accuracy in dynamic environments has become one of the core challenges in this field.

[0003] Objects in a dynamic environment can be divided into dynamic objects (such as pedestrians and vehicles), semi-static objects (such as objects on the table and temporary structures) and static objects (such as buildings and roads) according to their dynamic characteristics. For objects with different characteristics, existing VSLAM methods mainly directly identify dynamic objects and eliminate their feature points through target detection or semantic segmentation. However, these methods have limitations when processing semi-static objects. For example, a stationary desktop object may be transformed into a dynamic object due to environmental changes (such as being picked up by a person), thereby interfering with the stability of the system. In addition, existing methods generally ignore the dynamic change characteristics of depth information in the process of processing dynamic targets, resulting in insufficient recognition accuracy of dynamic targets.

[0004] To address the interference of dynamic objects, existing research has mostly used semantic segmentation, optical flow analysis, or robust feature matching to detect and remove dynamic objects. However, these methods often suffer from two shortcomings: First, some algorithms rely on complex models or computationally expensive global optimization strategies, making it difficult to meet real-time requirements; second, for pseudo-static or semi-static objects, existing methods lack effective dynamic judgment strategies, which can easily lead to misjudgment of feature points, thus affecting system performance. Therefore, a VSLAM algorithm that can detect dynamic objects in a dynamic environment in real time and accurately identify semi-static objects is urgently needed. Summary of the Invention

[0005] In response to the above-mentioned prior art, the present invention provides a dynamic environment VSLAM method based on semantic segmentation network and depth change analysis to improve the robustness and accuracy in dynamic environments.

[0006] In order to solve the above technical problems, the present invention proposes a dynamic environment VSLAM method based on semantic segmentation network and depth change analysis. The dynamic environment VSLAM system adopted includes ORB-SLAM3, and the ORB-SLAM3 includes a front-end tracking module, a local mapping module, a loop detection module and a global optimization module; a semantic segmentation network and a dynamic target detection module are introduced at the front end of the tracking module; the semantic segmentation network extracts information from the input image, fuses multi-level features and performs final target segmentation; the semantic segmentation network includes a feature extraction module, a feature fusion module and a semantic segmentation module; the feature extraction module includes several convolution blocks Conv and cross convolution blocks Cross-block, and the cross convolution block Cross-block is based on Cross convolution design; the feature fusion module The block includes several conventional convolution blocks and feature fusion blocks FFB; the semantic segmentation module includes a convolution layer, an upsampling layer and a receptive field enhancement block RFEB, and the receptive field enhancement block RFEB includes five parallel branches, two of which use conventional convolution operations, two branches use void convolution operations, and the other branch uses pooling operations to capture global context information; the segmented targets are divided into dynamic targets, semi-static targets and static targets; the dynamic target refers to the target that moves during the observation period, the static target refers to the target that does not move during the observation period, and the semi-static target refers to the target that changes from static to dynamic during the observation period; the dynamic target detection module filters the feature points on the target; the ORB-SLAM3 performs feature point optimization processing on the filtered feature points, and finally generates a trajectory map corresponding to the RGB image.

[0007] The dynamic environment VSLAM method comprises the following steps:

[0008] Step 1, data input processing and preprocessing: The dynamic environment VSLAM system obtains RGB images and depth images from the RGB-D camera and performs the following preprocessing: First, the RGB image is Gaussian filtered to remove noise; then, the RGB image and the depth image are time synchronized to ensure that the RGB image and the depth image are consistent in time stamp; the preprocessed RGB image is input to the semantic segmentation network to classify the objects in the scene and distinguish between dynamic objects, semi-static objects and static objects; the preprocessed depth image is input to the dynamic object detection module to determine whether the object is a dynamic object;

[0009] Step 2: Segment the image into different semantic categories, including: the feature extraction module extracts deep and shallow semantic information from the input RGB image, and captures local and global features in the image through a combination of convolution blocks Conv and cross-convolution blocks Cross-block, thereby obtaining semantic information of the target object; the feature fusion module uses a cross-layer feature fusion method to perform multi-scale feature fusion on features from different network levels, and learns shallow detail features and deep semantic information in the features; the fused features are used as input to the semantic segmentation module, and the semantic segmentation module assigns each pixel in the image to a corresponding semantic category through convolution and upsampling operations; the receptive field enhancement block RFEB is used to capture global context information, and the spatial resolution of the image is restored through upsampling operations, and finally the image is segmented into different semantic categories;

[0010] Step 3: Differentiate the segmented targets into dynamic targets, semi-static targets, and static targets;

[0011] Step 4: screening the feature points on the distinguished targets, including: for the dynamic target, removing the feature points related to dynamics from the target, retaining other feature points, and recording them as feature points A; for the static target, retaining all feature points in the target, and recording them as feature points B; screening the semi-static target according to the dynamic probability score result, including: the dynamic target detection module performs deep feature modeling on the semi-static target, and determines the state of the semi-static target in the N-frame time series according to the dynamic probability score result. If the target is a dynamic target, the feature points on the target related to dynamics are removed; if the target is a static target, the feature points on the target are retained, and recorded as feature points C;

[0012] Step 5: Optimize the feature points to obtain a trajectory map: The ORB-SLAM3 system tracks feature points A, B, and C, performs local mapping, loop closure detection, map fusion, and global optimization, and finally generates a trajectory map corresponding to the RGB image.

[0013] Furthermore, the dynamic environment VSLAM method described in the present invention, wherein: in step 4, the specific process of screening the semi-static target according to the dynamic probability score result is: calculating the depth value of the semi-static target of N consecutive frames to obtain the depth change rate of the N-frame time series, and then calculating the depth acceleration; dividing the semi-static target of each frame into local grids, calculating the depth mean and standard deviation of each grid to characterize the depth fluctuation degree of a single feature point in each grid; calculating the dynamic probability score of a single feature point based on the depth fluctuation degree of the single feature point, the depth change rate and depth acceleration of the N-frame time series; performing weighted averaging on the dynamic scores of all feature points to obtain the dynamic probability score of the semi-static target, and judging the state of the semi-static target in the N-frame time series based on the dynamic probability score.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] The method of the present invention uses RGB images and depth images as input, classifies scene targets through a lightweight semantic segmentation network, and divides the targets into dynamic targets, static targets, and semi-static targets. For dynamic targets, relevant feature points are removed to reduce interference with pose estimation; for static targets, their feature points are retained for use in VSLAM subsequent processes; for semi-static targets, a dynamic target detection strategy based on time series depth changes is used to build a depth change model and extract dynamic features (including depth change rate, acceleration, and consistency), calculate the dynamic probability score, and determine the target dynamics by comparing with the threshold and remove dynamic feature points. After removing the dynamic feature points, the remaining data is input into the standard process of ORB-SLAM3 to complete operations such as tracking, local mapping, closed-loop detection, map fusion, and global optimization. This method improves the dynamic target detection capability through the designed semantic segmentation network, and proposes a time series depth change analysis strategy to solve the interference of pseudo-static or semi-static targets on the SLAM system, thereby achieving higher positioning and mapping accuracy in a dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention is a dynamic environment VSLAM method based on semantic segmentation network and depth change analysis;

[0017] Figure 2 Schematic diagram of the semantic segmentation network structure in the present invention;

[0018] Figure 3 yes Figure 2 The schematic diagram of the Cross block structure is shown in ;

[0019] Figure 4 yes Figure 2 The schematic diagram of the FFB module structure is shown in FIG;

[0020] Figure 5 yes Figure 2 Schematic diagram of RFEB module structure shown in;

[0021] Figure 6 This is a diagram showing the effect of the method of the present invention on dynamic target detection on the KITTI dataset;

[0022] Figure 7 This is the effect diagram of removing dynamic feature points on the TUM dataset by the method of the present invention;

[0023] Figures 8-1 to 8-6 This is a comparison chart of the trajectories of ORBSLAM3 and the method of the present invention, wherein:

[0024] Figure 8-1 is the trajectory on the s_xyz sequence, Figure 8-2 is the trajectory on the w_static sequence, Figure 8-3 is the relative trajectory error (translation) on s_xyz, Figure 8-4 is the relative trajectory error (translation) on w_static, Figure 8-5 is the relative trajectory error (rotation) on s_xyz, Figure 8-6 is the relative trajectory error (rotation) on w_static;

[0025] Figures 9-1 to 9-6 This is the trajectory diagram of the method of the present invention in the KITTI dataset (6 sequences); where:

[0026] Figure 9-1 It is the KITTI-00 sequence experiment result, Figure 9-2 It is the result of KITTI-02 sequence experiment. Figure 9-3 It is the result of KITTI-03 sequence experiment. Figure 9-4 It is the result of KITTI-04 sequence experiment. Figure 9-5 It is the result of KITTI-05 sequence experiment. Figure 9-6 These are the results of KITTI-08 sequence experiments. DETAILED DESCRIPTION

[0027] The present invention proposes a dynamic environment VSLAM method based on a semantic segmentation network and deep change analysis. Its design concept is as follows: ORB-SLAM3 is one of the current mainstream VSLAM systems, mainly including core modules such as front-end tracking, local mapping, loop detection, and global optimization. However, this system assumes that the environment is static by default. In dynamic environments, it is easily disturbed by dynamic targets, leading to problems such as mismatching, pose drift, and map instability. To address these problems, the present invention designs an improved dynamic environment VSLAM. A semantic segmentation network and a dynamic target detection module are designed at the front end of the ORB-SLAM3 tracking module. The semantic segmentation network consists of three parts: a feature extraction module, a feature fusion module, and a semantic segmentation module. The semantic segmentation network is used to segment targets and distinguish between dynamic targets, semi-static targets, and static targets. Dynamic targets refer to objects that move or change rapidly in a short period of time; semi-static targets refer to objects that are usually stationary but may move under specific circumstances; and static targets refer to objects that do not move during observation. The dynamic target detection module determines the dynamic nature of semi-static targets by modeling their deep characteristics. If the target is dynamic, its related feature points are removed; if the target is static, its feature points are retained for subsequent SLAM processing. Tracking, local mapping, loop closure detection, map merging, and global BA optimization are performed based on the filtered feature points.

[0028] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention in any way.

[0029] The dynamic environment VSLAM method of the present invention adopts a dynamic environment VSLAM system, such as Figure 1 As shown, the system includes ORB-SLAM3, which includes a front-end tracking module, a local mapping module, a loop detection module and a global optimization module; a semantic segmentation network and a dynamic target detection module are introduced at the front end of the tracking module.

[0030] The semantic segmentation network extracts information from the input image, fuses multi-level features, and performs final object segmentation.

[0031] The semantic segmentation network contains the following modules:

[0032] Feature extraction module: includes several convolution blocks Conv and cross convolution blocks Cross-block, and the cross convolution block Cross-block is designed based on cross convolution.

[0033] Feature fusion module: includes several conventional convolution blocks and feature fusion blocks FFB.

[0034] Semantic segmentation module: includes convolutional layers, upsampling layers and receptive field enhancement blocks RFEB (Receptive Field Enhancement Block). The receptive field enhancement block RFEB consists of five parallel branches, two of which use conventional convolution operations, two branches use void convolution operations, and the other branch uses pooling operations to capture global context information.

[0035] The semantic segmentation module divides the segmented targets into dynamic targets, semi-static targets and static targets; in the present invention, the dynamic target refers to the target that moves during the observation period, the static target refers to the target that does not move during the observation period, and the semi-static target refers to the target that changes from static to dynamic during the observation period; then, the dynamic target detection module screens the feature points on the target; in the present invention, the dynamic target detection strategy mainly includes: constructing a depth change model of the feature points based on the time series; extracting the depth change rate, acceleration and consistency features, and calculating the dynamic probability score; comparing the dynamic probability score with the preset threshold to judge the dynamic nature of the target; eliminating the feature points with a dynamic probability score higher than the threshold, and retaining the feature points with a dynamic probability score lower than the threshold.

[0036] The ORB-SLAM3 performs feature point optimization processing on the selected feature points, and finally generates a trajectory map corresponding to the RGB image.

[0037] The dynamic environment VSLAM method of the present invention comprises the following steps:

[0038] S1. Based on the existing ORB-SLAM3 framework, the present invention designs a semantic segmentation network and a dynamic target detection module to form a VSLAM system suitable for dynamic environments. The semantic segmentation network is responsible for classifying the targets in the scene, while the dynamic target detection module combines depth change analysis to eliminate dynamic targets, thereby enhancing the robustness and positioning accuracy of the system in dynamic scenes. Figure 1 shown.

[0039] S2. Data Input Processing and Preprocessing: RGB images and depth images are acquired from the RGB-D camera as input data for the dynamic environment VSLAM system. Preprocessing this input data ensures input quality for subsequent processes and lays the foundation for target classification and feature point extraction. First, the RGB image is Gaussian filtered for denoising. Then, the RGB image and depth image are time-synchronized to ensure that their timestamps are consistent. The preprocessed RGB image is input into the semantic segmentation network to classify objects in the scene, distinguishing between dynamic, semi-static, and static objects. The preprocessed depth image is input into the dynamic object detection module to determine whether the object is dynamic.

[0040] S3, target state classification processing, using the semantic segmentation network to perform semantic segmentation on the input RGB image, and the segmented targets are divided into dynamic targets, semi-static targets and static targets according to the dynamic characteristics of the targets. In order to achieve efficient semantic segmentation in real-time applications, the present invention designs a semantic segmentation algorithm. Figure 2 As shown,

[0041] The feature extraction module is responsible for extracting rich semantic information from the input image. The feature extraction module extracts deep and shallow semantic information from the input RGB image, and captures local and global features in the image through the combination of convolution blocks Conv and cross-convolution blocks Cross-block, thereby obtaining the semantic information of the target object while keeping the network lightweight. Figure 3 As shown. The Cross Block module is designed based on the Cross convolution. The horizontal and vertical gradient information is explored through two parallel asymmetric convolutions to focus on edge information mining. It has fewer parameters and computational complexity and the same receptive field. Compared with the traditional feature extraction network, the feature extraction branch in the present invention includes two parallel asymmetric convolutions, which are used to extract edge information in the horizontal and vertical directions, respectively, to reduce computational complexity and improve the lightweight performance of the network. While fully extracting the semantic information of the image, it can maintain low computational complexity and parameter quantity. What is important is that it has better structure retention capabilities and can be plugged in and used in the network. The formula for asymmetric convolution is as follows:

[0042]

[0043] In formula (1), k 1×m Represents a convolution kernel with a column (vertical) size of 1×m. m×1 Represents a convolution kernel with a column (vertical) size of m×1, is the input feature map. Represents the convolution operation. b represents the bias term.

[0044] The feature fusion module fuses features from different levels to obtain a more comprehensive and accurate semantic representation, thereby improving the accuracy and comprehensiveness of semantic segmentation. Shallow feature maps contain low-level features such as edges and textures, while deep feature maps contain stronger semantic information but have lower spatial resolution. The feature fusion module uses a cross-layer feature fusion method to fuse features from different network levels at multiple scales, learning shallow detail features and deep semantic information in the features. The feature fusion module FFB performs a concatenation operation on feature maps from different levels to form a combined feature map. The structure diagram is shown in the figure. Figure 4 shown.

[0045] The feature fusion block (FFB) designed in this paper combines feature maps at different levels, fusing shallow physical features with deep contextual features, which can achieve multi-level and multi-angle semantic understanding of the image, thereby improving the performance of the model in tasks such as semantic segmentation. The FFB superimposes the two partial feature maps through the concatenation method and then calculates the weighted features, which plays a role in feature selection and combination. The weighted calculation selects and combines shallow physical features and deep contextual features to achieve feature selection and optimization. The calculation is as follows:

[0046] Y=X+(σ(Conv 1×1 (SiLU(Conv 1×1 (Pool 1×1 (X)))))⊙Conv k×k (X)) (2)

[0047] In formula (2), Pool 1×1 (X) represents the global pooling operation. Conv 1×1 Represents the convolution operation, SiLU is the activation function, σ is the Sigmoid function, and ⊙ is the element-by-element multiplication.

[0048] Finally, the semantic segmentation module uses the fused features as the input of the semantic segmentation module, and the semantic segmentation module assigns each pixel in the image to the corresponding semantic category through convolution and upsampling operations. In the present invention, RFEB is used to expand the receptive field and improve the nonlinear ability to enhance the accuracy of semantic segmentation. The structure diagram is shown in FIG. Figure 5 As shown in Figure 2, the receptive field enhancement block (RFEB) is used to capture global contextual information and restore the spatial resolution of the image through upsampling. Finally, the image is segmented into different semantic categories. The RFEB module consists of five branches: two regular convolution branches for capturing local features; two dilated convolution branches for capturing multi-scale information, with a dilated convolution kernel size of 3×3; and a pooling branch for processing global information. Each branch uses the SiLU activation function to increase nonlinearity, and the outputs of all branches are fused into a comprehensive feature map through a splicing operation. After feature extraction and fusion, a lightweight segmentation head is used to achieve semantic segmentation of the image. This head combines feature information at different scales and performs fine segmentation of the image in a cascaded manner, ensuring accurate classification of dynamic objects and providing accurate feature point information for subsequent steps. The RFEB module aims to achieve better nonlinearity, expand the receptive field, and multi-scale fusion. RFEB improves the distinguishability of features through nonlinear operations such as dilated convolution, making the feature representations between different semantic categories more distinct and clear. Enhancing the receptive field at the entrance of the segmentation head enables it to capture a wider range of contextual information. The calculation is as follows:

[0049]

[0050] In formula (3), AtrousConv 3×3,d＝3 is the dilated convolution, d is the dilation coefficient, and Comcat is the concatenation operation, which is performed along the channel direction.

[0051] In the present invention, the feature extraction module, feature fusion module and semantic segmentation module work together to form the overall framework of the semantic segmentation network, aiming to achieve efficient, accurate and fast semantic segmentation.

[0052] The objects in the scene are divided into three categories through the semantic segmentation network:

[0053] (1) Dynamic targets: such as pedestrians and vehicles, will significantly affect the positioning accuracy of the VSLAM system.

[0054] (2) Static targets: such as buildings, roads, etc., which serve as the main reference objects of the VSLAM system.

[0055] (3) Semi-static targets: targets whose states may change, whose dynamics need to be further determined.

[0056] The semantic segmentation network improves the accuracy and efficiency of semantic segmentation through asymmetric convolution and multi-scale feature fusion, ensuring the accuracy of target classification.

[0057] S4. Process the differentiated target, that is, filter the feature points on the target.

[0058] Dynamic target processing: Based on the target classification results, dynamic targets (such as pedestrians and vehicles) in the image are identified. Feature points associated with these targets are removed to prevent them from interfering with pose estimation. This improves the SLAM system's adaptability to dynamic environments. The remaining feature points, denoted as feature points A, are retained and used as input for subsequent processing by the SLAM system.

[0059] Static target processing: For static targets (such as buildings and roads), feature points associated with the static targets are retained and recorded as feature points B. They serve as reliable input for subsequent processing of the SLAM system. Static target feature points will be used in key steps such as pose estimation, mapping, and loop closure detection, providing stable reference information.

[0060] Semi-static target processing: For semi-static targets that may have dynamic characteristics, an innovative method based on time series deep change analysis is proposed. The semi-static targets are screened based on the dynamic probability scoring results, including: the dynamic target detection module performs deep characteristic modeling on the semi-static target, and judges the state of the semi-static target in the N-frame time series based on the dynamic probability scoring results. If the target is a dynamic target, the dynamic-related feature points on the target are removed. If the target is a static target, the feature points on the target are retained and recorded as feature points C, which serve as reliable input for subsequent processing of the SLAM system. The specific steps are as follows:

[0061] (1) Depth change rate and acceleration calculation: By analyzing the depth change rate and acceleration in the time series, dynamic features are extracted. The depth value of the semi-static target of N consecutive frames is calculated to obtain the depth change rate of the N-frame time series, and then the depth acceleration is calculated. The depth change rate R of each feature point is calculated. d and depth acceleration A d :

[0062]

[0063] By jointly considering R d and A d , which can detect sudden changes in depth values ​​more sensitively. Setting an abnormal threshold τ for acceleration a If A d >τ a , it is determined to be a dynamic feature point.

[0064] (2) Spatial consistency detection: Divide the target area into local grids and calculate the mean μ of the depth value of the feature points in each grid. d and standard deviation σ d , used to characterize the depth fluctuation of a single feature point in each grid and detect significant fluctuation areas. d Significant fluctuations (exceeding the set threshold τ s ), the region is considered to contain dynamic feature points.

[0065] (3) Dynamic probability scoring, including local scoring and global scoring.

[0066] Local score: Define the dynamic score P of the feature point dyn The weighted sum of its depth change rate and acceleration:

[0067] P dyn =α·R d +β·A d (5)

[0068] Among them, α and β are adjustable weight parameters used to control the contribution of rate of change and acceleration to the score.

[0069] Global score: The dynamic scores of all feature points in the semi-static object area are weighted averaged to obtain the regional dynamic score S dyn :

[0070]

[0071] Among them, w i is the weight of the feature point, which is usually set based on the depth consistency of its adjacent points. dyn >τ s , the entire object area is determined to be a dynamic target, and all feature points in the area are removed.

[0072] The dynamic probability score of the feature points is calculated by combining the depth change rate and acceleration, and the dynamic score of the target area is obtained by weighted averaging. The dynamic probability score of each feature point is calculated based on the depth fluctuation of the individual feature point, the depth change rate, and the depth acceleration of the N-frame time series. The dynamic probability score of all feature points is weighted averaged to obtain the dynamic probability score of the semi-static target. The state of the semi-static target in the N-frame time series is determined based on this dynamic probability score.

[0073] S5. Optimize the feature points to obtain the trajectory map

[0074] The data after target classification and dynamic detection and removal of dynamic feature points (including the feature points A, B, and C) are input into the ORB-SLAM3 standard processing flow to complete the SLAM processing steps such as feature point tracking, local mapping, closed-loop detection, map fusion, and global optimization, and finally generate a trajectory map corresponding to the RGB image.

[0075] The present invention conducts a series of experiments based on the KITTI dataset and the TUM dataset to verify the effectiveness and superiority of the method. The experiments include dynamic target detection on the KITTI dataset to evaluate the detection effect of the present invention. Figure 6 As shown in Figure 2; dynamic feature point removal experiments were conducted on the TUM dataset to verify the improvement of trajectory estimation by removing dynamic features, as shown in Figure 2. Figure 7 As shown in Figure 2, the trajectory estimation performance of ORB-SLAM3 and the method of the present invention on the KITTI dataset is compared, highlighting the accuracy advantage of the method in dynamic scenes. Figures 8-1 to 8-6 Finally, a comprehensive experiment was conducted on trajectory estimation of 6 sequences of KITTI dataset, demonstrating the stability and robustness of the proposed method. Figures 9-1 to 9-6Experimental results show that the present invention has significant advantages in dynamic target detection, feature point removal and trajectory estimation, providing a reliable technical guarantee for visual SLAM applications in dynamic scenes.

[0076] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many improvements and changes without departing from the purpose of the present invention, which are all protected by the present invention.

Claims

1. A dynamic environment VSLAM method based on semantic segmentation network and depth change analysis, characterized in that: The dynamic environment VSLAM system used includes ORB-SLAM3, which includes a front-end tracking module, a local mapping module, a loop detection module, and a global optimization module; a semantic segmentation network and a dynamic target detection module are introduced at the front end of the tracking module; The semantic segmentation network extracts information from the input image, fuses multi-level features and performs final target segmentation; the semantic segmentation network includes a feature extraction module, a feature fusion module and a semantic segmentation module; The feature extraction module includes several convolution blocks Conv and cross convolution blocks Cross-block, and the cross convolution block Cross-block is designed based on Cross convolution; The feature fusion module includes several conventional convolution blocks and feature fusion blocks FFB; The semantic segmentation module includes a convolutional layer, an upsampling layer and a receptive field enhancement block RFEB. The receptive field enhancement block RFEB includes five parallel branches, two of which use conventional convolution operations, two branches use void convolution operations, and the other branch uses pooling operations to capture global context information. The segmented targets are classified into dynamic targets, semi-static targets, and static targets; dynamic targets refer to targets that move during the observation period, static targets refer to targets that do not move during the observation period, and semi-static targets refer to targets that change from static to dynamic during the observation period; then, the dynamic target detection module screens feature points on the targets; The ORB-SLAM3 performs feature point optimization processing on the selected feature points, and finally generates a trajectory map corresponding to the RGB image.

2. The dynamic environment VSLAM method according to claim 1, wherein The following steps are involved: Step 1: Data input processing and preprocessing: The dynamic environment VSLAM system obtains RGB images and depth images from the RGB-D camera and performs the following preprocessing: first, Gaussian filtering is performed on the RGB image to remove noise; then, the RGB image and the depth image are time-synchronized to ensure that the RGB image and the depth image are consistent in time stamp; The pre-processed RGB image is input into the semantic segmentation network to classify the objects in the scene and distinguish between dynamic objects, semi-static objects and static objects; Inputting the preprocessed depth image into the dynamic target detection module to determine whether the target is a dynamic object; Step 2: Segment the image into different semantic categories, including: The feature extraction module extracts deep and shallow semantic information from the input RGB image, and captures local and global features in the image through a combination of convolution blocks Conv and cross-convolution blocks Cross-block, thereby obtaining semantic information of the target object; The feature fusion module uses a cross-layer feature fusion method to fuse features from different network layers at multiple scales, learning shallow detail features and deep semantic information in the features; The fused features serve as input to the semantic segmentation module, which assigns each pixel in the image to a corresponding semantic category through convolution and upsampling operations. The receptive field enhancement block (RFEB) is used to capture global context information, and the spatial resolution of the image is restored through upsampling operations. Finally, the image is segmented into different semantic categories. Step 3: Differentiate the segmented targets into dynamic targets, semi-static targets, and static targets; Step 4: Screen the feature points on the distinguished target, including: 4-1) For the dynamic target, the feature points related to the dynamics are eliminated, and the remaining feature points are retained and recorded as feature points A; 4-2) retaining all feature points of the static target and recording them as feature points B; 4-3) screening the semi-static target based on the dynamic probability scoring result, including: the dynamic target detection module performs deep feature modeling on the semi-static target, and determines the state of the semi-static target in the N-frame time series based on the dynamic probability scoring result. If the target is a dynamic target, the dynamic-related feature points on the target are eliminated; if the target is a static target, the feature points on the target are retained and recorded as feature points C; Step 5: Optimize the feature points to obtain the trajectory map: The ORB-SLAM3 system tracks feature points A, B, and C, performs local mapping, loop closure detection, map fusion, and global optimization, and ultimately generates a trajectory map corresponding to the RGB image.

3. Dynamic environment VSLAM method according to claim 1, is characterized in that, The specific process of step 4-3) is: 4-3-1) Calculate the depth value of the semi-static target for N consecutive frames to obtain the depth change rate of the N-frame time series, and then calculate the depth acceleration; 4-3-2) Divide the semi-static target in each frame into local grids and calculate the depth mean and standard deviation of each grid to characterize the depth fluctuation of a single feature point in each grid; 4-3-3) Calculating a dynamic probability score for a single feature point based on the depth fluctuation degree of the single feature point, the depth change rate and depth acceleration of the N-frame time series; 4-3-4) Perform a weighted average of the dynamic scores of all feature points to obtain a dynamic probability score of the semi-static target, and judge the state of the semi-static target in the N-frame time series based on the dynamic probability score.

Images