Dynamic environment VSLAM method based on semantic segmentation network and depth change analysis

Abstract

The invention relates to a dynamic environment VSLAM method based on a semantic segmentation network and depth change analysis, and the method comprises the steps: introducing a semantic segmentation network and a dynamic target detection module on the basis of ORB-SLAM3, firstly processing an input RGB image and a depth image, and dividing targets in a scene into a dynamic target, a static target and a semi-static target; and then, screening feature points on the target, removing feature points of a dynamic target, retaining feature points of a static target, and for a semi-static target, constructing a depth time sequence model through a time sequence depth change analysis strategy, extracting dynamic features, and calculating a dynamic probability score to remove the dynamic feature points. And enabling the screened feature points to enter an SLAM (Simultaneous Localization and Mapping) process to realize tracking, local mapping, closed-loop detection, map fusion and global optimization so as to obtain a trajectory diagram corresponding to the RGB image. According to the method, the problem of dynamic target interference can be effectively solved, and the robustness and precision of the VSLAM system in a complex dynamic environment are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and robot navigation. Specifically, it relates to a visual simultaneous localization and mapping method applicable to dynamic environments, especially a method that combines a lightweight semantic segmentation network and depth change analysis technology to improve the robustness and positioning accuracy of VSLAM in dynamic environments. Background Art

[0002] In the fields of robot autonomous navigation, driverless vehicles, Augmented Reality (AR), etc., the Visual Simultaneous Localization and Mapping (VSLAM) technology has become a hot research topic because it can achieve real-time localization and map construction in unknown environments. However, most traditional VSLAM systems assume that the environment is static, and this assumption is difficult to hold in practical applications. Moving objects in dynamic environments will interfere with the extraction and matching of feature points, thereby affecting the pose estimation and map accuracy of the system, and even causing the system to fail. How to achieve high-robustness and high-precision VSLAM in dynamic environments has become one of the core challenges in this field.

[0003] Objects in dynamic environments can be classified into dynamic objects (such as pedestrians, vehicles), semi-static objects (such as items on a table, temporarily constructed structures), and static objects (such as buildings, roads) according to their dynamic characteristics. For these objects with different characteristics, existing VSLAM methods mainly directly identify dynamic objects through object detection or semantic segmentation and remove their feature points. However, these methods have limitations in dealing with semi-static objects. For example, stationary items on a table may turn into dynamic objects due to environmental changes (such as being picked up by someone), thus interfering with the system stability. In addition, existing methods usually ignore the dynamic change characteristics of depth information during the processing of dynamic targets, resulting in insufficient recognition accuracy for dynamic targets.

[0004] In response to the interference of dynamic targets, existing research mostly detects and removes dynamic targets through semantic segmentation, optical flow analysis, or robust feature matching. However, these methods often have two deficiencies: First, some algorithms rely on complex models or computationally expensive global optimization strategies and are difficult to meet the real-time requirements; Second, for pseudo-static or semi-static targets, existing methods lack effective dynamic judgment strategies, which easily lead to misjudgments in feature point processing, thereby affecting the system performance. Therefore, there is an urgent need for a VSLAM algorithm that can detect dynamic targets in real time in dynamic environments and accurately identify semi-static targets at the same time. Summary of the Invention

[0005] In view of the above-mentioned prior art, the present invention provides a dynamic environment VSLAM method based on semantic segmentation network and depth change analysis to improve the robustness and accuracy in a dynamic environment.

[0006] To solve the above technical problems, a dynamic environment VSLAM method based on semantic segmentation network and depth change analysis proposed by the present invention uses a dynamic environment VSLAM system including ORB-SLAM3. The ORB-SLAM3 includes a front-end tracking module, a local mapping module, a loop closing detection module, and a global optimization module. A semantic segmentation network and a dynamic object detection module are introduced at the front end of the tracking module. The semantic segmentation network extracts information from the input image, fuses multi-level features, and performs final object segmentation. The semantic segmentation network includes a feature extraction module, a feature fusion module, and a semantic segmentation module. The feature extraction module includes several convolutional blocks Conv and cross convolutional blocks Cross-block, and the cross convolutional block Cross-block is designed based on Cross convolution. The feature fusion module includes several conventional convolutional blocks and feature fusion blocks FFB. The semantic segmentation module includes a convolutional layer, an upsampling layer, and a receptive field enhancement block RFEB. The receptive field enhancement block RFEB includes five parallel branches, two of which use conventional convolution operations, two use dilated convolution operations, and the other uses pooling operations to capture global context information. The segmented objects are classified into dynamic objects, semi-static objects, and static objects. The dynamic object refers to an object that moves during the observation period, the static object refers to an object that does not move at all during the observation period, and the semi-static object refers to an object that changes from static to dynamic during the observation period. The dynamic object detection module screens the feature points on the object. The ORB-SLAM3 performs feature point optimization processing on the screened feature points and finally generates a trajectory map corresponding to the RGB image.

[0007] The dynamic environment VSLAM method includes the following steps:

[0008] Step 1: Data input processing and preprocessing: The dynamic environment VSLAM system obtains RGB images and depth images from an RGB-D camera and performs the following preprocessing: First, Gaussian filtering is performed on the RGB image to remove noise. Then, time synchronization is performed on the RGB image and the depth image to ensure that the RGB image and the depth image are consistent in timestamp. The preprocessed RGB image is input into the semantic segmentation network to classify the objects in the scene and distinguish dynamic objects, semi-static objects, and static objects. The preprocessed depth image is input into the dynamic object detection module to determine whether the object is a dynamic object.

[0009] Step 2: Segment the image into different semantic categories, including: The feature extraction module extracts deep and shallow semantic information from the input RGB image. Through the combination of convolutional blocks Conv and cross-convolutional blocks Cross-block, local and global features in the image are captured, thereby obtaining the semantic information of the target object. The feature fusion module uses the method of cross-layer feature fusion to perform multi-scale feature fusion on features from different network levels, learning the detailed features in the shallow layer and the semantic information in the deep layer of the features. The fused features are used as the input of the semantic segmentation module. The semantic segmentation module assigns each pixel in the image to the corresponding semantic category through convolution and upsampling operations. The receptive field enhancement block RFEB is used to capture global context information, and the spatial resolution of the image is restored through upsampling operations. Finally, the image is segmented into different semantic categories;

[0010] Step 3: Classify the segmented targets into the aforementioned dynamic targets, semi-static targets, and static targets;

[0011] Step 4: Screen the feature points on the above-mentioned classified targets, including: For the dynamic target, remove the feature points related to dynamics in the target and retain other feature points, denoted as feature point A; For the static target, retain all the feature points in the target, denoted as feature point B; For the semi-static target, screen according to the dynamic probability scoring result, including: The dynamic target detection module performs deep feature modeling on the semi-static target, and judges the state of the semi-static target in the N-frame time series according to the dynamic probability scoring result. If the target is a dynamic target, remove the feature points related to dynamics on the target. If the target is a static target, retain the feature points on the target, denoted as feature point C;

[0012] Step 5: Optimize the feature points to obtain a trajectory map: The ORB-SLAM3 system tracks, locally maps, performs loop detection, fuses maps, and globally optimizes feature point A, feature point B, and feature point C, and finally generates a trajectory map corresponding to the RGB image.

[0013] Furthermore, in the dynamic environment VSLAM method of the present invention, in step 4, the specific process of screening the semi-static target according to the dynamic probability scoring result is as follows: for the semi-static target in consecutive N frames, calculate the depth change rate of the N-frame time series by calculating the depth value, and then calculate the depth acceleration; divide the semi-static target of each frame into local grids, calculate the depth mean and standard deviation of each grid to characterize the depth fluctuation degree of a single feature point in each grid; according to the depth fluctuation degree of the single feature point, the depth change rate and depth acceleration of the N-frame time series, calculate the dynamic probability score of the single feature point; perform weighted average on the dynamic scores of all feature points to obtain the dynamic probability score of the semi-static target, and judge the state of the semi-static target in the N-frame time series according to this dynamic probability score.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] The method of the present invention takes RGB images and depth images as inputs, classifies scene targets through a lightweight semantic segmentation network, and divides the targets into dynamic targets, static targets, and semi-static targets. For dynamic targets, relevant feature points are removed to reduce interference with pose estimation; for static targets, their feature points are retained for subsequent VSLAM processes; for semi-static targets, a dynamic target detection strategy based on depth change in time series is used to construct a depth change model and extract dynamic features (including depth change rate, acceleration, and consistency), calculate the dynamic probability score, and determine the target dynamicity by comparing with a threshold and remove dynamic feature points. After removing the dynamic feature points, the remaining data is input into the standard process of ORB-SLAM3 to complete operations such as tracking, local mapping, loop detection, map fusion, and global optimization. This method improves the dynamic target detection ability through the designed semantic segmentation network and proposes a time series depth change analysis strategy to solve the interference of pseudo-static or semi-static targets to the SLAM system, thereby achieving higher positioning and mapping accuracy in a dynamic environment. Description of the Drawings

[0016] Figure 1 is the dynamic environment VSLAM method of the present invention based on semantic segmentation network and depth change analysis;

[0017] Figure 2 is the schematic diagram of the semantic segmentation network structure in the present invention;

[0018] Figure 3 is Figure 2 the schematic diagram of the Cross block structure shown in

[0019] Figure 4 is Figure 2 the schematic diagram of the FFB module structure shown in

[0020] Figure 5 is Figure 2 the schematic diagram of the RFEB module structure shown in

[0021] Figure 6 is the detection effect diagram of dynamic targets on the KITTI dataset by the method of the present invention;

[0022] Figure 7 is the effect diagram of removing dynamic feature points on the TUM dataset by the method of the present invention;

[0023] Figures 8-1 to 8-6 is the trajectory comparison diagram between ORBSLAM3 and the method of the present invention in the present invention, where:

[0024] Figure 8-1 is the trajectory on the s_xyz sequence, Figure 8-2 is the trajectory on the w_static sequence, Figure 8-3 is the relative trajectory error (translation) on s_xyz, Figure 8-4 is the relative trajectory error (translation) on w_static, Figure 8-5 is the relative trajectory error (rotation) on s_xyz, Figure 8-6 is the relative trajectory error (rotation) on w_static;

[0025] Figures 9-1 to 9-6 is the trajectory diagram of the method of the present invention on the KITTI dataset (6 sequences); where:

[0026] Figure 9-1 is the experimental result of the KITTI-00 sequence, Figure 9-2 is the experimental result of the KITTI-02 sequence, Figure 9-3 is the experimental result of the KITTI-03 sequence, Figure 9-4 is the experimental result of the KITTI-04 sequence, Figure 9-5 is the experimental result of the KITTI-05 sequence, Figure 9-6 is the experimental result of the KITTI-08 sequence. Detailed implementation manners

[0027] A dynamic environment VSLAM method based on semantic segmentation network and depth change analysis proposed by the present invention has the following design concept: ORB-SLAM3 is one of the current mainstream VSLAM systems, mainly including core modules such as front-end tracking, local mapping, loop detection, and global optimization. However, this system default assumes that the environment is static and is prone to interference from dynamic objects in a dynamic environment, resulting in problems such as incorrect matching, pose drift, and unstable maps. In view of these problems, the present invention designs an improved dynamic environment VSLAM. At the front end of the ORB-SLAM3 tracking module, a semantic segmentation network and a dynamic object detection module are designed. The semantic segmentation network includes three parts: a feature extraction module, a feature fusion module, and a semantic segmentation module. The target is segmented through the semantic segmentation network to distinguish dynamic objects, semi-static objects, and static objects. A dynamic object refers to an object that moves or changes rapidly in a short period of time. A semi-static object refers to an object that is usually stationary but may move under specific circumstances. A static object refers to an object that does not move at all during the observation period. The dynamic object detection module models the depth characteristics of semi-static objects to judge their dynamicity. If the object is a dynamic object, the feature points related to it are removed; if the object is a static object, its feature points are retained for subsequent SLAM processing. Tracking, local mapping, loop closure detection, map merging, and global BA optimization are performed based on the filtered feature points.

[0028] The following further describes the present invention with reference to the accompanying drawings and specific embodiments, but the following embodiments are by no means limiting to the present invention.

[0029] The dynamic environment VSLAM method of the present invention adopts a dynamic environment VSLAM system, as Figure 1 shown. The system includes ORB-SLAM3, and the ORB-SLAM3 includes a front-end tracking module, a local mapping module, a loop detection module, and a global optimization module; a semantic segmentation network and a dynamic object detection module are introduced at the front end of the tracking module.

[0030] The semantic segmentation network extracts information from the input image, fuses multi-level features, and performs final target segmentation.

[0031] The semantic segmentation network includes the following modules:

[0032] Feature extraction module: It includes several convolutional blocks Conv and cross convolutional blocks Cross-block, and the cross convolutional block Cross-block is designed based on Cross convolution.

[0033] Feature fusion module: It includes several conventional convolutional blocks and feature fusion blocks FFB.

[0034] Semantic segmentation module: It includes a convolutional layer, an upsampling layer, and a receptive field enhancement block RFEB (Receptive Field Enhancement Block). The receptive field enhancement block RFEB includes five parallel branches. Among them, two branches use conventional convolution operations, two branches use dilated convolution operations, and the other branch uses pooling operations to capture global context information.

[0035] The semantic segmentation module classifies the segmented targets into dynamic targets, semi-static targets, and static targets. In the present invention, the dynamic target refers to a target that moves during the observation period, the static target refers to a target that has no movement during the observation period, and the semi-static target refers to a target that changes from static to dynamic during the observation period. Then, the dynamic target detection module filters the feature points on the target. In the present invention, the dynamic target detection strategy mainly includes: constructing a depth change model of the feature points according to the time series; extracting the depth change rate, acceleration, and consistency features, and calculating the dynamic probability score; comparing the dynamic probability score with a preset threshold to judge the dynamicity of the target; removing the feature points with a dynamic probability score higher than the threshold, and retaining the feature points with a dynamic probability score lower than the threshold.

[0036] The ORB-SLAM3 performs feature point optimization processing on the selected feature points, and finally generates a trajectory map corresponding to the RGB image.

[0037] The dynamic environment VSLAM method of the present invention includes the following steps:

[0038] S1. Based on the existing ORB-SLAM3 framework, the present invention designs a semantic segmentation network and a dynamic target detection module to form a VSLAM system suitable for dynamic environments. The semantic segmentation network is responsible for classifying the targets in the scene, and the dynamic target detection module combines depth change analysis to remove dynamic targets, thereby enhancing the robustness and positioning accuracy of the system in dynamic scenes, as Figure 1 shown.

[0039] S2. Data input processing and preprocessing: Obtain the RGB image and the depth image from the RGB-D camera as the input data of the dynamic environment VSLAM system. Through the preprocessing of the input data, the input quality of the subsequent process is ensured, and a foundation is laid for target classification and feature point extraction. First, perform Gaussian filtering denoising on the RGB image; then, synchronize the time of the RGB image and the depth image to ensure that the RGB image and the depth image are consistent in timestamp. The preprocessed RGB image is input into the semantic segmentation network to classify the targets in the scene, distinguishing dynamic targets, semi-static targets, and static targets; the preprocessed depth image is input into the dynamic target detection module to judge whether the target is a dynamic object.

[0040] S3. Target state classification processing: Use a semantic segmentation network to perform semantic segmentation on the input RGB image, and divide the segmented targets into the dynamic targets, semi-static targets, and static targets according to the dynamic characteristics of the targets. In order to achieve efficient semantic segmentation in real-time applications, a semantic segmentation algorithm is designed in the present invention. As Figure 2 shown

[0041] The feature extraction module is responsible for extracting rich semantic information from the input image. The feature extraction module extracts deep and shallow semantic information from the input RGB image, and captures local and global features in the image through the combination of convolutional blocks Conv and cross-convolutional blocks Cross-block, so as to obtain the semantic information of the target object while keeping the network lightweight. The structure is as Figure 3 shown. Based on the Cross convolution, the Cross Block module is designed to explore the horizontal and vertical gradient information through two-way parallel asymmetric convolutions to focus on edge information mining, with fewer parameters and computational complexity and the same receptive field. Compared with traditional feature extraction networks, the feature extraction branch in the present invention includes two-way parallel asymmetric convolutions, which are respectively used to extract edge information in the horizontal and vertical directions to reduce computational complexity and improve the lightweight performance of the network. While fully extracting the semantic information of the image, it can also maintain a low computational complexity and the number of parameters. Importantly, it has better structure preservation ability and can be plugged in and used in the network at will. The formula for the asymmetric convolution is as follows:

[0042]

[0043] In formula (1), k 1×m represents a convolution kernel with a column direction (vertical) size of 1×m. k m×1 represents a convolution kernel with a column direction (vertical) size of m×1, is the input feature map. represents the convolution operation. b represents the bias term.

[0044] The feature fusion module fuses features from different levels to obtain a more comprehensive and accurate semantic representation, thereby improving the accuracy and comprehensiveness of semantic segmentation. The shallow feature map contains low-level features such as edges and textures, while the deep feature map contains stronger semantic information but lower spatial resolution. The feature fusion module uses the method of cross-layer feature fusion to perform multi-scale feature fusion on features from different network levels, and learns the detailed features in the shallow layer and the semantic information in the deep layer of the features. The feature fusion module FFB performs a Concate operation on the feature maps from different levels to form a combined feature map. The structure diagram is as Figure 4 shown.

[0045] The feature fusion block (FFB) designed in the present invention combines feature maps at different levels, fuses shallow physical features and deep context features, and can achieve multi-level and multi-angle semantic understanding of images, thereby improving the performance of the model in tasks such as semantic segmentation. The FFB superimposes two partial feature maps by the Concate method and then calculates the weighted features, which plays a role in feature selection and combination. By calculating the weights, it selects and combines shallow physical features and deep context features to achieve feature selection and optimization. The calculation is as follows:

[0046] Y = X+(σ(Conv 1×1 (SiLU(Conv 1×1 (Pool 1×1 (X)))))⊙Conv k×k (X)) (2)

[0047] In formula (2), Pool 1×1 (X) represents the global pooling operation. Conv 1×1 represents the convolution operation, SiLU is the activation function, σ is the Sigmoid function, and ⊙ is the element-wise multiplication.

[0048] Finally, the semantic segmentation module takes the fused features as the input of the semantic segmentation module. The semantic segmentation module assigns each pixel in the image to the corresponding semantic category through convolution and upsampling operations. In the present invention, the RFEB is used to expand the receptive field and improve the non-linear ability, enhancing the accuracy of semantic segmentation. The structural diagram is as Figure 5 shown. The receptive field enhancement block (RFEB) is used to capture global context information, and the spatial resolution of the image is restored through upsampling operations. Finally, the image is segmented into different semantic categories. The RFEB module includes the following five branches: two conventional convolution branches for capturing local features; two dilated convolution branches for capturing multi-scale information, and the size of the dilated convolution kernel is 3×3; one pooling branch for processing global information. Each branch uses the SiLU activation function to increase the non-linear ability, and the outputs of all branches are fused into a comprehensive feature map through a concatenation operation. After feature extraction and fusion, a lightweight segmentation head is used to achieve semantic segmentation of the image. The segmentation head combines feature information at different scales and performs fine segmentation of the image in a cascaded manner to ensure accurate classification of dynamic objects and provide accurate feature point information for subsequent steps. The RFEB module aims to achieve functions such as good non-linearity, expanded receptive field, and multi-scale fusion. The RFEB improves the distinguishability of features through non-linear operations such as dilated convolution, making the feature representations between different semantic categories more obvious and clear. The receptive field at the entrance of the enhanced segmentation head is enhanced so that it can capture a wider range of context information. The calculation is as follows:

[0049]

[0050] In Equation (3), AtrousConv 3×3,d＝3 is the atrous convolution, and d is the dilation coefficient. Comcat is the concatenation operation, which is performed along the channel direction.

[0051] In the present invention, the feature extraction module, the feature fusion module, and the semantic segmentation module cooperate with each other to jointly constitute the overall framework of the semantic segmentation network, aiming to achieve efficient, accurate, and fast semantic segmentation.

[0052] The targets in the scene are classified into three categories through the semantic segmentation network:

[0053] (1) Dynamic targets: such as pedestrians, vehicles, etc., which will significantly affect the positioning accuracy of the VSLAM system.

[0054] (2) Static targets: such as buildings, roads, etc., which serve as the main reference objects for the VSLAM system.

[0055] (3) Semi-static targets: such as targets whose states may change, and their dynamics need to be further determined.

[0056] The semantic segmentation network improves the accuracy and efficiency of semantic segmentation through asymmetric convolution and multi-scale feature fusion to ensure the accuracy of target classification.

[0057] S4. Process the distinguished targets, that is, screen the feature points on the targets.

[0058] Processing of dynamic targets: According to the target classification results, identify the dynamic targets (such as pedestrians and vehicles) in the image. For these targets, eliminate the feature points associated with the dynamic targets to avoid their interference with pose estimation. Improve the adaptability of the SLAM system to dynamic environments. Retain other feature points, denoted as feature point A, as the input for the subsequent processing of the SLAM system.

[0059] Processing of static targets: For static targets (such as buildings and roads), retain the feature points associated with the static targets, denoted as feature point B, as the reliable input for the subsequent processing of the SLAM system. The feature points of static targets will be used in key steps such as pose estimation, mapping, and loop detection to provide stable reference information.

[0060] Semi-static target processing. For semi-static targets that may have dynamic characteristics, an innovative method based on time-series depth change analysis is proposed. The semi-static targets are screened according to the dynamic probability scoring results, including: the dynamic target detection module models the depth characteristics of the semi-static targets, and judges the state of the semi-static targets in the N-frame time series according to the dynamic probability scoring results. If the target is a dynamic target, the feature points related to dynamics on the target are removed. If the target is a static target, the feature points on the target are retained, denoted as feature point C, as the reliable input for the subsequent processing of the SLAM system. The specific steps are as follows:

[0061] (1) Depth change rate and acceleration calculation: By analyzing the depth change rate and acceleration in the time series, dynamic characteristics are extracted. The depth values of the semi-static targets in N consecutive frames are calculated to obtain the depth change rate of the N-frame time series, and then the depth acceleration is calculated. Calculate the depth change rate R d and depth acceleration A d :

[0062]

[0063] By jointly considering R d and A d , the mutation of the depth value can be detected more sensitively. Set an abnormal threshold τ a for the acceleration. If A d > τ a , it is determined as a dynamic feature point.

[0064] (2) Spatial consistency detection: Divide the target area into local grids, and calculate the mean value μ d and standard deviation σ d of the depth values of the feature points in each grid to characterize the depth fluctuation degree of a single feature point in each grid and detect significant fluctuation areas. If σ d in the grid shows significant fluctuation (exceeding the set threshold τ s ), it is considered that this area contains dynamic feature points.

[0065] (3) Dynamic probability scoring, including local scoring and global scoring.

[0066] Local scoring: Define the dynamic scoring P dyn of the feature point as the weighted sum of its depth change rate and acceleration:

[0067] P dyn = α·R d + β·A d (5)

[0068] Among them, α and β are adjustable weight parameters used to control the contributions of the change rate and acceleration to the score.

[0069] Global score: The dynamic scores of all feature points within the semi-static object region are weighted and averaged to obtain the regional dynamic score S dyn :

[0070]

[0071] Among them, w i is the weight of the feature point, usually set based on the depth consistency of its adjacent points. If S dyn > τ s , then the entire object region is determined to be a dynamic target, and all feature points within the region are removed.

[0072] Calculate the dynamic probability score of the feature point by combining the depth change rate and acceleration, and obtain the dynamic score of the target region through weighted average. Calculate the dynamic probability score of a single feature point according to the depth fluctuation degree of the single feature point, the depth change rate and depth acceleration of the N-frame time series; perform weighted average on the dynamic scores of all feature points to obtain the dynamic probability score of the semi-static target, and judge the state of the semi-static target in the N-frame time series according to this dynamic probability score.

[0073] S5. Optimize the feature points to obtain the trajectory map

[0074] Input the data (including the feature points A, B, and C) after removing the dynamic feature points through target classification and dynamic detection into the standard processing flow of ORB-SLAM3 to complete the SLAM processing steps such as feature point tracking, local mapping, loop detection, map fusion, and global optimization, and finally generate a trajectory map corresponding to the RGB image.

[0075] The present invention has carried out a series of experiments based on the KITTI dataset and the TUM dataset to verify the effectiveness and superiority of the method. The experiments include dynamic target detection on the KITTI dataset to evaluate the detection effect of the present invention, as Figure 6 shown; carry out dynamic feature point removal experiments on the TUM dataset to verify the improvement of trajectory estimation after removing dynamic features, as Figure 7 shown; compare the trajectory estimation performance of ORB-SLAM3 and the method of the present invention on the KITTI dataset to highlight the accuracy advantage of the present method in dynamic scenarios, as Figures 8-1 to 8-6 shown; finally, conduct a comprehensive experiment on the trajectory estimation of 6 sequences of the KITTI dataset to demonstrate the stability and robustness of the method of the present invention, as Figures 9-1 to 9-6As shown. The experimental results show that the present invention has significant advantages in dynamic target detection, feature point removal, and trajectory estimation, providing a reliable technical guarantee for visual SLAM applications in dynamic scenarios.

[0076] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many improvements and changes without departing from the gist of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A dynamic environment VSLAM method based on semantic segmentation network and depth change analysis, characterized in that The adopted dynamic environment VSLAM system includes ORB-SLAM3, and the ORB-SLAM3 includes a front-end tracking module, a local mapping module, a loop closure detection module, and a global optimization module; a semantic segmentation network and a dynamic object detection module are introduced at the front end of the tracking module; The semantic segmentation network extracts information from the input image, fuses multi-level features, and performs final object segmentation; the semantic segmentation network includes a feature extraction module, a feature fusion module, and a semantic segmentation module; The feature extraction module includes several convolutional blocks Conv and cross convolutional blocks Cross-block, and the cross convolutional block Cross-block is designed based on Cross convolution; The feature fusion module includes several conventional convolutional blocks and feature fusion blocks FFB; The semantic segmentation module includes a convolutional layer, an upsampling layer, and a receptive field enhancement block RFEB. The receptive field enhancement block RFEB includes five parallel branches, two of which use conventional convolutional operations, two use dilated convolutional operations, and the other uses pooling operations to capture global context information; The segmented objects are classified into dynamic objects, semi-static objects, and static objects; the dynamic object refers to an object that moves during the observation period, the static object refers to an object that does not move at all during the observation period, and the semi-static object refers to an object that changes from static to dynamic during the observation period; then, the dynamic object detection module filters the feature points on the object; The ORB-SLAM3 performs feature point optimization processing on the filtered feature points, and finally generates a trajectory map corresponding to the RGB image.

2. The dynamic environment VSLAM method according to claim 1, wherein It includes the following steps: Step 1, data input processing and preprocessing: The dynamic environment VSLAM system obtains RGB images and depth images from an RGB-D camera and performs the following preprocessing: First, perform Gaussian filtering denoising on the RGB image; then, perform time synchronization on the RGB image and the depth image to ensure that the RGB image and the depth image are consistent in timestamp; The preprocessed RGB image is input into the semantic segmentation network for classifying the objects in the scene into dynamic objects, semi-static objects, and static objects; The preprocessed depth image is input into the dynamic object detection module to determine whether the object is a dynamic object; Step 2, segment the image into different semantic categories, including: The feature extraction module extracts deep and shallow semantic information from the input RGB image, and captures the local and global features in the image through the combination of convolutional blocks Conv and cross convolutional blocks Cross-block, so as to obtain the semantic information of the target object; The feature fusion module uses the method of cross-layer feature fusion to perform multi-scale feature fusion on the features from different network levels, and learns the detailed features in the shallow layer and the semantic information in the deep layer of the features; The fused features are used as the input of the semantic segmentation module. The semantic segmentation module assigns each pixel in the image to the corresponding semantic category through convolution and upsampling operations; uses the receptive field enhancement block (RFEB) to capture global context information, and restores the spatial resolution of the image through upsampling operations. Finally, the image is segmented into different semantic categories; Step 3: Classify the segmented target areas into the above-mentioned dynamic targets, semi-static targets, and static targets; Step 4: Screen the feature points on the above-mentioned distinguished targets, including: 4-1) Remove the feature points related to dynamics in the dynamic target, and retain other feature points, denoted as feature points A; 4-2) Retain all the feature points in the static target, denoted as feature points B; 4-3) Screen the semi-static target according to the dynamic probability scoring result, including: the dynamic target detection module performs deep feature modeling on the semi-static target, and judges the state of the semi-static target in the N-frame time series according to the dynamic probability scoring result. If the target is a dynamic target, remove the feature points related to dynamics on the target. If the target is a static target, retain the feature points on the target, denoted as feature points C; Step 5: Optimize the feature points to obtain a trajectory map: The ORB-SLAM3 system tracks, locally maps, performs loop detection, fuses maps, and globally optimizes the feature points A, feature points B, and feature points C, and finally generates a trajectory map corresponding to the RGB image.

3. The dynamic environment VSLAM method according to claim 1, wherein The specific process of step 4-3) is: 4-3-1) Calculate the depth change rate of the N-frame time series by calculating the depth values of the continuous N frames of semi-static targets, and then calculate the depth acceleration; 4-3-2) Divide each frame of the semi-static target into local grids, and calculate the depth mean and standard deviation of each grid to characterize the depth fluctuation degree of a single feature point in each grid; 4-3-3) Calculate the dynamic probability score of a single feature point according to the depth fluctuation degree of the single feature point, the depth change rate, and the depth acceleration of the N-frame time series; 4-3-4) Perform weighted averaging on the dynamic scores of all feature points to obtain the dynamic probability score of the semi-static target, and judge the state of the semi-static target in the N-frame time series according to this dynamic probability score.

Images