A SLAM method based on two-stream feature fusion

Abstract

This invention discloses a SLAM method based on dual-stream feature fusion. First, video data is acquired and stacked to form a temporal sequence of video frames, constructing a video background and extracting dynamic and static object features. Then, an attention mechanism module performs channel and spatial attention weighted fusion to obtain dual-stream features, which are then passed to a YOLO head network to output the bounding box and attribute table of the target in each frame, generating dynamic semantic labels. Next, the ORB-SLAM3 system is used to extract ORB feature points for each frame. Then, the camera pose is calculated based on the static ORB feature points to determine keyframes and construct a local map. Finally, global BA operations are performed, along with dynamic pollution cleanup, and the g2o graph optimization method is used to further optimize the global map. This invention improves robustness and accuracy in complex dynamic scenes; it also introduces an attention mechanism to strengthen key dynamic information and improve the accuracy of semantic map construction.

Description

Technical Field

[0001] This invention belongs to the technical field of simultaneous localization and mapping, specifically relating to a SLAM method based on dual-stream feature fusion. Background Technology

[0002] With the rapid development of robotics technology, SLAM is widely used in autonomous driving, robot autonomous navigation, and environmental perception. Traditional SLAM algorithms perform well in static environments, achieving high-precision localization and map building through visual feature point matching. However, in dynamic environments, moving pedestrians, vehicles, and other dynamic objects can lead to problems such as cumulative localization errors, map contamination, and insufficient real-time performance. For example, dependency elimination techniques that remove dynamic feature points from environmental data ignore dynamic environmental information and fail to achieve complete environmental perception. Feature-point-based visual SLAM often suffers from localization drift due to insufficient detectable information under conditions of low illumination or sparse structural details. Spatiotemporal Dual-Stream Network (STDN-VO) (Xu C, Zeng T, Luo Y, et al. Spatiotemporal Dual-Stream Network for Visual Odometry[J]. IEEE Robotics and Automation Letters, 2025.) extracts global contextual information from images through spatial flow and temporal context from consecutive frames through temporal flow, but it is only applicable to monocular input and lacks depth information, and its performance tends to degrade when encountering dynamic scenes or regions with sparse texture. Dual-vision inertial SLAM networks (DVI-SLAM) (Peng X, Liu Z, Li W, et al. Dvi-slam: A dual visual inertial slam network[C] / / 2024 IEEE International Conference on Robotics and Automation (ICRA).IEEE, 2024:12020-12026.) may exhibit performance degradation without explicit mechanisms for handling dynamic objects or occlusion. Therefore, they suffer from limitations in complex and dynamic scenes, including insufficient feature representation capabilities, poor adaptability to dynamic scenes, inadequate processing of temporal information, and low feature fusion efficiency. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a SLAM method based on dual-stream feature fusion. By co-optimizing the improved optical flow network and the target detection network, and combining the improved CBAM attention mechanism, the dual-stream feature weighted fusion is achieved to generate a dynamic semantic map, accurately distinguish between static structures and dynamic entities, solve the mapping problem in dynamic environments, and significantly improve the tracking accuracy and robustness of dynamic targets.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A SLAM method based on two-stream feature fusion includes the following steps:

[0006] Video data is acquired through video capture devices on mobile platforms and stacked into a video frame time sequence.

[0007] The video background is constructed based on the temporal sequence of video frames using a Gaussian mixture model. The dynamic object features and predicted trajectories of each frame are extracted by a dual-stream feature extraction model with a dual-branch structure. At the same time, the static object features of each frame are extracted using an object detection network.

[0008] Dynamic and static object features are fused separately using channel and spatial attention weighted fusion through an improved CBAM attention mechanism module to obtain dual-stream features.

[0009] The dual-stream features are passed to the YOLO head network to output the target bounding box and its attribute table in each frame, generating dynamic semantic labels; the targets include dynamic targets and static targets; the bounding box is a binary image, annotating the pixel region of the target in each frame; the pixel region of the dynamic target has a value of 1, and the other static regions have a value of 0; the attribute table records the category, velocity, position, confidence, tracking ID and predicted trajectory of each target;

[0010] Based on the target's bounding box and attribute table, the ORB-SLAM3 system is used to perform multi-scale pyramid processing on each frame in the video frame time sequence to extract ORB feature points for each frame; the ORB feature points include dynamic ORB feature points and static ORB feature points.

[0011] The camera pose is solved using the PnP algorithm based on static ORB feature points;

[0012] Keyframes are determined based on the number of static ORB feature points in each frame of the video frame time sequence; if the number of static ORB feature points in a frame is greater than a set number of feature points, it is inserted as a keyframe; 3D map points are generated by triangulating the ORB feature points of the keyframes, and their dynamic semantic labels are annotated. The 3D map points generated by the static ORB feature points are added to the construction of the local map; the 3D positions generated by the dynamic ORB feature points are temporarily stored and not added to the local map.

[0013] Perform global BA operations on all keyframes and 3D map points in the local map, and perform dynamic pollution cleanup. Accept static target categories, associate keyframes with 3D map points to construct a graph structure, and use the g2o graph optimization method to further optimize the global map.

[0014] As a preferred technical solution, the Gaussian mixture model is composed of multiple single Gaussian models, and the brightness distribution of pixels at different times is described by weighted summation;

[0015] The video background is constructed as follows:

[0016] Initialize the Gaussian mixture model, including randomly initializing the mean, setting the variance, and assigning weights;

[0017] For each frame in the video frame time sequence, the deviation of each pixel value in the image is compared with the mean of each single Gaussian model. It is then determined whether the deviation is less than the variance of the corresponding single Gaussian model by a set multiple. If so, the mean, standard deviation, and weight of the matched single Gaussian model are updated. If all deviations are greater than or equal to the variance of the corresponding single Gaussian model by a set multiple, the single Gaussian model with the smallest weight is discarded, and a new single Gaussian model is established. The mean of the newly established single Gaussian model is the current pixel value, the standard deviation is set based on a preset empirical value, the global statistical standard deviation of the image, or the derived value of the sensor noise characteristics, and the weight is the initial weight of the discarded single Gaussian model.

[0018] The individual Gaussian models are sorted in descending order according to the ratio of weight to standard deviation, and the Gaussian distribution of the top B individual Gaussian models is selected as the background pattern; the cumulative weight of the top B individual Gaussian models is greater than a set weight threshold.

[0019] In the image, each pixel value is checked to see if it matches the Gaussian distribution of the selected B single Gaussian models. If they match, the pixel is a background point; otherwise, it is a foreground point, thus obtaining the background sequence of the image.

[0020] Obtain the background sequence of each frame in the time sequence of the video frame, and calculate the average to obtain the video background.

[0021] As a preferred technical solution, the dual-branch dual-flow feature extraction model adopts an improved FlowNet2.0 convolutional optical flow network. Based on the original FlowNet2.0 network, an attention mechanism layer is added between the conv2 layer and the conv2_1 layer of the stacked third module FlowNetS.

[0022] The dual-branch dual-stream feature extraction model is trained using endpoint error as the loss function.

[0023] As a preferred technical solution, the improved CBAM attention mechanism module refers to introducing an interaction layer before the channel attention module, and using three different sizes of convolution kernels, 3×3, 5×5 and 7×7, in parallel in the spatial attention module to extract fine-grained edge, medium-sized region and global context features respectively.

[0024] The interaction layer performs convolution operations on dynamic object features and static object features respectively, and after weighted summation, it fuses with the concatenated features of dynamic object features and static object features to obtain a joint feature map.

[0025] The joint feature map is used to generate a channel attention map through the channel attention module;

[0026] The input feature map of the spatial attention module is obtained by multiplying the channel attention map and the joint feature map.

[0027] The spatial attention module generates the weights of three parallel convolutional kernels based on the global mean of the input feature map. After the input feature map passes through the three parallel convolutional kernels of the spatial attention module, multi-scale features are dynamically weighted and fused to obtain the spatial attention map.

[0028] Multiply the spatial attention map with the input feature map to output two-stream features.

[0029] As a preferred technical solution, the YOLO head network includes detection heads at different levels, corresponding to the detection of small, medium and large targets respectively;

[0030] The target localization loss of the different levels of detection heads is constrained using the complete intersection-union loss.

[0031] As a preferred technical solution, the step of using the ORB-SLAM3 system to perform multi-scale pyramid processing on each frame in the video frame time sequence to extract ORB feature points specifically involves:

[0032] The ORB-SLAM3 system receives the bounding boxes and attribute tables of dynamic targets output by the YOLO header network, and ensures the synchronization of the timestamps of video frames, dynamic masks, and attribute tables through ROS or a custom message queue; the dynamic mask is obtained by converting the bounding boxes of dynamic targets.

[0033] Multi-scale pyramid processing is performed on each frame in the temporal sequence of video frames to extract ORB feature points for each frame; the ORB feature points include keypoints and descriptors.

[0034] Iterate through the ORB feature points of each frame, remove the ORB feature points whose pixel coordinates are located in the pixel region of the dynamic target in each frame, and keep only the ORB feature points of the static region in each frame.

[0035] As a preferred technical solution, 3D map points generated by static ORB feature points are added to the local map, and the areas that the dynamic target may cover in the future are marked as temporary restricted areas according to the predicted trajectory in the dynamic target attribute table. The mobile platform actively avoids the temporary restricted areas when planning the path.

[0036] As a preferred technical solution, the global BA operation specifically includes:

[0037] Based on the target category output by the YOLO head network, assign weights to 3D map points of different categories;

[0038] A global basis-assimilation (BA) operation is performed to minimize the reprojection error of all 3D map points based on the poses and assigned weights of all keyframes.

[0039] As a preferred technical solution, the dynamic pollution cleanup specifically involves: obtaining the last observation time of each map point in the global map based on the temporarily stored dynamic target 3D position, and calculating the unobserved duration of each map point with the time when the dynamic target 3D position was observed; if the unobserved duration of a map point is greater than a set interval threshold and its distance from the dynamic target 3D position is less than a set distance threshold, then the map point is determined to be a dynamic pollution point, and a deletion operation is performed to clean up the dynamic pollution.

[0040] Dynamic contamination cleaning is performed within a set cleaning frequency based on the mobile platform's movement status.

[0041] As a preferred technical solution, when the mobile platform is detected to have returned to a previously visited area, new constraints are added to further optimize the global map;

[0042] The newly acquired data is then compared with the list of keyframes, and the data is merged with the existing global map to update the map information.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] 1. This invention addresses the problem of dynamic objects being mixed into maps due to the difficulty in distinguishing between static and dynamic features. It introduces an optical flow network and adds an attention mechanism to the second convolutional layer of its stacked third module to model pixel-level motion information between frames. Combined with a target detection network, it accurately identifies potential dynamic entities in the image, providing a more intelligent perceptual foundation for downstream tasks such as path planning and behavior prediction. Furthermore, by continuously tracking and updating the features of targets in the video frame sequence, it significantly reduces recognition breaks and tracking loss caused by target occlusion, rapid movement, and other factors.

[0045] 2. Based on the output of a dual-stream network, this invention employs an improved CBAM attention mechanism to weightedly fuse dual-stream features. Specifically, the channel attention mechanism weights and fuses dynamic and static features, while the spatial attention mechanism uses three different convolutional kernels in parallel to extract information. This fully leverages the complementarity of spatial and channel information, enhances feature representation capabilities, and effectively addresses the problem of insufficient feature representation in complex dynamic scenes. Compared to traditional linear stitching methods, this mechanism adaptively focuses on the feature channels and regions that contribute most to map construction, strengthens key dynamic information, and suppresses redundant or noisy features, thereby improving the accuracy of semantic map construction. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is an overall flowchart of the SLAM method based on dual-stream feature fusion in an embodiment of the present invention.

[0048] Figure 2 This is a schematic diagram of the video stacking process in an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of the structure of the improved CBAM attention mechanism module in an embodiment of the present invention. Detailed Implementation

[0050] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0051] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0052] like Figure 1 As shown in the figure, this embodiment of a SLAM method based on dual-stream feature fusion includes the following steps:

[0053] S1. Acquire video data through the video capture device of the mobile platform; because the video data consists of continuous frames but lacks inter-frame correlation information, it is necessary to stack the continuous frames to form a video frame time sequence, retain short-term motion information, and help to combine historical time sequence information to better predict subsequent actions.

[0054] In this embodiment, the stacking process of consecutive frames is as follows: Figure 2 As shown, firstly, for the video data to be processed, the OpenCV VideoCapture function is used to read each frame of the video, and the frame size is adjusted as needed; then, each adjusted frame is added to a frame list, and finally, the OpenCV vstack function is used to stack the video frames to obtain a video frame time sequence and save it. Furthermore, this embodiment selects a robot as the mobile platform for illustration, which is equipped with a video capture device to capture video data during its movement; however, the mobile platform of this application is not limited to this, and other mobile platforms that need to achieve autonomous positioning and environmental awareness also fall within the protection scope of this application.

[0055] S2. Based on the Gaussian mixture model, a video background is constructed from the temporal sequence of video frames. The dual-stream feature extraction model with a dual-branch structure is used to extract the dynamic object features and predicted trajectories of each frame. At the same time, an object detection network is used to extract the static object features of each frame.

[0056] Furthermore, this invention first employs a Gaussian mixture model to construct the video background. The Gaussian mixture model consists of multiple single Gaussian models, and its weighted summation describes the brightness distribution of pixels at different times. Therefore, the video background is constructed as follows:

[0057] S2.1.1 Initialization of Gaussian Mixture Model: Randomly initialize the mean of multiple single Gaussian models, set the variance, and assign weights.

[0058] S2.1.2 For each frame in the video frame time sequence, compare the value of each pixel in the image with the mean of each single Gaussian model to calculate the deviation, and determine whether the deviation is less than the variance of the corresponding single Gaussian model by a set multiple. If so, update the mean of the matched single Gaussian model. m Standard deviation s and weight w The update method is as follows:

[0059] ,

[0060] ,

[0061] ,

[0062] ,

[0063] in, For the first i The mean after updating a single Gaussian model m i For the first i The mean of a single Gaussian model before updating. For the first i The standard deviation after updating a single Gaussian model For the first i The standard deviation of a single Gaussian model before update I t For image t Pixel value at time, e For adaptive learning rate, For the first i The weights updated by a single Gaussian model w i For the first i Weights before updating a single Gaussian model For the first j The weights updated by a single Gaussian model w j For the first j Weights before updating a single Gaussian model d The global learning rate and model confidence adjustment factor are used to control the update rate of the single Gaussian model. Since the sum of the weights of multiple single Gaussian models is 1, when the weight of one model changes, the weights of the other single Gaussian models also need to be adjusted to ensure that the sum of the weights of all models is 1.

[0064] If all deviations after comparison are greater than or equal to the variance of the corresponding single Gaussian model set by multiple, then the single Gaussian model with the smallest weight is discarded, and a new single Gaussian model is established. The mean of the newly established single Gaussian model is initialized to the current pixel value, the standard deviation is set according to the preset empirical value, the global statistical standard deviation of the image, or the derived value of the sensor noise characteristics, and the weight is the initial weight of the discarded single Gaussian model.

[0065] S2.1.3, categorize each Gaussian model according to the ratio of its weight to its standard deviation (i.e., ... w i / s i Sort the data in descending order; select the Gaussian distributions of the first B single Gaussian models as the background patterns. The cumulative weight of the first B single Gaussian models is greater than a set weight threshold.

[0066] S2.1.4. In the image, determine whether the pixel value matches the Gaussian distribution of the selected B single Gaussian models. If they match, the pixel is a background point; otherwise, it is a foreground point, thus obtaining the background sequence of the image.

[0067] S2.1.5 Perform the above operation on each frame image in the video frame time sequence to obtain the background sequence of each frame image, and finally calculate the average to obtain the video background in the video frame time sequence scene.

[0068] In this embodiment, the Gaussian mixture model consists of 5 single Gaussian models. Initially, the variance is set to 9, and the weight is assigned a value of 1 / 5. When comparing each pixel value in the image with the mean of each single Gaussian model, if the deviation is less than 2.5 times the variance of the corresponding single Gaussian model, it is determined to be a matching single Gaussian model, and the mean of the single Gaussian model is updated. m Standard deviation s and weight w If each deviation is greater than or equal to 2.5 times the variance of the corresponding single Gaussian model, then the newly created single Gaussian model has the mean of the current pixel value, the standard deviation of 9, and the weight of the minimum weight. When selecting the background mode, the weight threshold is usually set in the range of 0.5 to 1. In this embodiment, the weight threshold is set to 0.6.

[0069] Furthermore, after acquiring the video background, it is input into a dual-branch dual-flow feature extraction model to extract dynamic object features. The dual-branch dual-flow feature extraction model of this invention employs an improved FlowNet2.0 convolutional optical flow network.

[0070] For the original FlowNet2.0 convolutional optical flow network, the first half performs convolution and pooling operations, gradually decreasing the resolution of the output feature map. The second half uses deconvolution and feature concat fusion operations, gradually increasing the resolution of the output feature map. The FlowNetC architecture contains two independent convolutional network branches, which extract features from the input image pairs respectively. Then, a correlation layer is used to calculate the similarity between feature maps, thereby capturing large displacement motion information. FlowNetC is responsible for handling large displacement motion in image pairs, FlowNetSA and FlowNetS are responsible for refining optical flow estimation, and FlowNet-SD is responsible for handling small displacement motion in image pairs.

[0071] Since the original optical flow network is insufficient in perceiving key motion regions when processing optical flow in complex scenes, this invention adds an attention mechanism layer between the conv2 layer and conv2_1 layer of the stacked third module FlowNetS, based on the original FlowNet2.0 convolutional optical flow network. This attention mechanism may include SimAM, SE, etc.

[0072] The input feature map of the attention mechanism layer in this embodiment X Dimensions C × W × H (Number of channels × width × height), firstly, a query is generated using three independent 1×1 convolutions. q ), key k ), value v ) matrix; use PyTorch's view function to... q and k Convert to shape C ×( W × H )of q 1 and k A matrix is ​​formed, and then adjusted using the permute function. q 1 dimension obtained q 2; Calculate the energy matrix using matrix multiplication. e Its elements e ( i , j ) represents the feature map of the first j Position relative to the first i The influence of location; e The attention matrix is ​​obtained by row-by-row softmax normalization. a Ensure that the sum of each row is 1; convert the value matrix v Converted via the view function C ×( W × H ) shapev After 1, with a Matrix weighted summation, and through learnable coefficients c (Initial value 0) Fuse the original feature maps X Final output Y = c ·( a · v 1 T )+ X This enables the model to dynamically learn the dependencies between global locations in the feature map.

[0073] In this invention, the training of the dual-branch structure dual-flow feature extraction model uses End-Point Error as the loss function, and optimizes the model by calculating the pixel-level error between the predicted optical flow field and the true value.

[0074] Finally, an object detection network is used to process the current frame to extract static object features. In this embodiment, the object detection network is a standard YOLOv11 neural network. For the input features, the YOLOv11 neural network first extracts features through multiple convolutional layers in the backbone network. The model initially contains several basic convolutional layers and C3K2 modules alternately connected to extract low-level features. After the feature extraction part, it enters the SPPF module, which further enriches the feature representation through multi-scale spatial pyramid pooling. Next, the C2PSA module uses two PSA modules to aggregate the features, integrating information from different levels to enhance the overall semantic understanding capability. The C2PSA module refines the model's ability to selectively focus on regions of interest by applying spatial attention to the extracted features. In the Neck part, an upsampling operation is first performed to recover a higher resolution feature map. Then, multiple Concat operations are used to fuse features from different levels. These fused feature maps are further processed by the C3K2 module to obtain features with stronger expressive power.

[0075] S3. Dynamic object features and static object features are fused by channel and spatial attention weighting respectively through the improved CBAM attention mechanism module to obtain dual-stream features.

[0076] Furthermore, this invention improves the CBAM attention mechanism. The original CBAM attention mechanism includes a channel attention module and a spatial attention module. To enhance the response of key regions, this invention improves it by introducing an interaction layer before the channel attention module and using three different sizes of convolutional kernels (3×3, 5×5, and 7×7) in parallel in the spatial attention module to extract fine-grained edge, medium-sized region, and global contextual features, respectively.

[0077] The interaction layer performs convolution operations on dynamic and static object features respectively, and then fuses the weighted sum with the concatenated features of the dynamic and static object features to obtain a joint feature map; the process is described as follows:

[0078] F fusion = α · Conv ( F motion ) + β · Conv ( F static ) + Concat ( F motion , F static ),

[0079] In the formula, F fusion For joint feature maps, F motion Features of dynamic objects F static Characteristics of static objects. Conv For convolution operations, adjust the channel dimensions to fit the fusion requirements; Concat To stitch feature maps along the channel dimension, the integrity of the original information is preserved; α , β These are learnable parameters used to balance the contributions of motion and static characteristics.

[0080] Then, the joint feature map is used to generate a channel attention map through the channel attention module; described as:

[0081] ,

[0082] in, AvgPool and MaxPool These represent global average pooling and max pooling operations, respectively. MLP This represents a multilayer perceptron. s This represents the Sigmoid activation function.

[0083] Next, the channel attention map and the joint feature map are multiplied to obtain the input feature map of the spatial attention module. F ;

[0084] Subsequently, the spatial attention module first analyzes the input feature map. F The global mean is used to generate the weights of three parallel convolutional kernels. w 1. w 2. w 3. Achieve adaptive multi-scale fusion; then input the feature map. FAfter passing through three parallel convolutional kernels in the spatial attention module, multi-scale features are dynamically weighted and fused to obtain a spatial attention map, improving the ability to capture small and fast-moving targets. Described as:

[0085] ,

[0086] F ’ = [ AvgPool ( F ); MaxPool ( F )],

[0087] Finally, the spatial attention map With input feature map F Perform multiplication to output dual-stream features.

[0088] S4. The dual-stream features are passed to the YOLO head network to output the target bounding box and its attribute table for each frame, generating dynamic semantic labels. The targets include dynamic targets and static targets. The bounding box is a binary image, annotating the pixel region of the target (such as pedestrians and vehicles) in each frame. The pixel region of dynamic targets has a value of 1, and the other static regions have a value of 0. The attribute table records information such as the category, speed, position, confidence, tracking ID and predicted trajectory of each target.

[0089] Furthermore, the YOLO head network in this invention includes detection heads at different levels, corresponding to the detection of small, medium, and large targets, respectively. Each level of the detection head adjusts the number of channels through convolutional operations to generate a feature map adapted to the current scale. Specifically, each level of the detection head outputs the class probability distribution of each anchor point through a 1×1 convolution, with dimensions […]. B , N cls , H , W ], N cls The number of categories is used; then, each bounding box parameter (such as center coordinates, width and height) is output through a 1×1 convolution, with dimensions [ B ,4, H , W ].

[0090] In the YOLO head network, the target localization loss of different levels of detection heads is constrained by the Complete Intersection over Union (CIoU) Loss to optimize localization accuracy. Finally, the target's bounding box and its attribute table are output for dynamic object filtering in SLAM.

[0091] S5. Based on the target's bounding box and attribute table, use the ORB-SLAM3 system to perform multi-scale pyramid processing on each frame in the video frame time sequence, and extract ORB feature points for each frame, including dynamic ORB feature points and static ORB feature points.

[0092] Furthermore, the ORB-SLAM3 system receives the bounding boxes and attribute tables of dynamic targets output by the YOLO head network. First, it uses ROS or a custom message queue to ensure strict synchronization of the timestamps of the video frames, dynamic masks, and attribute tables, avoiding pose estimation errors caused by data latency. The dynamic mask is derived from the bounding boxes of the dynamic targets.

[0093] Next, multi-scale pyramid processing is performed on each frame to extract ORB feature points (including key points and descriptors) for each frame.

[0094] Finally, the ORB feature points of each frame are traversed, and ORB feature points whose pixel coordinates are located in the pixel region of the dynamic target are removed, retaining only the ORB feature points of the static region in each frame. The removal method is as follows:

[0095] F static = { f i ∈ F = M ( x , y )=0}, where, f i For the first i ORB feature points in a static region F This is the set of ORB feature points belonging to static regions in a video frame. M ( x , y ) = YOLO _ Mask ( x , y )∧ Flow _ Mask ( x , y ), YOLO _ Mask This refers to the bounding box region for dynamic targets detected by the YOLO head network. Flow _ Mask The significant motion region is obtained by thresholding the optical flow amplitude output by the dual-flow feature extraction model.

[0096] S6. Solve the camera pose (rotation matrix) using the Perspective-n-Point (PnP) algorithm based on static ORB feature points. R Translation vector t).

[0097] Furthermore, when using the PnP algorithm to solve for camera pose, the calculation relies solely on static ORB feature points, and the pose estimation formula is:

[0098] ,

[0099] in, SE (3) is the set of all possible rotation + translation transformations. F static This is a set of static map points filtered by dynamic object bounding boxes. r (·) is a robust kernel function used to suppress false matching of dynamic objects; π (·) represents the camera projection function, which includes the intrinsic parameter matrix. K , P i For the first i A static 3D point (a point retained after being filtered by dynamic masking and trajectory prediction). p i For the current frame and P i Matched 2D feature point pixel coordinates.

[0100] S7. Keyframes are determined based on the number of static ORB feature points in each frame of the video frame time sequence. If the number of static ORB feature points in a frame exceeds a set number, it is inserted as a keyframe. 3D map points are generated by triangulating the ORB feature points of the keyframes, and their dynamic semantic labels are annotated. The 3D map points generated from the static ORB feature points are added to the construction of the local map. The 3D positions generated from the dynamic ORB feature points are temporarily stored but not added to the local map. Since the robot needs to reconstruct the 3D position of the dynamic target in the image to calculate the safe distance when performing operations such as obstacle avoidance, temporarily storing the 3D position of the dynamic target eliminates the need for additional computational overhead.

[0101] Furthermore, 3D map points generated from static ORB feature points are added to the local map, and areas that the dynamic target may cover in the future are marked as temporary restricted zones based on the predicted trajectory in the dynamic target attribute table. The robot actively avoids these historical restricted zones during path planning. The definition of temporary restricted zones is as follows:

[0102] ,

[0103] in T predict The predicted trajectory is in the dynamic target attribute table. This indicates a region expansion operation. D safeTo maintain a safe distance, the robot actively avoids these temporary restricted areas during path planning, even if the dynamic object in the current frame has already left. In this embodiment, when the number of static ORB feature points in a frame is greater than 100, the corresponding frame is considered a keyframe.

[0104] S8. Perform global BA operations on all keyframes and 3D map points of the local map, and perform dynamic pollution cleanup. Accept static target categories, keyframes and 3D map points to construct a graph structure, and use the g2o graph optimization method to further optimize the global map.

[0105] Furthermore, a global BA operation is performed to construct a global map, specifically as follows:

[0106] S8.1 First, based on the target category output by the YOLO head network, such as "wall" or "shelf", assign weights to 3D map points of different categories. w j For example, long-term stable static objects (such as walls and shelves) are assigned high weights, while static objects that may be temporarily obscured (such as doors and movable tables and chairs) are assigned low weights. In this embodiment, long-term stable static objects are assigned a weight of 0.8, and static objects that may be temporarily obscured are assigned a weight of 0.2.

[0107] S8.2 Finally, a global BA operation is performed to minimize the reprojection error of all 3D map points based on the poses and assigned weights of all keyframes. Minimizing the reprojection error is expressed as:

[0108] ,

[0109] in, N The number of keyframes. T i For the first i The pose of each keyframe. S For the first i Number of 3D map points in each keyframe P j For the first j The world coordinates of a 3D map point. P ij For the first j 3D map points in keyframes i The observed pixel coordinates, r This is a robust kernel function.

[0110] Furthermore, if a 3D map point has not been observed for a long period of time, and its location overlaps with the historical trajectory of a dynamic target, it is identified as a dynamic contaminated point and deleted. Specifically:

[0111] Based on the temporarily stored dynamic target 3D position Qm =( x , y , z , t current Get map points in the global map. P j The last observation time That is, the video frame in which the map point is closest to the 3D position of the moving target. t Calculate the unobserved duration for each map point based on the observed time. When ∆ is satisfied at the same time t j > T t and Make || P j - Q m ||< d m Then determine the map point P j For dynamic contamination points, perform a deletion operation. P n_map = P map { P j | IsLongTermDirty ( P j )= True Dynamic pollution treatment is carried out, among which... T t To set an interval threshold, d m To set a distance threshold, P n_map This is a set of map points after dynamic pollution cleanup. P map This is a set of map points before dynamic pollution cleanup. IsLongTermDirty () is the dynamic contamination point determination function; finally, dynamic contamination cleaning is performed within a set cleaning frequency based on the movement state. In this embodiment, an interval threshold is set. T t =10s, set distance threshold d m =0.8m, the robot performs dynamic pollution cleaning every 5 seconds when it is moving, and performs dynamic pollution cleaning every 30 seconds when it is stationary.

[0112] Furthermore, keyframes and map points are constructed into a graph structure, and the global map is further optimized using the g2o graph optimization method. After obtaining the global map, the robot performs path planning and moves. If the robot is detected to have returned to a previously visited area, new constraints are added to further optimize the global map and eliminate accumulated errors. At the same time, the newly acquired data is compared with the keyframe list and merged with the existing map to update the map information.

[0113] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0114] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0115] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A SLAM method based on dual-stream feature fusion, characterized in that, Includes the following steps: Video data is acquired through video capture devices on mobile platforms and stacked into a video frame time sequence. The video background is constructed based on the temporal sequence of video frames using a Gaussian mixture model. The dynamic object features and predicted trajectories of each frame are extracted by a dual-stream feature extraction model with a dual-branch structure. At the same time, the static object features of each frame are extracted using an object detection network. Dynamic and static object features are fused separately using channel and spatial attention weighted fusion through an improved CBAM attention mechanism module to obtain dual-stream features. The dual-stream features are passed to the YOLO head network to output the target bounding box and its attribute table in each frame, generating dynamic semantic labels; the targets include dynamic targets and static targets; the bounding box is a binary image, annotating the pixel region of the target in each frame; the pixel region of the dynamic target has a value of 1, and the other static regions have a value of 0; the attribute table records the category, velocity, position, confidence, tracking ID and predicted trajectory of each target; Based on the target's bounding box and attribute table, the ORB-SLAM3 system is used to perform multi-scale pyramid processing on each frame in the video frame time sequence to extract ORB feature points for each frame; the ORB feature points include dynamic ORB feature points and static ORB feature points. The camera pose is solved using the PnP algorithm based on static ORB feature points; Keyframes are determined based on the number of static ORB feature points in each frame of the video frame time sequence; if the number of static ORB feature points in a frame is greater than a set number of feature points, it is inserted as a keyframe; 3D map points are generated by triangulating the ORB feature points of the keyframes, and their dynamic semantic labels are annotated. The 3D map points generated by the static ORB feature points are added to the construction of the local map; the 3D positions generated by the dynamic ORB feature points are temporarily stored and not added to the local map. Perform global BA operations on all keyframes and 3D map points in the local map, and perform dynamic pollution cleanup. Accept static target categories, associate keyframes with 3D map points to construct a graph structure, and use the g2o graph optimization method to further optimize the global map.

2. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, The Gaussian mixture model is composed of multiple single Gaussian models, and describes the brightness distribution of pixels at different times by weighted summation; The video background is constructed as follows: Initialize the Gaussian mixture model, including randomly initializing the mean, setting the variance, and assigning weights; For each frame in the video frame time sequence, the deviation of each pixel value in the image is compared with the mean of each single Gaussian model. It is then determined whether the deviation is less than the variance of the corresponding single Gaussian model by a set multiple. If so, the mean, standard deviation, and weight of the matched single Gaussian model are updated. If all deviations are greater than or equal to the variance of the corresponding single Gaussian model by a set multiple, the single Gaussian model with the smallest weight is discarded, and a new single Gaussian model is established. The mean of the newly established single Gaussian model is the current pixel value, the standard deviation is set based on a preset empirical value, the global statistical standard deviation of the image, or the derived value of the sensor noise characteristics, and the weight is the initial weight of the discarded single Gaussian model. The individual Gaussian models are sorted in descending order according to the ratio of weight to standard deviation, and the Gaussian distribution of the top B individual Gaussian models is selected as the background pattern; the cumulative weight of the top B individual Gaussian models is greater than a set weight threshold. In the image, each pixel value is checked to see if it matches the Gaussian distribution of the selected B single Gaussian models. If they match, the pixel is a background point; otherwise, it is a foreground point, thus obtaining the background sequence of the image. Obtain the background sequence of each frame in the time sequence of the video frame, and calculate the average to obtain the video background.

3. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, The dual-branch dual-flow feature extraction model adopts an improved FlowNet2.0 convolutional optical flow network. Based on the original FlowNet2.0 network, an attention mechanism layer is added between the conv2 layer and the conv2_1 layer of the stacked third module FlowNetS. The dual-branch dual-stream feature extraction model is trained using endpoint error as the loss function.

4. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, The improved CBAM attention mechanism module refers to introducing an interaction layer before the channel attention module, and using three different sizes of convolution kernels (3×3, 5×5, and 7×7) in parallel in the spatial attention module to extract fine-grained edge, medium-sized region, and global context features respectively. The interaction layer performs convolution operations on dynamic object features and static object features respectively, and after weighted summation, it fuses with the concatenated features of dynamic object features and static object features to obtain a joint feature map. The joint feature map is used to generate a channel attention map through the channel attention module; The input feature map of the spatial attention module is obtained by multiplying the channel attention map and the joint feature map. The spatial attention module generates the weights of three parallel convolutional kernels based on the global mean of the input feature map. After the input feature map passes through the three parallel convolutional kernels of the spatial attention module, multi-scale features are dynamically weighted and fused to obtain the spatial attention map. Multiply the spatial attention map with the input feature map to output two-stream features.

5. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, The YOLO head network includes detection heads at different levels, corresponding to the detection of small, medium and large targets respectively; The target localization loss of the different levels of detection heads is constrained using the complete intersection-union loss.

6. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, The process of using the ORB-SLAM3 system to perform multi-scale pyramid processing on each frame in the video frame time sequence and extract ORB feature points specifically involves: The ORB-SLAM3 system receives the bounding boxes and attribute tables of dynamic targets output by the YOLO header network, and ensures the synchronization of the timestamps of video frames, dynamic masks, and attribute tables through ROS or a custom message queue; the dynamic mask is obtained by converting the bounding boxes of dynamic targets. Multi-scale pyramid processing is performed on each frame in the temporal sequence of video frames to extract ORB feature points for each frame; the ORB feature points include keypoints and descriptors. Iterate through the ORB feature points of each frame, remove the ORB feature points whose pixel coordinates are located in the pixel region of the dynamic target in each frame, and keep only the ORB feature points of the static region in each frame.

7. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, The 3D map points generated by static ORB feature points are added to the local map, and the areas that the dynamic target may cover in the future are marked as temporary restricted areas based on the predicted trajectory in the dynamic target attribute table. The mobile platform actively avoids the temporary restricted areas when planning the path.

8. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, The aforementioned global BA operation specifically refers to: Based on the target category output by the YOLO head network, assign weights to 3D map points of different categories; A global basis-assimilation (BA) operation is performed to minimize the reprojection error of all 3D map points based on the poses and assigned weights of all keyframes.

9. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, The dynamic pollution cleanup specifically involves: obtaining the last observation time of each map point in the global map based on the temporarily stored 3D location of the dynamic target, and calculating the unobserved duration of each map point by comparing it with the time when the 3D location of the dynamic target was observed. If the unobserved duration of a map point exceeds the set interval threshold and its distance from the 3D position of the dynamic target is less than the set distance threshold, then the map point is determined to be a dynamic contaminated point, and a deletion operation is performed to clean up the dynamic contamination. Dynamic contamination cleaning is performed within a set cleaning frequency based on the mobile platform's movement status.

10. The SLAM method based on dual-stream feature fusion according to claim 1, characterized in that, When a mobile platform is detected to have returned to a previously visited area, new constraints are added to further optimize the global map. The newly acquired data is then compared with the list of keyframes, and the data is merged with the existing global map to update the map information.

Images