A dynamic scene visual slam method based on transformer and multi-modal fusion

Abstract

The application discloses a dynamic scene visual SLAM method based on a Transformer and multi-modal fusion, and relates to the technical field of real-time positioning and map construction. The method extracts an initial semantic mask by using a semantic segmentation network based on the Transformer, and performs time sequence consistency optimization through a ConvLSTM; then, a light flow dynamic mask is generated by using adaptive morphological erosion; meanwhile, a geometric consistency mask is constructed based on a re-projection error, a triangulation error and a depth consistency error; finally, the masks of the three modes are weighted and fused according to the accuracy of a verification set, and a high-reliability dynamic region mask is generated through a multi-condition judgment rule. Through the complementation and fusion of multi-modal information, the defects of single mode in the dynamic scene, such as inaccurate recognition, noise sensitivity and insufficient stability, are effectively overcome, and the positioning robustness and mapping precision of the visual SLAM in a complex dynamic environment are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of real-time localization and mapping technology, specifically to a dynamic scene visual SLAM method. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is a core technology in robotics, autonomous driving, and augmented reality. Its goal is to enable mobile platforms to simultaneously estimate their own position and build an environmental map in unknown environments using their onboard sensors. Visual SLAM, as an important implementation method, primarily relies on image sequences captured by cameras. It completes localization and mapping tasks through steps such as feature extraction, matching, and optimization, achieving satisfactory accuracy in static or weakly dynamic environments.

[0003] For dynamic targets commonly found in real-world scenarios, such as pedestrians and vehicles, existing technologies have proposed a series of processing methods to improve the robustness of SLAM systems in dynamic environments. The mainstream approach utilizes semantic segmentation networks to identify and classify objects in images, directly eliminating feature points classified as dynamic, thus preventing them from participating in pose estimation. Another common method is based on optical flow analysis, which distinguishes between static backgrounds and moving regions by detecting the motion patterns of pixels in an image sequence, thereby filtering out dynamic interference. Furthermore, some approaches employ geometric consistency constraints, such as using epipolar constraints or triangulation depth consistency in multi-view geometry to determine whether feature points belong to static structures, thereby identifying and eliminating abnormal matching points.

[0004] While the aforementioned methods can mitigate the impact of dynamic objects under certain conditions, they mostly rely on single-modal information processing and each has significant limitations. The performance of semantic segmentation methods is highly dependent on the completeness of the training data and the model's generalization ability. In complex real-world scenarios, they are prone to missegmentation or omissions, leading to incomplete removal of dynamic features or incorrect removal of static features. Optical flow methods are sensitive to image noise, lighting changes, and non-rigid motion; the calculated optical flow field often contains significant noise, resulting in a marked decrease in performance in fast-moving or texture-deficient regions. Methods based on geometric consistency are limited by the accuracy and stability of sensor depth measurements; their discriminative ability weakens considerably when depth information is unreliable or feature tracking is short.

[0005] Therefore, the above scheme cannot simultaneously meet the dual requirements of technical feasibility (accurate dynamic identification, stable tracking, and strong real-time performance) and engineering feasibility (universal method and light computational burden), resulting in its less than ideal application effect in real dynamic environments. Summary of the Invention

[0006] This invention proposes a dynamic scene visual SLAM method based on Transformer and multimodal fusion. Its purpose is to simultaneously achieve the accuracy of dynamic feature recognition, the stability of system operation, and the real-time performance of computation in real scenes containing complex dynamic interference, thereby meeting the dual requirements of technical feasibility and engineering feasibility, and improving the overall robustness and practicality of visual SLAM systems in dynamic environments.

[0007] The technical solution of this invention is as follows:

[0008] A dynamic scene visual SLAM method based on Transformer and multimodal fusion first obtains static masks for each frame from a sequence of consecutive multi-frame images, and then performs SLAM localization and mapping based on the static masks. The steps for obtaining the static masks include:

[0009] Step S1: Acquire a sequence of environmental images containing dynamic and static targets in multiple consecutive frames, and simultaneously acquire depth images as raw input data;

[0010] Step S2: Use a Transformer-based semantic segmentation network to perform dynamic object recognition on each frame of color image to obtain the corresponding initial semantic mask. Then, obtain semantic features based on the initial semantic mask, and then optimize the continuous semantic features through the ConvLSTM temporal module to obtain the optimized semantic mask for each frame.

[0011] Step S3: Determine the motion state of the pixel based on the optical flow amplitude, and generate the optical flow dynamic mask for each frame by combining the semantic mask and adaptive morphological erosion.

[0012] Step S4: Use geometric consistency constraints to make a judgment and generate a geometric consistency mask for each frame;

[0013] Step S5: Perform weighted fusion of semantic mask, optical flow dynamic mask and geometric consistency mask, generate the final dynamic mask for each frame according to the set multi-condition judgment rules, and obtain the static mask based on the final dynamic mask.

[0014] As a further improvement to the dynamic scene visual SLAM method based on Transformer and multimodal fusion: the image sequence obtained in step S1 is represented as follows: ;

[0015] In the above formula, Indicates the first Frame color image, Indicates the first Frame depth map This represents the total number of frames in the image sequence. For frame index, the value range is: arrive ;

[0016] In step S2, the initial semantic mask is obtained as follows:

[0017] A semantic segmentation network incorporating Transformer is used as the semantic analysis module to perform semantic parsing on the input color image, extracting object regions with dynamic features to obtain the initial semantic mask. ;

[0018] Specifically, based on a predefined set of dynamic categories Obtain the initial semantic mask Values ​​of all pixels in:

[0019]

[0020] In the above formula, Represents the coordinates of a pixel in an image; This indicates that the Transformer network is at the pixel level. The output of the first Confidence level of the category; This is a category index, representing different object categories in the image; A dynamic collection of categories; Indicates that solving makes The category index that yields the maximum value; Represents the initial semantic mask In pixels The value at that pixel When it belongs to the dynamic category, the value is taken as Otherwise .

[0021] As a further improvement to the dynamic scene visual SLAM method based on Transformer and multimodal fusion, in step S2, the optimized semantic mask is obtained as follows:

[0022] First, semantic features are obtained based on the initial semantic mask, and then the... The semantic features corresponding to the initial semantic mask of the frame are denoted as: ;

[0023] When continuous semantic features are input into the ConvLSTM temporal module, the state update process of ConvLSTM satisfies:

[0024]

[0025] In the above formula, For the first The input gate state of a frame is used to control the amount of input information for the current feature map. For the first The forget gate state of a frame is used to control the amount of historical cell states retained; For the first The candidate cell state of the frame is used to store the feature information of the current frame; For the first The cell state of the frame is fused with information from historical cell states and current candidate cell states; For the first The output gate state of a frame is used to control the amount of output from the cell state to the hidden state; For the first The hidden state of the frame is used to output optimized feature information; For the first The hidden state of a frame; For the first Cellular state of the frame; , , , , , , , These are all weight matrices, used for linear transformation of input features and historical states; , , , All are bias terms; The sigmoid activation function is used to map the output value to... A time zone is used to control the proportion of information flow. The hyperbolic tangent activation function is used to map values ​​to... interval; This represents the convolution operation; This indicates element-wise multiplication.

[0026] Subsequently, through the analysis of the first Hidden state of a frame Perform convolution mapping and Sigmoid function processing to obtain the first... Temporally optimized semantic mask of frames :

[0027]

[0028] In the above formula, Represents the convolution mapping weight matrix; Represents the convolutional mapping bias term; This represents the semantic mask after time-order optimization, where the value range of each element is [value range missing]. .

[0029] As a further improvement to the aforementioned dynamic scene visual SLAM method based on Transformer and multimodal fusion: when training the ConvLSTM temporal module, a temporal consistency loss is used. Measure the continuity of semantic masks across multiple consecutive frames;

[0030] Specifically, during training, semantic features from at least three frames of images are memorized and propagated over time, and temporal consistency loss is calculated:

[0031]

[0032] In the above formula, For the first Semantic mask of a frame; For the first Semantic mask of a frame; For the first Frame and the Optical flow vectors between frames of color images; For optical flow vector The mask timing alignment operator is used to align the first... The semantic mask of the frame is aligned to the first... The coordinate system of the frame; for Norm.

[0033] As a further improvement to the dynamic scene visual SLAM method based on Transformer and multimodal fusion, in step S3, each frame of image is processed according to the following steps:

[0034] Step S3.1: Use the dense bidirectional optical flow algorithm to calculate the current frame color image of adjacent frames to obtain the current frame number. Frame optical flow vector Then based on the optical flow vector Calculate the optical flow amplitude of each pixel;

[0035] Let the optical flow vector be... The median coordinate is The pixel-level optical flow vector of the pixel is , express The component in the horizontal direction, express The component in the vertical direction has the following coordinates: Pixel optical flow amplitude Calculate using the following formula:

[0036] ;

[0037] Step S3.2, for the first The set of optical flow amplitude values ​​of all pixels in a frame Find the mean with standard deviation Construct a dynamic threshold range:

[0038]

[0039] In the above formula, greater than The adjustment coefficient is used to control the width of the dynamic threshold range; This represents the lower limit of the dynamic threshold range; This represents the upper limit of the dynamic threshold range;

[0040] Step S3.3: Divide the motion state of each pixel according to the optical flow amplitude and dynamic threshold range:

[0041]

[0042] In the above formula, Represents a static set of regions. Represents the set of intermediate regions. Represents a dynamic set of regions;

[0043] Step S3.4: Morphological erosion of the semantic mask using adaptive structuring elements.

[0044] As a further improvement to the dynamic scene visual SLAM method based on Transformer and multimodal fusion, in step S3.4, the coordinates are... erosion radius of the pixel Calculate using the following formula:

[0045]

[0046] In the above formula, This is the preset minimum erosion radius constant; This is the preset maximum erosion radius constant;

[0047] Morphological erosion operations are performed on each pixel according to its corresponding erosion radius to obtain the dynamic optical flow mask value for that pixel. This allows us to obtain the complete optical flow dynamic mask for the current frame. :

[0048]

[0049] In the above formula, Represents the morphological erosion operator. The coordinates in the semantic mask of the current frame are The pixel value.

[0050] As a further improvement to the dynamic scene visual SLAM method based on Transformer and multimodal fusion: In step S4, the reprojection error, triangulation error and depth consistency error of each pixel are calculated for each frame of image. The comprehensive geometric error is obtained by weighting the above three errors. Then, the corresponding geometric consistency mask pixel value is obtained based on the comprehensive geometric error, thereby obtaining the geometric consistency mask.

[0051] As a further improvement to the dynamic scene visual SLAM method based on Transformer and multimodal fusion, in step S4:

[0052] The reprojection error is calculated as follows:

[0053] First, depth backprojection and pose transformation are performed to obtain the observed 3D point coordinates of each pixel in the current frame and the predicted 3D point coordinates in the previous frame.

[0054] Specifically, let the camera intrinsic parameter matrix be... The current frame is the first frame. Pixels in the depth map of a frame The depth value The coordinates of the observed 3D point for each pixel are calculated through back projection:

[0055]

[0056] In the above formula, This is the inverse of the camera intrinsic parameter matrix; Represents pixels homogeneous coordinates For pixels In the The coordinates of the observed three-dimensional points in the frame;

[0057] Let the pose transformation between adjacent frames be denoted as Then observe the coordinates of the three-dimensional point. In the previous frame, i.e. The corresponding predicted 3D point coordinates in the frame coordinate system are:

[0058]

[0059] In the above formula, Indicates by the first Frame observation of three-dimensional point coordinates The value predicted by pose transformation in the first position is... 3D point coordinates in the frame camera coordinate system; Indicates the first Frame to the The pose transformation matrix of a frame belongs to a special Euclidean group. ; Let be a rotation matrix, describing the first... Frame to the Frame pose changes; Let be the translation vector, describing the first... Frame to the Frame position change; It is the transpose of the zero vector;

[0060] Then, the predicted 3D point coordinates of each pixel are... Projected to the first From the frame image plane, obtain the corresponding non-homogeneous pixel coordinates. ;

[0061] Specifically, first calculate the homogeneous pixel coordinates:

[0062]

[0063] The non-homogeneous pixel coordinates are further obtained through dehomogenization:

[0064]

[0065] In the above formula, , , Represents the three components of the reprojected homogeneous coordinates; , Represents the non-homogeneous coordinates of the reprojection;

[0066] Finally, the reprojection error of each pixel is calculated:

[0067]

[0068] In the above formula, Represents pixels The reprojection error reflects the difference between the reprojected coordinates and the observed pixel coordinates; Represents pixels In the The actual observed location in the frame image; for Norm, used to calculate the Euclidean distance between two coordinate vectors;

[0069] The triangulation error is calculated as follows:

[0070] For pixels By triangulating images from different perspectives, triangulated 3D point coordinates can be obtained. Then the triangulation error corresponding to that pixel for:

[0071] ;

[0072] The depth consistency error is calculated as follows:

[0073] For pixels Triangulate the coordinates of three-dimensional points of Components can be used as pixels The prediction depth is denoted as The corresponding depth consistency error Defined as:

[0074]

[0075] In the above formula, Represents pixels In the Pixel values ​​in the frame depth map;

[0076] The calculation method for the comprehensive geometric error is as follows:

[0077]

[0078] In the above formula, For pixels The combined geometric error, , , These are weighting coefficients used to adjust the proportion of each individual error in the overall error. ;

[0079] The geometric consistency mask is obtained as follows:

[0080] For pixels ,when Greater than the preset geometric consistency judgment threshold When a feature point is determined to be a dynamic or outlier point that does not meet geometric consistency, the corresponding geometric consistency mask pixel value is determined. ,otherwise Finally, the geometrically consistent mask is obtained. .

[0081] As a further improvement to the dynamic scene visual SLAM method based on Transformer and multimodal fusion, in step S5, each frame of image is processed according to the following steps:

[0082] Step S5.1: Calculate the multimodal dynamic confidence of each pixel in the current frame. ;

[0083] Step S5.2: Generate the final dynamic mask pixel value corresponding to each pixel using a multi-condition joint judgment rule. The final dynamic mask that makes up this frame of the image:

[0084]

[0085] In the above formula, This is an indicator function; the value is set to true when the condition within the parentheses is true. Otherwise ; For multimodal dynamic confidence; , , The preset judgment threshold; This represents the logical AND operation; Represents a logical OR operation; For pixels semantic mask; For pixels Geometric consistency mask; For pixels Optical flow amplitude;

[0086] Step S5.3: Generate a static mask based on the final dynamic mask;

[0087] Static region mask Medium pixel mask value for:

[0088] .

[0089] As a further improvement to the dynamic scene visual SLAM method based on Transformer and multimodal fusion, in step S5-1, the multimodal dynamic confidence level... The calculation method is as follows:

[0090]

[0091] In the above formula, Represents the dynamic confidence of the semantic modality, taking the pixel value from the corresponding semantic mask. ; Represents the dynamic confidence level of optical flow modes, measured by the optical flow amplitude of that pixel. The result is obtained after normalization, and the normalization range is... ; Represents the geometric modality dynamic confidence level, which is the overall geometric error of the pixel. Mapped to The interval is obtained as follows: ,in This represents the maximum value of the combined geometric error of all pixels in the current frame. , , These are the fusion weights for semantic, optical flow, and geometric modalities, respectively. This represents the multimodal fusion score, with a value range of [value range missing]. ;

[0092] Fusion weights Based on the accuracy of the corresponding mask on the validation set , , The calculation yielded:

[0093]

[0094] In the above formula, This represents the dynamic region detection accuracy of the semantic mask on the validation set. This represents the dynamic region detection accuracy of the optical flow mask on the validation set. This represents the dynamic region detection accuracy of the geometric mask on the validation set.

[0095] Compared with the prior art, the present invention has the following beneficial effects:

[0096] 1. This invention introduces a Transformer-based semantic segmentation network and combines optical flow analysis and geometric consistency constraints for multimodal information fusion, effectively solving the problems of feature mismatch, trajectory drift, and decreased mapping accuracy caused by moving objects in dynamic environments in traditional visual SLAM. Compared with single-modal dynamic processing methods, this invention fully utilizes the category discrimination capability of semantic information, the motion sensitivity of optical flow information, and the structural consistency advantage of geometric constraints. Through weighted fusion and multi-condition judgment mechanisms, it significantly improves the accuracy and robustness of dynamic region recognition, thus maintaining high-precision localization and mapping performance even in complex scenes containing dynamic targets such as pedestrians and vehicles.

[0097] 2. This invention employs ConvLSTM to optimize the temporal consistency of semantic masks for consecutive frames. By memorizing and propagating semantic features over time, it compensates for mask breaks and missed detections caused by occlusion, viewpoint changes, and motion blur, ensuring the semantic mask remains continuous and stable both spatially and temporally. This design not only improves the reliability of the semantic mask but also provides a more accurate initial region for the subsequent generation of optical flow and geometric masks, thereby reducing the false positive rate in dynamic detection.

[0098] 3. In the optical flow dynamic mask generation stage, this invention proposes an adaptive thresholding and morphological erosion strategy based on optical flow amplitude. By calculating the mean and standard deviation of the optical flow amplitude for each frame, the threshold range is dynamically determined, and the erosion radius is adaptively adjusted according to the pixel motion intensity. This weakens noise interference in areas with low motion intensity and maintains mask integrity in areas with high motion intensity. This method effectively suppresses the impact of optical flow noise on dynamic detection and improves the accuracy of motion region boundaries.

[0099] 4. In the process of generating the geometric consistency mask, this invention comprehensively utilizes reprojection error, triangulation error, and depth consistency error to construct a multi-error weighted evaluation mechanism. By weighting and fusing different geometric errors according to their reliability, a comprehensive geometric error index is formed, and dynamic or abnormal feature points are determined based on a preset threshold. This method fully utilizes the complementarity of multi-view geometric constraints, enhances the ability to distinguish between dynamic and static points, and maintains good discrimination stability, especially when depth information is limited or feature tracking is short.

[0100] 5. This invention further integrates the detection results of semantic, optical flow, and geometric modalities through multimodal dynamic confidence fusion and multi-condition joint decision rules. The fusion weights are adaptively calculated based on the accuracy of each modality on the validation set, making the fusion process scene-adaptive; while the multi-condition decision rules further ensure the comprehensiveness and anti-interference of dynamic region detection by setting confidence and motion intensity thresholds for different modalities, avoiding system performance degradation caused by the failure of a single modality. Detailed Implementation

[0101] The technical solution of the present invention will be described in detail below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0102] A dynamic scene visual SLAM method based on Transformer and multimodal fusion includes:

[0103] Step S1: Acquire a sequence of environmental images containing dynamic and static targets in multiple consecutive frames, and simultaneously acquire depth images as raw input data.

[0104] Specifically, the image sequence is represented as .

[0105] In the above formula, Indicates the first Frame color image, Indicates the first Frame depth map This represents the total number of frames in the image sequence. For frame index, the value range is: arrive .

[0106] In RGB-D mode, depth image filtering preprocessing can be used to improve the quality of depth data, reduce depth noise, outliers and irregular transitions, thereby improving the accuracy of subsequent geometric consistency judgment.

[0107] Step S2: Use a Transformer-based semantic segmentation network to perform dynamic object recognition on each frame of color image to obtain the corresponding initial semantic mask. Then, obtain semantic features based on the initial semantic mask, and optimize the continuous semantic features through the ConvLSTM temporal module to obtain the optimized semantic mask for each frame.

[0108] (1) The method for obtaining the initial semantic mask is as follows:

[0109] A semantic segmentation network containing Transformers (such as Mask2Former) is used as the semantic analysis module to perform semantic parsing on the input color image, extracting object regions with dynamic features such as pedestrians and vehicles to obtain the initial semantic mask. .

[0110] Specifically, based on a predefined set of dynamic categories Obtain the initial semantic mask Values ​​of all pixels in:

[0111]

[0112] In the above formula, Represents the coordinates of a pixel in an image; This indicates that the Transformer network is at the pixel level. The output of the first Confidence level of the category; This is a category index, representing different object categories in the image; It is a dynamic category set, containing object categories with dynamic characteristics such as pedestrians and vehicles; Indicates that solving makes The category index that yields the maximum value; Represents the initial semantic mask In pixels The value at that pixel When it belongs to the dynamic category, the value is taken as Otherwise .

[0113] (2) The optimized semantic mask is obtained as follows:

[0114] First, semantic features are obtained based on the initial semantic mask (either through a convolutional network or by directly using the initial semantic mask as semantic features). The semantic features corresponding to the initial semantic mask of the frame are denoted as: .

[0115] When continuous semantic features are input into the ConvLSTM temporal module, the state update process of ConvLSTM satisfies:

[0116]

[0117] In the above formula, For the first The input gate state of a frame is used to control the amount of input information for the current feature map. For the first The forget gate state of a frame is used to control the amount of historical cell states retained; For the first The candidate cell state of the frame is used to store the feature information of the current frame; For the first The cell state of the frame is fused with information from historical cell states and current candidate cell states; For the first The output gate state of a frame is used to control the amount of output from the cell state to the hidden state; For the first The hidden state of the frame is used to output optimized feature information; For the first The hidden state of a frame; For the first Cellular state of the frame; , , , , , , , These are all weight matrices, used for linear transformation of input features and historical states; , , , All are bias terms; The sigmoid activation function is used to map the output value to... A time zone is used to control the proportion of information flow. The hyperbolic tangent activation function is used to map values ​​to... interval; This represents the convolution operation; This indicates element-wise multiplication.

[0118] Subsequently, through the analysis of the first Hidden state of a frame Perform convolution mapping and Sigmoid function processing to obtain the first... Temporally optimized semantic mask of frames :

[0119]

[0120] In the above formula, Represents the convolution mapping weight matrix; Represents the convolutional mapping bias term; This represents the semantic mask after time-order optimization, where the value range of each element is [value range missing]. The closer the value is The higher the confidence level, the more likely the pixel belongs to the dynamic region.

[0121] Furthermore, when training the ConvLSTM time-series module, a time-series consistency loss is used. Measure the continuity of semantic masks across multiple consecutive frames.

[0122] Specifically, during training, semantic features from at least three frames of images are memorized and propagated over time, and temporal consistency loss is calculated:

[0123]

[0124] In the above formula, For the first Semantic mask of a frame; For the first Semantic mask of a frame; For the first Frame and the Optical flow vectors between frames of color images; For optical flow vector The mask timing alignment operator is used to align the first... The semantic mask of the frame is aligned to the first... The coordinate system of the frame; for Norms are used to calculate the difference between two masks.

[0125] This constraint can compensate for mask breaks and missed detections caused by occlusion, changes in viewpoint, and changes in lighting, so that the semantic mask can maintain continuity and stability in space and time.

[0126] Step S3: Determine the motion state of the pixel based on the optical flow amplitude, and generate the optical flow dynamic mask for each frame by combining the semantic mask and adaptive morphological erosion.

[0127] Specifically, each frame of the image is processed according to the following steps:

[0128] Step S3.1: Use a dense bidirectional optical flow algorithm (e.g., Farnebäck algorithm) to calculate the current frame of color image from adjacent frames. Frame optical flow vector Then based on the optical flow vector Calculate the optical flow amplitude of each pixel.

[0129] Let the optical flow vector be... The median coordinate is The pixel-level optical flow vector of the pixel is , express The component in the horizontal direction, express The component in the vertical direction has the following coordinates: Pixel optical flow amplitude Calculate using the following formula:

[0130] .

[0131] Step S3.2, for the first The set of optical flow amplitude values ​​of all pixels in a frame Find the mean with standard deviation Construct a dynamic threshold range:

[0132]

[0133] In the above formula, greater than The adjustment coefficient is used to control the width of the dynamic threshold range; This represents the lower limit of the dynamic threshold range; This represents the upper limit of the dynamic threshold range. Each frame of the image has its own threshold range.

[0134] Step S3.3: Divide the motion state of each pixel according to the optical flow amplitude and dynamic threshold range:

[0135]

[0136] In the above formula, Represents a static set of regions. Represents the set of intermediate regions. Represents a dynamic set of regions.

[0137] Step S3.4: Morphological erosion of the semantic mask using adaptive structuring elements.

[0138] Coordinates are erosion radius of the pixel Calculate using the following formula:

[0139]

[0140] In the above formula, This is the preset minimum erosion radius constant; This is the preset maximum erosion radius constant.

[0141] Morphological erosion operations are performed on each pixel according to its corresponding erosion radius to obtain the dynamic optical flow mask value for that pixel. This allows us to obtain the complete optical flow dynamic mask for the current frame. :

[0142]

[0143] In the above formula, Represents the morphological erosion operator. The coordinates in the semantic mask of the current frame are The pixel value.

[0144] Step S4: Use geometric consistency constraints to make a judgment and generate a geometric consistency mask for each frame.

[0145] Specifically, for each frame of the image, the reprojection error, triangulation error, and depth consistency error of each pixel are calculated. The comprehensive geometric error is obtained by weighting the above three errors. Then, the corresponding geometric consistency mask pixel value is obtained based on the comprehensive geometric error, thus obtaining the geometric consistency mask.

[0146] (1) The reprojection error is calculated as follows:

[0147] First, depth backprojection and pose transformation are performed to obtain the observed 3D point coordinates of each pixel in the current frame and the predicted 3D point coordinates in the previous frame.

[0148] Specifically, let the camera intrinsic parameter matrix be... (Including parameters such as camera focal length and principal point coordinates), the current frame is the [number]th frame. Pixels in the depth map of a frame The depth value The coordinates of the observed 3D point for each pixel are calculated through back projection:

[0149]

[0150] In the above formula, This is the inverse of the camera intrinsic parameter matrix; Represents pixels homogeneous coordinates For pixels In the The coordinates of the observed three-dimensional points in the frame.

[0151] Furthermore, the pose transformation between adjacent frames is denoted as... Then observe the coordinates of the three-dimensional point. In the previous frame, i.e. The corresponding predicted 3D point coordinates in the frame coordinate system are:

[0152]

[0153] In the above formula, Indicates by the first Frame observation of three-dimensional point coordinates The value predicted by pose transformation in the first position is... 3D point coordinates in the frame camera coordinate system; Indicates the first Frame to the The pose transformation matrix of a frame belongs to a special Euclidean group. ; Let be a rotation matrix, describing the first... Frame to the Frame pose changes; Let be the translation vector, describing the first... Frame to the Frame position change; It is the transpose of the zero vector.

[0154] Then, the predicted 3D point coordinates of each pixel are... Projected to the first From the frame image plane, obtain the corresponding non-homogeneous pixel coordinates. .

[0155] Specifically, first calculate the homogeneous pixel coordinates:

[0156]

[0157] The non-homogeneous pixel coordinates are further obtained through dehomogenization:

[0158]

[0159] In the above formula, , , Represents the three components of the reprojected homogeneous coordinates; , Represents the non-homogeneous coordinates of the reprojection.

[0160] Finally, the reprojection error of each pixel is calculated. :

[0161]

[0162] In the above formula, Represents pixels The reprojection error reflects the difference between the reprojected coordinates and the observed pixel coordinates; Represents pixels In the The actual observed location in the frame image, which is obtained through dense optical flow tracing or feature point matching methods; for Norms are used to calculate the Euclidean distance between two coordinate vectors.

[0163] (2) The triangulation error is calculated as follows:

[0164] For pixels By triangulating images from different perspectives, triangulated 3D point coordinates can be obtained. Then the triangulation error corresponding to that pixel for:

[0165] .

[0166] (3) The depth consistency error is calculated as follows:

[0167] For pixels Triangulate the coordinates of three-dimensional points of Components can be used as pixels The prediction depth is denoted as The corresponding depth consistency error Defined as:

[0168]

[0169] In the above formula, Represents pixels In the Pixel values ​​in the frame depth map.

[0170] (4) The calculation method for the comprehensive geometric error is as follows:

[0171]

[0172] In the above formula, For pixels The combined geometric error, , , These are weighting coefficients used to adjust the proportion of each individual error in the overall error. This can be obtained through optimization of the validation set data.

[0173] (5) The geometric consistency mask is obtained as follows:

[0174] For pixels ,when Greater than the preset geometric consistency judgment threshold When (obtained through experimental data calibration), the feature point is determined to be a dynamic or outlier point that does not meet geometric consistency, thereby determining the geometric consistency mask pixel value corresponding to that pixel. ,otherwise Finally, the geometrically consistent mask is obtained. .

[0175] Step S5: Perform weighted fusion of semantic mask, optical flow dynamic mask and geometric consistency mask, generate the final dynamic mask for each frame according to the set multi-condition judgment rules, and obtain the static mask based on the final dynamic mask.

[0176] Specifically, each frame of the image is processed according to the following steps:

[0177] Step S5.1: Calculate the multimodal dynamic confidence of each pixel in the current frame.

[0178] For pixels Define the dynamic confidence scores of three modalities, and obtain the multimodal dynamic confidence score of the pixel through weighted fusion. :

[0179]

[0180] In the above formula, To represent the dynamic confidence of the semantic modality, the pixel value is directly taken from the corresponding semantic mask. ; Represents the dynamic confidence level of optical flow modes, measured by the optical flow amplitude of that pixel. The result is obtained after normalization, and the normalization range is... The larger the optical flow amplitude, the higher the confidence level. Represents the geometric modality dynamic confidence level, which is the overall geometric error of the pixel. Mapped to The interval is obtained as follows: The larger the error, the higher the confidence level. The mapping method can employ linear normalization, where... This represents the maximum value of the combined geometric error of all pixels in the current frame. , , These are the fusion weights for semantic, optical flow, and geometric modalities, respectively. This represents the multimodal fusion score, with a value range of [value range missing]. .

[0181] Furthermore, the fusion weights Based on the accuracy of the corresponding mask on the validation set , , The calculation yielded:

[0182]

[0183] And the constraints are satisfied: .

[0184] In the above formula, This represents the dynamic region detection accuracy of the semantic mask on the validation set. This represents the dynamic region detection accuracy of the optical flow mask on the validation set. This represents the dynamic region detection accuracy of the geometric mask on the validation set. The accuracy can be obtained by comparing each obtained mask with the labeled mask and calculating the intersection-over-union (IoU).

[0185] Step S5.2: Generate the final dynamic mask pixel value corresponding to each pixel using a multi-condition joint judgment rule. The final dynamic mask that makes up this frame of the image:

[0186]

[0187] In the above formula, This is an indicator function; the value is set to true when the condition within the parentheses is true. Otherwise ; For multimodal dynamic confidence; , , The preset judgment threshold; This represents the logical AND operation; Represents a logical OR operation; For pixels semantic mask; For pixels Geometric consistency mask; For pixels The optical flow amplitude. This rule aims to highlight the true dynamic region and reduce noise-induced false triggering.

[0188] Optionally, the final dynamic mask can be... Morphological closing operations are performed, followed by dilation and erosion operations to smooth edges, remove isolated noise points, and improve the stability of dynamic region detection.

[0189] Step S5.3: Generate a static mask based on the final dynamic mask.

[0190] Static region mask Medium pixel mask value for:

[0191] .

[0192] Step S6: Implement SLAM localization and mapping based on static masks.

[0193] By using the final static mask to constrain the feature point selection and pose estimation process in the visual SLAM system, only feature points in static regions are used for camera pose estimation and map construction, thereby reducing interference from dynamic targets.

[0194] The specific process is as follows:

[0195] Step S6.1, optimization with mask constraints.

[0196] Construct an objective function for reprojection error optimization that relies solely on static mask weighting, and solve the bundle adjustment optimization problem with mask constraints:

[0197]

[0198] In the above formula, Indicates the first The pose of the frame camera; Indicates the first The three-dimensional coordinates of a map point; Indicates the first The map point at the ... The projected pixel coordinates on the frame image, which are derived from the three-dimensional coordinates of the map points through the first... The frame camera pose and camera intrinsic parameter matrix were calculated using a pinhole camera projection model. express The static mask value of the corresponding pixel; Indicates the index for all frames and map point index Perform a summation operation; This indicates the minimum value operation, which is to find the optimal camera pose. and map points Minimize the objective function; This represents the camera projection function.

[0199] Step S6.2: Output the results.

[0200] By solving the above optimization problem, the camera pose sequence is output. With map point set This enables robust SLAM construction in dynamic scenarios.

[0201] This method is applicable to monocular, binocular, and RGB-D input modes. Under different sensor configurations, it uses the same multimodal fusion mechanism and the aforementioned weighting formula to eliminate dynamic targets. Based on minimizing the feature reprojection error under static mask constraints, it achieves robust feature tracking and high-precision pose estimation.

[0202] To further optimize the dynamic region detection performance, after completing the basic multimodal fusion process, the following optimization steps can be performed:

[0203] In the process of detecting moving objects, the overlapping region of the semantic mask and the optical flow intensity analysis results are combined, and the region is selected as the most effective dynamic region. To facilitate subsequent feature point extraction, morphological post-processing operations such as erosion, dilation, and connected component filtering can be performed on this overlapping region. Experimental tests show that the normalized optical flow intensity... Belongs to the interval Within a certain range, dynamic region detection performs best. Pixels within this range can be dynamically marked first to guide the removal of feature points.

[0204] To further improve the adaptability and local accuracy of multimodal fusion, the initially generated dynamic mask (such as...) can be... The image is divided into several non-overlapping or partially overlapping image blocks, either using an optical flow intensity map or a single image block. Within each block, the semantic intensity is... Optical flow intensity and geometric errors By comprehensively fitting and analyzing features such as [features], and by searching or optimizing on the validation data of this region, the weight combination that achieves the best dynamic detection effect in this region is selected. , , This serves as the local fusion weight for that block, replacing the globally unified fusion weight.

[0205] By applying the aforementioned fusion and elimination method based on local weight optimization to multiple sets of continuous image sequences under different dynamic scenes, experimental results show that the consistency between the final dynamic / static judgment results and the actual static background annotations is higher than that of methods using only simple semantic detection. This demonstrates that the multimodal fusion and adaptive optimization method provided in this invention can be effectively applied to visual SLAM in dynamic scenes of varying complexity, significantly improving the robustness and localization accuracy of the system.

[0206] It should be noted that, as will be apparent to those skilled in the art, the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics thereof. The scope of the present invention is defined by the claims rather than the foregoing description.

Claims

1. A dynamic scene visual SLAM method based on Transformer and multi-modal fusion, first obtaining a static mask of each frame based on a continuous multi-frame image sequence, and then realizing SLAM positioning and mapping based on the static mask, characterized in that, The steps to obtain a static mask include: Step S1: Acquire a sequence of environmental images containing dynamic and static targets in multiple consecutive frames, and simultaneously acquire depth images as raw input data; Step S2: Use a Transformer-based semantic segmentation network to perform dynamic object recognition on each frame of color image to obtain the corresponding initial semantic mask. Then, obtain semantic features based on the initial semantic mask, and then optimize the continuous semantic features through the ConvLSTM temporal module to obtain the optimized semantic mask for each frame. Step S3: Determine the motion state of the pixel based on the optical flow amplitude, and generate the optical flow dynamic mask for each frame by combining the semantic mask and adaptive morphological erosion. Step S4: Use geometric consistency constraints to make a judgment and generate a geometric consistency mask for each frame; Step S5: Perform weighted fusion of semantic mask, optical flow dynamic mask and geometric consistency mask, generate the final dynamic mask for each frame according to the set multi-condition judgment rules, and obtain the static mask based on the final dynamic mask; In step S5, each frame of the image is processed according to the following steps: Step S5.1, calculating a multi-modal dynamic confidence for each pixel of the current frame ; In step S5.1, the multi-modal dynamic confidence The calculation is: ； In the above formula, Represents the dynamic confidence of the semantic modality, taking the pixel value from the corresponding semantic mask. ; Represents the dynamic confidence level of optical flow modes, measured by the optical flow amplitude of that pixel. The result is obtained after normalization, and the normalization range is... ; Represents the geometric modality dynamic confidence level, which is the overall geometric error of the pixel. Mapped to The interval is obtained as follows: ,in This represents the maximum value of the combined geometric error of all pixels in the current frame. , , These are the fusion weights for semantic, optical flow, and geometric modalities, respectively. This represents the multimodal fusion score, with a value range of [value range missing]. ; Fusion weights Based on the accuracy of the corresponding mask on the validation set , , The calculation yielded: ； In the above formula, This represents the dynamic region detection accuracy of the semantic mask on the validation set. This represents the dynamic region detection accuracy of the optical flow mask on the validation set. This represents the dynamic region detection accuracy of the geometric mask on the validation set; Step S5.2, generating the final dynamic mask pixel value corresponding to each pixel using the multi-condition joint decision rule , the final dynamic mask constituting the frame image: ； In the above formula, This is an indicator function; the value is set to true when the condition within the parentheses is true. Otherwise ; For multimodal dynamic confidence; , , The preset judgment threshold; This represents the logical AND operation; Represents a logical OR operation; For pixels semantic mask; For pixels Geometric consistency mask; For pixels Optical flow amplitude; Step S5.3: Generate a static mask based on the final dynamic mask; Static region mask Medium pixel mask value for: 。 2. The dynamic scene visual SLAM method based on the fusion of Transformers and multi-modalities according to claim 1, characterized in that: The image sequence obtained in step S1 is represented as follows: ; In the above formula, Indicates the first Frame color image, Indicates the first Frame depth map This represents the total number of frames in the image sequence. For frame index, the value range is: arrive ; In step S2, the initial semantic mask is obtained as follows: The semantic segmentation network comprising the Transformer is used as a semantic analysis module to perform semantic analysis on an input color image, extract an object region with dynamic characteristics, and obtain an initial semantic mask ; In particular, according to a predefined dynamic set of categories obtaining an initial semantic mask values of all pixels in the image ； In the above formula, Represents the coordinates of a pixel in an image; This indicates that the Transformer network is at the pixel level. The output of the first Confidence level of the category; This is a category index, representing different object categories in the image; A dynamic collection of categories; Indicates that solving makes The category index that yields the maximum value; Represents the initial semantic mask In pixels The value at that pixel When it belongs to the dynamic category, the value is taken as Otherwise .

3. The dynamic scene visual SLAM method based on the fusion of Transformer and multi-modal according to claim 2, wherein, In step S2, the optimized semantic mask is obtained as follows: First, semantic features are obtained based on the initial semantic mask, and then the... The semantic features corresponding to the initial semantic mask of the frame are denoted as . ; When continuous semantic features are input into the ConvLSTM temporal module, the state update process of ConvLSTM satisfies: ； In the above formula, For the first The input gate state of a frame is used to control the amount of input information for the current feature map. For the first The forget gate state of a frame is used to control the amount of historical cell states retained; For the first The candidate cell state of the frame is used to store the feature information of the current frame; For the first The cell state of the frame is fused with information from historical cell states and current candidate cell states; For the first The output gate state of a frame is used to control the amount of output from the cell state to the hidden state; For the first The hidden state of the frame is used to output optimized feature information; For the first The hidden state of a frame; For the first Cellular state of the frame; , , , , , , , These are all weight matrices, used for linear transformation of input features and historical states; , , , All are bias terms; The sigmoid activation function is used to map the output value to... A time zone is used to control the proportion of information flow. The hyperbolic tangent activation function is used to map values ​​to... interval; This represents the convolution operation; This indicates element-wise multiplication. Subsequently, through the analysis of the first Hidden state of a frame Perform convolution mapping and Sigmoid function processing to obtain the first... Temporally optimized semantic mask of frames : ； In the above formula, Represents the convolution mapping weight matrix; Represents the convolutional mapping bias term; This represents the semantic mask after time-order optimization, where the value range of each element is [value range missing]. .

4. The dynamic scene visual SLAM method based on the fusion of Transformers and multi-modalities according to claim 3, characterized in that: when training the ConvLSTM temporal module, through a temporal consistency loss measuring continuity of consecutive semantic masks; Specifically, during training, semantic features from at least three frames of images are memorized and propagated over time, and temporal consistency loss is calculated: ； In the above formula, For the first Semantic mask of a frame; For the first Semantic mask of a frame; For the first Frame and the Optical flow vectors between frames of color images; For optical flow vector The mask timing alignment operator is used to align the first... The semantic mask of the frame is aligned to the first... The coordinate system of the frame; for Norm.

5. The dynamic scene visual SLAM method based on the fusion of Transformers and multi-modalities according to claim 1, characterized in that, In step S3, each frame of the image is processed according to the following steps: Step S3.1: Use the dense bidirectional optical flow algorithm to calculate the current frame color image of adjacent frames to obtain the current frame number. Frame optical flow vector Then based on the optical flow vector Calculate the optical flow amplitude of each pixel; Let the optical flow vector be... The median coordinate is The pixel-level optical flow vector of the pixel is , express The component in the horizontal direction, express The component in the vertical direction has the following coordinates: Pixel optical flow amplitude Calculate using the following formula: ； Step S3.2, for each pixel in the frame Set of optical flow magnitude values for all pixels of the frame Mean value Standard deviation Dynamic threshold interval is constructed: ； In the above formula, greater than The adjustment coefficient is used to control the width of the dynamic threshold range; This represents the lower limit of the dynamic threshold range; This represents the upper limit of the dynamic threshold range; Step S3.3: Divide the motion state of each pixel according to the optical flow amplitude and dynamic threshold range: ； In the above formulae, denotes a set of static regions, denotes a set of intermediate regions, denotes a set of dynamic regions; Step S3.4: Morphological erosion of the semantic mask using adaptive structuring elements.

6. The dynamic scene visual SLAM method based on Transformer and multimodal fusion as described in claim 5, characterized in that, In step S3.4, the coordinates are... erosion radius of the pixel Calculate using the following formula: ； In the above formula, is a preset minimum erosion radius constant; is a preset maximum erosion radius constant; Respectively, the corresponding erosion radius is carried out morphological erosion operation on each pixel, and a pixel corresponding optical flow dynamic mask value is obtained , so as to obtain the complete optical flow dynamic mask of the current frame : ； In the above formulae, denotes a morphological erosion operator, denotes the pixel value in the semantic mask of the current frame at coordinates .

7. The dynamic scene visual SLAM method based on the fusion of Transformers and multi-modalities according to claim 1, characterized in that: In step S4, the reprojection error, triangulation error, and depth consistency error of each pixel are calculated for each frame of the image. The comprehensive geometric error is obtained by weighting the above three errors. Then, the corresponding geometric consistency mask pixel value is obtained based on the comprehensive geometric error, thereby obtaining the geometric consistency mask.

8. The dynamic scene visual SLAM method based on Transformer and multimodal fusion as described in claim 7, characterized in that, In step S4: The reprojection error is calculated as follows: First, depth backprojection and pose transformation are performed to obtain the observed 3D point coordinates of each pixel in the current frame and the predicted 3D point coordinates in the previous frame. Specifically, let the camera intrinsic parameter matrix be... The current frame is the first frame. Pixels in the depth map of a frame The depth value The coordinates of the observed 3D point for each pixel are calculated through back projection: ； In the above formula, This is the inverse of the camera intrinsic parameter matrix; Represents pixels homogeneous coordinates For pixels In the The coordinates of the observed three-dimensional points in the frame; Let the pose transformation between adjacent frames be denoted as Then observe the coordinates of the three-dimensional point. In the previous frame, i.e. The corresponding predicted 3D point coordinates in the frame coordinate system are: ； In the above formula, Indicates by the first Frame observation of three-dimensional point coordinates The value predicted by pose transformation in the first position is... 3D point coordinates in the frame camera coordinate system; Indicates the first Frame to the The pose transformation matrix of a frame belongs to a special Euclidean group. ; Let be a rotation matrix, describing the first... Frame to the Frame pose changes; Let be the translation vector, describing the first... Frame to the Frame position change; It is the transpose of the zero vector; Then, the predicted three-dimensional point coordinates of each pixel are projected onto the first frame image plane respectively, to obtain corresponding non-homogeneous pixel coordinates ;​​ Specifically, first calculate the homogeneous pixel coordinates: ； The non-homogeneous pixel coordinates are further obtained through dehomogenization: ； In the above formulae, , , denotes the three components of the reprojected homogeneous coordinates; , denotes the non-homogeneous coordinates of the re-projection. Finally, the reprojection error of each pixel is calculated: ； In the above formula, Represents pixels The reprojection error reflects the difference between the reprojected coordinates and the observed pixel coordinates; Represents pixels In the The actual observed location in the frame image; for Norm, used to calculate the Euclidean distance between two coordinate vectors; The triangulation error is calculated as follows: For a pixel , the triangulation error of the pixel is , where is the triangulated 3D point coordinate of the pixel ； The depth consistency error is calculated as follows: For pixels Triangulate the coordinates of three-dimensional points of Components can be used as pixels The prediction depth is denoted as The corresponding depth consistency error Defined as: ； In the above formula, representing a pixel In the first pixel value in the frame depth map; The calculation method for the comprehensive geometric error is as follows: ； In the above formula, For pixels The combined geometric error, , , These are weighting coefficients used to adjust the proportion of each individual error in the overall error. ; The geometric consistency mask is obtained as follows: For pixels ,when Greater than the preset geometric consistency judgment threshold When a pixel is identified as a dynamic or outlier point that does not meet geometric consistency, the corresponding geometric consistency mask pixel value is determined. ,otherwise Finally, the geometrically consistent mask is obtained. .

Images