Visual slam method and system based on lightweight semantic gate inhibition

Abstract

The application discloses a visual SLAM method and system based on lightweight semantic gate inhibition. The method comprises the following steps: acquiring an image frame and additional modal information of a mapping environment; using a real-time semantic segmentation network to extract semantic information of the image frame to obtain a corresponding semantic mask; performing pyramid scaling processing on the current image frame, the semantic mask and the additional modal information, and performing timestamp alignment; calculating an extraction factor of different feature extraction regions; performing gate threshold inhibition on the current feature extraction region based on the extraction factor to dynamically extract an effective feature point set; and tracking the feature point set to construct a map. The application improves the robustness and perception accuracy of the system in a high dynamic environment through a real-time semantic segmentation network; and the threshold gate inhibition mechanism is constructed by fusing the semantic information and the perception data of different input modalities to selectively inhibit and retain the feature points.

Description

Technical Field

[0001] This invention belongs to the technical field of simultaneous localization and mapping, specifically relating to a visual SLAM method and system based on lightweight semantic gating inhibition. Background Technology

[0002] In the field of autonomous navigation and environmental perception, Visual Simultaneous Localization and Mapping (VSLAM) is often achieved by fusing RGB images with depth information or by calculating binocular parallax using images acquired by the right eye of a stereo camera. These methods and strategies can significantly enhance the system's ability to perceive its environment and improve the accuracy and robustness of map construction.

[0003] However, most traditional visual SLAM systems are based on the assumption of static scenes, which is obviously very different from real-world applications. They rely on feature point matching between frames and epipolar constraints on camera estimation, both of which are based on grayscale invariance. In highly dynamic environments, such as randomly moving or potentially moving feature point regions like those of pedestrians, visual SLAM systems often fail to accurately evaluate the camera's pose, and dynamic objects can be inappropriately incorporated into the map construction. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies in terms of real-time performance, accuracy, and robustness in the mapping process under high dynamic environments, and to provide a visual SLAM method and system based on lightweight semantic gating inhibition.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: One aspect of the present invention provides a visual SLAM method based on lightweight semantic gating inhibition, comprising the following steps: Image frames and additional modal information of the mapping environment are acquired based on different camera sensors mounted on the mobile platform; A real-time semantic segmentation network is used to extract semantic information from the image frames input from the current camera, resulting in a corresponding semantic mask. The real-time semantic segmentation network comprises, in sequence, parallel spatial coding and semantic coding branches, a channel merging module, convolutional layers, mapping layers, and upsampling layers. The spatial coding branch downsamples and compresses the resolution of the image frames using several tensor features and extracts spatial feature tensors. The semantic coding branch includes a four-stage encoder module for extracting feature tensors at different scales and semantic levels. The feature tensors output from the encoder modules of the last three stages of the semantic coding branch are passed through a channel-enhanced dual-attention module, and the semantic segmentation loss at each stage is calculated. These are then weighted to construct the total loss function of the real-time semantic segmentation network. The channel-enhanced dual-attention module includes parallel spatial attention and channel-enhanced attention branches, used to simultaneously capture information in both channel and spatial dimensions at different semantic levels through spatial attention mechanisms and channel-enhanced attention. The acquired current image frame, semantic mask, and additional modal information are subjected to pyramid-style scaling at different scales, and timestamp alignment is performed. Based on the current image frame at the pyramid scale, the semantic mask, and the additional modal information, a set of semantic information of the semantic mask and the set of intersecting regions under different modal information are constructed, and the extraction factors of different feature extraction regions are calculated. Based on the obtained extraction factor, the current feature extraction region is subjected to gating threshold suppression to dynamically extract an effective set of feature points. Track the set of feature points and build a map.

[0006] As a preferred technical solution, the spatial coding branch receives RGB image frames acquired from the current camera unit in the form of a list, and extracts the feature tensors of the image frames through several tensor feature downsampling. Each tensor feature downsampling includes a downsampling module and a convolution module for learning spatial features. For the n Tensor feature downsampling is represented as: T 1= Conv 3×3 ( Tensor input ); T 2= Conv 3×3 ( T 1); in, The feature tensor output by the downsampling module. Conv 3×3 For general 3×3 convolution, For the set of real numbers, C n+1 For the firstn The number of tensor channels for tensor feature downsampling. H and W The spatial resolution size of the RGB image frame. Tensor input The input feature tensor; The feature tensor output by the convolutional module for learning spatial features.

[0007] As a preferred technical solution, the semantic coding branch receives RGB image frames acquired from the current camera unit in the form of a list; in the four-stage encoder module, the first-stage encoder module includes three general 3×3 convolutions stacked sequentially; the specific processes of the latter three stages of the encoder module are as follows: In the i In the encoder module of each stage, i Take {2,3,4} as an input feature tensor. Pointwise convolution is performed, and a general 3×3 convolution is used for preliminary semantic information extraction, represented as: X 1= Conv 3×3 ( Conv 1×1 ( X )); in, For the set of real numbers, Conv 1×1 For pointwise convolution, Conv 3×3 For general 3×3 convolution, C i-1 Input feature tensor X The number of tensor channels, H and W The spatial resolution size of the RGB image frame; For feature tensors Perform channel separation operation to obtain feature tensor ,in C i For the first i The number of tensor channels in the encoder module at each stage; Let the characteristic tensor X 2. Semantic feature information is learned by sequentially applying two asymmetric depthwise separable convolutions of 3×1 and 1×3; and the feature tensor is set... X 3. First, semantic features are learned by sequentially applying two asymmetric depthwise separable convolutions of 1×5 and 5×1, and then by sequentially applying 1×5 and 5×1 asymmetric depthwise separable convolutions with added dilation, for different receptive fields. This can be represented as follows: X 4= DConv1×3 ( DConv 3×1 ( X 2)); X 5= DConv 5×1 ( DConv 1×5 ( X 3)); X 5'= DDConv 5×1 ( DDConv 1×5 ( X 5)); in, These are the output feature tensors of the two branches, respectively. DConv 3×1 and DConv 1×3 This represents two asymmetric depthwise separable convolutions, 3×1 and 1×3. DConv 1×5 and DConv 5×1 This represents two asymmetric depthwise separable convolutions, 3×1 and 1×3. DDConv 1×5 and DDConv 5×1 This represents 1×5 and 5×1 asymmetric depthwise separable convolutions with added holes; feature tensor X 4 and X 5' Perform channel merging, compress the channels through pointwise convolution, and then add the compressed channels element-wise with the input feature tensor to obtain semantic features with different receptive fields. Finally, shuffle the channels, as shown below: X 6= Conv 1×1 ( Cat ( X 4, X 5'))+ Tensor input ; X 6'= Shuffle ( X 6); in, For the first i The semantic feature tensor ultimately output by the encoder module at each stage. Cat ( • This is a channel merging operation. Tensor input For the input feature tensor, Shuffle ( •This is a channel shuffling operation.

[0008] As a preferred technical solution, the spatial attention branch is specifically as follows: In relation to the i In the spatial attention branch connected to the encoder module at each stage, i Take {2,3,4}, and perform pointwise convolution on the input feature tensor. Perform channel communication: Y 1= Conv 1×1 ( Y ); in, For the set of real numbers, C i For the first i The number of tensor channels in the encoder module at each stage. H and W The spatial resolution size of the RGB image frame. Conv 1×1 For pointwise convolution; feature tensor The inputs are fed into two 3×3 dilated convolutions, and the resulting two feature tensors are multiplied by a matrix and then processed. softmax The operation calculates the spatial attention matrix of the current input feature tensor. Attention 1: Attention 1= SoftMax ( Mul ( DilConv 3×3 ( Y 1), DilConv 3×3 ( Y 1))); in, SoftMax ( • )for SoftMax operate, Mul ( • ) represents matrix multiplication. DilConv 3×3 ( • ) is a 3×3 dilated convolution; feature tensor Y 1. The feature tensor is obtained by multiplying the result of pointwise convolution with the spatial attention matrix. : Y 2= Mul ( Conv 1×1 ( Y 1), Attention 1); feature tensor Y 1 and Y 2. After element-wise addition, the input is fed into pointwise convolution and combined with the feature tensor. Y After summing, pointwise convolution is performed to obtain the spatial attention-weighted feature tensor. : Y 3= Conv 1×1 ( Conv 1×1 ( Y 1+ Y 2)+ Y 1); The channel-enhanced attention branch is specifically as follows: In relation to the i In the channel-enhanced attention branch connected to the encoder module at each stage, the input feature tensor... Perform global averaging, and pass through two fully connected layers. FC 1 and FC 2; The fully connected layer FC 1 includes a linear layer connected in sequence, ReLU An activation function and another linear layer reduce the dimension of the weight vector from the number of input channels to 1 / 16 of the number of input channels, and use... Sigmoid The function is normalized and represented as follows: F FC1 = sigmoid ( linear ( relu ( linear ( Avgpool ( Y ))))); in, F FC1 For fully connected layers FC Output of 1 Avgpool ( • (This is the global average.) linear ( • ) is a linear layer, relu ( • )for ReLU Activation function sigmoid ( • )for Sigmoid function; The fully connected layer FC 2 includes a 3×3 convolutional layer connected sequentially. ReLU An activation function and another 3×3 convolutional layer reduce the dimension of the weight vector from the number of input channels to 1 / 16 of the number of input channels, and use... SigmoidThe function is normalized and represented as follows: F FC2 = sigmoid ( conv ( relu ( conv ( Avgpool ( Y ))))); in, F FC2 For fully connected layers FC Output of 2 conv ( • () is a 3×3 convolutional layer; Fully connected layer FC 1 and FC The outputs of 2 are added together, and then multiplied one by one with the input feature maps to obtain the output feature map of enhanced channel interest. F CA , represented as: F CA = Conv 1×1 ( F FC1 + F FC2 )× Y ; Enhance the output feature map of channel interest F CA Spatial attention-weighted feature tensor obtained from the spatial attention branch Y Add 3 together to get the result of the first addition. i The channels connected to the encoder modules at each stage enhance the final output of the dual attention module.

[0009] As a preferred technical solution, the encoder modules of the last three stages of the semantic coding branch introduce multi-stage auxiliary loss calculation mechanisms to calculate the semantic segmentation loss in stages, specifically: L total = α 2 L 2+ α 3 L 3+ α 4 L 4+ L main ; in, L total For the total loss, L 2. L 3. L 4 represents the loss of the encoder modules in the second, third, and fourth stages, respectively, with corresponding weights. α 2.α 3. α 4; L main The final output of the real-time semantic segmentation network is represented by the main loss calculated using the main classification head and the true labels. Let the label corresponding to the input image frame be... , representing the semantic category label of each pixel, in the . i The predictions output by the encoder modules in stages ∈{2,3,4} are: ,in C i For the first i The number of tensor channels in the encoder module at each stage, i.e., the number of semantic categories. It is the set of real numbers; H and W Let be the spatial resolution size of the RGB image frame, then the... i The cross-entropy loss for each stage is defined as follows: ; in, As an indicator function, when pixel ( x,y The actual label is a dynamic category. c The value is 1 if it is true, and 0 otherwise. Represents pixels ( x,y (Belongs to the dynamic category) c The predicted probability, log ( • ) is a logarithmic function; Main loss L main Input feature tensor F main Represented as: F main = Project ( ConvBlock 3×3 ( Cat ( F seg , F sp ))); in, Project ( • ) represents the feature mapping operation of the mapping layer. ConvBlock 3×3 ( • ) represents the function operations of the fusion module. Cat ( • This is a channel merging operation. F seg For semantic encoding branch, F spThis is a spatial coding branch.

[0010] As a preferred technical solution, when the added modal information is a depth image, the semantic information of the constructed semantic mask and the set of intersecting regions under different modal information are used to calculate the extraction factors of different feature extraction regions, specifically: Spatial alignment of the semantic mask with the depth image; Based on the aligned regions, the depth information of the matched semantic labels is diffused or propagated in the spatial neighborhood to cover more unlabeled but structurally consistent neighboring regions, represented as: Ω1={( x , y )| M ( x , y )= c ∧ D ( x , y )>0}; Where Ω1 represents the semantic mask. M ( x , y ) and depth image D ( x , y The intersecting region of ) serves as the seed domain for deep diffusion; c For dynamic categories; For seed point ( x 0, y 0)∈Ω1 to its neighborhood Diffusion semantic tags, with the condition for deep diffusion being: | D ( x , y )- D ( x 0, y 0)|< δ 1; in, δ 1 represents the set depth difference tolerance threshold; Define a semantic diffusion weight function for pixels that satisfy the depth diffusion condition: ; in, For semantic diffusion weights, exp( • ) is an exponential function, σ 1 represents the sensitivity coefficient of the semantic diffusion weight function; The semantic score of diffusion is calculated using semantic diffusion weights. S ( x , yThe total semantic depth fusion factor is obtained by adding the pixel area of ​​Ω1 to the total semantic depth fusion factor. S fusion , represented as: ; ; in, As an indicator function, when pixel ( x , y The value is 1 if the diffusion condition is met by the seed pixel, and 0 otherwise; total( • () represents a total operation; Total semantic depth fusion factor S fusion Total pixels of the currently extracted region S extract Divide them to obtain the depth extraction factor of the current feature extraction region. F 1 extract , represented as: .

[0011] As a preferred technical solution, when the additional modal information is the right-eye acquired image, the semantic information of the constructed semantic mask and the set of intersecting regions under different modal information are used to calculate the extraction factors of different feature extraction regions, specifically: The left eye unit of the binocular camera is used as the input image, and the image acquired by the right eye is defined as the auxiliary image input; Stereo matching calculations are performed between the input image and the auxiliary image to obtain a binocular disparity map; Costs are aggregated within a sliding window, and path consistency is optimized on the cost volume using a semi-global matching algorithm to obtain the optimal disparity result with global coherence. For each pixel, the disparity value with the lowest cost is selected as the output binocular disparity map. ; in, P ( x , y ) represents the output binocular disparity map, with displacement in pixels; C ( x , y,d Calculate different disparity values ​​for the sliding window. d The matching cost function below; argmin This indicates the value of the independent variable required to minimize the objective function; semantic mask M ( x , y ) and binocular parallax map P ( x, y Spatial alignment is performed, defined as the semantic-parallax joint effective region Ω2: Ω2={( x , y )|M( x , y )= c ∧ P ( x , y )>0}; Wherein, Ω2 is the set of pixels that simultaneously possess both effective dynamic category and effective disparity value. c For dynamic categories; Using the disparity value of each pixel in the semantic-disparity joint effective region Ω2 as the initial seed for structural diffusion, a local disparity consistency diffusion operation is performed, specifically as follows: For seed point ( x 0', y For each 0')∈Ω2, disparity values ​​are diffused within its spatial neighborhood, and the diffusion range is controlled according to the local disparity similarity criterion: | P ( x , y )- P ( x 0', y 0')|< δ 2; in, δ 2 represents the tolerance threshold for disparity differences; Define a disparity diffusion weight function for each neighboring pixel that satisfies the local disparity similarity criterion: ; in, For disparity diffusion weights, exp( • ) is an exponential function, σ 2 represents the sensitivity coefficient of the disparity diffusion weighting function; For all pixels that satisfy the local disparity similarity criterion, calculate the disparity diffusion consistency score of the region. S’ ( x , y The total semantic disparity fusion factor is obtained by adding the pixel area of ​​Ω2 to Ω2. S’ fusion : ; Among them, total( • () represents a total operation; Total semantic disparity fusion factor S’ fusion Total pixel area of ​​the current regionS extract Divide to define the stereo extraction factor for the current feature extraction region. F 2 extract : .

[0012] As a preferred technical solution, if the current input does not contain additional modal information, i.e., the input is a monocular camera unit, the semantic information for constructing the semantic mask and the set of intersecting regions under different modal information, and the extraction factors for different feature extraction regions are calculated, specifically as follows: The mask region for the current dynamic category is defined as follows: Ω mask ={( x , y )| M ( x , y )= c}; Ω mask This represents the set of all semantically valid pixel locations in the current region. M ( x , y () is a semantic mask. c For dynamic categories; Define the monocular extraction factor for the current feature extraction region. F 3 extract for: total( • () represents the total operation.

[0013] As a preferred technical solution, the gating suppression of the current feature extraction region threshold based on the obtained extraction factor specifically includes: Extraction factors based on each feature extraction region F extract Calculate the regional gating threshold , represented as: ; Among them, extraction factors F extract Including monocular extraction factors, binocular extraction factors, and depth extraction factors; SFactor This represents a predefined system scaling factor used to adjust the sensitivity of the threshold; ln is the natural logarithm. Regional gating threshold Perform a restrictive check to determine if it exceeds the set threshold limit, and then execute the following steps: If the regional gate threshold If the value is greater than or equal to the upper threshold, the feature point extraction process for that region is skipped, and no further calculations are performed. If the regional gate threshold If the threshold is less than the upper limit, the regional gating threshold is optimized and adjusted to obtain the final gating threshold. θ r , represented as: θ r =[2- max (0.05, )], max (•) is the maximum value operation; Using the above-mentioned gating threshold θ r Perform feature point extraction operations using the current scale image and a gate threshold. θ r As input, we obtain the weighted feature point set in the output region after threshold filtering; Save the weighted feature point set to the feature point set storage unit of the current region.

[0014] Another aspect of the present invention provides a visual SLAM system based on lightweight semantic gating inhibition, applied to the above-mentioned visual SLAM method based on lightweight semantic gating inhibition, including a data acquisition module, a real-time semantic segmentation network, a scaling and alignment module, an extraction factor calculation module, a gating threshold inhibition module, and a map construction module. The data acquisition module is used to acquire image frames and additional modal information of the mapping environment based on different camera sensors mounted on the mobile platform. The real-time semantic segmentation network is used to extract semantic information from the image frames input from the current camera to obtain the corresponding semantic mask. It includes, in sequence, a parallel spatial coding branch and a semantic coding branch, a channel merging module, a convolutional layer, a mapping layer, and an upsampling layer. The spatial coding branch downsamples and compresses the resolution of the image frame using several tensor features and extracts spatial feature tensors. The semantic coding branch includes a four-stage encoder module for extracting feature tensors at different scales and semantic levels. The feature tensors output by the encoder modules in the last three stages of the semantic coding branch are passed through a channel-enhanced dual-attention module, and the semantic segmentation loss at each stage is calculated. These are then weighted to construct the total loss function of the real-time semantic segmentation network. The channel-enhanced dual-attention module includes a parallel spatial attention branch and a channel-enhanced attention branch, used to simultaneously capture information in both the channel dimension and spatial dimension at different semantic levels through spatial attention mechanisms and channel-enhanced attention. The scaling and alignment module is used to perform pyramid-style scaling processing on the acquired current image frame, semantic mask, and additional modal information at different scales, and to perform timestamp alignment. The extraction factor calculation module is used to construct the semantic information of the semantic mask and the set of intersecting regions under different modal information based on the current image frame at the pyramid scale, the semantic mask, and the additional modal information, and to calculate the extraction factor of different feature extraction regions. The gate threshold suppression module is used to perform gate threshold suppression on the current feature extraction region based on the obtained extraction factor, and dynamically extract an effective set of feature points. The map building module is used to track the set of feature points and build a map.

[0015] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) To address the stringent requirements of high real-time performance and high accuracy in practical applications of visual simultaneous localization and mapping (vSLAM) systems, this paper designs and proposes a lightweight semantic segmentation network model. This network, while maintaining low computational overhead, possesses excellent semantic recognition capabilities, enabling efficient and accurate identification and segmentation of dynamic targets in a scene, thus providing more reliable semantic support for subsequent feature extraction and map construction. By introducing this module, the robustness and perceptual accuracy of the system in highly dynamic environments are significantly improved, providing a strong guarantee for achieving stable and real-time visual SLAM.

[0016] (2) This invention constructs a threshold-gated suppression mechanism for dynamic feature point extraction by fusing semantic information with perceptual data from different input modalities. This mechanism can dynamically adjust the feature point extraction threshold according to the coupling relationship between semantic categories and modal features in the scene, thereby achieving selective suppression and retention of feature points. Specifically, for key feature points at the boundary between dynamic target edges and static regions, the system can effectively retain them due to their strong spatial correlation in localization; while for feature points in the edge regions that contribute weakly to localization, the gating mechanism is used to suppress them, thereby improving the accuracy and robustness of the overall feature extraction.

[0017] (3) By introducing a semantic gating suppression mechanism, the present invention enables joint perception of data under different input modalities and adaptively filters the effectiveness of edge feature points of dynamic objects. This allows the system to accurately retain key positioning features at the boundary between dynamic and static regions and effectively suppress misleading edge features. Attached Figure Description

[0018] Figure 1 This is a flowchart of a visual SLAM method based on lightweight semantic gating inhibition in an embodiment of the present invention.

[0019] Figure 2 This is a framework diagram of a lightweight real-time semantic segmentation network in an embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the channel-enhanced dual attention module in an embodiment of the present invention.

[0021] Figure 4 This is a schematic diagram of the feature extraction module in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram illustrating the combination of gating suppression methods under different modes in an embodiment of the present invention.

[0023] Figure 6 This is a schematic diagram of the adaptive gating threshold extracted in an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0025] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0026] Example: like Figure 1 As shown, this embodiment provides a visual SLAM method based on lightweight semantic gating inhibition, including the following steps: S1. Perceive the mapping environment based on the different camera sensors on the mobile platform.

[0027] Furthermore, the current RGB image frame can be obtained through a monocular or binocular camera sensor, or the current RGB image frame and depth image can be obtained simultaneously through an RGB-D camera sensor. In this embodiment, the mobile platform is a robot equipped with the above three different camera sensors to perceive the surrounding environment. However, the mobile platform is not limited to this, and other mobile platforms that need to achieve autonomous positioning and environmental perception are also within the protection scope of this application.

[0028] S2. A real-time semantic segmentation network is used to extract semantic information from the current camera input image frames to obtain their corresponding semantic masks. The real-time semantic segmentation network is based on a dual-branch coding structure, employing a fusion of channel enhancement and spatial attention mechanisms, and multi-stage loss convergence. It possesses high real-time performance and high-precision semantic recognition capabilities, accurately labeling the position and contours of dynamically categorized objects.

[0029] Further, in step S2, as Figure 2 As shown, the real-time semantic segmentation network comprises, in sequence: a parallel spatial encoding branch and a semantic encoding branch, a channel merging module, a convolutional layer, a mapping layer, and an upsampling layer. The spatial encoding branch downsamples and compresses the resolution of image frames using several (preferably three) tensor features and extracts spatial feature tensors. The semantic encoding branch includes a four-stage encoder module for extracting feature tensors at different scales and semantic levels. The feature tensors output by the encoder modules of the last three stages of the semantic encoding branch are respectively processed by a channel-enhanced dual-attention module, and the semantic segmentation loss of each stage is calculated. These are then weighted to construct the total loss function of the real-time semantic segmentation network. The real-time semantic segmentation network integrates a channel enhancement mechanism and a multi-stage supervised loss strategy, exhibiting good feature representation capabilities and training stability. The channel merging module performs channel merging operations on the feature tensors output by the spatial encoding branch and the semantic encoding branch. The convolutional layer includes two 3×3 convolutions.

[0030] S201. The RGB image frames acquired from the current camera unit are input into a lightweight real-time semantic segmentation network in list form. The tensor features are downsampled in several stages through a 3×3 convolution module via the spatial coding branch. The image size is compressed to reduce the image resolution and the spatial features of the image are extracted to obtain the feature tensor of the current image frame. Each stage of tensor feature downsampling includes a downsampling module and a convolution module for learning spatial features. For the n Tensor feature downsampling at each stage is represented as: T 1= Conv 3×3 ( Tensor input ); T 2= Conv 3×3 ( T 1); in, The feature tensor output by the downsampling module. Conv 3×3 For general 3×3 convolution, For the set of real numbers, C n+1For the first n The number of tensor channels for tensor feature downsampling. H and W This refers to the spatial resolution of the RGB image frame, which is also the height and width of the image. Tensor input This is the input tensor from the previous stage; The feature tensor output by the convolutional module for learning spatial features; S202. The image feature tensor corresponding to the current image frame is input into the semantic coding branch for processing. This coding branch consists of four encoder modules connected in sequence, corresponding to the four semantic abstraction stages in the semantic segmentation network. By extracting information at different scales and semantic levels layer by layer, a multi-scale feature tensor representation is finally obtained for subsequent semantic discrimination and fusion.

[0031] The first-stage encoder module consists of three general 3×3 convolutions stacked sequentially, responsible for extracting low-level edge and texture features. The semantic extraction modules (encoder modules) used in the second, third, and fourth stages are as follows... Figure 4 As shown, the specific process is as follows: In the i In the encoder module of each stage, i Take {2,3,4} as an input feature tensor. Node-wise convolution is performed to expand the channel dimension information, followed by a general 3×3 convolution for preliminary semantic information extraction. The operation process is represented as follows: X 1= Conv 3×3 ( Conv 1×1 ( X )); in, For the set of real numbers, Conv 1×1 For pointwise convolution, C i-1 Input feature tensor X The number of tensor channels; Furthermore, for the feature tensor Perform channel separation operation to obtain feature tensor Among them, the feature tensor X 2. Semantic feature information is learned by sequentially applying two asymmetric depthwise separable convolutions, 3×1 and 1×3. Feature tensor X 3. First, semantic features are learned from different receptive fields using two asymmetric depthwise separable convolutions, 1×5 and 5×1, sequentially. Then, dilation is added to the original 1×5 and 5×1 asymmetric depthwise separable convolutions to further expand the module's receptivity. XSemantic feature learning is performed using the receptive field of 3. The operation process is represented as follows: X 4= DConv 1×3 ( DConv 3×1 ( X 2)); X 5= DConv 5×1 ( DConv 1×5 ( X 3)); X 5'= DDConv 5×1 ( DDConv 1×5 ( X 5)); in, These are the output feature tensors of the two branches, respectively. DConv 3×1 and DConv 1×3 This represents two asymmetric depthwise separable convolutions, 3×1 and 1×3. DConv 1×5 and DConv 5×1 This represents two asymmetric depthwise separable convolutions, 3×1 and 1×3. DDConv 1×5 and DDConv 5×1 This represents 1×5 and 5×1 asymmetric depthwise separable convolutions with added holes (in this embodiment, the hole rate is 2).

[0032] Finally, the feature tensor X 4 and X 5' Perform channel merging by compressing the channels through pointwise convolution. After compression, add the compressed channels element-wise with the input feature tensor to obtain semantic features from different receptive fields. Then, perform channel shuffling to improve the overall expressive efficiency and perceptual ability of the model. The operation process is represented as follows: X 6= Conv 1×1 ( Cat ( X 4, X 5'))+ Tensor input ; X 6'= Shuffle ( X 6); in, For the firsti The semantic feature tensor ultimately output by the encoder module at each stage. Cat ( • This is a channel merging operation. Tensor input For the input feature tensor, Shuffle ( • This is a channel shuffling operation.

[0033] S203. In stages 2, 3, and 4 of the semantic encoding branch, the feature tensors output from each stage are sequentially input into the channel-enhanced dual-attention module proposed in this invention to further enhance the responsiveness of key semantic features. For example... Figure 3 As shown, the channel-enhanced dual attention module combines channel-enhanced attention with an improved spatial attention mechanism, enabling it to simultaneously capture important information from both the channel and spatial dimensions at different semantic levels, effectively improving the discriminative and semantically focused capabilities of feature representations. Specifically, each channel-enhanced dual attention module performs the following operations on its input feature tensor: Transform the input feature tensor The inputs are respectively fed into the spatial attention branch and the channel-enhanced attention branch.

[0034] Operations in the spatial attention branch: In relation to the i In the spatial attention branch connected to the encoder module at each stage, i Take {2,3,4}, and perform pointwise convolution on the input feature tensor. Perform channel communication: Y 1= Conv 1×1 ( Y ); feature tensor The inputs are fed into two 3×3 dilated convolutions, and the resulting two feature tensors are multiplied by a matrix and then processed. softmax The operation calculates the spatial attention matrix of the current input feature tensor. Attention 1: Attention 1= SoftMax ( Mul ( DilConv 3×3 ( Y 1), DilConv 3×3 ( Y 1))); in, SoftMax ( • )for SoftMax operate, Mul ( •) represents matrix multiplication. DilConv 3×3 ( • ) is a 3×3 dilated convolution (in this embodiment, the dilation rate is 2); feature tensor Y 1. The feature tensor obtained by multiplying the feature tensor result after pointwise convolution with the spatial attention matrix is ​​then obtained. : Y 2= Mul ( Conv 1×1 ( Y 1), Attention 1); feature tensor Y 1 and Y 2. After element-wise addition, the input is fed into pointwise convolution and then combined with the feature tensor again. Y After summing, pointwise convolution is performed to obtain the spatial attention-weighted feature tensor. : Y 3= Conv 1×1 ( Conv 1×1 ( Y 1+ Y 2)+ Y 1); Operations in the channel-enhanced attention branch: In relation to the i In the channel-enhanced attention branch connected to the encoder module at each stage, the input feature tensor... Perform a global average, then pass it through two fully connected layers. FC 1 and FC 2.

[0035] Fully connected layer FC 1 includes a linear layer connected in sequence, ReLU An activation function and another linear layer reduce the dimension of the weight vector from the number of input channels to 1 / 16 of the number of input channels, then use... Sigmoid The function is normalized. FC The formula for the output feature map of 1 is: F FC1 = sigmoid ( linear ( relu ( linear ( Avgpool ( Y ))))); in, F FC1 For fully connected layersFC Output of 1 Avgpool ( • (This is the global average.) linear ( • ) is a linear layer, relu ( • )for ReLU Activation function sigmoid ( • )for Sigmoid function; Fully connected layer FC 2 includes a 3×3 convolutional layer connected sequentially. ReLU The process consists of an activation function and another 3×3 convolutional layer, which reduces the dimension of the weight vector from the number of input channels to 1 / 16 of the number of input channels, and then uses... Sigmoid The function is normalized. FC The formula for the output feature map of 2 is: F FC2 = sigmoid ( conv ( relu ( conv ( Avgpool ( Y ))))); in, F FC2 For fully connected layers FC Output of 2 conv ( • () is a 3×3 convolutional layer; Fully connected layer FC 1 and FC The outputs of the two features are summed, and then multiplied one by one with the input feature map. The result is expanded to match the shape of the input feature map, thus obtaining an output feature map that enhances channel focus. F CA , can be represented as: F CA = Conv 1×1 ( F FC1 + F FC2 )× Y ; Finally, the output feature map of enhanced channel interest will be... F CA Spatial attention-weighted feature tensor obtained from the spatial attention branch Y Add 3 together to get the result of the first addition. i The channels connected to the encoder modules at each stage enhance the final output of the dual attention module.

[0036] S204. A multi-stage auxiliary loss calculation mechanism is introduced in stages 2, 3, and 4 of the semantic encoding branch. Specifically, the feature tensor output from each stage is input into the corresponding auxiliary classification head, compared with the true semantic labels, and the stage-specific semantic segmentation loss is calculated. These stage-specific losses participate in the construction of the final total loss function with certain weights. Let the encoder module losses for stages 2, 3, and 4 be respectively... L 2. L 3. L 4, and its corresponding weight is α 2. α 3. α 4; Simultaneously, the final output after the dual-branch semantic fusion is also used to calculate the main loss using the main classification head and the true label. L main The training loss of the entire semantic segmentation network can then be expressed as: L total = α 2 L 2+ α 3 L 3+ α 4 L 4+ L main ; in, L total For the total loss, α 2. α 3. α 4. Experiments verified that the values ​​were set to 0.3, 0.3, and 0.4 respectively. In this invention, the semantic coding branches at stages 2, 3, and 4 are respectively connected to auxiliary segmentation heads to perform semantic prediction on the intermediate feature tensors, and the cross-entropy loss function is introduced into the final segmentation head and all auxiliary segmentation heads.

[0037] Specifically, let the label corresponding to the input image frame be... , representing the semantic category label of each pixel, in the . i The predictions output by the encoder modules in stages ∈{2,3,4} are: ,in C i For the first i The number of tensor channels in the encoder module at each stage, i.e., the number of semantic categories. H and W Let be the spatial resolution size of the RGB image frame, which is the image height and width. Then the ... i The cross-entropy loss for each stage is defined as follows: ; in, As an indicator function, when the pixel position ( x,y The actual label is the category. c The value is 1 if it is true, and 0 otherwise. Represents pixels ( x,y ) belongs to category c The predicted probability, log ( • The function is logarithmic. For the output of the main semantic segmentation branch (i.e., the final segmentation result after fusing the two coding branches), its corresponding main loss is also calculated. L main They use the same cross-entropy calculation method.

[0038] Among them, the main loss L main Input feature tensor This is achieved by using the output feature tensor of the fourth stage of the semantic encoding branch. With the output feature tensor of the third stage of the spatial coding branch After channel merging, the features are fused through a fusion module consisting of two general 3×3 convolutions, and then mapped through a mapping layer to obtain the input feature tensor of the main loss. F main Its specific operation can be expressed by the formula: F main = Project ( ConvBlock 3×3 ( Cat ( F seg , F sp ))); in, Project ( • ) represents the feature mapping operation of the mapping layer. ConvBlock 3×3 ( • ) represents the function operations of the fusion module. Cat ( • This is a channel merging operation. F seg For semantic encoding branch, F sp This is a spatial coding branch.

[0039] S3. After acquiring the current image frame, semantic mask, and additional modal information (such as depth map or right eye acquisition image), perform pyramid-style multi-scale scaling on the above data and align the timestamps.

[0040] Furthermore, in step S3, after completing the semantic segmentation operation and obtaining the semantic mask corresponding to the current frame image, considering that there is a certain delay in the reasoning of semantic information (i.e., the generation of semantic results is temporarily delayed relative to the acquisition of image frames), the present invention introduces a timestamp alignment mechanism to ensure the synchronization of semantic information with the corresponding image frame and the supplementary unit modal information.

[0041] Specifically, the system records the corresponding timestamp information for each image frame during acquisition and processing. The generated semantic mask is marked and cached according to the same timestamp as the original image frame. In subsequent processing, the system can accurately call the semantic mask corresponding to the current frame based on the timestamp matching rules, avoiding feature alignment offset caused by semantic inference delays and improving the overall system stability and perception accuracy.

[0042] Subsequently, the system performs image pyramid scaling on the time-aligned semantic mask. Specifically, this invention scales the semantic mask and other input information according to multiple scales (original size 1.0x, magnified 1.5x, reduced 0.75x) to generate multi-scale semantic information representations for adapting to image feature maps of different scales. This pyramid scaling strategy helps achieve unified processing of cross-scale information in the feature fusion and dynamic region suppression stages, improving the system's ability to perceive multi-scale targets and dynamic boundaries.

[0043] S4. Based on the current image frame information at the pyramid scale, the semantic mask, and the additional modal information input (including depth map or right eye acquisition image; for monocular camera systems, this modal information is not included), the system constructs a fused perceptual feature map (i.e., a set of intersecting regions under the semantic information of the semantic mask and different modal information), and calculates the extraction factors of different feature extraction regions based on the preset function module.

[0044] Furthermore, in step S4, based on the image frame at the current pyramid scale, its corresponding semantic mask, and supplementary modal input (including depth map or right-eye acquired image; for monocular camera systems, this modal information is not included), the different information is combined to generate extraction factors for different feature extraction regions of the current image, specifically: S401. First, determine whether the current input contains supplementary modal information units to determine the processing method for subsequent feature fusion strategies. If the modal information is a depth image, the system executes the following processing flow: like Figure 5 As shown, the system first defines the semantic mask region ( Figure 5 (b) in the image and the depth image region ( Figure 5(c) Spatial alignment is performed. This process can be understood as performing a pixel-level "intersection" operation in two image spaces, that is, retaining those location regions that simultaneously have valid semantic labels and valid depth values, thereby assigning corresponding semantic category labels to the depth information. This operation ensures that the depth data can be constrained in the semantic space, enabling the system to analyze depth changes under the premise of known category semantics, and enhancing the ability to model dynamic regions. Next, based on the above aligned regions, the system introduces a depth diffusion mechanism, that is, the depth information with matched semantic labels is diffused or propagated in the spatial neighborhood, thereby covering more unlabeled but structurally consistent neighboring regions. This diffusion mechanism achieves effective fusion of semantic and geometric information while maintaining depth continuity and spatial consistency, generating a fused feature tensor containing rich contextual information. The fused feature tensor will be used as input to the subsequent feature factor calculation module to dynamically adjust the feature point extraction intensity of different regions, achieving more refined gating discrimination. Its operation can be expressed as: Ω1={( x , y )| M ( x , y )= c ∧ D ( x , y )>0}; Where Ω1 represents the semantic mask. M ( x , y ) and depth image D ( x , y The intersecting regions of ) serve as seed domains for deep diffusion, possessing dynamic categories. c And the effective pixel position of depth.

[0045] S402. Based on step S401, the semantic label of each location in this region is further used as a diffusion source. For seed points ( x 0, y 0)∈Ω1 to its neighborhood Diffusion semantic tags, with the condition for deep diffusion being: | D ( x , y )- D ( x 0, y 0)|< δ 1; in, δ 1 represents the set depth difference tolerance threshold, which is the criterion for determining "depth similarity". In this embodiment... δ 1 is set to 0.15.σ 1 is set to 0.73.

[0046] Then, a semantic diffusion weight function is defined for pixels that satisfy the depth diffusion condition: ; in, For semantic diffusion weights, exp( • ) is an exponential function, σ 1 represents the sensitivity coefficient of the semantic diffusion weight function; The semantic score of diffusion is calculated using semantic diffusion weights. S ( x , y The total semantic depth fusion factor is obtained by adding the pixel area of ​​Ω1 to the total semantic depth fusion factor. S fusion .

[0047] ; ; in, As an indicator function, when the pixel position ( x , y The value of total( ) is 1 if it satisfies the diffusion condition with the seed pixel, and 0 otherwise; • () represents the total operation.

[0048] Ultimately, the total semantic depth fusion factor will be... S fusion Total pixels of the currently extracted region S extract Divide them to obtain the depth extraction factor of the current feature extraction region. F 1 extract The formula is expressed as: .

[0049] S403. First, determine whether the current input contains supplementary modal information units to determine the processing method for subsequent feature fusion strategies. If the modal information is acquired from the right eye image, the system executes the following processing flow: This invention uses the left eye unit of a binocular camera as the input image to the segmentation network, and the right eye unit is defined as the auxiliary image input (the image acquired by the right eye). The left and right images acquired by the binocular cameras are processed through stereo matching to obtain a binocular disparity map, which represents the horizontal displacement difference of objects in the same scene between the left and right eye images. To improve matching robustness, the system aggregates costs within a sliding window and optimizes the path consistency of the cost volume using a semi-global matching (SGM) algorithm, thereby obtaining the optimal disparity result with global coherence. The binocular camera estimates the scene depth by calculating the pixel disparity between the left and right views. After camera calibration, the intrinsic and extrinsic parameters of the left and right cameras can be obtained, which are used to convert image pixel coordinates into three-dimensional coordinates. However, for distant objects, the depth estimation error increases significantly due to the very small disparity value. To accurately estimate the correspondence between pixels, fine matching of pixels in the left and right images is still required. Typically, a matching cost function is used to evaluate the similarity between two pixels; a smaller cost indicates a better match and higher similarity, thus helping to improve the accuracy of depth estimation.

[0050] For each pixel, select the disparity value with the lowest cost as the output binocular disparity map: ; in, P ( x , y ) represents the output binocular disparity map, with displacement in pixels; C ( x , y,d Calculate different disparity values ​​for the sliding window. d The matching cost function below, argmin This indicates the value of the independent variable required to minimize the objective function. For example... Figure 5 As shown, the system first applies a semantic mask. M ( x , y )( Figure 5 (f) and binocular disparity map P ( x , y Spatial alignment is performed. Through this alignment process, the system obtains a preliminary fusion of semantic category information and disparity information, specifically defined as the semantic-disparity joint effective region Ω2: Ω2={( x , y )|M( x , y )= c ∧ P ( x , y )>0}; Here, Ω2 is a set of pixels that simultaneously possess effective dynamic categories and effective disparity values. This joint region is used as a seed region for the next step of structural analysis and disparity diffusion, in order to further enhance the structural consistency modeling and dynamic region discrimination capabilities within the region.

[0051] S404. Further, based on step S403, the present invention uses the disparity value of each pixel in the semantic-disparity joint effective region Ω2 as the initial seed for structural diffusion, and performs a local disparity consistency diffusion operation. Specifically, for the seed point ( x 0', y For each 0')∈Ω2, disparity values ​​are diffused within its spatial neighborhood, and the diffusion range is controlled according to the local disparity similarity criterion: | P ( x , y )- P ( x 0', y 0')|< δ 2; in, δ 2 represents the tolerance threshold for disparity differences. In this embodiment, it is recommended to set it to 0.02 (disparity values ​​have been normalized to [0,1]), which represents the range of disparity consistency.

[0052] Furthermore, a disparity diffusion weighting function is defined for each neighboring pixel that satisfies the local disparity similarity criterion: ; in, For disparity diffusion weights, σ 2 represents the sensitivity coefficient of the weighting function, which is recommended to be set to 0.73 in this embodiment to maintain consistency with the depth diffusion parameter, ensuring sensitivity to subtle structural differences while maintaining smooth diffusion. Next, the system calculates the disparity diffusion consistency score for all pixels satisfying the local disparity similarity criterion. S’ ( x , y The total semantic disparity fusion factor is obtained by adding the pixel area of ​​Ω2 to Ω2. S’ fusion : ; To further achieve dynamic filtering of feature points at the region level, the above total semantic disparity fusion factor will be used. S’ fusion Total pixel area of ​​the current region S extract Divide to define the stereo extraction factor for the current feature extraction region. F 2extract : ; S405. First, determine whether the current input contains supplementary modal information units to determine the processing method for subsequent feature fusion strategies. If there is no supplementary modal information input, it is in monocular camera unit mode, and the following operations are performed: In the case of a monocular camera, since there is no additional spatial structure information such as depth maps or disparity maps, the system relies solely on semantic masks. M ( x , y The semantic region information provided. Let the mask region of the current dynamic category be: Ω mask ={( x , y )| M ( x , y )= c}; Ω mask This represents the set of all semantically valid pixel locations in the current region. The system directly defines the monocular extraction factor for the current feature extraction region. F 3 extract for: .

[0053] S5, such as Figure 6 As shown, based on the obtained extraction factor, the current feature extraction region is subjected to gating threshold suppression, and the effective set of feature points is dynamically extracted, namely edge feature points (feature points at the junction of the edge of the dynamic object and the static scene).

[0054] Furthermore, in step S5, the region gate threshold for the extracted region under the current pyramid scaling ratio is calculated based on the extraction factor obtained in step S4, specifically as follows: Extracting factors based on the characteristics of each region F extract Calculate the regional gating threshold The specific calculation formula is as follows: ; Among them, extraction factors F extract Including deep extraction factors F 1 extract Binocular extraction factor F 2 extract and monocular extraction factors F 3 extract ;SFactor This represents a predefined system scaling factor used to adjust the sensitivity of the threshold. In this embodiment, it is recommended to set it between 0.5 and 1.5, but the specific value can be adjusted according to the needs of the scenario; ln is the natural logarithm. Subsequently, the region gating threshold is... Perform a limiting judgment to determine whether it exceeds the upper limit of the threshold (set to 0.95 in this embodiment), and then execute the following steps: If the regional gate threshold If the value is greater than or equal to the upper limit of the threshold, the system considers the feature point extraction requirements for the current region to be too high, and the region may lack stable features. Therefore, the feature point extraction process for this region is skipped, and no further calculations are performed.

[0055] If the regional gate threshold If the threshold is less than the upper limit, the regional gating threshold is further optimized and adjusted to obtain the final gating threshold. θ r The formula is expressed as: θ r =[2- max (0.05, )], max (•) is the maximum value operation; The above formula ensures the gating threshold of the region. θ r The threshold should fall within a stable range, ensuring that it is not too low, so as to avoid extracting a large number of unstable feature points.

[0056] Using the above-mentioned gating threshold θ r Perform feature point extraction operations using the current scale image and a gate threshold. θ r As input, the weighted feature point set after threshold filtering is obtained in the output region.

[0057] Finally, the system will extract and weight the feature point set, and save it to the feature point set storage unit of the current area for subsequent SLAM localization and mapping processes.

[0058] S6. Track visual feature points and build a map.

[0059] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0060] In another embodiment of this application, a visual SLAM system based on lightweight semantic gating suppression is provided. The system includes a data acquisition module, a real-time semantic segmentation network, a scaling and alignment module, an extraction factor calculation module, a gating threshold suppression module, and a map building module. The data acquisition module is used to acquire image frames and additional modal information of the mapping environment based on different camera sensors mounted on the mobile platform. The real-time semantic segmentation network is used to extract semantic information from the image frames input from the current camera to obtain the corresponding semantic mask. It includes, in sequence, a parallel spatial coding branch and a semantic coding branch, a channel merging module, a convolutional layer, a mapping layer, and an upsampling layer. The spatial coding branch downsamples and compresses the resolution of the image frame using several tensor features and extracts spatial feature tensors. The semantic coding branch includes a four-stage encoder module for extracting feature tensors at different scales and semantic levels. The feature tensors output by the encoder modules in the last three stages of the semantic coding branch are passed through a channel-enhanced dual-attention module, and the semantic segmentation loss at each stage is calculated. These are then weighted to construct the total loss function of the real-time semantic segmentation network. The channel-enhanced dual-attention module includes a parallel spatial attention branch and a channel-enhanced attention branch, used to simultaneously capture information in both the channel dimension and spatial dimension at different semantic levels through spatial attention mechanisms and channel-enhanced attention. The scaling and alignment module is used to perform pyramid-style scaling processing on the acquired current image frame, semantic mask, and additional modal information at different scales, and to perform timestamp alignment. The extraction factor calculation module is used to construct the semantic information of the semantic mask and the set of intersecting regions under different modal information based on the current image frame at the pyramid scale, the semantic mask, and the additional modal information, and to calculate the extraction factor of different feature extraction regions. The gate threshold suppression module is used to perform gate threshold suppression on the current feature extraction region based on the obtained extraction factor, and dynamically extract an effective set of feature points. The map building module is used to track the set of feature points and build a map.

[0061] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above. This system is a visual SLAM method based on lightweight semantic gating inhibition applied to the above embodiments.

[0062] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0063] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A visual SLAM method based on lightweight semantic gating inhibition, characterized in that, Includes the following steps: The image frames and additional modal information of the mapping environment are acquired based on the different camera sensors equipped on the mobile platform. When a monocular camera is used, the input is an image frame without additional modal information. When a binocular camera is used, the input is the image acquired by the left eye, and the image acquired by the right eye is used as additional modal information. When equipped with an RGB-D camera sensor, the input is an image frame, with depth images used as additional modal information; A real-time semantic segmentation network is used to extract semantic information from the image frames input from the current camera, resulting in a corresponding semantic mask. The real-time semantic segmentation network comprises, in sequence, parallel spatial coding and semantic coding branches, a channel merging module, convolutional layers, mapping layers, and upsampling layers. The spatial coding branch downsamples and compresses the resolution of the image frames using several tensor features and extracts spatial feature tensors. The semantic coding branch includes a four-stage encoder module for extracting feature tensors at different scales and semantic levels. The feature tensors output from the last three stages of the encoder module of the semantic coding branch are passed through a channel-enhanced dual-attention module, and the semantic segmentation loss at each stage is calculated. These are then weighted to construct the total loss function of the real-time semantic segmentation network. The channel-enhanced dual-attention module includes parallel spatial attention and channel-enhanced attention branches, used to simultaneously capture information in both channel and spatial dimensions at different semantic levels through spatial attention mechanisms and channel-enhanced attention. The acquired current image frame, semantic mask, and additional modal information are subjected to pyramid-style scaling at different scales, and timestamp alignment is performed. Based on the current image frame at the pyramid scale, the semantic mask, and the additional modal information, a set of the semantic information of the semantic mask and the intersection regions under different modal information are constructed as feature extraction regions, and the extraction factors of different feature extraction regions are calculated. Based on the obtained extraction factor, the current feature extraction region is subjected to gating threshold suppression to dynamically extract an effective set of feature points. Track the set of feature points and build a map.

2. The visual SLAM method based on lightweight semantic gating inhibition according to claim 1, characterized in that, The spatial coding branch receives RGB image frames acquired from the current camera unit in the form of a list, and extracts the feature tensors of the image frames through several tensor feature downsampling. Each tensor feature downsampling includes a downsampling module and a convolution module for learning spatial features. For the nth tensor feature downsampling, it is represented as: T1=Conv 3×3 (Tensor input ); T2=Conv 3×3 (T1); in, Conv is the feature tensor output by the downsampling module. 3×3 For general 3×3 convolution, For the set of real numbers, C n+1 H represents the number of tensor channels downsampled for the nth tensor feature, and H and W represent the spatial resolution of the RGB image frame. input The input feature tensor; The feature tensor output by the convolutional module for learning spatial features.

3. The visual SLAM method based on lightweight semantic gating inhibition according to claim 1, characterized in that, The semantic coding branch receives RGB image frames acquired from the current camera unit in the form of a list; in the four-stage encoder module, the first stage encoder module includes three general 3×3 convolutions stacked sequentially; the specific processes of the latter three stages encoder modules are as follows: In the encoder module of the i-th stage, where i takes the values ​​{2, 3, 4}, the input feature tensor... Perform pointwise convolution, using a general 3×3 convolution for preliminary semantic information extraction, represented as: X1= Conv 3×3 (Conv 1×1 (X)); in, For the set of real numbers, Conv 1×1 For pointwise convolution, Conv 3×3 For general 3×3 convolution, C i-1 H represents the number of tensor channels of the input feature tensor X, and H and W represent the spatial resolution of the RGB image frame. For feature tensors Perform channel separation operation to obtain feature tensor C i Let be the number of tensor channels of the encoder module in the i-th stage; Let feature tensor X2 be passed sequentially through two asymmetric depthwise separable convolutions of 3×1 and 1×3 to learn semantic feature information; and let feature tensor X3 be passed sequentially through two asymmetric depthwise separable convolutions of 1×5 and 5×1, and then sequentially through asymmetric depthwise separable convolutions of 1×5 and 5×1 with dilation to learn semantic features with different receptive fields, as shown below: X4=DConv 1×3 (DConv 3×1 (X2)); X5=DConv 5×1 (DConv 1×5 (X3)); X5'= DDConv 5×1 (DDConv 1×5 (X5)); in, These are the output feature tensors of the two branches, DConv. 3×1 and DConv 1×3 DConv represents two asymmetric depthwise separable convolutions, 3×1 and 1×3. 1×5 and DConv 5×1 DDConv represents two asymmetric depthwise separable convolutions, 3×1 and 1×3. 1×5 and DDConv 5×1 This represents 1×5 and 5×1 asymmetric depthwise separable convolutions with added holes; The feature tensors X4 and X5' are channel-merged, and then compressed through pointwise convolution. After compression, they are added element-wise with the input feature tensor to obtain semantic features with different receptive fields. Finally, the channels are shuffled, as shown below: X6=Conv 1×1 (Cat(X4, X5’))+Tensor input ; X6'=Shuffle(X6); in, Let be the semantic feature tensor output by the encoder module in the i-th stage, and let Cat(•) be the channel merging operation. input The input feature tensor is shuffle(•), and the channel shuffle operation is shuffle(•).

4. The visual SLAM method based on lightweight semantic gating inhibition according to claim 1, characterized in that, The spatial attention branch is specifically as follows: In the spatial attention branch connected to the encoder module of the i-th stage, where i takes the value {2,3,4}, the input feature tensor is processed by pointwise convolution. Perform channel communication: Y1=Conv 1×1 (Y); in, For the set of real numbers, C i Let H be the number of tensor channels of the encoder module in the i-th stage, and let H and W be the spatial resolution of the RGB image frame. 1×1 For pointwise convolution; feature tensor The inputs are fed into two 3×3 dilated convolutions, and the resulting two feature tensors are multiplied by a matrix and then subjected to a softmax operation to calculate the spatial attention matrix Attention1 for the current input feature tensor. Attention1=SoftMax(Mul(DilConv 3×3 (Y1),DilConv 3×3 (Y1))); Where SoftMax(•) is the SoftMax operation, Mul(•) is the matrix multiplication operation, and DilConv... 3×3 (•) represents a 3×3 dilated convolution; The feature tensor is obtained by multiplying the result of pointwise convolution of the feature tensor Y1 with the spatial attention matrix. : Y2=Mul(Conv 1×1 (Y1), Attention1); The feature tensors Y1 and Y2 are element-wise added and then fed into a pointwise convolution. After being added to the feature tensor Y1, a pointwise convolution is performed to obtain the spatial attention-weighted feature tensor. : Y3= Conv 1×1 (Conv 1×1 (Y1+Y2)+Y1); The channel-enhanced attention branch is specifically as follows: In the channel-enhanced attention branch connected to the encoder module of the i-th stage, the input feature tensor is... Perform global averaging through two fully connected layers FC1 and FC2; The fully connected layer FC1 comprises a linear layer, a ReLU activation function, and another linear layer connected in sequence. The dimension of the weight vector is reduced from the number of input channels to 1 / 16 of the number of input channels, and normalized using the Sigmoid function, as follows: F FC1 =sigmoid(liner(relu(liner(Avgpool(Y))))); Among them, F FC1 The output of the fully connected layer FC1 is represented by Avgpool(•), which is the global average, liner(•) is the linear layer, relu(•) is the ReLU activation function, and sigmoid(•) is the Sigmoid function. The fully connected layer FC2 consists of a 3×3 convolutional layer, a ReLU activation function, and another 3×3 convolutional layer connected in sequence. This reduces the dimension of the weight vector from the number of input channels to 1 / 16 of the number of input channels and normalizes it using the Sigmoid function, as follows: F FC2 =sigmoid(conv(relu(conv(Avgpool(Y))))); Among them, F FC2 This is the output of the fully connected layer FC2, and conv(•) is a 3×3 convolutional layer; The outputs of fully connected layers FC1 and FC2 are summed, and then multiplied one by one with the input feature maps to obtain the output feature map F of enhanced channel interest. CA , represented as: F CA =Conv 1×1 (F FC1 +F FC2 )×Y; The output feature map F of the enhanced channel focus CA The feature tensor Y3 obtained by spatial attention weighting from the spatial attention branch is added to obtain the final output of the channel-enhanced dual attention module connected to the encoder module of the i-th stage.

5. The visual SLAM method based on lightweight semantic gating inhibition according to claim 1, characterized in that, The encoder modules in the last three stages of the semantic coding branch introduce multi-stage auxiliary loss calculation mechanisms to calculate the semantic segmentation loss in stages, specifically: L total =α2 L2+α3 L3+α4 L4+L main ; Among them, L total The total loss is represented by L2, L3, and L4, which are the losses of the encoder modules in the second, third, and fourth stages, respectively, with corresponding weights of α2, α3, and α4. main The final output of the real-time semantic segmentation network is represented by the main loss calculated using the main classification head and the true labels. Let the label corresponding to the input image frame be... Let represent the semantic category label of each pixel, and let the prediction output by the encoder module in the i-th stage ({2,3,4}) be . C i Let be the number of tensor channels of the encoder module in the i-th stage, i.e., the number of semantic categories. Let H be the set of real numbers; H and W be the spatial resolution of the RGB image frames. Then, the cross-entropy loss for the i-th stage is defined as follows: ； in, This is an indicator function, which is 1 when the true label of pixel (x,y) is the dynamic category c, and 0 otherwise; This represents the predicted probability that pixel (x,y) belongs to dynamic category c, and log(•) is the logarithmic function; The feature tensor F used to calculate the main loss Lmain main Represented as: F main =Project(ConvBlock 3×3 (Cat(F seg ,F sp ))); Where Project(•) represents the feature mapping operation of the mapping layer, ConvBlock 3×3 (•) represents the function operation of the fusion module, Cat(•) represents the channel merging operation, and F seg For the semantic encoding branch, F sp For spatial encoding branch; the feature tensor F main The final output of the real-time semantic segmentation network is obtained after upsampling.

6. The visual SLAM method based on lightweight semantic gating inhibition according to claim 1, characterized in that, When the added modal information is a depth image, the semantic information of the constructed semantic mask and the set of intersecting regions under different modal information are used to calculate the extraction factors of different feature extraction regions, specifically: Spatial alignment of the semantic mask with the depth image; Based on the aligned regions, the depth information of the matched semantic labels is diffused or propagated in the spatial neighborhood to cover more unlabeled but structurally consistent neighboring regions, represented as: Ω1={(x,y)|M(x,y)=c∧D(x,y)>0}; Where Ω1 represents the intersection region of the semantic mask M(x,y) and the depth image D(x,y), serving as the seed region for depth diffusion; c represents the dynamic category; For a seed point (x0, y0) ∈ Ω1, the direction of its neighborhood is... Diffusion semantic tags, with the condition for deep diffusion being: |D(x,y)- D(x0,y0)|<δ1; Where δ1 is the set depth difference tolerance threshold; Define a semantic diffusion weight function for pixels that satisfy the depth diffusion condition: ； in, σ1 represents the semantic diffusion weight, exp(•) is the exponential function, and σ1 is the sensitivity coefficient of the semantic diffusion weight function. The semantic score S(x,y) of the diffusion is calculated using the semantic diffusion weight, and then added to the pixel area of ​​Ω1 to obtain the total semantic depth fusion factor S. fusion , represented as: ； ； in, The indicator function is 1 if the pixel (x,y) and the seed pixel satisfy the diffusion condition, and 0 otherwise; total(•) is the total operation. The total semantic depth fusion factor S fusion Compared with the total pixels S of the currently extracted region extract Divide by the product to obtain the depth extraction factor F of the current feature extraction region. 1 extract , represented as: 。 7. The visual SLAM method based on lightweight semantic gating inhibition according to claim 1, characterized in that, When the additional modal information is the right-eye acquired image, the semantic information of the constructed semantic mask and the set of intersecting regions under different modal information are used to calculate the extraction factors of different feature extraction regions, specifically: The left eye unit of the binocular camera is used as the input image, and the image acquired by the right eye is defined as the auxiliary image input; Stereo matching calculations are performed between the input image and the auxiliary image to obtain a binocular disparity map; Costs are aggregated within a sliding window, and path consistency is optimized on the cost volume using a semi-global matching algorithm to obtain the optimal disparity result with global coherence. For each pixel, the disparity value with the lowest cost is selected as the output binocular disparity map. ； Where P(x,y) represents the output binocular disparity map, with displacement in pixels; C(x,y,d) is the matching cost function calculated by the sliding window for different disparity values ​​d; argmin represents the value of the independent variable required to minimize the objective function. The semantic mask M(x,y) and the binocular disparity map P(x,y) are spatially aligned and defined as the semantic-disparity joint effective region Ω2: Ω2={(x,y)|M(x,y)=c∧P(x,y)>0}; Where Ω2 is the set of pixels that simultaneously have effective dynamic categories and effective disparity values, and c is the dynamic category; Using the disparity value of each pixel in the semantic-disparity joint effective region Ω2 as the initial seed for structural diffusion, a local disparity consistency diffusion operation is performed, specifically as follows: For a seed point (x0', y0') ∈ Ω2, disparity values ​​are diffused within its spatial neighborhood, and the diffusion range is controlled according to the local disparity similarity criterion: |P(x,y)-P(x0', y0')|<δ2; Wherein, δ2 represents the tolerance threshold for disparity differences; Define a disparity diffusion weight function for each neighboring pixel that satisfies the local disparity similarity criterion: ； in, σ² represents the disparity diffusion weight, exp(•) is the exponential function, and σ² is the sensitivity coefficient of the disparity diffusion weight function. For all pixels satisfying the local disparity similarity criterion, calculate the disparity diffusion consistency score S'(x,y) of the region, and add it to the pixel area of ​​Ω2 to obtain the total semantic disparity fusion factor S'. fusion : ； The total(•) function performs a totaling operation. The total semantic disparity fusion factor S' fusion With the total pixel area S of the current region extract Divide by the product to define the stereo extraction factor F for the current feature extraction region. 2 extract : 。 8. The visual SLAM method based on lightweight semantic gating inhibition according to claim 1, characterized in that, If the current input does not contain additional modal information, i.e., the input is a monocular camera unit, the semantic information for constructing the semantic mask and the set of intersecting regions under different modal information are used, and the extraction factors for different feature extraction regions are calculated, specifically as follows: The mask region for the current dynamic category is defined as follows: Ω mask ={(x,y)|M(x,y)=c}; Ω mask M(x,y) represents the set of all semantically valid pixel locations in the current region, where M(x,y) is the semantic mask and c is the dynamic category. Define the monocular extraction factor F for the current feature extraction region. 3 extract for: total(•) is the total operation, S extract This represents the total pixel area of ​​the current region.

9. The visual SLAM method based on lightweight semantic gating inhibition according to claim 1, characterized in that, The gating suppression based on the obtained extraction factor on the current feature extraction region threshold is specifically as follows: Based on the extraction factor F for each feature extraction region extract Calculate the regional gating threshold , represented as: ； Among them, the extraction factor F extract This includes monocular extraction factor, binocular extraction factor, and depth extraction factor; SFactor represents a predefined system scaling factor used to adjust the sensitivity of the threshold; ln is the natural logarithm. Regional gating threshold Perform a restrictive check to determine if it exceeds the set threshold limit, and then execute the following steps: If the regional gate threshold If the value is greater than or equal to the upper threshold, the feature point extraction process for that region is skipped, and no further calculations are performed. If the regional gate threshold If the threshold is less than the upper limit, the regional gating threshold is optimized and adjusted to obtain the final gating threshold θ. r , represented as: θ r =[2-max(0.05, )], max(•) is the maximum value operation; Using the above-mentioned gate threshold θ r Perform feature point extraction operation with the current scale image and a gate threshold θ. r As input, we obtain the weighted feature point set in the output region after threshold filtering; Save the weighted feature point set to the feature point set storage unit of the current region.

10. A visual SLAM system based on lightweight semantic gating inhibition, characterized in that, The visual SLAM method based on lightweight semantic gating inhibition applied to any one of claims 1-9 includes a data acquisition module, a real-time semantic segmentation network, a scaling and alignment module, an extraction factor calculation module, a gating threshold inhibition module, and a map construction module; The data acquisition module is used to acquire image frames and additional modal information of the mapping environment based on different camera sensors mounted on the mobile platform. The real-time semantic segmentation network is used to extract semantic information from the image frames input by the current camera to obtain the corresponding semantic mask. It includes the following components connected in sequence: a parallel spatial coding branch and a semantic coding branch, a channel merging module, a convolutional layer, a mapping layer, and an upsampling layer. The spatial coding branch downsamples and compresses the resolution of the image frame using several tensor features and extracts spatial feature tensors. The semantic coding branch includes a four-stage encoder module for extracting feature tensors at different scales and semantic levels. The feature tensors output by the encoder modules of the last three stages of the semantic coding branch are respectively processed by a channel-enhanced dual attention module, and the semantic segmentation loss of each stage is calculated. The weighted sums are used to construct the total loss function of the real-time semantic segmentation network. The channel-enhanced dual attention module includes a parallel spatial attention branch and a channel-enhanced attention branch, which are used to simultaneously capture information in the channel dimension and spatial dimension at different semantic levels through spatial attention mechanism and channel-enhanced attention. The scaling and alignment module is used to perform pyramid-style scaling processing on the acquired current image frame, semantic mask and additional modal information at different scales, and to perform timestamp alignment. The extraction factor calculation module is used to construct the semantic information of the semantic mask and the set of intersecting regions under different modal information based on the current image frame at the pyramid scale, the semantic mask, and the additional modal information, and to calculate the extraction factor of different feature extraction regions. The gate threshold suppression module is used to perform gate threshold suppression on the current feature extraction region based on the obtained extraction factor, and dynamically extract an effective set of feature points. The map building module is used to track the set of feature points and build a map.

Images