An object-level semantic visual slam method and system based on a hybrid attention mechanism target detection network and an ellipsoid model

Abstract

The application relates to the field of computer vision and mobile robot navigation, and discloses an object-level semantic visual SLAM method and system based on a hybrid attention mechanism target detection network and an ellipsoid model. The system comprises five parallel thread modules of semantic perception, visual tracking and repositioning, local mapping and fusion, loop detection and global consistency optimization. The application extracts local and global features of an image in parallel through a target detection network, and outputs high-precision semantic observations; when tracking is lost, a 2D inscribed ellipse-3D object ellipsoid dual geometric constraint is used to realize fast repositioning in cooperation with a P3P algorithm and an IoU cost function. In the mapping process, a Gaussian-Wasserstein distance is introduced to measure semantic re-projection errors, and a joint objective function containing map point geometric errors and object semantic errors is constructed to perform bundle adjustment. The application effectively solves the problem of feature extraction failure in motion blur and weak texture scenes, and significantly improves the construction accuracy of a semantic map and the survival ability of the system.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and autonomous navigation technology for mobile robots, specifically to an object-level semantic visual SLAM method and system based on a hybrid attention mechanism target detection network and an ellipsoidal model. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) is the core foundation for mobile robots to achieve autonomous navigation and intelligent interaction. Traditional visual SLAM systems mainly rely on low-level geometric features such as points and lines for pose estimation. However, in practical applications, mobile robots often face "observation degradation" environments such as low texture (e.g., white walls, dimly lit corridors), motion blur, and dynamic occlusion. In such unstructured scenarios, traditional visual SLAM systems face severe challenges:

[0003] First, there are limitations and semantic gaps in feature extraction. Traditional systems rely on low-level pixel features, which are difficult to extract stably or prone to mismatches in degraded observation scenarios, leading to easy loss of tracking capabilities. Furthermore, the sparse point cloud maps they construct lack high-level semantic descriptions of scene object attributes (such as category and scale), making it difficult to support robots in performing advanced semantic tasks.

[0004] Secondly, object detection suffers from insufficient accuracy and adaptability in SLAM applications. Object-level SLAM schemes rely on object detection networks to acquire object observations, but existing algorithms have significant drawbacks: traditional convolutional neural networks (CNNs) focus on local receptive fields, making it difficult to capture global contextual information; while Transformer-based architectures can establish long-range dependencies, they lack in extracting local detailed features. This limitation in feature extraction capabilities leads to low accuracy of object bounding boxes output by the system in complex backgrounds, resulting in significant pose and scale errors in the object ellipsoid model constructed in the backend, making it difficult to serve as a reliable landmark in system optimization.

[0005] Third, the robustness and survivability of relocalization are relatively weak. Once visual tracking is lost due to observation degradation, existing relocalization methods based on bag-of-words models or pure geometric feature matching often fail. Existing object-level SLAM research focuses more on semantic enrichment of maps, failing to explore the potential of object models in terms of geometric constraints, and neglecting to use macroscopic object-level geometric constraints (such as view-invariant ellipsoidal projection constraints) to assist camera pose recovery when features are missing.

[0006] Finally, the limitations of constraint metrics in backend optimization. In object-level model optimization, the traditional Intersection over Union (IoU) cost function suffers from problems such as gradient discontinuity and loss of constraint effectiveness in non-overlapping states. This leads to slow convergence speed and low accuracy of object parameters in actual mapping, making it difficult to achieve deep joint optimization of geometric features and semantic objects.

[0007] In summary, existing technologies have shortcomings in the following aspects, which limit the performance of mobile robots in complex open environments:

[0008] (1) Feature fusion capability of the sensing end;

[0009] (2) Observe the relocation survival ability under degraded environment;

[0010] (3) There are still significant shortcomings in the accuracy of the back-end semantic-geometric joint optimization. Summary of the Invention

[0011] The purpose of this invention is to provide an object-level semantic visual SLAM method and system based on a hybrid attention mechanism target detection network and an ellipsoidal model, which is designed for observation degradation scenarios to enable robust localization, rapid relocalization, and high-precision semantic map construction for robots in complex environments.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] An object-level semantic visual SLAM system based on a hybrid attention mechanism object detection network and an ellipsoidal model is presented. The system is a SLAM system based on an improved ORB-SLAM2 architecture and includes five parallel thread modules.

[0014] The semantic awareness module includes a detection thread, which contains a target detection network based on a hybrid attention mechanism. This network performs real-time inference on the acquired image sequences and outputs semantic observation results that include object category labels, two-dimensional bounding box information, and confidence scores, providing high-precision semantic constraints for backend optimization.

[0015] The visual tracking and relocalization module includes a tracking thread and a relocalization unit. The tracking thread uses feature points and semantic observation results to jointly perform data association and track the camera pose of each frame of image. When tracking loss is detected, the relocalization unit uses a 3D object ellipsoid model to guide the matching of feature points in the current frame with global map points, thereby estimating the accurate camera pose to restore tracking.

[0016] Local mapping and fusion module: This includes a mapping thread, which comprises a mapping management unit and a local joint optimization unit. The mapping management unit includes functions for fusing map point mapping and object mapping, used to manage the updating and maintenance of local maps, estimate object ellipsoid models in 3D space and insert them into the map, and perform model fusion for the same physical entity. The local joint optimization unit includes constructing a joint objective function that includes the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid model, and performing joint bundle adjustment to simultaneously optimize local keyframe poses, map point coordinates, and object ellipsoid parameters.

[0017] Loop closure detection module: Includes a loop closure detection thread, which performs similarity retrieval on keyframes based on the DBoW2 bag-of-words model; when a loop closure is detected, it performs essence graph optimization to correct the cumulative error of the camera motion trajectory.

[0018] Global Consistency Optimization Module: This module includes a global bundle adjustment thread. After the loop closure detection module completes the essential graph optimization, the global bundle adjustment thread performs global bundle adjustment on the sparse semantic map structure maintained by the system to obtain a globally consistent camera pose distribution and object-level map structure.

[0019] An object-level semantic visual SLAM method based on a hybrid attention mechanism object detection network and an ellipsoidal model includes the following steps:

[0020] S1. Semantic Observation Extraction: Start the detection thread and capture images of the scene using a camera to obtain an image sequence; input the image sequence into a pre-built target detection network to obtain semantic observation data of the target object; the target detection network performs local feature extraction and global feature extraction based on a multi-head self-attention mechanism in parallel, and fuses the extracted local features with the global features, and then decodes the local features and global features to obtain semantic observation data containing object category labels and two-dimensional detection bounding boxes.

[0021] S2. Tracking and Relocalization: While running in step S1, the tracking thread performs camera pose tracking on the image sequence obtained in step S1. When camera tracking loss is detected, the relocalization process is initiated using the semantic observation data output in step S1. An inscribed ellipse is constructed based on the 2D detection bounding box of the semantic observation data. In the global map containing sparse map points and 3D object ellipsoid models, the same type of 3D object ellipsoid model is retrieved. The initial estimated pose is then filtered again using the P3P algorithm and the IoU cost function. Finally, the initial estimated pose is used to guide the matching of the feature points of the current frame with the map points in the global map, and the system tracking is restored.

[0022] S3. Object Mapping and Fusion: While running step S1, a mapping thread is run. Based on the semantic observation data obtained in step S1 and the camera pose recovered in step S2, a new 3D object ellipsoid model is initialized in 3D space and inserted into the global map. A semantic reprojection error function based on Gaussian-Wasserstein distance is constructed to optimize the redundant object models generated during initialization. Inclusion and fusion checks are performed to obtain a single ellipsoid model and update the global map.

[0023] S4. Local Joint Optimization: While running in steps S1 and S3, in the mapping thread, keyframe sequences are selected from the image sequence; a joint objective function is established that includes the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid model; and local bundle adjustment is performed on the keyframe sequence through the joint objective function to optimize and synchronously update various parameters in the visual tracking task, including camera pose, map point coordinates and object ellipsoid model parameters; redundant keyframes are removed to obtain the optimized keyframe sequence.

[0024] S5. Global Consistency Optimization: Run the loop closure detection thread and the global optimization thread to perform loop closure detection on the optimized keyframe sequence obtained in step S4; after confirming the loop closure, perform pose graph optimization and global bundle adjustment to eliminate accumulated errors and finally generate a globally consistent object-level semantic 3D scene map.

[0025] Preferably, the specific process of semantic observation extraction in step S1 includes:

[0026] First, build the object detection network. This includes:

[0027] Network architecture construction: The object detection network is based on the encoder-decoder model, where the encoder consists of a word embedding vector encoding module (Patch-PositionEmbed) and a feature extraction and fusion interaction module (TransformerBlock), constructing a 4-stage feature hierarchy to generate feature maps at different scales.

[0028] The construction of the overall network reasoning process is mainly divided into four stages:

[0029] In the first stage, the input image is initially processed by the Encoder's convolutional stemming module (Conv-stem). The Conv-stem consists of two... The convolutional layers consist of a batch normalization layer and a Hardwish activation function, which are applied after each convolution to generate a high-dimensional feature map of the image. = .

[0030] In stages 2 through 4, each stage stacks Patch-PositionEmbed and TransformerBlock. Patch-PositionEmbed processes the high-dimensional features output from stage 1 into embedding vectors through patch embedding and position embedding. TransformerBlock takes the local detail multi-scale features L extracted by the CNN and the global information T extracted based on a multi-head self-attention mechanism, and inputs them into the IntegratedInteractionModule. This module, based on a channel attention mechanism, dynamically adjusts the weights of different channels in the feature map to enhance the feature representation of important channels, thereby outputting the fused interactive feature A. Feature A is input into the Decoder for information decoupling to obtain the semantic observation data of the object.

[0031] The semantic observation data are:

[0032]

[0033] in, Represents a two-dimensional detection bounding box. Indicates object category labels, Indicates the confidence level.

[0034] S11. Obtain image sequence: Capture the scene with a camera to obtain an image sequence;

[0035] S12, Feature Extraction: The image sequence obtained in step S11 is processed by the object detection network to extract the local features L and global context information T of the object in the image sequence;

[0036] S13. Stitching and Preliminary Fusion: The local features obtained in step S12 are then stitched together. and global context information Stitching along the channel dimension, through Convolutional layers achieve feature fusion, and batch normalization is performed to obtain fused features. The specific process is as follows:

[0037]

[0038] in, This indicates splicing along the channel dimension. This represents matrix multiplication. express The convolutional layer weights, Indicates the batch normalization layer. This represents the LeakyReLU activation function;

[0039] S14, Channel Attention Interaction: The fused features obtained in step S13 are... Perform global average pooling, then pass through two... The convolutional layer encodes the fused feature U, and the channel interaction weights V are obtained by combining the LeakyReLU activation function. The specific process is as follows:

[0040]

[0041] Where H and W are the height and width of the fused feature map, and W2 and W3 represent the two... The weights of the convolutional layer, where δ() represents the Sigmoid activation function;

[0042] S15. Dynamic Weighted Output: Multiply the interaction weight V obtained in step S14 and the fusion feature U obtained in step S13 channel by channel to recalibrate and generate the interaction-adjusted edge features; add the fusion feature and the edge calibration feature through residual connection to finally obtain the edge features with fusion feature U and the target. The output feature A is calculated using the following formula:

[0043]

[0044] in, This is an element-wise inner product;

[0045] S16. Accuracy Optimization: The target detection network is trained and optimized based on a composite loss function that includes geometric constraints to improve the regression accuracy of observation data in scenarios with observation degradation.

[0046] Preferably, the specific process of accuracy optimization in step S16 includes:

[0047] The object detection network is trained and optimized based on a composite loss function that includes geometric constraints to improve the regression accuracy of observation data in scenarios with observation degradation. The specific process includes:

[0048] The composite loss function L is composed of a weighted average of the classification loss function, the regression loss function, and the confidence loss function, expressed as:

[0049]

[0050] in, , and These represent the weight coefficients corresponding to each loss function;

[0051] Classification loss function L c It is based on the modification of cross-entropy loss, expressed as:

[0052]

[0053] in, This represents the model's predicted probability for the current category. This represents a factor that weights the categories. This indicates the hyperparameters for adjusting the difficulty level of samples;

[0054] Regression loss function L l The MDIoU loss function is used to avoid the degradation and failure of the CIoU loss function when the center points of the predicted bounding box and the labeled bounding box coincide. It is expressed as:

[0055]

[0056] Where h and w represent the height and width of the image, respectively. , These represent the coordinates of the top-left and bottom-right points of the predicted bounding box, respectively. , These represent the coordinates of the top-left and bottom-right points of the actual labeled bounding box, respectively.

[0057] Confidence loss function L o A binary cross-entropy loss function is used to evaluate the deviation between the model's predicted probability of containing the target object and the true label, expressed as:

[0058]

[0059] in, This indicates the category label detected by the target detection result. This indicates the probability that the bounding box contains the target;

[0060] Preferably, the specific process of tracking and relocation in step S2 includes:

[0061] S21, Camera pose tracking: While running step S1, the tracking thread runs to perform camera pose tracking on the image sequence;

[0062] S22. Conditional Judgment: During step S21, when the system detects that the number of feature points matching the current frame and the local map is lower than the threshold, the relocalization process is activated; the object detection network is used to obtain the category information of the objects in the current frame and the 2D detection bounding box, and an inscribed 2D ellipse is constructed.

[0063] S23, 2D-3D Matching: Retrieve a 3D object ellipsoid model in the global map dynamically maintained by the system that matches the category label of the object detected in the current frame, and match it with the 3D object ellipsoid models already established in the global map according to the category to establish a 2D-3D matching pair; the global map includes sparse map points and 3D object ellipsoid models.

[0064] S24. Verification of initial pose solution set: Select the smallest set containing three pairs of "ellipse-ellipsoid" from the 2D-3D matching pairs obtained in step S23, and use the P3P algorithm to calculate the camera pose based on the center points of the ellipse and ellipsoid to obtain four sets of output pose solutions.

[0065] S25. Optimal Pose Solution Selection: To select the unique true pose from four candidate solutions, this embodiment abandons the traditional reprojection point error (which has a large error when the initial value is inaccurate) and designs a projection cost function based on IoU. The specific steps are as follows:

[0066] Traverse each group of candidate poses Using the dual quadratic surface projection formula, all visible 3D ellipsoids in the map are represented. Project back to the current image plane:

[0067]

[0068] in, Represents a two-dimensional dual conic section projected onto the image plane; Represents the dual quadric surface of a three-dimensional object in the world coordinate system; Indicates the first The camera projection matrix corresponding to the candidate poses, and ; Represents the camera intrinsic parameter matrix; and They represent the first The rotation matrix and translation vector of the candidate solutions.

[0069] Subsequently, the projected ellipse is obtained. circumscribed rectangle And construct the total cost function. :

[0070]

[0071] in, Indicates the first The total cost of reprojection of candidate poses; This indicates the total number of object matching pairs participating in the matching in the current frame; For matching index pairs; Indicates the first The two-dimensional bounding rectangle generated by projecting a three-dimensional ellipsoid of an object into the current candidate pose; Indicates the first The actual two-dimensional bounding boxes detected for each object in the current frame; This function calculates the intersection-union ratio of two rectangles; its value is the ratio of the area of ​​the intersection to the area of ​​the union of the two rectangles.

[0072] Finally, select The pose that is the smallest and less than a preset threshold (e.g., 0.6) is taken as the optimal initial estimated pose. .

[0073] S26. Feature point matching verification: The map points of the previous keyframe are projected onto the current frame using the initial estimated pose to guide the matching; if the number of matching points exceeds the preset threshold, the PnP algorithm is used to further optimize the camera pose and restore system tracking; otherwise, the initial estimated pose is used directly as the pose of the current frame.

[0074] Preferably, the specific process of constructing an optimization function based on semantic reprojection error in step S3, object mapping and fusion, includes:

[0075] The projection ellipse is obtained by projecting the 3D object ellipsoid model onto the image plane, and the inscribed ellipse of the detection box in the semantic observation data is obtained.

[0076] The Gaussian-Wasserstein distance is used to measure the distributional difference between the projected ellipse and the detected inscribed ellipse, and its error term is defined as:

[0077]

[0078] The first An object ellipsoid Projected to the The semantic reprojection error function of the frame image plane is expressed as:

[0079]

[0080] in, Indicates the first The first frame The inscribed ellipse of each target detection bounding box (as an observation). The coordinates of the center of this ellipse. The geometric center of the bounding box, the major semi-axis The minor axis is half the height of the bounding box. Half the width of the bounding box, direction angle Initially set to 0. Indicates the first The dual matrix of an object ellipsoidal model. Indicates the first The camera projection matrix of the keyframes, and . Represents the dual ellipsoid via camera projection matrix After projection, the two-dimensional dual conic section (i.e., the projected ellipse) formed on the image plane is used to calculate its corresponding Gaussian distribution parameters. . Indicates the first The first frame The detection confidence score of each object is used to adjust the impact of different observation qualities on optimization. This represents the squared Gaussian-Wasserstein distance as defined above.

[0081] Preferably, the specific process of establishing a joint objective function that includes the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid model in step S4 local joint optimization includes:

[0082] Establish a standard map point geometric reprojection error function For the first The first keyframe observed For each map point, the error is defined as:

[0083]

[0084] in, These are the observed coordinates of the feature points on the image; For camera projection function; For the first The camera pose of the frame; For the first The three-dimensional coordinates of a map point; For Huber robust kernel function;

[0085] Directly call the semantic reprojection error function obtained in step S3; and project the camera projection matrix. Substituting and expanding, the semantic reprojection error function of a single frame is expressed as:

[0086]

[0087] in, Indicates the first The first frame Gaussian distribution parameters of the inscribed ellipse of the target detection box; Indicates the first An ellipsoidal model of an object Camera position Gaussian distribution parameters formed by downward projection; This represents the detection confidence score; the error function uses the Gaussian-Wasserstein distance to measure the positional differences between the observed ellipse and the projected ellipse. ,shape and direction Overall differences;

[0088] The joint objective optimization function is constructed by linearly weighting and summing the map point geometric reprojection error function and the ellipsoidal semantic reprojection error function according to a preset weight ratio. :

[0089]

[0090] The beneficial effects of this invention are:

[0091] This invention establishes long-range dependencies between pixels by introducing Transformer branches and multi-head self-attention mechanisms in parallel, and designs a channel interaction fusion module. This "global-local" complementary architecture effectively solves the problem of feature extraction failure in motion blur scenes, significantly improves the robustness of front-end perception, breaks through the receptive field limitation of local feature extraction, and solves the problem that traditional CNNs are limited by the local receptive field when detecting large-scale objects or blurred boundaries.

[0092] This invention abandons unstable point feature constraints and instead utilizes object-level ellipsoidal constraints with viewpoint invariance. Through a cascaded selection strategy of P3P geometric solution and IoU cost function, even in extremely degenerate environments with very few feature points, it can achieve millisecond-level fast pose recovery using sparse object landmarks, solving the problem that existing SLAM systems are prone to relocalization failure in weakly textured regions in texture-deficient scenarios.

[0093] This invention innovatively introduces the Gaussian-Wasserstein distance metric to measure semantic reprojection error. Compared to traditional IoU metrics, the Gaussian-Wasserstein distance is more sensitive to object position and shape deviations, and can still provide effective optimization gradients when the predicted and observed boxes do not overlap at all, avoiding the nonlinear optimization process from getting trapped in local minima. The jointly optimized map not only possesses geometric consistency but also contains accurate object semantic parameters, providing data support for advanced robot interactions and constructing a high-dimensional semantic optimization field with gradient smoothness. Attached Figure Description

[0094] Figure 1 This is a schematic diagram of the overall system structure of Embodiment 1 of the present invention;

[0095] Figure 2This is the overall flowchart of Embodiment 2 of the present invention;

[0096] Figure 3 This is a schematic diagram of the overall architecture of the target detection network in Embodiment 2 of the present invention;

[0097] Figure 4 This is a schematic diagram of the specific structure and calculation process of feature fusion interaction in Embodiment 2 of the present invention;

[0098] Figure 5 This is a schematic diagram of the depth-separable residual block structure in Embodiment 2 of the present invention;

[0099] Figure 6 This is a performance comparison chart of the target detection network in Example 2 of the present invention;

[0100] Figure 7 This is a flowchart of robust relocation based on an object ellipsoid model in Embodiment 2 of the present invention;

[0101] Figure 8 This is a diagram illustrating the SLAM semantic graph construction effect in Embodiment 2 of the present invention.

[0102] Figure 9 This is a comparison chart of pose trajectories estimated by different SLAM systems. Detailed Implementation

[0103] Example 1

[0104] The following is a further explanation of the present invention in conjunction with specific embodiments, such as... Figure 1 As shown, this embodiment is an object-level semantic visual SLAM system based on a hybrid attention mechanism object detection network and an ellipsoidal model. The system is a SLAM system based on an improved ORB-SLAM2 architecture. To meet the real-time inference requirements of the hybrid attention mechanism object detection network and the computational load of large-scale joint optimization in the backend, the system is divided into five concurrently running thread modules in its engineering implementation:

[0105] The semantic awareness module includes a detection thread, which contains a target detection network based on a hybrid attention mechanism. This network performs real-time inference on the acquired image sequences and outputs semantic observation results that include object category labels, two-dimensional bounding box information, and confidence scores, providing high-precision semantic constraints for backend optimization.

[0106] To avoid blocking the main tracking process due to deep learning inference, the detection thread runs independently and in parallel. Its processing flow is as follows: it receives the original RGB image sequence as input and calls the hybrid attention mechanism object detection network for inference. This thread extracts local texture features and global context information of the image in parallel, and outputs semantic observation data containing object category labels, two-dimensional detection bounding boxes, and confidence scores in real time. This data is then written to a thread-safe buffer for the tracking thread to read.

[0107] The visual tracking and relocalization module includes a tracking thread and a relocalization unit. The tracking thread uses feature points and semantic observation results to jointly perform data association and track the camera pose of each frame of image. When tracking loss is detected, the relocalization unit uses a 3D object ellipsoid model to guide the matching of feature points in the current frame with global map points, thereby estimating the accurate camera pose to restore tracking.

[0108] This module is responsible for estimating the camera pose in real time and determining whether to insert keyframes. The processing flow is as follows:

[0109] Normal tracking: Joint tracking of camera pose and object pose is performed by combining the feature point method (ORB features) with the semantic observations passed in by the detection thread.

[0110] Robust relocalization: When visual tracking loss is detected (e.g., the number of matched feature points is below a threshold), this thread suspends normal tracking and initiates a relocalization process based on the object ellipsoid model. Utilizing the dual geometric constraints of "2D inscribed ellipse - 3D object ellipsoid", the camera pose is quickly calculated using the P3P algorithm and IoU cost function, and feature point matching is guided to restore system tracking.

[0111] Local mapping and fusion module: This includes a mapping thread, which comprises a mapping management unit and a local joint optimization unit. The mapping management unit includes functions for fusing map point mapping and object mapping, used to manage the updating and maintenance of local maps, estimate object ellipsoid models in 3D space and insert them into the map, and perform model fusion for the same physical entity. The local joint optimization unit includes constructing a joint objective function that includes the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid model, and performing joint bundle adjustment to simultaneously optimize local keyframe poses, map point coordinates, and object ellipsoid parameters.

[0112] This module is responsible for building and optimizing local maps. The processing flow is as follows:

[0113] Object mapping: Process keyframes generated by the tracking thread, initialize new 3D object ellipsoid models in 3D space, and construct semantic error functions using Gaussian-Wasserstein distance; at the same time, detect and fuse redundant ellipsoid models for the same physical entity.

[0114] Joint Bundle Adjustment (LocalBA): Maintains a local map and constructs a joint objective function that includes the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid. Local bundle adjustment is performed on the local keyframe sequence using a nonlinear optimization algorithm (such as Levenberg-Marquardt), simultaneously updating the camera pose, map point coordinates, and object ellipsoid geometric parameters.

[0115] Loop closure detection module: Includes a loop closure detection thread, which performs similarity retrieval on keyframes based on the DBoW2 bag-of-words model; when a loop closure is detected, it performs essence graph optimization to correct the cumulative error of the camera motion trajectory.

[0116] This module runs continuously in the background, responsible for detecting whether the robot has returned to a previously seen scene. Its processing flow is as follows: it uses the DBoW2 bag-of-words model to calculate the similarity between the current keyframe and historical keyframes. Once a loop closure is confirmed, it performs PoseGraphOptimization, calculates the Sim3 transformation to correct accumulated drift, and ensures the geometric closure of the trajectory.

[0117] Global Consistency Optimization Module: This module includes a global bundle adjustment thread. After the loop closure detection module completes the essential graph optimization, the global bundle adjustment thread performs global bundle adjustment on the sparse semantic map structure maintained by the system to obtain a globally consistent camera pose distribution and object-level map structure.

[0118] To avoid prolonged global optimization delays affecting front-end tracking, this thread module is separated into its own thread. The processing flow is as follows: it starts after the loop closure detection thread completes the pose graph optimization. This thread locks the entire map and performs Global Bundle Adjustment (GlobalBA), making fine adjustments to the poses of all keyframes, sparse map points, and all object ellipsoid models in the global map, ultimately generating a globally consistent object-level semantic map.

[0119] Example 2

[0120] This embodiment is based on the object-level semantic visual SLAM system proposed in Embodiment 1, which is a hybrid attention mechanism object detection network and an ellipsoidal model. It implements a SLAM method that fuses hybrid attention detection and the object ellipsoidal model. Figure 2 As shown, it includes the following steps:

[0121] S1. Semantic Observation and Extraction: Start the detection thread, capture images of the scene using the camera to obtain an image sequence; input the image sequence into a pre-built database such as... Figure 3The target detection network shown obtains semantic observation data of the target object. The target detection network performs local feature extraction L and global feature extraction T based on a multi-head self-attention mechanism in parallel, and fuses the extracted local features and global features. Then, it decodes the local features and global features to obtain semantic observation data containing object category labels and two-dimensional detection bounding boxes. The specific process includes:

[0122] S11. Obtain image sequence: Capture the scene with a camera to obtain an image sequence.

[0123] S12. Feature extraction: The image sequence obtained in step S11 is processed by the object detection network to extract the local features L and global context information T of the object in the image sequence.

[0124] Among them, the target detection network architecture is as follows Figure 3 As shown, the setup process includes:

[0125] Network architecture construction: The object detection network is based on the encoder-decoder model, where the encoder consists of a word embedding vector encoding module (Patch-PositionEmbed) and a feature extraction and fusion interaction module (TransformerBlock), constructing a 4-stage feature hierarchy to generate feature maps at different scales.

[0126] The construction of the overall network reasoning process is mainly divided into four stages:

[0127] In the first stage, the input image is initially processed by the Encoder's convolutional stemming module (Conv-stem). The Conv-stem consists of two... The convolutional layers consist of a batch normalization layer and a Hardwish activation function, which are applied after each convolution to generate a high-dimensional feature map of the image. = .

[0128] In stages 2 through 4, each stage stacks Patch-PositionEmbed and TransformerBlock. Patch-PositionEmbed processes the high-dimensional features output from stage 1 into embedding vectors through patch embedding and position embedding. TransformerBlock takes the local detail multi-scale features L extracted by the CNN and the global information T extracted based on the multi-head self-attention mechanism, and inputs them together into the IntegratedInteractionModule. Its structure and computational flow are as follows: Figure 4 As shown, the feature fusion interaction module first performs a convolutional transformation on the input local feature L to obtain a feature map U. Then, global average pooling is applied to U to extract channel statistics, and this is concatenated and fused with the global feature T. The fused feature is then processed through a series of fully connected layers, LeakyReLU activation function, and sigmoid function to generate channel attention weights. Finally, these weights are multiplied channel-by-channel with the feature map U to output the enhanced fused feature A. The entire process dynamically adjusts the importance of different channels through a channel attention mechanism, achieving effective interaction between local details and global semantics. Feature A is input to the Decoder for information decoupling to obtain semantic observation data of the object.

[0129] The semantic observation data are:

[0130]

[0131] in, Represents a two-dimensional detection bounding box. Indicates object category labels, Indicates the confidence level.

[0132] The method for extracting local detail multi-scale features L in step S12 specifically includes:

[0133] Design a depth-separable residual block structure, such as Figure 5 As shown, in order to effectively represent local multi-scale features, and to ensure that the computational cost of the network structure does not affect the real-time running efficiency of the subsequent SLAM system.

[0134] The depthwise separable residual block structure consists of two 1×1 convolutional layers and one 3×3 depthwise convolutional layer. The structure where the input is passed through the depthwise convolutional layer and then into the 1×1 convolutional layer can be considered a depthwise separable convolution. First, the input high-dimensional feature map is reduced in dimensionality by the 1×1 convolutional layer, and the feature channels are adjusted to improve computational efficiency. Second, a 3×3 depthwise convolutional layer (DWConv) is used to process the input feature map. Each channel is assigned a corresponding convolution kernel, and the convolution operation is performed independently. The calculation method for the output feature map Y is expressed as follows:

[0135]

[0136] In a standard 3×3 convolution, for each output channel, the convolution kernel performs a sliding window calculation across all channels of the input feature map. The calculation method for the output feature map Y is expressed as follows:

[0137]

[0138] Where (i,j) represents the position index of the output feature map, i.e., the pixel coordinate in the output feature map, and represents the index inside the convolution kernel, traversing the entire K×K filter. Comparing the calculation formulas of the two convolutions, depthwise convolution has relatively lower computational complexity while maintaining effective extraction of local spatial features. This method reduces computational overhead, reserving some computational resources for the real-time operation of the subsequent SLAM system. Finally, the convolutional layers are linearly combined along the channel dimension to restore the feature dimension. In addition, to enhance gradient flow and avoid information loss, this module further introduces residual connections, directly adding the input features to the output features. Finally, the feature map after residual connections is subjected to multi-scale pooling or concatenation to output local detail multi-scale features. It is used to characterize local geometric information such as texture and edges of an image.

[0139] The method for extracting global information T based on the multi-head self-attention mechanism in step S12 includes the following steps:

[0140] Patch concatenation: First, multiple frames of input images are acquired and preliminary feature extraction is performed through Conv-stem convolutional layers to generate high-dimensional feature maps. Second, this feature map is divided into a sequence of patches, and the sequence information is encoded through block embedding encoding layers and positional embedding encoding layers to form spatially aware embedding vector representations. The embedding vectors of the patches are mapped to query vector Q, key vector K, and value vector V. Let Fi = {F1, F2, ..., Fn} represent the high-dimensional feature map obtained after Conv-stem convolution, where Fn represents the patch block of the feature map, and n represents the number of patches. The QKV matrix obtained by word embedding encoding mapping for each feature map Fi can be represented as follows:

[0141]

[0142] Where i represents the index of the patch, and Lq, Lk, and Lv represent the mapping functions that map the patch to the corresponding vector. This mapping is implemented in the actual network layer through two-dimensional convolution.

[0143] Feature map matching: The matching process for feature maps Fi is achieved by calculating the similarity score between the query vector Q of each feature map and the key vector K of other feature maps. Taking the i-th feature map as an example, the similarity score between its query vector Qi and the key vector Kj of the j-th feature map can be calculated as follows:

[0144]

[0145] Where h, w, and c represent the height, width, and number of channels of the feature map, respectively. Next, the calculated similarity scores are converted into a probability distribution using the Softmax function. After Softmax processing, all scores are positive and sum to 1. This process can be represented as:

[0146]

[0147] in, This represents the sum of similarity scores between the i-th feature map and all other feature maps. Based on this, the final calculation is... This refers to the attention weight of the i-th feature map relative to the j-th feature map.

[0148] Global information extraction: By calculating the attention weight of each feature map relative to all other feature maps, and then summing this weight with the corresponding value vector V, the attention value Ti of the i-th feature map can be obtained, thus achieving effective extraction of global information from the input image. The weighted attention value is represented as:

[0149]

[0150] in, This represents the value vector of the j-th feature map. Each attention head can obtain the attention value corresponding to the feature map through the above calculation process. The attention values ​​output by multiple attention heads are concatenated by Concat and transformed through a linear layer to finally output the multi-head attention weights of the feature map, which are then converted into a vector representation, i.e., the global information T.

[0151] Then feature fusion is performed: the local features L of the object are fused with the global context information T to obtain the interaction weight V and the fused feature U. The specific process can be represented as follows:

[0152] S13. Stitching and Preliminary Fusion: This involves combining the local features obtained in the previous steps. and global context information Stitching along the channel dimension, through Convolutional layers achieve feature fusion, and batch normalization is performed to obtain fused features. The specific process is as follows:

[0153]

[0154] in, This indicates splicing along the channel dimension. This represents matrix multiplication. express The convolutional layer weights, Indicates the batch normalization layer. This represents the LeakyReLU activation function;

[0155] S14, Channel Attention Interaction: The fused features obtained in step S13 are... Perform global average pooling, then pass through two... The convolutional layer encodes the fused feature U, and the channel interaction weights V are obtained by combining the LeakyReLU activation function. The specific process is as follows:

[0156]

[0157] Where H and W are the height and width of the fused feature map, and W2 and W3 represent the two... The weights of the convolutional layer, where δ() represents the Sigmoid activation function;

[0158] S15. Dynamic Weighted Output: The interaction weight V obtained in step S14 is multiplied channel-by-channel with the fusion feature U obtained in step S13 to recalibrate and generate the interaction-adjusted edge features; the fusion feature is added to the edge calibration feature through residual connection to finally obtain the edge features with fusion feature U and the target. The output feature A is calculated using the following formula:

[0159]

[0160] in, This is an element-wise inner product;

[0161] S16. Accuracy Optimization: The target detection network is trained and optimized based on a composite loss function that includes geometric constraints to improve the regression accuracy of observation data in scenarios with observation degradation. The specific process includes:

[0162] The composite loss function L is composed of a weighted average of the classification loss function, the regression loss function, and the confidence loss function, expressed as:

[0163]

[0164] in, , and These represent the weight coefficients corresponding to each loss function.

[0165] Classification loss function L c It is based on the modification of cross-entropy loss, expressed as:

[0166]

[0167] in, This represents the model's predicted probability for the current category. This represents a factor that weights the categories. This indicates the hyperparameters for adjusting the difficulty level of samples;

[0168] Regression loss function The MDIoU loss function is used to avoid the degradation and failure of the CIoU loss function when the center points of the predicted bounding box and the labeled bounding box coincide. It is expressed as:

[0169]

[0170] Where h and w represent the height and width of the image, respectively. , These represent the coordinates of the top-left and bottom-right points of the predicted bounding box, respectively. , These represent the coordinates of the top-left and bottom-right points of the actual labeled bounding box, respectively.

[0171] Confidence loss function L o A binary cross-entropy loss function is used to evaluate the deviation between the model's predicted probability of containing the target object and the true label, expressed as:

[0172]

[0173] in, This indicates the category label detected by the target detection result. This indicates the probability that the bounding box contains the target.

[0174] To further verify the performance of the object detection network model in this embodiment, it was compared with other classic object detection models. The main evaluation metrics for comparison were mAP value and inference speed (FPS). In the TUMRGB-D dataset, specifically the fr1 / room, fr2 / desk, and fr3 / office sequences, the mAP and FPS values ​​of six object detection models were compared: Faster R-CNN, SSD, YOLOv5s, YOLOv7, ViT, and MPViT. Quantitative comparison results are as follows: Figure 6As shown, in terms of overall inference speed, this model outperforms the traditional two-stage detection framework Faster R-CNN and the multi-path Transformer architecture MPViT, indicating that this model can maintain relatively ideal real-time performance while maintaining high detection accuracy. In some sequences, it is comparable to SSD, YOLOv7 and ViT, although it is not as good, the difference is small. However, compared with the lightweight detection framework YOLOv5s, there is still a large gap.

[0175] S2, Tracking and Relocation: such as Figure 7 While step S1 is running, the tracking thread performs camera pose tracking on the image sequence obtained in step S1. When camera tracking loss is detected, the relocalization process is initiated using the semantic observation data output in step S1. An inscribed ellipse is constructed based on the 2D detection bounding box of the semantic observation data. In the global map containing sparse map points and 3D object ellipsoid models, the same type of 3D object ellipsoid model is retrieved. The initial estimated pose is then filtered again using the P3P algorithm and the IoU cost function. Finally, the initial estimated pose is used to guide the matching of feature points in the current frame with map points in the global map to restore the system tracking process. The specific process includes:

[0176] S21. Camera pose tracking: While running step S1, a tracking thread is run to perform camera pose tracking on the image sequence.

[0177] S22. Condition Judgment: During step S21, when the system detects that the number of feature points matching the current frame and the local map is lower than a threshold, the system is put into a "Lost" state, and the relocation process is activated. In this embodiment, the number of interior points matching the current frame and the local map is lower than a preset threshold. .

[0178] The object detection network obtains the category information and 2D detection bounding boxes of objects in the current frame, and constructs an inscribed 2D ellipse. Specifically, it uses the semantic observation data output by the detection thread in step S1 to construct the inscribed 2D ellipse for the current frame. The first detected in For each object, obtain its 2D detection bounding box. .

[0179] To establish a geometric relationship with the three-dimensional ellipsoid, this embodiment models the rectangular bounding box as an inscribed ellipse. The center coordinates of the ellipse The geometric center of the bounding box is defined by its major and minor axes, which are the heights of the bounding box, respectively. With width Half of it. This modeling method can transform discrete pixel boxes into continuous geometric parameters, which facilitates subsequent projection calculations.

[0180] S23, 2D-3D Matching: Retrieve 3D object ellipsoid models from the dynamically maintained global map that match the category label of the detected object in the current frame. Match these models with existing 3D object ellipsoid models in the global map according to category to establish 2D-3D matching pairs; construct a candidate matching pair set.

[0181]

[0182] in, Indicates the number of detected frames in the current frame. A 2D semantic object, This indicates the number of existing elements in the global map. A 3D ellipsoidal object model This represents the operation of retrieving the semantic labels of objects. This collection contains potential matches that are consistent across all category labels.

[0183] S24. Verification of Initial Pose Values: From the 2D-3D matching pairs obtained in step S23, select the smallest set containing three pairs of "ellipse-ellipsoid". Use the P3P algorithm to calculate the camera pose based on the center points of the ellipse and ellipsoid, obtaining four sets of output pose solutions; the specific process package is as follows:

[0184] From the matching set The smallest subset containing three pairs of "2D ellipse-3D ellipsoid" is randomly selected. To quickly solve for the camera pose, this embodiment utilizes the correspondence between the centers of the ellipse and the ellipsoid, transforming it into a classic P3P (Perspective-3-Point) problem:

[0185] Input: Coordinates of the centers of the three 3D ellipsoids and its corresponding 2D inscribed ellipse center projection coordinates .

[0186] Solution: Using the P3P algorithm, calculate the distances from the camera's optical center to the centers of the three objects, and then determine the camera's pose in the world coordinate system. .

[0187] Due to the multiple-solution nature of the P3P algorithm, it typically generates up to 4 candidate pose solutions.

[0188] S25. Selection of optimal pose solution:

[0189] To select the unique true pose from four candidate solutions, this embodiment abandons the traditional reprojection point error (which has a large error when the initial value is inaccurate) and designs a projection cost function based on IoU. The specific steps are as follows:

[0190] Traverse each group of candidate poses Using the dual quadratic surface projection formula, all visible 3D ellipsoids in the map are represented. Project back to the current image plane:

[0191]

[0192] in, Represents the two-dimensional dual conic section projected onto the image plane; Represents the dual quadric surface of a three-dimensional object in the world coordinate system; Indicates the first The camera projection matrix corresponding to the candidate poses, and ; Represents the camera intrinsic parameter matrix; and They represent the first The rotation matrix and translation vector of the candidate solutions.

[0193] Subsequently, the projected ellipse is obtained. circumscribed rectangle And construct the total cost function. :

[0194]

[0195] in, Indicates the first The total cost of reprojection of candidate poses; This indicates the total number of object matching pairs participating in the matching in the current frame; For matching index pairs; Indicates the first The two-dimensional bounding rectangle generated by projecting a three-dimensional ellipsoid of an object into the current candidate pose; Indicates the first The actual two-dimensional bounding boxes detected for each object in the current frame; This function calculates the intersection-union ratio of two rectangles; its value is the ratio of the area of ​​the intersection to the area of ​​the union of the two rectangles.

[0196] Finally, select The pose that is the smallest and less than a preset threshold (e.g., 0.6) is taken as the optimal initial estimated pose. .

[0197] S26. Feature Point Matching Verification: The map points from the previous keyframe are projected onto the current frame using the initial estimated pose to guide matching. If the number of matched points exceeds a preset threshold, the PnP algorithm is used to further optimize the camera pose and restore system tracking. Otherwise, the initial estimated pose is directly used as the pose of the current frame. The specific process is as follows:

[0198] Using the selected As prior values, sparse map points from the previous keyframe are projected onto the current frame's image plane. Because... It provides better initial values ​​and greatly reduces the projection range, thus enabling efficient matching of ORB feature points within an extremely narrow search window.

[0199] If the number of successfully matched feature points exceeds the recovery threshold (e.g., 30), then perform a single PnP (Perspective-n-Point) optimization that only optimizes the pose, refines the camera pose, and switches the system state back to "TrackingOK", declaring the relocalization successful.

[0200] S3. Object Mapping and Fusion: Simultaneously with step S1, a mapping thread runs. Based on the semantic observation data obtained in step S1 and the camera pose recovered in step S2, a new 3D object ellipsoid model is initialized in 3D space and inserted into the global map. A semantic reprojection error function based on Gaussian-Wasserstein distance is constructed to optimize redundant object models generated during initialization. Inclusion and fusion checks are performed to obtain a single ellipsoid model, and the global map is updated. The specific process is as follows:

[0201] S31. Object-level Data Association: Before initializing new objects, the system first determines whether the object detected in the current frame is an existing "old object" in the map. The system projects the candidate 3D object ellipsoid onto the current frame and calculates the IoU between the projected area and the current 2D detection box. If the maximum IoU exceeds the association threshold (0.5 in this example), the observation data is associated with the corresponding 3D object; if it is below the threshold, it is determined to be a new object, and the initialization process begins.

[0202] S32. Object Ellipsoid Model Initialization: For objects identified as newly added, the system needs to initialize their three-dimensional geometric parameters. This embodiment adopts a strategy of "first estimating a sphere and then expanding it into an ellipsoid," using multi-frame observation data for robust initialization. The specific steps are as follows:

[0203] S321. Calculation of the center of the sphere: First, the center point of the detection bounding box in each frame is obtained based on the target detection network in step S1. By utilizing the parallax between multiple frames of observation, the initial position of the object in the world coordinate system is calculated using the triangulation method, and this initial position is taken as the center position of the sphere. .

[0204] S322. Radius estimation: Obtain the center point of the detected bounding box based on the target detection network in step S1. The position of the center of the sphere is obtained through triangulation calculation. ,radius Determined by the average size of the bounding box:

[0205]

[0206] in, This indicates the number of frames in which the object is being tracked. Indicates the center of the object at the th position. Depth in a camera coordinate system and They represent the first Width and height of the detection box in the frame, and This indicates the camera's focal length.

[0207] S323. Construction of the dual ellipsoid: Based on the sphere center position calculated above... and radius Construct the initial dual ellipsoid matrix During the initialization phase, the object is modeled as a sphere in the world coordinate system.

[0208] The specific matrix construction process is as follows: First, define the standard spherical dual matrix in the local coordinate system. Then, pose transformation is used to convert it to the world coordinate system. The calculation formula is as follows:

[0209]

[0210] Among them, the standard sphere matrix ;

[0211] Transformation matrix Determined by the position of the ball's center:

[0212]

[0213] in, Represents the identity matrix; Represents a diagonal matrix; This represents the transpose of the transformation matrix.

[0214] S33. Semantic error construction based on Gaussian-Wasserstein distance:

[0215] An objective function based on the Gaussian-Wasserstein distance was designed to update objects by minimizing reprojection error. This distance measures the error between the inscribed ellipse of the object detection box and the ellipse formed by projecting the ellipsoid onto the image plane. In the Gaussian-Wasserstein distance, the parameters of the ellipse are considered to follow a two-dimensional Gaussian distribution. Its parameters are defined as follows:

[0216]

[0217] in, Represents the coordinates of the center of the ellipse. This represents the lengths of the major and minor axes of the ellipse. This indicates the direction angle of the ellipse.

[0218] Suppose we have two ellipses, then the Gaussian distributions of the two ellipses are... and The Gaussian-Wasserstein distance between them is defined as:

[0219]

[0220] Construct a single-frame semantic reprojection error function, and then... An object ellipsoid Projected to the The error function of the frame image plane is expressed as:

[0221]

[0222] in, Indicates the first The first frame The inscribed ellipse of each target detection bounding box (as an observation). The coordinates of the center of this ellipse. The geometric center of the bounding box, the major semi-axis The minor axis is half the height of the bounding box. Half the width of the bounding box, direction angle Initially set to 0. Indicates the first The dual matrix of an object ellipsoidal model. Indicates the first The camera projection matrix of the keyframes, and . Represents the dual ellipsoid via camera projection matrix After projection, the two-dimensional dual conic section (i.e., the projected ellipse) formed on the image plane is used to calculate its corresponding Gaussian distribution parameters. . Indicates the first The first frame The detection confidence score of each object is used to adjust the impact of different observation qualities on optimization. This represents the squared Gaussian-Wasserstein distance as defined above.

[0223] Objective function with respect to object ellipsoid parameters The derivative of the ellipse will serve as the gradient, guiding the direction of optimization iterations, thereby enabling high-precision updates to the shape (axis length), position (center), and orientation (rotation) of the ellipsoid.

[0224] S34. Redundant Object Detection and Fusion: Criterion 1: Does the IoU of the target detection bounding boxes corresponding to the two object ellipsoid models exceed a set threshold (IoU threshold is set to 0.2)?

[0225] Criterion 2: Do the two object ellipsoidal models share a certain number of 3D map points (the threshold for the number of map points is set to 20)?

[0226] Both criteria 1 and criterion 2 above use a threshold method to specify the constraint range. If two ellipsoidal models simultaneously satisfy the above two criteria, they are considered to be duplicate objects and need to be merged.

[0227] The fusion method is as follows:

[0228] Specifically, the detection bounding boxes of two objects in the keyframe are merged, and a new ellipsoid is re-initialized based on the bounding box information according to the initial estimation method. After merging, the original two ellipsoid models are removed. This fusion process is only performed on the keyframe.

[0229] S4. Local Joint Optimization: While running steps S1 and S3, in the mapping thread, keyframe sequences are selected from the image sequence; a joint objective function is established, including the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid model. Local bundle adjustment is then performed on the keyframe sequences using the joint objective function to optimize and synchronously update various parameters in the visual tracking task, including camera pose, map point coordinates, and object ellipsoid model parameters. Redundant keyframes are removed to obtain an optimized keyframe sequence. To improve map accuracy and integrate geometric and semantic information, this embodiment maintains a "Local Map" in the local mapping thread. This map contains the current keyframe and its adjacent keyframes connected in the covisibility graph, as well as all map points and object ellipsoids observed by these frames. The specific steps are as follows:

[0230] S41. Joint Factor Graph Construction: The system constructs a graph containing keyframe pose nodes. Map point nodes and object ellipsoid nodes The factor graph is connected using geometric error edges and semantic error edges.

[0231] S42. Establish the joint objective function:

[0232] First, establish a standard map point geometric reprojection error function. For the first The first keyframe observed For each map point, the error is defined as:

[0233]

[0234] in, These are the observed coordinates of the feature points on the image; For camera projection function; For the first The camera pose of the frame; For the first The three-dimensional coordinates of a map point; This is the Huber robust kernel function.

[0235] Secondly, the error form based on Gaussian-Wasserstein distance established in step S3 is directly called. The camera projection matrix is ​​then... Substituting and expanding, the semantic reprojection error function of a single frame is expressed as:

[0236]

[0237] in, Indicates the first The first frame Gaussian distribution parameters of the inscribed ellipse of the target detection box; Indicates the first An ellipsoidal model of an object Camera position Gaussian distribution parameters formed by downward projection; This represents the detection confidence score. The error function uses the Gaussian-Wasserstein distance to measure the positional differences between the observed ellipse and the projected ellipse. ,shape and direction The overall differences.

[0238] Then, the map point geometric reprojection error function and the ellipsoidal semantic reprojection error function are linearly weighted and summed according to a preset weight ratio to construct the joint objective optimization function. This is used for subsequent joint refinement of camera pose and ellipsoidal geometric parameters. The joint objective function is then... It can be represented as:

[0239]

[0240] in, A geometric observation set; A set of semantic observations; The balancing weight coefficient is set to 0.5 in this embodiment. This function also constrains the camera pose. Map point location and object ellipsoid parameters .

[0241] S43. Minimize the joint objective function: In order to minimize the above joint objective function This embodiment uses the Levenberg-Marquardt (LM) algorithm for iterative solution. The specific solution process is as follows:

[0242] S431. Jacobian matrix calculation:

[0243] Calculate the joint objective function separately The partial derivatives with respect to each state variable, i.e., the calculation error term relative to the camera pose. Map points and the dual matrix of the object ellipsoid The Jacobian matrix.

[0244] For the semantic error term, the derivative of the Gaussian-Wasserstein distance with respect to the parameters of the projected ellipse is calculated using the chain rule, and then the parameters of the projected ellipse with respect to the camera pose are calculated. and ellipsoid parameters The derivative of the derivative is used to obtain the gradient information needed for optimization.

[0245] S432, Parameter Update:

[0246] The increment is calculated based on the calculated gradient direction and the approximate inverse of the Hessian matrix. .

[0247]

[0248] Use this increment to update all state variables: Among them, the update of camera pose is related to Lie algebra. The process takes place in space; updates to the object ellipsoid and map points occur in Euclidean space.

[0249] S433. Convergence criterion:

[0250] Repeat the above calculation and update steps until the decrease in the objective function is less than a preset threshold or the maximum number of iterations is reached. At this point, the algorithm is considered to have converged. Output the optimized accurate camera pose, map point coordinates, and object ellipsoid model parameters, and update them to the global map.

[0251] S5. Global Consistency Optimization: Run the loop closure detection thread and the global optimization thread to perform loop closure detection on the optimized keyframe sequence obtained in step S4. After confirming loop closure, perform pose graph optimization and global bundle adjustment to eliminate accumulated errors, and finally generate a globally consistent object-level semantic 3D scene map. To eliminate accumulated drift caused by long-term operation, the system performs global consistency correction in the loop closure thread and the global optimization thread. The specific process is as follows:

[0252] S51. Loop Closure Detection and Verification: Calculate the similarity score between the current frame and historical keyframes using the DBoW2 bag-of-words model. If the score exceeds a threshold and passes geometric and temporal consistency verification, a loop closure is detected.

[0253] S52, Pose Graph Optimization: After detecting loop closure, pose graph optimization is performed first. This process uses keyframes as nodes and co-view relationships and loop closure constraints as edges. Through optimization on the Sim(3) group, the loop closure error is quickly amortized, and the scale drift and rigid body transformation deviation of the keyframe trajectory are corrected.

[0254] S53, Global Bundle Adjustment (GlobalBA): After pose graph optimization is complete, a global optimization thread is started. This thread locks the entire map and constructs a global factor graph containing all historical keyframes, all map points, and all object ellipsoids. It utilizes the same joint objective function as S4. Global iterative optimization is performed to ultimately generate a geometrically accurate and semantically consistent object-level semantic map.

[0255] To verify the semantic mapping performance of the proposed method in both unobserved and observed degradation scenarios, experiments were conducted on the publicly available TUMRGB-D dataset and a self-built campus indoor conference room scene. Representative sequences from the TUMRGB-D dataset, fr1 / room, fr2 / desk, and fr3 / office, were selected as benchmarks for the unobserved degradation scenario. This scenario possesses rich texture features and stable lighting conditions, and the output object mapping results are shown below. Figure 8As shown in (a). In the figure, (1) represents the input experimental scene sequence image; (2) represents the mapping effect of the object model in the scene, and the ellipsoids of different colors represent different object categories; (3) represents the map point cloud associated with the object; (4) represents the complete semantic mapping effect of fusing map points and object models. This semantic map not only contains the three-dimensional point cloud structure in the environment, but also represents the semantic objects in the scene through the ellipsoid model, which can provide information support for subsequent advanced tasks such as navigation and obstacle avoidance. In a more challenging real-world scene test, a typical sequence of a conference room on campus was selected for verification. This scene also contains low-texture areas, motion blur, and observation degradation such as camera occlusion, such as Figure 8 As shown in (b). In the figure, (1) represents the input experimental scene sequence image; (2) represents the mapping effect of the object model in the scene, and the ellipsoids of different colors represent different object categories; (3) represents the map point cloud associated with the object; and (4) represents the complete semantic mapping effect of fusing map points and object models. Even when running the system under such unstable observation conditions, it can still maintain stable mapping performance. The mapping effect shows that although the system frequently experiences tracking loss in the observation degradation scenario, it can still continuously complete the environmental mapping task. Its object model and map points can be effectively updated in the intermittent process of tracking loss-tracking recovery, maintaining the correct map structure, which also proves the advantages of the system in terms of mapping robustness.

[0256] To evaluate the localization accuracy of the proposed object-level SLAM system, experiments were conducted on the TUMRGB-D dataset and in an indoor conference room scene on a campus. The system was then compared with current mainstream visual SLAM systems. The visual SLAM algorithms compared included feature-point based ORB-SLAM2 and ORB-SLAM3, the direct method LSD-SLAM, and object-level algorithms Quadric-SLAM and SO-SLAM. Figure 9 The improved SLAM system and the camera pose trajectories output by the five comparison methods mentioned above are shown in the experimental scenario. ORB-SLAM2 and ORB-SLAM3 both operate in monocular mode. Figure 9 The mid-trajectory results show that, compared with the five SLAM algorithms compared, the improved system demonstrates higher accuracy in estimating pose trajectories (solid red lines) across all experimental sequences, with significantly smaller deviations from the actual trajectories (dashed black lines) compared to the comparative algorithms.

[0257] Example 3

[0258] This embodiment is a verification platform for implementing the method proposed in Embodiment 2 based on the system proposed in Embodiment 1. It mainly includes a mobile experimental carrier, on which a sensor group is set. The mobile experimental carrier and the sensor group are respectively connected to an onboard computing platform via signals.

[0259] The mobile experimental platform uses a general-purpose wheeled mobile robot chassis as its motion carrier. To actively simulate common observation degradation conditions in real-world scenarios, such as limited observation angles and rapid, large-amplitude turns, the mobile platform is specially equipped with a rotatable gimbal support (or rotatable platform). During the experiment, the camera's observation attitude is dynamically adjusted by controlling the gimbal, thus constructing a controllable degradation experimental environment.

[0260] The sensor group mainly includes a vision sensor and a ground truth sensor. The vision sensor uses a high frame rate industrial camera (preferably a USB 3.0 interface camera with a frame rate of 30fps) mounted on a rotatable platform as the main visual input source for the SLAM system of this invention, used to acquire high dynamic range image sequences. The ground truth sensor includes a synchronously mounted solid-state LiDAR. By running a high-precision laser odometry calculation method (such as the LOAM framework), the ground truth of the mobile platform is obtained as a benchmark reference for error analysis.

[0261] The onboard computing platform is a high-performance industrial control computer, using Ubuntu 20.04 (64-bit) as the operating system to run the system's various threads. To meet the real-time requirements of parallel inference in the hybrid attention mechanism object detection network and multi-threaded optimization of the SLAM backend, the computing unit is configured as follows:

[0262] Central Processing Unit (CPU): Equipped with a quad-core processor (2.0GHz or higher), responsible for logic control and thread scheduling;

[0263] Graphics Processing Unit (GPU): Equipped with a high-performance discrete graphics card (8GB or more of video memory, such as the NVIDIA RTX 30 series), utilizing CUDA acceleration technology to ensure the real-time performance of semantic observation extraction;

[0264] Memory: Equipped with 32GB of large-capacity memory to ensure real-time throughput of image stream data and global semantic map.

[0265] This hardware architecture enables real-time display of sensor data acquisition, system thread status monitoring, and SLAM trajectory and mapping results through a visual interactive interface.

[0266] The above is merely a further explanation of the present invention in conjunction with specific embodiments. All descriptions made do not imply any limitation on the scope of protection of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An object-level semantic visual SLAM system based on a hybrid attention mechanism object detection network and an ellipsoidal model, characterized in that, The system is a SLAM system based on an improved ORB-SLAM2 architecture, containing five thread modules: Semantic perception module: includes a detection thread, which includes a target detection network based on a hybrid attention mechanism, used to extract local features and global context information of image sequences in parallel, and output semantic observation results including object category labels, two-dimensional bounding boxes and confidence scores; Visual tracking and relocalization module: includes a tracking thread and a relocalization unit, wherein the tracking thread uses feature points and semantic observation results to jointly perform pose tracking; When tracking loss is detected, the relocalization unit constructs dual geometric constraints using a 3D object ellipsoid model in the map and the semantic observation of the current frame, and estimates the camera pose to restore tracking. Local mapping and fusion module: includes a mapping thread for estimating the object ellipsoid model in 3D space and inserting it into the map, and performing model fusion to eliminate redundancy; the local mapping module constructs a joint objective function that includes the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid model, and performs joint bundle adjustment to simultaneously optimize keyframe pose, map point coordinates and object ellipsoid parameters; The loop closure detection module includes a loop closure detection thread, which performs similarity retrieval on keyframes based on the bag-of-words model and performs essence graph optimization when a loop closure is detected. Global Consistency Optimization Module: Includes a global bundle adjustment thread, which performs global bundle adjustment on the sparse semantic map structure of the entire map after loop closure detection is completed, to obtain a globally consistent camera pose distribution and object-level map structure.

2. An object-level semantic visual SLAM method based on a hybrid attention mechanism object detection network and an ellipsoidal model, characterized in that, Includes the following steps: S1. Semantic observation extraction: Start the detection thread, capture images of the scene using a camera to obtain an image sequence; input the image sequence into the target detection network to obtain semantic observation data of the target object; S2. Tracking and Relocation: The tracking thread performs camera pose tracking on the image sequence obtained in step S1; when camera tracking is lost, the relocation process is started using the semantic observation data output in step S1 to restore the camera pose tracking task. S3. Object mapping and fusion: Run the mapping thread, and based on the semantic observation data obtained in step S1 and the camera pose recovered in step S2, initialize a new three-dimensional object ellipsoid model in three-dimensional space and insert it into the global map. The redundant object models generated during initialization are optimized to obtain a single ellipsoidal model, and the global map is updated. S4. Local Joint Optimization: In the mapping thread, key frame sequences are selected from the image sequence; a joint objective function is established that includes the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid model; and local bundle adjustment is performed on the key frame sequences through the joint objective function to optimize and synchronously update various parameters in the visual tracking task, thereby obtaining optimized key frame sequences. S5. Global Consistency Optimization: Run the loop closure detection thread and the global optimization thread to perform loop closure detection on the optimized keyframe sequence obtained in step S4; after confirming the loop closure, perform pose graph optimization and global bundle adjustment to eliminate accumulated errors and finally generate a globally consistent object-level semantic 3D scene map.

3. The object-level semantic visual SLAM method based on a hybrid attention mechanism target detection network and an ellipsoidal model according to claim 2, characterized in that: The specific process of semantic observation extraction in step S1 includes: S11. Obtain image sequence: Capture the scene with a camera to obtain an image sequence; S12, Feature Extraction: The image sequence obtained in step S11 is processed by the object detection network to extract the local features L and global context information T of the object in the image sequence; S13. Stitching and Preliminary Fusion: The local features obtained in step S12 are then stitched together. and global context information Stitching along the channel dimension, through Convolutional layers achieve feature fusion, and batch normalization is performed to obtain fused features. The specific process is as follows: in, This indicates splicing along the channel dimension. This represents matrix multiplication. express The convolutional layer weights, Indicates the batch normalization layer. This represents the LeakyReLU activation function; S14, Channel Attention Interaction: The fusion features obtained in step S13 are... Perform global average pooling, then pass through two... The convolutional layer encodes the fused feature U, and the channel interaction weights V are obtained by combining the LeakyReLU activation function. The specific process is as follows: Where H and W are the height and width of the fused feature map, and W2 and W3 represent the two... The weights of the convolutional layer, where δ() represents the Sigmoid activation function; S15. Dynamic Weighted Output: Multiply the interaction weight V obtained in step S14 and the fusion feature U obtained in step S13 channel by channel to recalibrate and generate the interaction-adjusted edge features; add the fusion feature and the edge calibration feature through residual connection to finally obtain the edge features with fusion feature U and the target. The output feature A is calculated using the following formula: in, This is an element-wise inner product; S16. Accuracy Optimization: The target detection network is trained and optimized based on a composite loss function that includes geometric constraints to improve the regression accuracy of observation data in scenarios with observation degradation.

4. The object-level semantic visual SLAM method based on a hybrid attention mechanism target detection network and an ellipsoidal model according to claim 3, characterized in that: The specific process of accuracy optimization in step S16 includes: The final composite loss function L is composed of a weighted average of the classification loss function, the regression loss function, and the confidence loss function, and is expressed as: in, , and These represent the weight coefficients corresponding to each loss function; The classification loss function L c It is based on the modification of cross-entropy loss, expressed as: in, This represents the model's predicted probability for the current category. This represents a factor that weights the categories. This indicates the hyperparameters for adjusting the difficulty level of samples; The regression loss function The MDIoU loss function is used to avoid the degradation and failure of the CIoU loss function when the center points of the predicted bounding box and the labeled bounding box coincide. It is expressed as: Where h and w represent the height and width of the image, respectively. , These represent the coordinates of the top-left and bottom-right points of the predicted bounding box, respectively. , These represent the coordinates of the top-left and bottom-right points of the actual labeled bounding box, respectively. The confidence loss function L o A binary cross-entropy loss function is used to evaluate the deviation between the model's predicted probability of containing the target object and the true label, expressed as: in, This indicates the category label detected by the target detection result. This indicates the probability that the bounding box contains the target.

5. The object-level semantic visual SLAM method based on a hybrid attention mechanism target detection network and an ellipsoidal model according to claim 4, characterized in that: The specific process of tracking and relocation in step S2 includes: S21, Camera Pose Tracking: Run the tracking thread to perform camera pose tracking tasks on the image sequence; S22. Conditional Judgment: During step S21, when the system detects that the number of feature points matching the current frame and the local map is lower than the threshold, the relocalization process is activated; the object detection network is used to obtain the category information of the objects in the current frame and the 2D detection bounding box, and an inscribed 2D ellipse is constructed. S23, 2D-3D matching: Retrieve a 3D object ellipsoid model in the global map dynamically maintained by the system that matches the category label of the detected object in the current frame, and match it with the 3D object ellipsoid models already established in the global map according to the category to establish a 2D-3D matching pair; the global map includes sparse map points and 3D object ellipsoid models. S24. Verification of initial pose solution set: Select the smallest set containing three pairs of "ellipse-ellipsoid" from the 2D-3D matching pairs obtained in step S23, and use the P3P algorithm to calculate the camera pose based on the center points of the ellipse and ellipsoid to obtain four sets of output pose solutions. S25. Optimal Pose Solution Selection: Based on the four pose solutions output in S24, a projection cost function based on IoU is designed to select the unique true pose. The specific process is as follows: Traverse each group of candidate poses Using the dual quadric surface projection formula, all visible three-dimensional ellipsoids in the map are represented. Project back to the current image plane: in, Represents the two-dimensional dual conic section projected onto the image plane; Represents the dual quadric surface of a three-dimensional object in the world coordinate system; Indicates the first The camera projection matrix corresponding to the candidate poses, and ; Represents the camera intrinsic parameter matrix; and They represent the first The rotation matrix and translation vector of the candidate solutions; Subsequently, the projected ellipse is obtained. circumscribed rectangle And construct the total cost function. : in, Indicates the first The total cost of reprojection of candidate poses; This indicates the total number of object matching pairs participating in the matching in the current frame; For matching index pairs; Indicates the first The two-dimensional bounding rectangle generated by projecting a three-dimensional ellipsoid of an object into the current candidate pose; Indicates the first The actual two-dimensional bounding boxes detected for each object in the current frame; This function calculates the intersection-union ratio of two rectangles; its value is the ratio of the area of ​​the intersection to the area of ​​the union of the two rectangles. Finally, select The pose that is the smallest and less than a preset threshold is taken as the optimal initial estimated pose. ; S26. Feature point matching verification: The map points of the previous keyframe are projected onto the current frame using the initial estimated pose to guide the matching; if the number of matching points exceeds a preset threshold, the PnP algorithm is used to further optimize the camera pose and restore system tracking; otherwise, the initial estimated pose is directly used as the pose of the current frame.

6. The object-level semantic visual SLAM method based on a hybrid attention mechanism target detection network and an ellipsoidal model according to claim 5, characterized in that: The specific process of initializing the new three-dimensional object ellipsoid model in three-dimensional space in step S3 is as follows: First, the center point of the detection bounding box in each frame is obtained based on the target detection network. By utilizing the parallax between multiple frames of observation, the initial position of the object in the world coordinate system is calculated through triangulation, and this position is taken as the center position of the sphere. Then, the center point of the detection bounding box is obtained according to the target detection network. The position of the center of the sphere is obtained through triangulation calculation. ,radius Determined by the average size of the bounding box: in, This indicates the number of frames in which the object is being tracked. Indicates the center of the object at the th position. Depth in a camera coordinate system; and They represent the first Width and height of the detection box in the frame. and Indicates the camera's focal length.

7. The object-level semantic visual SLAM method based on a hybrid attention mechanism object detection network and an ellipsoidal model according to claim 6, characterized in that: The specific process of constructing an optimization function based on semantic reprojection error in step S3, object mapping and fusion, includes: The projection ellipse is obtained by projecting the 3D object ellipsoid model onto the image plane, and the inscribed ellipse of the detection box in the semantic observation data is obtained. The Gaussian-Wasserstein distance is used to measure the distributional difference between the projected ellipse and the detected inscribed ellipse, and its error term is defined as: The first An object ellipsoid Projected to the The semantic reprojection error function of the frame image plane is expressed as: in, Indicates the first The first frame The inscribed ellipse of the target detection bounding box; the coordinates of the center of the ellipse. The geometric center of the bounding box, the major semi-axis The minor axis is half the height of the bounding box. Half the width of the bounding box, direction angle Initially set to 0; Indicates the first The dual matrix of an ellipsoidal model of an object; Indicates the first The camera projection matrix of the keyframes, and Represents the dual ellipsoid via camera projection matrix After projection, the two-dimensional dual conic section formed on the image plane, i.e., the projected ellipse, is used to calculate its corresponding Gaussian distribution parameters. Indicates the first The first frame The detection confidence score of each object is used to adjust the impact of different observation qualities on optimization. This represents the squared Gaussian-Wasserstein distance as defined above.

8. The object-level semantic visual SLAM method based on a hybrid attention mechanism target detection network and an ellipsoidal model according to claim 7, characterized in that: The specific process of establishing the joint objective function that includes the geometric reprojection error of map points and the semantic reprojection error of the object ellipsoid model in step S4 local joint optimization includes: Establish a standard map point geometric reprojection error function For the first The first keyframe observed For each map point, the error is defined as: in, These are the observed coordinates of the feature points on the image; For camera projection functions; For the first The camera pose of the frame; For the first The three-dimensional coordinates of a map point; For Huber robust kernel function; Directly call the semantic reprojection error function obtained in step S3; and project the camera projection matrix. Substituting and expanding, the semantic reprojection error function of a single frame is expressed as: in, Indicates the first The first frame Gaussian distribution parameters of the inscribed ellipse of the target detection box; Indicates the first An ellipsoidal model of an object Camera position Gaussian distribution parameters formed by downward projection; This represents the detection confidence score; the error function uses the Gaussian-Wasserstein distance to measure the positional differences between the observed ellipse and the projected ellipse. ,shape and direction Overall differences; The joint objective optimization function is constructed by linearly weighting and summing the map point geometric reprojection error function and the ellipsoidal semantic reprojection error function according to a preset weight ratio. : in, A geometric observation set; A set of semantic observations; These are the balancing weight coefficients.

Images