Simultaneous localization and mapping method and electronic device

Abstract

The present disclosure provides a simultaneous localization and mapping method and an electronic device, and relates to the technical field of computer vision. The simultaneous localization and mapping method comprises: acquiring an image frame sequence, the image frame sequence comprising a first image frame; performing feature extraction operations on geometric attributes and semantic attributes of the first image frame respectively to obtain feature points in the first image frame and semantic category features of the feature points; determining feature point weights of the feature points based on the semantic category features, the feature point weights comprising at least semantic weights of the feature points, the semantic weights representing static stability confidence of semantic categories to which the feature points belong, the semantic weights having a value range between a first value indicating complete untrustworthiness and a second value indicating complete trustworthiness; constructing a weighted optimization objective function based on the feature point weights of the feature points, and performing sensor pose estimation and map optimization based on the weighted optimization objective function.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, specifically to a method and electronic device for simultaneous localization and mapping. Background Technology

[0002] Simultaneous localization and mapping (SLAM) is a core technology for autonomous robot localization and navigation. Based on the SLAM scheme, robots can acquire environmental data through sensors, estimate their own pose, and build an environmental map to assist them in performing tasks such as path planning, autonomous exploration, and navigation.

[0003] Among related technologies, SLAM schemes have certain limitations. For example, manual feature SLAM schemes extract geometric features such as corners and edges for matching and optimization, but lack the understanding of the semantic entities to which the features belong, resulting in poor performance in weakly textured regions or dynamic scenes. Semantic SLAM schemes improve robustness in dynamic environments by dividing the scene into static and dynamic regions and removing features from dynamic regions, but they suffer from information waste in dynamic regions. Summary of the Invention

[0004] In view of this, the present disclosure provides a method and electronic device for simultaneous positioning and mapping.

[0005] Firstly, a method for simultaneous localization and mapping is provided, comprising: acquiring an image frame sequence, the image frame sequence including a first image frame; performing feature extraction operations on the geometric and semantic attributes of the first image frame respectively to obtain feature points in the first image frame and semantic category features of the feature points; determining feature point weights of the feature points based on the semantic category features, the feature point weights including at least the semantic weights of the feature points, the semantic weights representing the static stability confidence of the semantic category to which the feature points belong, the semantic weights taking values ​​between a first value indicating completely unreliable and a second value indicating completely reliable; constructing a weighted optimization objective function based on the feature point weights of the feature points, and performing sensor pose estimation and map optimization based on the weighted optimization objective function.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, semantic category features represent at least one semantic category corresponding to a feature point; based on the semantic category features, the feature point weights of the feature points are determined, including: determining the semantic prior weights of at least one semantic category; determining the semantic weights of the feature points based on the semantic prior weights of at least one semantic category; and determining the feature point weights of the feature points based on the semantic weights of the feature points.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the semantic weight of a feature point is determined based on the semantic prior weight of at least one semantic category, including: determining the geometric confidence factor of at least one semantic category, wherein the geometric confidence factor is determined based on the texture richness of the local region where the feature point is located; and determining the semantic weight of the feature point based on the geometric confidence factor and the semantic prior weight of at least one semantic category.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the feature point weight of a feature point is determined based on the semantic weight of the feature point, including: obtaining the weights of other dimensions of the feature point, which include at least one of geometric stability weight, dynamic weight, and temporal consistency weight; and fusing the semantic weight of the feature point with the other dimension weights to obtain the feature point weight of the feature point.

[0009] In conjunction with the first aspect, in certain implementations of the first aspect, feature extraction operations are performed on the geometric and semantic attributes of the first image frame to obtain feature points in the first image frame and semantic category features of the feature points. This includes: inputting the first image frame into a joint feature extraction network to obtain feature points and semantic category features of the feature points; wherein the joint feature extraction network includes an encoder, a first branch decoder, and a second branch decoder; the encoder is used to perform feature encoding operations on the first image frame to obtain image feature encoding; the first branch decoder is used to generate the location and descriptor of the feature points based on the image feature encoding; and the second branch decoder is used to generate the semantic category features of the feature points based on the image feature encoding.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, a weighted optimization objective function is constructed based on the feature point weights of the feature points, including: determining the threshold parameter in the robust kernel function based on the feature point weights of the feature points; and constructing a weighted optimization objective function for the reprojection error term of the feature points based on the robust kernel function and the feature point weights.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the image frame sequence further includes a second image frame; the synchronous localization and mapping method further includes: obtaining the semantic description vectors of the first image frame and the second image frame respectively, wherein the semantic description vectors are determined based on semantic category features; calculating the visual similarity between the first image frame and the second image frame, and selecting a loopback frame in the second image frame based on the visual similarity and the similarity between the semantic description vectors of the first image frame and the second image frame respectively; and performing loopback verification and global map optimization based on the first image frame and the loopback frame.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, loop closure verification and global map optimization are performed based on the first image frame and the loop closure frame, including: determining matching feature point pairs between the first image frame and the loop closure frame, wherein the matching feature point pairs include a first feature point in the first image frame and a second feature point in the loop closure frame; determining the loop closure edge weights between the first image frame and the loop closure frame based on the semantic category consistency between the semantic category features of the first feature point and the second feature point, and the feature point weights of the first feature point and the second feature point; constructing a loop closure optimization objective function based on the loop closure edge weights, and performing global map optimization based on the loop closure optimization objective function.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the synchronous localization and mapping method further includes: hierarchical management of feature points based on feature point weights; wherein, hierarchical management includes at least one of the following: using first-level feature points as long-term map points, which participate in the global map optimization process; using second-level feature points as temporary map points, which only participate in the local map optimization process; and using third-level feature points only for sensor pose estimation of the first image frame, and not as map points.

[0014] In a second aspect, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method provided in the first aspect by executing the executable instructions.

[0015] The simultaneous localization and mapping method disclosed herein constructs continuously valued semantic weights based on the semantic category features of feature points, thereby enabling differentiated utilization of objects with varying degrees of dynamics. This effectively suppresses the negative impact of dynamic features while leveraging some geometric constraints of mildly dynamic objects. It achieves a balance between maintaining localization robustness and maximizing information utilization. Attached Figure Description

[0016] Figure 1 The diagram shown illustrates a scenario applicable to an embodiment of this disclosure.

[0017] Figure 2 The diagram shown is a flowchart of a synchronous positioning and mapping method provided in an embodiment of this disclosure.

[0018] Figure 3 The diagram shown is a flowchart illustrating the steps for determining feature point weights based on semantic category features according to an embodiment of this disclosure.

[0019] Figure 4 The diagram shown is a flowchart illustrating the steps for determining the semantic weight of a feature point based on the semantic prior weight of at least one semantic category, according to an embodiment of this disclosure.

[0020] Figure 5 The diagram shown is a flowchart illustrating the steps for determining the feature point weights based on the semantic weights of feature points according to an embodiment of this disclosure.

[0021] Figure 6 The diagram shown is a flowchart illustrating the steps of constructing a weighted optimization objective function based on feature point weights according to an embodiment of this disclosure.

[0022] Figure 7 The diagram shown is a flowchart of a synchronous positioning and mapping method provided in another embodiment of this disclosure.

[0023] Figure 8 The diagram shown is a flowchart illustrating the steps of performing loop closure verification and global map optimization based on a first image frame and a loop closure frame, according to an embodiment of this disclosure.

[0024] Figure 9 The diagram shown is a schematic representation of a synchronous positioning and mapping device provided in an embodiment of this disclosure.

[0025] Figure 10 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0026] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0027] In related visual SLAM solutions, handcrafted feature SLAM schemes, such as ORB-SLAM and SIFT-based SLAM, only extract low-level geometric features such as corners and edges, lacking an understanding of the semantic entities to which these features belong. This leads to the extraction of unreliable features in weakly textured regions (such as white walls) and the assignment of excessively high confidence in strongly textured but semantically unstable regions (such as floor patterns or swaying curtains). Furthermore, this scheme applies uniform weights to all feature points, and when feature points belong to dynamic objects (such as pedestrians, vehicles, or swinging doors), they are still included in pose optimization, causing trajectory drift and map ghosting problems.

[0028] Semantic SLAM schemes, such as DynaSLAM and SG-SLAM, divide regions in a scene into static and dynamic categories based on semantics, and then use binary masks to remove feature points from dynamic regions, thereby suppressing dynamic interference. However, this approach suffers from information waste: stationary vehicles and fixed furniture, although semantically classified as dynamic, are actually static. Directly removing their feature points based on semantics results in the loss of usable geometric constraints. Furthermore, this approach typically employs a pipelined architecture of semantic segmentation, feature extraction, and final filtering. This discrete processing method leads to a disconnect between feature point weights and semantic confidence.

[0029] To address the above issues, this disclosure provides a simultaneous localization and mapping method, comprising: acquiring an image frame sequence, the image frame sequence including a first image frame; performing feature extraction operations on the geometric and semantic attributes of the first image frame respectively to obtain feature points in the first image frame and semantic category features of the feature points; determining feature point weights based on the semantic category features, the feature point weights including at least the semantic weights of the feature points, the semantic weights representing the static stability confidence of the semantic category to which the feature points belong, the semantic weights being between a first value indicating complete unreliability and a second value indicating complete reliability; constructing a weighted optimization objective function based on the feature point weights of the feature points, and performing sensor pose estimation and map optimization based on the weighted optimization objective function to achieve a balance between information utilization and robustness.

[0030] The following is a combination of... Figure 1 This invention introduces an application scenario to which an embodiment of the present disclosure is applicable.

[0031] Figure 1 The diagram illustrates a scenario applicable to an embodiment of this disclosure. Figure 1 As shown, this scenario includes a mobile device 110 and a processing unit 120. The processing unit 120 can be deployed on the mobile device 110 or connected via network communication.

[0032] For example, the mobile device 110 may be a robot, an augmented reality headset, or a handheld scanner. The mobile device 110 is equipped with at least one image acquisition device, such as an RGB camera, which may be a monocular camera, a binocular camera, or an RGB-D camera. Optionally, the mobile device 110 may also be equipped with an inertial measurement unit (IMU) and / or a wheel encoder for gravity orientation prior and motion prediction. When the mobile device 110 moves in the target environment, it acquires image frame sequences of the target environment in real time through the image acquisition device, and simultaneously estimates its own pose and constructs an environmental map using the synchronous positioning and mapping method provided in this embodiment.

[0033] The processing unit 120 may include one or more processors and is connected to at least one image acquisition device. When the processing unit 120 is a cloud server or an edge computing node, the mobile device 110 sends the acquired image frame sequence to the processing unit 120 via a wireless network. The processing unit 120 then executes a simultaneous localization and mapping (SLAM) method and returns the pose and map results to the mobile device 110 to support subsequent tasks such as path planning, autonomous exploration, navigation, or virtual-real fusion.

[0034] The synchronous positioning and mapping method provided in this disclosure will be described in detail below with reference to the accompanying drawings and embodiments.

[0035] First, this disclosure provides a method for synchronous positioning and mapping, which can be executed by any electronic device with computing capabilities, such as the processing unit described above.

[0036] Figure 2 The diagram shown is a schematic flowchart of a synchronous positioning and mapping method provided in an embodiment of this disclosure. Figure 2 As shown in the embodiments of this disclosure, the synchronous positioning and mapping method includes the following steps.

[0037] S210, acquire the image frame sequence.

[0038] Environmental images are captured by an image acquisition device mounted on a mobile device and formed into an image frame sequence in chronological order.

[0039] The image frame sequence includes a first image frame. The first image frame can be any frame in the image frame sequence; for example, it can be a keyframe or a regular frame recently acquired by the image acquisition device.

[0040] S220, feature extraction operations are performed on the geometric and semantic attributes of the first image frame to obtain the feature points in the first image frame and the semantic category features of the feature points.

[0041] Geometric attributes are visual information related to spatial structure, texture gradient, corner points, etc., in the first image frame. Semantic attributes are semantic information about the object categories included in the first image frame. By extracting features from the geometric and semantic attributes in the first image frame respectively, it is possible to simultaneously obtain feature points that can be used for localization and mapping, as well as semantic category features representing the object category to which the feature points belong.

[0042] In this context, feature points are locations in the first image frame that possess significant texture or geometric structure, such as corner points, edge points, or spots. Semantic category features are used to describe the probability or confidence level of a feature point belonging to one or more semantic categories; for example, they can be represented as a probability distribution vector.

[0043] For example, semantic category features It can be represented as , ,in This represents the confidence level of feature point i belonging to semantic category k. For example, for feature points on a wall, the confidence level of the semantic category "wall" is higher, while the confidence level of other categories such as "ground" and "pedestrian" is lower. Semantic categories can include wall, ground, ceiling, column, painting, picture frame, clock, door, window, sofa, cabinet, bed, pedestrian, vehicle, plant, curtain, etc. It should be noted that the above semantic categories are for illustrative purposes only. Those skilled in the art can add, delete, or refine the categories according to actual application scenarios, such as adding categories like "traffic signs" and "lane lines" in autonomous driving scenarios.

[0044] In this embodiment of the disclosure, feature extraction of geometric attributes can employ any hand-designed or learned feature detection and description method (such as ORB, SIFT, SuperPoint-like CNN, etc.). Feature extraction of semantic attributes can employ any semantic segmentation or object detection network (such as FCN, ASPP, YOLO, etc.).

[0045] S230 determines the feature point weights based on semantic category features.

[0046] Feature point weights are used to characterize the credibility or constraint contribution of feature points in the SLAM process. Feature point weights include at least the semantic weights of the feature points.

[0047] Semantic weight refers to the static stability confidence level of a feature point's semantic category, determined based on its semantic category features. The value of the semantic weight ranges between a first value indicating complete unreliability and a second value indicating complete reliability. Specifically, the first value (e.g., 0) indicates that the feature point is completely unreliable, such as when it is located on a highly dynamic object; the second value (e.g., 1) indicates that the feature point is completely reliable, such as when it is located on a fixed object. Semantic weight reflects the degree to which a feature point is suitable as a localization reference due to the inherent motion properties of the object it belongs to.

[0048] For example, feature points on paintings or clocks have extremely high static stability in their semantic categories, so their corresponding semantic weights are closer to the second value (e.g., 0.95). Feature points on people or pets, on the other hand, have lower static stability in their semantic categories, so their corresponding semantic weights are closer to the first value (e.g., 0.05).

[0049] It can be observed that the semantic weights range from a continuous interval between the first and second values, rather than simple binary labels. Therefore, it is possible to perform fine-grained modeling of intermediate states between completely trustworthy and completely untrustworthy. For example, a moderate weight can be assigned to objects with slight dynamics (such as plants swaying in the wind) to achieve differentiated utilization of feature points on objects with different degrees of dynamics and improve the information utilization rate of dynamic objects.

[0050] As an optional implementation, a mapping table between semantic categories and semantic weights can be pre-constructed, and the semantic weight of a feature point can be obtained by looking up the table based on the semantic category features of the feature point. Alternatively, the semantic weights can be predicted using a weight prediction network; this disclosure does not impose any specific limitations on this approach.

[0051] In addition, feature point weights can also include other types, such as geometric stability weights. In practical applications, semantic weights can be directly used as feature point weights in subsequent optimizations, or semantic weights can be fused with one or more other types of weights, and the fusion result can be used as the feature point weights.

[0052] S240 constructs a weighted optimization objective function based on the feature point weights of the feature points, and performs sensor pose estimation and map optimization based on the weighted optimization objective function.

[0053] After obtaining the feature point weights of each feature point, the first image frame is matched with other image frames in the image frame sequence (such as adjacent key frames before the first image frame) to establish the correspondence between feature points between frames. Then, the relative sensor pose transformation between adjacent frames is estimated using methods such as epipolar geometry and PnP. At the same time, the successfully matched feature points are triangulated and used as map points.

[0054] Next, the sensor pose and feature points are adjusted through backend optimization.

[0055] Specifically, within a sliding window, several keyframes and their jointly observed feature points are maintained. Based on the reprojection error of each feature point in each keyframe and the feature point weight, a weighted optimization objective function is constructed.

[0056] Based on a weighted optimization objective function, an optimizer is used to jointly iteratively solve for the sensor pose and feature points. The higher the feature point weight, the greater the cost of its reprojection error during the optimization process, forcing the optimizer to prioritize satisfying the geometric constraints of these feature points. Conversely, the lower the feature point weight, the smaller the cost of its reprojection error during the optimization process. Even if it has a large error, it will not have a decisive impact on the sensor pose and map structure, so that feature points with high static stability contribute stronger constraints, while feature points with low static stability contribute weaker constraints.

[0057] By minimizing this weighted optimization objective function, the optimizer can simultaneously solve for the current sensor pose and the spatial coordinates of feature points, thereby obtaining a globally consistent sensor trajectory and environment map.

[0058] In this embodiment, semantic weights with continuously varying values ​​are constructed based on the semantic category features of feature points, thereby enabling differentiated utilization of objects with varying degrees of dynamics. This effectively suppresses the negative impact of dynamic features while utilizing some geometric constraints of objects with mild dynamics. A balance is achieved between maintaining localization robustness and maximizing information utilization.

[0059] The following section introduces a specific implementation method for jointly sensing the geometric and semantic attributes of the first image frame.

[0060] In some embodiments, the step of performing feature extraction operations on the geometric and semantic attributes of the first image frame to obtain feature points in the first image frame and semantic category features of the feature points includes: inputting the first image frame into a joint feature extraction network to obtain feature points and semantic category features of the feature points.

[0061] The joint feature extraction network adopts an encoder-dual-branch decoder architecture, which includes a shared encoder, a first branch decoder, and a second branch decoder.

[0062] The encoder is used to perform feature encoding operations on the first image frame to obtain the image feature code.

[0063] The encoder can employ convolutional neural networks, such as ResNet-50 or ConvNeXt-TinyBackbone. When the first image frame (e.g., an RGB image of size 3×H×W) is input into the encoder, the encoder extracts hierarchical features from low-level edges and textures to high-level semantics through stacked convolutional layers, batch normalization layers, and activation function layers. Simultaneously, the spatial resolution of the feature map is progressively reduced (e.g., downsampled to 1 / 8 of the original size) through convolutional or pooling layers to increase the receptive field and reduce subsequent computation. Finally, the encoder outputs a high-dimensional shared feature map (e.g., size 256×H / 8×W / 8), denoted as the image feature encoding. This image feature encoding can be shared by the subsequent two branch decoders, thus achieving resource sharing and feature consistency.

[0064] The first branch decoder is used to generate the location of feature points and descriptors based on image feature encoding.

[0065] The first branch decoder can employ networks used for feature point detection and description, such as SuperPoint or ASLFeat. The first branch decoder further processes the image feature encoding to determine multiple feature points in the first image frame, along with the location coordinates and descriptors of each feature point. The descriptor of a feature point is a high-dimensional vector used to identify the local appearance of that feature point; in subsequent inter-frame matching, the correspondence is established by calculating the similarity between the descriptors. For example, the output dimension of the first branch decoder is N×(2+256), where N is the number of feature points.

[0066] The second branch decoder is used to generate semantic category features of feature points based on image feature encoding.

[0067] The second branch decoder runs in parallel with the first branch decoder. The second branch decoder can employ semantic segmentation or object detection networks, such as ASPP, SegFormer, etc. The second branch decoder performs semantic category prediction based on image feature encoding to determine the semantic category features of feature points. For example, the output dimension of the second branch decoder is N×K, where K is the number of semantic categories.

[0068] Next, the outputs of the first and second branch decoders are input together into the fully connected layer. The fully connected layer fuses the outputs of the first and second branch decoders, outputting feature points and their corresponding semantic category features and descriptors. For example, the output dimension of the fully connected layer is N×1.

[0069] In this embodiment, the shared encoder provides features to both the first and second branch decoders, and the dual branch decoders output in parallel, breaking the cascaded pattern of segmentation followed by feature extraction, thereby achieving semantic awareness in the feature extraction stage and avoiding the problem of feature point weights being disconnected from semantic confidence.

[0070] The joint feature extraction network described above can be trained using a multi-task learning approach. For example, a joint training objective comprising four loss functions can be constructed: in, This represents the feature point detection loss; This represents the descriptor distinguishability loss; This represents the semantic classification cross-entropy loss; λ represents the weighted regression loss; λ1, λ2, and λ3 are the weights of each loss.

[0071] Feature point detection loss is supervised based on the repeatability of feature points, encouraging the network to generate repeatable keypoints in texture-rich and geometrically stable locations; this can be represented by Focal Loss. Descriptor discriminative loss minimizes the descriptor distance of the same feature point across different viewpoints while maximizing the descriptor distance between different feature points; this can be represented by Triplet Loss. Semantic classification cross-entropy loss supervises the output of the second-branch decoder, ensuring it aligns with the semantic labels of corresponding points on the image frame. Weight regression loss supervises the error between the network's predicted weights and their ground truth values; it can be expressed as follows: in, This represents the predicted weight value for feature point i. The weight of feature point i represents the true value, which can be determined through offline annotation.

[0072] The following section introduces a specific implementation method for determining feature point weights.

[0073] Figure 3 The diagram shown is a flowchart illustrating the steps for determining feature point weights based on semantic category features according to an embodiment of this disclosure. Figure 3 As shown in this embodiment, the step of determining the feature point weights based on semantic category features includes the following steps.

[0074] S310, determine the semantic prior weights for at least one semantic category.

[0075] As described in the foregoing embodiments, the semantic category feature represents at least one semantic category corresponding to the feature point, and the probability that the feature point belongs to that semantic category.

[0076] Determine the semantic prior weights for each semantic category. The semantic prior weights refer to the static stability confidence scores pre-assigned to each semantic category, characterizing the reliability with which objects of that category maintain their spatial position and geometric shape over long time scales.

[0077] For example, a correspondence table between semantic categories and semantic prior weights can be pre-established, and the semantic prior weights corresponding to each semantic category can be determined based on the correspondence table.

[0078] Optionally, in this correspondence table, the semantic prior weights corresponding to different semantic categories, from high to low, are as follows: rigid, fixed, long-term stable semantic categories with rich geometric textures (such as paintings, picture frames, clocks, etc.); semi-static semantic categories that move occasionally but are usually stable (such as sofas, beds, etc.); absolutely static semantic categories with poor quality of weak texture features (such as walls, ceilings, etc.); semantic categories such as doors and windows that can be opened and closed and whose state is uncertain; semantic categories with repetitive and easily confused textures and whose positions may change (such as floors, carpets, etc.); flexible and swaying semantic categories with geometric instability (such as curtains, potted plants, etc.); and highly dynamic semantic categories (such as people, animals, etc.).

[0079] The semantic prior weights can be set to values ​​between a first value and a second value. For example, the first and second values ​​form the interval [0, 1], and the semantic prior weights corresponding to different semantic categories can be set from high to low as follows: 0.95, 0.85, 0.8, 0.5, 0.4, 0.3, and 0.05. It is understood that the above semantic prior weight values ​​are exemplary and can be adjusted in practical applications based on the object motion patterns of the specific scenario and engineering experience.

[0080] S320, determine the semantic weight of the feature point based on the semantic prior weight of at least one semantic category.

[0081] For each feature point, each component of its semantic category feature (i.e., the confidence that the feature point belongs to the semantic category) is multiplied by the semantic prior weight corresponding to the semantic category, and then all products are fused to obtain the semantic weight of the feature point.

[0082] S330 determines the feature point weights based on the semantic weights of the feature points.

[0083] In this step, the feature point weights can be directly used in subsequent optimizations, or the semantic weights can be fused with one or more other types of weights, and the fusion result can be used as the feature point weights.

[0084] In this embodiment, the semantic category corresponding to the feature point is first quantized into semantic prior weights, and then the different semantic prior weights are fused to obtain the semantic weight. The semantic prior weight of each semantic category has a clear physical meaning, strong interpretability, and is computationally efficient and easy to adapt to different scenarios.

[0085] Figure 4 The diagram illustrates a flowchart of a step in determining the semantic weights of feature points based on the semantic prior weights of at least one semantic category, according to an embodiment of this disclosure. Figure 4 As shown in this embodiment, the step of determining the semantic weight of a feature point based on the semantic prior weight of at least one semantic category includes the following steps.

[0086] S410, determine the geometric confidence factor for at least one semantic category.

[0087] The geometric confidence factor is determined based on the texture richness of the local region where the feature point is located. It is used to reflect the geometric observation quality of the local region of the feature point. Specifically, it can be quantified according to the texture richness, corner response intensity or depth continuity in the neighborhood of the feature point, and then normalized as the geometric confidence factor of the semantic category.

[0088] For example, the semantic category of a feature point includes the wall category: if the feature point is located in a textured area covered by wallpaper, it is assigned a higher geometric confidence factor (e.g., 1.0); if it is located in a white wall or a weak textured area, it is assigned a lower geometric confidence factor (e.g., 0.3).

[0089] For example, the semantic category of a feature point includes the ground category: if the feature point is located in a tiled area with obvious texture, it is assigned a higher geometric confidence factor (e.g., 0.9); if it is located in a smooth, textureless cement ground, it is assigned a lower geometric confidence factor (e.g., 0.2).

[0090] S420, determine the semantic weight of the feature point based on the geometric confidence factor and semantic prior weight of at least one semantic category.

[0091] The semantic weight of a feature point is obtained by weighting and summing the semantic prior weights and geometric confidence factors of multiple semantic categories based on the confidence that the feature point belongs to the semantic category.

[0092] Optionally, semantic weights can be calculated using the following formula.

[0093] in, The semantic weight of feature point i is represented. This represents the confidence level that feature point i belongs to semantic category k. The semantic prior weights of semantic category k are represented. The geometric confidence factor represents the semantic category k, where K represents the number of semantic categories.

[0094] In this embodiment, by introducing a geometric confidence factor within the semantic category, the semantic weights of feature points are locally fine-tuned based on the semantic prior weights, thus achieving refined modeling of differences within the semantic category. The semantic weights of feature points are automatically reduced in weakly textured regions to avoid interference from unreliable features in the subsequent optimization process, thereby achieving a joint evaluation of semantic priors and local geometric quality.

[0095] Building upon the aforementioned embodiments, to further enhance the comprehensiveness and robustness of feature point weights, the semantic weights of feature points can be fused with other weights reflecting observation quality or motion state to form the final feature point weights. This will be described in detail below.

[0096] Figure 5 The diagram shown is a flowchart illustrating the steps for determining feature point weights based on semantic weights of feature points, according to an embodiment of this disclosure. Figure 5 As shown in the embodiments of this disclosure, the step of determining the feature point weight based on the semantic weight of the feature point includes the following steps.

[0097] S510, obtain the weights of other dimensions of the feature points.

[0098] In some embodiments, the other dimensional weights of the feature points may include at least one of geometric stability weights, dynamic weights, and temporal consistency weights.

[0099] Geometric stability weights are used to measure the geometric observation quality of a local region of a feature point, such as texture richness, depth continuity, and corner response strength. For example, geometric stability weights can be calculated using the following formula.

[0100] in, The geometric stability weights of feature point i are represented. This represents the gradient variance of the neighborhood (e.g., 15×15 pixels) of feature point i. The larger the value, the richer the texture. These are the depth values ​​of feature point i and other feature points in the neighborhood of feature point i, respectively. The score represents the Shi-Tomasi corner response score. The higher the score, the more significant the corner structure of the feature point. α, β, and δ are weighting coefficients used to balance the contribution of each factor to the geometric stability weight. Empirical values ​​can be 0.4, 0.4, and 0.2, respectively. In practical applications, these values ​​can be adjusted according to the sensor characteristics or the scene. This represents the Sigmoid function, which normalizes the linear weighted sum within the parentheses to the (0, 1) interval, serving as the final geometric stability weights.

[0101] Dynamic weights are used to reflect whether feature points have moved at the current moment, and are typically calculated based on the optical flow residual or reprojection error between consecutive frames. For example, dynamic weights can be calculated using the following formula.

[0102] For two frames Feature points in Calculate the optical flow residual: in, Indicates optical flow residual; The first Frame and frame t; , These represent the feature point i at the th... Position coordinates in frame t and frame t, The speed of the camera's own movement can be estimated using an IMU or the pose of the previous frame; Δt represents the time interval between two frames.

[0103] Dynamic weights can be represented as: in, Represents the dynamic weight of feature point i; Indicates the standard deviation of optical flow noise (e.g., 3 pixels); This represents the static judgment threshold (e.g., 5 pixels). When the magnitude of the optical flow residual is less than this threshold, the feature point is considered to be static; when it is greater than or equal to the threshold, the feature point is considered to be a clearly dynamic object. This is the lowest weight to retain obviously dynamic objects, to avoid completely removing obviously dynamic objects that could lead to tracking loss or map fragmentation. Its value can be set to a small value, such as 0.1. This indicates an indicator function that takes the value 1 if the condition is true, and 0 otherwise.

[0104] Temporal consistency weights are used to characterize the stability of the same feature point across multiple frames of observation within a sliding window, and can be obtained by calculating the variance of the reprojection error sequence. For example, the temporal consistency weights can be calculated using the following formula.

[0105] First, calculate the reprojection error variance using the following formula: in, This represents the variance of the reprojection error; This represents the set of frames in which feature point i was observed; This represents the observed coordinates of feature point i in the j-th frame. This represents the sensor pose in the j-th frame; Represents the three-dimensional spatial coordinates of feature point i; This represents the projection function, used to project... The predicted pixel coordinates are obtained by projecting the sensor coordinate system onto the image plane. This represents the variance function.

[0106] The timing consistency weight can be expressed as: in, The temporal consistency weight represents feature point i; This represents the sensitivity coefficient, used to adjust the strength of the influence of variance on the weights; an empirical value of 0.5 is recommended.

[0107] This step can compute one or more of the above weights in parallel or serially. For example, the geometric stability weight depends only on the local texture and depth information of the current frame image and does not involve inter-frame correlation. Therefore, its computation can be arranged at the first processing end (e.g., the post-processing stage of the feature extraction network) and executed in parallel with feature point detection. The dynamic weight and temporal consistency weight depend on the optical flow information of the previous and next frames and the reprojection error statistics within a multi-frame window, respectively. They need to be processed across frames. Their computation can be arranged at the second processing end (e.g., the dynamic verification module before local optimization) and performed in conjunction with the front-end odometry or sliding window management.

[0108] S520 integrates the semantic weights of feature points with the weights of other dimensions to obtain the feature point weights.

[0109] The fusion method can employ weighted geometric average, weighted arithmetic average, or fuzzy logic operations, among others.

[0110] As an optional implementation, the feature point weights can be calculated using the following formula: in, The weight of feature point i represents the feature point weight. 、 、 、 These are the exponential coefficients of semantic weight, dynamic weight, geometric stability weight, and temporal consistency weight, respectively. Empirical values ​​for these four coefficients can be 0.3, 0.3, 0.2, and 0.2, respectively. If the weight of any dimension is close to 0, it will lead to a decrease in the feature point weight. The weight also approaches 0, thus automatically eliminating the unreliable feature; the overall weight only approaches 1 when all dimension weights are high. This achieves joint constraint of multi-source confidence, avoiding inaccurate weights caused by misjudgment of a single dimension.

[0111] As an optional implementation method, , , , The value can be determined based on the reinforcement learning process. For example, in a simulation environment, the proximal policy optimization module outputs an action based on the current state. The environment updates the fusion coefficients and runs for a period of time, calculating the cumulative reward, and then updates the policy network and value network. Through numerous iterations, the proximal policy optimization module can learn a policy that dynamically adjusts the fusion coefficients according to different scenarios.

[0112] The current state includes feature point weight distribution, tracking quality, and map quality, which can be obtained in real time through the operational status of the SLAM system's front-end and back-end. Actions include adjusting the tiny increment vectors of the four fusion coefficients, each increment taking a value within a continuous interval, and re-normalizing the updated coefficients so that the sum of the four coefficients equals 1. These actions are sampled by the policy network based on the current state. The reward includes the sum of localization accuracy and map stability.

[0113] It is understood that, in addition to the weights listed above, other auxiliary weights of other dimensions may be introduced in the embodiments of this disclosure. The weights of the above-mentioned dimensions can be fused with semantic weights in a similar manner. Any extension of the weight dimensions or substitution of the fusion method without departing from the principles of this disclosure shall fall within the protection scope of this invention.

[0114] This embodiment introduces a multi-dimensional weight fusion mechanism. The weights of the four dimensions of semantics, geometry, dynamics and time series respectively characterize the reliability of feature points from the perspectives of static priors of objects, local observation quality, instantaneous motion state and long-term observation consistency. This effectively suppresses the interference of unreliable features and realizes the joint evaluation of multi-source confidence information, providing more comprehensive feature support for the subsequent optimization process.

[0115] Next, we will introduce an implementation method based on feature point weights to construct a weighted optimization objective function, in order to improve the back-end optimization's ability to suppress dynamic interference and the fitting accuracy of high-confidence features. A detailed explanation follows.

[0116] Figure 6 The diagram shows a flowchart illustrating the steps of constructing a weighted optimization objective function based on feature point weights according to an embodiment of this disclosure. Figure 6 As shown in the embodiments of this disclosure, the step of constructing a weighted optimization objective function based on the feature point weights of feature points includes the following steps.

[0117] S610 determines the threshold parameter in the robust kernel function based on the feature point weights of the feature points.

[0118] Robust kernel functions (such as Huber kernel, Cauchy kernel, or Tukey kernel) are functions used to suppress the influence of outliers on optimization results. They behave as quadratic functions when the residuals are small (less than the threshold parameter) to preserve the least squares property, and become linear or constant when the residuals are large (greater than the threshold parameter) to weaken the influence of large residuals.

[0119] The larger the feature point weight, the stricter the threshold parameter is applied to the high-weight feature points to force an accurate fit; the smaller the feature point weight, the more lenient the threshold parameter is applied to the low-weight feature points to tolerate a certain degree of error.

[0120] Taking the Huber kernel as an example, based on feature point weights, the threshold parameter can be expressed in the following form: in, Indicates the threshold parameter; Indicates the basic threshold; This represents the adaptive intensity coefficient, used to control the weights of feature points. The degree of influence on the threshold parameter.

[0121] S620 constructs a weighted optimization objective function for the reprojection error term of feature points based on a robust kernel function and feature point weights.

[0122] The reprojection error term refers to the difference vector between the projected position of a feature point on a keyframe and its actual observed position. In weighted optimization, a robust kernel function can be used to perform a nonlinear mapping on the reprojection error term of each feature point to suppress the negative impact of excessively large residuals. Furthermore, feature point weights are used to weight all observation residuals of the corresponding feature point to achieve differentiated weighting for feature points with different confidence levels.

[0123] For example, weighted optimization objective function It can be represented in the following form: in, Represents a robust kernel function; This represents the reprojection error term. .

[0124] In this embodiment, adaptive differential processing for backend optimization is achieved by embedding feature point weights into the threshold parameter of a robust kernel function and the coefficients of the observation residuals. On one hand, a strict threshold parameter is applied to feature points with high feature point weights to ensure accurate fitting of high-confidence features; a more lenient threshold parameter is applied to feature points with low feature point weights to avoid distortion of the overall pose by dynamic or weak-texture features. On the other hand, the feature point weights weight the observation residuals, improving the positioning accuracy and grounding in dynamic and weak-texture scenes. Figure 1 To the point of being responsive.

[0125] Next, to eliminate the cumulative drift caused by long-term operation, loop closure detection and global optimization mechanisms can be further introduced. These will be discussed in detail below.

[0126] Figure 7 The diagram shown is a flowchart illustrating a synchronous positioning and mapping method provided in another embodiment of this disclosure. Figure 7 As shown in the embodiments of this disclosure, the synchronous positioning and mapping method further includes the following steps.

[0127] S710, obtain the semantic description vectors of the first image frame and the second image frame respectively.

[0128] The image frame sequence includes a second image frame, which can be any frame that is distinct from the first image frame, such as a historical keyframe preceding the first image frame.

[0129] Semantic description vectors are used to describe the semantic content distribution of the entire image frame and can be determined through the semantic category features of feature points. For example, semantic description vectors... It can be represented in the following form: in, The semantic category feature of feature point i The embedding vector can be obtained through Word2Vec or CLIP encoding; N represents the number of feature points in the frame.

[0130] S720, calculate the visual similarity between the first image frame and the second image frame, and select a loop frame in the second image frame based on the visual similarity and the similarity between the semantic description vectors of the first image frame and the second image frame respectively.

[0131] The visual similarity between the first and second image frames can be calculated using the traditional Bag-of-Words (BoW) model, which compares the similarity of the distribution of descriptors of feature points between the two frames to obtain a similarity score.

[0132] The similarity between the semantic description vectors of the first and second image frames can be expressed using cosine similarity or Euclidean distance.

[0133] The second image frame is identified as a loopback frame only when the visual similarity is greater than the first threshold and the similarity between the semantic description vectors is greater than the second threshold.

[0134] The S730 performs loopback verification and global map optimization based on the first image frame and the loopback frame.

[0135] After determining the loop closure frame, the authenticity of the loop closure constraint can be confirmed through loop closure verification (e.g., by feature point matching, calculating the relative pose using RANSAC, and checking the number of inliers and reprojection error). If the verification passes, the loop closure edges of the first image frame and the loop closure frame are constructed. The loop closure edges and their weights are added to the pose graph, and global map optimization is performed.

[0136] Traditional bag-of-words models rely solely on the distributional similarity of descriptors, which can easily lead to perceptual confusion in scenarios that are geometrically similar but semantically different. For example, two similar meeting rooms may have almost identical geometric layouts, but they are actually different rooms and cannot be effectively distinguished by geometric features alone.

[0137] In this embodiment of the disclosure, by introducing semantic description vectors, high-level semantic information of object category distribution in the scene is captured (for example, although two conference rooms have almost the same geometric layout, the semantic content of the decorative paintings hanging inside, the furniture placed, the switches on the wall or the whiteboard may be different), thereby effectively excluding second image frames that are geometrically similar but have different semantic distributions.

[0138] The following section presents a method to improve loop closure detection accuracy based on semantic category features. A detailed explanation follows.

[0139] Figure 8 The diagram shows a flowchart illustrating the steps of loop closure verification and global map optimization based on a first image frame and a loop closure frame, according to an embodiment of this disclosure. Figure 8 As shown in the embodiment of this disclosure, the steps of performing loop closure verification and global map optimization based on the first image frame and the loop closure frame include the following steps.

[0140] S810, determine the matching feature point pairs between the first image frame and the loopback frame.

[0141] The matching feature point pair includes a first feature point in the first image frame and a second feature point in the loopback frame, wherein the first feature point and the second feature point correspond to the same map point.

[0142] Matching feature point pairs can be determined by performing nearest neighbor matching on the descriptors of the feature points.

[0143] S820, based on the semantic category consistency between the semantic category features of the first feature point and the second feature point, and the feature point weights of the first feature point and the second feature point, determines the loop edge weights between the first image frame and the loop frame.

[0144] The lap edge weight is used to quantitatively characterize the reliability of the lap constraint.

[0145] Semantic category consistency refers to the consistency of the semantic categories to which the first feature point and the second feature point belong. When the semantic categories to which the first feature point and the second feature point belong are consistent, such as both being in the "hanging picture" category, it means that the matching pair is semantically homogeneous, and its feature point weights can participate normally in the calculation of loop edge weights. When the semantic categories to which the first feature point and the second feature point belong are inconsistent, such as the first feature point being in the "hanging picture" category and the second feature point being in the "wall" category, the feature point weights of the matching pair should be ignored or appropriately restricted to avoid low-quality loops polluting the global map.

[0146] For example, semantic category consistency can be determined by the following formula: in, , These represent the feature point weights of the first feature point i and the second feature point j, respectively. This represents the set of matching feature point pairs between the first image frame and the loopback frame; Let i and j represent the semantic categories of the first feature point i and the second feature point j, respectively. The category with the highest confidence among the semantic category features can be selected. This indicates an indicator function that takes the value 1 if the condition is true, and 0 otherwise.

[0147] S830 constructs a loop optimization objective function based on loop edge weights and performs global map optimization based on the loop optimization objective function.

[0148] The objective function for loop closure optimization is the total cost function for pose graph optimization, including odometry error and loop closure error terms. The optimization process can be expressed in the following form: in, This represents the optimal set of sensor poses, which is the result obtained after optimizing the poses of all keyframes, and serves as the optimization target. This indicates the search for an objective function that optimizes cyclic loops. The operation on the smallest variable T; This represents the odometry error term, which is the difference between the relative pose measurement between the current frame and the loopback frame and the relative pose calculated by the current optimization variables. The loop closure error term represents the difference between the relative pose measurement value given by the loop closure check and the relative pose calculated by the current optimization variable between the current frame and the loop closure frame.

[0149] In this embodiment, by minimizing the loop closure optimization objective function, it is possible to eliminate the cumulative drift caused by long-term operation while ensuring local smoothness, thereby obtaining a globally consistent sensor pose trajectory and a high-precision map. Specifically, determining the loop edge weights based on semantic consistency and feature point weights ensures that high-semantic-confidence loops dominate the global optimization, while semantically inconsistent or feature-unreliable loops are suppressed, thus improving the robustness of loop closure and map quality.

[0150] To further improve the efficiency of map management and the stability of long-term operation, feature points can be managed in a hierarchical manner based on their weights. This will be explained in detail below.

[0151] In some embodiments, the synchronous localization and mapping method further includes: hierarchical management of feature points based on feature point weights.

[0152] Feature points can be divided into multiple levels based on their weights. For example, two thresholds (e.g., 0.8, 0.3) can be set, and feature points can be divided into three levels based on these two thresholds. Hierarchical management includes at least one of the following.

[0153] First-level feature points are designated as long-term map points and participate in the global map optimization process. These first-level feature points have relatively high feature weights (e.g., greater than 0.8). These feature points are typically located on long-term, fixed objects with rich textures and stable semantics (e.g., paintings on a wall, corner points of structural columns). These feature points are fixed as long-term map points, added to the global map, participate in subsequent global optimization processes, and are used for relocalization and long-term tracking. Long-term map points are not easily removed unless they have not been observed for an extended period or the observation quality continuously deteriorates.

[0154] Second-level feature points are used as temporary map points, which only participate in the local map optimization process. The feature point weights of these second-level feature points are moderate (e.g., greater than 0.3 but less than 0.8). These feature points may be located in semi-static objects or areas with general textures (e.g., doors, windows, potted plants, etc.), and while they have some reference value, they are not very stable. Temporary map points participate in local optimization; their lifespan is monitored, and if their weight continuously decreases over multiple frames (e.g., 5 frames), they are removed to avoid introducing drift.

[0155] Third-level feature points are used only for sensor pose estimation in the first image frame and are not included as map points. Third-level feature points have relatively small feature point weights (e.g., less than 0.3). These feature points are only used for pose estimation in the current frame and are not included in the map.

[0156] It should be noted that the above thresholds and specific management strategies can be adjusted according to the application scenario and sensor characteristics, and this disclosure does not impose any specific restrictions on them.

[0157] In this embodiment of the disclosure, a hierarchical management method based on feature point weights is adopted. High-weight feature points are fixed as long-term map points and participate in global optimization to provide reliable core landmarks. Low-weight feature points are only used for tracking in the current frame to prevent tracking loss and avoid map pollution, which significantly improves the efficiency of map maintenance and the stability of long-term operation.

[0158] The above text combined Figures 1 to 8 The method embodiments of this disclosure have been described in detail below, in conjunction with... Figure 9 The apparatus embodiments of this disclosure are described in detail below. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be referred to the foregoing method embodiments.

[0159] Figure 9The diagram shown is a structural schematic of a synchronous positioning and mapping device provided in an embodiment of this disclosure. Figure 9 As shown, the synchronous positioning and mapping device 900 of this embodiment includes: an acquisition module 910, a feature extraction module 920, a determination module 930, and an optimization module 940.

[0160] The acquisition module 910 is configured to acquire an image frame sequence, the image frame sequence including a first image frame.

[0161] The feature extraction module 920 is configured to perform feature extraction operations on the geometric attributes and semantic attributes of the first image frame respectively, to obtain the feature points in the first image frame and the semantic category features of the feature points.

[0162] The determination module 930 is configured to determine the feature point weights of feature points based on semantic category features. The feature point weights include at least the semantic weights of the feature points. The semantic weights characterize the static stability confidence of the semantic category to which the feature points belong. The range of the semantic weights is between a first value indicating complete unreliability and a second value indicating complete reliability.

[0163] The optimization module 940 is configured to construct a weighted optimization objective function based on the feature point weights of the feature points, and perform sensor pose estimation and map optimization based on the weighted optimization objective function to achieve a balance between information utilization and robustness.

[0164] Below, for reference Figure 10 To describe an electronic device according to embodiments of the present disclosure. Figure 10 The diagram shown is a structural schematic of an electronic device provided according to an embodiment of this disclosure. Figure 10 As shown, the electronic device 1000 includes one or more processors 1010 and memory 1020.

[0165] The processor 1010 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1000 to perform desired functions.

[0166] The memory 1020 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1010 may execute the program instructions to implement the synchronous positioning and mapping methods of the various embodiments of this disclosure described above, and / or other desired functions.

[0167] In some embodiments, the electronic device 1000 may further include an input device 1030 and an output device 1040, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0168] The input device 1030 may include, for example, a touch screen, microphone, keyboard, mouse, etc. The output device 1040 may include, for example, a display, speaker, and communication network and its connected remote output devices, etc.

[0169] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 1000 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1000 may include any other suitable components depending on the specific application.

[0170] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the synchronous positioning and mapping methods according to various embodiments of this disclosure as described above.

[0171] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0172] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the synchronous positioning and mapping methods according to various embodiments of this disclosure described above.

[0173] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0174] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0175] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0176] It should also be noted that in the systems, apparatus, and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0177] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0178] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for simultaneous positioning and mapping, characterized in that, include: Obtain an image frame sequence, the image frame sequence including a first image frame; Feature extraction operations are performed on the geometric and semantic attributes of the first image frame to obtain feature points in the first image frame and the semantic category features of the feature points. Based on the semantic category features, the feature point weights of the feature points are determined. The feature point weights include at least the semantic weights of the feature points. The semantic weights characterize the static stability confidence of the semantic category to which the feature points belong. The value range of the semantic weights is between a first value indicating complete untrustworthiness and a second value indicating complete trustworthiness. Based on the feature point weights of the aforementioned feature points, a weighted optimization objective function is constructed, and sensor pose estimation and map optimization are performed based on the weighted optimization objective function. The construction of a weighted optimization objective function based on the feature point weights of the feature points includes: Based on the feature point weights of the aforementioned feature points, the threshold parameter in the robust kernel function is determined; Based on the robust kernel function and the feature point weights, a weighted optimization objective function for the reprojection error term of the feature points is constructed. The threshold parameter is the sum of the base threshold and the adjustment amount, where the adjustment amount is the product of the base threshold, the adaptive intensity coefficient, and the feature point weight.

2. The method according to claim 1, characterized in that, The semantic category feature represents at least one semantic category corresponding to the feature point; The step of determining the feature point weights of the feature points based on the semantic category features includes: Determine the semantic prior weights for each of the at least one semantic category; The semantic weights of the feature points are determined based on the semantic prior weights of each of the at least one semantic category. The feature point weights are determined based on the semantic weights of the feature points.

3. The method according to claim 2, characterized in that, Determining the semantic weights of the feature points based on the semantic prior weights of each of the at least one semantic category includes: Determine the geometric confidence factor for each of the at least one semantic category, wherein the geometric confidence factor is determined based on the texture richness of the local region where the feature point is located; The semantic weight of the feature point is determined based on the geometric confidence factor of each of the at least one semantic category and the semantic prior weight.

4. The method according to claim 2, characterized in that, The step of determining the feature point weights based on the semantic weights of the feature points includes: Obtain the weights of other dimensions of the feature points, wherein the other dimension weights include at least one of geometric stability weight, dynamic weight, and temporal consistency weight; The semantic weight of the feature point is fused with the weights of the other dimensions to obtain the feature point weight.

5. The method according to claim 1, characterized in that, The step of performing feature extraction operations on the geometric and semantic attributes of the first image frame to obtain feature points in the first image frame and the semantic category features of the feature points includes: The first image frame is input into a joint feature extraction network to obtain the feature points and the semantic category features of the feature points; The joint feature extraction network includes an encoder, a first branch decoder, and a second branch decoder. The encoder performs feature encoding on the first image frame to obtain image feature encoding. The first branch decoder generates the location and descriptor of the feature point based on the image feature encoding. The second branch decoder generates the semantic category feature of the feature point based on the image feature encoding.

6. The method according to claim 1, characterized in that, The image frame sequence also includes a second image frame; The method further includes: Obtain the semantic description vectors of the first image frame and the second image frame respectively, wherein the semantic description vectors are determined based on the semantic category features; Calculate the visual similarity between the first image frame and the second image frame, and based on the visual similarity and the similarity between the semantic description vectors of the first image frame and the second image frame, select a loop frame in the second image frame; Based on the first image frame and the loopback frame, loopback verification and global map optimization are performed.

7. The method according to claim 6, characterized in that, The step of performing loop closure verification and global map optimization based on the first image frame and the loop closure frame includes: Determine a matching feature point pair between the first image frame and the loopback frame, wherein the matching feature point pair includes a first feature point in the first image frame and a second feature point in the loopback frame; Based on the semantic category consistency between the semantic category features of the first feature point and the second feature point, and the feature point weights of the first feature point and the second feature point, the loop edge weights between the first image frame and the loop frame are determined. Based on the loop edge weights, a loop optimization objective function is constructed, and global map optimization is performed based on the loop optimization objective function.

8. The method according to claim 1, characterized in that, Also includes: Based on the feature point weights, the feature points are managed in a hierarchical manner; The hierarchical management includes at least one of the following: using first-level feature points as long-term map points, which participate in the global map optimization process; and using second-level feature points as temporary map points, which only participate in the local map optimization process. The third-level feature points are used only for sensor pose estimation in the first image frame and are not used as map points.

9. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 8 by executing the executable instructions.

Images