Visual simultaneous localization and mapping method based on meta-learning candidate key frame screening

Abstract

The application belongs to the technical field of computer vision and mobile robot navigation, and particularly discloses a visual simultaneous localization and mapping method based on meta-learning candidate key frame screening. In the offline training stage, the method firstly acquires historical key frames, and adopts an irrelevant meta-learning method to train a key frame relationship discrimination model. In the online running stage, firstly, the current key frame is determined, then the visual similarity between the current key frame and each historical key frame is calculated, and the candidate key frame is determined; then the key frame pair features of the current key frame and the candidate key frame are constructed, and input into the trained key frame relationship discrimination model to obtain the relationship confidence between them; the candidate key frame with the relationship confidence not lower than the relationship discrimination threshold is taken as the confidence candidate key frame; finally, the ORB feature matching and geometric consistency verification are performed on the confidence candidate key frame and the current key frame, and the loop constraint or multi-map fusion constraint is established, and then the key frame pose and the map point coordinates are updated.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and mobile robot navigation technology, specifically relating to a visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection. Background Technology

[0002] Visual simultaneous localization and mapping (VSLT) is a technique that uses a camera to continuously acquire environmental images and constructs an environmental map while estimating the camera's position and orientation. It is widely used in mobile robots, drones, and visual navigation devices. During the operation of a VSLT system, a subset of image frames is typically selected as keyframes from a series of consecutive image frames. These generated keyframes are then saved as historical keyframes in a keyframe database. When processing a new current keyframe, historical keyframes that are visually similar to the current keyframe are retrieved from the keyframe database to obtain candidate keyframes. Existing VSLT systems typically use the bag-of-words method for candidate keyframe retrieval. By comparing the bag-of-words representations of the current keyframe with those of historical keyframes, visually similar historical keyframes can be obtained. However, the bag-of-words method primarily compares images based on their visual appearance. When there are repetitive textures, structurally similar regions, changes in lighting, or changes in viewing angle in the environment, keyframes at different spatial locations may also have high visual similarity, resulting in the retrieval of historical keyframes that do not have a valid spatial correspondence among the candidate keyframes.

[0003] To further confirm the geometric relationship between the current keyframe and candidate keyframes, visual simultaneous localization and mapping (VSLAM) systems typically require geometric consistency verification processes such as feature descriptor matching, geometric transformation estimation, and map point projection matching for candidate keyframes. When the number of candidate keyframes obtained from visual bag-of-words retrieval is large, or when it includes many historical keyframes with similar visual appearances but invalid geometric relationships, the above geometric consistency verification processes need to be repeated for multiple candidate keyframes. This increases the computational load of the VSLAM system and also reduces the processing efficiency of loop closure detection and multi-map fusion. Summary of the Invention

[0004] The purpose of this invention is to propose a visual simultaneous localization and mapping method based on meta-learning candidate keyframe screening. It utilizes multiple visual scenes or multiple visual data sequences to construct a meta-learning task and pre-trains a keyframe relationship discrimination model offline. Then, it uses the trained keyframe relationship discrimination model to screen candidate keyframes and establishes loop closure constraints or multi-map fusion constraints based on geometric consistency verification results to reduce unnecessary candidate relationship verification calculations and improve the processing efficiency of loop closure detection and multi-map fusion.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: The visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection includes the following steps: Step 1. Obtain historical keyframes from the historical results of multiple visual scenes or multiple visual data sequences, preprocess the historical keyframes to obtain the features and corresponding labels of the training keyframe pairs; Step 2. Build a keyframe relationship discrimination model and use the model-independent meta-learning method to train it offline based on the features of the training keyframe pairs and the corresponding relationship labels; the keyframe relationship discrimination model is used to process the features of the training keyframe pairs and output the relationship confidence of the corresponding training keyframe pairs. Step 3. Acquire continuously captured image frames from the camera, and determine the current keyframe based on the keyframe insertion criteria; Calculate the visual similarity between the current keyframe and each historical keyframe, and construct a set of candidate keyframes corresponding to the current keyframe; Construct keyframe pair features between the current keyframe and candidate keyframes, and input them into the trained keyframe relationship discrimination model to obtain the confidence of the relationship between the current keyframe and candidate keyframes; Step 4. Filter candidate keyframes according to the relationship confidence level, and retain candidate keyframes with a relationship confidence level not lower than the preset relationship determination threshold to form a set of confidence candidate keyframes; Step 5. Perform ORB feature matching and geometric consistency verification on the current keyframe and the candidate keyframes in the set of confidence candidate keyframes, and establish loop closure constraints or multi-map fusion constraints; then update the keyframe pose and map point coordinates to obtain the optimized camera pose estimation results and map.

[0006] Furthermore, based on the aforementioned visual simultaneous localization and mapping (SSL) method based on meta-learning candidate keyframe selection, this invention also proposes a corresponding visual SSL system based on meta-learning candidate keyframe selection, which adopts the following technical solution: The visual simultaneous localization and mapping system based on meta-learning candidate keyframe selection includes the following modules: The preprocessing module is used to obtain historical keyframes from the historical running results of multiple visual scenes or multiple visual data sequences, preprocess the historical keyframes, and obtain the features and corresponding labels of the training keyframe pairs. The model training module is used to build a keyframe relationship discrimination model and uses a model-independent meta-learning method to train offline based on the features of the training keyframe pairs and their corresponding relationship labels. The keyframe relationship discrimination model is used to process the features of the training keyframe pairs and output the relationship confidence of the corresponding training keyframe pairs. The candidate retrieval and relationship confidence calculation module is used to acquire continuously captured image frames from the camera and determine the current keyframe based on the keyframe insertion criteria. Calculate the visual similarity between the current keyframe and each historical keyframe, and construct a set of candidate keyframes corresponding to the current keyframe; Construct keyframe pair features between the current keyframe and candidate keyframes, and input them into the trained keyframe relationship discrimination model to obtain the confidence of the relationship between the current keyframe and candidate keyframes; The candidate keyframe filtering module is used to filter candidate keyframes according to the relationship confidence level, and retain candidate keyframes with a relationship confidence level not lower than a preset relationship judgment threshold to form a set of confidence candidate keyframes. The geometric consistency verification, loop closure correction, and multi-map fusion module is used to perform ORB feature matching and geometric consistency verification on the current keyframe and candidate keyframes in the set of confidence candidate keyframes, and to establish loop closure constraints or multi-map fusion constraints; then, the keyframe pose and map point coordinates are updated to obtain the optimized camera pose estimation results and map.

[0007] The present invention has the following advantages: As described above, this invention proposes a visual simultaneous localization and mapping (SMR) method based on meta-learning candidate keyframe selection. This method incorporates a relationship confidence selection process based on a keyframe relationship discrimination model between visual bag-of-words candidate retrieval and subsequent geometric consistency verification. The trained keyframe relationship discrimination model calculates the relationship confidence between the current keyframe and historical keyframes in the candidate keyframe set, and further filters the candidate keyframe set based on a relationship determination threshold. This reduces the number of historical keyframes entering the subsequent ORB feature matching and geometric consistency verification processes, minimizing unnecessary feature matching, geometric transformation estimation, and projection matching, and reducing the computational load of the candidate relationship verification process. Simultaneously, this invention employs a model-independent meta-learning method to train the keyframe relationship discrimination model offline. Multiple meta-learning tasks composed of training keyframe pairs are used to train shared model parameters, resulting in fixed model parameters. During online operation, these fixed model parameters remain unchanged; only forward model computation is performed, without updating the model parameters, thus avoiding the additional computational processes caused by online backpropagation and model parameter updates. Furthermore, this invention uses the relationship confidence score output by the keyframe relationship discrimination model for candidate keyframe screening. For the historical keyframes retained after screening, ORB feature matching and geometric consistency verification are further performed. Based on the geometric consistency verification results, loop closure constraints or multi-map fusion constraints are established. This reduces the number of candidate keyframes entering the subsequent geometric verification process while retaining the processing steps for loop closure confirmation and multi-map fusion confirmation based on geometric relationships. In summary, this invention can reduce unnecessary computations in the candidate keyframe relationship verification process, reduce the computational load of subsequent ORB feature matching and geometric consistency verification, and improve the processing efficiency of loop closure detection and multi-map fusion in visual synchronous localization and mapping. Attached Figure Description

[0008] Figure 1 This is a flowchart of the visual synchronous localization and mapping method based on offline meta-learning candidate keyframe screening in an embodiment of the present invention; Figure 2 This is a flowchart of the offline training process for the keyframe relationship discrimination model of the present invention; Figure 3 The diagram shows a comparison of the absolute position trajectories of the method of this invention and the ORB-SLAM3 method; where (a) is the position estimation result in the X direction, (b) is the position estimation result in the Y direction, and (c) is the position estimation result in the Z direction. Figure 4 The diagram shows a comparison of attitude estimation results between the method of this invention and the ORB-SLAM3 method; where (a) is the roll angle estimation result, (b) is the pitch angle estimation result, and (c) is the yaw angle estimation result. Detailed Implementation

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Example 1 This embodiment 1 describes a visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection. The method includes an offline training phase and an online execution phase. In the offline training phase, historical keyframes are obtained from historical execution results of multiple visual scenes or multiple visual data sequences. Based on these historical keyframes, training keyframe pair features and corresponding labels are constructed, and a model-independent meta-learning method is used to train the keyframe relationship discrimination model. In the online execution phase, candidate keyframes are first retrieved from the historical keyframes in the keyframe database using the bag-of-words method. Then, keyframe pair features between the current keyframe and candidate keyframes are constructed, and the confidence score of the relationship between them is calculated using the trained keyframe relationship discrimination model. Candidate keyframes are selected based on the relationship confidence score and a relationship determination threshold. Finally, ORB feature matching and geometric consistency verification are performed on the selected and retained confident candidate keyframes and the current keyframe. Loop closure constraints or multi-map fusion constraints are established based on the geometric consistency verification results. This invention reduces the number of historical keyframes that enter the subsequent ORB feature matching and geometric consistency verification process by adding a candidate keyframe screening process based on relation confidence between visual bag-of-words candidate retrieval and subsequent geometric consistency verification. This reduces unnecessary candidate relation verification calculations and improves the processing efficiency of loop closure detection and multi-map fusion.

[0010] like Figure 1 As shown, the visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection includes the following steps: Step 1. Obtain historical keyframes from the historical results of multiple visual scenes or multiple visual data sequences, preprocess the historical keyframes to obtain the features and corresponding labels of the training keyframe pairs.

[0011] Step 1.1. Obtain historical keyframes from the historical execution results of multiple visual scenes or multiple visual data sequences.

[0012] Multiple visual scenes or visual data sequences are processed using a visual simultaneous localization and mapping (SLAM) system, and historical keyframes generated by the SLAM system are obtained from the historical running results of each visual scene or visual data sequence.

[0013] The historical keyframes include at least a keyframe identifier, a collection timestamp, and ORB feature information for subsequent visual bag-of-words vector construction; for historical visual data with a reference trajectory, the reference position information of the corresponding historical keyframes can be obtained based on the collection timestamp.

[0014] Step 1.2. Preprocess the acquired historical keyframes.

[0015] I. Construct training keyframe pairs.

[0016] From the set of historical keyframes corresponding to each visual scene or each visual data sequence, any two historical keyframes are selected to form a training keyframe pair, the expression of which is: ; in Indicates the first A key historical frame, Indicates the first A key historical frame, Indicates training keyframe pairs Corresponding relationship tags; Indicates the historical keyframes Historical keyframes and relationship tags The training keyframe pairs constituted.

[0017] Construct multiple training keyframe pairs in each visual scene or each visual data sequence.

[0018] For training keyframe pairs According to historical keyframes and historical keyframes The acquisition timestamps are used to obtain historical keyframes from the reference trajectories corresponding to the historical visual data. Reference position coordinates and historical keyframes Reference position coordinates : ; ; in , , They are respectively The reference position is located on the x, y, and z axes. , , They are respectively The reference position is located on the x, y, and z axes.

[0019] Calculate historical keyframes With historical keyframes Spatial distance between : ; in The L2 norm of a vector is used to represent the vector's L2 norm.

[0020] Define spatial distance threshold In this embodiment, a spatial distance threshold is set. .

[0021] when ≤ At that time, the training keyframe pairs Set as a positive sample, and let ;when > At that time, the training keyframe pairs Set as a negative sample, and let .

[0022] Using the above method, relationship labels corresponding to training keyframe pairs are generated based on the spatial positional relationship of historical keyframes in the reference trajectory, providing training samples for the offline training of subsequent keyframe relationship discrimination models.

[0023] II. Construct visual bag-of-words vectors.

[0024] The ORB feature extraction method is used to extract feature points of historical keyframes and their corresponding ORB feature descriptors;

[0025] Then, based on the pre-defined visual dictionary, each ORB feature descriptor is quantized into a corresponding visual word, and the TF-IDF method is used to determine the weight of each visual word in the historical keyframes, forming the original visual bag-of-words vector: ; ;

[0026] in Representing historical keyframes The original visual bag-of-words vector; Representing historical keyframes The original visual bag-of-words vector.

[0027] Indicates the first A visual word in a historical keyframe The weights in Indicates the first A visual word in a historical keyframe The weights in ;

[0028] Indicates the number of visual words included in the preset visual dictionary; superscript This represents the transpose of a vector. This embodiment uses a scale of... ORB visual dictionary.

[0029] ; ; in Indicates the first in the preset visual dictionary Inverse document frequency weights corresponding to each visual word; This indicates the history keyframes , No. The word frequency of visual words; This indicates the history keyframes , No. The word frequency of visual words; ; ; in Indicates the first A visual word in a historical keyframe The number of times it appears in Representing historical keyframes The total number of occurrences of all visual words in the text; Indicates the first A visual word in a historical keyframe The number of times it appears in Representing historical keyframes The total number of occurrences of all visual words in the text; Historical keyframes and historical keyframes Normalizing the original visual bag-of-words vectors using the 2-norm yields: ; ; in Representing historical keyframes Normalized visual bag-of-words vectors, Representing historical keyframes The normalized visual bag-of-words vector. The normalized visual bag-of-words vector satisfies... , .

[0030] The same preset visual dictionary, visual word quantization method, TF-IDF weight calculation method, and visual bag-of-words vector normalization method are used in both the offline training phase and the online running phase.

[0031] The visual bag-of-words vectors of historical keyframes mentioned below refer to the visual bag-of-words vectors after the above-mentioned L2 norm normalization process.

[0032] The same preset visual dictionary, visual word quantization method, visual word weight calculation method, and visual bag-of-words vector normalization method are used in both the offline training phase and the online running phase.

[0033] III. Construct training keyframe pair features.

[0034] For training keyframe pairs According to historical keyframes Visual bag-of-words vectors in front, historical keyframes The order of the visual bag-of-words vectors is then used to concatenate the normalized visual bag-of-words vectors to obtain the training keyframe pair features: ; in Indicates training keyframe pairs The corresponding training keyframe pairs feature.

[0035] When the dimension of the bag-of-words vector is The training keyframes pair features The dimension is .

[0036] In the online processing stage, following the same concatenation order as the training stage, the visual bag-of-words vectors of the current keyframe are arranged first, followed by the visual bag-of-words vectors of the candidate keyframes, in order to form keyframe pair features and input them into the keyframe relationship discrimination model.

[0037] Step 2. Build a keyframe relationship discrimination model and use the model-independent meta-learning method to train it offline based on the features and corresponding relationship labels of the training keyframes; the keyframe relationship discrimination model is used to process the features of the training keyframes obtained in Step 1 and output the confidence of the relationship between the training keyframes.

[0038] The keyframe relationship discrimination model consists of an input layer, a hidden layer, and an output layer.

[0039] The input layer is used to receive the training keyframe pairs of features obtained in step 1. The training keyframe pairs feature the data from historical keyframes. Normalized visual bag-of-words vectors and historical keyframes Normalized visual bag-of-words vectors The components are assembled in the order described in step 1.2, section III, i.e. .

[0040] in, , , , Represents the real number field.

[0041] The hidden layer is used to perform feature transformation on the features of the training keyframes. The calculation process of the hidden layer is as follows: ; in This represents the feature vector output by the hidden layer. This represents the weight matrix of the hidden layer. This represents the bias vector of the hidden layer; in this embodiment, the dimension of the hidden layer is set to 32, therefore... , , .

[0042] Represents a linear rectified function used to transform vectors. Each element in the process is subjected to nonlinear processing; For any element of the input linear rectifier function , ; when hour, Output ;when hour, Output 0.

[0043] The output layer is used to calculate the discrimination values ​​corresponding to the training keyframe pairs based on the feature vectors output by the hidden layer. The specific calculation formula is as follows: ; in This represents the discriminant value calculated by the output layer. This represents the weight matrix of the output layer. This indicates the bias of the output layer; in this embodiment, the hidden layer dimension is set to 32, therefore... , .

[0044] The discriminant value is determined using the Sigmoid function. Converting to a confidence level between 0 and 1, we get: ; in This indicates that the keyframe relationship discrimination model is trained on keyframe pairs. The output relation confidence score represents the training keyframe pair. The degree to which it belongs to a positive label relationship. The larger the keyframe relationship discrimination model is, the better it judges the training keyframe pairs. The higher the probability of a positive label relationship.

[0045] During the offline training phase, the relationship confidence level... Relation labels used to combine training keyframe pairs Calculate the classification loss and use it to train the keyframe relationship discrimination model. After offline training, use the fixed model parameters obtained from the training to calculate the relationship confidence during the online run phase.

[0046] In this embodiment, the Model-Independent Meta-Learning (MAML) method is used to train the keyframe relationship discrimination model offline.

[0047] Each visual scene or visual data sequence corresponds to a pair of training keyframes, which are then used to construct a meta-learning task. The training keyframes in each meta-learning task are divided into a support set and a query set.

[0048] Definition of the first The individual learning task is: ; in Indicates the first Individual learning tasks; Indicates the first Support sets for individual meta-learning tasks, used to update model parameters within a single meta-learning task; Indicates the first The query set of each meta-learning task is used to evaluate the updated model within the task and update the model parameters shared by multiple meta-learning tasks accordingly.

[0049] The keyframe relationship discrimination model is trained offline based on meta-learning tasks corresponding to multiple visual scenes or multiple visual data sequences. The offline training process is as follows: Figure 2 As shown, the trained keyframe relationship discrimination model is used to calculate the confidence level of the relationship between the current keyframe and candidate keyframes during the online operation of visual synchronous localization and mapping. This confidence level is then used for subsequent candidate keyframe selection.

[0050] To evaluate the output of the keyframe relationship discrimination model based on the labels of the trained keyframe pairs, and to provide an optimization basis for updating intra-task parameters on the support set and updating shared model parameters on the query set, a binary cross-entropy loss function is used to calculate the classification loss.

[0051] During model training, for any set of samples Its classification loss is: ; in This indicates that the keyframe relationship discrimination model is in the sample set. Classification loss on Represents the sample set The number of training keyframe pairs; and These represent the features and their relational labels for the training keyframe pairs obtained in step 1.

[0052] The model parameters are: At that time, the keyframe relationship discrimination model targets the features of the keyframes during training. Output relation confidence; This represents the model parameters shared across multiple meta-learning tasks, including the hidden layer weight matrix. Hidden layer bias vector Output layer weight matrix and output layer bias .

[0053] For the Individual meta-learning tasks, using model parameters shared across multiple meta-learning tasks. As initial task parameters ,Right now .

[0054] According to the support set The classification loss is used to update the in-task parameters of the task. The update within this task is as follows: ; in Indicates the first The first generation of the meta-learning task was executed. This task updates the parameters of the previous task. Indicates the first Individual learning execution Updated task parameters within this task; Indicates the number of updates within the task. Indicates the learning rate within the task; ) indicates the use of parameters Time model in support set Classification loss on The support set classification loss is expressed with respect to the parameters. The gradient.

[0055] Complete the preset number of times After the task update, the first... Task adaptation parameters for individual learning tasks .

[0056] Using task adaptation parameters Calculate the corresponding query set The loss is calculated, and the shared model parameters are updated based on the classification loss of the query set of multiple meta-learning tasks selected in the first-order training iteration: ; in Indicates the use of task adaptation parameters The time model in the query set Classification loss on This indicates that the classification loss of the query set is related to the shared model parameters. The gradient; Indicates the learning rate for shared model parameters. This indicates the number of meta-learning tasks selected in a single-dimensional training iteration.

[0057] Different visual scenes or visual data sequences constitute different meta-learning tasks, so that the shared model parameters are simultaneously constrained by the query set loss corresponding to multiple meta-learning tasks during the training process. This helps to reduce the dependence of the keyframe relationship discrimination model on the appearance features of a single visual scene and improve the model's adaptability and generalization ability to different visual scenes.

[0058] In this embodiment, the in-task learning rate Each learning task execution Intra-task parameter updates; shared model parameter learning rate Each meta-training iteration selects Each learning task is performed, with a total of 500 training iterations.

[0059] In this embodiment, the support set and query set for each meta-learning task each include 16 training keyframe pairs. Among them, the relation labels... The training keyframe pairs are positive samples, and the relationship labels are... The training keyframe pairs are negative samples; the ratio of positive samples to negative samples in both the support set and the query set is 1:1, that is, both the support set and the query set include 8 positive samples and 8 negative samples respectively.

[0060] The trained shared model parameters are used as the fixed model parameters for the keyframe relationship discrimination model: ; in This represents the fixed model parameters obtained after completing meta-learning training. This represents the hidden layer weight matrix after training. This represents the hidden layer bias vector after training. This represents the output layer weight matrix after training. This indicates the output layer bias after training is complete.

[0061] The fixed model parameters The keyframe relationship discrimination model is loaded into the system, resulting in a trained keyframe relationship discrimination model. During the online operation of the visual synchronous localization and mapping system, the fixed model parameters are... The model remains unchanged; only the forward computation of the keyframe relationship discrimination model is performed, and the model parameters are not updated.

[0062] Step 3. Acquire continuously captured image frames from the camera, and when the current image frame meets the keyframe insertion conditions, determine it as the current keyframe.

[0063] Construct visual bag-of-words vectors for the current keyframe and historical keyframes in the keyframe database, calculate the visual similarity between the current keyframe and each historical keyframe, and select historical keyframes from high to low visual similarity to form a candidate keyframe set corresponding to the current keyframe.

[0064] Then, keyframe pair features are constructed based on the normalized visual bag-of-words vectors of the current keyframe and candidate keyframes, and the keyframe pair features are input into the keyframe relationship discrimination model trained in step 2 to obtain the confidence of the relationship between the current keyframe and candidate keyframes.

[0065] Step 3.1. Acquire continuously captured image frames from the camera, and when the current image frame meets the keyframe insertion conditions, determine it as the current keyframe.

[0066] The keyframe insertion condition is as follows: ; in This represents the number of intra-map points retained after the current image frame is matched with the local map and optimized for pose. This indicates the number of map points already tracked in the reference keyframe. The reference keyframe refers to the established keyframe used for local map tracking during the current image frame tracking process.

[0067] Step 3.2. Calculate the visual similarity between the current keyframe and historical keyframes, and determine the set of candidate keyframes corresponding to the current keyframe.

[0068] Define the current keyframe as Any historical keyframe in the keyframe database is .

[0069] First, the current keyframe is constructed using the same preset visual dictionary, visual word quantization method, and L2 normalization method as in step 1. Normalized visual bag-of-words vectors and historical keyframes in the keyframe database Normalized visual bag-of-words vectors ;and , .

[0070] Then, the visual similarity between the current keyframe and each historical keyframe in the keyframe database is calculated using cosine similarity. The specific calculation formula is as follows: ; in Indicates the current keyframe With historical keyframes Visual similarity between them Represents the visual bag-of-words vector The transpose of .

[0071] Historical keyframes were sorted from highest to lowest visual similarity, and those at the top of the list were selected. Use one historical keyframe as the current keyframe. The corresponding candidate keyframes form a candidate keyframe set. ,Right now: .

[0072] When the number of historical keyframes in the keyframe database is less than In this case, all historical keyframes are selected as candidate keyframes. .

[0073] Step 3.3. Construct keyframe pair features based on the normalized visual bag-of-words vectors of the current keyframe and candidate keyframes, and calculate the confidence of the relationship between them using the trained keyframe relationship discrimination model.

[0074] For the selected candidate keyframe set Historical keyframes Hereinafter referred to as candidate keyframes ; For the candidate keyframe set formed in step 3.2 Any candidate keyframe Following the same vector concatenation order as in step 1, i.e., the visual bag-of-words vector of the current keyframe first, followed by the visual bag-of-words vector of the candidate keyframes, the current keyframe is concatenated... Normalized visual bag-of-words vectors With the candidate keyframe Normalized visual bag-of-words vectors Perform splicing to construct keyframe pairs of features .

[0075] in Indicates the current keyframe With historical keyframes Corresponding keyframe features; and These represent the current keyframes. and candidate keyframes The normalized visual bag-of-words vectors are constructed using the same pre-defined visual dictionary, visual word quantization method, visual word weight calculation method, and L2 normalization method, and have the same vector dimension.

[0076] Key features Input the keyframe relationship discrimination model trained in step 2, and obtain: ; in This represents the current keyframe output by the keyframe relationship discrimination model. With candidate keyframes Confidence of the relationship The model parameters are: Keyframe relationship discrimination model; This indicates that the fixed model parameters obtained after the meta-learning training in step 2 remain unchanged during the online operation of the visual synchronous localization and mapping system.

[0077] The confidence level of the relationship It is used for subsequent filtering of historical keyframes in the candidate keyframe set, and is not directly used as the confirmation result of loop closure constraints or multi-map fusion constraints.

[0078] Step 4. Based on the confidence level, filter multiple candidate keyframes and retain candidate keyframes with a confidence level not lower than the preset relationship determination threshold to form a set of confidence candidate keyframes.

[0079] Define the relationship determination threshold as In this embodiment .

[0080] For the candidate keyframe set Any candidate keyframe The relationship confidence obtained in step 3.3 Relationship determination threshold Compare them.

[0081] When the relation determination condition At that time, retain the corresponding candidate keyframes. Otherwise, discard the corresponding candidate keyframe.

[0082] All candidate keyframes that satisfy the aforementioned relationship determination conditions are combined into a confidence candidate keyframe set. , .

[0083] When candidate keyframe set If no candidate keyframes satisfy the relationship determination conditions are found, subsequent ORB feature matching and geometric consistency verification will not be performed, and the visual synchronous localization and mapping system will continue to perform normal tracking and mapping processing.

[0084] Step 5. Perform ORB feature matching and geometric consistency verification on the current keyframe and the candidate keyframes in the confidence candidate keyframe set; for keyframes that pass the geometric consistency verification, establish loop closure constraints and perform loop closure correction when the two belong to the same map, and establish multi-map fusion constraints and perform multi-map fusion when the two belong to different maps; update the keyframe pose and map point coordinates to obtain the optimized camera pose estimation results and map.

[0085] Step 5.1. Perform ORB feature matching.

[0086] Define any candidate keyframe in the set of confidence candidate keyframes as a confidence candidate keyframe. .

[0087] First, extract the current keyframe. and confidence candidate keyframes ORB feature points and their corresponding ORB feature descriptors; Calculate the current keyframe Any ORB feature descriptor and confidence candidate keyframe The Hamming distance between each ORB feature descriptor is used to obtain the minimum Hamming distance. Distance between the second and third smallest Hamming Then calculate the nearest neighbor distance ratio. .

[0088] when Not greater than the preset nearest neighbor distance ratio threshold At that time, the minimum Hamming distance The two corresponding ORB feature points are retained as candidate matching feature point pairs; otherwise, the matching feature point pairs are discarded. In this embodiment... .

[0089] Then, based on the ORB feature directions of the two ORB feature points in the candidate matching feature point pair, direction consistency screening is performed to eliminate candidate matching feature point pairs with inconsistent direction relationships, resulting in the initial matching feature point pairs. The direction consistency screening is based on the difference in ORB feature directions between a pair of matching ORB feature points to determine whether their direction relationships are consistent.

[0090] When the number of initial matched feature point pairs is not less than the preset threshold for the number of matched feature point pairs. If the condition is met, continue with geometric transformation estimation (i.e., step 5.2); otherwise, determine the current keyframe. With confidence candidate keyframes Geometric consistency verification failed. In this embodiment... .

[0091] Step 5.2. Perform initial geometric transformation estimation.

[0092] Based on the initial matching feature point pairs obtained in step 5.1, the map points associated with each matching feature point are obtained from the visual synchronous localization and mapping system, and the initial matching feature point pairs where both matching feature points are associated with map points are retained; the two map points associated with each pair of matching feature points are combined into a matching map point pair (the system will save this during the mapping process).

[0093] The map point refers to a three-dimensional point established and saved by the visual synchronous positioning and mapping system during the tracking and mapping process, which is used to represent the three-dimensional spatial position of the scene.

[0094] Obtain the 3D coordinates of the matched map points and the current keyframe. and confidence candidate keyframes The pose already determined in the visual synchronous localization and mapping system; Based on the pose of the corresponding keyframe, perform a conventional coordinate transformation to convert the two map points in each matched map point pair to confidence candidate keyframes. and the current keyframe From the camera coordinate system, obtain the corresponding three-dimensional coordinates. and ; in, This indicates that the corresponding map point in the matched map point pair is in the confidence candidate keyframe. 3D coordinates in the camera coordinate system This indicates the map point that matches it in the current keyframe. Three-dimensional coordinates in the camera coordinate system.

[0095] Then, based on the Sim(3) solution method with random sampling consistency, according to... and Estimating confidence candidate keyframes Up to the current keyframe The initial geometric transformation.

[0096] Specifically, a matching map point pair is randomly selected, and the Sim(3) geometric transformation is calculated based on the selected matching map point pair. The Sim(3) geometric transformation obtained from each random selection is recorded as a candidate geometric transformation.

[0097] Based on the candidate geometric transformation, the matching map points are projected onto the image plane of the corresponding keyframe. The distance between the projected position of the map point and the position of the corresponding matching ORB feature point in the image is calculated, and the distance is used as the reprojection error.

[0098] When the reprojection error is not greater than a preset reprojection error threshold When the corresponding matching map point pair is determined to have geometric consistency with the candidate geometric transformation, the matching map point pair is identified as a matching interior point, and the number of matching interior points is counted.

[0099] In this embodiment A pixel is defined as a matching map point pair when the distance between the projection position of a map point on the corresponding keyframe image plane and the position of the corresponding matching ORB feature point in the image is no more than 3 pixels.

[0100] Repeatedly perform random selection of matching map point pairs, calculation of candidate geometric transformations, and determination of matching interior points, and retain the candidate geometric transformation with the most matching interior points.

[0101] Define the preset threshold for the number of matched intrapoints as follows: .

[0102] When the maximum number of matched interior points obtained within the preset maximum number of iterations is not less than When the corresponding candidate geometric transformation is determined, it is identified as a confidence candidate keyframe. Up to the current keyframe Initial geometric transformation ; in Indicates the scale factor. This represents the rotation matrix estimated from the matching map point pairs. This represents the translation vector estimated from the matched map point pairs.

[0103] The initial geometric transformation represents a confidence candidate keyframe. Camera coordinate system to current keyframe The spatial transformation relationship of the camera coordinate system satisfies: .

[0104] When the maximum number of matched interior points obtained within the preset maximum number of iterations is less than At that time, determine the current keyframe With confidence candidate keyframes The geometric consistency verification at this stage has not been passed.

[0105] In this embodiment, the confidence level of the random sampling consistency method is set to 0.99, the maximum number of iterations is set to 300, and the threshold for the number of matched interior points is set. .

[0106] Step 5.3. Perform coarse projection matching.

[0107] Obtain and believable candidate keyframes from the keyframe co-view relationships established by the visual synchronous localization and mapping system. Locally shared keyframes with shared viewing relationships are identified, and confidence candidate keyframes are selected. and the map points associated with its local co-view keyframes;

[0108] In this context, a co-view relationship is formed when two keyframes jointly observe the same map point. The number of jointly observed map points is used to characterize the strength of the co-view relationship. The local co-view keyframe is a confidence candidate keyframe. Several keyframes with a large number of map points are observed together.

[0109] The initial geometric transformation estimated in step 5.2 Project the map points onto the current keyframe. The image plane.

[0110] For any projected map point, the predicted ORB scale layer corresponding to that map point is used as a reference. Centered on its projection position, with To determine the search radius, a projection matching method based on visual synchronous localization and mapping is used, based on the current keyframe extracted in step 5.1. Candidate ORB feature points are determined from the ORB feature points; in , Indicates the first Scale factor corresponding to each ORB scale layer; predict ORB scale layer It refers to the ORB image pyramid hierarchy determined by the visual synchronous localization and mapping system based on the distance between the projected map point and the current keyframe camera, as well as the existing scale information of the map point.

[0111] In this embodiment, the scale layer is selected from the ORB feature points located within the search range as the ORB prediction scale layer. or the ORB feature points from each ORB scale layer are used as candidate ORB feature points.

[0112] Calculate the Hamming distance between the ORB feature descriptor of the projected map point and the ORB feature descriptor of each candidate ORB feature point within the search range, and select the ORB feature with the smallest Hamming distance as the candidate match.

[0113] For ORB features within the projection search range, calculate the Hamming distance between the ORB feature descriptor corresponding to the projected map point and the ORB feature descriptor of the ORB feature. ; in This represents the ORB feature descriptor corresponding to the projected map point. This represents the ORB feature descriptor obtained from the search. The number of binary bits representing the ORB feature descriptor; Indicates an indicator function; when the first of two descriptors... When the binary bits are different Otherwise, it is 0.

[0114] When the minimum Hamming distance is not greater than the preset coarse projection distance threshold At this time, candidate matches are retained to obtain coarse projection matching results. In this embodiment... .

[0115] The number of candidate matches obtained from coarse projection matching is counted. When the number of coarse projection matches is not less than a preset threshold for the number of coarse projection matches... If the condition is met, continue with geometric transformation optimization (i.e., step 5.4); otherwise, determine the current keyframe. With confidence candidate keyframes Geometric consistency verification failed. In this embodiment... .

[0116] Step 5.4. Perform geometric transformation optimization.

[0117] Using the coarse projection matching result obtained in step 5.3 as the observation, and the reprojection error of the matching points as the optimization basis, the initial geometric transformation obtained in step 5.2 is applied. Sim(3) optimization is performed to obtain the optimized geometric transformation. .

[0118] The number of matching interior points retained after Sim(3) optimization is counted. When the number of matching interior points is not less than a preset threshold for the number of interior points... If the condition is met, perform fine projection matching (i.e., step 5.5); otherwise, determine the current keyframe. With confidence candidate keyframes Geometric consistency verification failed. In this embodiment... .

[0119] Step 5.5. Perform fine projection matching.

[0120] Based on the optimized geometric transformation in step 5.4 The confidence candidate keyframes selected in step 5.3 are then re-selected. The map points associated with the local co-view keyframe are projected onto the current keyframe. The image plane.

[0121] For any projected map point, the predicted ORB scale layer corresponding to that map point is used as a reference. Centered on its projection position, with The search radius is defined in the current keyframe. Re-search ORB features; where , It is a constant, in this embodiment .

[0122] Calculate the Hamming distance between the ORB feature descriptor of the projected map point and the ORB feature descriptor of each ORB feature point within the search range, and determine the ORB feature point corresponding to the minimum Hamming distance.

[0123] When the minimum Hamming distance is not greater than the preset fine projection distance threshold At that time, the corresponding matches are retained to obtain fine projection matching. In this embodiment... .

[0124] The number of fine projection matches is counted, and when the number of fine projection matches is not less than the preset threshold for the number of fine projection matches... At that time, determine the current keyframe With confidence candidate keyframes Complete the current sub-geometric consistency verification; otherwise, determine that neither has passed the current sub-geometric consistency verification. In this embodiment... .

[0125] Step 5.6. Perform geometric consistency verification.

[0126] Obtain the current keyframe from the shared-view relationships of keyframes already established by the visual synchronous localization and mapping system. Several shared keyframes, and according to the current keyframe The number of jointly observed map points is verified by projection in sequence.

[0127] Specifically, for any of the aforementioned shared keyframes Based on the optimized geometric transformation obtained in step 5.4 Combined with the current keyframe With the co-view keyframe Based on the existing relative poses, determine the confidence candidate keyframes. With the co-view keyframe Geometric transformations between the two are performed, and projection verification is performed based on the geometric transformations.

[0128] The projection verification refers to the process of verifying confidence candidate keyframes. The corresponding map points are projected onto the shared keyframe. The image plane is used for ORB feature matching.

[0129] When the number of matches obtained in a single projection verification is not less than the preset verification match number threshold When this occurs, it is recorded as a successful geometric consistency verification. In this embodiment... .

[0130] For the same confidence candidate keyframe Sequentially process the current keyframe The projection verification is performed on different co-view keyframes and the number of successful geometric consistency verifications is accumulated.

[0131] When the number of successful verifications is not less than a preset geometric consistency verification threshold At that time, confirm the current keyframe. With confidence candidate keyframes Geometric consistency verification was performed. (This embodiment...) .

[0132] Step 5.7. For the confidence candidate keyframes that have passed the geometric consistency verification, based on the current keyframe... With confidence candidate keyframes To determine if the maps belong to the same location, establish loop closure constraints or multi-map fusion constraints, then update the keyframe pose and map point coordinates to obtain the optimized camera pose estimation results and map.

[0133] For a confidence candidate keyframe that fails the geometric consistency verification, no loop closure constraints or multi-map fusion constraints are established based on the confidence candidate keyframe. The visual synchronous localization and mapping process continues to track and map subsequent input image frames.

[0134] For a confidence candidate keyframe that passes the geometric consistency verification, based on the current keyframe With confidence candidate keyframes If the maps belong to the same map, perform loop closure correction or multi-map fusion.

[0135] I. When the current keyframe With confidence candidate keyframes When they belong to the same map, the optimized geometric transformation obtained in step 5.4 is applied. Determine the current keyframe With confidence candidate keyframes The relative pose constraints between them are established, and closure constraints are created.

[0136] Loop closure constraints are new constraints added to the same map to constrain the current keyframe. With confidence candidate keyframes Constraints on relative pose relationships.

[0137] Correct the current keyframe based on the closure constraint. The pose is determined, and the pose correction is propagated to the co-view keyframe of the current keyframe while maintaining the original relative pose relationship between the current keyframe and its co-view keyframe.

[0138] Set the current keyframe The pose-corrected keyframes and their co-view keyframes after pose correction propagation are collectively referred to as pose-corrected keyframes. Any pose-corrected keyframe is denoted as... .

[0139] Based on keyframes The pose changes before and after correction are then corrected, and the coordinates of the associated map points are adjusted accordingly. The corrected coordinates of the map points are as follows: ; in This represents the coordinates of a map point in the world coordinate system after calibration. This indicates the coordinates of a map point in the world coordinate system before correction. and These represent keyframes. World coordinate system to keyframe before and after correction Geometric transformation of the camera coordinate system; Keyframe Geometric transformation from camera coordinate system to world coordinate system.

[0140] The map points observed repeatedly on both sides of the loop are fused together, and the current keyframe is then used. With confidence candidate keyframes The lapsing constraints between keyframes are added to the keyframe pose graph; the keyframe pose graph has keyframes as nodes and relative pose constraints between keyframes as edges.

[0141] Then, the pose graph optimization method of the visual synchronous localization and mapping system is adopted. Based on the newly added closure constraints and the relative pose constraints between the original keyframes, the pose of the keyframes is optimized as a whole, and the coordinates of the corresponding map points are updated according to the optimized keyframe poses.

[0142] II. When the current keyframe With confidence candidate keyframes When the maps belong to different maps, apply the optimized geometric transformation obtained in step 5.4. Establish multi-map fusion constraints as an alignment relationship between two maps.

[0143] Multi-map fusion constraints refer to the constraints used to integrate the current keyframes. Map and Confidence Candidate Keyframes Geometric constraints for transforming the map to the same coordinate system.

[0144] Based on the multi-map fusion constraints, the current keyframe The keyframe poses in the map are transformed by coordinates, and the coordinate transformation is propagated to the corresponding map points, so that the keyframes and map points in the two maps are in the same coordinate system. For two maps after coordinate transformation, duplicate map points representing the same scene features are merged, and keyframes and map points in the two maps are combined.

[0145] Then, the map optimization method of the visual synchronous localization and mapping system is adopted to optimize the keyframe pose and map point coordinates after fusion, so as to obtain the unified map after fusion.

[0146] After the above loop closure correction or multi-map fusion, the corresponding keyframe pose and map point coordinates are updated to obtain the optimized camera pose estimation results and map.

[0147] The method of this invention maintains fixed model parameters of the keyframe relationship discrimination model during the online operation phase, and uses the relationship confidence score output by the model to filter candidate keyframes retrieved by the bag-of-words method, thereby reducing the number of historical keyframes entering the subsequent ORB feature matching and geometric consistency verification processes. For the historical keyframes retained after filtering, ORB feature matching and geometric consistency verification are performed according to step 5, and loop closure constraints or multi-map fusion constraints are established based on the verification results. Therefore, the keyframe relationship discrimination model is used for calculating and filtering the relationship confidence scores of candidate keyframes, and step 5 is used for the final confirmation of the geometric relationships between candidate keyframes and subsequent loop closure correction or multi-map fusion. Together, they constitute the online candidate keyframe processing flow of this invention.

[0148] To verify the actual performance of the method of this invention, this embodiment uses ORB-SLAM3 as the specific implementation platform to implement and verify the visual synchronous localization and mapping method based on meta-learning candidate keyframe selection described in this invention. During online operation, the keyframe relationship discrimination model trained offline is used to calculate and filter the relationship confidence of candidate keyframes. The remaining candidate keyframes after filtering are further subjected to ORB feature matching, geometric consistency verification, and subsequent loop closure correction or multi-map fusion. The fixed model parameters of the keyframe relationship discrimination model remain unchanged during online operation.

[0149] This embodiment uses the original ORB-SLAM3 system, which does not employ the candidate keyframe relationship confidence screening method described in this invention, as a comparison method, and conducts experiments on BONN RGB-D test data. Absolute trajectory error is used as the evaluation index for trajectory estimation results, and the maximum error, average error, median error, root mean square error, and standard deviation of error are statistically analyzed. The experimental results are shown in Table 1.

[0150] Table 1. Comparison of Absolute Trajectory Error (ATE) between the Invention Method and the Comparative Method on BONN RGB-D Test Data

[0151] As shown in Table 1, compared with the ORB-SLAM3 method, the maximum error of the method of the present invention is reduced from 1.933 meters to 1.313 meters, a reduction of approximately 32.06%; the root mean square error is reduced from 0.688 meters to 0.592 meters, a reduction of approximately 13.91%; and the standard deviation of the error is reduced from 0.500 meters to 0.324 meters, a reduction of approximately 35.20%. These results demonstrate that under the test conditions of this embodiment, the method of the present invention can reduce large deviations in trajectory estimation and decrease the overall fluctuation of the error.

[0152] To further demonstrate the trajectory estimation results of different methods, Figure 3 The estimation results of the method of this invention and the ORB-SLAM3 method in three positional directions are presented. Figure 4 Attitude estimation results for both trajectories in the roll, pitch, and yaw directions are presented, with the actual trajectories used as a comparison reference. Figure 3 , Figure 4 As shown in Table 1, the method of the present invention can complete camera trajectory and attitude estimation, and reduce large trajectory estimation deviations and error fluctuations.

[0153] In summary, under the test data and experimental conditions used in this embodiment, the maximum absolute trajectory error, root mean square error, and standard deviation of the error of the method of the present invention are all lower than those of the ORB-SLAM3 method. The method of the present invention can reduce large trajectory estimation bias and reduce error fluctuation.

[0154] Example 2 This embodiment 2 describes a visual synchronous localization and mapping system based on meta-learning candidate keyframe screening. This system is based on the same inventive concept as the visual synchronous localization and mapping method based on meta-learning candidate keyframe screening in embodiment 1 above.

[0155] The visual simultaneous localization and mapping system based on meta-learning candidate keyframe selection includes the following modules: The preprocessing module is used to obtain historical keyframes from the historical running results of multiple visual scenes or multiple visual data sequences, preprocess the historical keyframes, and obtain the features and corresponding labels of the training keyframe pairs. The model training module is used to build a keyframe relationship discrimination model and uses a model-independent meta-learning method to train offline based on the features of the training keyframe pairs and their corresponding relationship labels. The keyframe relationship discrimination model is used to process the features of the training keyframe pairs and output the relationship confidence of the corresponding training keyframe pairs. The candidate retrieval and relationship confidence calculation module is used to acquire continuously captured image frames from the camera and determine the current keyframe based on the keyframe insertion criteria. Calculate the visual similarity between the current keyframe and each historical keyframe, and construct a set of candidate keyframes corresponding to the current keyframe; Construct keyframe pair features between the current keyframe and candidate keyframes, and input them into the trained keyframe relationship discrimination model to obtain the confidence of the relationship between the current keyframe and candidate keyframes; The candidate keyframe filtering module is used to filter candidate keyframes according to the relationship confidence level, and retain candidate keyframes with a relationship confidence level not lower than a preset relationship judgment threshold to form a set of confidence candidate keyframes. The geometric consistency verification, loop closure correction, and multi-map fusion module is used to perform ORB feature matching and geometric consistency verification on the current keyframe and candidate keyframes in the set of confidence candidate keyframes, and to establish loop closure constraints or multi-map fusion constraints; then, the keyframe pose and map point coordinates are updated to obtain the optimized camera pose estimation results and map.

[0156] It should be noted that any content not mentioned in the above-described functional modules of the system described in Embodiment 2 can be referred to the step description of the corresponding method in Embodiment 1 above, and will not be repeated in detail here.

[0157] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection, characterized in that, Includes the following steps: Step 1. Obtain historical keyframes from the historical results of multiple visual scenes or multiple visual data sequences, preprocess the historical keyframes to obtain the features and corresponding labels of the training keyframe pairs; Step 2. Build a keyframe relationship discrimination model and use the model-independent meta-learning method to train the features and corresponding relationship labels offline based on the training keyframes; The keyframe relationship discrimination model is used to process the features of the training keyframe pairs and output the relationship confidence of the corresponding training keyframe pairs. Step 3. Acquire continuously captured image frames from the camera, and determine the current keyframe based on the keyframe insertion criteria; Calculate the visual similarity between the current keyframe and each historical keyframe, and construct a set of candidate keyframes corresponding to the current keyframe; Construct keyframe pair features between the current keyframe and candidate keyframes, and input them into the trained keyframe relationship discrimination model to obtain the confidence of the relationship between the current keyframe and candidate keyframes; Step 4. Filter candidate keyframes according to the relationship confidence level, and retain candidate keyframes with a relationship confidence level not lower than the preset relationship determination threshold to form a set of confidence candidate keyframes; Step 5. Perform ORB feature matching and geometric consistency verification on the current keyframe and the candidate keyframes in the set of confidence candidate keyframes, and establish loop closure constraints or multi-map fusion constraints. Then, the keyframe pose and map point coordinates are updated to obtain the optimized camera pose estimation results and map.

2. The visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection according to claim 1, characterized in that, In step 1, the preprocessing includes: I. Construct training keyframe pairs; A training keyframe pair is formed by any two historical keyframes corresponding to each visual scene or each visual data sequence, and its expression is: ; in Indicates the first A key historical frame, Indicates the first A key historical frame, Indicates training keyframe pairs Corresponding relationship tags; Indicates the historical keyframes Historical keyframes and relationship tags The training keyframe pairs constituted; For training keyframe pairs According to historical keyframes and historical keyframes The acquisition timestamps are used to obtain historical keyframes from the reference trajectories corresponding to the historical visual data. Reference position coordinates and historical keyframes Reference position coordinates : ； ； in , , They are respectively The reference position is located on the x, y, and z axes. , , They are respectively The reference position is located on the x, y, and z axes. Calculate historical keyframes With historical keyframes Spatial distance between : ； in The second norm of a vector; Define spatial distance threshold ; when ≤ At that time, the training keyframe pairs Set as a positive sample, and let ; when > At that time, the training keyframe pairs Set as a negative sample, and let ; II. Construct visual bag-of-words vectors; The ORB feature extraction method is used to extract feature points of historical keyframes and their corresponding ORB feature descriptors; Then, based on the pre-defined visual dictionary, each ORB feature descriptor is quantized into a corresponding visual word, and the TF-IDF method is used to determine the weight of each visual word in the historical keyframes, forming the original visual bag-of-words vector: ； ； in Representing historical keyframes The original visual bag-of-words vector; Representing historical keyframes The original visual bag-of-words vector; Indicates the first A visual word in a historical keyframe The weights in Indicates the first A visual word in a historical keyframe The weights in ; Indicates the number of visual words included in the preset visual dictionary; superscript Indicates vector transpose; Historical keyframes and historical keyframes Normalizing the original visual bag-of-words vectors using the 2-norm yields: ； ； in Representing historical keyframes Normalized visual bag-of-words vectors, Representing historical keyframes Normalized visual bag-of-words vectors; The normalized visual bag-of-words vector satisfies , ; The visual bag-of-words vectors of historical keyframes mentioned below all refer to the visual bag-of-words vectors after the above-mentioned L2 normalization process. III. Construct training keyframe pair features; For training keyframe pairs According to historical keyframes Visual bag-of-words vectors in front, historical keyframes The order of the visual bag-of-words vectors is then used to concatenate the normalized visual bag-of-words vectors to obtain the training keyframe pair features: ; in Indicates training keyframe pairs The corresponding training keyframe pairs feature.

3. The visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection according to claim 2, characterized in that, In step 2, the keyframe relationship discrimination model includes an input layer, a hidden layer, and an output layer; The input layer is used to receive the training keyframe pairs of features obtained in step 1. ; The hidden layer is used to perform feature transformation on the features of the training keyframes. The calculation process of the hidden layer is as follows: ； in This represents the feature vector output by the hidden layer. This represents the weight matrix of the hidden layer. This represents the bias vector of the hidden layer; Represents a linear rectified function used to transform vectors. Each element in the process is subjected to nonlinear processing; For any element of the input linear rectifier function , ; The output layer is used to calculate the discrimination values ​​corresponding to the training keyframe pairs based on the feature vectors output by the hidden layer. The specific calculation formula is as follows: ； in This represents the discriminant value calculated by the output layer. This represents the weight matrix of the output layer. Indicates the bias of the output layer; The discriminant value is determined using the Sigmoid function. Converting to a confidence level between 0 and 1, we get: ； in This indicates that the keyframe relationship discrimination model is trained on keyframe pairs. The confidence level of the output relationship.

4. The visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection according to claim 3, characterized in that, In step 2, the keyframe relationship discrimination model is trained using a model-independent meta-learning method. Each visual scene or visual data sequence corresponds to a training keyframe pair to construct a meta-learning task, and the training keyframe pair in each meta-learning task is divided into a support set and a query set. Definition of the first The individual learning task is: ; in Indicates the first Individual learning tasks; Indicates the first Support sets for individual meta-learning tasks, used to update model parameters within a single meta-learning task; Indicates the first The query set of each meta-learning task is used to evaluate the updated model within the task and update the model parameters shared by multiple meta-learning tasks accordingly. During model training, the binary cross-entropy loss function is used to calculate the classification loss for any set of samples. Its classification loss is: ； in This indicates that the keyframe relationship discrimination model is in the sample set. Classification loss on Represents the sample set The number of training keyframe pairs; The model parameters are: At that time, the keyframe relationship discrimination model targets the features of the keyframes during training. Output relation confidence; This represents the model parameters shared across multiple meta-learning tasks, including the hidden layer weight matrix. Hidden layer bias vector Output layer weight matrix and output layer bias ; For the Individual meta-learning tasks, using model parameters shared across multiple meta-learning tasks. As initial task parameters ,Right now ; According to the support set The classification loss is used to update the in-task parameters of the task. The update within this task is as follows: ； in Indicates the first The first generation of the meta-learning task was executed. This task updates the parameters of the previous task. Indicates the first Individual learning execution Updated task parameters within this task; Indicates the number of updates within the task. Indicates the learning rate within the task; ) indicates the use of parameters Time model in support set Classification loss on The support set classification loss is expressed with respect to the parameters. The gradient; Complete the preset number of times After the task update, the first... Task adaptation parameters for individual learning tasks ; Using task adaptation parameters Calculate the corresponding query set The loss is calculated, and the shared model parameters are updated based on the classification loss of the query set of multiple meta-learning tasks selected in the first-order training iteration: ; in Indicates the use of task adaptation parameters The time model in the query set Classification loss on This indicates that the classification loss of the query set is related to the shared model parameters. The gradient; Indicates the learning rate for shared model parameters. This indicates the number of meta-learning tasks selected in a single-dimensional training iteration.

5. The visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection according to claim 4, characterized in that, Step 3 specifically involves: Step 3.

1. Acquire continuously captured image frames from the camera, and when the current image frame meets the keyframe insertion conditions, determine it as the current keyframe; The keyframe insertion condition is as follows: ; in This represents the number of intra-map points retained after the current image frame is matched with the local map and optimized for pose. This indicates the number of map points already tracked in the reference keyframe; the reference keyframe refers to the keyframe that has been established for local map tracking during the current image frame tracking process. Step 3.

2. Calculate the visual similarity between the current keyframe and historical keyframes in the keyframe database, and determine the candidate keyframe set corresponding to the current keyframe; Define the current keyframe as Any historical keyframe in the keyframe database is ; First, construct the current keyframe. Normalized visual bag-of-words vectors and historical keyframes Normalized visual bag-of-words vectors ; Then, the visual similarity between the current keyframe and each historical keyframe in the keyframe database is calculated. The specific calculation formula is as follows: ； in Indicates the current keyframe With historical keyframes Visual similarity between them; Historical keyframes were sorted from highest to lowest visual similarity, and those at the top of the list were selected. Use one historical keyframe as the current keyframe. The corresponding candidate keyframes form a candidate keyframe set. ; When the number of historical keyframes in the keyframe database is less than When this is the case, all historical keyframes are selected as candidate keyframes; Step 3.

3. Based on the normalized bag-of-words vector of the current keyframe and the normalized bag-of-words vector of the candidate keyframe, construct keyframe pair features and input them into the trained keyframe relationship discrimination model to obtain the confidence of the relationship between the current keyframe and the candidate keyframe. For the selected candidate keyframe set Historical keyframes Hereinafter referred to as candidate keyframes ; For the candidate keyframe set Any candidate keyframe The visual bag-of-words vectors of the current keyframe are arranged first, followed by the visual bag-of-words vectors of the candidate keyframes, to construct keyframe feature pairs. ; Keyframes and features The keyframe relationship discrimination model, after being trained, is processed to obtain: ; in This represents the current keyframe output by the keyframe relationship discrimination model. With candidate keyframes Confidence of the relationship The model parameters are: Keyframe relationship discrimination model; This represents the fixed model parameters obtained after completing meta-learning training.

6. The visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection according to claim 5, characterized in that, Step 4 specifically involves: Define the relationship determination threshold as ; For the candidate keyframe set Any candidate keyframe The relationship confidence obtained in step 3.3 Relationship determination threshold Compare; When the relationship determination condition is met At that time, retain the corresponding candidate keyframes. Otherwise, discard the corresponding candidate keyframe. All candidate keyframes that satisfy the aforementioned relationship determination conditions are combined into a confidence candidate keyframe set. ; When candidate keyframe set If no candidate keyframes satisfy the relationship determination conditions are found, subsequent ORB feature matching and geometric consistency verification will not be performed, and the visual synchronous localization and mapping system will continue to perform normal tracking and mapping processing.

7. The visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection according to claim 6, characterized in that, Step 5 specifically involves: Step 5.

1. Perform ORB feature matching; Define any candidate keyframe in the set of confidence candidate keyframes as a confidence candidate keyframe. ; First, extract the current keyframe. and confidence candidate keyframes ORB feature points and their corresponding ORB feature descriptors; Calculate the current keyframe Any ORB feature descriptor and confidence candidate keyframe The Hamming distance between each ORB feature descriptor is used to obtain the minimum Hamming distance. Distance between the second and third smallest Hamming Then calculate the nearest neighbor distance ratio. ; when Not greater than the preset nearest neighbor distance ratio threshold At that time, the minimum Hamming distance The two corresponding ORB feature points are retained as candidate matching feature point pairs; Then, based on the ORB feature directions of the two ORB feature points in the candidate matching feature point pairs, the direction consistency is filtered to remove candidate matching feature point pairs with inconsistent direction relationships, thus obtaining the initial matching feature point pairs. When the number of initial matched feature point pairs is not less than the preset threshold for the number of matched feature point pairs. At that time, continue performing the initial geometric transformation; Otherwise, determine the current keyframe. With confidence candidate keyframes Geometric consistency verification failed; Step 5.

2. Perform initial geometric transformation estimation; Based on the initial matching feature point pairs obtained in step 5.1, obtain the map points associated with each matching feature point from the visual synchronous localization and mapping system, and retain the initial matching feature point pairs where both matching feature points are associated with map points. A pair of map points associated with a pair of matching feature points are combined into a matching map point pair. Obtain the 3D coordinates of the matched map points and the current keyframe. and confidence candidate keyframes The pose already determined in the visual synchronous localization and mapping system; Based on the pose of the corresponding keyframe, perform a conventional coordinate transformation to convert the two map points in each matched map point pair to confidence candidate keyframes. and the current keyframe From the camera coordinate system, obtain the corresponding three-dimensional coordinates. and ; in, This indicates that the corresponding map point in the matched map point pair is in the confidence candidate keyframe. 3D coordinates in the camera coordinate system This indicates the map point that matches it in the current keyframe. Three-dimensional coordinates in the camera coordinate system; Then, based on the Sim(3) solution method with random sampling consistency, according to... and Estimating confidence candidate keyframes Up to the current keyframe The initial geometric transformation; Specifically, a matching map point pair is randomly selected, and the Sim(3) geometric transformation is calculated based on the selected matching map point pair. The Sim(3) geometric transformation obtained by each random selection is recorded as a candidate geometric transformation. Based on the candidate geometric transformation, the matching map points are projected onto the image plane of the corresponding key frame. The distance between the projected position of the map point and the position of the corresponding matching ORB feature point in the image is calculated, and the distance is used as the reprojection error. When the reprojection error is not greater than a preset reprojection error threshold When the corresponding matching map point pair is determined to have geometric consistency with the candidate geometric transformation, the matching map point pair is identified as a matching interior point, and the number of matching interior points is counted. Repeatedly perform random selection of matching map point pairs, calculation of candidate geometric transformations, and determination of matching interior points, and retain the candidate geometric transformation with the most matching interior points; Define the preset threshold for the number of matched intrapoints as follows: ; When the maximum number of matched interior points obtained within the preset maximum number of iterations is not less than When the corresponding candidate geometric transformation is determined, it is identified as a confidence candidate keyframe. Up to the current keyframe Initial geometric transformation ; in Indicates the scale factor. This represents the rotation matrix estimated from the matching map point pairs. This represents the translation vector estimated from the matched map point pairs; And the initial geometric transformation satisfies: ; When the maximum number of matched interior points obtained within the preset maximum number of iterations is less than At that time, determine the current keyframe With confidence candidate keyframes The geometric consistency verification at this stage has not been passed; Step 5.

3. Perform coarse projection matching; Obtain and believable candidate keyframes from the keyframe co-view relationships established by the visual synchronous localization and mapping system. Locally shared keyframes with shared viewing relationships are identified, and confidence candidate keyframes are selected. and the map points associated with its local co-view keyframes; The initial geometric transformation estimated in step 5.2 Project the map points onto the current keyframe. The image plane; For any projected map point, the predicted ORB scale layer corresponding to that map point is used as a reference. Centered on its projection position, with To determine the search radius, a projection matching method based on visual synchronous localization and mapping is used, based on the current keyframe extracted in step 5.

1. Candidate ORB feature points are determined from the ORB feature points; in , Indicates the first Scale factor corresponding to each ORB scale layer; predict ORB scale layer It refers to the ORB image pyramid hierarchy determined by the visual synchronous localization and mapping system based on the distance between the projected map point and the current keyframe camera, as well as the existing scale information of the map point. Calculate the Hamming distance between the ORB feature descriptor of the projected map point and the ORB feature descriptor of each candidate ORB feature point within the search range, and select the ORB feature with the smallest Hamming distance as the candidate match. When the minimum Hamming distance is not greater than the preset coarse projection distance threshold At this time, candidate matches are retained to obtain coarse projection matching results; The number of candidate matches obtained from coarse projection matching is counted. When the number of coarse projection matches is not less than a preset threshold for the number of coarse projection matches... If the condition is met, continue with geometric transformation optimization; otherwise, determine the current keyframe. With confidence candidate keyframes Geometric consistency verification failed; Step 5.

4. Perform geometric transformation optimization; Using the coarse projection matching result obtained in step 5.3 as the observation, and the reprojection error of the matching points as the optimization basis, the initial geometric transformation obtained in step 5.2 is applied. Sim(3) optimization is performed to obtain the optimized geometric transformation. ; The number of matching interior points retained after Sim(3) optimization is counted. When the number of matching interior points is not less than a preset threshold for the number of interior points... If the condition is met, perform fine projection matching; otherwise, determine the current keyframe. With confidence candidate keyframes Geometric consistency verification failed; Step 5.

5. Perform fine-grained projection matching; Based on the optimized geometric transformation in step 5.4 The confidence candidate keyframes selected in step 5.3 are then re-selected. The map points associated with the local co-view keyframe are projected onto the current keyframe. The image plane; For any projected map point, the predicted ORB scale layer corresponding to that map point is used as a reference. Centered on its projection position, with The search radius is the current keyframe extracted in step 5.

1. Re-search for ORB features among the ORB feature points; where , It is a constant; Calculate the Hamming distance between the ORB feature descriptor of the projected map point and the ORB feature descriptor of each ORB feature point within the search range, and determine the ORB feature point corresponding to the minimum Hamming distance; When the minimum Hamming distance is not greater than the preset fine projection distance threshold When this happens, the corresponding match is retained to obtain a fine projection match; The number of fine projection matches is counted, and when the number of fine projection matches is not less than the preset threshold for the number of fine projection matches... At that time, determine the current keyframe With confidence candidate keyframes Complete the current geometric consistency verification; otherwise, determine that neither of them has passed the current geometric consistency verification. Step 5.

6. Perform geometric consistency verification; Obtain the current keyframe from the shared-view relationships of keyframes already established by the visual synchronous localization and mapping system. Several shared keyframes, and according to the current keyframe The number of jointly observed map points is sequentially projected and verified. Specifically, for any of the aforementioned shared keyframes Based on the optimized geometric transformation obtained in step 5.4 Combined with the current keyframe With the co-view keyframe Based on the existing relative poses, determine the confidence candidate keyframes. With the co-view keyframe Geometric transformations between the two, and projection verification based on the geometric transformations; When the number of matches obtained in a single projection verification is not less than the preset verification match number threshold When this happens, it is recorded as a successful geometric consistency verification. For the same confidence candidate keyframe Sequentially process the current keyframe The projection verification is performed on different co-view keyframes and the number of successful geometric consistency verifications is accumulated. When the number of successful verifications is not less than a preset geometric consistency verification threshold At that time, confirm the current keyframe. With confidence candidate keyframes Verification of geometric consistency was passed; Step 5.

7. For the confidence candidate keyframes that have passed the geometric consistency verification, based on the current keyframe... With confidence candidate keyframes To determine if the maps belong to the same location, establish loop closure constraints or multi-map fusion constraints, then update the keyframe pose and map point coordinates to obtain the optimized camera pose estimation results and map.

8. The visual simultaneous localization and mapping method based on meta-learning candidate keyframe selection according to claim 7, characterized in that, Step 5.7 specifically involves: I. When the current keyframe With confidence candidate keyframes When they belong to the same map, according to the optimized geometric transformation Determine the current keyframe With confidence candidate keyframes Establish relative pose constraints between them, and establish closure constraints; Correct the current keyframe based on the closure constraint. The pose is determined, and while maintaining the original relative pose relationship between the current keyframe and its co-view keyframes, the pose correction is propagated to the current keyframe. Shared keyframes; Set the current keyframe The pose-corrected keyframes and their co-view keyframes after pose correction propagation are collectively referred to as pose-corrected keyframes. Any pose-corrected keyframe is denoted as... ; Based on keyframes The pose changes before and after correction are used to correct the coordinates of the associated map points; the corrected coordinates of the map points are: ； in This represents the coordinates of a map point in the world coordinate system after calibration. This indicates the coordinates of a map point in the world coordinate system before correction. and These represent keyframes. World coordinate system to keyframe before and after correction Geometric transformation of the camera coordinate system; Keyframe Geometric transformation from camera coordinate system to world coordinate system; The map points observed repeatedly on both sides of the loop are fused together, and the current keyframe is then used. With confidence candidate keyframes Add the closure constraints between them to the keyframe pose graph; Then, the pose graph optimization method of the visual synchronous localization and mapping system is adopted. Based on the newly added closure constraints and the relative pose constraints between the original keyframes, the pose of the keyframes is optimized as a whole, and the coordinates of the corresponding map points are updated according to the optimized keyframe poses. II. When the current keyframe With confidence candidate keyframes When belonging to different maps, the optimized geometric transformation will be applied. Establish multi-map fusion constraints as an alignment relationship between two maps; Based on the multi-map fusion constraints, the current keyframe The keyframe poses in the map are transformed by coordinates, and the coordinate transformation is propagated to the corresponding map points, so that the keyframes and map points in the two maps are in the same coordinate system. For two maps after coordinate transformation, duplicate map points representing the same scene features are merged, and keyframes and map points in the two maps are combined. Then, the map optimization method of the visual synchronous localization and mapping system is adopted to optimize the keyframe pose and map point coordinates after fusion, so as to obtain the unified map after fusion. After the above loop closure correction or multi-map fusion, the corresponding keyframe pose and map point coordinates are updated to obtain the optimized camera pose estimation results and map.

9. A visual simultaneous localization and mapping system based on meta-learning candidate keyframe selection, characterized in that, Includes the following modules: The preprocessing module is used to obtain historical keyframes from the historical running results of multiple visual scenes or multiple visual data sequences, preprocess the historical keyframes, and obtain the features and corresponding labels of the training keyframe pairs. The model training module is used to build a keyframe relationship discrimination model and uses a model-independent meta-learning method to train features and corresponding relationship labels offline based on training keyframes. The keyframe relationship discrimination model is used to process the features of the training keyframe pairs and output the relationship confidence of the corresponding training keyframe pairs. The candidate retrieval and relationship confidence calculation module is used to acquire continuously captured image frames from the camera and determine the current keyframe based on the keyframe insertion criteria. Calculate the visual similarity between the current keyframe and each historical keyframe, and construct a set of candidate keyframes corresponding to the current keyframe; Construct keyframe pair features between the current keyframe and candidate keyframes, and input them into the trained keyframe relationship discrimination model to obtain the confidence of the relationship between the current keyframe and candidate keyframes; The candidate keyframe filtering module is used to filter candidate keyframes according to the relationship confidence level, and retain candidate keyframes with a relationship confidence level not lower than a preset relationship judgment threshold to form a set of confidence candidate keyframes. The geometric consistency verification, loop closure correction, and multi-map fusion module is used to perform ORB feature matching and geometric consistency verification on the current keyframe and candidate keyframes in the set of confidence candidate keyframes, and to establish loop closure constraints or multi-map fusion constraints. Then, the keyframe pose and map point coordinates are updated to obtain the optimized camera pose estimation results and map.

Images