Method and apparatus for instant localization and mapping

Abstract

Provided are a simultaneous localization and mapping method and device. The simultaneous localization and mapping method comprises: acquiring a current frame image input by a camera; performing scene recognition on the current frame image to obtain a key frame image having the greatest similarity to the current frame image in a global map; and determining a camera pose of the current frame image according to the key frame image. The simultaneous localization and mapping method comprises: acquiring a current frame image input by a camera; determining a key frame image having a similarity to the current frame image; acquiring a feature point matching relationship between the current frame image and the key frame image; and calculating a camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image. According to the simultaneous localization and mapping method and device of the example embodiments of the present disclosure, the camera pose can be calculated by a traditional method, and the camera pose can also be determined by loop detection and repositioning through a deep learning model.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology. More specifically, this disclosure relates to a method and apparatus for real-time localization and mapping. Background Technology

[0002] Simultaneous Localization and Mapping (SLAM) systems capture input from various sensors (LiDAR, camera, IMU, GPS, Kinect, etc.) to estimate pose in real time and construct 3D scenes simultaneously. It is widely used in autonomous driving, robot navigation, and augmented reality. Cameras are extensively used in SLAM systems because they can acquire richer texture information and are unaffected by distance or building occlusion. The ability to find data correlations between two images is crucial to the accuracy of camera tracking in SLAM systems, as well as the robustness of tracking within the system. Scene recognition technology calculates the similarity between any two images, finding the most similar image. This can be used to find potentially congruent images in large-scale scenes for subsequent feature point matching, playing a vital role in loop closure detection and re-localization within SLAM systems. Summary of the Invention

[0003] Exemplary embodiments of this disclosure provide a real-time localization and mapping method and apparatus for determining camera pose through loop closure detection and repositioning.

[0004] According to an exemplary embodiment of the present disclosure, a real-time localization and mapping method is provided, comprising: acquiring a current frame image input through a camera; performing scene recognition on the current frame image to obtain a key frame image with the highest similarity to the current frame image in a global map; and determining the camera pose of the current frame image based on the key frame image.

[0005] Optionally, the step of determining the camera pose of the current frame image based on the keyframe image may include: obtaining the feature point matching relationship between the current frame image and the keyframe image; and calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the keyframe image.

[0006] Optionally, the step of scene recognition for the current frame image may include: obtaining the global descriptor of the current frame image; and querying the keyframe image with the highest similarity to the current frame image in the global map based on the global descriptor of the current frame image.

[0007] Optionally, the step of querying the keyframe image with the highest similarity to the current frame image in the global map based on the global descriptor of the current frame image may include: calculating the distance between the global descriptor of each keyframe image in the global map and the global descriptor of the current frame image; determining the similarity between the global descriptor of the keyframe image and the global descriptor of the current frame image based on the distance between the global descriptor of the keyframe image and the global descriptor of the current frame image, wherein the closer the distance between the global descriptor of the keyframe image and the global descriptor of the current frame image, the greater the similarity between the keyframe image and the current frame image.

[0008] Optionally, the step of obtaining the feature point matching relationship between the current frame image and the key frame image may include: obtaining the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; obtaining the scene structure of the current frame image and the key frame image based on the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; and obtaining the feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image.

[0009] Optionally, the step of obtaining the local 3D point cloud of the current frame image may include: obtaining the 2D feature points and depth values ​​of the current frame image; and using pose priors to convert the 2D feature points and depth values ​​of the current frame image into the local 3D point cloud of the current frame image.

[0010] Optionally, the step of obtaining the global 3D point cloud of the keyframe image may include: converting the 2D feature points and depth values ​​of the keyframe image into a global 3D point cloud based on the camera pose of the keyframe image.

[0011] Optionally, the step of obtaining the feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image may include: encoding the local 3D point cloud and the global 3D point cloud respectively through a 3D position encoder to obtain local high-dimensional features and global high-dimensional features; obtaining the feature point descriptors of the current frame image and the key frame image, and inputting the local high-dimensional features and global high-dimensional features, as well as the feature point descriptors of the current frame image and the key frame image, into a feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

[0012] Optionally, the step of calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image may include: calculating the camera pose of the current frame image using local bundle adjustment or global bundle adjustment based on the feature point matching relationship between the current frame image and the key frame image.

[0013] Optionally, the step of obtaining the global descriptor of the current frame image may include: obtaining the global descriptor of the current frame image through an image depth feature extraction network based on Intersection over Union (IOU).

[0014] Optionally, the method may further include: training an IOU-based image depth feature extraction network. The step of training the IOU-based image depth feature extraction network may include: obtaining anchor images and positive sample images from training images; and training the IOU-based image depth feature extraction network according to the anchor images, positive sample images, and the similarity between the anchor images and positive sample images.

[0015] Optionally, the steps of obtaining anchor images and positive sample images from training images may include: randomly sampling anchor images from training images; randomly sampling to obtain the similarity between anchor images and positive sample images; and sampling images from training images that have a similarity to the anchor images equal to the obtained similarity as positive sample images.

[0016] Optionally, the similarity between the anchor image and the positive sample image is IOU.

[0017] Optionally, after obtaining the anchor image and positive sample image from the training image, the method may further include performing a projective transformation on one or more of the anchor image and positive sample image.

[0018] According to an exemplary embodiment of the present disclosure, a real-time localization and mapping method is provided, comprising: acquiring a current frame image input through a camera; determining a key frame image that has the greatest similarity to the current frame image; acquiring a feature point matching relationship between the current frame image and the key frame image; and calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image.

[0019] Optionally, the step of obtaining the feature point matching relationship between the current frame image and the key frame image may include: obtaining the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; obtaining the scene structure of the current frame image and the key frame image based on the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; and obtaining the feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image.

[0020] Optionally, the step of obtaining the local 3D point cloud of the current frame image may include: obtaining the 2D feature points and depth values ​​of the current frame image; and using pose priors to convert the 2D feature points and depth values ​​of the current frame image into the local 3D point cloud of the current frame image.

[0021] Optionally, the step of obtaining the global 3D point cloud of the keyframe image may include: converting the 2D feature points and depth values ​​of the keyframe image into a global 3D point cloud based on the camera pose of the keyframe image.

[0022] Optionally, the step of obtaining the feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image may include: encoding the local 3D point cloud and the global 3D point cloud respectively through a 3D position encoder to obtain local high-dimensional features and global high-dimensional features; obtaining the feature point descriptors of the current frame image and the key frame image, and inputting the local high-dimensional features and global high-dimensional features, as well as the feature point descriptors of the current frame image and the key frame image, into a feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

[0023] Optionally, the step of calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image may include: calculating the camera pose of the current frame image using local bundle adjustment or global bundle adjustment based on the feature point matching relationship between the current frame image and the key frame image.

[0024] According to an exemplary embodiment of the present disclosure, an instantaneous localization and mapping apparatus is provided, comprising: an image acquisition unit configured to acquire a current frame image input through a camera; a scene recognition unit configured to perform scene recognition on the current frame image to obtain a key frame image with the highest similarity to the current frame image in a global map; and a pose calculation unit configured to determine the camera pose of the current frame image based on the key frame image.

[0025] Optionally, the pose calculation unit may include: a matching relationship acquisition unit, configured to acquire the feature point matching relationship between the current frame image and the key frame image; and a pose calculation subunit, configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image.

[0026] Optionally, the scene recognition unit may include: a descriptor acquisition unit configured to acquire a global descriptor of the current frame image; and a keyframe query unit configured to query a keyframe image with the highest similarity to the current frame image in the global map based on the global descriptor of the current frame image.

[0027] Optionally, the keyframe query unit can be configured to: calculate the distance between the global descriptor of each keyframe image in the global map and the global descriptor of the current frame image; determine the similarity between the global descriptor of the keyframe image and the global descriptor of the current frame image based on the distance between the global descriptor of the keyframe image and the global descriptor of the current frame image, wherein the closer the distance between the global descriptor of the keyframe image and the global descriptor of the current frame image, the greater the similarity between the keyframe image and the current frame image.

[0028] Optionally, the matching relationship acquisition unit may include: a 3D point cloud acquisition unit configured to acquire a local 3D point cloud of the current frame image and a global 3D point cloud of the key frame image; a scene structure acquisition unit configured to obtain the scene structure of the current frame image and the key frame image based on the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; and a feature point matching unit configured to obtain the feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image.

[0029] Optionally, the 3D point cloud acquisition unit can be configured to: acquire the two-dimensional feature points and depth values ​​of the current frame image; and use pose priors to convert the two-dimensional feature points and depth values ​​of the current frame image into a local 3D point cloud of the current frame image.

[0030] Optionally, the 3D point cloud acquisition unit can be configured to convert the two-dimensional feature points and depth values ​​of the keyframe image into a global 3D point cloud based on the camera pose of the keyframe image.

[0031] Optionally, the feature point matching unit can be configured to: encode the local 3D point cloud and the global 3D point cloud respectively through a 3D position encoder to obtain local high-dimensional features and global high-dimensional features; obtain the feature point descriptors of the current frame image and the key frame image, and input the local high-dimensional features, the global high-dimensional features, the feature point descriptors of the current frame image and the key frame image into the feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

[0032] Optionally, the pose calculation subunit can be configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image using local bundle adjustment or global bundle adjustment.

[0033] Optionally, the descriptor acquisition unit can be configured to acquire the global descriptor of the current frame image through an image depth feature extraction network based on Intersection over Union (IOU).

[0034] Optionally, the apparatus may further include an extraction network training unit configured to: acquire anchor images and positive sample images from training images; and train an IOU-based image depth feature extraction network based on the anchor images, positive sample images, and the similarity between the anchor images and positive sample images.

[0035] Optionally, the network training unit can be configured to: randomly sample anchor images from training images; randomly sample the similarity between anchor images and positive sample images; and sample images from training images that have a similarity to the anchor images equal to the obtained similarity as positive sample images.

[0036] Optionally, the similarity between the anchor image and the positive sample image is IOU.

[0037] Optionally, the extraction network training unit is further configured to perform a projective transformation on one or more of the anchor images and positive sample images after obtaining the anchor images and positive sample images from the training images.

[0038] According to an exemplary embodiment of the present disclosure, an instantaneous localization and mapping apparatus is provided, comprising: an image acquisition unit configured to acquire a current frame image input through a camera; a keyframe selection unit configured to determine a keyframe image having the greatest similarity to the current frame image; a matching relationship acquisition unit configured to acquire a feature point matching relationship between the current frame image and the keyframe image; and a pose calculation subunit configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the keyframe image.

[0039] Optionally, the matching relationship acquisition unit may include: a 3D point cloud acquisition unit configured to acquire a local 3D point cloud of the current frame image and a global 3D point cloud of the key frame image; a scene structure acquisition unit configured to obtain the scene structure of the current frame image and the key frame image based on the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; and a feature point matching unit configured to obtain the feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image.

[0040] Optionally, the 3D point cloud acquisition unit can be configured to: acquire the two-dimensional feature points and depth values ​​of the current frame image; and use pose priors to convert the two-dimensional feature points and depth values ​​of the current frame image into a local 3D point cloud of the current frame image.

[0041] Optionally, the 3D point cloud acquisition unit can be configured to convert the two-dimensional feature points and depth values ​​of the keyframe image into a global 3D point cloud based on the camera pose of the keyframe image.

[0042] Optionally, the feature point matching unit can be configured to: encode the local 3D point cloud and the global 3D point cloud respectively through a 3D position encoder to obtain local high-dimensional features and global high-dimensional features; obtain the feature point descriptors of the current frame image and the key frame image, and input the local high-dimensional features, the global high-dimensional features, the feature point descriptors of the current frame image and the key frame image into the feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

[0043] Optionally, the pose calculation subunit can be configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image using local bundle adjustment or global bundle adjustment.

[0044] According to exemplary embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements an instantaneous localization and mapping method according to exemplary embodiments of the present disclosure.

[0045] According to an exemplary embodiment of the present disclosure, a computing device is provided, including: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, it implements an instantaneous localization and mapping method according to an exemplary embodiment of the present disclosure.

[0046] According to exemplary embodiments of the present disclosure, a computer program product is provided, wherein the instructions in the computer program product are executable by a processor of a computer device to perform an instantaneous localization and mapping method according to exemplary embodiments of the present disclosure.

[0047] The real-time localization and mapping method and apparatus according to exemplary embodiments of the present disclosure acquires a current frame image input through a camera; performs scene recognition on the current frame image to obtain a keyframe image with the highest similarity to the current frame image in a global map; and determines the camera pose of the current frame image based on the keyframe image. Thus, for images captured from the camera, both traditional algorithms can be used to extract feature points and descriptors for tracking, followed by calculation of the camera pose using bundle adjustment; and deep learning models can be used for loop closure detection and relocalization. The method selects the most similar keyframe image from a global map composed of keyframe images for the current frame image captured from the camera, and determines the camera pose of the current frame image based on the keyframe image.

[0048] The real-time localization and mapping method and apparatus according to exemplary embodiments of the present disclosure acquires a current frame image input through a camera; determines a key frame image with similarity to the current frame image; acquires a feature point matching relationship between the current frame image and the key frame image; and calculates the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image. Thus, for images captured from the camera, both traditional algorithms can be used to extract feature points and descriptors for tracking, followed by calculation of the camera pose using bundle adjustment; and deep learning models can be used for loop closure detection and relocalization. The method determines the most similar key frame image to the current frame image captured from the camera and calculates the feature point matching relationship (i.e., data association) between the current frame image and the determined key frame image.

[0049] Furthermore, the real-time localization and mapping method and apparatus according to the exemplary embodiments of this disclosure can obtain a model with better generalization ability by utilizing an unsupervised deep learning model, while improving the expressive power of deep features through an IOU-related image deep feature extraction method, so as to find better similar images more accurately, and find the feature point matching relationship between the current frame image and the queried key frame image through a scene structure-based data association model, so as to recover the camera pose of the current frame image.

[0050] Further aspects and / or advantages of the general concept of this disclosure will be set forth in part in the description which follows, and in part will be clear from the description or may be learned by practice of the general concept of this disclosure. Attached Figure Description

[0051] The above and other objects and features of exemplary embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, which exemplarily illustrate the embodiments, wherein:

[0052] Figure 1 A flowchart illustrating an exemplary embodiment of the present disclosure of a real-time localization and mapping method is shown.

[0053] Figure 2 A schematic diagram illustrating an instantaneous localization and mapping process according to an exemplary embodiment of the present disclosure is shown.

[0054] Figure 3 A flowchart illustrating one training iteration of an IOU-based image depth feature extraction network according to an exemplary embodiment of the present disclosure is provided.

[0055] Figure 4 A schematic diagram illustrating the training of an IOU-based image depth feature extraction network according to an exemplary embodiment of the present disclosure is shown.

[0056] Figure 5A schematic diagram illustrating the positive sample sampling principle during training of an IOU-based image depth feature extraction network according to an exemplary embodiment of the present disclosure.

[0057] Figure 6 A schematic diagram illustrating the training principle of an IOU-based image depth feature extraction network according to an exemplary embodiment of the present disclosure is shown.

[0058] Figure 7 This diagram illustrates an exemplary embodiment of the present disclosure, in which an image depth feature extraction network with IOU-related features searches for the keyframe image with the highest similarity to the current frame image in a global map.

[0059] Figure 8 A schematic diagram of a scene structure-preserving feature point matching network according to an exemplary embodiment of the present disclosure is shown.

[0060] Figure 9 A flowchart illustrating a scene structure-preserving feature point matching network calculating feature point matching relationships according to an exemplary embodiment of the present disclosure is shown.

[0061] Figure 10 A flowchart illustrating an instantaneous localization and mapping method according to another exemplary embodiment of the present disclosure is shown.

[0062] Figure 11 A flowchart illustrating an instantaneous localization and mapping method according to another exemplary embodiment of the present disclosure is shown.

[0063] Figure 12 A block diagram of an instantaneous positioning and mapping apparatus according to an exemplary embodiment of the present disclosure is shown.

[0064] Figure 13 A block diagram of an instantaneous positioning and mapping apparatus according to another exemplary embodiment of the present disclosure is shown.

[0065] Figure 14 A block diagram of an instantaneous positioning and mapping apparatus according to another exemplary embodiment of the present disclosure is shown.

[0066] Figure 15 A schematic diagram of a computing device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0067] Exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same components. The embodiments will now be described with reference to the accompanying drawings in order to explain the present disclosure.

[0068] Currently, scene recognition technology extracts traditional ORB feature points and BRIEF descriptors, uses the Bag-of-Words (BOW) model to find co-view frames, and then uses the Random Sampling Consensus (RANSAC) algorithm to filter out erroneous matches between feature points extracted from two images to obtain data associations. However, due to the limitations of descriptor expressive power, the accuracy of finding similar frames is not high, and erroneous feature point matches also reduce the accuracy of the initial value-sensitive RANSAC algorithm. Global features learned by the deep learning network NetVLAD suffer from domain adaptation problems and poor generalization ability in new scenes. The most outstanding deep learning networks, SuperPoint and SuperGlue, first extract feature points and descriptors using the SuperPoint network, and then use the SuperGlue network to find feature point matches between two images using a graph neural network, ultimately calculating the camera pose. They perform well in various scenes.

[0069] However, the traditional method of calculating camera pose by finding similar frames using ORB+BRIER+BOW is limited by the expressive power of the BRIEF descriptor, often resulting in incorrect matching results. This leads to low accuracy when using BOW to find similar frames, and the incorrect matching relationships provide incorrect initial values ​​for the initial value-sensitive RANSAC algorithm, often failing to find feature point matching relationships and thus failing to recover the camera pose. Global features learned through the NetVLAD deep learning network suffer from domain adaptation problems, exhibiting poor generalization ability in new scenes, and its network structure and training methods have been surpassed by current algorithms. In contrast, deep learning networks like SuperPoint and SuperGlue often obtain rich feature point and descriptor information. SuperGlue, through graph neural networks, can find matching relationships between two images. However, SuperGlue only utilizes the 2D coordinate information of feature points, ignoring scene structure information, i.e., the 3D point cloud formed by the 3D coordinates corresponding to each feature point in the image. This is easily obtained during SLAM system operation and is very helpful for inter-frame feature point matching.

[0070] Therefore, a hybrid SLAM system is needed, which can quickly achieve the tracking, localization, and mapping functions of SLAM using traditional methods, while also providing loop closure detection and repositioning to recover camera pose in case of tracking failure during SLAM operation. This requires the system to have strong generalization ability, be able to operate in different scenarios, and fully utilize the scene structure information recovered during SLAM operation to help the SLAM system better recover camera pose.

[0071] The purpose of this disclosure is to find the most similar keyframes in the global map constructed during the operation of the SLAM system from the images input from the camera, establish feature point matching between them, and calculate the camera pose, providing more constraint information for the loop closure detection and relocalization modules of the SLAM system. On the one hand, the images obtained from the camera will select keyframes through a keyframe selection strategy to construct a global map, and calculate the camera pose through local bundle adjustment and global bundle adjustment methods, realizing real-time localization and mapping of the SLAM system. On the other hand, in order to obtain a deep learning network with strong generalization ability to obtain image depth features, an image depth feature extraction network with IOU correlation trained in unsupervised manner is proposed. The correlation between the input images is explicitly calculated to constrain the image depth features learned by the network. The similarity between the two frames enables unsupervised deep learning networks to help SLAM systems find the keyframe most similar to the query frame from the global map. On the other hand, since the SuperPoint+SuperGlue deep learning network structure only uses the 2D coordinate system of feature points and ignores the 3D coordinate information, a feature point matching network based on scene structure is proposed. This network transforms the 3D coordinate information of feature points into point clouds to obtain the corresponding scene structure, and then feeds it along with the descriptors into the SuperGlue network to learn the feature point matching relationship between images. Finally, using the obtained matching relationship between feature points, the relative pose between the input image and the co-view frame is recovered, which can then be used in local bundle adjustment and global bundle adjustment for localization and mapping in SLAM systems.

[0072] Figure 1 A flowchart illustrating an exemplary embodiment of the present disclosure of a real-time localization and mapping method is shown.

[0073] Figure 2 This diagram illustrates a real-time localization and mapping process according to exemplary embodiments of the present disclosure. The real-time localization and mapping method of the present disclosure can be used in applications such as, but not limited to, autonomous driving, robot navigation, and augmented reality.

[0074] Reference Figure 1 In step S101, the current frame image input through the camera is acquired.

[0075] Specifically, in application scenarios such as autonomous driving, robot navigation, and augmented reality, cameras capture images in real time.

[0076] In step S102, scene recognition is performed on the current frame image to obtain the key frame image with the highest similarity to the current frame image in the global map.

[0077] In an exemplary embodiment of this disclosure, when performing scene recognition on the current frame image, a global descriptor of the current frame image can be obtained first, and then, based on the global descriptor of the current frame image, a keyframe image with the highest similarity to the current frame image can be queried in the global map. Here, the keyframe images are stored in the global map. Figure 2 As shown, the global map may include a set of keyframes and a set of map points. The set of keyframes may include the camera pose and feature point descriptors for each keyframe image. The set of map points may include 3D map points.

[0078] In an exemplary embodiment of this disclosure, when querying the global map for the keyframe image with the highest similarity to the current frame image based on the global descriptor of the current frame image, the distance between the global descriptor of each keyframe image in the global map and the global descriptor of the current frame image can be calculated first. Then, the similarity between the global descriptors of the keyframe images and the current frame images can be determined based on the distance between them. Here, the closer the global descriptors of the keyframe images are to the global descriptors of the current frame images, the greater the similarity between them.

[0079] Specifically, when tracking and recognizing the current frame image, the keyframe image most similar to the current frame image can be searched in the global map as the common view frame image of the current frame image. Scene recognition is performed based on the common view frame image, and the scene structure is restored, thereby calculating the camera pose.

[0080] In an exemplary embodiment of this disclosure, when obtaining the global descriptor of the current frame image, the global descriptor of the current frame image can be obtained through an image depth feature extraction network based on Intersection Over Union (IOU).

[0081] In exemplary embodiments of this disclosure, an image depth feature extraction network based on IOU can also be trained.

[0082] The requirement for transforming an image into a feature vector using a deep learning network is to distinguish between positive and negative samples, ensuring that the distance between positive samples continuously decreases while the distance between negative samples increases during training, ultimately making the distance between positive samples smaller than the distance between negative samples, thus guaranteeing the deep learning network's ability to differentiate between them. Previous deep learning networks, when optimizing this problem, did not consider the differences between positive samples, assuming that errors or gradients caused by the distance between positive samples during training should be treated equally. This disclosure generalizes the problem, suggesting that the deep learning network should be constrained by similarity during optimization. Specifically, for positive samples, the more similar the positive samples, the closer their distances should be; the less similar the positive samples, the slightly larger their distances should be, but still smaller than their distances to negative samples.

[0083] In this disclosure, it is argued that the similarity between two images should be related to the overlapping area within the field of view of the two images. Here, IOU is used as a measure of similarity between images. Since IOU is also a commonly used metric function in other fields such as segmentation and detection, the objective function has two uses: on the one hand, to distinguish between positive and negative samples; on the other hand, to rank positive samples based on IOU.

[0084] In an exemplary embodiment of this disclosure, when training an IOU-based image depth feature extraction network, anchor images and positive sample images can be obtained from training images first, and then the IOU-based image depth feature extraction network can be trained based on the anchor images, positive sample images, and the similarity between the anchor images and positive sample images.

[0085] In an exemplary embodiment of this disclosure, when obtaining anchor images and positive sample images from training images, anchor images can first be randomly sampled from training images, and the similarity between anchor images and positive sample images can be randomly sampled. Then, images with a similarity to the obtained similarity between the anchor images and training images are sampled from the training images as positive sample images.

[0086] In an exemplary embodiment of this disclosure, the similarity between the anchor image and the positive sample image can be IOU.

[0087] In an exemplary embodiment of this disclosure, after obtaining the anchor image and the positive sample image from the training image, a projective transformation may be performed on one or more of the anchor image and the positive sample image.

[0088] Figure 3 A flowchart illustrating one training iteration of an IOU-based image depth feature extraction network according to an exemplary embodiment of the present disclosure is provided. Figure 4A schematic diagram illustrating the training of an image depth feature extraction network based on IOU according to an exemplary embodiment of the present disclosure is shown. Figure 4 Negative samples from the old feature queue of China and Israel To represent the features of negative samples. Figure 5 This diagram illustrates the principle of positive sample sampling when training an image depth feature extraction network based on IOU, according to an exemplary embodiment of this disclosure. Figure 5 The input image is used to represent the input training image. Figure 6 A schematic diagram illustrating the training principle of an IOU-based image depth feature extraction network according to an exemplary embodiment of the present disclosure is shown.

[0089] like Figure 3 As shown, in one training iteration, in step S301, the anchor image and positive sample image are obtained from the training image using a random sampling strategy based on IOU.

[0090] In an exemplary embodiment of this disclosure, when obtaining anchor images and positive sample images from training images using an IOU-based random sampling strategy, anchor images can be obtained by random sampling from the training images first, and IOUs can be obtained by random sampling. Then, images with an IOU equal to the intersection-union ratio of the anchor images are sampled from the training images as positive sample images.

[0091] Specifically, IOU can be obtained by random sampling using, for example, but not limited to, the average random distribution algorithm. For example, {IOU} can be obtained by randomly sampling twice within the range [0 to 1]. p1 ,IOU p2}

[0092] like Figure 5 As shown, for each input training image, random cropping is performed, and the cropped image is used as the anchor image; cropping is performed on the training image such that the IOU between the cropped image and the anchor image is equal to {IOU}. p1 ,IOU p2 The result of the cropping is taken as a positive sample, resulting in two positive samples {p1, p2}. In this disclosure, two positive samples are used as an example for ease of description. In actual application, there may be multiple positive samples, and this disclosure does not limit this.

[0093] In step S302, the IOU between the anchor image and the positive sample image is used as the target similarity. For example, the IOU can be obtained by randomly selecting two values ​​between [0 and 1]. p1 ,IOU p2} as target similarity.

[0094] In an exemplary embodiment of this disclosure, after obtaining the anchor image and the positive sample image from the training image, a projective transformation can be performed on the anchor image and the positive sample image, and the projective transformed anchor image and positive sample image can be used as the anchor image and the positive sample image, respectively.

[0095] The algorithm determines whether a warp transformation is needed on the anchor image and the two positive samples {p1, p2}. If so, a quadrilateral consisting of four points is randomly selected from the four rectangular regions of the image for cropping and photographic transformation. Then, other data augmentation transformations commonly used in deep learning are performed, such as random cropping and resizing, random color and luminance changes, random grayscale changes, random Gaussian blur, and random horizontal flipping, to obtain the final anchor and positive samples {p1, p2}.

[0096] In step S303, feature extraction is performed on the anchor point image and the positive sample image using an online convolutional neural network and a target convolutional neural network, respectively, to obtain anchor point features and positive sample features.

[0097] like Figure 4 As shown, the anchor point and positive samples {p1, p2} are fed into the online convolutional neural network and the target convolutional neural network, respectively, to obtain the corresponding features f. a and

[0098] In step S304, the anchor features and positive sample features are transformed from the feature space to the metric space using a multilayer perceptron.

[0099] like Figure 6 As shown, the Online Convolutional Neural Network and the Target Convolutional Neural Network have identical structures, with the last two layers being a Multi-Layer Perceptron (MLP). The feature f... a and The features are transformed from the feature space to the metric space by a multilayer perceptron (MLP), and the features in the metric space are normalized to obtain f′. a and

[0100] In step S305, the negative sample features of the current training iteration are determined based on the positive sample features from previous training iterations.

[0101] In an exemplary embodiment of this disclosure, when determining the negative sample features of the current training iteration based on the positive sample features from previous training iterations, the negative sample features are zero in the first training iteration; in each subsequent training iteration, the negative sample features of the current training iteration are determined based on the positive and negative sample features from the previous training iteration.

[0102] like Figure 4 As shown, the feature queue from previous training iterations can be used. The anchor feature f′ will be calculated using the negative sample features. a and negative sample features The cosine distance between the anchor images is used as the similarity between the anchor images and the negative samples, thus obtaining the similarity between the anchor images and both the positive and negative samples. in

[0103] In step S306, the cosine distance between the anchor point features, positive sample features, and negative sample features is calculated as the similarity.

[0104] Specifically, calculating the anchor point feature f′ a and positive sample features The cosine distance between the two images is used as the similarity between the anchor image and the positive sample image. exist Figure 3 In this context, similarity is represented by feature similarity scores.

[0105] In step S307, based on similarity and target similarity, the loss and gradient are calculated using the cross-entropy loss function to backpropagate and update the online convolutional neural network.

[0106] For example, it can be based on target similarity. As a monitoring signal, i.e. As shown in the following equation, the loss and gradient are calculated using the cross-entropy loss function for backpropagation and updating of the online convolutional neural network. Here, m represents the number of positive samples, and k represents the number of negative samples. The features of the positive samples are randomly selected... One is selected and sent to a feature queue with a fixed length. The negative sample features are used as the basis for the next iteration.

[0107]

[0108] In step S308, the parameters θ of the online convolutional neural network are used through an exponentially weighted average algorithm. Online For the parameters θ of the target convolutional neural network Target Update.

[0109] For example, it can be based on the formula θTarget =mθ Target +(1-m)θ Online The exponentially weighted average algorithm utilizes the parameters θ of the online convolutional neural network. Online For the parameters θ of the target convolutional neural network Target Target The system is updated, with the momentum parameter m = 0.999, which enables the target convolutional neural network (Target) to acquire better feature learning capabilities as it learns. At the same time, the exponentially weighted average algorithm ensures that the target convolutional neural network (Target) has a stable feature space, guaranteeing that the online convolutional neural network (Online) can converge during training.

[0110] In step S309, when the online convolutional neural network reaches the preset termination condition, the online convolutional neural network and the target convolutional neural network are used as an image depth feature extraction network based on IOU.

[0111] The remaining part after removing the final multilayer perceptron (MLP) of the online convolutional neural network is used as an IOU-related image depth feature extraction network. It can be used in the loop closure detection and relocalization module of the SLAM system for scene recognition and to find similar keyframe images as co-view frames.

[0112] Figure 7 This diagram illustrates an exemplary embodiment of the present disclosure, in which an image depth feature extraction network with IOU-related characteristics searches for the keyframe image with the highest similarity to the current frame image in a global map. For example... Figure 7 As shown, the shared IOU-related image depth feature extraction network first extracts the global descriptor of the current frame image and the global descriptor of each keyframe image in the keyframe set. Then, the IOU-related error function is calculated based on the global descriptor, and the keyframe image with the highest similarity to the current frame image in the global map is obtained according to the error function.

[0113] Return to reference Figure 1 In step S103, the camera pose of the current frame image is determined based on the keyframe image.

[0114] In an exemplary embodiment of this disclosure, when determining the camera pose of the current frame image based on the keyframe image, the feature point matching relationship between the current frame image and the keyframe image can be obtained first, and then the camera pose of the current frame image can be calculated based on the feature point matching relationship between the current frame image and the keyframe image.

[0115] Calculating the feature point matching relationship (i.e., data association) between two images is a crucial process in real-time localization and mapping. Traditional methods for calculating this relationship include two approaches: Method 1 involves extracting feature points and descriptors, selecting an initial matching relationship based on the similarity between descriptors, and finally using the RANSAC algorithm to filter out incorrect matches to obtain the final data association; Method 2 uses optical flow, where feature points from the previous frame are compared to similar feature points in the neighborhood of the next frame, and then RANSAC is used to filter out incorrect matches to obtain the final data association. However, optical flow only works within the neighborhood, and it cannot find data associations when there is significant motion between the two frames. For loop closure detection and relocalization, traditional methods can only achieve this through the first approach, as traditional descriptors have limited expressive power, often failing to find correct matching relationships. Currently, the SuperGlue network, through graph neural networks, considers both descriptors and positional constraints between feature points, significantly improving the success rate of finding data associations. However, the SuperGlue network only considers the two-dimensional coordinates of feature points when considering their positional constraints. This ignores the depth information of the feature points, resulting in the neglect of scene structure information when searching for data associations. Therefore, this disclosure proposes a feature point matching network based on scene structure, which not only makes full use of the 3D point cloud containing scene structure information reconstructed during the SLAM system operation, but also improves the accuracy in the process of searching for data associations.

[0116] In an exemplary embodiment of this disclosure, when obtaining the feature point matching relationship between the current frame image and the key frame image, the local three-dimensional point cloud of the current frame image and the global three-dimensional point cloud of the key frame image can be obtained first. The scene structure of the current frame image and the key frame image can be obtained based on the local three-dimensional point cloud of the current frame image and the global three-dimensional point cloud of the key frame image. Then, the feature point matching relationship between the current frame image and the key frame image can be obtained based on the scene structure of the current frame image and the key frame image.

[0117] In an exemplary embodiment of this disclosure, when acquiring a local three-dimensional point cloud of the current frame image, the two-dimensional feature points and depth values ​​of the current frame image can be acquired first, and then the pose prior can be used to convert the two-dimensional feature points and depth values ​​of the current frame image into a local three-dimensional point cloud of the current frame image.

[0118] In an exemplary embodiment of this disclosure, when acquiring the global three-dimensional point cloud of the keyframe image, the two-dimensional feature points and depth values ​​of the keyframe image are converted into a global three-dimensional point cloud based on the camera pose of the keyframe image.

[0119] In an exemplary embodiment of this disclosure, when obtaining the feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image, the local 3D point cloud and the global 3D point cloud can be encoded first by a 3D position encoder to obtain local high-dimensional features and global high-dimensional features. Then, the feature point descriptors of the current frame image and the key frame image are obtained. The local high-dimensional features and global high-dimensional features, as well as the feature point descriptors of the current frame image and the key frame image, are input into the feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

[0120] Figure 8 A schematic diagram of a scene structure-preserving feature point matching network according to an exemplary embodiment of the present disclosure is shown. Figure 9 A flowchart illustrating a scene structure-preserving feature point matching network calculating feature point matching relationships according to an exemplary embodiment of the present disclosure is shown.

[0121] like Figure 8 As shown, a feature point matching network based on scene structure preservation may include a 3D point cloud generator, a 3D position encoder, and a SuperGlue network.

[0122] like Figure 9 As shown, firstly, a 3D point cloud converter transforms the feature points and their depths on the most similar keyframe images into 3D point clouds using the camera pose (coordinates) of the keyframe images. These high-dimensional features are then concatenated with descriptors via a 3D position encoder. Simultaneously, the feature points and their depths on the current frame image are transformed into 3D point clouds using the camera prior of that image frame to obtain a sub-global map (e.g., ...). Figure 2As shown, high-dimensional features and descriptors are connected together after being encoded by a 3D position encoder. If ORB feature points extracted from a SLAM system are used, their depth can be automatically obtained during the SLAM system's operation. If SuperPoint network is used to extract feature points and descriptors, the depth corresponding to the feature points is obtained through several adjacent frames using the Structure From Motion (SFM) algorithm based on their camera pose, thus transforming the feature points into a 3D point cloud to obtain a sub-global map. Then, inner edges are constructed between any feature points within an image, and outer edges are constructed between any two feature points between two images, thus constructing the input to the graph neural network. Attention mechanisms are used to aggregate between inner edges and between outer edges, enabling the selection of discriminative neighborhood structures between inner edges and potential matching relationships with visual similarity between outer edges. Finally, the weight matrices of the constructed inner and outer edges are calculated by the graph neural network as the criteria for judging the existence of each edge, thereby obtaining the feature point matching relationship between the two images.

[0123] In an exemplary embodiment of this disclosure, when calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image, the camera pose of the current frame image can be calculated using the local bundle adjustment method or the global bundle adjustment method based on the feature point matching relationship between the current frame image and the key frame image.

[0124] Figure 10 A flowchart illustrating an instantaneous localization and mapping method according to another exemplary embodiment of the present disclosure is shown.

[0125] Reference Figure 10 In step S1001, the current frame image input through the camera is acquired.

[0126] In step S1002, the current frame image is tracked.

[0127] In step S1003, it is determined whether the tracking was successful. Tracking of the current frame image can be achieved through either of the following two methods: Method 1: Extract ORB feature points and corresponding descriptors from the current frame image, compare them with the descriptors of the previous frame image, and obtain the feature point matching relationship between the two frames; Method 2: Using the feature points of the previous frame image, find the corresponding feature points in the current frame using optical flow, and filter out incorrect feature point matches using descriptors to obtain the feature point matching relationship between the two frames. If the number of matching feature points obtained in the current frame image is less than a set threshold, feature points and descriptors will be extracted again in the current frame image.

[0128] In step S1004, if tracking is successful, the tracking status between the current frame image and at least the previous frame image is obtained. When determining the tracking status between two frames, the number of tracked feature points in the current frame image and the number of newly extracted feature points are compared. If the number of tracked feature points is less than a given value or the number of newly extracted feature points exceeds a given value, then the current frame image is used as the keyframe image.

[0129] In step S1005, based on the tracking state between the current frame image and at least the previous frame image, it is determined whether to use the current frame image as a keyframe image. This can be determined by comparing the tracking states between the previous and next frame images. If selected as a keyframe image, the current frame image will be added to the global map for use in subsequent local and global bundle adjustment, as well as loop closure detection and relocalization modules.

[0130] In step S1006, when it is determined that the current frame image will not be used as a keyframe image, the camera pose of the current frame image is calculated using local bundle adjustment based on the feature point matching relationship between the current frame image and at least the previous frame image. The camera pose and 3D coordinates of the feature points corresponding to each of the latest input frames can be updated using local bundle adjustment based on the feature point matching relationship between them.

[0131] In step S1007, when it is determined that the current frame image will be used as a keyframe image, the current frame image is added to the global map. Based on the feature point matching relationship between all keyframe images in the global map, the camera pose of each keyframe image is calculated using the global bundle adjustment method. The camera coordinates corresponding to each keyframe image can be calculated using the global bundle adjustment method based on the feature point matching relationship between all keyframe images.

[0132] In step S1008, the camera pose of non-keyframe images is updated based on the camera pose of each keyframe image. This allows for updating the camera pose of non-keyframe images in local bundle adjustment using relative poses, achieving data unification between local bundle adjustment and global bundle adjustment.

[0133] In step S1009, if tracking fails, scene recognition is performed on the current frame image to obtain the keyframe image with the highest similarity in the global map. The global descriptor of the current frame image is obtained using an IOU-related image depth feature extraction model. The current frame image is input into the IOU-related image depth feature extraction model to obtain its global descriptor. The most similar keyframe image to the current frame image is queried in the global map. The global descriptors corresponding to each keyframe image stored in the global map are extracted using the IOU-related image depth feature extraction model. The current frame image is then compared to the global descriptors corresponding to all keyframe images to find the global descriptor with the smallest distance, which is then used as the most similar global descriptor, thus obtaining the most similar keyframe image for the current frame image, which is used as the common-view frame image.

[0134] In step S10010, the feature point matching relationship between the current frame image and the keyframe image with the highest similarity to the current frame image in the global map is obtained. Feature points and descriptors on the current frame image and its co-view frames can be extracted using the SuperPoint network. Based on their camera poses, the depth corresponding to the feature points is obtained through several adjacent frames using the SFM algorithm, thus converting the feature points into a 3D point cloud. Alternatively, feature points can be extracted and the 3D point cloud restored without using the SuperPoint network. ORB feature points used during SLAM operation can also be used, as the corresponding 3D point clouds have already been restored during SLAM system operation, eliminating the need for additional SFM algorithm calculations of the depth corresponding to the feature points.

[0135] In step S10011, the camera pose of the current frame image is calculated based on the feature point matching relationship between the current frame image and the keyframe image. First, the 3D point cloud can be encoded using a 3D position encoder. The encoded result and descriptor are then fed into the SuperGlue network as input to obtain the feature point matching relationship under scene structure constraints. The feature point matching relationship between the current frame image and the co-view frame image can be used to recover the camera coordinates of the current input image, and can also be used as a constraint in calculating the camera coordinates of the current frame image in local bundle adjustment and global bundle adjustment methods.

[0136] Figure 11 A flowchart illustrating an instantaneous localization and mapping method according to another exemplary embodiment of the present disclosure is shown.

[0137] Reference Figure 11 In step S1101, the current frame image input through the camera is acquired. Specifically, in application scenarios such as autonomous driving, robot navigation, and augmented reality, the camera captures images in real time.

[0138] In step S1102, a keyframe image that is similar to the current frame image is determined. Specifically, the keyframe image that is similar to the current frame image can be determined by any method, and this disclosure does not impose any restrictions on this.

[0139] In step S1103, the feature point matching relationship between the current frame image and the key frame image is obtained.

[0140] In an exemplary embodiment of this disclosure, when obtaining the feature point matching relationship between the current frame image and the key frame image, the local three-dimensional point cloud of the current frame image and the global three-dimensional point cloud of the key frame image can be obtained first. The scene structure of the current frame image and the key frame image can be obtained based on the local three-dimensional point cloud of the current frame image and the global three-dimensional point cloud of the key frame image. Then, the feature point matching relationship between the current frame image and the key frame image can be obtained based on the scene structure of the current frame image and the key frame image.

[0141] In an exemplary embodiment of this disclosure, when acquiring a local three-dimensional point cloud of the current frame image, the two-dimensional feature points and depth values ​​of the current frame image can be acquired first, and then the pose prior can be used to convert the two-dimensional feature points and depth values ​​of the current frame image into a local three-dimensional point cloud of the current frame image.

[0142] In an exemplary embodiment of this disclosure, when acquiring the global three-dimensional point cloud of the keyframe image, the two-dimensional feature points and depth values ​​of the keyframe image can be converted into a global three-dimensional point cloud based on the camera pose of the keyframe image.

[0143] In an exemplary embodiment of this disclosure, when obtaining the feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image, the local 3D point cloud and the global 3D point cloud can be encoded first by a 3D position encoder to obtain local high-dimensional features and global high-dimensional features. Then, the feature point descriptors of the current frame image and the key frame image are obtained. The local high-dimensional features and global high-dimensional features, as well as the feature point descriptors of the current frame image and the key frame image, are input into the feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

[0144] In step S1104, the camera pose of the current frame image is calculated based on the feature point matching relationship between the current frame image and the key frame image.

[0145] In an exemplary embodiment of this disclosure, when calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image, the camera pose of the current frame image can be calculated using the local bundle adjustment method or the global bundle adjustment method based on the feature point matching relationship between the current frame image and the key frame image.

[0146] Furthermore, according to exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed, implements the real-time positioning and mapping method according to exemplary embodiments of the present disclosure.

[0147] In an exemplary embodiment of this disclosure, the computer-readable storage medium may carry one or more programs that, when executed, can perform the following steps: acquiring a current frame image input through a camera; performing scene recognition on the current frame image to obtain a keyframe image with the highest similarity to the current frame image in a global map; acquiring a feature point matching relationship between the current frame image and the keyframe image; and calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the keyframe image.

[0148] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a computer program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof. A computer-readable storage medium can be included in any apparatus; it can also exist independently without being assembled into that apparatus.

[0149] Furthermore, according to exemplary embodiments of the present disclosure, a computer program product is also provided, wherein the instructions in the computer program product can be executed by a processor of a computer device to perform an instantaneous localization and mapping method according to exemplary embodiments of the present disclosure.

[0150] The above has been combined Figures 1 to 11An exemplary embodiment of the present disclosure has been described. Hereinafter, reference will be made to... Figures 12 to 14 An instantaneous positioning and mapping apparatus and its components according to exemplary embodiments of the present disclosure will be described.

[0151] Figure 12 A block diagram of an instantaneous positioning and mapping apparatus according to an exemplary embodiment of the present disclosure is shown.

[0152] Reference Figure 12 The real-time positioning and mapping device includes an image acquisition unit 121, a scene recognition unit 122, and a pose calculation unit 123.

[0153] The image acquisition unit 121 is configured to acquire the current frame image input through the camera.

[0154] The scene recognition unit 122 is configured to perform scene recognition on the current frame image and obtain the key frame image with the highest similarity to the current frame image in the global map.

[0155] In an exemplary embodiment of this disclosure, the scene recognition unit 122 may include: a descriptor acquisition unit (not shown), configured to acquire a global descriptor of the current frame image; and a keyframe query unit (not shown), configured to query a keyframe image with the highest similarity to the current frame image in a global map based on the global descriptor of the current frame image.

[0156] In an exemplary embodiment of this disclosure, the keyframe query unit may be configured to: calculate the distance between the global descriptor of each keyframe image in the global map and the global descriptor of the current frame image; and determine the similarity between the global descriptor of the keyframe image and the global descriptor of the current frame image based on the distance between the global descriptor of the keyframe image and the global descriptor of the current frame image. Here, the closer the global descriptor of the keyframe image is to the global descriptor of the current frame image, the greater the similarity between the keyframe image and the current frame image.

[0157] In an exemplary embodiment of this disclosure, the descriptor acquisition unit may be configured to acquire a global descriptor of the current frame image through an image depth feature extraction network based on Intersection over Union (IOU).

[0158] In an exemplary embodiment of this disclosure, the apparatus may further include an extraction network training unit (not shown), configured to: acquire anchor images and positive sample images from training images; train an IOU-based image depth feature extraction network based on the anchor images, positive sample images, and the similarity between the anchor images and positive sample images. When training the IOU-based image depth feature extraction network, the IOU score between the anchor images and positive sample images can be used as the target similarity; feature extraction is performed on the anchor images and positive sample images using an online convolutional neural network and a target convolutional neural network, respectively, to obtain anchor features and positive sample features; the anchor features and positive sample features are transformed from the feature space to the metric space using a multilayer perceptron; the negative sample features of the current training iteration are determined based on the positive sample features from previous training iterations; the cosine distance between the anchor features, positive sample features, and negative sample features is calculated as the similarity; based on the similarity and target similarity, the loss and gradient are calculated using a cross-entropy loss function to backpropagate and update the online convolutional neural network; and the parameters θ of the online convolutional neural network are used using an exponentially weighted average algorithm. Online For the parameters θ of the target convolutional neural network Target Update the network; when the online convolutional neural network reaches the preset termination condition, use the online convolutional neural network and the target convolutional neural network as an image depth feature extraction network based on IOU.

[0159] In an exemplary embodiment of this disclosure, the network training unit may be configured to: randomly sample anchor images from training images; randomly sample the similarity between anchor images and positive sample images; and sample images from training images that have a similarity to the anchor images equal to the obtained similarity as positive sample images.

[0160] In an exemplary embodiment of this disclosure, the network training unit can be configured such that: in the first training iteration, the negative sample features are zero; and in each subsequent training iteration, the negative sample features in the current training iteration are determined based on the positive and negative sample features from the previous training iteration.

[0161] In an exemplary embodiment of this disclosure, the similarity between the anchor image and the positive sample image can be IOU.

[0162] In an exemplary embodiment of this disclosure, the network training extraction unit may also be configured to: after obtaining anchor images and positive sample images from training images, perform a projective transformation on one or more of the anchor images and positive sample images, and use the projective transformed anchor images and positive sample images as anchor images and positive sample images, respectively.

[0163] The pose calculation unit 123 is configured to determine the camera pose of the current frame image based on the keyframe image.

[0164] In an exemplary embodiment of this disclosure, the pose calculation unit 123 may include a matching relationship acquisition unit and a pose calculation subunit. The matching relationship acquisition unit may be configured to acquire the feature point matching relationship between the current frame image and the keyframe image. The pose calculation subunit may be configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the keyframe image.

[0165] In an exemplary embodiment of this disclosure, the matching relationship acquisition unit may include: a 3D point cloud acquisition unit configured to acquire a local 3D point cloud of the current frame image and a global 3D point cloud of the key frame image; a scene structure acquisition unit configured to obtain a scene structure of the current frame image and the key frame image based on the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; and a feature point matching unit configured to obtain a feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image.

[0166] In an exemplary embodiment of this disclosure, the 3D point cloud acquisition unit is configured to: acquire two-dimensional feature points and depth values ​​of the current frame image; and convert the two-dimensional feature points and depth values ​​of the current frame image into a local 3D point cloud of the current frame image using pose priors.

[0167] In an exemplary embodiment of this disclosure, the three-dimensional point cloud acquisition unit may be configured to: convert the two-dimensional feature points and depth values ​​of the keyframe image into a global three-dimensional point cloud based on the camera pose of the keyframe image.

[0168] In an exemplary embodiment of this disclosure, the feature point matching unit may be configured to: encode the local 3D point cloud and the global 3D point cloud respectively through a 3D position encoder to obtain local high-dimensional features and global high-dimensional features; obtain the feature point descriptor of the current frame image and the feature point descriptor of the key frame image, and input the local high-dimensional features and global high-dimensional features, as well as the feature point descriptor of the current frame image and the feature point descriptor of the key frame image, into the feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

[0169] In an exemplary embodiment of this disclosure, the pose calculation subunit may be configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image using local bundle adjustment or global bundle adjustment.

[0170] Figure 13 A block diagram of an instantaneous positioning and mapping apparatus according to another exemplary embodiment of the present disclosure is shown.

[0171] Reference Figure 13The real-time positioning and mapping device 1300 includes an image acquisition unit 131, an image tracking unit 132, a tracking status acquisition unit 133, a key frame determination unit 134, a first calculation unit 135, a second calculation unit 136, a pose update unit 137, a key frame search unit 138, a matching relationship acquisition unit 139, and a third calculation unit 1310.

[0172] The image acquisition unit 131 is configured to acquire the current frame image input through the camera.

[0173] The image tracking unit 132 is configured to track the current frame image.

[0174] The tracking state acquisition unit 133 is configured to acquire the tracking state between the current frame image and at least the previous frame image when tracking is successful.

[0175] The keyframe determination unit 134 is configured to determine whether to use the current frame image as a keyframe image based on the tracking state between the current frame image and at least the previous frame image.

[0176] The first computing unit 135 is configured to calculate the camera pose of the current frame image by means of local bundle adjustment when it is determined that the current frame image is not to be used as a keyframe image.

[0177] The second calculation unit 136 is configured to add the current frame image to the global map when it is determined that the current frame image is to be used as a key frame image, and calculate the camera pose of each key frame image by means of the global bundle adjustment method based on the feature point matching relationship between all key frame images in the global map.

[0178] The pose update unit 137 is configured to update the camera pose of non-keyframe images based on the camera pose of each keyframe image.

[0179] The keyframe search unit 138 is configured to perform scene recognition on the current frame image in the event of tracking failure, and obtain the keyframe image with the highest similarity to the current frame image in the global map.

[0180] The matching relationship acquisition unit 139 is configured to acquire the feature point matching relationship between the current frame image and the key frame image.

[0181] The third calculation unit 1310 is configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image.

[0182] Figure 14 A block diagram of an instantaneous positioning and mapping apparatus according to another exemplary embodiment of the present disclosure is shown.

[0183] Reference Figure 14 The real-time positioning and mapping device includes an image acquisition unit 141, a key frame selection unit 142, a matching relationship acquisition unit 143, and a pose calculation subunit 144.

[0184] The image acquisition unit 141 is configured to acquire the current frame image input through the camera.

[0185] The keyframe selection unit 142 is configured to determine the keyframe image that has the greatest similarity to the current frame image.

[0186] The matching relationship acquisition unit 143 is configured to acquire the feature point matching relationship between the current frame image and the key frame image.

[0187] In an exemplary embodiment of this disclosure, the matching relationship acquisition unit 143 may include: a three-dimensional point cloud acquisition unit configured to acquire a local three-dimensional point cloud of the current frame image and a global three-dimensional point cloud of the key frame image; a scene structure acquisition unit configured to obtain a scene structure of the current frame image and the key frame image based on the local three-dimensional point cloud of the current frame image and the global three-dimensional point cloud of the key frame image; and a feature point matching unit configured to obtain a feature point matching relationship between the current frame image and the key frame image based on the scene structure of the current frame image and the key frame image.

[0188] In an exemplary embodiment of this disclosure, the 3D point cloud acquisition unit may be configured to: acquire two-dimensional feature points and depth values ​​of the current frame image; and convert the two-dimensional feature points and depth values ​​of the current frame image into a local 3D point cloud of the current frame image using pose priors.

[0189] In an exemplary embodiment of this disclosure, the three-dimensional point cloud acquisition unit may be configured to: convert the two-dimensional feature points and depth values ​​of the keyframe image into a global three-dimensional point cloud based on the camera pose of the keyframe image.

[0190] In an exemplary embodiment of this disclosure, the feature point matching unit may be configured to: encode the local 3D point cloud and the global 3D point cloud respectively through a 3D position encoder to obtain local high-dimensional features and global high-dimensional features; obtain the feature point descriptor of the current frame image and the feature point descriptor of the key frame image, and input the local high-dimensional features and global high-dimensional features, as well as the feature point descriptor of the current frame image and the feature point descriptor of the key frame image, into the feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

[0191] The pose calculation subunit 144 is configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image.

[0192] In an exemplary embodiment of this disclosure, the pose calculation subunit 144 may be configured to calculate the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image using local bundle adjustment or global bundle adjustment.

[0193] The above has been combined Figures 12 to 14 An instantaneous localization and mapping apparatus according to exemplary embodiments of the present disclosure has been described. Next, in conjunction with... Figure 15 A computing device according to exemplary embodiments of the present disclosure will be described.

[0194] Figure 15 A schematic diagram of a computing device according to an exemplary embodiment of the present disclosure is shown.

[0195] Reference Figure 15 The computing device 15 according to an exemplary embodiment of the present disclosure includes a memory 151 and a processor 152. The memory 151 stores a computer program that, when executed by the processor 152, implements an instantaneous localization and mapping method according to an exemplary embodiment of the present disclosure.

[0196] In an exemplary embodiment of this disclosure, when the computer program is executed by the processor 152, the following steps can be implemented: acquiring a current frame image input through a camera; performing scene recognition on the current frame image to obtain a key frame image with the highest similarity to the current frame image in a global map; acquiring the feature point matching relationship between the current frame image and the key frame image; and calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image.

[0197] The computing device in this disclosure may include, but is not limited to, devices such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 15 The computing device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0198] The above has been referred to Figures 1 to 15 A real-time localization and mapping method and apparatus according to exemplary embodiments of the present disclosure are described. However, it should be understood that: Figures 12 to 14 The real-time positioning and mapping device and its units shown can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. Figure 15 The computing device shown is not limited to the components shown above, but some components may be added or removed as needed, and the above components may also be combined.

[0199] The real-time localization and mapping method and apparatus according to exemplary embodiments of the present disclosure acquires a current frame image input through a camera; performs scene recognition on the current frame image to obtain a keyframe image with the highest similarity to the current frame image in a global map; and determines the camera pose of the current frame image based on the keyframe image. Thus, for images captured from the camera, both traditional algorithms can be used to extract feature points and descriptors for tracking, followed by calculation of the camera pose using bundle adjustment; and deep learning models can be used for loop closure detection and relocalization. The method selects the most similar keyframe image from a global map composed of keyframe images for the current frame image captured from the camera, and determines the camera pose of the current frame image based on the keyframe image.

[0200] The real-time localization and mapping method and apparatus according to exemplary embodiments of the present disclosure acquires a current frame image input through a camera; determines a key frame image with similarity to the current frame image; acquires a feature point matching relationship between the current frame image and the key frame image; and calculates the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the key frame image. Thus, for images captured from the camera, both traditional algorithms can be used to extract feature points and descriptors for tracking, followed by calculation of the camera pose using bundle adjustment; and deep learning models can be used for loop closure detection and relocalization. The method determines the most similar key frame image to the current frame image captured from the camera and calculates the feature point matching relationship (i.e., data association) between the current frame image and the determined key frame image.

[0201] Furthermore, the real-time localization and mapping method and apparatus according to the exemplary embodiments of this disclosure can obtain a model with better generalization ability by utilizing an unsupervised deep learning model, while improving the expressive power of deep features through an IOU-related image deep feature extraction method, so as to find better similar images more accurately, and find the feature point matching relationship between the current frame image and the queried key frame image through a scene structure-based data association model, so as to recover the camera pose of the current frame image.

[0202] Although this disclosure has been specifically shown and described with reference to exemplary embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. A real-time localization and mapping method, comprising: Acquire the current frame image input through the camera; Perform scene recognition on the current frame image to obtain the keyframe image with the highest similarity to the current frame image in the global map; as well as The camera pose of the current frame image is determined based on the keyframe images. The step of determining the camera pose of the current frame image based on the keyframe image includes: Obtain the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; Based on the local 3D point cloud of the current frame image and the global 3D point cloud of the keyframe image, obtain the feature point matching relationship between the current frame image and the keyframe image; and The camera pose of the current frame image is calculated based on the feature point matching relationship between the current frame image and the key frame image. The step of obtaining the feature point matching relationship between the current frame image and the keyframe image includes: The local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image are encoded by a 3D position encoder to obtain the local high-dimensional features of the current frame image and the global high-dimensional features of the key frame image. The feature point descriptors of the current frame image and the key frame image are obtained. The local high-dimensional features of the current frame image and the global high-dimensional features of the key frame image, as well as the feature point descriptors of the current frame image and the key frame image, are input into the feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

2. The method according to claim 1, wherein, The steps for scene recognition in the current frame image include: Get the global descriptor of the current frame image; Based on the global descriptor of the current frame image, search for the keyframe image with the highest similarity to the current frame image in the global map.

3. The method according to claim 2, wherein, The steps for querying the keyframe image with the highest similarity to the current frame image in the global map based on the global descriptor of the current frame image include: Calculate the distance between the global descriptor of each keyframe image in the global map and the global descriptor of the current frame image; The similarity between the global descriptors of the keyframe image and the current frame image is determined based on the distance between the global descriptors of the keyframe image and the global descriptors of the current frame image. The closer the global descriptor of the keyframe image is to the global descriptor of the current frame image, the greater the similarity between the keyframe image and the current frame image.

4. The method according to claim 1, wherein, The step of obtaining the feature point matching relationship between the current frame image and the keyframe image further includes: The scene structure of the current frame image and the key frame image is obtained based on the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image. The feature point matching relationship between the current frame image and the key frame image is obtained based on the scene structure of the current frame image and the key frame image.

5. The method according to claim 1, wherein, The steps to obtain a local 3D point cloud of the current frame image include: Obtain the 2D feature points and depth values ​​of the current frame image; and Based on pose prior, the two-dimensional feature points and depth values ​​of the current frame image are converted into a local three-dimensional point cloud of the current frame image.

6. The method according to claim 1, wherein, The steps for obtaining the global 3D point cloud of the keyframe image include: Based on the camera pose of the keyframe image, the two-dimensional feature points and depth values ​​of the keyframe image are converted into a global three-dimensional point cloud.

7. The method according to claim 1, wherein, The steps for calculating the camera pose of the current frame image based on the feature point matching relationship between the current frame image and the keyframe image include: The camera pose of the current frame image is calculated using either local bundle adjustment or global bundle adjustment based on the feature point matching relationship between the current frame image and the key frame image.

8. The method according to claim 2, wherein, The steps to obtain the global descriptor of the current frame image include: The global descriptor of the current frame image is obtained by an image depth feature extraction network based on Intersection over Union (IOU).

9. The method according to claim 8, further comprising: Train an image deep feature extraction network based on IOU. The steps for training the image deep feature extraction network based on IOU include: Obtain anchor images and positive sample images from training images; The image deep feature extraction network based on IOU is trained based on the anchor image, the positive sample image, and the similarity between the anchor image and the positive sample image.

10. The method according to claim 9, wherein, The steps to obtain anchor images and positive sample images from training images include: Anchor point images are obtained by randomly sampling from training images; Random sampling is used to obtain the similarity between anchor point images and positive sample images; Images obtained by sampling the similarity between the training images and the anchor images are used as positive sample images.

11. The method according to claim 9 or 10, wherein the similarity between the anchor image and the positive sample image is IOU.

12. The method according to claim 9, after obtaining the anchor image and the positive sample image from the training image, further comprising: Perform a projective transformation on one or more of the anchor point image and the positive sample image.

13. A real-time localization and mapping method, comprising: Acquire the current frame image input through the camera; Identify the keyframe image that has the greatest similarity to the current frame image; Obtain the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image; Based on the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image, obtain the feature point matching relationship between the current frame image and the key frame image; The camera pose of the current frame image is calculated based on the feature point matching relationship between the current frame image and the keyframe image. The steps for obtaining the local 3D point cloud of the current frame image include: obtaining the 2D feature points and depth values ​​of the current frame image; and converting the 2D feature points and depth values ​​of the current frame image into a local 3D point cloud of the current frame image based on pose prior. The step of obtaining the feature point matching relationship between the current frame image and the keyframe image includes: The local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image are encoded by a 3D position encoder to obtain the local high-dimensional features of the current frame image and the global high-dimensional features of the key frame image. The feature point descriptors of the current frame image and the key frame image are obtained. The local high-dimensional features of the current frame image and the global high-dimensional features of the key frame image, as well as the feature point descriptors of the current frame image and the key frame image, are input into the feature point matching network to obtain the feature point matching relationship between the current frame image and the key frame image.

14. The method according to claim 13, wherein, The step of obtaining the feature point matching relationship between the current frame image and the keyframe image further includes: The scene structure of the current frame image and the key frame image is obtained based on the local 3D point cloud of the current frame image and the global 3D point cloud of the key frame image. The feature point matching relationship between the current frame image and the key frame image is obtained based on the scene structure of the current frame image and the key frame image.

15. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the real-time positioning and mapping method according to any one of claims 1 to 14.

16. A computing device, comprising: At least one processor; At least one memory storing a computer program that, when executed by at least one processor, implements the real-time positioning and mapping method according to any one of claims 1 to 14.

Images