SLAM method, system, device and storage medium based on feature matching network

Abstract

The present invention discloses a SLAM method, system, device, and storage medium based on a feature matching network. A feature extraction network originally designed for matching tasks is trained using the SLAM operating mode. The trained matching-task-based feature extraction network is then embedded into the SLAM system framework, optimizing the existing SLAM system framework and improving the robustness of the designed SLAM system, enabling it to effectively cope with the challenges of various SLAM scenarios while ensuring positioning and mapping accuracy. Furthermore, the trained matching-task-based feature extraction network is used to extract feature points from images captured by a monocular camera. This extraction process only requires a monocular camera, thereby reducing the cost of smart vehicles. The above-mentioned solution has low overall cost, does not require modification to the environment or the vehicle itself, and is highly portable.

Description

Technical Field

[0001] The present invention relates to the fields of intelligent driving technology and vehicles, and in particular to a SLAM method, system, device and storage medium based on a feature matching network. Background Art

[0002] With the advancement of technology, vehicles, robots, and other technological products are becoming increasingly intertwined with people's lives. Autonomous driving technology is also maturing. However, the main challenge facing autonomous driving today lies in ensuring good positioning and navigation performance in complex scenarios, such as those involving changing lighting and rapid rotation. Consequently, SLAM (Simultaneous Localization and Mapping) systems in complex scenarios have become a key research direction in the field. The foundation for this is the design of more robust and stable features. Commonly used hand-crafted features offer advantages in certain scenarios, but they still have drawbacks. The ORB (Oriented FAST and Rotated BEIEF) feature is extremely fast but lacks rotation and scale invariance. The SIFT (Scale Invariant Feature Transform) feature demonstrates satisfactory performance in a variety of complex scenarios, but struggles to meet real-time requirements. In recent years, a large number of feature extraction networks have emerged that outperform hand-crafted features. However, these networks are mostly designed for image matching tasks, and simply applying these feature extraction networks to SLAM systems often yields mediocre performance. Specifically, unlike the matching task for 2D-2D matching, the SLAM system needs to complete the mapping work at the same time. The SLAM system usually uses the matching of map points (3D points) and image points (2D points), and then estimates the posture by optimizing the reprojection error. Therefore, the requirements of the feature-based SLAM system are essentially different from the matching task, resulting in some deep learning-based visual features that cannot be fully and effectively adapted to the SLAM system. Therefore, designing a SLAM method based on a feature matching network can effectively utilize the advanced achievements in the field of deep learning and greatly contribute to the development of autonomous driving technology. On the other hand, if autonomous driving technology can be completed using a lower-cost monocular camera, the technology cost will be greatly reduced. However, the problem that comes with it is the decline in image quality, which makes feature extraction difficult.

[0003] Currently, the main technical solutions are as follows:

[0004] 1) The patent "Loop Detection and Visual SLAM Method Based on Optical Flow Tracking and Feature Matching" (Publication No.: CN113888603A) utilizes binocular cameras, uses optical flow to collect point features, and the LSD algorithm to extract line features, achieving better results than ORB-SLAM2. However, binocular cameras are relatively expensive.

[0005] 2) The patent "A visual SLAM positioning method for weak texture and dynamic scenes based on RGBD cameras" (Publication No.: CN113379842A) uses RGBD cameras to perceive the environment and achieves good results in weak texture and dynamic scenes. However, RGB-D cameras are generally expensive, and the solution does not address challenging scenarios such as large field of view changes.

[0006] 3) The patent "A method and related device for realizing SLAM positioning based on monocular vision" (publication number: CN111928842A) uses the brisk operator to extract feature points of each frame of at least two frames, and then matches the feature points of adjacent images to further realize subsequent positioning and mapping work. However, the brisk operator lacks good rotation invariance and scale invariance. In complex scenes, it is easy for the two frames of images to not match, resulting in tracking failure.

[0007] In summary, the main problems with the current solution are as follows:

[0008] 1) How to reduce sensor costs while ensuring high-quality positioning accuracy.

[0009] 2) Feature point matching is difficult in complex scenes, and tracking failure is likely to occur.

[0010] 3) How to effectively apply feature extraction networks to ensure high-quality positioning accuracy. Summary of the Invention

[0011] The purpose of the present invention is to provide a SLAM method, system, device and storage medium based on a feature matching network, which fully considers the requirements of cost and accuracy and has the advantages of strong robustness, low cost and high accuracy.

[0012] The purpose of the present invention is achieved through the following technical solutions:

[0013] A SLAM method based on a feature matching network, comprising:

[0014] Using deep learning methods, the feature extraction network based on matching tasks is trained through the SLAM operation mode;

[0015] Embed the trained matching task-based feature extraction network into the SLAM system framework;

[0016] The trained matching-based feature extraction network extracts feature points from images captured by a monocular camera and matches the corresponding feature points of two adjacent frames. The corresponding rotation matrix and translation vector are solved for the matched feature point pairs to locate the vehicle. The position of the object point in the world coordinate system is obtained through triangulation to complete the mapping task.

[0017] A SLAM system based on a feature matching network, the system comprising:

[0018] A network training unit is used to train a feature extraction network based on a matching task using a deep learning method through a SLAM operation mode;

[0019] A SLAM system framework update unit, used to embed the trained matching task-based feature extraction network into the SLAM system framework;

[0020] The positioning and mapping unit is used to extract feature points from images taken by a monocular camera using a trained matching-based feature extraction network, match the corresponding feature points of two adjacent frames of the extracted images, and solve the corresponding rotation matrix and translation vector for the matched feature point pairs to locate the vehicle. The position of the object point in the world coordinate system is obtained through triangulation to complete the mapping task.

[0021] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0023] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0024] As can be seen from the technical solution provided by the present invention, a feature extraction network originally designed for matching tasks is trained through the SLAM operating mode, and the trained matching task-based feature extraction network is embedded into the SLAM system framework, thereby optimizing the original SLAM system framework and improving the robustness of the designed SLAM system, enabling it to effectively cope with the challenges of various SLAM scenarios and ensuring positioning and mapping accuracy. In addition, feature point extraction is performed from images captured by a monocular camera using the trained matching task-based feature extraction network. Since only a monocular camera is required for this purpose, the cost of the smart car is reduced. The above solution has a low overall cost, does not require modification to the environment or the vehicle itself, and has strong portability. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 A flow chart of a SLAM method based on a feature matching network provided in an embodiment of the present invention;

[0027] Figure 2 A flowchart of training a feature extraction network based on a matching task provided by an embodiment of the present invention;

[0028] Figure 3 The following is a workflow diagram of the SLAM system framework provided by the embodiment of the present invention;

[0029] Figure 4 A schematic diagram of test results on the Hptaches dataset, a matching task dataset, provided by an embodiment of the present invention;

[0030] Figure 5 A schematic diagram of test results on the outdoor dataset KITTI dataset provided by an embodiment of the present invention;

[0031] Figure 6 A schematic diagram of test results on the Euroc indoor dataset provided by an embodiment of the present invention;

[0032] Figure 7 Schematic diagram of a SLAM system based on a feature matching network provided by an embodiment of the present invention

[0033] Figure 8 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0035] First, the following terms may be used in this article:

[0036] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0037] The following describes in detail a SLAM method, system, device, and storage medium based on a feature matching network provided by the present invention. Any details not described in the embodiments of the present invention are prior art known to those skilled in the art. In the embodiments of the present invention, if specific conditions are not specified, the conditions are carried out according to conventional conditions in the art or the conditions recommended by the manufacturer.

[0038] Example 1

[0039] Since both the image matching task and the feature-based SLAM system rely on feature points to complete pixel association and camera pose estimation, they provide important support and foundation for the perception and fusion of unmanned vehicles. In order to overcome the problem of feature tracking failure and reduce the gap between the matching task and the SLAM system, an embodiment of the present invention provides a SLAM method based on a feature matching network. By referring to the operating mode of the SLAM system, an effective training strategy is involved to train the feature extraction network based on the matching task, so that the trained feature extraction network based on the matching task can better adapt to the SLAM system, thereby helping autonomous driving technology. Figure 1 FIG. 1 is a flow chart of a SLAM method based on a feature matching network provided by an embodiment of the present invention, which mainly includes the following steps:

[0040] Step 1: Use deep learning methods to train the feature extraction network based on the matching task through the SLAM operation mode.

[0041] like Figure 2 As shown, the preferred implementation of this step is as follows:

[0042] 1) Build a feature extraction network based on the matching task and select a training dataset containing RGB images and corresponding depth images.

[0043] Exemplarily, in an embodiment of the present invention, the feature extraction network based on the matching task can adopt a twin network, specifically a SuperPoint network, which is called a Rein-SuperPoint network after training.

[0044] For example, the fr2_pioneer_slam sequence in the TUM dataset can be used as training data.

[0045] 2) For each RGB image in the training dataset, an image matching pair is formed by randomly using adjacent images; the two images in an image matching pair are called the first image and the second image.

[0046] In an embodiment of the present invention, for the k-th RGB image in the training data set, the v-th RGB image is randomly selected from the next three frames to form an image matching pair; where v∈{k+1,k+2,k+3}; the k-th RGB image is called the first image, and the v-th RGB image is called the second image.

[0047] For example, 1500 image matching pairs may be selected and the order of these image matching pairs may be disrupted.

[0048] 3) For each image matching pair, input it into the feature extraction network based on the matching task to obtain a heat map pair, denoted as H1 and H2; randomly select n key points on the heat maps H1 and H2 respectively; perform nearest neighbor matching on the key points on the heat maps H1 and H2 to obtain a preliminary point matching result M, and randomly select point matching pairs from the preliminary matching result M according to the set ratio. The selected point matching pair set is denoted as M s .

[0049] like Figure 2 As shown in the figure, a twin network structure is adopted, and each single image in the image matching pair is input into a feature extraction network based on the matching task; the two feature extraction networks based on the matching task share network parameters.

[0050] For example, n may be set to 1000, and the ratio may be set to 50%, that is, half of the point matching pairs are selected.

[0051] 4) Point matching pair set M s Each point matching pair (p i ,p′ j ), where p i ∈H1,p′ j ∈H2,p i =(x i ,y i ),p′ j =(x′ j ,y′ j ), indicating p i and p′ j are the key points on the heat maps H1 and H2, (x i ,y i ) represents the key point p iThe position coordinates in the image coordinate system, (x′ j ,y′ j ) represents the key point p′ j Position coordinates in the image coordinate system; obtain the key point p from the corresponding depth image according to the frame number of the first image i Depth d pi , and then use the camera's intrinsic parameter matrix K contained in the training data set to obtain the key point p i Position coordinates (x i ,y i ) corresponds to the position point P of the first image in the camera coordinate system i The 3D coordinates (X i ,Y i ,Z i ), and combined with the annotation information in the training data set to determine the true pose of the first image and the second image, obtain the position point P i The corresponding second image position point P in the camera coordinate system j The 3D coordinates (X j ,Y j ,Z j ); Then, the position point P is converted to j The 3D coordinates (X j ,Y j ,Z j ) is projected onto the image coordinate system of the second image to obtain the corresponding position point p j The position coordinates (x j ,y j ).

[0052] Those skilled in the art will understand that if a dataset includes an RGB image and a corresponding depth image, it means that the dataset is collected using an RGB-D camera, and the dataset includes the intrinsic parameter matrix K of the RGB-D camera.

[0053] This stage is for single point matching pairs (p i ,p′ j ) is introduced, and all other point matching needs to follow the steps in the above introduction; the main steps are described as follows:

[0054] First, according to the sequence number of the frame (the sequence numbers of the first image and the second image are recorded as k and v here), the key point p can be obtained from the depth map of the corresponding sequence number. i Depth d pi , and then use the camera's intrinsic parameter matrix K to get the key point p i Position coordinates (x i ,y i ) corresponds to the position point P of the first image in the camera coordinate system iThe 3D coordinates (X i ,Y i ,Z i ) is expressed as:

[0055]

[0056]

[0057]

[0058] Among them, (c x ,c y ) is the offset of the optical axis from the center of the projection plane coordinate, (f x ,f y ) are the focal lengths of the camera in the x and y directions.

[0059] Then, the true pose T of the first image and the second image can be obtained from the groundtruth (annotation information) of the TUM dataset. wk and T wv , thereby calculating the transformation matrix T between them vk :

[0060]

[0061] in, Indicates T wv The inverse matrix of .

[0062] Then use the transformation matrix T vk , the position point P can be calculated i The corresponding second image position point P in the camera coordinate system j The 3D coordinates (X j ,Y j ,Z j ):

[0063] P j =T vk P i .

[0064] Finally, the position point P is converted to j The 3D coordinates (X j ,Y j ,Z j ) is projected onto the image coordinate system of the second image to obtain the corresponding position point p j The position coordinates (x j ,y j ), expressed as:

[0065]

[0066] 5) Calculate the position coordinates (x j ,y j ) and (x′ j ,y′ j ) as the distance between the point matching pairs (p i ,p′ j ) matching error, comprehensive point matching pair set M s The loss function is constructed based on the matching errors of all point matching pairs, and the feature extraction network based on the matching task is trained.

[0067] The matching point position coordinates (x j ,y j ), which is often the coordinate of the matching point obtained by nearest neighbor matching (x′ j ,y′ j ) are inconsistent. The distance between the two position coordinates is calculated by the following formula as the point matching pair (p i ,p′ j ) matching error;

[0068]

[0069] Point matching pair set M s Calculate the matching errors of all point matching pairs and construct the following loss function:

[0070]

[0071] Based on the above loss function, the network parameters of the feature extraction network based on the matching task are updated through back propagation. The processes involved in this part can be implemented with reference to conventional technologies, so they will not be described in detail.

[0072] The above describes the loss function calculation scheme for a batch size of 1. In actual applications, the batch size (for example, 1, 2, 4, 8, etc.) will be designed based on computing resources. Taking the 1500 image matching pairs provided in the previous article as an example, the loss function is calculated by taking the corresponding number of image matching pairs from the designed batch size until the adjustment is stopped (for example, the number of iterations reaches the set maximum number or the loss function converges).

[0073] The above training method provided by the embodiment of the present invention is inspired by the operation mode of SLAM, which trains the network by optimizing the projection error from map points (3D points) to image points (2D). In the posture tracking process of the SLAM system, the SLAM system generally estimates an initial posture transformation T by assuming uniform velocity. vk , then follow the above steps, p i We can get its corresponding point p in H2j , then the SLAM system is based on p j and p′ j (p i Optimize T based on the distance between matching pairs obtained by nearest neighbor matching) vk , and obtain the final pose estimate.

[0074] Step 2: Embed the trained matching task-based feature extraction network into the SLAM system framework.

[0075] In an embodiment of the present invention, for a monocular camera, the checkerboard calibration method is used to calibrate the internal parameters of the camera to obtain the camera's intrinsic parameter matrix and distortion coefficient vector d, where K is a 3×3 matrix and d is a 4×1 vector, and the camera's calibration parameters are stored; the monocular camera is installed on the vehicle. Since autonomous driving technology is based on the vehicle body coordinate system, the transformation from the camera coordinate system to the vehicle body coordinate system is calibrated and solved, and subsequent processes are all based on the camera coordinate system.

[0076] In the embodiment of the present invention, the SLAM system framework can use the ORB SLAM2 framework, and the feature extraction network based on the matching task obtained by training in the above steps is embedded into the SLAM system framework.

[0077] Step 3: The trained matching-based feature extraction network is used to extract feature points from the image captured by the monocular camera, and the corresponding feature points of the two adjacent frames of the extracted image are matched. The corresponding rotation matrix and translation vector are solved for the matched feature point pairs to locate the vehicle, and the position of the object point in the world coordinate system is obtained through triangulation to complete the mapping task.

[0078] In this embodiment, a matching-based feature extraction network is used to extract feature points (keypoints and descriptors) from images captured by a monocular camera, replacing the ORB feature points used in the original SLAM system framework. Euclidean distance is used to calculate the matching error during feature matching. The remaining positioning and mapping solutions are essentially the same as in the existing process.

[0079] As mentioned previously, the SLAM system framework can use the ORB SLAM2 framework. ORB-SLAM2 primarily consists of three threads: tracking, local mapping, and loop closure detection. These threads are responsible for pose estimation and positioning, building a sparse environment map, and detecting loop closures and optimizing trajectories, respectively. The specific implementation methods of these three threads can be referenced in conventional techniques.

[0080] Figure 3The main process of the SLAM system framework is demonstrated; the feature extraction network based on the matching task obtained through training is used to extract feature points from the image taken by the monocular camera and match the corresponding feature points of two adjacent frames. If the SLAM is in the initialization stage, the matched 2D features are triangulated to obtain their corresponding map (3D) points, and a connection is established with the key frame to optimize the pose (rotation matrix and translation matrix) estimation; if the SLAM is in the pose tracking state, the map points corresponding to the features of the previous frame are matched with the next frame, and a connection is established with the key frame to optimize the pose estimation, thereby obtaining the vehicle's position information and completing vehicle positioning. The local mapping thread runs simultaneously to build a sparse map. The closed-loop detection thread is responsible for optimizing the closed-loop trajectory and system pose. Specifically, through closed-loop detection and global BA (Bundle Ajustment), the cumulative error of positioning and mapping is reduced, further improving the accuracy of the SLAM system.

[0081] The above-mentioned solution provided by the embodiment of the present invention improves the robustness of the designed SLAM system by training the feature extraction network originally for the matching task in the SLAM operation mode, so that it can well cope with the challenges of various SLAM scenarios; further optimizes the positioning and mapping framework in SLAM, and designs a SLAM system with better performance (that is, the original feature extraction network is replaced by the feature extraction network based on the matching task trained in the above steps); the above-mentioned solution has low overall cost, does not require modification of the environment and the vehicle itself, and has strong portability.

[0082] In order to illustrate the effect of the above solution of the embodiment of the present invention, experiments were conducted to verify it.

[0083] The test results on the matching task dataset Hptaches dataset are as follows Figure 4 The left side shows the matching results of the original SuperPoint for feature points in the three image groups, and the right side shows the matching results of the Rein-SuperPoint for feature points in the three image groups. In each image group, two matching points connected by a dark line indicate an incorrect match, and two matching points connected by a light line indicate a correct match. It can be clearly seen that Rein-SuperPoint achieves better matching results under varying illumination and large field of view.

[0084] The test results on the outdoor dataset KITTI dataset are as follows Figure 5 As shown. Figure 5As shown in the left part, on the KITTT01 sequence, SP-SLAM (SLAM system using the original SuperPoint integration) will lose tracking in the last part of the sequence, while Rein-SLAM (SLAM system framework using the Rein-SuperPoint integration) can complete the positioning and mapping of the entire sequence. Figure 5 As shown in the right part, on the KITTI09 sequence, Rein-SLAM achieved significantly better results than SP-SLAM.

[0085] The test results on the indoor dataset Euroc dataset are as follows Figure 6 As shown in the figure, the left part and the right part respectively represent the test results on the MH_01 sequence and the MH_04 sequence. Figure 6 From the test results shown, it can be seen that Rein-SLAM achieved significantly better results than SP-SLAM in both the MH_01 sequence and the MH_04 sequence.

[0086] It should be noted that the specific network type (SuperPoint) provided in the above text description and related drawings is only an example and does not constitute a limitation. In actual applications, users can choose other types of feature matching networks according to actual conditions or needs.

[0087] Example 2

[0088] The present invention also provides a SLAM system based on a feature matching network, which is mainly implemented based on the method provided in the above embodiment 1. Figure 7 As shown, the system mainly includes:

[0089] A network training unit is used to train a feature extraction network based on a matching task using a deep learning method through a SLAM operation mode;

[0090] A SLAM system framework update unit, used to embed the trained matching task-based feature extraction network into the SLAM system framework;

[0091] The positioning and mapping unit is used to extract feature points from images taken by a monocular camera using a trained matching-based feature extraction network, match the corresponding feature points of two adjacent frames of the extracted images, and solve the corresponding rotation matrix and translation vector for the matched feature point pairs to locate the vehicle. The position of the object point in the world coordinate system is obtained through triangulation to complete the mapping task.

[0092] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0093] It should be noted that the relevant technical details involved in each unit in the above system have been introduced in detail in the previous embodiment 1, so they will not be repeated here.

[0094] Example 3

[0095] The present invention also provides a processing device, such as Figure 8 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0096] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0097] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0098] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0099] The output device may be a display terminal;

[0100] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0101] Example 4

[0102] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0103] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0104] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A SLAM method based on feature matching network, characterized in that, include: Using deep learning methods, the feature extraction network based on matching tasks is trained through the SLAM operation mode; Embed the trained matching task-based feature extraction network into the SLAM system framework; The trained matching-based feature extraction network extracts feature points from images captured by a monocular camera and matches the corresponding feature points of two adjacent frames. The corresponding rotation matrix and translation vector are solved for the matched feature point pairs to locate the vehicle. The position of the object point in the world coordinate system is obtained through triangulation to complete the mapping task. The step of using a deep learning method to train a feature extraction network for a matching task through a SLAM operation mode includes: Build a feature extraction network based on the matching task and select a training dataset containing RGB images and corresponding depth images; For each RGB image in the training dataset, an image matching pair is formed by randomly using adjacent images; the two images in an image matching pair are called the first image and the second image; For each image matching pair, input it into the feature extraction network based on the matching task to obtain a heat map pair, which is recorded as H1 and H2; randomly select n key points on the heat maps H1 and H2 respectively; perform nearest neighbor matching on the key points on the heat maps H1 and H2 to obtain the preliminary point matching result M, and randomly select point matching pairs from the preliminary point matching result M according to the set ratio. The selected point matching pair set is recorded as M s ; Point matching pair set M s A single point matching pair (p i ,p′ j ), where p i ∈H1,p′ j ∈H2,p i =(x i ,y i ),p′ j =(x′ j ,y′ j ), indicating p i and p′ j are the key points on the heat maps H1 and H2, (x i ,y i ) represents the key point p i The position coordinates in the image coordinate system, (x′ j ,y′ j ) represents the key point p′ j Position coordinates in the image coordinate system; obtain the key point p from the corresponding depth image according to the frame number of the first image i Depth Then, with the help of the camera’s intrinsic parameter matrix K contained in the training data set, the key point p is obtained. i Position coordinates (x i ,y i ) corresponds to the position point P of the first image in the camera coordinate system i The 3D coordinates (X i ,Y i ,Z i ), and combined with the annotation information in the training data set to determine the true pose of the first image and the second image, obtain the position point P i The corresponding second image position point P in the camera coordinate system j The 3D coordinates (X j ,Y j ,Z j ); Then, the position point P is converted to j The 3D coordinates (X j ,Y j ,Z j ) is projected onto the image coordinate system of the second image to obtain the corresponding position point p j The position coordinates (x j ,y j ); Calculate the position coordinates (x j ,y j ) and (x′ j ,y′ j ) as the distance between the point matching pairs (p i ,p′ j ) matching error, comprehensive matching pair set M s The loss function is constructed based on the matching errors of all point matching pairs, and the feature extraction network based on the matching task is trained.

2. A SLAM method based on a feature matching network according to claim 1, characterized in that, For each RGB image frame in the training data set, randomly using adjacent images to form an image matching pair includes: For the k-th RGB image in the training dataset, randomly select the v-th RGB image from the next three frames to form an image matching pair; where v∈{k+1,k+2,k+3}; the k-th RGB image is called the first image, and the v-th RGB image is called the second image.

3. A SLAM method based on a feature matching network according to claim 1 or 2, characterized in that, The key point p is obtained by using the camera's intrinsic parameter matrix K i Position coordinates (x i ,y i ) corresponds to the position point P of the first image in the camera coordinate system i The 3D coordinates (X i ,Y i ,Z i ) is expressed as: Among them, (c x ,c y ) is the offset of the optical axis from the center of the projection plane coordinate, (f x ,f y ) are the focal lengths of the camera in the x and y directions.

4. A SLAM method based on a feature matching network according to claim 1 or 2, characterized in that, The real poses of the first and second images are determined by combining the annotation information in the training data set to obtain the position point P i The corresponding second image position point P in the camera coordinate system j The 3D coordinates (X j ,Y j ,Z j )include: Determine the true pose T of the first image and the second image based on the annotation information in the training data set wk and T wv , calculate the transformation matrix T between the image matching pairs vk : in, Indicates T wv The inverse matrix of Using the transformation matrix T vk , calculate the position point P i The corresponding second image position point P in the camera coordinate system j The 3D coordinates (X j ,Y j ,Z j ): P j =T vk P i 。 5. A SLAM method based on a feature matching network according to claim 1 or 2, characterized in that, The camera's intrinsic parameter matrix K is used to convert the position point P j The 3D coordinates (X j ,Y j ,Z j ) is projected onto the image coordinate system of the second image to obtain the corresponding position point p j The position coordinates (x j ,y j ) is expressed as:

6. A SLAM method based on a feature matching network according to claim 1 or 2, characterized in that, The position coordinates (x j ,y j ) and (x′ j ,y′ j ) as the distance between the point matching pairs (p i ,p′ j ) matching error, comprehensive matching pair set M s The matching error of all point matching pairs in constructing the loss function includes: Calculate the position coordinates (x j ,y j ) and (x′ j ,y′ j ) distance d ij′ , as a point matching pair (p i ,p′ j )’s matching error: Point matching pair set M s Calculate the matching errors of all point matching pairs and construct the following loss function:

7. A SLAM system based on a feature matching network, characterized in that: The method according to any one of claims 1 to 6 is implemented, and the system comprises: A network training unit is used to train a feature extraction network based on a matching task using a deep learning method through a SLAM operation mode; A SLAM system framework update unit, used to embed the trained matching task-based feature extraction network into the SLAM system framework; The positioning and mapping unit is used to extract feature points from images taken by a monocular camera using a trained matching-based feature extraction network, match the corresponding feature points of two adjacent frames of the extracted images, and solve the corresponding rotation matrix and translation vector for the matched feature point pairs to locate the vehicle. The position of the object point in the world coordinate system is obtained through triangulation to complete the mapping task.

8. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images