A stable feature point recognition method for visual SLAM system

Abstract

The present invention relates to the technical field of feature point recognition, and more specifically, to a stable feature point recognition method for a visual SLAM system, comprising the following steps: S1. constructing a training sample; S2. establishing a prediction neural network and using the training sample for pre-training, and then predicting the image to be processed through the prediction neural network. The present invention can retain highly repeatable visual feature points for irregular environmental changes, thereby reducing memory consumption and erroneous data association, and improving the robustness of mobile robot visual navigation.

Description

Technical Field

[0001] The present invention relates to the technical field of feature point recognition, and more specifically, to a stable feature point recognition method for a visual SLAM system. Background Art

[0002] Visual sensors have the characteristics of large amount of information and low cost, and play an important role in many mobile robot applications, such as visual SLAM system, scene recognition, visual navigation and positioning, etc. Among them, visual SLAM system and visual navigation are the two key technologies to realize the autonomy of mobile robots.

[0003] At present, in order to achieve visual navigation and positioning of mobile robots, most solutions will first use visual SLAM system technology to build a map of the environment, then extract the visual feature points of the current frame image and establish correct data association with the feature points in the map through descriptor matching and RANSAC elimination to achieve the positioning function. Due to the limitations of traditional visual feature point detection methods, changes in the environment make many feature points only be repeatedly detected and matched in a short time span. Even after just a few hours, the number of feature points that can be correctly associated will be greatly reduced. Most of the existing visual SLAM system solutions based on feature point methods do not consider the subsequent changes in illumination, object movement, and vegetation growth in the environment during operation. The generated map will contain a large number of feature points that become unstable over time. For robot applications that need to perform routine tasks in the same area for a long time, such as inspection and cleaning, this kind of map often introduces a large number of false associations in the navigation and positioning process. These false associations will cause the number of iterations for RANSAC to find the correct hypothesis to become longer, and if the error ratio exceeds a certain percentage, RANSAC will also fail, causing the robot to be unable to work reliably. In addition, when mapping large scene environments, storing unstable feature points and their descriptors often consumes a lot of memory space, which is also a huge limitation for mobile robots with limited resources.

[0004] The prior art discloses a method and device for tracking the trajectory of a capsule gastroscope, which comprises: obtaining a sequence of video images taken by a capsule gastroscope; inputting each frame of the video image sequence into a trained image feature extraction model in turn to obtain image feature points and corresponding feature descriptors of each frame of the image; and tracking the trajectory of the capsule gastroscope through the ORB SLAM3 algorithm according to the image feature points and the feature descriptors. The present invention combines a deep learning network with the ORB SLAM3 algorithm to achieve matching and tracking of image features through the ORBSLAM3 algorithm on the basis of extracting feature vectors using a convolutional neural network, thereby improving the robustness of the extracted features, thereby achieving precise positioning of the trajectory tracking of the capsule gastroscope.

[0005] However, the above scheme cannot be used to identify stable feature points of the visual SLAM system. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a stable feature point recognition method for a visual SLAM system, which can retain highly repeatable visual feature points for irregular environmental changes, thereby reducing memory consumption and erroneous data association, and improving the robustness of mobile robot visual navigation.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A stable feature point recognition method for a visual SLAM system is provided, comprising the following steps:

[0009] S1. Constructing training samples; wherein step S1 includes the following steps:

[0010] S11. Selecting an image dataset with known image poses, and selecting a plurality of image sequences from the image dataset;

[0011] S12. Extracting multiple feature points from all images in each image sequence;

[0012] S13. Selecting an image sequence having a separated perspective from the image sequence of extracted feature points;

[0013] S14. performing feature point matching on the image sequence from which feature points have been extracted by reprojection, obtaining stable feature points in the image with separated viewing angles, and randomly selecting an equal number of unstable feature points to form training samples;

[0014] S2. Establish a prediction neural network and use the training samples for pre-training, and then predict the image to be processed through the prediction neural network; wherein step S2 includes the following steps:

[0015] S21. Establishing a prediction neural network, and then using the training samples obtained in step S1 for pre-training; wherein the prediction neural network has a feature extraction module and a logic judgment module;

[0016] S22. Extracting multiple feature points from the image to be processed;

[0017] S23. Selecting an image region of the image to be processed from which feature points have been extracted, and then inputting the region into the feature extraction module to obtain a first feature vector;

[0018] S24. Standardizing the descriptor of the feature point to obtain a second feature vector;

[0019] S25. Concatenate the first feature vector and the second feature vector, and then input the concatenated feature vector into the logic judgment module to obtain a possible expression of the stability of the feature point.

[0020] Further, in step S11, the time span of the image sequence is greater than or equal to 90 days.

[0021] Furthermore, the step S13 specifically includes the following steps:

[0022] S131. Select one of the image sequences from which feature points have been extracted, and select multiple images at a certain time interval therein to obtain a first separated perspective image sequence;

[0023] S132. According to each image in the first separated-view image sequence, select the nearest view in the remaining image sequences from which feature points have been extracted one by one to obtain the corresponding remaining separated-view image sequences.

[0024] Furthermore, in step S132, the conditions for selecting the nearest viewing angle are met: the viewing angle of the image is less than or equal to 1 meter from the viewing angle of the image in the first separated viewing angle image sequence, and the yaw angle is between 0 and 30°, and in the same separated viewing angle image sequence, if there is more than one image that meets the conditions for selecting the nearest viewing angle corresponding to the same image in the first separated viewing angle image sequence, then the image with the closest viewing angle is selected.

[0025] Furthermore, the step S14 specifically includes the following steps:

[0026] S141. Sparsely reconstruct each image sequence from which feature points have been extracted, so that each feature point in each separated view image sequence obtains corresponding three-dimensional coordinates;

[0027] S142. Randomly select a separated view image sequence, and project the feature points with three-dimensional coordinates therein to the images of the corresponding view in the remaining separated view image sequences through the camera model to form projection points, and then search for matching feature points in the images of the corresponding view through the KD tree. If no matching feature points that meet the matching requirements can be found in the images of the corresponding view in the remaining separated view image sequences, execute step S143, otherwise execute step S144;

[0028] S143. Mark the feature point as an unstable feature point, and then execute step S145;

[0029] S144. If at least one matching feature point that meets the matching requirement can be found on the images of the corresponding perspectives in the remaining separated perspective image sequences, the feature point is marked as a stable feature point, otherwise the feature point is not marked; then step S145 is executed;

[0030] S145. Repeat steps S142 to S144 until all images in the separated view image sequence have completed feature point marking, and then execute step S146;

[0031] S146. According to the number of obtained stable feature points, an equal number of unstable feature points are randomly selected to form training samples.

[0032] Further, in step S142, the matching requirement is that the distance between the projection point and the feature point to be matched is less than 5 pixels, and the distance between the feature point descriptor vectors is less than 0.7 threshold.

[0033] Furthermore, in step S21, the feature extraction module is a ResNet18 network without a fully connected layer, and the logic judgment module is an MLP network consisting of two fully connected layers.

[0034] Furthermore, the step S23 specifically includes: extracting an image area (R) around the feature points from the image to be processed from which the feature points have been extracted. 64×64×3 ), and then input it into the ResNet18 network to obtain a 512-dimensional feature vector.

[0035] Furthermore, the step S24 specifically includes: performing Z-Score normalization processing on each dimension of the descriptor of the feature point to obtain a 128-dimensional feature vector.

[0036] Furthermore, the step S25 specifically includes: concatenating the 512-dimensional feature vector and the 128-dimensional feature vector to obtain a 640-dimensional feature vector, and then inputting the 640-dimensional feature vector into the MLP network, and obtaining a possibility representation of feature point stability through logistic regression.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention provides a stable feature point recognition method for a visual SLAM system, which can retain highly repeatable visual feature points for irregular environmental changes during the mapping process of the SLAM system, thereby reducing memory consumption and erroneous data association, and improving the robustness of the visual navigation of the mobile robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a stable feature point recognition method for a visual SLAM system according to the present invention;

[0040] Figure 2 The present invention is a flow chart of step S2 of a method for identifying stable feature points in a visual SLAM system. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with specific implementation methods. The accompanying drawings are only used for exemplary descriptions and are only schematic diagrams, not actual drawings, and cannot be understood as limiting this patent; in order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the accompanying drawings may be omitted.

[0042] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the drawings, it is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limitations on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0043] Example 1

[0044] like Figure 1 The first embodiment of the present invention is a stable feature point recognition method for a visual SLAM system, comprising the following steps:

[0045] S1. Constructing training samples; wherein step S1 includes the following steps:

[0046] S11. Selecting an image dataset with known image poses, and selecting multiple image sequences in the image dataset;

[0047] S12. Extracting multiple feature points from all images in each image sequence;

[0048] S13. Selecting an image sequence having a separated perspective from the image sequence of extracted feature points;

[0049] S14. Perform feature point matching on the image sequence from which feature points have been extracted by reprojection, obtain stable feature points in the image with separated viewing angles, and randomly select an equal number of unstable feature points to form training samples.

[0050] S2. Establish a prediction neural network and use the training samples for pre-training, and then predict the image to be processed through the prediction neural network; wherein step S2 includes the following steps:

[0051] S21. Establish a prediction neural network, and then use the training samples obtained in step S1 for pre-training; wherein the prediction neural network has a feature extraction module and a logic judgment module;

[0052] S22. Extracting multiple feature points from the image to be processed;

[0053] S23. Selecting an image region of the image to be processed from which feature points have been extracted, and then inputting the image region into a feature extraction module to obtain a first feature vector;

[0054] S24. Standardizing the descriptors of the feature points to obtain a second feature vector;

[0055] S25. Concatenate the first feature vector and the second feature vector, and then input the concatenated feature vector into a logic judgment module to obtain a possible expression of the stability of the feature point.

[0056] The present invention provides a stable feature point recognition method for a visual SLAM system, which can retain highly repeatable visual feature points for irregular environmental changes during the mapping process of the SLAM system, thereby reducing memory consumption and erroneous data association, and improving the robustness of the visual navigation of the mobile robot.

[0057] Example 2

[0058] This embodiment is similar to Embodiment 1, except that, in this embodiment, step S1 specifically includes the following steps:

[0059] S11. Select an image dataset with known image poses, and select multiple image sequences from the image dataset; wherein the time span of the image sequence is greater than or equal to 90 days; the setting of greater than 90 days allows the selected image sequence to have seasonal changes.

[0060] S12. Extracting multiple SIFT feature points from all images in each image sequence;

[0061] S13. Selecting an image sequence with a separated viewing angle from the image sequence from which feature points have been extracted; step S13 specifically comprises the following steps:

[0062] S131. Select one of the image sequences from which feature points have been extracted, and select multiple images at a certain time interval therein to obtain a first separated perspective image sequence;

[0063] S132. According to each image in the first separated perspective image sequence, the nearest perspective is selected one by one in the remaining image sequences from which feature points have been extracted, to obtain the corresponding remaining separated perspective image sequences; wherein, the conditions for satisfying the nearest perspective selection are: the perspective of the image is less than or equal to 1 meter from the perspective of the image in the first separated perspective image sequence, and the yaw angle is between 0 and 30°, and in the same separated perspective image sequence, if there is more than one image satisfying the nearest perspective selection conditions corresponding to the same image in the first separated perspective image sequence, then an image with a smaller perspective distance and a smaller yaw angle is selected.

[0064] S14. Perform feature point matching on the image sequence from which feature points have been extracted by reprojection, obtain stable feature points in the image with separated viewing angles, and randomly select an equal number of unstable feature points to form a training sample; step S14 specifically includes the following steps:

[0065] S141. Sparsely reconstruct each image sequence from which feature points have been extracted, so that each feature point in each separated view image sequence obtains corresponding three-dimensional coordinates;

[0066] S142. Randomly select a separated perspective image sequence, and project the feature points with three-dimensional coordinates therein to the images of the corresponding perspectives in the remaining separated perspective image sequences through the camera model to form projection points, and then search for matching feature points in the images of the corresponding perspectives through the KD tree; if no matching feature points that meet the matching requirements can be found on the images of the corresponding perspectives in the remaining separated perspective image sequences, execute step S143, otherwise execute step S144; wherein the matching requirements are: the distance between the projection point and the feature point to be matched is less than 5 pixels, and the distance between the feature point descriptor vectors is less than 0.7 threshold value.

[0067] S143. Mark the feature point as an unstable feature point, and then execute step S145;

[0068] S144. If at least one matching feature point that meets the matching requirement can be found on the images of the corresponding perspectives in the remaining separated perspective image sequences, the feature point is marked as a stable feature point, otherwise the feature point is not marked; then step S145 is executed;

[0069] S145. Repeat steps S142 to S144 until all images in the separated view image sequence have completed feature point marking, and then execute step S146;

[0070] S146. According to the number of obtained stable feature points, an equal number of unstable feature points are randomly selected to form training samples.

[0071] Specifically, the image dataset selected in step S11 is the CMU Seasons image dataset, which includes 12 image sequences, which are a series of images taken by a vehicle-mounted camera at 12 different times, from the center of Pittsburgh (USA) to the suburbs, and acquired in sequence. The 12 image sequences obtained have different degrees of environmental changes, including lighting, weather, seasons, etc., and also have real image poses. In this embodiment, three of the image sequences are selected for the construction of training samples, numbered as the first image sequence, the second image sequence, and the third image sequence; and the selected three image sequences need to meet the cross-season time span, that is, greater than 90 days.

[0072] Specifically, in step S12, 2000 SIFT feature points are extracted from the first image sequence, the second image sequence, and the third image sequence.

[0073] Specifically, in step S13, a certain number of images are selected from the first image sequence at the same time interval according to the real position of the image to obtain a first separated perspective image sequence; for each image in the first separated perspective image sequence, the image with the closest viewing angle distance is found in the second image sequence and the third image sequence one by one, respectively, to obtain the corresponding second separated perspective image sequence and the third separated perspective image sequence; it should be noted that the viewing angle condition that the selected image with the closest viewing angle distance needs to meet is: an image with a viewing angle less than or equal to 1 meter from the image in the first separated perspective image sequence and a yaw angle between 0 and 30°. For each viewing angle, it is necessary to simultaneously include one image from each of the three sequences of the first, second, and third separated perspective image sequences. If for a certain image in the first separated perspective image sequence, an image that meets the viewing angle condition cannot be found in the second image sequence or the third image sequence, then the image needs to be deleted from the first separated perspective image sequence; when the number of images in the first separated perspective image sequence is small, then the first image sequence is reselected at another time interval to obtain a new first separated perspective image sequence.

[0074] Specifically, in step S14, SFM is used to perform sparse reconstruction on the first image sequence, the second image sequence, and the third image sequence to obtain the three-dimensional coordinates corresponding to each feature point; the operation of SFM in this embodiment includes: image feature point extraction, image matching, and triangulation; to ensure that all images are in the same coordinate system, the real position of the image is used in this embodiment to triangulate the matched feature points to obtain the three-dimensional coordinates corresponding to the feature points. Next, the feature points with three-dimensional coordinates in the first separated perspective image sequence are projected onto the images of the corresponding perspectives in the second and third separated perspective image sequences through the camera model to form projection points, and then the KD tree is used to quickly search in the two images of the corresponding perspectives to find out whether there are matching feature points that meet the matching requirements:

[0075] If at least one matching feature point can be found on the images of the corresponding perspectives in the second and third separated perspective image sequences, the feature point on the image in the first separated perspective image sequence is marked as a stable feature point;

[0076] If no matching feature point can be found on the images of the corresponding perspectives in the second and third separated perspective image sequences, the feature point on the image in the first separated perspective image sequence is marked as an unstable feature point;

[0077] If it does not belong to the above two situations, the feature point will not be marked.

[0078] After all feature points in the first separated view image sequence are marked, all feature points with three-dimensional coordinates in the second separated view image sequence are projected into the images of the first and third separated view image sequences through the camera model for matching feature point search, and then stable feature points and unstable feature points are obtained in the second separated view image sequence. Next, all feature points with three-dimensional coordinates in the third separated view image sequence are projected into the images of the first and second separated view image sequences through the camera model for matching feature point search, and stable feature points and unstable feature points are also obtained; at this time, the marking of stable feature points and unstable feature points is completed. Finally, in order to generate training samples with balanced categories, an equal number of unstable feature points are randomly selected according to the number of stable feature points obtained to form the final training samples.

[0079] Example 3

[0080] This embodiment is similar to Embodiment 2, except that step S2 in this embodiment specifically includes the following steps:

[0081] S21. Establish a prediction neural network, and then use the training samples obtained in step S1 for pre-training; wherein the prediction neural network has a feature extraction module and a logic judgment module; in this embodiment, the feature extraction module is a ResNet18 network without a fully connected layer, and the logic judgment module is an MLP network composed of two fully connected layers. It should be noted that the initial parameters of the ResNet18 network have been pre-trained on the ImageNet dataset, and in order to reduce the amount of training parameters, the fully connected layer of the ResNet18 network is deleted in this embodiment.

[0082] S22. Extract K SIFT feature points from the image to be processed. In this embodiment, the image to be processed may be an image captured by a camera on the mobile robot during its movement.

[0083] S23. Extracting the image area around the feature points from the image to be processed from which the feature points have been extracted (R 64×64×3 ), and scaled to a size of 224×224 and then input into the ResNet18 network to obtain a first eigenvector. In this embodiment, the first eigenvector is a 512-dimensional eigenvector.

[0084] S24. Process each dimension of the descriptor of the SIFT feature point according to Z-Score normalization with a mean equal to 0 and a variance equal to 1 to obtain a second feature vector. In this embodiment, the second feature vector is a 128-dimensional feature vector.

[0085] S25. The 512-dimensional feature vector and the 128-dimensional feature vector are concatenated to obtain a 640-dimensional feature vector, and then the 640-dimensional feature vector is input into an MLP network consisting of two fully connected layers. Finally, the MLP network obtains a score expression in the range of 0 to 1 as a possible expression of the stability of the feature point through logistic regression. According to the annotation situation, when the stable feature point is marked as 1 and the unstable feature point is marked as 0, a threshold of 0.5 can be set at this time; if the output score is less than or equal to 0.5, the feature point is considered to be an unstable feature point; if the output score is greater than 0.5, the feature point is considered to be a stable feature point.

[0086] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A stable feature point recognition method for a visual SLAM system, characterized in that: The steps include: S1. Constructing training samples; wherein step S1 includes the following steps: S11. Selecting an image dataset with known image poses, and selecting a plurality of image sequences from the image dataset; S12. Extracting multiple feature points from all images in each image sequence; S13. Selecting an image sequence having a separated perspective from the image sequence of extracted feature points; S14. performing feature point matching on the image sequence from which feature points have been extracted by reprojection, obtaining stable feature points in the image with separated viewing angles, and randomly selecting an equal number of unstable feature points to form training samples; S2. Establish a prediction neural network and use the training samples for pre-training, and then predict the image to be processed through the prediction neural network; wherein step S2 includes the following steps: S21. Establishing a prediction neural network, and then using the training samples obtained in step S1 for pre-training; wherein the prediction neural network has a feature extraction module and a logic judgment module; S22. Extracting multiple feature points from the image to be processed; S23. Selecting an image region of the image to be processed from which feature points have been extracted, and then inputting the region into the feature extraction module to obtain a first feature vector; S24. Standardizing the descriptor of the feature point to obtain a second feature vector; S25. Concatenate the first feature vector and the second feature vector, and then input the concatenated feature vector into the logic judgment module to obtain a possible expression of the stability of the feature point.

2. The stable feature point identification method for a visual SLAM system according to claim 1, characterized in that: In step S11, the time span of the image sequence is greater than or equal to 90 days.

3. The stable feature point identification method for a visual SLAM system according to claim 1, characterized in that: The step S13 specifically includes the following steps: S131. Select one of the image sequences from which feature points have been extracted, and select multiple images at a certain time interval therein to obtain a first separated perspective image sequence; S132. According to each image in the first separated-view image sequence, select the nearest view in the remaining image sequences from which feature points have been extracted one by one to obtain the corresponding remaining separated-view image sequences.

4. The stable feature point identification method for a visual SLAM system according to claim 3, characterized in that: In step S132, the conditions for selecting the nearest viewing angle are: the viewing angle of the image is less than or equal to 1 meter from the viewing angle of the image in the first separated viewing angle image sequence, and the yaw angle is between 0 and 30°; and in the same separated viewing angle image sequence, if there is more than one image that meets the nearest viewing angle selection conditions corresponding to the same image in the first separated viewing angle image sequence, then the image with the closest viewing angle is selected.

5. The stable feature point identification method for a visual SLAM system according to claim 3 or 4, characterized in that: The step S14 specifically includes the following steps: S141. Sparsely reconstruct each image sequence from which feature points have been extracted, so that each feature point in each separated view image sequence obtains corresponding three-dimensional coordinates; S142. Randomly select a separated view image sequence, and project the feature points with three-dimensional coordinates therein to the images of the corresponding view in the remaining separated view image sequences through the camera model to form projection points, and then search for matching feature points in the images of the corresponding view through the KD tree. If no matching feature points that meet the matching requirements can be found in the images of the corresponding view in the remaining separated view image sequences, execute step S143, otherwise execute step S144; S143. Mark the feature point as an unstable feature point, and then execute step S145; S144. If at least one matching feature point that meets the matching requirement can be found on the images of the corresponding perspectives in the remaining separated perspective image sequences, the feature point is marked as a stable feature point, otherwise the feature point is not marked; then step S145 is executed; S145. Repeat steps S142 to S144 until all images in the separated view image sequence have completed feature point marking, and then execute step S146; S146. According to the number of obtained stable feature points, an equal number of unstable feature points are randomly selected to form training samples.

6. The stable feature point identification method for a visual SLAM system according to claim 5, characterized in that: In step S142, the matching requirement is that the distance between the projection point and the feature point to be matched is less than 5 pixels, and the distance between the feature point descriptor vectors is less than 0.7 threshold.

7. The stable feature point identification method for a visual SLAM system according to claim 1, characterized in that: In step S21, the feature extraction module is a ResNet18 network without a fully connected layer, and the logic judgment module is an MLP network consisting of two fully connected layers.

8. The stable feature point identification method for a visual SLAM system according to claim 7, characterized in that: The step S23 specifically includes: extracting the image area around the feature points from the image to be processed from which the feature points have been extracted. , and then input it into the ResNet18 network to obtain a 512-dimensional feature vector.

9. The stable feature point identification method for a visual SLAM system according to claim 8, characterized in that: The step S24 specifically includes: performing Z-Score normalization processing on each dimension of the descriptor of the feature point to obtain a 128-dimensional feature vector.

10. The stable feature point identification method for a visual SLAM system according to claim 9, characterized in that: The step S25 specifically includes: concatenating the 512-dimensional feature vector and the 128-dimensional feature vector to obtain a 640-dimensional feature vector, and then inputting the 640-dimensional feature vector into the MLP network to obtain a possibility representation of feature point stability through logistic regression.

Images