A key point selection method and system based on neural network

Abstract

The present invention discloses a feature point selection method and system based on a neural network. The present invention takes an image as input, and after data preprocessing, extracts a feature map from the image through a convolutional neural network, and then uses a fully connected neural network to extract feature points from the feature map. The present invention uses the constraints of posture and canonical phase to calculate the feature points on each image. Due to the introduction of posture constraints, the correlation between feature point extraction and posture estimation can be greatly improved, and the goal of accurate positioning can be achieved while selecting fewer feature points, greatly reducing the consumption of edge computing when executing SLAM programs. It solves the problem of how to select appropriate map points and reduce the number of feature points without affecting positioning accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of feature point matching, visual odometry, simultaneous positioning and mapping in the field of three-dimensional vision. This method is a method of selecting and extracting feature points using a neural network, which can improve the positioning accuracy of the visual odometry or SLAM system and reduce the amount of calculation. It can be used in VR / AR or the metaverse field in the future. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) system refers to a type of system that simultaneously completes mapping and positioning. SLAM system is commonly used in fields such as robots and drones. It can map the environment around the robot and realize the robot's self-positioning while mapping.

[0003] There are many different approaches and technologies for SLAM systems, including those based on sensors such as LiDAR, vision, and inertial measurement units (IMUs). LiDAR is a very useful technology with widespread application in a variety of real-world scenarios. For example, it can be used in devices such as sweeping robots to help them perceive their surroundings and plan paths. However, LiDAR is expensive, making it unsuitable for all applications. Furthermore, LiDAR can be affected by factors such as interference, which can affect its performance. Therefore, people are looking for alternative solutions to these problems. Visual SLAM is a widely researched alternative that has begun to gain public attention and finds increasing application. This approach, based on computer vision technology, can perform functions such as environmental perception and path planning by analyzing image and video data.

[0004] Feature point SLAM is a crucial technology within visual SLAM. Its basic principle is to estimate the robot's trajectory by calculating feature point matches between adjacent images and using these matches to construct a map of the robot's environment. This process can be divided into two main parts: the front end and the back end. The front end is primarily responsible for extracting feature points and using them to perform feature matching between adjacent frames to estimate the robot's trajectory. The back end, on the other hand, primarily uses optimization algorithms to refine the estimated trajectory and map, thereby improving positioning and mapping accuracy. The advantages of feature point SLAM are its high positioning and mapping accuracy. It can handle robot positioning and mapping in complex environments, and can achieve very high accuracy under good lighting conditions.

[0005] However, it also has some drawbacks, such as proneness to matching errors in complex environments like changing lighting and dynamic scenes, and slow processing speeds due to high computational effort. A large number of feature points can provide more pose constraints and improve accuracy, but this also significantly increases the computational effort. This is particularly evident on embedded devices with limited computing power. As map size increases, the large number of map points creates significant computational pressure, preventing the system from running in real time. Therefore, selecting appropriate map points and reducing the number of feature points without compromising positioning accuracy is a crucial and pressing issue. Summary of the Invention

[0006] To address the shortcomings of the existing technology, the present invention provides a neural network-based key point selection method and system. This method uses pose and canonical phase constraints to calculate the feature points on each image. The introduction of pose constraints significantly improves the correlation between feature point extraction and pose estimation, enabling precise positioning while selecting fewer feature points and significantly reducing the edge computing consumption when executing SLAM programs. This method solves the problem of how to select appropriate map points and reduce the number of feature points without affecting positioning accuracy.

[0007] The present invention takes an image as input and, after data preprocessing, extracts a feature map from the image using a convolutional neural network. Feature points are then extracted from the feature map using a fully connected neural network. The present invention uses the matching poses between the two images to supervise the selection of feature points. The feature points between the two images are matched, and then the pose between the two images is calculated using the essential matrix. The error between the poses is used as the loss to supervise the training. In the present invention, the error between the losses also includes the position regularization term of the feature points, the score and position regularization term of the feature points, and the regularization term of the descriptor.

[0008] In a first aspect, the present invention provides a feature point selection method based on a neural network, comprising the following steps:

[0009] Step (1) feature extraction based on feature extraction network;

[0010] In this stage, we use a convolutional neural network as the feature extraction network to extract feature maps from the input image. This convolutional neural network compresses the input image's width and height to a quarter of its original resolution, increasing the number of channels from 3 to 128. Compared to fully connected layers, convolutional neural networks require less computation, are faster, and can easily extract information from different channels. Therefore, at this stage, we can extract feature maps using minimal computing resources.

[0011] Step (2) feature point extraction based on fully connected neural network;

[0012] In this stage, we use an encoder to further compress the feature map, reducing the number of channels to 8. Due to the varying resolutions of the input images, an adaptive pooling layer is used to compress the feature map to a uniform size of 32x32x4. Subsequently, after converting the feature map into a 4096-dimensional vector, a fully connected neural network with 3 hidden layers is used to predict the feature points in each image. The output of the fully connected neural network is a 200x3 vector representing the coordinates and confidence levels of the 200 feature points. In this stage, we use convolutional neural networks and fully connected neural networks to extract the coordinates of the 200 feature points and their confidence levels from the image.

[0013] Step (3) Loss design based on homography transformation;

[0014] In this stage, we use homography to map the feature points obtained using the fast method into another space. The transformed feature points are then projected onto the original image. The feature points extracted from each space are combined as the true value of the feature point. This enriches the detected feature points and ensures a certain degree of homography invariance. Next, the relative pose between the two adjacent images is calculated by computing the orb descriptors corresponding to the feature points in the two adjacent images. The calculated relative pose is compared with the true value and the pose error is calculated. To ensure the quality of the feature points, feature point position and score regularization terms are added to the loss of the feature point selection network to further constrain feature point generation. The feature point selection network consists of the feature extraction network in step 1 and the fully connected neural network in step 2. In this stage, we design the loss of the entire neural network, using the pose between image matches and some regularization terms to constrain the feature point generation process.

[0015] Step (4) training the feature point selection network according to the designed loss;

[0016] During the network training process, the input image will first pass through the feature extraction network to obtain the feature map, and then extract the feature points through the fully connected neural network. After obtaining the feature points, the descriptor is calculated in combination with the feature map. Then, the PnP (Perspective-n-Point) algorithm and adjacent frames are used to calculate the pose of the image, and the loss is calculated with the true value of the pose to supervise the network training.

[0017] In a second aspect, the present invention provides a key point selection system based on a neural network, comprising a feature extraction module, a feature point extraction module, a homography transformation module, and a training module;

[0018] The feature extraction module uses a convolutional neural network as a feature extraction network to extract feature maps from the input image. The convolutional neural network compresses the input image's length and width to a quarter of the original resolution, increasing the number of channels from 3 to 128.

[0019] The feature point extraction module uses an encoder to further compress the feature map, reducing the number of channels to 8. Due to the varying resolutions of the input images, an adaptive pooling layer is used to compress the feature map to a uniform size of 32x32x4. After converting the feature map to a 4096-dimensional vector, a fully connected neural network with three hidden layers is used to predict the feature points in each image.

[0020] The homography transformation module uses homography transformation to map the feature points obtained by the fast method to another space, and then projects the transformed feature points onto the original image. The feature points extracted in each space are combined as the true value of the feature point.

[0021] The training module designs a loss based on the true value obtained by homography transformation; and trains the feature extraction module and the feature point extraction module according to the designed loss;

[0022] In a third aspect, the present invention provides a key point selection device based on a neural network, comprising a storage device and a processor, wherein the storage device is used to store a computer program, and the processor runs the computer program to enable the virtual wall setting device to perform the above-mentioned key point selection method based on the neural network.

[0023] In a fourth aspect, the present invention provides a storage medium storing a computer program used in the aforementioned key point selection device based on a neural network, wherein the computer program implements the aforementioned key point selection method based on a neural network when executed by a processor.

[0024] The beneficial effects of the present invention are as follows:

[0025] (1) The present invention uses homography transformation to map the feature points on the image to another space and then project them back to the source image as the true value of the feature points. This method helps to increase the true value of the feature points and also brings a certain degree of homography deformation to the feature points, thereby improving the robustness of the network in extracting feature points.

[0026] (2) The present invention introduces the pose of the image to constrain the selection of feature points. In previous work, feature point extraction and pose estimation were two separate processes. However, the feature points extracted by the feature point extraction network are not necessarily the most suitable combination for pose estimation, so it is difficult to directly evaluate the contribution of feature points to pose estimation. After introducing the pose error of the image into the loss function, this method directly and strongly associates the extraction of feature points with the trajectory of the pose, thereby ensuring that the extracted feature points contribute to pose estimation and that the entire process is differentiable, which is very important for updating the neural network.

[0027] (3) The present invention introduces regularization terms of feature point positions and confidence levels into the network. Due to the non-convexity of images, in image optimization problems, we need to find ways to avoid falling into local optimality. Therefore, when designing the loss function, we add regularization terms of feature point positions, hoping that feature points can be selected according to a trend of more uniform distribution in space. At the same time, we add regularization terms of feature point confidence levels, hoping to select feature points with higher confidence levels. The way we choose to add regularization terms to the loss function can guide feature points to the rules we choose, but not completely select them according to the rules of the loss function. This can add artificial constraints to the selection of feature points while retaining a certain degree of freedom, making the selection of feature points more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a diagram of the feature extraction network structure. The figure shows the structure of the feature extraction network and the specific structure of its residual block. For specific explanations, please refer to the subsequent text description;

[0029] Figure 2 It is a schematic diagram of extracting feature points through a fully connected neural network;

[0030] Figure 3 It is a schematic diagram of using homography transformation to obtain the complete truth value;

[0031] Figure 4 It is a flowchart of training feature point selection network to extract feature points. DETAILED DESCRIPTION

[0032] The following describes the specific implementation of the invention patent in conjunction with the drawings of this patent

[0033] A feature point selection method based on a neural network comprises the following steps:

[0034] Step (1) feature extraction phase based on feature extraction network;

[0035] In the feature extraction stage, we use convolutional neural network as the feature extraction network to extract features from the original input image. In order to ensure the effect of convolutional neural network, the convolutional neural network is composed of two residual blocks connected. The structure of the residual block is as follows Figure 1 As shown, the first residual block uses two convolutions with a kernel of 3x3 and a stride of 2 to reduce the dimension of the original input image to half of the original, and the number of channels increases from 3 channels to 64 channels, that is, 64 different feature maps are extracted from the image. The two convolutions are connected after a layer of Batch Norm and a layer of ReLu activation function. The second residual block structure is the same as the first residual block, reducing the dimension of the feature map output by the first residual block to half of the original, and reducing the number of channels from 64 channels to 128 channels. A total of two residual blocks are used to reduce the resolution of the image to one-quarter of the original, and increase the number of channels from the original 3 channels to 128 channels. Using a neural network with a residual structure can achieve better convergence, and using a convolutional network can significantly reduce the amount of computation while ensuring the quality of feature extraction.

[0036] Step (2) feature point extraction stage based on fully connected neural network;

[0037] After the feature extraction stage, the amount of data in the feature graph is still large. If it is directly put into the fully connected neural network, the amount of calculation will increase significantly, which will greatly affect the real-time performance of feature extraction. Therefore, Figure 2 As shown in the figure, at the beginning of this stage, an encoder is used to compress the feature map processed in the previous stage. The encoder uses a feature extraction network to compress the size of the feature map to one-eighth of the original size, and the number of channels becomes 8 channels. This network consists of a 3x3 convolutional layer with a step size of 1, a padding of 1, and a channel of 64, a 2x2 pooling layer with a step size of 2, and a 3x3 convolutional layer with a step size and padding of 1 and a channel of 8. Since the size and size of the input images are different, an adaptive pooling layer is used to unify the dimensions of the feature map output by the encoder into 32x32x4, which is convenient for the subsequent extraction of the fully connected layer. After the complete feature map is compressed and unified, use the following Figure 2 The fully connected layer shown extracts a feature vector of dimension 200x3. The 200 represent the 200 feature points extracted from the image. The first two dimensions represent the coordinates of the feature points on the image, and the last dimension represents the confidence level of the feature points. This allows us to easily obtain the feature points in the image and the confidence level of each feature point.

[0038] Step (3) calculates the true value using homography transformation;

[0039] Since artificial standards or automatically generated feature points may be incomplete, if the true value of the feature points in the data set is not complete, it will affect the subsequent training. Therefore, in this method, homography transformation is used to generate complete true value of feature points (see the detailed process for details). Figure 3 ), perform homography transformation on the original image and the feature map extracted by the feature extraction network, map the original image to different spaces, extract feature points in different spaces, and finally superimpose the feature points in all spaces and transform them back to the original space, thereby obtaining the complete true value of the feature points.

[0040] Step (4) neural network loss function design;

[0041] The feature point selection network loss function of the present invention consists of a feature point part and a pose part. The feature point part consists of a feature point position regularization term and a feature point confidence regularization term. The feature point position regularization term is as follows:

[0042]

[0043]

[0044]

[0045] The first formula is the value of the overall regularization term, Represents the regular term of the x-axis, represents the regularization term of the y-axis. The last two formulas are how to calculate the regularization term, where: It represents the x-axis coordinate of the i-th feature point. represents the y-axis coordinate of the i-th feature point, M is the number of feature points in the image, and i is the index of the feature point. The last two formulas describe how to calculate the regularization term. Its purpose is to make the distribution of the extracted feature points more uniform in space.

[0046] The feature point confidence regularization term is designed as follows:

[0047]

[0048] Among them, α is the coefficient of each term, the first term It is the difference between the position of the extracted feature point and the true value. The second item The third item is the difference between the confidence of the feature point and the true value. is the regularization term of the confidence, where K is the number of feature points and k is the index of the feature point.

[0049] The pose part consists of two parts, R,t,, which are rotation and translation. After the feature points are extracted, the descriptor is extracted from the feature map in the first stage, and then matched to calculate the relative pose between two adjacent images. The calculated relative pose and the true value are used to calculate the translation error and rotation error.

[0050] Finally, the specific training flow chart of this method is shown in Figure 4 In the present invention, we use the algorithm in OrbSLAM2 to calculate the pose through feature points. By matching the feature points in the image and other images, we use the PnP (Perspective-n-Point) algorithm to calculate the pose of the image. In the neural network part, the present invention uses pytorch as the network framework, uses the Euroc and tumvi datasets as training datasets, and uses the Adam optimizer to optimize the learning rate. The input image obtains a feature map after passing through the feature extraction network, and the feature extraction network extracts feature points from the feature image. On the one hand, the feature points directly calculate the loss function with the true value, and on the other hand, the descriptor is extracted on the feature image to match the previous and next frame images, calculate the loss function of the image extrinsic parameters, and calculate the interpolation between the estimated extrinsic parameters and the true value of the extrinsic parameters.

[0051] The training of the pose-based loss supervision network is supervised, so that the feature points learned by the network are conducive to pose estimation. Compared with traditional methods, the training of the pose-supervised network can more directly establish the relationship between pose and feature points, and has better performance in actual performance.

[0052] Step (6) replaces the feature point extraction part of the feature point method of the existing method with the trained feature point selection network to achieve robot positioning and map construction in complex environments;

[0053] In one embodiment, the present invention further provides a key point selection system based on a neural network, comprising a feature extraction module, a feature point extraction module, a homography transformation module, and a training module;

[0054] The feature extraction module uses a convolutional neural network as a feature extraction network to extract feature maps from the input image. The convolutional neural network compresses the input image's length and width to a quarter of the original resolution, increasing the number of channels from 3 to 128.

[0055] The feature point extraction module uses an encoder to further compress the feature map, reducing the number of channels to 8. Due to the varying resolutions of the input images, an adaptive pooling layer is used to compress the feature maps to a uniform size of 32x32x4. After converting the feature map to a 4096-dimensional vector, a fully connected neural network with three hidden layers is used to predict the feature points in each image.

[0056] The homography transformation module uses homography transformation to map the feature points obtained by the fast method to another space, and then projects the transformed feature points onto the original image. The feature points extracted in each space are combined as the true value of the feature point.

[0057] The training module designs the loss based on the true value obtained by the homography transformation; and trains the feature extraction module and feature point extraction module according to the designed loss;

[0058] In one embodiment, the present invention also provides a key point selection device based on a neural network, including a storage device and a processor, wherein the storage device is used to store a computer program, and the processor runs the computer program to enable the virtual wall setting device to perform the above-mentioned key point selection method based on the neural network.

[0059] In one embodiment, the present invention further provides a storage medium storing a computer program used in the aforementioned neural network-based key point selection device, wherein the computer program implements the aforementioned neural network-based key point selection method when executed by a processor.

[0060] In one embodiment, the method of the present invention is added to the common two-feature point method, replacing the feature point extraction part of the original feature point method, and is tested on a popular data set. It can be seen that after the method of the present invention is added and applied, the positioning indicators are improved in most scenarios, and it can also run scenarios that the SVO method cannot complete.

[0061]

[0062] Where svo is the method in the following reference number 1, and droid vo is the method in reference number 2.

[0063] 1. Forster C, Pizzoli M, Scaramuzza D.SVO: Fast semi-direct monocular visual odometry[C] / / 2014IEEE international conference on robotics and automation (ICRA). IEEE, 2014:15-22.

[0064] 2.Teed Z,Lipson L,Deng J.Deep patch visual odometry[J].Advances inNeural Information Processing Systems,2024,36.

[0065] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0066] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A key point selection method based on neural network, characterized in that: The following steps are involved: Step (1) feature extraction based on feature extraction network; A convolutional neural network is used as a feature extraction network to extract feature maps from the input image. The convolutional neural network compresses the length and width of the input image to a quarter of the original resolution, and the number of channels increases from the original 3 to 128 channels. Step (2) feature point extraction based on fully connected neural network; An encoder is used to further compress the feature map, reducing the number of channels to 8. Due to the varying resolutions of the input images, an adaptive pooling layer is used to compress the feature map to a uniform size of 32x32x4. The feature map is then converted to a 4096-dimensional vector, and a fully connected neural network with three hidden layers is used to predict the feature points in each image. The output of the fully connected neural network is a 200x3 vector representing the coordinates and confidence scores of the 200 feature points. Step (3) loss design based on homography transformation; Using homography transformation, the feature points obtained by the fast method are mapped to another space, and then the transformed feature points are projected onto the original image. The feature points extracted in each space are combined as the true value of the feature point; then, the relative pose between the two adjacent images is calculated by calculating the orb descriptors corresponding to the feature points in the two adjacent images; the calculated relative pose is compared with the true value and the pose error is calculated; in order to ensure the quality of the feature points, the feature point position and score regularization terms are added to the loss of the feature point selection network to further constrain the generation of feature points; the feature point selection network is composed of the feature extraction network in step 1 and the fully connected neural network in step 2; Step (4) training the feature point selection network according to the designed loss; During the network training process, the input image will first pass through the feature extraction network to obtain the feature map, and then extract the feature points through the fully connected neural network. After obtaining the feature points, the descriptor is calculated in combination with the feature map. Then, the PnP algorithm and adjacent frames are used to calculate the pose of the image, and the loss is calculated with the true value of the pose to supervise the network training.

2. A key point selection method based on a neural network according to claim 1, characterized in that: The convolutional neural network consists of two residual blocks connected together; the first residual block uses two convolutions with a kernel size of 3x3 and a stride of 2 to reduce the dimension of the original input image to half of the original, and the number of channels increases from 3 channels to 64 channels, that is, 64 different feature maps are extracted from the image. The two convolutions are followed by a layer of Batch Norm and a layer of ReLu activation function; the second residual block structure is the same as the first residual block, reducing the dimension of the feature map output by the first residual block to half of the original, and increasing the number of channels from 64 channels to 128 channels; A total of two residual blocks are used to reduce the image resolution to a quarter of the original, and the number of channels is increased from 3 to 128.

3. A key point selection method based on a neural network according to claim 2, characterized in that: The specific method of step (2) is as follows: An encoder is used to compress the feature map processed in the previous stage. The encoder uses a feature extraction network to compress the size of the feature map to one-eighth of its original size, and the number of channels becomes 8. This network consists of a 3x3 convolutional layer with a stride of 1, a padding of 1, and 64 channels, a 2x2 pooling layer with a stride of 2, and a 3x3 convolutional layer with a stride and padding of 1 and 8 channels. Since the input images vary in size and scale, an adaptive pooling layer is used to unify the dimensions of the feature map output by the encoder to 32x32x4. After the complete feature map is compressed and unified, a fully connected layer is used to extract a feature vector of dimension 200x3. The number 200 represents the 200 feature points extracted from the image, the first two dimensions of the three dimensions are the coordinates of the feature points on the image, and the last dimension is the confidence of the feature points.

4. A key point selection method based on a neural network according to claim 3, characterized in that: Homography transformation is used to generate complete true values ​​of feature points. The original image and the feature map extracted by the feature extraction network are transformed by homography, the original image is mapped to different spaces, feature points are extracted in different spaces, and finally the feature points in all spaces are superimposed and transformed back to the original space, thereby obtaining the complete true values ​​of the feature points.

5. A key point selection method based on a neural network according to claim 4, characterized in that: The feature point selection network loss function consists of a feature point part and a pose part; the feature point part consists of a feature point position regularization term and a feature point confidence regularization term; the feature point position regularization term is as follows: The first formula is the value of the overall regularization term, Represents the regular term of the x-axis, represents the regularization term of the y-axis; the last two formulas are how to calculate the regularization term, where: It represents the x-axis coordinate of the i-th feature point. It represents the y-axis coordinate of the i-th feature point, M is the number of feature points in the image, and i is the index of the feature point; the last two formulas are how to calculate the regularization term; its purpose is to make the extracted feature points more evenly distributed in space; The feature point confidence regularization term is designed as follows: Among them, α is the coefficient of each term, the first term It is the difference between the position of the extracted feature point and the true value. The second item The third item is the difference between the confidence of the feature point and the true value. is the regular term of confidence, where K is the number of feature points and k is the index of the feature point; The pose part consists of two parts, R,t,, which are rotation and translation. After the feature points are extracted, the descriptor is extracted from the feature map in the first stage, and then matched to calculate the relative pose between two adjacent images. The calculated relative pose and the true value are used to calculate the translation error and rotation error.

6. A system based on a neural network-based key point selection method according to any one of claims 1 to 5, characterized in that: Including feature extraction module, feature point extraction module, homography transformation module and training module; The feature extraction module uses a convolutional neural network as a feature extraction network to extract a feature map from the input image; the convolutional neural network compresses the length and width of the input image to a quarter of the original resolution, and the number of channels increases from the original 3 channels to 128 channels; The feature point extraction module uses an encoder to further compress the feature map, reducing the number of channels to 8. Due to the varying resolutions of the input images, an adaptive pooling layer is used to compress the feature map to a uniform size of 32x32x4. Subsequently, after converting the feature map into a 4096-dimensional vector, a fully connected neural network with 3 hidden layers is used to predict the feature points in each image. The homography transformation module uses homography transformation to map the feature points obtained by the fast method into another space, and then projects the transformed feature points onto the original image. The feature points extracted in each space are combined as the true value of the feature point; The training module designs a loss based on the true value obtained by the homography change transformation; and trains the feature extraction module and the feature point extraction module according to the designed loss.

7. A key point selection device based on a neural network, characterized in that: It includes a storage device and a processor, the storage device is used to store a computer program, and the processor runs the computer program to implement the key point selection method based on a neural network according to any one of claims 1 to 5.

8. A storage medium, characterized in that: It stores a computer program used in the neural network-based key point selection device, and when the computer program is executed by a processor, it implements the neural network-based key point selection method according to any one of claims 1 to 5.

Images