A method and system for constructing a key point extraction network of a neural network architecture

Abstract

The present invention discloses a method and system for deep key point extraction based on neural network architecture search, which utilizes neural network architecture search technology to automatically design a model for deep key point extraction. The model can simultaneously detect and describe key points, and ultimately search for a lighter and better-performing key point extraction network model.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular to a method and system for constructing a key point extraction network of a neural network architecture. Background Art

[0002] Key point detection is a basic step in computer vision tasks and an important part of many applications. In the past few decades, manual key point extraction has been very mature, including key point detection and calculation of descriptors based on local information of feature points. However, manual key points are too dependent on human experience and have certain limitations. More and more researchers are turning their attention to the field of deep learning. Key point extraction methods based on deep learning rarely use supervised methods, because key points are not clearly defined and it is difficult to calibrate the true value. Therefore, most of the existing key point extraction models based on deep learning are self-supervised, which is more feasible.

[0003] However, most of the existing network models for key point extraction do not consider performance, model parameter quantity and computational complexity at the same time. In actual use, the relatively large network models are too heavy for subsequent visual tasks, such as SLAM systems. Summary of the invention

[0004] The present invention aims to solve the problem that existing methods cannot well balance model size, model calculation amount and model performance.

[0005] The technical solution adopted by the present invention is: a method for constructing a key point extraction network based on a neural network architecture search, comprising the following steps:

[0006] S1, prepare image data I and image data V, the image data V includes a source image, a target image and the real homography transformation relationship between the two, the source image S in the image data I is transformed by homography to obtain the target image T, and the source image S and the target image T constitute the training data;

[0007] S2, obtain sub-network model A by uniformly sampling each block operation of the hyper-network model, input the training data into sub-network model A, obtain the key point scores, key point position coordinates and descriptors corresponding to the key points of the training data, and use the above three to construct a loss function, calculate the error through the loss function, perform back propagation to update the network parameters of the hyper-network model, and perform iterative updates until the loss function no longer decreases;

[0008] S3, using the controller model to output the operands selected in each block of the super network model, generate a sub-network model B according to the operands and the network parameters of the super network model iteratively updated in S2, extract a pair of source image and target image from the image data V and input them into the sub-network model B, obtain the key point score, key point position coordinates and descriptors corresponding to the key points of the image pair, and then calculate the evaluation index of the image pair in combination with the true value of the homography transformation relationship in the image data V, and then input the remaining image pairs in the image data V into the sub-network model B one by one and calculate its evaluation index, use the evaluation index of all image pairs as the evaluation index of the sub-network model B, and use the evaluation index of the sub-network model B as the reward value to update the parameters of the controller model;

[0009] S4, repeat S3 until the preset number of iterations is reached or the performance of sub-network model B is no longer improved;

[0010] S5, using the iterated controller model to output N sub-network models C, and selecting the one with the best performance as the output result.

[0011] Preferably, in S2, the loss function is:

[0012]

[0013] Among them, α, β, γ, and λ are used to balance the weights between different error terms. The definition of each error term is as follows.

[0014]

[0015] L pos is the error used to learn the keypoint positions, Indicates the key points of the source image are warped to the corresponding positions of the target image. Indicates the distance from the target image The nearest key point, i represents the key point number.

[0016]

[0017] In order to assist the key point location learning, the image gradient error L is added grad For supervision, I(i,j) represents the gray value, Represents the sum of the gradients of the image position (i, j) in the horizontal, vertical and diagonal directions.

[0018]

[0019] L score is the error used to learn the keypoint scores, s i and Represent the scores of the corresponding key points in the source image and the target image respectively, d represents the Euclidean distance between matching points, and L represents the total number of matching points.

[0020]

[0021] L desc is the error used to learn the descriptor, k i is the descriptor in the source image, is the corresponding positive sample in the target image, is a negative sample in the target image, and m is a threshold, which indicates the maximum distance limit between dissimilar key points in the description subspace.

[0022] As a preferred method, the evaluation indicators of the S3 neutron model include repeatability score (Rep), positioning error (Loc), matching score (M score ), homography accuracy (H Cor3 ), and finally combine them as the reward value for controller update, which is defined as follows:

[0023]

[0024] Preferably, the evaluation index of the sub-network model in S3 is the average value of the evaluation indexes of all image pairs.

[0025] Preferably, N in S5 is a positive integer less than or equal to 10.

[0026] Preferably, in S3, the evaluation index of the sub-network model B is obtained and then recorded.

[0027] A deep key point extraction system based on neural network architecture search includes a hypernetwork model and a controller model, wherein the hypernetwork model includes a Stem module, a Middle module and a Tail module, wherein the Middle module is composed of a combination of a Convolution Block (CB) and a Reduction Block (RB). The Stem module is used for initial data processing, the Middle module is used as an encoder to extract features, and the Tail module is used as a decoder to output key point related information, including scores, positions and descriptors.

[0028] Preferably, the Convolution Block includes 6 parallel combinations of convolution operations with different parameters, and the Reduction Block includes 3 parallel combinations of convolution downsampling operations.

[0029] Compared with the existing technology, the present invention has the following advantages and effects:

[0030] The present invention automatically designs a key point extraction model based on a neural network architecture search, which reduces the number of model parameters and the amount of calculation on the one hand, and ensures the performance of the model on the other hand. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0032] Figure 1 This is a schematic diagram of the super network structure of the present invention;

[0033] Figure 2 Operation diagrams included in the Convolution Block module and the ReductionBlock module;

[0034] Figure 3 A schematic diagram of loss function calculation for self-supervised training of the present invention;

[0035] Figure 4 It is a schematic diagram of the controller model;

[0036] Figure 5 It is a schematic diagram of the output result of the present invention;

[0037] Figure 6 Comparison of the key performances of the present invention and other methods;

[0038] Figure 7 Comparison of the complexity and performance of the present invention with existing advanced network structures;

[0039] Figure 8 To compare the performance of the present invention applied to a closed loop SLAM system;

[0040] Fig. 9 The present invention is applied to the performance comparison of the SLAM system with open loop. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] Embodiment 1:

[0043] A deep key point extraction system for neural network architecture search, including a hypernetwork structure and a controller model. Figure 1As shown in Figure 1, the hypernetwork structure includes the Stem module, the Middle module, and the Tail module. The Stem module is used for initial data processing, including a convolution operation with a kernel size of 7x7, a stride of 1, and a padding of 3, followed by a batch normalization operation. The Middle module is used as an encoder to extract features, and is composed of a combination of Convolution Block (CB) and Reduction Block (RB). The specific arrangement is as follows: Figure 1 As shown in Figure 1, CB has 7 layers and RB has 3 layers. The size and number of channels of the feature map output by CB are consistent with those of the input feature map. The size of the feature map output by RB is halved compared to the input feature map, but the number of channels is doubled. Figure 2 As shown in the figure, CB contains 6 parallel combinations of convolution operations with different parameters, and RB contains 3 parallel combinations of convolution downsampling operations. The Tail module acts as a decoder to output the score, position, and descriptor of the key point. Both the key point score decoder and the key point position decoder contain a channel attention module, followed by a convolution operation with a kernel size of 3x3, a stride of 1, and a padding of 1. The descriptor decoder contains a channel attention module, followed by a PixelShuffle operation, followed by a channel attention module, and the last layer is a convolution layer with a kernel size of 3x3, a stride of 1, and a padding of 1.

[0044] like Figure 1-Figure 5 As shown, a method for constructing a key point extraction network of a neural network architecture includes the following steps.

[0045] Step 1: Prepare unlabeled image data I, such as the MS COCO dataset, which contains 118,287 images. Each image in the dataset is used as a source image. By artificially randomly generating a homography transformation (HT) matrix within a certain range, the target image corresponding to the source image is generated to obtain an image pair. Prepare a verification dataset V, which contains perspective transformation and illumination transformation, and the dataset itself provides source images and target images, as well as the HT transformation relationship between them, such as the HPatches dataset.

[0046] Step 2: Training of the hypernetwork model:

[0047] 2.1, extract some of the image pairs from step 1 as the training data of the current batch. Preferably, the batch size is set to 8.

[0048] 2.2, followed by the sampling of sub-network model A. The specific approach is to uniformly sample the block operations of each layer of the Middle module of the hypernetwork to select a candidate operation. After the selection of all blocks is completed, a path from input to output is formed, and this path is used as the sub-network model A of the current training batch.

[0049] 2.3. Input the batch training data in 2.1 into the sub-network model A sampled in 2.2 to obtain the scores, positions and descriptors of the key points of the source image and the target image.

[0050] 2.4, then use the following loss function to calculate the error. The schematic diagram of the error construction is as follows Figure 3 As shown, back propagation is then performed to update the network parameters, and the loss function is defined as follows:

[0051]

[0052] L pos is the error used to learn the key point position, the image gradient error L grad Used to assist key point location learning, L score is the error used to learn the keypoint scores, L desc is the error used to learn the descriptor, and each term is defined as follows:

[0053]

[0054]

[0055]

[0056]

[0057] in, Indicates the key points of the source image are warped to the corresponding positions of the target image. Indicates the distance from the target image The nearest key point, i represents the key point number; I(i, j) represents the gray value, Represents the sum of the gradients of the image position (i, j) in the horizontal, vertical and diagonal directions; s i and Represent the scores of the corresponding key points in the source image and the target image respectively, d represents the Euclidean distance between matching points, L represents the total number of matching points; k i is the descriptor in the source image, is the corresponding positive sample in the target image, is a negative sample in the target image, and m is a threshold, which indicates the maximum distance limit between dissimilar key points in the description subspace; preferably, m = 0.2, α = 1.0, β = 1.0, γ = 1.0, and λ = 2.0.

[0058] 2.5. Repeat 2.1-2.4 for training iterations until the loss function no longer decreases. After multiple iterations, ideally, all candidate operations of each block are fully trained.

[0059] Step 3: Search for the model:

[0060] 3.1, the controller model is used to generate the code of the sub-network model B. The code is a sequence that represents the number of operations selected in each layer of the block. The code and the sub-network model B are in a one-to-one correspondence. The process diagram is as follows Figure 4 shown.

[0061] 3.2. After confirming the structure of sub-network model B, its model parameters are inherited from the network parameters of the corresponding structure in the trained super-network model in step 2, and no retraining is required. This weight sharing method reduces the time consumption caused by the need to retrain the sub-network model and improves efficiency.

[0062] 3.3, the structure of sub-network model B is determined by 3.1, and the network parameters of sub-network model B are determined by 3.2. Then, any pair of source image and target image is extracted from the verification data set V and input into the above sub-network model to obtain the key point scores, key point position coordinates and descriptors of the source image and target image, and then the evaluation index of the current image pair is calculated in combination with the true HT value provided in the verification data set V. The calculation formula of the evaluation index is as follows:

[0063]

[0064] Among them, Rep is the repeatability score, Loc is the positioning error, and M score is the matching score, H Cor3 is the homography accuracy.

[0065] Then continue to randomly select a pair of image pairs and calculate their evaluation index according to the method in 3.3. Repeat this process until the evaluation index of all image pairs in the validation dataset V is calculated. The average value of the evaluation index calculated for all image pairs is used as the final evaluation index of the current sub-network model to measure the performance of the sub-network model. Then you can proceed directly to step 4, or you can record the structure and evaluation index of the sub-network model before proceeding.

[0066] Step 3.4, after completing steps 3.1-3.3, you can get an evaluation index of a sub-network model. In order to reduce the impact of noise on the controller model training, the evaluation index of sub-network model B is used as the reward value to update the parameters of the controller model. As a preferred method, the controller samples 4 sub-network models each time, and uses their average performance index as the reward value of the controller to update the model parameters of the controller. Specifically, the policy gradient in reinforcement learning is used to update the parameters, and finally the performance of the self-network model sampled by the controller is continuously improved.

[0067] Step 4, then repeat 3.1-3.4 for iteration. When the preset number of iterations is reached or the performance of the sub-network model output by the controller model no longer improves, the training of the controller ends.

[0068] Step 5: Use the controller model trained in step 4 to output N sub-network models, where N is preferably an integer less than or equal to 10, and select the one with the best performance as the final search result.

[0069] like Figure 6 and Figure 7 As shown in Figure , the comparison results with other advanced network models show that the key points predicted by the present invention have better performance in terms of evaluation indicators. Figure 8 and Fig. 9 As shown, the present invention is applied to a mature SLAM system (ORB-SLAM2), and compared with the existing methods, the positioning accuracy and robustness are stronger.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a key point extraction network of a neural network architecture, characterized in that: The following steps are involved: S1, prepare image data I and image data V, the image data V includes a source image, a target image and the real homography transformation relationship between the two, the source image S in the image data I is transformed by homography to obtain the target image T, and the source image S and the target image T constitute the training data; S2, obtain sub-network model A by uniformly sampling each block operation of the hyper-network model, input the training data into sub-network model A, obtain the key point scores, key point position coordinates and descriptors corresponding to the key points of the training data, and use the above three to construct a loss function, calculate the error through the loss function, perform back propagation to update the network parameters of the hyper-network model, and perform iterative updates until the loss function no longer decreases; S3, using the controller model to output the operands selected in each block of the super network model, generate a sub-network model B according to the operands and the network parameters of the super network model iteratively updated in S2, extract a pair of source image and target image from the image data V and input them into the sub-network model B, obtain the key point score, key point position coordinates and descriptors corresponding to the key points of the image pair, and then calculate the evaluation index of the image pair in combination with the true value of the homography transformation relationship in the image data V, and then input the remaining image pairs in the image data V into the sub-network model B one by one and calculate its evaluation index, use the evaluation index of all image pairs as the evaluation index of the sub-network model B, and use the evaluation index of the sub-network model B as the reward value to update the parameters of the controller model; S4, repeat S3 until the preset number of iterations is reached or the performance of sub-network model B is no longer improved; S5, using the iterated controller model to output N sub-network models C, and selecting the one with the best performance as the output result.

2. The method for constructing a key point extraction network of a neural network architecture according to claim 1, characterized in that: In step S2, the loss function is: Among them, α, β, γ, and λ are used to balance the weights between different error terms; L pos is the error used to learn the keypoint positions, Indicates the key points of the source image are warped to the corresponding positions of the target image. Indicates the distance from the target image The nearest key point, i represents the key point number; L grad is the image gradient error, I(i, j) represents the gray value, Represents the sum of the gradients of the image position (i, j) in the horizontal, vertical and diagonal directions; L score is the error used to learn the keypoint scores, s i and Represent the scores of the corresponding key points in the source image and the target image respectively, d represents the Euclidean distance between matching points, and L represents the total number of matching points; L desc is the error used to learn the descriptor, k i is the descriptor in the source image, is the corresponding positive sample in the target image, is a negative sample in the target image, and m is a threshold, which indicates the maximum distance limit between dissimilar key points in the description subspace.

3. The key point extraction network construction method of the neural network architecture according to claim 1 is characterized in that: The calculation formula of the evaluation index of the image pair in S3 is as follows: Among them, Rep is the repeatability score, Loc is the positioning error, and M score is the matching score, H Cor3 is the homography accuracy.

4. The method for constructing a key point extraction network of a neural network architecture according to claim 1, characterized in that: The evaluation index of the sub-network model in S3 is the average of the evaluation index of all image pairs.

5. The method for constructing a key point extraction network of a neural network architecture according to claim 1, characterized in that: N in S5 is a positive integer less than or equal to 10.

6. The method for constructing a key point extraction network of a neural network architecture according to claim 1, characterized in that: In S3, the evaluation index of the sub-network model B is obtained and then recorded.

7. A key point extraction network construction system for a neural network architecture, used to implement the key point extraction network construction method for a neural network architecture according to any one of claims 1 to 6, characterized in that: It includes a hypernetwork model and a controller model, wherein the hypernetwork model includes a Stem module, a Middle module, and a Tail module; the Middle module is composed of a Convolution Block and a Reduction Block, and the Middle module is used as an encoder to extract features; the Stem module is used for initial data processing; and the Tail module is used as a decoder to output key point related information.

8. According to the key point extraction network construction system of the neural network architecture of claim 7, ConvolutionBlock includes 6 parallel combinations of convolution operations with different parameters, and Reduction Block includes 3 parallel combinations of convolution downsampling operations.

Images