Method and system for rapidly extracting and matching features in low-illumination weak-texture environment

Abstract

The invention discloses a rapid feature extraction and matching method and system in a low-illumination weak-texture environment, and the method comprises the steps: carrying out the preprocessing of unlabeled image data, and constructing an unlabeled data set; constructing an image feature extraction model, and carrying out iterative optimization training on the image feature extraction model by adopting a self-supervised training mechanism based on the data set; inputting the data of the two collected images into the trained feature extraction model, removing mismatching points from the output of the feature extraction model, and obtaining a matched feature point set; the problems that an existing feature extraction and matching method is poor in scene adaptability, low in detection precision, difficult in data labeling and the like are solved, image features can be learned in a self-adaptive mode, and the method has high robustness and generalization ability.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and system for rapid feature extraction and matching in a low-light and weak-texture environment. Background Art

[0002] Feature point extraction is an important task in the field of computer vision. Whether it is VR / AR or visual positioning in SLAM / 3D reconstruction, it is all based on accurate, stable and fast feature point extraction and feature matching. Fast and accurate feature extraction is a hot topic in current research and is the basis for realizing downstream visual characters. However, the characteristics of drastic changes in lighting and different changes in texture features in actual scenes are important challenges for feature point extraction.

[0003] Traditional image feature point extraction and matching methods rely on manually designed feature descriptors to extract pixel points with more drastic gradient changes in the image, such as image corners and edge points, which can represent obvious signs of the image. Methods such as SIFT and SURF are invariant to image rotation and scale changes and can adapt to most scenes, but they perform poorly in complex environments and lighting changes. They have the disadvantages of failure in scenes with drastic lighting changes, low extraction efficiency, and inaccurate matching.

[0004] In recent years, with the development of deep learning, feature extraction algorithms based on neural networks have made important breakthroughs. Among them, the SuperPoint method is the mainstream feature point detection method based on deep learning, but most methods still rely on a large amount of labeled data, and are difficult to adapt to changes in different scales and lighting conditions, and have weak generalization capabilities. At the same time, they still do not exceed the traditional SIFT algorithm in detection and matching accuracy. Therefore, it is particularly important to develop a feature point extraction and matching method that can adaptively learn image features and has strong robustness and generalization capabilities. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for fast feature extraction and matching in a low-light and weak-texture environment, so as to solve the problems of poor scene adaptability, low detection accuracy and difficulty in data labeling of existing feature extraction and matching methods, and to adaptively learn image features with strong robustness and generalization ability.

[0006] The technical solution to achieve the purpose of the present invention is:

[0007] A fast feature extraction and matching method in a low-light and weak-texture environment, comprising:

[0008] Step 1: preprocess the unlabeled image data to construct an unlabeled dataset;

[0009] Step 2: construct an image feature extraction model, and use a self-supervised training mechanism to iteratively optimize the image feature extraction model based on the data set;

[0010] Step 3: Input the collected two image data into the trained feature extraction model, remove the mismatched points from the output of the feature extraction model, and obtain a set of matched feature points.

[0011] Furthermore, the image feature extraction model includes a multi-scale convolution layer, a feature pyramid network, a feature point extraction module and a feature matching module. The feature matching module uses convolutions of different scales to extract multi-level feature information. The feature pyramid network is used to fuse the extracted multi-level feature information and output a fused feature map. The feature point extraction module is used to extract feature points of the feature map. The feature matching module combines the constraints of the self-supervised training mechanism to eliminate feature point matching pairs that do not meet the constraints, and obtain optimized matching pairs that minimize photometric errors.

[0012] Furthermore, the multi-scale convolution layer includes convolution layers of three different scales, each layer includes a convolution layer, normalization, an activation function and a residual connection, and outputs feature maps of three different scales.

[0013] Furthermore, the feature pyramid network fuses feature maps of different scales through upsampling and downsampling operations, and outputs the fused feature map through a fully connected layer and a pooling layer.

[0014] Furthermore, the feature point extraction module uses non-maximum suppression and threshold segmentation methods to extract feature points in the feature map.

[0015] Furthermore, when the self-supervised training mechanism is used to iteratively optimize the image feature extraction model, it includes: generating images of different resolutions through an image pyramid, and then performing adaptive rotation and translation transformations on the image, judging whether the images before and after the transformation meet the designed constraints. If so, the images before and after the transformation are input into the image feature extraction model, and the image feature extraction model is trained with minimization of the designed photometric error function as the training optimization goal.

[0016] Furthermore, the constraints of the self-supervised training mechanism are designed as follows:

[0017]

[0018] in, For distance, is the angle, , are the image coordinates before and after the transformation, respectively.

[0019] Furthermore, the designed photometric error function is:

[0020]

[0021] in, is the window size around the feature point, are all pixels within the feature point window, , are the image coordinates before and after the transformation, i and j represent the distance from the feature point in the X and Y directions, , Respectively represent the grayscale values ​​of the corresponding pixels of the image before and after the transformation.

[0022] Furthermore, the unlabeled image data is preprocessed, specifically including:

[0023] Step 1-1, use Gaussian filtering to denoise the image data and obtain:

[0024]

[0025] in, is the standard deviation of the Gaussian distribution, are the horizontal and vertical coordinates of the image;

[0026] Step 1-2, grayscale the image using the histogram equalization method, that is,

[0027]

[0028] in, Represents the grayscale range, is the number of pixels in the image, Grayscale The number of pixels;

[0029] Steps 1-3 use bilinear interpolation method to adjust the image size.

[0030] A fast feature extraction and matching system in a low-light and weak-texture environment, comprising:

[0031] A data set construction unit preprocesses the unlabeled image data and constructs an unlabeled data set;

[0032] Image feature extraction model construction and training unit: constructs an image feature extraction model and uses a self-supervised training mechanism to iteratively optimize and train the image feature extraction model based on the data set;

[0033] The feature point matching unit uses the trained feature extraction model to perform feature extraction and matching on the two collected image data, eliminates the mismatched points, and obtains a set of matched feature points.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1) The present invention uses unlabeled image data to generate positive and negative sample pairs through image transformation (such as rotation, scaling, translation, etc.), and uses the constructed photometric consistency constraints (i.e., the image feature points should remain consistent before and after the transformation) and geometric consistency constraints (i.e., the geometric relationship between the image feature points before and after the transformation should remain consistent) to optimize the training image feature extraction model, thereby realizing self-supervised training of the feature extraction network;

[0036] 2) In the feature extraction process, the present invention adopts a multi-scale convolutional neural network structure to extract multi-level feature information in the image through convolution kernels of different scales; then, the features of different scales are fused using a feature pyramid network (FPN) to obtain a feature map containing rich contextual information, thereby improving the accuracy and robustness of feature point extraction;

[0037] 3) In the process of model feature matching, the present invention introduces a minimization optimization function based on photometric error. This function optimizes the matching result by calculating the photometric difference of pixels around the matching feature points and minimizing the photometric error. This method can more effectively handle illumination changes and noise interference, and improve the accuracy and stability of matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the fast feature extraction and matching method in a low-light and weak-texture environment of the present invention.

[0039] Figure 2 It is a schematic diagram of the feature point extraction effect of the present invention.

[0040] Figure 3 It is a schematic diagram of the feature point matching effect of the present invention.

[0041] Figure 4 The feature point map is extracted from the images collected at different locations in scenes with poor lighting conditions.

[0042] Figure 5 This is the feature point matching result graph of images collected at different locations. DETAILED DESCRIPTION

[0043] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings to help those skilled in the art better understand the present invention. It should be particularly noted that in the following description, in order to highlight the core content of the present invention, the details of the known functions and designs will not be described in detail if they do not directly affect the understanding of the present invention.

[0044] This application is aimed at the technology of image feature point extraction and matching, and proposes a fast feature extraction and matching method in low-light and weak-texture environment. This method is innovatively based on self-supervised training, multi-scale feature fusion, and optimization method to minimize photometric error and geometric error, aiming to improve the accuracy of image feature point extraction and the robustness of matching, combined with Figure 1 , the method specifically comprises the steps of:

[0045] Step 1, data preprocessing, mainly includes denoising, grayscale (if the input is a color image), histogram equalization, and size adjustment of the input image to ensure that the image quality meets the requirements of subsequent network input. Specifically include:

[0046] Step 1-1: Use Gaussian filtering to remove noise and input image , the coordinates of the image are expressed as , and Gaussian filtering is:

[0047]

[0048] in, It is the standard deviation of the Gaussian distribution, and the noise is reduced by performing operations on each pixel with the Gaussian kernel.

[0049] Step 1-2: The histogram equalization method is used to grayscale the image (color image) to improve the contrast of the image, make the image grayscale distribution more uniform, and increase the significance of the feature points. , Represents the grayscale range, usually 256, and the number of pixels in the image is , grayscale is The number of pixels is , the histogram equalization operation is:

[0050]

[0051] Step 1-3, image resizing, uses bilinear interpolation method to adjust the image to a suitable size.

[0052] Step 2: Design a self-supervised training mechanism. The core is to use the inherent structure and relationship of the data itself to train the model, without the need to manually label a large amount of feature point information, thereby reducing the data labeling cost and improving the generalization ability of the model. Specifically include:

[0053] Step 2-1, generate images of different resolutions through image pyramid to simulate the effect of human eyes observing objects at different distances and provide a basis for multi-scale feature extraction. The original image is , the coordinates of the image are expressed as , the scaling factor is , represents the Gaussian kernel, The layer image can be represented as:

[0054]

[0055] Step 2-2, rotation, translation and scaling of the image. The purpose of adaptive rotation and translation is to ensure that the data set is diverse, covering different scenes, lighting conditions and object types, enhancing the diversity of the data and improving the generalization ability of the model.

[0056] The original image is , the coordinates of the image are expressed as , rotation angle , the translation vector , the coordinates after rotation and translation for:

[0057]

[0058] Step 2-3: The transformed image is constrained for photometric consistency, and the photometric change threshold of the pixels around the feature point before and after the transformation is used as a measure of feature point matching.

[0059] Image before transformation , the transformed image , The coordinates of the image feature points are , The coordinates of the image feature points are , after rotation, translation and scaling transformation, the photometric error is calculated:

[0060]

[0061] in, is the window size around the feature point, are all pixels within the feature point window, , are the image coordinates before and after the transformation, i and j represent the distance from the feature point in the X and Y directions, , Respectively represent the grayscale values ​​of the corresponding pixels of the image before and after the transformation.

[0062] Step 2-4, geometric consistency constraint, is to determine whether the set constraint relationship of feature points before and after the transformation meets the set threshold. Image before transformation , the transformed image , The coordinates of the image feature points are , The coordinates of the image feature points are , after rotation, translation and scaling transformation, the distance and angle , by setting a threshold, it is determined whether the geometric consistency constraints are met.

[0063]

[0064] Step 3: Build an image feature extraction model for image feature extraction and matching.

[0065] The image feature extraction model mainly uses a self-supervised training mechanism combined with a multi-scale feature fusion network to extract image feature points, and then extracts and matches feature points by minimizing photometric error optimization. The image feature extraction model includes a multi-scale convolution layer, a feature pyramid network, a feature point extraction module, and a feature matching module. By using convolution kernels of different scales, different levels of features from local details to global structures can be captured. After these features are fused by the feature pyramid network (FPN), they can better reflect the overall feature distribution of the image. Image feature extraction and matching specifically include:

[0066] Step 3-1, obtaining multi-level feature information of different scales through a multi-scale convolutional layer, wherein the multi-scale convolutional layer includes three convolutional layers of different scales, each layer including a convolutional layer, normalization, an activation function and a residual connection.

[0067] Let the input image be , the size is , Indicates the image height, Indicates the image width, It represents the number of channels and uses three different scales of convolution to extract multi-level feature information.

[0068] Small-scale convolution uses convolution kernel , the step length is , filled with , convolution operation , normalization operation and activation operation Get feature map :

[0069]

[0070] After the small-scale convolution layer operation, part of the information of the original input image is connected through the residual Get feature map :

[0071]

[0072] The mid-scale convolution uses the convolution kernel , the step length is , filled with , convolution operation , normalization operation and activation operation Get feature map :

[0073]

[0074] After the medium-scale convolution layer operation, part of the information of the original input image is connected through the residual Get feature map :

[0075]

[0076] Large-scale convolution uses convolution kernel , the step length is , filled with , convolution operation , normalization operation and activation operation Get feature map :

[0077]

[0078] After the large-scale convolution layer operation, part of the information of the original input image is connected through the residual Get feature map

[0079]

[0080] Step 3-2: Multi-scale features are fused through feature pyramid network (FPN). Feature pyramid network is a network that combines feature maps by upsampling and downsampling operations. and feature map Fusion to feature map And get the output information through the fully connected layer and the pooling layer.

[0081] The feature map Downsampling and feature maps Upsampling and feature map Consistent size, and then get the fused feature map , and then global average pooling and fully connected Operations get rich contextual information of output .

[0082]

[0083] Step 3-3, feature point extraction. Feature extraction is based on the output results of the feature pyramid network, and stable feature points are extracted using non-maximum suppression and threshold segmentation methods.

[0084] Non-maximum suppression (NMS): Assume that the response value of each pixel in the feature map is , for each local area centered on a pixel, if the response value of the pixel If is the maximum value of all pixel response values ​​in the neighborhood and is greater than the set threshold, the pixel is retained as a potential feature point. Otherwise, the pixel is suppressed, that is, it is not used as a feature point.

[0085] Threshold segmentation: The set of potential feature points retained after non-maximum suppression is , for each potential feature point , if the response value is greater than the set threshold, the point is finally determined to be a stable feature point.

[0086] Step 3-4, feature matching, combines the geometric consistency of the self-supervision mechanism to check the angle and distance between feature matching pairs, eliminate matching pairs that do not meet the constraints, and then construct optimized matching pairs that minimize the photometric error.

[0087] Step 4: Data post-processing to remove mismatched points, aiming to improve the accuracy, reliability and efficiency of matching. Specifically includes:

[0088] Step 4-1, cross matching filtering. Suppose from image The feature point set extracted from , and its corresponding feature descriptor set is ; From the image The feature point set extracted from , and its corresponding feature descriptor set is .

[0089] For images The descriptor corresponding to the feature point in the image is searched through K nearest neighbors Descriptors corresponding to the feature points in the calculation of descriptor distance Then reversely calculate the descriptor distance ,Judge whether the distance consistency meets the threshold, and retain it if it meets the threshold, otherwise delete it.

[0090] Step 4-2, RANSAC removes outliers. Image after cross filtering The feature point set in is , and its corresponding feature descriptor set is ;image The feature point set in is , and its corresponding feature descriptor set is There is a correspondence between them. The RANSAC algorithm can be used to estimate the correspondence matrix, and the mismatched points can be removed by repeated iterations. The correspondence matrix of the planar scene is the homography matrix, and the correspondence matrix of the non-planar scene is the essential matrix. Taking the essential matrix as an example, the EPNP algorithm is used to calculate the essential matrix for the candidate matching point pairs, and then the number of matching points whose photometric errors of all matching points are less than a certain threshold is calculated. If it exceeds a certain number, the estimated homography matrix is ​​considered accurate, and the matching points whose reprojection errors are greater than are removed as outliers.

[0091] The present invention also includes a fast feature extraction and matching system in a low-light and weak-texture environment, comprising:

[0092] A data set construction unit preprocesses the unlabeled image data and constructs an unlabeled data set;

[0093] Image feature extraction model construction and training unit: constructs an image feature extraction model and uses a self-supervised training mechanism to iteratively optimize and train the image feature extraction model based on the data set;

[0094] The feature point matching unit uses the trained feature extraction model to perform feature extraction and matching on the two collected image data, eliminates the mismatched points, and obtains a set of matched feature points.

[0095] In order to verify the application effect of the fast feature extraction and matching method in low-light and weak-texture environment in robot positioning, this method is tested on public data sets and real scene pictures collected by the robot's own camera. The input image is preprocessed, including noise reduction and other operations. Then, a self-supervised training mechanism is used to extract image feature points in combination with a multi-scale feature fusion network. Finally, image feature point matching is performed. Figure 1 The method of the present invention is tested on the TUM public dataset, where TUM is a commonly used dataset for robot & autonomous driving SLAM, covering a wide range and rich data types. The feature extraction results are shown in Figure 2 As shown, the matching results are as follows Figure 3 As shown in the figure, the method of the present invention is verified in real scenes captured by the robot camera. For the real scene data, the robot is specifically selected to conduct experiments in indoor scenes with poor lighting conditions, weak texture and uneven exposure. In scenes with poor lighting conditions, accurate feature points can be effectively extracted, such as Figure 4 As shown in the figure, this is an example of feature points extracted from pictures collected at different locations in a real scene. In addition, the method can accurately match feature points to provide assistance for robot positioning, such as Figure 5 As shown in FIG. 1 , the feature point matching results of images collected at different positions are shown.

[0096] The method of the present invention uses unlabeled images for training based on a self-supervised training mode, and simultaneously performs multi-scale feature fusion and image feature point extraction and matching that minimizes photometric errors, thereby improving the accuracy of feature detection and matching.

[0097] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0098] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A fast feature extraction and matching method in a low-light and weak-texture environment, characterized in that: include: Step 1: preprocess the unlabeled image data to construct an unlabeled dataset; Step 2: construct an image feature extraction model, and use a self-supervised training mechanism to iteratively optimize the image feature extraction model based on the data set; Step 3: Input the collected two image data into the trained feature extraction model, remove the mismatched points from the output of the feature extraction model, and obtain a set of matched feature points.

2. According to the method of claim 1, the method is characterized in that: The image feature extraction model includes a multi-scale convolution layer, a feature pyramid network, a feature point extraction module and a feature matching module. The feature matching module uses convolutions of different scales to extract multi-level feature information. The feature pyramid network is used to fuse the extracted multi-level feature information and output a fused feature map. The feature point extraction module is used to extract feature points of the feature map. The feature matching module combines the constraints of the self-supervised training mechanism to eliminate feature point matching pairs that do not meet the constraints and obtain optimized matching pairs that minimize photometric errors.

3. According to the method of fast feature extraction and matching in low light and weak texture environment of claim 2, it is characterized by: The multi-scale convolution layer includes three convolution layers of different scales, each layer includes a convolution layer, normalization, an activation function and a residual connection, and outputs feature maps of three different scales.

4. According to the method of fast feature extraction and matching in low light and weak texture environment of claim 2, it is characterized by: The feature pyramid network fuses feature maps of different scales through upsampling and downsampling operations, and outputs the fused feature map through a fully connected layer and a pooling layer.

5. According to the method of fast feature extraction and matching in low light and weak texture environment of claim 2, it is characterized in that: The feature point extraction module uses non-maximum suppression and threshold segmentation methods to extract feature points in the feature map.

6. According to claim 2, a fast feature extraction and matching method in a low-light and weak-texture environment, characterized in that: When the self-supervised training mechanism is used to iteratively optimize the image feature extraction model, it includes: generating images of different resolutions through an image pyramid, and then performing adaptive rotation and translation transformations on the image, judging whether the images before and after the transformation meet the designed constraints. If so, the images before and after the transformation are input into the image feature extraction model, and the image feature extraction model is trained with minimization of the designed photometric error function as the training optimization goal.

7. According to claim 6, a fast feature extraction and matching method in a low-light and weak-texture environment, characterized in that: The constraints of the self-supervised training mechanism are designed as follows: ； in, For distance, is the angle, , are the image coordinates before and after the transformation, respectively.

8. According to claim 6, a fast feature extraction and matching method in a low-light and weak-texture environment, characterized in that: The designed photometric error function is: ； in, is the window size around the feature point, are all pixels within the feature point window, , are the image coordinates before and after the transformation, i and j represent the distance from the feature point in the X and Y directions, , Respectively represent the grayscale values ​​of the corresponding pixels of the image before and after the transformation.

9. According to the method of claim 1, a fast feature extraction and matching method in a low-light and weak-texture environment, characterized in that: Preprocess the unlabeled image data, including: Step 1-1, use Gaussian filtering to denoise the image data and obtain: ； in, is the standard deviation of the Gaussian distribution, are the horizontal and vertical coordinates of the image; Step 1-2, grayscale the image using the histogram equalization method, that is, ； in, Represents the grayscale range, is the number of pixels in the image, Grayscale The number of pixels; Steps 1-3 use bilinear interpolation method to adjust the image size.

10. A fast feature extraction and matching system in a low-light and weak-texture environment for implementing the method described in any one of claims 1 to 9, characterized in that: include: A data set construction unit preprocesses the unlabeled image data and constructs an unlabeled data set; Image feature extraction model construction and training unit: constructs an image feature extraction model and uses a self-supervised training mechanism to iteratively optimize and train the image feature extraction model based on the data set; The feature point matching unit uses the trained feature extraction model to perform feature extraction and matching on the two collected image data, eliminates the mismatched points, and obtains a set of matched feature points.

Images