Image feature point matching method and system

Abstract

The application relates to the technical field of computer vision, and discloses an image feature point matching method and system to improve matching accuracy. The method comprises the following steps: extracting shallow features by using an encoder-decoder; performing channel dimension aggregation on the output of the decoder, generating a key point attention map through a unit subtraction convolution operation, then performing residual fusion on the key point attention map and the output feature map of the decoder to obtain a fused feature map, and then obtaining the integral pixel coordinates of each key point by grid sampling on the key point heat map obtained based on the fused feature map; extracting deeper semantic features and multi-scale features of an image from the output of the encoder, fusing the semantic features and the multi-scale features to generate semantic descriptors of each key point, fusing the semantic descriptors and basic descriptors obtained according to the output of the encoder into final descriptors of each key point; and performing feature point matching on the final descriptors of each key point and outputting.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to an image feature point matching method and system. BACKGROUND

[0002] Image feature point detection and matching is one of the key technologies for automatic driving systems to improve environmental perception and decision-making capabilities. This technology helps vehicles estimate the three-dimensional information of the target objects in front by detecting and matching feature points in binocular images, thereby providing reference data for obstacle distance measurement, drivable area detection, and other tasks. Especially in various complex scenarios such as urban streets, highways, and dynamic obstacles, this technology can help autonomous vehicles respond quickly to changes in the surrounding environment, thereby providing a safer and more reliable autonomous driving experience.

[0003] For example, during vehicle travel, by real-time detection and matching of left and right binocular images captured by binocular cameras, the system can estimate the distance of the vehicle in front, identify the slope and undulation of the road, and other features. When encountering obstacles or changes in road surface undulation, combined with depth and motion information, it assists in path planning to ensure stable driving of the vehicle in complex road conditions.

[0004] However, the existing methods mainly have two problems, one is that in the feature point detection stage, most methods only focus on local details in the image, ignoring the rich context semantic information in the entire image, resulting in insufficient learning of the descriptor, and prone to false matching in the matching process due to noise interference. The second is that most existing models only rely on two frames of images for matching, which is difficult to cope with complex and dynamic scenes, prone to matching interruption, and unable to guarantee the consistency and continuity of feature points, with poor stability. SUMMARY

[0005] The present application aims to disclose an image feature point matching method and system to improve the accuracy of matching.

[0006] To achieve the above purpose, the image feature point matching method disclosed by the present application comprises constructing a data set, dividing the data set into a training set and a validation set, network structure construction, training and validation, and deploying the trained network structure for testing. The network structure includes an encoder-decoder architecture, a difference interaction attention module, a hierarchical feature extraction module, and an output module. The method comprises:

[0007] Step S1, shallow features are extracted using the encoder-decoder architecture;

[0008] Step S2, the output of the decoder in the encoder-decoder architecture is extracted by a differential interaction attention module to extract key points in the image, and the processing process comprises: firstly aggregating the channel dimension of the output feature map of the decoder, then generating a key point attention map through a unit subtraction convolution operation, then performing residual fusion on the key point attention map and the original output feature map of the decoder to obtain a fusion feature map with enhanced key point attention, and then obtaining the integer pixel coordinates of each key point through grid sampling based on the fusion feature map;

[0009] Step S3, the output of the encoder in the encoder-decoder architecture is extracted by a hierarchical feature extraction module to extract deeper semantic features and multi-scale features of the image, the semantic features and the multi-scale features are fused to generate semantic descriptors of each key point, and the semantic descriptors and basic descriptors obtained according to the output of the encoder are fused to generate final descriptors of each key point;

[0010] Step S4, the final descriptors of each key point are matched based on the output module and output.

[0011] Preferably, in step S4, it comprises:

[0012] Step S41, key points are detected in the image A, the image B and the image C, and corresponding descriptors are extracted;

[0013] Step S42, a MNN (Mutual Nearest Neighbors) algorithm with an introduced distance ratio screening mechanism is applied to obtain the matching relationship among the three images , and ;

[0014] Step S43, the index of the image A is extracted from and , and the intersection is obtained to obtain the common first key point index;

[0015] Step S44, a first type index mapping of the image A to the image B and the image C is constructed, the position of the first key point index in the three images A, B and C is obtained, and all other key points whose index is not in the first key point index are removed;

[0016] Step S45, a second type index mapping of the image B to the image C is constructed, and the key points whose corresponding points in the image B are also in the second type index are screened out from the first key point index:

[0017] Step S46, respectively verifying whether the target points of the respective key points belonging to the intersection of the first type index and the second type index screened from the image A and remapped to the image C via the matching points in the image B are consistent with the matching points, if consistent, retaining the corresponding matching pairs, otherwise, filtering the corresponding matching relationship.

[0018] Preferably, the image A and the image B are a pair of images synchronously collected by a binocular camera, and the image C is a monocular image collected by the binocular camera and adjacent in time sequence to the image pair.

[0019] Preferably, the encoder performs three downsampling operations on the input image by three downsampling modules; each time the downsampling is performed, the size of the feature map is reduced by 2 times, and the number of channels of the feature map is increased.

[0020] Preferably, the specific operation of the sampling module is to increase the number of channels of the feature map through a 3x3 standard convolution operation, then apply batch normalization processing, and use a ReLU activation function, and finally perform a maximum pooling operation on the feature map to reduce the width and height of the feature map by 2 times.

[0021] Preferably, the decoder is composed of two upsampling modules, and if , , respectively represent the feature maps of the encoder output , and size, the decoding process is specifically as follows: first, the feature map is upsampled:

[0022] ;

[0023] Among them, represents the upsampled feature map , represents a bilinear interpolation operation, will be spliced along the channel dimension with the feature map of the corresponding scale of the encoder, represents a multi-modal bilinear pooling to generate a fusion feature containing cross-modal interaction information, and then a 3x3 standard convolution is used to adjust the number of channels of the feature map to obtain the feature map :

[0024] ;

[0025] Among them, represents a splicing operation; and the following upsampling operation is performed to finally output a high-resolution feature map :

[0026]

[0027] wherein the features outputted by the decoder will be used to extract the key points, the features outputted by the encoder will be used to obtain the descriptors.

[0028] Preferably, in step S2, comprising:

[0029] Step S21, performing aggregation on the channel dimension of the output feature map of the decoder , and then generating a key point attention map through a unit subtraction convolution operation, the calculation formula being:

[0030] ;

[0031] wherein, represents the key point attention map, A is the receptive field area of the convolution kernel, is the coordinate position in the feature map, represents the offset in the convolution kernel;

[0032] Step S22, performing residual fusion between and the original feature map to obtain a feature map , so as to enhance the attention of the model to the key points:

[0033] ;

[0034] wherein, represents an element-wise multiplication operation;

[0035] Step S23, performing bilinear interpolation once on the feature map to restore the input image size, and then performing a standard convolution to restore the channel number to 1, and outputting a key point heat map;

[0036] Step S24, constructing a 3x3 grid with the key point coordinate (x, y) as the center, mapping each point to the key point heat map, and obtaining the score value of each grid point in the key point heat map from the key point heat map. Softmax Step S25, performing normalization processing on the score obtained by sampling to obtain the probability distribution of each point in the grid :

[0037] ;

[0038] wherein, e is the base of natural logarithm, represents the score value of the i-th row and j-th column point in the 3x3 grid centered on the key point (x, y) in the key point heat map, and m and n are the row and column indexes traversing the grid;

[0039] Step S25, according to the score probability, the weight of each point in the grid is calculated, and the key point coordinates of sub-pixel accuracy are weighted and averaged :

[0040] .

[0041] Preferably, in step S3, specifically includes:

[0042] Step S31, cross-scale feature enhancement: using three parallel grouped convolutions to learn the features of the input feature map at different scales, the convolution kernel size of each convolution layer is 3, 5 and 7 respectively, and the corresponding padding strategy is used to keep the spatial dimension of the feature map unchanged, and the three features are spliced with the original feature ; At the same time, a multi-layer perceptron composed of two convolution layers is designed, the first convolution layer expands the channel number of the spliced feature map to four times the input channel number, and then performs nonlinear transformation through the ReLU activation function, and the second convolution layer reduces the channel number to half of the output channel number; The multi-scale features output by the cross-scale feature enhancement branch are represented as:

[0043]

[0044] Wherein, represents a grouped convolution with a convolution kernel size of ; MLP represents a multi-layer perceptron;

[0045] Step S32, semantic feature enhancement: first, the feature map input by the encoder part is applied with a sliding window average pooling operation, which slides a fixed size window on the feature map and calculates the average value in the window to capture local spatial information; After sliding window average pooling, the feature map is sent to a multi-layer perceptron composed of two 1x1 convolutions to further extract the interaction information between channels, and then the Sigmoid activation function is used to generate the attention weight of each channel, reflecting the importance of each channel; Finally, the up-sampling attention weight map is multiplied with the original feature map , so as to adjust the activation strength of each channel, enhance important features and suppress unimportant features; The semantic features output by the semantic feature enhancement branch are represented as:

[0046] ;

[0047] Wherein, SWAP represents a sliding window average pooling operation, represents a Sigmoid activation function;

[0048] ​​Step S33, feature fusion: feature fusion is divided into two stages, one is the fusion of semantic features and multi-scale features , and the other is the fusion of intermediate features output by step (1) and hierarchical features; first, semantic features and multi-scale features are spliced along the channel dimension, and after passing through an instance normalization layer and a convolution layer, they are recorded as hierarchical features : ;

[0049] Among them, IN represents an instance normalization operation; bicubic interpolation is used on feature vectors and to perform spatial interpolation according to the key points and the original image size to generate feature descriptors matching the target size: semantic descriptors and basic descriptors; finally, the semantic descriptors and the basic descriptors are input into a fusion layer to output the final descriptors :

[0050] ;

[0051] Among them, respectively represent semantic descriptors and basic descriptors, and FusL represents a fusion layer.

[0052] Preferably, the fusion layer successively comprises a first fully connected layer, a RELU activation function processing layer, a Dropout layer, a second fully connected layer, and a Sigmoid activation function processing layer.

[0053] To achieve the above purpose, the application further discloses an image feature point matching system, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above method when executing the computer program.

[0054] The application has the following beneficial effects:

[0055] 1. In the application, the difference interaction attention module can be regarded as imitating the behavior of the human visual system when identifying images, that is, starting from global features and then gradually focusing on local detailed features. This coarse-to-fine processing method helps to identify images under different perspectives. The module has fewer learnable parameters, which are mainly concentrated in the batch normalization layer. Since the number of parameters is small, the calculation amount and computational overhead during forward propagation are low, thereby effectively improving the running efficiency of the model.

[0056] 2. The hierarchical feature extraction module extracts deeper semantic features and multi-scale features of the image, fuses the semantic features and the multi-scale features to generate semantic descriptors of each key point, and generates the final descriptors of each key point by fusing the semantic descriptors and the basic descriptors obtained according to the output of the encoder, thereby enhancing the response to important features.

[0057] 3、The matching of the application is not only suitable for feature point matching of left and right images in binocular vision, but also suitable for feature point matching between multiple frames of related images, which improves the matching accuracy, the expansibility and the adaptability to complex and variable dynamic scenes, and guarantees the consistency, continuity and stability of the feature points.

[0058] The application will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0059] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application. The embodiments of the application illustrated in the drawings, and their description thereto, are presented to explain the application and are not intended to limit the application unduly.

[0060] Figure 1 is a schematic block diagram of the network structure of the image feature point matching disclosed in embodiment 1 of the application.

[0061] Figure 2 is a schematic block diagram of the structure of the shallow feature extraction module disclosed in embodiment 1 of the application.

[0062] Figure 3 is a schematic flow chart of the multi-frame matching algorithm disclosed in embodiment 1 of the application. DETAILED DESCRIPTION

[0063] The embodiments of the application will be described in detail below with reference to the accompanying drawings, but the application can be implemented in various different ways limited and covered by the claims.

[0064] Embodiment 1

[0065] This embodiment adopts the mode of detecting first and then matching, and discloses an image feature point matching method.

[0066] As shown in Figure 1 , in the detection stage, the shallow features of the image are first extracted by a feature extraction network similar to the U-Net structure, then the key points with representative in the image are automatically focused by the difference interactive attention module, and finally the deeper semantic and multi-scale hierarchical features of the image are extracted in a local-to-global manner by the hierarchical feature enhancement module, so as to obtain dense descriptors; as shown in Figure 3 , in the matching stage, the key points matched by the MNN method are screened by using the distance ratio idea, the key points with low confidence are filtered, and the method is applied to multiple frames of images to correct the wrong matching in the two frames of images. The specific implementation steps are as follows:

[0067] I. Extract the shallow features of the image for subsequent detection of key points and descriptors.

[0068] AsFigure 1 As shown in Part A, the feature extraction network consists of two parts: an encoder and a decoder.

[0069] The encoder consists of three downsampling modules, and its specific structure is as follows: Figure 2 The encoder section is shown. The input image undergoes three downsampling operations. With each downsampling, the feature map size is reduced by a factor of 2, while the number of channels in the feature map is increased. Specifically, the number of channels in the feature map is increased through a standard 3×3 convolution operation, followed by batch normalization and the ReLU activation function. Finally, a max-pooling operation is performed on the feature map, reducing its width and height by a factor of 2.

[0070] Let the input image be Repeat this downsampling encoding operation three times:

[0071]

[0072] in, Conv This indicates a convolution operation with a kernel size of 3×3 and padding of 1. BN This indicates a batch normalization operation. ReLU This represents the activation function. MaxPool This indicates a max pooling operation with a pooling kernel size of 2×2. , , These represent the encoder outputs. , and Feature map of size.

[0073] Decoder: Consists of two upsampling modules, with the specific structure as follows Figure 2 The decoder section is shown below. Its input is a sequence of multi-scale feature maps output by the encoder. First, the feature maps... Perform upsampling operation:

[0074] ;

[0075] in, express Upsampled feature map This indicates a bilinear interpolation operation. The feature maps will be concatenated along the channel dimension and the corresponding scale of the encoder, and then the number of channels of the feature maps will be adjusted by a standard 3×3 convolution to obtain the final feature map. :

[0076] ;

[0077] in, This indicates a stitching operation. The following operations then output a high-resolution feature map. :

[0078]

[0079] represents the multi-modal bilinear pooling to generate the fusion features containing cross-modal interaction information. Through the processing of the above feature extraction network, the shallow features of the input image are extracted, in which the features output by the decoder will be used to extract the key points, and the features output by the encoder are used to obtain the descriptors.

[0080] II. Key point extraction based on decoding features.

[0081] In order to capture the most discriminative and representative key regions in the image and improve the accuracy of subsequent matching, the difference interaction attention module is used to extract the key points in the image. As shown in part B of Figure 1 , first, the output feature map of the decoder is aggregated in the channel dimension, and then a unit subtraction convolution operation is performed to generate a key point attention map. Unlike traditional convolution, the unit subtraction convolution highlights the key points in the feature map through subtraction rather than addition, and the calculation formula is:

[0082] ;

[0083] wherein, represents the key point attention map, A is the receptive field area of the convolution kernel, is the coordinate position in the feature map, represents the offset in the convolution kernel, which is used for movement and calculation in the feature map. The is residually fused with the original feature map to obtain the feature map to enhance the model's attention to key points:

[0084] ;

[0085] wherein, represents the element-wise multiplication operation. Then the feature map is bilinearly interpolated once to restore it to the size of the input image. After a standard convolution restores the channel number to 1, the key point heat map is output. Finally, grid sampling is used to obtain the coordinates of each key point from the key point heat map.

[0086] Among them, grid sampling is a coordinate extraction method. The specific process is: the model outputs a key point heat map (the size is the same as the original picture, usually HxW), and each pixel position has a "key point possibility score"; grid sampling is to establish a regular pixel grid on the key point heat map, and to extract the position with the highest score from the grid and convert it to a specific coordinate in a method similar to sampling a specific value from a probability distribution.

[0087] In order to obtain more accurate key point positions, a 3x3 grid is constructed around the key point coordinates (x, y), and each point is mapped to the key point heat map to obtain the score value of each grid point in the key point heat map The scores obtained by sampling are subjected to Softmax normalization to obtain the probability distribution of each point in the grid :

[0088] ;

[0089] This distribution reflects the probability of the key point at each position in the grid, where e is the base of the natural logarithm, represents the score value of the i-th row and j-th column point in the 3x3 grid centered on the key point (x, y) in the key point heat map, and m and n are the row and column indices of the grid traversal. Finally, according to the score probability, the weight of each point in the grid is calculated, and the key point coordinates with sub-pixel accuracy are obtained by weighted averaging :

[0090] .

[0091] The above key point optimization method takes into account the score values of multiple points around the key point, avoiding the uncertainty of a single peak point, and helps to improve the robustness of the algorithm.

[0092] III. Descriptor acquisition based on encoded features.

[0093] After extracting the key points, a hierarchical feature enhancement module is used to extract deeper semantic and multi-scale hierarchical features of the image to generate dense descriptors. For example Figure 1 As shown in Part C of the above-mentioned method, the module is composed of cross-scale feature enhancement, semantic feature enhancement and feature fusion. The design of each part is as follows:

[0094] Cross-scale feature enhancement: three parallel grouped convolutions are used to learn the features of the input feature map at different scales, and the kernel size of each convolution layer is 3, 5 and 7 respectively. The corresponding padding strategy is used to keep the spatial dimension of the feature map unchanged, and the three features are combined with the original feature is concatenated; in order to further extract and integrate multi-scale features, a multi-layer perceptron composed of two convolutional layers is designed, the first convolutional layer expands the channel number of the concatenated feature map to four times the input channel number, followed by a ReLU activation function for nonlinear transformation, and the second convolutional layer reduces the channel number to half of the output channel number. The multi-scale features output by the cross-scale feature enhancement branch may be represented as:

[0095]

[0096] wherein, represents a convolution kernel size of grouped convolution.

[0097] Semantic feature enhancement: traditional channel attention mechanism mainly relies on global average pooling operation to capture global clues, while ignoring the importance of local dependence. This embodiment adopts sliding window average pooling operation, effectively constructing local dependence, which is crucial for feature description task. As shown in part C of Figure 1 , firstly, the feature map input by the encoder part is applied to the sliding window average pooling operation, which captures local spatial information by sliding a fixed size window on the feature map and calculating the average value in the window; after sliding window average pooling, the feature map is sent to a multi-layer perceptron composed of two 1x1 convolutions, further extracting the interaction information between channels, and then passing through the Sigmoid activation function to generate the attention weight of each channel, reflecting the importance of each channel; finally, the up-sampled attention weight map is multiplied by the original feature map to adjust the activation strength of each channel, enhance important features and suppress unimportant features. The semantic features output by the semantic feature enhancement branch may be represented as:

[0098] ;

[0099] wherein, SWAP represents a sliding window average pooling operation, MLP and represent a multi-layer perceptron and a Sigmoid activation function, respectively.

[0100] Feature fusion: as shown in part C of Figure 1 , feature fusion is divided into two stages, one is the fusion of semantic features and multi-scale features , and the other is the fusion of intermediate features output by step (1) and hierarchical features. Firstly, the semantic features and multi-scale features are concatenated along the channel dimension, and then recorded as hierarchical features after passing through an instance normalization layer and a convolutional layer :

[0101] ;

[0102] where IN denotes instance normalization operation. For feature vector and , bicubic interpolation is used to perform spatial interpolation according to the key points and the original image size to generate feature descriptors matching the target size (usually set to 128 dimensions) : semantic descriptors and base descriptors; finally, the semantic descriptors and base descriptors are input into the fusion layer to output the final descriptors :

[0103] ;

[0104] where denote semantic descriptors and base descriptors respectively, and FusL denotes the fusion layer, as shown in Figure 1 , which consists of two fully connected layers, RELU activation function, Dropout layer and Sigmoid activation function.

[0105] Four, feature point matching based on key points and descriptors.

[0106] In the matching stage, a distance ratio screening mechanism is introduced to optimize the MNN algorithm. Let the feature point sets extracted from image A and image B by the above steps be and , and the descriptor sets be and , and the matching set obtained by the MNN algorithm be . For each descriptor , find its nearest neighbor and second nearest neighbor in , and calculate the Euclidean distance between them:

[0107] .

[0108] Then calculate the ratio of the nearest neighbor distance to the second nearest neighbor distance for each feature point: .

[0109] Set a threshold , if , keep the matching pair, otherwise filter it out, and finally get the matching pair set: .

[0110] By calculating the ratio of the nearest neighbor distance and the second nearest neighbor distance, the algorithm of the embodiment effectively filters out unreliable matches with a ratio below a set threshold, which is an important supplement to the nearest neighbor matching algorithm, making the matching process more delicate and robust.

[0111] In order to better maintain the consistency and continuity of the feature points, the embodiment further discloses a multi-frame image feature point matching method, which further corrects the false matches by using the matching relationship among three images, and the detailed process of the algorithm is as shown in Figure 3 , and the specific steps are as follows:

[0112] (a) Key point detection is performed on the images A, B and C, and corresponding descriptors are extracted.

[0113] (b) The optimized MNN algorithm is applied to obtain the matching relationship among the three images , and .

[0114] (c) The indexes of the image A are extracted from and , and the intersection is obtained to obtain the common key point index , wherein i, j and k are the i-th feature point of the image A, the j-th feature point of the image B and the k-th feature point of the image C.

[0115] (d) The index mapping of the image A to the images B and C is constructed and , the position of the common key point index in the images A, B and C is obtained, and all key points whose indexes are not in are removed.

[0116] (e) The index mapping of the image B to the image C is constructed , and it is checked whether the corresponding point of the key point in the image A in the image B is also in : .

[0117] (f) For each key point in the image A, it is verified whether the point in the image C to which the corresponding point of the key point in the image B is mapped through is consistent with the point obtained through the matching of : .

[0118] If consistent, the matching pair is reserved, otherwise, the matching is filtered out, wherein steps (e) and (f) are collectively referred to as matching consistency verification.

[0119] After distance ratio screening and multi-frame image correction matching, a high-confidence key point matching result is obtained, and the entire feature point matching process is completed.

[0120] In summary, the matching method of the embodiment uses a distance ratio screening mechanism and proposes a multi-frame image feature point matching strategy. By constructing the matching relationship between three images and verifying the matching consistency, the robustness of the matching is further enhanced. In particular, when dealing with dynamic scenes or long-time sequence images, the consistency and continuity of the feature points can be better maintained. Embodiment

[0121] Corresponding to the above embodiment, the embodiment discloses an image feature point matching system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method is realized, which comprises constructing a data set, dividing the data set into a training set and a validation set, network structure construction, training and validation, and deploying the trained network structure for testing. The network structure comprises an encoder-decoder architecture, a difference interaction attention module, a hierarchical feature extraction module, and an output module. The method comprises:

[0122] Step S1, shallow features are extracted using the encoder-decoder architecture.

[0123] Step S2, the output of the decoder in the encoder-decoder architecture is used to extract key points in the image using the difference interaction attention module. The processing process includes: first aggregating the output feature map of the decoder in the channel dimension, then generating a key point attention map through unit subtraction convolution operation, then performing residual fusion on the key point attention map and the original decoder output feature map to obtain a fusion feature map with enhanced key point attention, and then obtaining the integer pixel coordinates of each key point through grid sampling based on the fusion feature map.

[0124] Step S3, the output of the encoder in the encoder-decoder architecture is used to extract deeper semantic features and multi-scale features of the image based on the hierarchical feature extraction module. After fusing the semantic features and the multi-scale features, a semantic descriptor of each key point is generated. The semantic descriptor and the base descriptor obtained according to the output of the encoder are fused to generate a final descriptor of each key point.

[0125] Step S4, feature point matching is performed on the final descriptor of each key point based on the output module and output.

[0126] Preferably, in step S4, it comprises:

[0127] Step S41, key point detection is performed on image A, image B, and image C, and corresponding descriptors are extracted.

[0128] Step S42, the MNN algorithm applying the introduced distance ratio screening mechanism obtains the matching relationship among the three images , and .

[0129] Step S43, the index of image A is extracted from and , and the intersection is obtained to obtain the common first key point index.

[0130] Step S44, the first type index mapping of image A to image B and image C is constructed, the position of the first key point index in the three images A, B and C is obtained, and all other key points whose index is not in the first key point index are removed.

[0131] Step S45, the second type index mapping of image B to image C is constructed, and the key points in image B whose corresponding points are also in the second type index are screened out from the first key point index.

[0132] Step S46, the target points of the key points in image A screened out and belonging to the intersection of the first type index and the second type index are verified whether they are consistent with the points matched by the matching points in image B and remapped to image C, if consistent, the corresponding matching pairs are reserved, otherwise, the corresponding matching relationship is filtered.

[0133] Preferably, image A and image B are a pair of images synchronously collected by a binocular camera, and image C is a monocular image collected by the binocular camera and adjacent in time sequence to the image pair.

[0134] Preferably, the encoder performs three downsampling operations on the input image by three downsampling modules; each time the downsampling is performed, the size of the feature map is reduced by 2 times, and the number of channels of the feature map is increased.

[0135] Preferably, the specific operation of the sampling module is: the number of channels of the feature map is increased by a 3x3 standard convolution operation, then batch normalization processing is applied, and a ReLU activation function is used, and finally a maximum pooling operation is performed on the feature map to reduce the width and height of the feature map by 2 times.

[0136] Preferably, the decoder is composed of two upsampling modules, and if , , respectively represent the feature maps with sizes of , and output by the encoder, the decoding process is specifically: first, the feature map ​Up-sampling operation is performed:

[0137] ;

[0138] wherein, represents the up-sampled feature map, represents a bilinear interpolation operation, will be spliced along the channel dimension with the feature map of the corresponding scale of the encoder, represents a multi-modal bilinear pooling to generate a fusion feature containing cross-modal interaction information, and then a 3×3 standard convolution is performed to adjust the channel number of the feature map to obtain the feature map :

[0139] ;

[0140] wherein, represents a splicing operation; and then the following up-sampling operation is performed to finally output a high-resolution feature map :

[0141]

[0142] wherein, the feature output by the decoder will be used to extract key points, and the feature output by the encoder is used to obtain descriptors.

[0143] Preferably, in step S2, the following steps are included:

[0144] Step S21, aggregate the output feature map of the decoder in the channel dimension, and then generate a key point attention map through a unit subtraction convolution operation, and the calculation formula is:

[0145] ;

[0146] wherein, represents the key point attention map, A is the receptive field area of the convolution kernel, is the coordinate position in the feature map, represents the offset in the convolution kernel.

[0147] Step S22, perform residual fusion on and the original feature map to obtain a feature map to enhance the attention of the model to the key points:

[0148] ;

[0149] wherein, represents an element-wise multiplication operation.​

[0150] Step S23: Process the feature map Perform a bilinear interpolation to restore the image to its original size, then perform a standard convolution to restore the number of channels to 1, and finally output a keypoint heatmap.

[0151] Step S24: Construct a 3×3 grid centered on the keypoint coordinates (x, y), map each point onto the keypoint heatmap, and obtain the score value of each grid point from the keypoint heatmap. The scores obtained from sampling are then analyzed. Softmax Normalization is performed to obtain the probability distribution of each point within the grid. :

[0152] ;

[0153] Where e is the base of the natural logarithm, This represents the score of the point in the i-th row and j-th column within a 3×3 grid centered at the keypoint (x, y) in the keypoint heatmap, where m and n are the row and column indices for traversing the grid.

[0154] Step S25: Calculate the weight of each point in the grid based on the score probability, and calculate the key point coordinates with sub-pixel accuracy by weighted averaging. :

[0155] .

[0156] Preferably, step S3 specifically includes:

[0157] Step S31, Cross-scale Feature Enhancement: Three parallel grouped convolutions are used to learn features of the input feature map at different scales. The kernel sizes of each convolutional layer are 3, 5, and 7, respectively. Appropriate padding strategies are used to maintain the spatial dimension of the feature map. The three features are then combined with the original features. The feature maps are concatenated; a multilayer perceptron consisting of two convolutional layers is designed. The first convolutional layer expands the number of channels in the concatenated feature map to four times the number of input channels, followed by a non-linear transformation using the ReLU activation function. The second convolutional layer reduces the number of channels to half the number of output channels. The multi-scale features output by the cross-scale feature enhancement branch are further elaborated. Represented as:

[0158] ;

[0159] in, Indicates the kernel size as Grouped convolution.

[0160] Step S32, Semantic Feature Enhancement: First, the feature map input to the encoder is partially enhanced. The sliding window average pooling operation is applied to capture local spatial information by sliding a fixed size window on the feature map and calculating the average value in the window. After the sliding window average pooling, the feature map is sent to a multi-layer perceptron composed of two 1x1 convolutions to further extract the interaction information between channels, and then the Sigmoid activation function is used to generate the attention weight of each channel, reflecting the importance of each channel. Finally, the up-sampled attention weight map is multiplied with the original feature map to adjust the activation strength of each channel, enhance important features and suppress unimportant features. The semantic feature enhancement branch outputs the semantic feature

[0161] ;

[0162] wherein, SWAP represents the sliding window average pooling operation, MLP and represent the multi-layer perceptron and the Sigmoid activation function, respectively.

[0163] Step S33, feature fusion: the feature fusion is divided into two stages, one is the fusion of the semantic feature and the multi-scale feature , and the other is the fusion of the intermediate feature output by step (1) and the hierarchical feature; first, the semantic feature and the multi-scale feature are spliced along the channel dimension, and then an instance normalization layer and a convolution layer are used to generate the hierarchical feature :

[0164] ;

[0165] wherein, IN represents the instance normalization operation; the bicubic interpolation is used for the feature vectors and to perform spatial interpolation according to the key points and the original image size to generate feature descriptors with a target size, i.e., the semantic descriptor and the basic descriptor; finally, the semantic descriptor and the basic descriptor are input into the fusion layer to output the final descriptor :

[0166] ;

[0167] wherein, represent the semantic descriptor and the basic descriptor, respectively, and FusL represents the fusion layer.

[0168] Preferably, the fusion layer is composed of a first fully connected layer, a RELU activation function processing layer, a Dropout layer, a second fully connected layer and a Sigmoid activation function processing layer in sequence.​​

[0169] In summary, the method and system disclosed in the above embodiments of the present application have at least the following beneficial effects:

[0170] 1. In the present application, the difference interaction attention module can be regarded as imitating the behavior of the human visual system when identifying images, i.e., starting from global features and then gradually focusing on local detailed features. This coarse-to-fine processing method helps to identify images under different perspectives. The module has fewer learnable parameters, mainly concentrated in the batch normalization layer. Due to the small number of parameters, the computational cost during forward propagation is low, thereby effectively improving the running efficiency of the model.

[0171] 2. The hierarchical feature extraction module extracts deeper semantic features and multi-scale features of the image, fuses the semantic features and multi-scale features to generate semantic descriptors of each key point, and generates the final descriptors of each key point by fusing the semantic descriptors and the basic descriptors obtained according to the output of the encoder, thereby enhancing the response to important features.

[0172] 3. The matching of the present application is not only suitable for feature point matching of left and right eye images in binocular vision, but also suitable for feature point matching between multiple related images. While improving the matching accuracy, the expansibility and adaptability to complex and variable dynamic scenes are also improved, and the consistency, continuity and stability of the feature points can be guaranteed.

[0173] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image feature point matching method, comprising constructing a dataset, dividing the dataset into a training set and a validation set, constructing, training, and validating a network structure, and deploying the trained network structure for testing, characterized in that, The network structure includes an encoder-decoder architecture, a differential interaction attention module, a hierarchical feature extraction module, and an output module. The method includes: Step S1: Extract shallow features using the encoder-decoder architecture; Step S2: Extract key points from the image using the differential interactive attention module from the output of the decoder in the encoder-decoder architecture. The process includes: first, aggregating the channel dimension of the output feature map of the decoder, then generating a key point attention map through unit subtraction convolution, then performing residual fusion of the key point attention map and the original output feature map of the decoder to obtain a fused feature map that enhances the attention to key points, and then obtaining the integer pixel coordinates of each key point from the key point heatmap obtained based on the fused feature map through grid sampling. Step S3: Extract deeper semantic features and multi-scale features of the image based on the hierarchical feature extraction module from the output of the encoder in the encoder-decoder architecture. After fusing the semantic features and the multi-scale features, generate semantic descriptors for each key point. Then, fuse the semantic descriptors with the basic descriptors obtained from the output of the encoder to generate the final descriptors for each key point. Step S4: Based on the output module, perform feature point matching on the final descriptors of each key point and output the results.

2. The image feature point matching method according to claim 1, characterized in that, Step S4 includes: Step S41: Perform key point detection on images A, B, and C, and extract the corresponding descriptors; Step S42: Apply the MNN algorithm with a distance ratio filtering mechanism to obtain the matching relationship between the three frames. , and ; Step S43, from and Extract the index of image A, and find the intersection to obtain the common first keypoint index; Step S44: Construct the first type of index mapping from image A to image B and image C respectively, obtain the position of the first key point index in the three images A, B and C, and remove all other key points whose index is not in the first key point index; Step S45: Construct a second type of index mapping from image B to image C, and filter out key points from the first key point index whose corresponding points in B are also in the second type of index: Step S46: Verify whether each key point selected from image A belonging to the intersection of the first type of index and the second type of index, after being matched by the matching point in image B and then mapped to the target point in image C, matches the target point in image C. If the points obtained from the matching are consistent, the corresponding matching pair is retained; otherwise, the corresponding matching relationship is filtered out.

3. The image feature point matching method according to claim 2, characterized in that, Image A and Image B are a pair of images acquired synchronously by a stereo camera, and Image C is a monocular image acquired by the stereo camera that is temporally adjacent to the pair of images.

4. The image feature point matching method according to any one of claims 1 to 3, characterized in that, The encoder performs three downsampling operations on the input image using three downsampling modules; each downsampling operation reduces the size of the feature map by a factor of 2, while increasing the number of channels in the feature map.

5. The image feature point matching method according to claim 4, characterized in that, The specific operation of the sampling module is as follows: the number of channels of the feature map is increased by a standard 3×3 convolution operation, then batch normalization is applied, and the ReLU activation function is used. Finally, a max pooling operation is performed on the feature map to reduce the width and height of the feature map by a factor of 2.

6. The image feature point matching method according to claim 5, characterized in that, The decoder consists of two upsampling modules. , , These represent the encoder outputs. , and The feature map of a certain size is decoded as follows: First, the feature map is processed... Perform upsampling operation: ； in, express Upsampled feature map This represents the bilinear interpolation operation. It will stitch together the feature maps along the channel dimension and the corresponding scale of the encoder. Multimodal bilinear pooling is used to generate fused features that include cross-modal interaction information, and then the number of channels in the feature map is adjusted by a standard 3×3 convolution to obtain the feature map. : ； in, This indicates a concatenation operation; the following upsampling operation is then used to finally output a high-resolution feature map. : Among them, the features output by the decoder The features output by the encoder will be used to extract key points. This is used to retrieve the descriptor.

7. The image feature point matching method according to claim 6, characterized in that, Step S2 includes: Step S21: Process the output feature map of the decoder. Aggregate along the channel dimension, and then generate a keypoint attention map through a unit subtraction convolution operation. The calculation formula is as follows: ； in, This represents the keypoint attention map, where A is the receptive field region of the convolution kernel. It is the coordinate position in the feature map. This represents the offset within the convolution kernel; Step S22, to Compared with the original feature map Feature maps are obtained by performing residual fusion. To enhance the model's focus on key points: ； in, This represents the element-wise multiplication operation; Step S23: Process the feature map Perform a bilinear interpolation to restore the input image size, then perform a standard convolution to restore the number of channels to 1, and output a key point heatmap; Step S24: Construct a 3×3 grid centered on the keypoint coordinates (x, y), map each point onto the keypoint heatmap, and obtain the score value of each grid point from the keypoint heatmap. The scores obtained from sampling are then analyzed. Softmax Normalization is performed to obtain the probability distribution of each point within the grid. : ； Where e is the base of the natural logarithm, This represents the score of the point in the i-th row and j-th column within a 3×3 grid centered at the keypoint (x, y) in the keypoint heatmap, where m and n are the row and column indices for traversing the grid. Step S25: Calculate the weight of each point in the grid based on the score probability, and calculate the key point coordinates with sub-pixel accuracy by weighted averaging. : 。 8. The image feature point matching method according to claim 5, characterized in that, Step S3 specifically includes: Step S31, Cross-scale Feature Enhancement: Three parallel grouped convolutions are used to learn features of the input feature map at different scales. The kernel sizes of each convolutional layer are 3, 5, and 7, respectively. Appropriate padding strategies are used to maintain the spatial dimension of the feature map. The three features are then combined with the original features. The feature maps are concatenated; a multilayer perceptron consisting of two convolutional layers is designed. The first convolutional layer expands the number of channels in the concatenated feature map to four times the number of input channels, followed by a non-linear transformation using the ReLU activation function. The second convolutional layer reduces the number of channels to half the number of output channels. The multi-scale features output by the cross-scale feature enhancement branch are further elaborated. Represented as: in, Indicates the kernel size as Grouped convolution, MLP Represents a multilayer perceptron; Step S32, Semantic Feature Enhancement: First, the feature map input to the encoder is partially enhanced. A sliding window average pooling operation is applied, which captures local spatial information by sliding a fixed-size window across the feature map and calculating the average value within the window. After sliding window average pooling, the feature map is fed into a multilayer perceptron consisting of two 1×1 convolutions to further extract the interaction information between channels. Then, a sigmoid activation function is used to generate attention weights for each channel, reflecting the importance of each channel. Finally, the attention weight map is upsampled and compared with the original feature map. Multiplication is used to adjust the activation intensity of each channel, enhancing important features and suppressing unimportant features; semantic feature enhancement branches output semantic features. Represented as: ； in, SWAP This indicates a sliding window average pooling operation. This represents the Sigmoid activation function; Step S33, Feature Fusion: Feature fusion is divided into two stages, the first being semantic features. Multiscale features The integration, and the intermediate features Fusion with hierarchical features: First, semantic features and multi-scale features are concatenated along the channel dimension. After passing through an instance normalization layer and a convolutional layer, these are denoted as hierarchical features. : ； Where IN represents instance normalization operation; for feature vectors and Bicubic interpolation is used to spatially interpolate based on keypoints and the original image size to generate feature descriptors that match the target size: a semantic descriptor and a base descriptor. Finally, the semantic descriptor and the base descriptor are input into the fusion layer to output the final descriptor. : ； in, These represent the semantic descriptor and the basic descriptor, respectively. This indicates the fusion layer.

9. The image feature point matching method according to claim 8, characterized in that, The fusion layer consists of a first fully connected layer, a ReLU activation function processing layer, a Dropout layer, a second fully connected layer, and a Sigmoid activation function processing layer.

10. An image feature point matching system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 9.

Images