An image matching method based on rearrangement of cropped regions of image feature points

Abstract

The present invention discloses a method for image matching based on rearrangement of cropped regions of image feature points. In view of image rotation and blur, a local image and rearrangement method is used to enhance the stability of matching. It mainly includes image cropping, data enhancement part, segmented image rearrangement module and feature description module. In the image cropping module, we obtain a local area of ​​fixed size around the feature point according to the position of the feature point to avoid irrelevant information of the whole image interfering with feature extraction. In the data enhancement part, we Gaussian blur and random angle rotation of the original image to obtain enhanced data. In the segmented image rearrangement module, we sort these small blocks in a clockwise direction according to the distance of each segmented area relative to the center point of the image. In this way, the error caused by image rotation can be effectively reduced.

Description

Technical Field

[0001] The present invention relates to image analysis, and in particular to an image matching method based on rearrangement of cropped areas of image feature points. Background Art

[0002] Finding and describing image feature points is of great significance in the field of computer vision. By identifying key feature points in an image, we can achieve many applications, such as object recognition, image registration, and 3D reconstruction. Common manual extraction methods include Harris corner detection, SIFT (Scale-Invariant Feature Transform), and ORB (Oriented FAST and Rotated BRIEF). These methods detect corners, edges, or spots in an image and extract their local feature descriptors, making the image stable under different scales, rotations, and lighting conditions. Finding and describing feature points is widely used in computer vision. For example, in object recognition, objects are matched by comparing feature points in different images, thereby achieving object detection and recognition; in image stitching, seamless image stitching is achieved by matching feature points across images; and in SLAM (Simultaneous Localization and Mapping), feature points are extracted from images to enable robot localization and map construction. Therefore, accurate description of image feature points is crucial for achieving various computer vision tasks.

[0003] Robustness against rotation and blur is crucial in image matching. Rotational robustness ensures that the algorithm can identify and match the same object in an image rotated at different angles, which is particularly critical in practical applications because the shooting angle is often uncontrollable. Blur robustness ensures that the algorithm can still accurately match even when faced with blurred or low-resolution images, which is particularly important in low-light conditions and motion blur. Image matching algorithms that are robust to rotation and blur can handle a variety of scenarios more stably, improving matching accuracy and reliability, making them more widely applicable in computer vision tasks such as object detection, image stitching, and scene reconstruction.

[0004] However, traditional manual feature point extraction has certain disadvantages. First, the feature extractor must be manually designed, making it difficult to adapt to different scenarios and variations. Second, for complex images, feature points may be unstable or unable to extract valid information. In contrast, the advantage of neural networks lies in their ability to automatically learn abstract features from images, eliminating the need for manually designed feature extractors and avoiding human bias. Furthermore, deep learning models can be trained end-to-end using large amounts of data, adapting to a variety of complex scenarios and variations, and performing even better. Therefore, neural network feature point extraction has broad application prospects in image processing and computer vision. Summary of the Invention

[0005] The present invention solves the problems existing in the prior art and proposes an image matching method based on rearrangement of image feature point cropping areas, which is a feature point description method based on an improved Vision Transformer network model.

[0006] The present invention describes the network based on feature points, so SIFT (Scale Invariant Feature Transform) is used as a feature point search method and SIFT is used as a feature point position acquisition module.

[0007] The present invention uses a feature point as the center and obtains a local area of ​​24*24 pixels around the feature point by cropping, which serves as the image basis for feature description. This is because using the local area around the feature point rather than the entire image area in image description has multiple advantages. First, the local area description can more accurately capture the detailed information around the feature point, while the full image description may contain a large amount of redundant information, increasing the computational complexity. Second, the local area description is more robust to partial occlusion and background changes of the image because it only focuses on specific key areas and is not easily disturbed by other parts of the full image.

[0008] In order to enhance the stability of image feature points, the present invention adopts a series of image processing steps. First, the present invention performs a rotation transformation on the input local image to obtain a rotated cropped image. This step helps to make the feature points more stable in different directions. Secondly, the original image is Gaussian blurred to generate a blurred image. The impact of noise on feature point detection is reduced, thereby improving the stability of feature points. Finally, the original local image, the rotated local image and the blurred local image are input as homologous images to ensure that the algorithm can identify stable feature points under different conditions. Through these processing steps, we can more reliably obtain the feature points of the image and achieve better results in subsequent image processing or computing tasks.

[0009] The Vision Transformer model modifies the Transformer model and applies it to image classification, citing its significant advantages in image description. Through its self-attention mechanism, the Transformer captures long-range dependencies and complex contextual information within an image, enabling a more comprehensive understanding of both global features and local details. Its parallel processing capabilities improve computational efficiency and adaptability to large-scale data. Furthermore, the Transformer eliminates the need for hand-crafted feature extractors and automatically learns features, resulting in more flexible and accurate performance.

[0010] In the feature point description module of the present invention, targeted improvements are made to the Vision Transformer model. Specifically, we removed the original sequence part used for classification tasks in the design of the model. In the traditional Vision Transformer model, the classification task involves label prediction for the image, which is not required in our application scenario. Therefore, we removed this part of the structure and redesigned the decoder part of the model. After processing by the decoder, we splice the results output by the decoder, and then process them through a fully connected dimensionality reduction module to finally generate a feature descriptor for the input local image. This process effectively converts the feature information in the image into a compact descriptor form, which is convenient for subsequent feature matching and recognition tasks.

[0011] A method for image matching based on rearrangement of cropped regions of image feature points comprises the following steps:

[0012] 1) Obtain the original image, randomly rotate the original image to obtain a rotated image, randomly blur the original image to obtain a blurred image, and then splice the original image, the rotated image, and the blurred image to obtain an enhanced dataset;

[0013] 2) Use the scale-invariant feature transformation method to obtain the feature point positions from the original image in the enhanced dataset. Based on the feature point positions, obtain the local area around the feature point positions from the original image, rotated image, and blurred image in the enhanced dataset as the feature description image;

[0014] 3) cropping and rearranging the feature description image to obtain a rearranged cropped region;

[0015] 4) Input the rearranged cropped areas into the Vision Transformer model to obtain the descriptors corresponding to each cropped area. The descriptors corresponding to each cropped area are concatenated and sorted in descending order to obtain the final description vector of the local image of the feature point, thereby obtaining the feature description network.

[0016] 5) Optimize the feature description network through the loss function to obtain the optimized feature description model;

[0017] 6) Match the two images to be matched using the optimized feature description model.

[0018] In step 1), the original image is randomly rotated to obtain a rotated image, which specifically includes:

[0019] The original image is randomly rotated within the range of 50 degrees clockwise to 50 degrees counterclockwise to obtain a rotated image.

[0020] In step 1), the original image is randomly blurred to obtain a blurred image, which specifically includes:

[0021] The original image is subjected to Gaussian blurring with a random convolution kernel size of 3*3 or 5*5 to obtain a blurred image.

[0022] In step 2), the local area around the feature point position refers to a (20-28)*(20-28) pixel area within the feature point position range.

[0023] In step 3), the feature description image is cropped and rearranged, specifically including:

[0024] 3.1) First, crop the obtained feature description image to obtain multiple cropped areas and number them;

[0025] 3.2) Based on the coordinates of the center point of the cropped area, find the cropped area closest to the center point of the image and mark it as 1. Then, mark the cropped areas in the circle outside the cropped image clockwise as 2 to 4, and the circle outside as 6 to 9. Rearrange the cropped areas from small to large according to the numbering.

[0026] In step 3.2), the center coordinates of the cropped area are determined by the formula:

[0027]

[0028] Among them, (x i ,y i ) is the coordinate of the center point of the cropping area, i is the cropping area numbered from 1 to 9, (x_left, y_left) is the coordinate of the upper left corner of the cropping area, w and h are the two side lengths of the cropping area, w and h are both 8 pixels, and the size of the feature description image is 24*24 pixels.

[0029] In step 5), the final description vectors of the local image of the feature points are divided into groups of three and calculated separately. The loss function is as follows:

[0030] Loss=Loss1+Loss2; (1)

[0031]

[0032] In formula (1), Loss is the final loss function value used for back propagation;

[0033] In formula (2), n is the number of feature points of the original image, that is, the final description vector of the local image of n feature points; j is the image type, where 0 represents the original image, and when j is 1-2, it represents a rotated image and a blurred image respectively; is the final description vector of the local image of the feature point; mean() means finding the mean.

[0034] In formula (3), n is the number of feature points of the original image, that is, the final description vector of the local image of n feature points; j is the image type, when j is 1-3, it represents the original image, rotated image, and blurred image respectively; is the final description vector of the local image corresponding to the feature point; Max is the corrected loss value, ensuring that the entire loss value Loss2 is non-negative; mean() means finding the mean.

[0035] In the encoding module, we use the standard transformer module. The specific architecture of this part is as follows Figure 1 As shown in Figure 2, the transformer encoding module extracts high-level features from the image through a self-attention mechanism and a feedforward network, ensuring the accuracy and robustness of feature description.

[0036] The complete process of the network is as follows Figure 2 As shown, it includes various processing steps from input image to final feature descriptor. Figure 3 The original structure of the Vision Transformer model is shown to facilitate comparison and understanding of the improvements we have made. In the model of this invention, we focus on optimizing the accuracy and efficiency of feature point descriptions, so that it can better serve feature matching and recognition tasks in practical applications.

[0037] In order to enhance the rotation stability of the image, this paper proposes a method based on segmentation and sorting. First, the input local image is segmented into several small blocks. Then, these small blocks are sorted in a clockwise direction according to the distance of each segmented area relative to the center point of the image. In this way, the error caused by image rotation can be effectively reduced, thereby improving the robustness of the model to rotated images. The specific process is as follows: Figure 4 shown.

[0038] In the loss function design, the present invention adopts the sum of the similarity between homologous local images and the difference between the same images as the loss function of back propagation to enhance the ability of the network.

[0039] For homologous local images, the descriptors obtained by the network for the original image, the rotated image, and the blurred image are required to be as close as possible. The network matching accuracy is positively correlated with the similarity between the two. Therefore, the loss function (Loss1) is calculated as follows: n is the number of feature points in the original image, that is, the final description vector of the local image with n feature points; j is the image type, where 0 represents the original image, and when j is 1-2, it represents the rotated image and the blurred image respectively. is the final description vector of the local image of the feature point; mean() means finding the mean:

[0040]

[0041] For descriptors obtained from local images within the same image, we need the descriptors to have as large a gap as possible to better distinguish different feature points. Network matching accuracy is negatively correlated with the similarity between the two, so we set a large value (Max). The specific loss function Loss2 is expressed as follows: n is the number of feature points in the original image, i.e., the final description vector of the local image with n feature points; j is the image type, and when j is 1-3, it represents the original image, rotated image, and blurred image, respectively. is the final description vector of the local image corresponding to the feature point; Max is the corrected loss value, ensuring that the entire loss value Loss2 is non-negative; mean() means finding the mean:

[0042]

[0043] Finally, the final loss function Loss used for back propagation, where n1 is calculated as follows:

[0044] Loss = Loss1 + Loss2

[0045] This invention relates to a feature point cropping region description method based on a Vision Transformer neural network that addresses rotation and blur. The method comprises image enhancement, local image segmentation and shifting, and a feature point description network. In the data enhancement component, the original image is rotated and Gaussian blurred to create three homologous images. In the local image segmentation and shifting module, the segmented images are rearranged based on their position within the complete image, enhancing the robustness of the feature description network to rotation. In the feature description module, the present invention modifies the Vision Inference Time (VIT) model by removing the classification branch module and focusing on image description. The descriptors of the segmented parts are concatenated to obtain a complete description of the local image. This descriptor is then subjected to dimensionality reduction to obtain the final local region descriptor. The loss function consists of two parts: loss1, which measures the similarity between descriptors of local images within the same image, and loss2, which measures the similarity between homologous local images. Loss2 should be as large as possible to maximize the differences between feature point descriptors and improve matching accuracy. The sum of loss1 and loss2 is then used as the final loss value.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] (1) A local image description method based on the Vision Transformer neural network is proposed. Compared with other neural network methods, it greatly enhances the stability and accuracy of image rotation and blur.

[0048] (2) Using local images to obtain the descriptors of feature points enables the network to focus on the features of the area around the feature points, reducing the details of irrelevant information in other areas of the entire image and enhancing the accuracy of the description.

[0049] (3) The use of data enhancement technology to generate homologous images can improve the neural network's ability to recognize feature points at different angles and blur levels, enhance the network's generalization ability, and make it more robust in actual scenarios.

[0050] (4) After the input image is segmented, the segmentation map is rearranged according to the relative positions of its feature points, which enhances the network's robustness to rotation.

[0051] (5) In the loss function, the sum of the similarity between corresponding feature points and the difference between non-corresponding feature points is used as the loss value, and no standard label is required. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is the transformer encoding module;

[0053] Figure 2 It is the overall network flow diagram of the present invention;

[0054] Figure 3 Original Vision Transformer neural network flow diagram;

[0055] Figure 4 Crop image reordering instructions;

[0056] Figure 5 is the matching result between the original image and the Gaussian blurred image;

[0057] Figure 6 is the matching result between the original image and the rotated image; DETAILED DESCRIPTION

[0058] The present invention is further described in detail below with reference to the embodiments, but the protection scope of the present invention is not limited thereto.

[0059] The present invention relates to an improved feature point cropping region description method based on a Vision Transformer neural network model, which mainly includes image cropping, a data enhancement part, a segmented image rearrangement module and a feature description module.

[0060] The present invention describes the network based on feature points, so SIFT (Scale Invariant Feature Transform) is used as a feature point search method, and the SIFT method is used to obtain the position information of the feature points.

[0061] The present invention takes the feature point as the center and obtains a local area of ​​24*24 pixels around the feature point by cropping, which serves as the image basis for feature description. This is because the local area description can more accurately capture the detailed information around the feature point, while the full image description may contain a large amount of redundant information, which increases the computational complexity. Secondly, the local area description is more robust to partial occlusion and background changes of the image because it only focuses on specific key areas and is not easily interfered with by other parts of the full image. The reason for choosing an area of ​​24*24 pixels is that an area of ​​this size can focus on the detailed information around the feature point, avoid including too much irrelevant background or other features, and at the same time process less data, thereby improving computational efficiency.

[0062] In the data augmentation phase, the present invention rotates the input partial image to obtain a rotated image. The original image is then Gaussian blurred to generate a blurred image. Finally, the original image, the rotated image, and the blurred image are input as homologous images. The original image list of length n is augmented to an augmented image of length 3n.

[0063] In the segmented image rearrangement module, the input full image size is 3*224*224 pixels, and the local area size is 3*24*24 pixels. The local area is divided into 9 3×3 cropping regions starting from the upper left corner. The channel width and height of each cropping region are 3*8*8 pixels. Then we get the coordinates of the center point of each cropping region. The specific formula is as follows, where i is the cropping region number from 1 to 9, (x left ,y left ) is the coordinate of the upper left corner of the cropping area, w,h is the width of the local area, here is 24 pixels:

[0064]

[0065] Based on the obtained cropping region coordinates, the distance from each region to the image center point (112, 112) is calculated. The corner region with the closest coordinates is identified as 1. This region is then numbered 2 to 4 clockwise in the outermost circle of the cropped image, and 6 to 9 in the outermost circle. The cropping regions are then rearranged from smallest to largest according to their numbering. This ensures relative invariance between regions even when the image is rotated, effectively reducing errors caused by image rotation.

[0066] The resulting cropped patches are flattened into a sequence and fed into the feature description module one by one. The feature description module consists of three stacked Transformer encoders, each of which is composed of a multi-head self-attention mechanism and a feed-forward neural network.

[0067] Multi-head attention has h heads. For each head i, calculate the attention weight matrix A i The expression is as follows, where Q i ,k i , and V i are query, key, and value matrices, d i is the dimension of each head. :

[0068]

[0069] The output of the self-attention mechanism is passed to the feedforward neural network, which consists of two linear layers and a nonlinear activation function. The mathematical expression is as follows:

[0070] F x =Relu(w1*x+b1)*w2+b2

[0071] This network generates descriptor vectors for each segmented area. Because a complete local image is composed of these segments, the segmented segments' descriptor vectors are concatenated to obtain [x1, x2…x8, x9], where x1 through x9 are the feature vectors of the cropped area. This is then reduced through a fully connected module to obtain the final 16-dimensional descriptor.

[0072] In order to enhance the matching stability of the network, in the loss function design, the present invention adopts the sum of the similarity between homologous local images and the difference between the same images as the backpropagation loss function to strengthen the network's capabilities.

[0073] For homologous local images, the descriptors obtained by the network for the original image, the rotated image, and the blurred image are required to be as close as possible. The network matching accuracy is positively correlated with the similarity between the two. Therefore, the loss function for this part is calculated as follows, where n is the number of feature points in the original image, i.e., the final description vector of the local image with n feature points; j is the image type, where 0 represents the original image, and j = 1-2 represents the rotated image and the blurred image, respectively. is the final description vector of the local image of the feature point; mean() means finding the mean:

[0074]

[0075] For descriptors obtained from local images within the same image, we need the descriptors to have as large a gap as possible to better distinguish different feature points. Network matching accuracy is negatively correlated with the similarity between the two, so we set a large value (Max). The specific loss function Loss2 is expressed as follows: n is the number of feature points in the original image, i.e., the final description vector of the local image with n feature points; j is the image type, and when j is 1-3, it represents the original image, rotated image, and blurred image, respectively. is the final description vector of the local image corresponding to the feature point; Max is the corrected loss value, ensuring that the entire loss value Loss2 is non-negative; mean() means finding the mean:

[0076]

[0077] The final loss function Loss calculation formula used for back propagation is as follows:

[0078] Loss = Loss1 + Loss2

[0079] 1) Experimental conditions

[0080] The model was implemented using the PyTorch framework. All experiments were conducted on a Microsoft Windows Server 2019 Datacenter system running version 10.0.17763. The CPU was an Intel(R) Xeon(R) Gold 6326 CPU @ 2.90 GHz, with 256 GB of memory. Training was accelerated by an RTX A6000. The training images were from the CalTech101 dataset, which consists of images of objects from 101 categories. Each category contains approximately 40 to 800 images, for a total of approximately 9,000 images. Training images were uniformly resized to 224 × 224 pixels before being fed into the model. The model was trained end-to-end using the Adam optimization algorithm with a learning rate of 1e-4 and a batch size of 15. Convergence was achieved after 200 epochs.

[0081] 2) Experimental results

[0082] Figure 5 This is the result of feature point matching of blurred image. It can be seen that the network has a high accuracy in matching feature points of blurred image.

[0083] Figure 6 This is the feature point matching result of the image rotated at a large angle. It can be seen that the network has a high accuracy in matching feature points of the image rotated at a small angle.

[0084] The present invention adopts matching accuracy as the evaluation index.

[0085] Table 1 shows the matching accuracy of the model proposed in the present invention under 3*3 convolution kernel Gaussian blur, 5*5 convolution kernel Gaussian blur, range random angle rotation and range random angle rotation plus 3*3 convolution kernel Gaussian blur.

[0086] Table 1

[0087]

[0088] Comprehensive experimental results show that the matching model proposed in this paper has good learning and generalization capabilities; in terms of quantitative indicators and qualitative segmentation results, compared with classic and popular matching methods in the field, it has higher precision, more accurate matching effects and faster inference speed, and can effectively and efficiently cope with various complex matching scenarios.

[0089] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0093] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0094] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for image matching based on rearrangement of cropped regions of image feature points, characterized in that: The following steps are involved: 1) Obtain the original image, randomly rotate the original image to obtain a rotated image, randomly blur the original image to obtain a blurred image, and then splice the original image, rotated image and blurred image to obtain an enhanced dataset; 2) Use the scale-invariant feature transformation method to obtain the feature point positions from the original image in the enhanced dataset. Based on the feature point positions, obtain the local area around the feature point positions from the original image, rotated image, and blurred image of the enhanced dataset as the feature description image; 3) Crop and rearrange the feature description image to obtain a rearranged cropped region; 4) Input the rearranged cropped areas into the Vision Transformer model to obtain the descriptors corresponding to each cropped area. The descriptors corresponding to each cropped area are concatenated and sorted in descending order to obtain the final description vector of the local image of the feature point, completing the construction of the feature description network. Crop and rearrange the feature description image, including: 3.1) First, crop the obtained feature description image to obtain multiple cropped areas and number them; 3.2) Based on the coordinates of the center point of the cropped area, find the cropped area closest to the center point of the image and mark it as 1. Then, mark the cropped areas in the circle outside the cropped image clockwise as 2 to 4, and the circle outside as 6 to 9. Rearrange the cropped areas in ascending order according to the numbering. The coordinates of the center point of the clipping area are determined by the formula: ； in, is the center point coordinate of the cropping area, i is the cropping area number from 1 to 9, ( , ) is the coordinate of the upper left corner of the cropping area, w and h are the two side lengths of the cropping area, w and h are both 8 pixels, and the size of the feature description image is 24*24 pixels; 5) Optimize the feature description network through the loss function to obtain the optimized feature description model; The loss function is as follows: ； （1） （2） （3）； In formula (1), Loss is the final loss function value used for back propagation; In formula (2), n is the number of feature points of the original image; j is the image type; is the final description vector of the local image of the feature point; Indicates finding the mean; In formula (3), n is the number of feature points of the original image; j is the image type; is the final description vector of the local image corresponding to the feature point; Max is the corrected loss value; Indicates finding the mean; 6) Match the two images to be matched using the optimized feature description model.

2. The image matching method based on rearrangement of image feature point cropping regions according to claim 1, characterized in that: In step 1), the original image is randomly rotated to obtain a rotated image, which specifically includes: The original image is randomly rotated within the range of 50 degrees clockwise to 50 degrees counterclockwise to obtain a rotated image.

3. The image matching method based on rearrangement of image feature point cropping regions according to claim 1, characterized in that: In step 1), the original image is randomly blurred to obtain a blurred image, which specifically includes: The original image is subjected to Gaussian blurring with a random convolution kernel size of 3*3 or 5*5 to obtain a blurred image.

4. The image matching method based on rearrangement of image feature point cropping regions according to claim 1, characterized in that: In step 2), the local area around the feature point position refers to the (20~28)*(20~28) pixel area within the feature point position range.

Images