A matching method, device and storage medium for a significantly rotated image

Abstract

The present invention relates to a matching method, device, and storage medium for significantly rotated images. The method comprises the following steps: inputting a reference image and a target image and performing preprocessing; based on the inherent translation invariance of a convolutional neural network, simplifying the rotated image matching problem to a corresponding point matching problem using a group equivariant neural network modeled based on the plane rotation group SO2; constructing and training a rotation equivariant image feature extractor; establishing two matching routes, and applying a nearest neighbor matching algorithm to the image features of the images in each matching route; and performing a primary filtering, backprojection, merging, and secondary filtering on the matching results of the two matching routes using a dual rotation equivariant excitation mixer, to output a matching relationship of dense corresponding points in the reference and target images. Compared with the prior art, the present invention can overcome the effects of significant illumination changes and achieve image matching under all rotation angles.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a matching method, device and storage medium for a significantly rotated image. Background Art

[0002] Image matching methods based on deep learning have become mainstream techniques in many computer vision and robotics applications because they are robust to large illumination and viewpoint changes and are widely used in camera motion estimation, multi-view Figure 3 dimensional reconstruction, remote sensing image stitching and other fields. However, it is difficult for traditional convolutional neural networks to extract rotation-equivariant local features. Recent studies have shown that group equivariant neural networks are very effective in image matching even when the image is rotated by a large amplitude. However, the inherent shape of the convolution kernel (square) leads to a performance bottleneck for this rotation-equivariant neural network, which can only perform well at a few fixed angles, and exhibits alternating peaks and troughs between 0 and 360°. Therefore, how to effectively utilize equivariant neural networks and design a reasonable solution based on the application characteristics of image matching methods so that the algorithm can offset the performance loss in the trough situation and achieve good performance from 0 to 360° has become the key to the problem, and no effective solution has been given in the existing technology. Summary of the Invention

[0003] The purpose of the present invention is to provide a matching method, device and storage medium for images rotated to a large extent, which uses a rotational equivariant neural network to achieve high-precision matching of images rotated at all angles and can overcome large-scale illumination changes.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A matching method for a significantly rotated image comprises the following steps:

[0006] S1: input reference image and target image;

[0007] S2: perform image preprocessing by averaging and standardization;

[0008] S3: Based on the inherent translation invariance of convolutional neural networks, the problem of matching rotated images is simplified to a corresponding point matching problem using a group equivariant neural network based on the plane rotation group SO2;

[0009] S4: Build and train a rotation-equivariant image feature extractor to extract image features of the reference image, target image, and 45° rotated reference image respectively;

[0010] S5: establishing two matching routes, and applying the nearest neighbor matching algorithm to the image features of the images in each matching route for matching, wherein the first matching route is matching the reference image and the target image, and the second matching route is matching the 45° rotated reference image and the target image;

[0011] S6: Based on the dual rotation equivariant excitation mixer, the matching results of the two matching routes are filtered, back-projected, merged, and filtered twice to output the matching relationship of dense corresponding points in the reference image and the target image.

[0012] In view of the characteristics of the application of the rotation image matching method to digital images and considering the fact that the convolution kernel of the convolutional neural network is square, the plane rotation group adopts a pure rotation group composed of 4 rotations.

[0013] The rotation equivariant image feature extractor performs a lifting operation on the input image and then inputs the image into a rotation equivariant backbone network to obtain a rotation equivariant feature map. The rotation equivariant feature map is converted into a rotation invariant feature map by using a group maximum pooling operation. The rotation invariant feature map is subjected to pixel-by-pixel bi-norm and quadratic power operations to obtain a descriptor and a new feature map. The new feature map is then subjected to a full connection operation in the channel dimension to obtain a reliability map and a repeatability map, respectively.

[0014] The lifting operation lifts the input image in the two-dimensional Euclidean space to the rotation group space.

[0015] The rotational equivariant backbone network is implemented using five convolutional layers, where each layer includes convolution, batch normalization, and nonlinear activation operations. The convolution kernel size of the first four layers is 3×3, and the last layer flexibly adjusts the receptive field according to the actual application. The third and fourth layers use dilated convolutions to effectively expand the receptive field without increasing the number of parameters and computational complexity.

[0016] When the input image is rotated 90°, the rotation equivariant feature map output by the rotation equivariant backbone network will also be rotated 90° in space, and in the rotation equivariant channel dimension, it appears as a permutation of the channel order.

[0017] The step S6 comprises the following steps:

[0018] S61: filtering the matching results of the two matching routes based on the RANdom SAmple Consensus (RANSAC) algorithm to remove outliers;

[0019] S62: performing a back-projection operation on the matching result of the second matching route after the first filtering, and merging the result with the matching result of the first matching route after the first filtering;

[0020] S63: Perform secondary filtering on the merged result based on the RANSAC algorithm to filter out outliers again, and output the matching relationship between the dense corresponding points in the reference image and the target image.

[0021] Based on the matching relationship between the reference image and the dense corresponding points in the target image output by the matching method, combined with the camera intrinsic parameters, the pose transformation relationship between the images is calculated and applied to various fields of computer vision.

[0022] A matching device for a greatly rotated image comprises a memory, a processor, and a program stored in the memory, wherein the processor implements the above-mentioned method when executing the program.

[0023] A storage medium stores a program thereon, and when the program is executed, the method described above is implemented.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) This paper proposes a unified framework based on group equivariant neural networks, which overcomes the problem that the performance of traditional convolutional neural networks is greatly reduced under large-scale rotations and achieves high-precision rotation matching.

[0026] (2) The present invention designs a dual-rotational equivariant excitation mixer, which can effectively solve the problem of periodic performance of group equivariant neural networks in image matching problems (that is, it can only achieve excellent performance near a few fixed angles, and performs very poorly at angles outside of these). It performs excellently at all angles, can adapt to large-scale illumination changes, and has a wide range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 Schematic diagram of the method framework of the present invention;

[0029] Figure 3 This is a comparative test diagram of the present invention under different rotation conditions. DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0031] This embodiment provides a matching method for a significantly rotated image, such as Figure 1 As shown, the following steps are included:

[0032] S1: Input reference image and target image.

[0033] S2: Perform image preprocessing by averaging and standardization.

[0034] S3: A group equivariant neural network is modeled based on the plane Lie group (SE2). At the same time, based on the inherent translation invariance of convolutional neural networks, the matching problem of rotated images is simplified to a corresponding point matching problem using a group equivariant neural network based on the plane rotation group (SO2).

[0035] In view of the characteristics of the application of the rotation image matching method to digital images and considering the fact that the convolution kernel of the convolutional neural network is square, this embodiment only uses a pure rotation group consisting of 4 rotations to construct a group equivariant neural network.

[0036] S4: Construct and train a rotation-equivariant image feature extractor to extract image features of the reference image, target image, and 45° rotated reference image respectively.

[0037] In this embodiment, the rotation equivariant image feature extractor is implemented using E2CNN code.

[0038] like Figure 2 As shown in the figure, the rotation equivariant image feature extractor performs a lifting operation on the input image and then inputs it into the rotation equivariant backbone network to obtain a rotation equivariant feature map. The group maximum pooling operation is used to convert the rotation equivariant feature map into a rotation invariant feature map. The rotation invariant feature map is subjected to pixel-by-pixel bi-norm and quadratic power operations to obtain a descriptor and a new feature map. The new feature map is then subjected to a full connection operation in the channel dimension to obtain a reliability map and a repeatability map, respectively.

[0039] In this embodiment, the lifting operation lifts the input image in the two-dimensional Euclidean space to the rotation group space.

[0040] In this embodiment, the rotational equivariant backbone network is implemented using five convolutional layers, where each layer includes convolution, batch normalization, and nonlinear activation operations. The convolution kernel size of the first four layers is 3×3, and the last layer flexibly adjusts the receptive field according to actual applications. The third and fourth layers use dilated convolutions to effectively expand the receptive field without increasing the number of parameters and computational complexity.

[0041] When the input image is rotated 90°, the rotation equivariant feature map output by the rotation equivariant backbone network will also be rotated 90° in space, and in the rotation equivariant channel dimension (usually a multiple of 4), it appears as a permutation of the channel order.

[0042] The rotation-equivariant image feature extractor uses five layers of rotation-equivariant convolution and group-max pooling operations to ensure that the features retain rotation equivariance during the multi-layer convolutional network extraction process, and when the features are finally output, they are converted into rotation-invariant feature maps through group-max pooling.

[0043] When training the rotation-equivariant image feature extraction network, the Adam optimizer was used, with the maximum number of iterations of the training dataset set to 40, the batch size to 12, the learning rate to 0.0001, and the weight decay exponent to 0.0005. If the loss did not decrease after five consecutive evaluation rounds, the training would be terminated early to prevent overfitting.

[0044] S5: Establish two matching routes, and apply the nearest neighbor (NN) matching algorithm to the image features of the images in each matching route for matching. The first matching route is the matching of the reference image and the target image, and the second matching route is the matching of the 45° rotated reference image and the target image.

[0045] S6: Based on the dual rotation equivariant excitation mixer, the matching results of the two matching routes are filtered, back-projected, merged, and filtered twice to output the matching relationship of dense corresponding points in the reference image and the target image.

[0046] S61: filtering the matching results of the two matching routes based on the RANSAC algorithm to remove outliers;

[0047] S62: performing a back-projection operation on the matching result of the second matching route after the first filtering, and merging the result with the matching result of the first matching route after the first filtering;

[0048] S63: Perform secondary filtering on the merged result based on the RANSAC algorithm to filter out outliers again, and output a matching relationship between the reference image and the dense corresponding points in the target image with excellent performance in the range of 0 to 360 degrees.

[0049] A matching effect of this embodiment is as follows: Figure 3 shown.

[0050] Based on the matching relationship between the dense corresponding points in the reference image and the target image output by the matching method, combined with the camera intrinsic parameters, the pose transformation relationship between the images is calculated and applied to various fields of computer vision, such as visual positioning and visual odometry.

[0051] The present invention only uses the classic nearest neighbor matching algorithm and random sampling consistency filtering in the matching stage. The matching method is relatively mature, has a small amount of calculation, and is simple to implement.

[0052] This method overcomes the problems of traditional image matching methods that perform poorly under varying lighting conditions, and learning-based image matching methods that perform poorly under large rotations. It achieves image matching that can handle both large lighting variations and rotations at all angles. As the most advanced rotation-equivariant image matching method, this method is beneficial for applications such as rapid motion estimation from a bird's-eye view, motion estimation of high-speed drone cameras with large rotations, and remote sensing image stitching.

[0053] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0054] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A matching method for a significantly rotated image, characterized in that: The following steps are involved: S1: input reference image and target image; S2: perform image preprocessing by averaging and standardization; S3: Based on the inherent translation invariance of convolutional neural networks, the matching problem of rotated images is simplified to a corresponding point matching problem using a group equivariant neural network based on the plane rotation group SO2; S4: Build and train a rotation-equivariant image feature extractor to extract image features of the reference image, target image, and 45° rotated reference image respectively; The rotation equivariant image feature extractor performs a lifting operation on the input image and then inputs it into the rotation equivariant backbone network to obtain a rotation equivariant feature map, and uses a group maximum pooling operation to convert the rotation equivariant feature map into a rotation invariant feature map. The rotation invariant feature map is subjected to pixel-by-pixel two-norm and quadratic power operations to obtain a descriptor and a new feature map. The new feature map is then subjected to a full connection operation in the channel dimension to obtain a reliability map and a repeatability map, respectively. The lifting operation lifts the input image in the two-dimensional Euclidean space to the rotation group space. The rotation equivariant backbone network is implemented using a 5-layer convolutional layer, wherein each layer includes convolution, batch normalization and nonlinear activation operations. The convolution kernel size of the first four layers is 3×3, and the last layer flexibly adjusts the receptive field according to actual applications. The third and fourth layers use void convolution to effectively expand the receptive field without increasing the number of parameters and the amount of calculation. S5: establishing two matching routes, and applying the nearest neighbor matching algorithm to the image features of the images in each matching route for matching, wherein the first matching route is matching the reference image and the target image, and the second matching route is matching the 45° rotated reference image and the target image; S6: Based on the dual rotation equivariant excitation mixer, the matching results of the two matching routes are filtered, back-projected, merged, and filtered twice to output the matching relationship of dense corresponding points in the reference image and the target image.

2. The matching method for a significantly rotated image according to claim 1, characterized in that: In view of the characteristics of the application of the rotation image matching method to digital images and considering the fact that the convolution kernel of the convolutional neural network is square, the plane rotation group adopts a pure rotation group composed of 4 rotations.

3. The matching method for a significantly rotated image according to claim 1, characterized in that: When the input image is rotated 90°, the rotation equivariant feature map output by the rotation equivariant backbone network will also be rotated 90° in space, and in the rotation equivariant channel dimension, it appears as a permutation of the channel order.

4. The method for matching a significantly rotated image according to claim 1, wherein: The step S6 comprises the following steps: S61: filtering the matching results of the two matching routes based on the RANSAC algorithm to remove outliers; S62: performing a back-projection operation on the matching result of the second matching route after the first filtering, and merging the result with the matching result of the first matching route after the first filtering; S63: Perform secondary filtering on the merged result based on the RANSAC algorithm to filter out outliers again, and output the matching relationship between the dense corresponding points in the reference image and the target image.

5. The matching method for a significantly rotated image according to claim 1, wherein: Based on the matching relationship between the reference image and the dense corresponding points in the target image output by the matching method, combined with the camera intrinsic parameters, the pose transformation relationship between the images is calculated and applied to various fields of computer vision.

6. A matching device for a significantly rotated image, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 5 is implemented.

7. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 5 is implemented.

Images