Image alignment method and device, computer-readable storage medium, and electronic device

Abstract

The present application relates to the field of image processing technology, and provides an image alignment method and device, a computer-readable storage medium, and an electronic device. The image alignment method includes: based on a source image and a target image, using a neural network model to calculate multiple weights; using the multiple weights to sum the corresponding multiple optical flow bases to obtain a first homography optical flow; using the first homography optical flow to align the source image to the target image; each weight in the method corresponds to an optical flow base of the first homography optical flow, and the first homography optical flow is the optical flow corresponding to the homography matrix between the source image and the target image. The method significantly improves the effect of image alignment through high-efficiency and high-precision homography estimation, and the method has good robustness and can adapt to different scenes in the image.

Claims

1. An image alignment method, characterized in that: include: Based on the source image and the target image, a plurality of weights are calculated using a neural network model; wherein each weight corresponds to an optical flow basis of a first homography optical flow, where the first homography optical flow is an optical flow corresponding to a homography matrix between the source image and the target image; Using the multiple weights, summing the corresponding multiple optical flow bases to obtain the first homography optical flow; The source image is aligned to the target image using the first homography optical flow.

2. The image alignment method according to claim 1, wherein: The number of the optical flow bases is no less than 8.

3. The image alignment method according to claim 2, wherein: The number of the optical flow bases is 8. Before calculating the multiple weights based on the source image and the target image using the neural network model, the method further includes: Generate 8 original optical flow bases; wherein, the size of the original optical flow base is the same as the size of the source image and the size of the target image, and the pixels in the 8 original optical flow bases are respectively taken in the following 8 ways: (1,0), (0,1), (x,0), (0,y), (y,0), (0,x), (x 2 ,xy)、(xy,y 2 ), x and y represent the horizontal and vertical coordinates of the pixel respectively; The 8 original optical flow bases are orthogonalized to obtain corresponding 8 optical flow bases.

4. The image alignment method according to any one of claims 1 to 3, characterized in that: The neural network model includes at least one low-rank reconstruction module, which is used to reduce the rank of the input features of the module by using subspace projection.

5. The image alignment method according to claim 4, characterized in that: The low-rank reconstruction module includes: a first branch, a second branch and a projection structure; The first branch is used to construct a basis of a subspace based on the input features and pass the basis of the subspace to the projection structure. The second branch is used to directly pass the input features to the projection structure. The projection structure is used to project the input features into the subspace according to the basis of the subspace and output the projected features. The dimension of the subspace is smaller than the dimension of the vector space where the input features are located.

6. The image alignment method according to claim 4, characterized in that: The neural network model includes: a feature extraction unit, a feature fusion unit and a weight prediction unit; The neural network model is used to calculate multiple weights based on the source image and the target image, including: respectively extracting basic features of the source image and basic features of the target image using the feature extraction unit; fusing the basic features of the source image and the basic features of the target image using the feature fusion unit to obtain fused basic features; The weight prediction unit is used to calculate the multiple weights based on the fused basic features; wherein the weight prediction unit includes the at least one low-rank reconstruction module.

7. The image alignment method according to claim 6, characterized in that: The loss function corresponding to the neural network model includes a triplet loss, wherein the anchor feature in the triplet loss is a feature of the target training image obtained after being processed by the feature extraction unit, the positive sample feature is a feature obtained after the source training image is first processed by the feature extraction unit and then distorted using the second homography optical flow, and the negative sample feature is a feature obtained after the source training image is processed by the feature extraction unit; The source training image and the target training image are both images used to train the neural network model, and the second homography optical flow is the optical flow corresponding to the homography matrix between the source training image and the target training image, which is calculated using the neural network model during training.

8. The image alignment method according to claim 7, characterized in that: The loss function corresponding to the neural network model also includes a feature identity loss, which characterizes the difference between a first feature and a second feature. The first feature is the feature obtained by first processing the source training image by the feature extraction unit and then distorting it using the second homography optical flow. The second feature is the feature obtained by first distorting the source training image using the second homography optical flow and then processing it by the feature extraction unit.

9. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 8 is executed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the method according to any one of claims 1 to 8 is executed.

11. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer program instructions, and when the computer program instructions are read and executed by the processor, the method according to any one of claims 1 to 8 is executed.

Images