Image alignment method and device

By acquiring the similarity characteristics of image features and using the offset prediction convolution layer, the problem of insufficient initial value of offset prediction in image alignment is solved, and the accuracy and stability of the alignment result are improved.

CN115393405BActive Publication Date: 2024-11-26BEIJING ZITIAO NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110557632.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-05-21
Publication Date
2024-11-26
Estimated Expiration
2041-05-21

AI Technical Summary

Technical Problem

In the prior art, the image alignment method lacks initial value when predicting the offset, resulting in a large difference between the alignment result of the deformable convolutional layer and the truth value, especially when the image quality is poor, the alignment result is unstable.

Method used

By acquiring the similarity characteristics of the target feature and the reference feature, the offset prediction convolution layer is used to obtain the offset, and combining the deformable convolution layer to align the reference image and the target image, the similarity characteristics guide the offset prediction to improve accuracy sex.

Benefits of technology

The accuracy of the deformable convolutional layer alignment results is improved, the difference between the alignment results and the truth value is reduced, and the stability of image alignment is enhanced, especially under low-quality image conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115393405B_ABST
    Figure CN115393405B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides an image alignment method and device, which relates to the field of image processing technology. The method includes: obtaining a target feature including feature points corresponding to pixel points in a target image and a reference feature including feature points corresponding to pixel points in a reference image; obtaining a similarity feature based on the target feature and the reference feature; the similarity feature includes: the similarity between each feature point in the target feature and the corresponding related feature point; predicting a convolution layer based on the similarity feature, the target feature and the offset, obtaining the offset of the target feature and the reference feature; aligning the reference feature with the target feature based on the offset and the deformable convolution layer. The embodiment of the present invention is used to solve the problem in the prior art that there is a lack of initial values ​​when predicting the offset, which leads to a large difference between the alignment result of the deformable convolution layer and the true value of the alignment result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image alignment method and device. Background Art

[0002] Image alignment refers to the process of determining the variation parameters between the reference image and the target image, and deforming the target image into the same spatial layout as the reference image according to the variation parameters. Image alignment is widely used in video restoration, image fusion, image stitching, target recognition and other fields. For example, in video restoration, by aligning adjacent image frames, the information between adjacent image frames can be effectively used to obtain more detailed information of the image frames, thereby obtaining a clearer video with richer details.

[0003] The traditional image alignment method is: calculate the optical flow field between the target image and the reference image, and use the optical flow field as the dense registration relationship between the target image and the reference image, and finally align the target image to the reference image by back-warping. Although this optical flow-based image alignment method can also achieve image alignment, the amount of calculation when calculating the optical flow field is very large, so the efficiency is low. In order to improve the problem of low image alignment efficiency, a feature-based image alignment method is proposed in the prior art, which specifically includes: obtaining the reference feature of the reference image and the target feature of the target image, then predicting the offset according to the reference feature and the target feature, and then aligning the reference feature with the target feature based on the deformable convolution layer by the offset control to obtain the final alignment result. Compared with the optical flow-based image alignment method, the feature-based image alignment method does not need to calculate the optical flow field, so it is more efficient and can directly align the image features. However, due to the lack of initial values ​​when predicting the offset according to the reference feature and the target feature, when the image quality of the reference image and the target image is poor, the alignment result of the deformable convolution layer will be greatly different from the true value of the alignment result. Summary of the invention

[0004] In view of this, the present invention provides an image alignment method and device for solving the problem in the prior art that there is a lack of initial values ​​when predicting offsets, which leads to a large difference between the alignment result of the deformable convolutional layer and the true value of the alignment result.

[0005] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present invention provides an image alignment method, comprising:

[0007] Acquire target features and reference features; the target features include feature points corresponding to pixels in the target image, and the reference features include feature points corresponding to pixels in the reference image;

[0008] According to the target feature and the reference feature, a similarity feature is obtained; the similarity feature includes: the similarity between each feature point in the target feature and the corresponding related feature point, and the related feature points corresponding to the feature point in the target feature include: feature points in the reference feature with pixel coordinates that are the same as or adjacent to the pixel coordinates of the feature point in the target feature;

[0009] Predicting a convolution layer according to the similarity feature, the target feature, and the offset to obtain an offset between the target feature and the reference feature;

[0010] The reference image is aligned with the target image according to the offset and the deformable convolution layer.

[0011] As an optional implementation of the embodiment of the present invention, the related feature points corresponding to the first feature point in the target feature include: the second feature point and feature points within the neighborhood of the first preset value of the second feature point;

[0012] The second feature point is a feature point in the reference feature whose pixel coordinates are the same as the pixel coordinates of the first feature point.

[0013] As an optional implementation manner of the embodiment of the present invention, the acquiring of a similarity feature according to the target feature and the reference feature includes:

[0014] Determine a first spatial domain corresponding to a third feature point in the target feature, where the first spatial domain is a spatial domain formed by feature points in a neighborhood of the third feature point and a second preset value of the third feature point;

[0015] Determine a second spatial domain corresponding to a fourth feature point in the reference feature, where the second spatial domain is a spatial domain formed by feature points in a neighborhood of the fourth feature point and a second preset value of the fourth feature point; the fourth feature point is a related feature point corresponding to the third feature;

[0016] Calculating the outer product of the feature points in each feature group to obtain the outer product of each feature group; the feature group includes feature points belonging to the first spatial domain and feature points belonging to the second spatial domain, and the position of the feature points belonging to the first spatial domain in the first spatial domain is the same as the position of the feature points belonging to the second spatial domain in the second spatial domain;

[0017] The outer products of each feature group are summed to obtain the similarity between the third feature point and the fourth feature point.

[0018] As an optional implementation of the embodiment of the present invention, the target feature is a feature obtained by extracting features from pixels in the target image, and the reference feature is a feature obtained by extracting features from pixels in the reference image;

[0019] or;

[0020] The target feature is a feature obtained by extracting features from pixel points in the target image and downsampling the extracted features at a preset downsampling rate; the reference feature is a feature obtained by extracting features from pixel points in the reference image and downsampling the extracted features at the preset downsampling rate.

[0021] As an optional implementation of the embodiment of the present invention, the predicting convolution layer according to the similarity feature, the target feature and the offset to obtain the offset of the target feature and the reference feature includes:

[0022] Connecting the similarity feature and the target feature in series in the dimension of the channel to obtain an offset prediction feature;

[0023] Inputting the offset prediction feature into the offset prediction convolutional layer;

[0024] The output of the offset prediction convolutional layer is obtained as the offset of the target feature and the reference feature.

[0025] As an optional implementation of the embodiment of the present invention, the target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the similarity feature includes sub-similarity features of multiple spatial scales;

[0026] The predicting convolution layer according to the similarity feature, the target feature and the offset to obtain the offset of the target feature and the reference feature includes:

[0027] A convolution layer is predicted according to the sub-similarity features of the multiple spatial scales, the sub-target features of the multiple spatial scales, and the corresponding offsets of the multiple spatial scales to obtain sub-offsets of the sub-target features and sub-reference features of the multiple spatial scales.

[0028] As an optional implementation manner of the embodiment of the present invention, aligning the reference image with the target image according to the offset and the deformable convolution layer includes:

[0029] Inputting the reference feature into the deformable convolution layer, and controlling the shape of the convolution kernel of the deformable convolution layer by the offset;

[0030] Obtaining the output of the deformable convolutional layer as an alignment result of the reference feature and the target feature;

[0031] The reference image is aligned with the target image according to the alignment result of the reference feature and the target feature.

[0032] As an optional implementation manner of the embodiment of the present invention, the target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the offset includes sub-offsets of multiple spatial scales;

[0033] The step of aligning the reference image with the target image according to the offset and the deformable convolution layer comprises:

[0034] Acquire alignment results of sub-target features and sub-reference features of the multiple spatial scales according to the sub-offsets of the multiple spatial scales and the corresponding deformable convolutional layers of the multiple spatial scales;

[0035] The reference image is aligned with the target image according to the alignment results of the sub-target features and the sub-reference features at the multiple spatial scales.

[0036] As an optional implementation manner of the embodiment of the present invention, the reference image is the nth image frame of the video to be repaired, and the reference image is the n+1th image frame of the video to be repaired; n is a positive integer.

[0037] In a second aspect, an embodiment of the present invention provides an image alignment device, comprising:

[0038] A feature acquisition unit, used to acquire target features and reference features; the target features include feature points corresponding to pixels in the target image, and the reference features include feature points corresponding to pixels in the reference image;

[0039] A similarity acquisition unit is used to acquire a similarity feature according to the target feature and the reference feature; the similarity feature includes: the similarity between each feature point in the target feature and the corresponding related feature point, and the related feature points corresponding to the feature point in the target feature include: feature points in the reference feature with pixel coordinates that are the same as or adjacent to the pixel coordinates of the feature point in the target feature;

[0040] An offset acquisition unit, configured to acquire an offset of the target feature and the reference feature according to the similarity feature, the target feature and the offset prediction convolution layer;

[0041] A processing unit is used to align the reference image with the target image according to the offset and the deformable convolution layer.

[0042] As an optional implementation of the embodiment of the present invention, the related feature points corresponding to the first feature point in the target feature include: the second feature point and feature points within the neighborhood of the first preset value of the second feature point;

[0043] The second feature point is a feature point in the reference feature whose pixel coordinates are the same as the pixel coordinates of the first feature point.

[0044] As an optional implementation manner of the embodiment of the present invention, the similarity acquisition unit is specifically used to determine the first spatial domain corresponding to the third feature point in the target feature, the first spatial domain being the spatial domain formed by the third feature point and the feature points in the neighborhood of the second preset value of the third feature point; determine the second spatial domain corresponding to the fourth feature point in the reference feature, the second spatial domain being the spatial domain formed by the fourth feature point and the feature points in the neighborhood of the second preset value of the fourth feature point; the fourth feature point is a related feature point corresponding to the third feature; calculate the outer product of the feature points in each feature group to obtain the outer product of each feature group; the feature group includes feature points belonging to the first spatial domain and feature points belonging to the second spatial domain, and the position of the feature points belonging to the first spatial domain in the first spatial domain is the same as the position of the feature points belonging to the second spatial domain in the second spatial domain; sum the outer products of each feature group to obtain the similarity between the third feature point and the fourth feature point.

[0045] As an optional implementation of the embodiment of the present invention, the target feature is a feature obtained by extracting features from pixels in the target image, and the reference feature is a feature obtained by extracting features from pixels in the reference image;

[0046] or;

[0047] The target feature is a feature obtained by extracting features from pixel points in the target image and downsampling the extracted features at a preset downsampling rate; the reference feature is a feature obtained by extracting features from pixel points in the reference image and downsampling the extracted features at the preset downsampling rate.

[0048] As an optional implementation manner of an embodiment of the present invention, the offset acquisition unit is specifically used to concatenate the similarity feature and the target feature in the dimension of the channel to obtain an offset prediction feature; input the offset prediction feature into the offset prediction convolution layer; and obtain the output of the offset prediction convolution layer as the offset of the target feature and the reference feature.

[0049] As an optional implementation manner of the embodiment of the present invention, the target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the offset includes sub-offsets of multiple spatial scales;

[0050] The offset acquisition unit is specifically used to predict the convolution layer according to the sub-similarity features of the multiple spatial scales, the sub-target features of the multiple spatial scales, and the corresponding offsets of the multiple spatial scales, to obtain the sub-offsets of the sub-target features and sub-reference features of the multiple spatial scales.

[0051] As an optional implementation of the embodiment of the present invention, the processing unit is specifically used to input the reference feature into the deformable convolution layer, and control the shape of the convolution kernel of the deformable convolution layer through the offset;

[0052] The output of the deformable convolutional layer is obtained as an alignment result of the reference feature and the target feature.

[0053] As an optional implementation manner of the embodiment of the present invention, the target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the offset includes sub-offsets of multiple spatial scales;

[0054] The processing unit is specifically used to obtain the alignment results of the sub-target features and the sub-reference features of the multiple spatial scales according to the sub-offsets of the multiple spatial scales and the corresponding deformable convolution layers of the multiple spatial scales; and align the reference image with the target image according to the alignment results of the sub-target features and the sub-reference features of the multiple spatial scales.

[0055] As an optional implementation manner of the embodiment of the present invention, the reference image is the nth image frame of the video to be repaired, and the reference image is the n+1th image frame of the video to be repaired; n is a positive integer.

[0056] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the image alignment method described in the first aspect or any optional implementation of the first aspect when calling the computer program.

[0057] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image alignment method described in the first aspect or any optional implementation of the first aspect.

[0058] In a fifth aspect, an embodiment of the present invention provides a computer program product. When the computer program product runs on a computer, the computer implements the image alignment method described in the first aspect or any optional implementation of the first aspect.

[0059] The image alignment method provided by the embodiment of the present invention first obtains a target feature including feature points corresponding to pixel points in the target image and a reference feature including feature points corresponding to pixel points in the reference image, then obtains a feature including the similarity between each feature point in the target feature and the corresponding related feature point based on the target feature and the reference feature, and then predicts a convolution layer based on the similarity feature, the target feature and the offset to obtain the offset between the target feature and the reference feature, and finally aligns the reference image with the target image based on the offset and the deformable convolution layer. Since the similarity between each feature point in the target feature and the corresponding related feature point has a strong correlation with the optical flow field between the target image and the reference image, using the similarity feature as a guide for the offset can more accurately predict the offset between the target feature and the reference feature, thereby solving the problem that the alignment result of the deformable convolution layer is greatly different from the true value of the alignment result. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0062] Figure 1 A flowchart of the steps of the image alignment method provided by an embodiment of the present invention;

[0063] Figure 2 One of the schematic diagrams of related feature points corresponding to the feature points provided in the embodiment of the present invention;

[0064] Figure 3 A second schematic diagram of related feature points corresponding to feature points provided in an embodiment of the present invention;

[0065] Figure 4 A schematic diagram of a first spatial domain corresponding to a feature point provided in an embodiment of the present invention;

[0066] Figure 5 A schematic diagram of a second spatial domain corresponding to a feature point provided in an embodiment of the present invention;

[0067] Figure 6 One of the flowcharts of the image alignment method provided by an embodiment of the present invention;

[0068] Figure 7 A second flowchart of the image alignment method provided by an embodiment of the present invention;

[0069] Figure 8 A third flowchart of the image alignment method provided by an embodiment of the present invention;

[0070] Fig. 9 A schematic diagram of a graphics processing device provided by an embodiment of the present invention;

[0071] Fig.10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the scheme of the present invention will be further described below. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0073] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present invention, rather than all of the embodiments.

[0074] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present invention, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0075] The overall inventive concept of the embodiment of the present invention is: in actual application, when the input image of the deformable convolution layer has a relatively serious degradation process (blur, fog, noise), the alignment result of the deformable convolution layer is very unstable, and the gradient explosion phenomenon is easy to occur. The root cause of the instability is that in the early stage of training, due to the uncertainty of the initial value of the offset, the initial predicted offset is greatly different from the actual true value. Therefore, in order to stabilize the training process, in the embodiment of the present invention, the correlation layer in the optical flow network is introduced, and the similarity feature of the target feature and the reference feature obtained by the correlation layer is used as a guide for the offset. Since the similarity feature has a close relationship with the optical flow, the problem of a large difference between the initial predicted offset and the actual true value can be avoided by guiding the similarity feature.

[0076] The embodiment of the present invention provides an image alignment method, referring to Figure 1 As shown, the image alignment method includes the following steps:

[0077] S101. Obtain target features and reference features.

[0078] The target features include feature points corresponding to pixel points in the target image; and the reference features include feature points corresponding to pixel points in the reference image.

[0079] Optionally, the target feature and the reference feature may be adjacent image frames in a video, that is, the reference image is the nth image frame of the video to be repaired, and the reference image is the n+1th image frame of the video to be repaired, where n is a positive integer.

[0080] S102: Acquire a similarity feature according to the target feature and the reference feature.

[0081] Among them, the similarity feature includes: the similarity between each feature point in the target feature and the corresponding related feature point, and the related feature points corresponding to the feature points in the target feature include: feature points whose pixel coordinates in the reference feature are the same as or adjacent to the pixel coordinates of the feature points in the target feature.

[0082] Specifically, the pixel coordinates of a feature point in the embodiment of the present invention refer to the pixel coordinates of a pixel point corresponding to the feature point in the corresponding image.

[0083] For example: the pixel I with pixel coordinates (1, 1) in the reference image 11 The corresponding feature point is F a11 , then the feature point F a11 The pixel coordinates are (1, 1).

[0084] Another example: the pixel point I with pixel coordinates (2, 3) in the target image 23 The corresponding feature point is Fb23 , then the feature point F b23 The pixel coordinates are (2, 3).

[0085] It should be noted that the pixel coordinates and pixel coordinates in the embodiment of the present invention refer to two pixels belonging to the same preset coordinate range, and are limited to not including other pixel coordinates between the two pixel coordinates.

[0086] As an optional implementation of the embodiment of the present invention, the related feature points corresponding to the first feature point in the target feature include: the second feature point and feature points within the neighborhood of the first preset value of the second feature point;

[0087] The second feature point is a feature point in the reference feature whose pixel coordinates are the same as the pixel coordinates of the first feature point.

[0088] Assume that the first preset value is d, then the related feature points corresponding to the first feature point include: the second feature point in the reference feature whose pixel coordinates are the same as the pixel coordinates of the first feature point, and the feature points within the d*d neighborhood of the second feature point.

[0089] Exemplarily, d may be 9. That is, the related feature points corresponding to the first feature point include: the second feature point in the reference feature whose pixel coordinates are the same as the pixel coordinates of the first feature point, and the feature points in the 9*9 neighborhood of the second feature point.

[0090] For example, refer to Figure 2 As shown, Figure 2 In the example, the resolution of the target image and the reference image is 6*6 and the first preset value is 3, and the related feature points corresponding to the feature points in the target feature are described. Figure 2 As shown, the pixel coordinates of the feature point Fa33 in the target feature 21 are (3, 3), and the feature point with pixel coordinates (3, 3) in the reference feature 22 is Fb33. Therefore, when the first feature point is Fa33, the second feature point is Fb33, and the feature points within the 3*3 neighborhood of the second feature point Fb33 include: Fb22, Fb23, Fb24, Fb32, Fb34, Fb42, Fb43, and Fb44. Therefore, the related feature points corresponding to the feature point Fa33 in the target feature 21 include 9 feature points in the reference feature, which are: Fb22, Fb23, Fb24, Fb32, Fb33, Fb34, Fb42, Fb43, and Fb44.

[0091] For example, refer to Figure 3 As shown, Figure 3In the example, the resolution of the target image and the reference image is still 6*6, and the first preset value is 3, and the related feature points corresponding to the feature points in the target feature are described. Figure 3 As shown, the pixel coordinates of the feature point Fa46 in the target feature 31 are (4, 6), and the feature point with pixel coordinates (4, 6) in the reference feature 32 is Fb46. Therefore, when the first feature point is Fa46, the second feature point is Fb46, and the feature points within the 3*3 neighborhood of the second feature point Fb46 include: Fb35, Fb36, Fb45, Fb55, and Fb56. Therefore, the related feature points corresponding to the feature point Fa46 in the target feature 21 include the 6 feature points in the reference feature, which are: Fb35, Fb36, Fb45, Fb46, Fb55, and Fb56.

[0092] According to the same method as above, each feature point (Fa11, Fa12, Fa13...Fa65, Fa66) in the target feature is used as the first feature point one by one, and then the relevant feature points corresponding to each feature point in the target feature can be determined.

[0093] In the above embodiment, the second feature point in the reference feature whose pixel coordinates are the same as the pixel coordinates of the first feature point and the feature points in the neighborhood of the first preset value of the second feature point are determined as related feature points corresponding to the first feature point. Compared with only determining the second feature point as the related feature point corresponding to the first feature point, the above embodiment can increase the receptive field of the similarity feature when obtaining the similarity feature, and thus when the true value of the offset between the reference image and the target image is large, the obtained similarity feature is inaccurate.

[0094] Furthermore, in the above step S102, obtaining similarity features according to the target features and the reference features includes the following steps a to d.

[0095] Step a: determine the first spatial domain corresponding to the third feature point in the target feature.

[0096] The first spatial domain is a spatial domain formed by the third feature point and feature points in the neighborhood of a second preset value of the third feature point; and the third feature point is any feature point in the target features.

[0097] Assume that the second preset value is k, then the first spatial domain is the spatial domain formed by the third feature point and the feature points within the k*k neighborhood of the third feature point.

[0098] For example, refer to Figure 4 As shown, Figure 4 In the example, the resolution of the target image and the reference image is 6*6 and the second preset value is 3, and the first spatial domain corresponding to the feature points in the target feature is described. Figure 4 As shown, the pixel points within the 3*3 neighborhood of the feature point Fa33 in the target feature 41 include: Fa22, Fa23, Fa24, Fa32, Fa34, Fa42, Fa43, and Fa44. Therefore, the first spatial domain corresponding to the feature point Fa33 is the spatial domain 400 composed of Fa22, Fa23, Fa24, Fa32, Fa33, Fa34, Fa42, Fa43, and Fa44.

[0099] Step b: determining a second spatial domain corresponding to the fourth feature point in the reference feature.

[0100] The second spatial domain is a spatial domain formed by the fourth feature point and feature points in the neighborhood of a second preset value of the fourth feature point; and the fourth feature point is a related feature point corresponding to the third feature.

[0101] Similarly, assuming that the second preset value is k, the second spatial domain is the spatial domain formed by the fourth feature point and feature points in the k*k neighborhood of the fourth feature point.

[0102] For example, refer to Figure 5 As shown, Figure 5 In the example, the second spatial domain corresponding to the feature points in the reference feature is described by taking the case where the resolution of the target image and the reference image are both 6*6 and the second preset value is 3. Figure 5 As shown, reference feature 52 Figure 4 The pixel points within the 3*3 neighborhood of the related feature point Fa22 corresponding to the feature point Fa33 in the target feature 41 shown include: Fb11, Fb12, Fb13, Fb21, Fb23, Fb31, Fb32, and Fb33. Therefore, the second spatial domain corresponding to the feature point Fb22 is the space 500 composed of Fb11, Fb12, Fb13, Fb21, Fb22, Fb23, Fb31, Fb32, and Fb33.

[0103] Step c: Calculate the outer product of the feature points in each feature group to obtain the outer product of each feature group.

[0104] The feature group includes feature points belonging to the first spatial domain and feature points belonging to the second spatial domain, and positions of the feature points belonging to the first spatial domain in the first spatial domain are the same as positions of the feature points belonging to the second spatial domain in the second spatial domain.

[0105] In the embodiment of the present invention, the outer product refers to the vector product of two eigenvectors. With features The outer product can be expressed as:

[0106] Continued Figure 4 , Figure 5 As shown, the feature points with the same position in their respective spatial domains include: Fa22 and Fb11, Fa23 and Fb12, Fa24 and Fb13, Fa32 and Fb21, Fa33 and Fb22, Fa34 and Fb23, Fa42 and Fb31, Fa43 and Fb32, Fa44 and Fb33, so the feature group (Fa22, Fb11), feature group (Fa23, Fb12), feature group (Fa24, Fb13), (Fa32, Fb21 ), the outer product of feature group (Fa33, Fb22), feature group (Fa34, Fb23), feature group (Fa42, Fb31), feature group (Fa43, Fb32), and feature group (Fa44, Fb33) obtains Fa22×Fb11, Fa23×Fb12, Fa24×Fb13, Fa32×Fb21, Fa33×Fb22, Fa34×Fb23, Fa42×Fb31, Fa43×Fb32, and Fa44×Fb33.

[0107] Step d: summing the outer products of each feature group to obtain the similarity between the third feature point and the fourth feature point.

[0108] As described in the above example, the outer product of the feature group includes: Fa22×Fb11, Fa23×Fb12, Fa24×Fb13, Fa32×Fb21, Fa33×Fb22, Fa34×Fb23, Fa42×Fb31, Fa43×Fb32, Fa44×Fb33, so the similarity between the third feature point Fa33 and the fourth feature point Fb22 is: Fa22×Fb11+Fa23×Fb12+Fa24×Fb13+Fa32×Fb21+Fa33×Fb22+Fa34×Fb23+Fa42×Fb31+Fa43×Fb32+Fa44×Fb33.

[0109] That is, for the feature point x in the target feature 1 and the feature point x in the reference feature 1 The relevant feature points x 2 , x 1 With x 2 The similarity can be calculated by the following formula:

[0110] c(x 1 , x 2 )=∑[f 1 (x 1 +o)×f 2 (x 2 +o)];

[0111] o∈[-1 / 2k, 1 / 2k]×[-1 / 2k, 1 / 2k].

[0112] Among them, c(x 1 , x 2 ) is the feature point x 1 and x 2 similarity, k is a constant, f 1 (x 1 +o) indicates that x 1 and x 1 The feature points in the k*k neighborhood of f 2 (x 2 +o) indicates that x 2 and x 2 Feature points within the k*k neighborhood of .

[0113] Based on the same method mentioned above, the other related feature points corresponding to the third feature point are taken as the fourth feature point one by one, so that the similarity between the third feature point and the corresponding related feature point can be obtained, and then each feature point in the target feature (Fa11, Fa12, Fa13...Fa65, Fa66) is taken as the third feature point one by one, so that the similarity between each feature point in the target feature and the corresponding related feature point can be obtained, and then the similarity feature can be obtained.

[0114] When calculating the similarity between the third feature point and the fourth feature point, the above embodiment first determines the first spatial domain corresponding to the third feature point and the second spatial domain corresponding to the fourth feature point, then calculates the outer product of the feature points in each feature group, obtains the outer product of each feature group, and finally sums the outer products of each feature group, and uses the summed result as the similarity between the third feature point and the fourth feature point. Compared with directly calculating the outer product of the third feature point and the fourth feature point, and using the outer product of the third feature point and the fourth feature point as the similarity between the third feature point and the fourth feature point, the above embodiment can increase the dimension of obtaining the similarity, thereby improving the robustness of similarity feature acquisition.

[0115] S103. Predict a convolution layer according to the similarity feature, the target feature, and the offset to obtain an offset between the target feature and the reference feature.

[0116] Optionally, an implementation of step S103 includes the following steps 1 to 3:

[0117] Step 1: Concatenate the similarity feature and the target feature in the dimension of the channel to obtain an offset prediction feature.

[0118] In the embodiment of the present invention, the number of channels of a feature refers to the number of feature maps contained in the feature. A channel of a feature is a feature map obtained by extracting a feature based on a certain dimension, so the channel of a feature is a feature map in a specific sense. The similarity feature is connected in series with the target feature in the dimension of the channel to obtain the offset prediction feature, that is, the feature map of the similarity feature is connected in series with the feature map of the target feature in sequence, thereby obtaining the offset prediction feature including all feature maps of the target feature and all feature maps of the similarity feature in sequence.

[0119] Step 2: Input the offset prediction feature into the offset prediction convolutional layer.

[0120] Step 3: Obtain the output of the offset prediction convolution layer as the offset of the target feature and the reference feature.

[0121] S104: Align the reference image with the target image according to the offset and the deformable convolution layer.

[0122] Optionally, an implementation of the above step S104 includes the following steps I to III:

[0123] Step I: input the reference feature into the deformable convolution layer, and control the shape of the convolution kernel of the deformable convolution layer by the offset.

[0124] Step II: obtaining the output of the deformable convolutional layer as the alignment result of the reference feature and the target feature.

[0125] Step III: Align the reference feature with the target feature, and align the reference image with the target image.

[0126] In summary, the reference feature is represented as F1, the target feature is represented as F2, the similarity feature is represented as Fc, the offset prediction feature is represented as Ft, the offset is represented as Off, the alignment result is represented as Fa, the module for obtaining the similarity feature is called the correlation layer, and the module for obtaining the offset prediction feature is represented as the concatenation layer. The process of the image alignment method provided in the above embodiment is shown in Figure 6:

[0127] First, the reference feature F1 and the target feature F2 are input into the correlation layer 61, and the output of the correlation layer 61 is obtained as the similarity feature Fc.

[0128] Secondly, the similarity feature Fc and the target feature F2 are input into the concatenation layer 62, and the output of the concatenation layer 62 is obtained as the offset prediction feature Ft.

[0129] Again, the offset prediction feature Ft is input into the offset prediction convolution layer 63, and the output of the offset prediction convolution layer 63 is obtained as the offset Off.

[0130] Finally, the offset Off and the target feature F2 are input into the deformable convolution layer 64, and the output of the deformable convolution layer 64 is obtained as the alignment result Fa.

[0131] Furthermore, assuming that the feature dimensions of the reference feature F1 and the target feature F2 are C×H×W, the feature dimension of the similarity feature Fc is (d*d)×H×W, and the feature dimension of the offset prediction feature Ft is (d*d+C)×H×W; wherein d is a first preset value.

[0132] The image alignment method provided by the embodiment of the present invention first obtains a target feature including feature points corresponding to pixel points in the target image and a reference feature including feature points corresponding to pixel points in the reference image, then obtains a feature including the similarity between each feature point in the target feature and the corresponding related feature point based on the target feature and the reference feature, and then predicts a convolution layer based on the similarity feature, the target feature and the offset to obtain the offset between the target feature and the reference feature, and finally aligns the reference image with the target image based on the offset and the deformable convolution layer. Since the similarity between each feature point in the target feature and the corresponding related feature point has a strong correlation with the optical flow field between the target image and the reference image, using the similarity feature as a guide for the offset can more accurately predict the offset between the target feature and the reference feature, thereby solving the problem that the alignment result is greatly different from the true value of the alignment result.

[0133] As an optional implementation of the embodiment of the present invention, the target feature is a feature obtained by extracting features from pixels in the target image, and the reference feature is a feature obtained by extracting features from pixels in the reference image.

[0134] That is, the images extracted from the target image and the reference image are used as the target features and the reference features, respectively.

[0135] As an optional implementation manner of an embodiment of the present invention, the target feature is a feature obtained by performing feature extraction on pixel points in the target image and downsampling the extracted image at a preset downsampling rate; the reference feature is a feature obtained by performing feature extraction on pixel points in the reference image and downsampling the extracted image at the preset downsampling rate.

[0136] Exemplarily, the preset downsampling rate may be 1 / 16.

[0137] That is, the features extracted from the target image are downsampled to 1 / 16 of the original features as target features, and the features extracted from the reference image are downsampled to 1 / 16 of the original features as reference features.

[0138] In the above embodiment, when the first preset value is set to be relatively large, the offset prediction convolution layer can obtain a sufficiently large receptive field, but at the same time, it will also increase the amount of calculation for calculating the similarity feature, thereby affecting the image alignment efficiency. In order to solve the above problem, the above embodiment downsamples the features extracted from the target image and the features extracted from the target image at a preset downsampling rate, so that while ensuring a sufficiently large receptive field, the amount of calculation for the similarity feature can be avoided to be too large, thereby avoiding affecting the image alignment efficiency.

[0139] In addition, after downsampling the target feature and the reference feature at a preset downsampling rate and obtaining the similarity feature, the similarity feature and the target feature need to be concatenated in the dimension of the channel to obtain the offset prediction feature, and the dimensions of the features concatenated in the dimension of the channel need to be the same, so the similarity feature also needs to be upsampled to a feature with the same dimension as the target feature. Figure 7 As shown, Figure 6 Based on the process shown in FIG. 1 , the image alignment method provided in the embodiment of the present invention further includes: performing a downsampling operation on the target feature F2 and the reference feature F1 ( Figure 7 denoted by a downward arrow), and upsampling the similarity features ( Figure 7 indicated by an upward arrow).

[0140] Furthermore, in an embodiment of the present invention, a cascade pyramid architecture may be used to progressively align target features and reference features from multiple different spatial scales.

[0141] That is, the target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the similarity feature includes sub-similarity features of multiple spatial scales. The above step S103 (predicting the convolution layer according to the similarity feature, the target feature, and the offset to obtain the offset of the target feature and the reference feature) includes:

[0142] A convolution layer is predicted according to the sub-similarity features of the multiple spatial scales, the sub-target features of the multiple spatial scales, and the corresponding offsets of the multiple spatial scales to obtain sub-offsets of the sub-target features and sub-reference features of the multiple spatial scales.

[0143] Since the above embodiment progressively obtains the offset between the features of the reference image and the features of the target image from multiple spatial scales, the above embodiment can more accurately obtain the offset between the features of the reference image and the features of the target image.

[0144] The target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, the offset includes sub-offsets of multiple spatial scales, and the above step S104 (aligning the reference image with the target image according to the offset and the deformable convolution layer) includes:

[0145] Acquire alignment results of sub-target features and sub-reference features of the multiple spatial scales according to the sub-offsets of the multiple spatial scales and the corresponding deformable convolutional layers of the multiple spatial scales;

[0146] The reference image is aligned with the target image according to the alignment results of the sub-target features and the sub-reference features at the multiple spatial scales.

[0147] Since the above embodiment progressively aligns the reference image with the target image from multiple spatial scales, the above embodiment can improve the accuracy of the alignment result between the reference image and the target image.

[0148] Optionally, aligning the reference image with the target image according to the offset and the deformable convolution layer includes:

[0149] Acquire a target offset of the n-th spatial scale according to the sub-offset of the n-th spatial scale and the sub-offset of the n-1-th spatial scale;

[0150] Acquire an alignment result of the n-th spatial scale according to the target offset of the n-th spatial scale and the deformable convolution layer corresponding to the n-th spatial scale;

[0151] Acquire a target alignment result of the n-th spatial scale according to the alignment result of the n-th spatial scale and the target alignment result of the n-1-th spatial scale;

[0152] Acquire an alignment result of the reference image and the target image according to the target alignment result of the first spatial scale;

[0153] The n-1th level spatial scale is smaller than the nth level spatial scale.

[0154] Reference Figure 8 As shown, Figure 8 In the figure, a three-level cascade pyramid architecture is used to progressively align target features and reference features at different spatial scales.

[0155] First, sub-reference features F1_1, F1_2, F1_3 and sub-target features F2_1, F2_2 and F2_3 are obtained from the spatial scales corresponding to the first, second and third levels.

[0156] Then, starting from the third spatial scale, the sub-offset Off_3 of the sub-reference feature F1_1 and the sub-target feature F2_1 and the result Fa_3 are obtained through the image alignment method provided in the above embodiment. Since the third spatial scale does not include the previous spatial scale, the sub-offset Off_3 directly acts on the deformable convolution layer corresponding to the third spatial scale, and the alignment result of the third spatial scale is the same as its target result.

[0157] Secondly, through the image alignment method provided in the above embodiment, the sub-offset Off_2 of the sub-reference feature F1_2 and the sub-target feature F2_2 of the second-level spatial scale is obtained, and the target offset of the second-level spatial scale is generated according to the sub-offset Off_3 and the sub-offset Off_2, and then the target offset of the second-level spatial scale is input into the deformable convolution layer of the second-level spatial scale to obtain the second-level alignment result Fa_2, and the alignment result Fa_2 and the target alignment result Fa_3 of the third-level spatial scale are combined to obtain the target alignment result of the second-level spatial scale.

[0158] Finally, through the image alignment method provided in the above embodiment, the sub-offset Off_1 of the sub-reference feature F1_1 and the sub-target feature F2_1 of the first-level spatial scale is obtained, and the target offset of the first-level spatial scale is generated according to the sub-offset Off_2 and the sub-offset Off_1, and then the target offset of the first-level spatial scale is input into the deformable convolution layer of the first-level spatial scale to obtain the alignment result Fa_1 of the first level, and the alignment result Fa_1 and the target alignment result of the second-level spatial scale are combined to obtain the target alignment result of the first-level spatial scale (the final alignment result).

[0159] It should be noted that Figure 8 In the embodiment, the target feature and the reference feature are progressively aligned from three different spatial scales as an example, but the embodiment of the present invention is not limited thereto. On the basis of the above embodiment, the target feature and the reference feature can also be progressively aligned from other different spatial scales. For example, the target feature and the reference feature are progressively aligned from two different spatial scales, the target feature and the reference feature are progressively aligned from five different spatial scales, etc., which are not limited in the embodiment of the present invention.

[0160] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present invention further provides an image alignment device. The device embodiment corresponds to the above method embodiment. For ease of reading, the device embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the image alignment device in this embodiment can correspond to all the contents of the above method embodiment.

[0161] An embodiment of the present invention provides an image alignment device. Fig. 9 is a structural diagram of the image alignment device, such as Fig. 9 As shown, the image alignment device 900 includes:

[0162] A feature acquisition unit 91 is used to acquire target features and reference features; the target features include feature points corresponding to pixels in the target image, and the reference features include feature points corresponding to pixels in the reference image;

[0163] A similarity acquisition unit 92 is used to acquire a similarity feature according to the target feature and the reference feature; the similarity feature includes: the similarity between each feature point in the target feature and the corresponding related feature point, and the related feature points corresponding to the feature points in the target feature include: feature points in the reference feature with pixel coordinates that are the same as or adjacent to the pixel coordinates of the feature points in the target feature;

[0164] An offset acquisition unit 93, configured to acquire an offset between the target feature and the reference feature according to the similarity feature, the target feature and the offset prediction convolution layer;

[0165] The processing unit 94 is configured to align the reference image with the target image according to the offset and the deformable convolution layer.

[0166] As an optional implementation of the embodiment of the present invention, the related feature points corresponding to the first feature point in the target feature include: the second feature point and feature points within the neighborhood of the first preset value of the second feature point;

[0167] The second feature point is a feature point in the reference feature whose pixel coordinates are the same as the pixel coordinates of the first feature point.

[0168] As an optional implementation manner of the embodiment of the present invention, the similarity acquisition unit 92 is specifically used to determine the first spatial domain corresponding to the third feature point in the target feature, the first spatial domain being the spatial domain formed by the third feature point and the feature points in the neighborhood of the second preset value of the third feature point; determine the second spatial domain corresponding to the fourth feature point in the reference feature, the second spatial domain being the spatial domain formed by the fourth feature point and the feature points in the neighborhood of the second preset value of the fourth feature point; the fourth feature point is a related feature point corresponding to the third feature; calculate the outer product of the feature points in each feature group to obtain the outer product of each feature group; the feature group includes feature points belonging to the first spatial domain and feature points belonging to the second spatial domain, and the position of the feature points belonging to the first spatial domain in the first spatial domain is the same as the position of the feature points belonging to the second spatial domain in the second spatial domain; sum the outer products of each feature group to obtain the similarity between the third feature point and the fourth feature point.

[0169] As an optional implementation of the embodiment of the present invention, the target feature is a feature obtained by extracting features from pixels in the target image, and the reference feature is a feature obtained by extracting features from pixels in the reference image;

[0170] or;

[0171] The target feature is a feature obtained by extracting features from pixel points in the target image and downsampling the extracted features at a preset downsampling rate; the reference feature is a feature obtained by extracting features from pixel points in the reference image and downsampling the extracted features at the preset downsampling rate.

[0172] As an optional implementation of an embodiment of the present invention, the offset acquisition unit 93 is specifically used to concatenate the similarity feature and the target feature in the channel dimension to obtain an offset prediction feature; input the offset prediction feature into the offset prediction convolution layer; and obtain the output of the offset prediction convolution layer as the offset of the target feature and the reference feature.

[0173] As an optional implementation of the embodiment of the present invention, the target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the similarity feature includes sub-similarity features of multiple spatial scales;

[0174] The offset acquisition unit 93 is specifically used to predict the convolution layer according to the sub-similarity features of the multiple spatial scales, the sub-target features of the multiple spatial scales and the corresponding offsets of the multiple spatial scales, to obtain the sub-offsets of the sub-target features and sub-reference features of the multiple spatial scales.

[0175] As an optional implementation of the embodiment of the present invention, the processing unit 94 is specifically used to input the reference feature into the deformable convolution layer, and control the shape of the convolution kernel of the deformable convolution layer through the offset;

[0176] The output of the deformable convolutional layer is obtained as an alignment result of the reference feature and the target feature.

[0177] As an optional implementation manner of the embodiment of the present invention, the target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the offset includes sub-offsets of multiple spatial scales;

[0178] The processing unit 94 is specifically used to obtain the alignment results of the sub-target features and the sub-reference features of the multiple spatial scales according to the sub-offsets of the multiple spatial scales and the corresponding deformable convolution layers of the multiple spatial scales; and align the reference image with the target image according to the alignment results of the sub-target features and the sub-reference features of the multiple spatial scales.

[0179] As an optional implementation manner of the embodiment of the present invention, the reference image is the nth image frame of the video to be repaired, and the reference image is the n+1th image frame of the video to be repaired; n is a positive integer.

[0180] The image alignment device provided in this embodiment can execute the image alignment method provided in the above method embodiment. Its implementation principle and technical effect are similar and will not be described in detail here.

[0181] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device. Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig.10 As shown, the electronic device provided in this embodiment includes: a memory 101 and a processor 102, wherein the memory 101 is used to store a computer program; and the processor 102 is used to execute the image alignment method provided in the above embodiment when calling the computer program.

[0182] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements the image alignment method provided in the above embodiment.

[0183] An embodiment of the present invention further provides a computer program product. When the computer program product is run on a computer, the computer implements the image alignment method provided by the embodiment.

[0184] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code.

[0185] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0186] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0187] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image alignment method, It is characterized in that include: Obtain target features and reference features; The target features include feature points corresponding to pixels in the target image, and the reference features include feature points corresponding to pixels in the reference image; Acquire a similarity feature according to the target feature and the reference feature; The similarity feature includes: the similarity between each feature point in the target feature and the corresponding related feature point, and the related feature points corresponding to the feature point in the target feature include: feature points in the reference feature with pixel coordinates that are the same as or adjacent to the pixel coordinates of the feature point in the target feature; Predicting a convolution layer according to the similarity feature, the target feature, and the offset to obtain an offset between the target feature and the reference feature; Aligning the reference image with the target image according to the offset and the deformable convolution layer; The related feature points corresponding to the first feature point in the target feature include: the second feature point and feature points in the neighborhood of the first preset value of the second feature point; the first feature point is any feature point in the target feature; The second feature point is a feature point in the reference feature whose pixel coordinates are the same as the pixel coordinates of the first feature point.

2. The method according to claim 1, It is characterized in that The acquiring of a similarity feature according to the target feature and the reference feature includes: Determine a first spatial domain corresponding to a third feature point in the target feature, where the first spatial domain is a spatial domain formed by feature points in a neighborhood of the third feature point and a second preset value of the third feature point; the third feature point is any feature point in the target feature; Determine a second spatial domain corresponding to a fourth feature point in the reference feature, where the second spatial domain is a spatial domain formed by feature points in a neighborhood of the fourth feature point and the second preset value of the fourth feature point; the fourth feature point is a related feature point corresponding to the third feature; Calculating the outer product of the feature points in each feature group to obtain the outer product of each feature group; the feature group includes feature points belonging to the first spatial domain and feature points belonging to the second spatial domain, and the position of the feature points belonging to the first spatial domain in the first spatial domain is the same as the position of the feature points belonging to the second spatial domain in the second spatial domain; The outer products of each feature group are summed to obtain the similarity between the third feature point and the fourth feature point.

3. The method according to claim 1, It is characterized in that The target feature is a feature obtained by extracting features from pixels in the target image, and the reference feature is a feature obtained by extracting features from pixels in the reference image; or; The target feature is a feature obtained by extracting features from pixel points in the target image and downsampling the extracted features at a preset downsampling rate; the reference feature is a feature obtained by extracting features from pixel points in the reference image and downsampling the extracted features at the preset downsampling rate.

4. The method according to any one of claims 1 to 3, It is characterized in that The predicting convolution layer according to the similarity feature, the target feature and the offset to obtain the offset of the target feature and the reference feature includes: Connecting the similarity feature and the target feature in series in the dimension of the channel to obtain an offset prediction feature; Inputting the offset prediction feature into the offset prediction convolutional layer; The output of the offset prediction convolutional layer is obtained as the offset of the target feature and the reference feature.

5. The method according to any one of claims 1 to 3, It is characterized in that The target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the similarity feature includes sub-similarity features of multiple spatial scales; The predicting convolution layer according to the similarity feature, the target feature and the offset to obtain the offset of the target feature and the reference feature includes: A convolution layer is predicted according to the sub-similarity features of the multiple spatial scales, the sub-target features of the multiple spatial scales, and the corresponding offsets of the multiple spatial scales to obtain sub-offsets of the sub-target features and sub-reference features of the multiple spatial scales.

6. The method according to any one of claims 1 to 3, It is characterized in that The step of aligning the reference image with the target image according to the offset and the deformable convolution layer comprises: Inputting the reference feature into the deformable convolution layer, and controlling the shape of the convolution kernel of the deformable convolution layer by the offset; Obtaining the output of the deformable convolutional layer as an alignment result of the reference feature and the target feature; The reference image is aligned with the target image according to the alignment result of the reference feature and the target feature.

7. The method according to any one of claims 1 to 3, It is characterized in that The target feature includes sub-target features of multiple spatial scales, the reference feature includes sub-reference features of multiple spatial scales, and the offset includes sub-offsets of multiple spatial scales; The step of aligning the reference image with the target image according to the offset and the deformable convolution layer comprises: Acquire alignment results of sub-target features and sub-reference features of the multiple spatial scales according to the sub-offsets of the multiple spatial scales and the corresponding deformable convolutional layers of the multiple spatial scales; The reference image is aligned with the target image according to the alignment results of the sub-target features and the sub-reference features at the multiple spatial scales.

8. The method according to any one of claims 1 to 3, It is characterized in that The target image is the nth image frame of the video to be repaired, and the reference image is the n+1th image frame of the video to be repaired; n is a positive integer.

9. An image alignment device, It is characterized in that include: A feature acquisition unit, used for acquiring target features and reference features; The target features include feature points corresponding to pixels in the target image, and the reference features include feature points corresponding to pixels in the reference image; A similarity acquisition unit, used for acquiring a similarity feature according to the target feature and the reference feature; The similarity feature includes: the similarity between each feature point in the target feature and the corresponding related feature point, and the related feature points corresponding to the feature point in the target feature include: feature points in the reference feature with pixel coordinates that are the same as or adjacent to the pixel coordinates of the feature point in the target feature; An offset acquisition unit, configured to acquire an offset of the target feature and the reference feature according to the similarity feature, the target feature and the offset prediction convolution layer; a processing unit, configured to align the reference image with the target image according to the offset and the deformable convolution layer; The related feature points corresponding to the first feature point in the target feature include: a second feature point and feature points within a neighborhood of a first preset value of the second feature point; the first feature point is any feature point in the target feature; wherein the second feature point is a feature point in the reference feature with the same pixel coordinates as the first feature point.

10. An electronic device, It is characterized in that include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the image alignment method described in any one of claims 1 to 8 when calling the computer program.

11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the image alignment method according to any one of claims 1 to 8.

12. A computer program product, It is characterized in that When the computer program product is run on a computer, the computer is enabled to implement the image alignment method according to any one of claims 1 to 8.