Image alignment method, device, electronic device and storage medium

Abstract

The embodiment of the present application provides an image alignment method, device, electronic device and storage medium, the method comprising: repeatedly performing a training operation until the training of the image alignment network is completed, the training operation comprising: inputting a training image into the image alignment network to obtain a predicted aligned image output by the image alignment network; calculating the edge feature loss between the predicted aligned image and the target image based on the edge feature information obtained of the predicted aligned image and the edge feature information obtained of the target image; updating the parameter value of the image alignment network based on the edge feature loss; inputting the image to be processed into the image alignment network to obtain an aligned image output by the image alignment network. The accuracy of the image alignment network after training is improved, and accordingly, the alignment effect of the aligned image obtained by using the image alignment network after training is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to an image alignment method, device, electronic device and storage medium. Background Art

[0002] Image alignment is a fundamental problem in the field of computer vision. In the fields of image stitching, video stitching, SLAM, etc., it is necessary to align the images to be aligned to obtain aligned images related to the images to be aligned. Image alignment networks based on unsupervised learning are widely used due to their great savings in training costs and high accuracy.

[0003] At present, when training an image alignment network based on unsupervised learning, the difference between the predicted aligned image and the target image in pixel values ​​of all pixels is usually used to represent the difference between the predicted aligned image and the target image, and the parameter values ​​of related parameters in the image alignment network are updated according to the difference between the predicted aligned image and the target image.

[0004] However, in the areas related to light changes in the predicted aligned image and the target image, even when the alignment effect of the predicted aligned image is good, the pixel values ​​of the pixels in the areas related to light changes in the predicted aligned image and the target image still have a large difference. The difference in pixel values ​​of pixels in the areas related to light changes in the predicted aligned image and the target image is caused by light changes, and has a low correlation with the alignment effect. The difference with a low correlation with the alignment effect is used to update the parameter values ​​of the relevant parameters in the image alignment network, which makes it difficult to accurately update the parameter values ​​of the relevant parameters in the image alignment network, affecting the accuracy of the image alignment network. Accordingly, the alignment effect of the aligned image obtained using the image alignment network after training is affected. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present application provides an image alignment method, device, electronic device and storage medium.

[0006] According to a first aspect of an embodiment of the present application, there is provided an image alignment method, comprising:

[0007] The training operation is repeatedly performed until the training of the image alignment network is completed, and the training operation includes: inputting the training image into the image alignment network to obtain a predicted alignment image related to the training image output by the image alignment network; obtaining edge feature information of the predicted alignment image and edge feature information of a target image related to the training image; based on the edge feature information of the predicted alignment image and the edge feature information of the target image, calculating the edge feature loss between the predicted alignment image and the target image, wherein the edge feature loss indicates the difference between the predicted alignment image and the target image in edge features; based on the edge feature loss, updating the parameter value of the image alignment network;

[0008] The image to be processed is input into the image alignment network to obtain the aligned image output by the image alignment network.

[0009] According to a second aspect of an embodiment of the present application, there is provided an image alignment device, comprising:

[0010] A training unit is configured to repeatedly perform a training operation until the training of the image alignment network is completed, the training operation comprising: inputting a training image into the image alignment network to obtain a predicted alignment image related to the training image output by the image alignment network; acquiring edge feature information of the predicted alignment image and edge feature information of a target image related to the training image; calculating an edge feature loss between the predicted alignment image and the target image based on the edge feature information of the predicted alignment image and the edge feature information of the target image, wherein the edge feature loss indicates a difference in edge features between the predicted alignment image and the target image; and updating a parameter value of the image alignment network based on the edge feature loss;

[0011] The alignment unit is configured to input the image to be processed into the image alignment network to obtain the aligned image output by the image alignment network.

[0012] The image alignment method and device provided by the embodiments of the present application realizes updating the parameter values ​​of the relevant parameters in the image alignment network by using the edge feature loss which indicates the difference in edge features between the predicted aligned image and the target image as the alignment target, thereby accurately updating the parameter values ​​of the relevant parameters in the image alignment network. The accuracy of the image alignment network after training is improved. Accordingly, the alignment effect of the aligned images obtained by using the image alignment network after training is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] Figure 1 A flow chart of an image alignment method provided by an embodiment of the present application is shown;

[0015] Figure 2 The structure block diagram of the image alignment device provided by the embodiment of the present application is shown;

[0016] Figure 3 A structural block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0019] Figure 1 A flowchart of an image alignment method provided in an embodiment of the present application is shown, and the method includes:

[0020] Step 101, repeatedly perform the training operation until the training of the image alignment network is completed.

[0021] In the present application, the image alignment network is an image alignment network based on unsupervised learning, for example, A Fast and Robust Homography Estimation Model.

[0022] Before using the image alignment network to perform alignment processing on the image to be processed, the image alignment network is trained. The training of the image alignment network can be completed when the accuracy of the image alignment network reaches an accuracy threshold.

[0023] The image alignment network is trained each time using one training image and a target image related to the training image.

[0024] Each time the image alignment network is trained, the training images used are different. Accordingly, each time the image alignment network is trained, the target images used are different.

[0025] For each training image, a target image related to the training image is used as the alignment target of the training image.

[0026] Each time, the image alignment network is trained using a training image and a target image related to the training image. The actual output of the image alignment network is a predicted alignment image related to the training image, and the target image related to the training image serves as the target output of the image alignment network.

[0027] The training operation is performed each time the image alignment network is trained.

[0028] In the present application, the training operation includes: inputting a training image into an image alignment network to obtain a predicted aligned image related to the training image output by the image alignment network; obtaining edge feature information of the predicted aligned image related to the training image and edge feature information of a target image related to the training image; calculating an edge feature loss between the predicted aligned image related to the training image and the target image based on the edge feature information of the predicted aligned image related to the training image and the edge feature information of the target image related to the training image, wherein the edge feature loss indicates the difference in edge features between the predicted aligned image related to the training image and the target image; and updating a parameter value of the image alignment network based on the edge feature loss between the predicted aligned image related to the training image and the target image.

[0029] Each time the image alignment network is trained, a training image is input into the image alignment network, and the image alignment network outputs a predicted aligned image related to the training image.

[0030] Each time the image alignment network is trained, a Sobel operator, such as an X-direction Sobel operator or a Y-direction Sobel operator, may be used to obtain edge feature information of the alignment image associated with the training image and edge feature information of the target image associated with the training image.

[0031] The Sobel operator Sobel-X in the X direction can be expressed as:

[0032]

[0033] The Sobel operator Sobel-Y in the Y direction can be expressed as:

[0034]

[0035] In the present application, the edge feature information of the predicted aligned image related to the training image may be a Sobel calculation result image corresponding to the predicted aligned image related to the training image.

[0036] When obtaining edge feature information of the predicted aligned image related to the training image, the Sobel operator can be used to calculate the Sobel calculation result of each pixel in the predicted aligned image related to the training image to obtain the Sobel calculation result image corresponding to the predicted aligned image related to the training image.

[0037] For a predicted alignment image related to a training image, each Sobel calculation result image pixel in a Sobel calculation result image corresponding to the predicted alignment image corresponds to a predicted alignment image pixel in the predicted alignment image.

[0038] For each Sobel calculation result image pixel in the Sobel calculation result image corresponding to the predicted aligned image, the position of the Sobel calculation result image pixel in the Sobel calculation result image corresponding to the predicted aligned image is the same as the position of the predicted aligned image pixel corresponding to the Sobel calculation result image pixel in the predicted aligned image.

[0039] For each Sobel calculation result image pixel in the Sobel calculation result image corresponding to the predicted aligned image, the pixel value of the Sobel calculation result image pixel is the Sobel calculation result of the predicted aligned image pixel corresponding to the Sobel calculation result image pixel.

[0040] In the present application, the edge feature information of the target image related to the training image may be a Sobel calculation result image corresponding to the target image related to the training image.

[0041] When acquiring edge feature information of a target image related to a training image, a Sobel operator may be used to calculate a Sobel calculation result of each pixel in the target image related to the training image to obtain a Sobel calculation result image corresponding to the target image related to the training image.

[0042] For a target image related to a training image, each Sobel calculation result image pixel in the Sobel calculation result image corresponding to the target image corresponds to a target image pixel in the target image.

[0043] For each Sobel calculation result image pixel in the Sobel calculation result image corresponding to the target image, the position of the Sobel calculation result image pixel in the Sobel calculation result image corresponding to the target image is the same as the position of the target image pixel corresponding to the Sobel calculation result image pixel in the target image.

[0044] For each Sobel calculation result image pixel in the Sobel calculation result image corresponding to the target image, the pixel value of the Sobel calculation result image pixel is the Sobel calculation result of the target image pixel corresponding to the Sobel calculation result image pixel.

[0045] In the present application, each time the image alignment network is trained, for a predicted aligned image related to the training image and a target image related to the training image, the edge feature loss between the predicted aligned image and the target image can be calculated based on the edge feature information of the predicted aligned image and the edge feature information of the target image.

[0046] The L1 norm loss function can be used to calculate the first L1 loss between the Sobel calculation result image corresponding to the predicted aligned image and the Sobel calculation result image corresponding to the target image, and the first L1 loss between the Sobel calculation result image corresponding to the predicted aligned image and the Sobel calculation result image corresponding to the target image is used as the edge feature loss between the predicted aligned image and the target image.

[0047] When using the L1 norm loss function to calculate the first L1 loss between the Sobel calculation result image corresponding to the predicted aligned image and the Sobel calculation result image corresponding to the target image, the Sobel calculation result image corresponding to the predicted aligned image can be firstly vectorized to obtain a vector representing the Sobel calculation result image corresponding to the predicted aligned image, the vector representing the Sobel calculation result image corresponding to the predicted aligned image includes a vector element corresponding to each pixel in the Sobel calculation result image corresponding to the predicted aligned image, and the value of the vector element is the pixel value of the pixel corresponding to the vector element. At the same time, the Sobel calculation result image corresponding to the target image is vectorized to obtain a vector representing the Sobel calculation result image corresponding to the target image, the vector representing the Sobel calculation result image corresponding to the target image includes a vector element corresponding to each pixel in the Sobel calculation result image corresponding to the target image, and the value of the vector element is the pixel value of the pixel corresponding to the vector element.

[0048] Then, the L1 distance between the vector representing the Sobel calculation result image corresponding to the predicted aligned image and the vector representing the Sobel calculation result image corresponding to the target image can be calculated, and the L1 distance can be used as the first L1 loss between the Sobel calculation result image corresponding to the predicted aligned image and the Sobel calculation result image corresponding to the target image.

[0049] In the present application, each time the image alignment network is trained, for a predicted alignment image related to the training image and a target image related to the training image participating in the training, the parameter value of the image alignment network can be updated based on the edge feature loss between the predicted alignment image and the target image.

[0050] The image alignment network based on unsupervised learning predicts the aligned image using the homography matrix. The parameters related to the edge feature loss in the image alignment network include: parameters of the unit used to generate the homography matrix in the image alignment network based on unsupervised learning.

[0051] By continuously training the image alignment network, the parameter values ​​of the network used to generate the homography matrix can be continuously updated so that the network used to generate the homography matrix can generate an accurate homography matrix, so as to use the accurate homography matrix to complete the accurate alignment process. The process of continuously updating the parameter values ​​of the network used to generate the homography matrix so that the network used to generate the homography matrix can generate an accurate homography matrix can be called a regression process based on an unsupervised homography matrix.

[0052] By continuously using the edge feature loss to update the parameter value of the image alignment network, the edge feature loss becomes smaller and smaller.

[0053] In some embodiments, edge feature information of a predicted aligned image related to a training image includes: a horizontal Sobel calculation result image corresponding to the predicted aligned image related to the training image, and a vertical Sobel calculation result image corresponding to the predicted aligned image related to the training image; and edge feature information of a target image related to the training image includes: a horizontal Sobel calculation result image corresponding to the target image related to the training image, and a vertical Sobel calculation result image corresponding to the target image related to the training image.

[0054] In the present application, the Sobel operator in the horizontal direction, i.e., the Sobel operator in the X direction, and the Sobel operator in the vertical direction, i.e., the Sobel operator in the Y direction, can be used to obtain edge feature information of the predicted aligned image related to the training image and edge feature information of the target image related to the training image.

[0055] In the present application, for a predicted aligned image related to a training image, the Sobel operator in the X direction can be used to calculate the Sobel calculation result of each pixel in the predicted aligned image in the horizontal direction, i.e., the X direction, to obtain the horizontal Sobel calculation result image corresponding to the predicted aligned image.

[0056] Each Sobel calculation result image pixel in the Sobel calculation result image corresponding to the horizontal direction of the predicted aligned image corresponds to a predicted aligned image pixel in the predicted aligned image.

[0057] For each Sobel calculation result image pixel in the horizontal Sobel calculation result image corresponding to the predicted aligned image, the position of the Sobel calculation result image pixel in the horizontal Sobel calculation result image corresponding to the predicted aligned image is the same as the position of the predicted aligned image pixel corresponding to the Sobel calculation result image pixel in the predicted aligned image.

[0058] For each Sobel calculation result image pixel in the horizontal direction Sobel calculation result image corresponding to the predicted aligned image, the pixel value of the Sobel calculation result image pixel is the Sobel calculation result of the predicted aligned image pixel corresponding to the Sobel calculation result image pixel calculated by the X direction Sobel operator.

[0059] In the present application, for a predicted aligned image related to a training image, each Sobel calculation result image pixel in a vertical Sobel calculation result image corresponding to the predicted aligned image corresponds to a predicted aligned image pixel in the predicted aligned image.

[0060] For each Sobel calculation result image pixel in the vertical Sobel calculation result image corresponding to the predicted aligned image, the position of the Sobel calculation result image pixel in the vertical Sobel calculation result image corresponding to the predicted aligned image is the same as the position of the predicted aligned image pixel corresponding to the Sobel calculation result image pixel in the predicted aligned image.

[0061] For each Sobel calculation result image pixel in the vertical Sobel calculation result image corresponding to the predicted aligned image, the pixel value of the Sobel calculation result image pixel is the Sobel calculation result of the predicted aligned image pixel corresponding to the Sobel calculation result image pixel calculated by the Sobel operator in the Y direction.

[0062] In the present application, for a target image related to a training image, the Sobel operator in the X direction can be used to calculate the Sobel calculation result in the X direction for each pixel in the target image to obtain the Sobel calculation result image in the horizontal direction corresponding to the target image.

[0063] Each Sobel calculation result image pixel in the Sobel calculation result image corresponding to the target image in the horizontal direction corresponds to a target image pixel in the target image.

[0064] For each Sobel calculation result image pixel in the horizontal Sobel calculation result image corresponding to the target image, the position of the Sobel calculation result image pixel in the horizontal Sobel calculation result image corresponding to the target image is the same as the position of the target image pixel corresponding to the Sobel calculation result image pixel in the target image.

[0065] For each Sobel calculation result image pixel in the horizontal direction Sobel calculation result image corresponding to the target image, the pixel value of the Sobel calculation result image pixel is the Sobel calculation result of the target image pixel corresponding to the Sobel calculation result image pixel calculated by the Sobel operator in the X direction.

[0066] In the present application, for a target image related to a training image, the Sobel operator in the Y direction can be used to calculate the Sobel calculation result in the Y direction for each pixel in the target image to obtain the Sobel calculation result image in the vertical direction corresponding to the target image.

[0067] Each Sobel calculation result image pixel in the Sobel calculation result image corresponding to the vertical direction of the target image corresponds to a target image pixel in the target image.

[0068] For each Sobel calculation result image pixel in the vertical direction Sobel calculation result image corresponding to the target image, the position of the Sobel calculation result image pixel in the vertical direction Sobel calculation result image corresponding to the target image is the same as the position of the target image pixel corresponding to the Sobel calculation result image pixel in the target image.

[0069] For each Sobel calculation result image pixel in the vertical direction Sobel calculation result image corresponding to the target image, the pixel value of the Sobel calculation result image pixel is the Sobel calculation result of the target image pixel corresponding to the Sobel calculation result image pixel calculated by the Sobel operator in the Y direction.

[0070] Each time the image alignment network is trained, for a predicted alignment image related to the training image and a target image related to the training image, when calculating the edge feature loss between the predicted alignment image and the target image, the L1 norm loss function can be used to calculate the L1 loss between the horizontal Sobel calculation result image corresponding to the predicted alignment image and the horizontal Sobel calculation result image corresponding to the target image. At the same time, the L1 norm loss function can be used to calculate the L1 loss between the vertical Sobel calculation result image corresponding to the predicted alignment image and the vertical Sobel calculation result image corresponding to the target image. The edge feature loss between the predicted alignment image and the target image can be calculated based on the L1 loss between the horizontal Sobel calculation result image corresponding to the predicted alignment image and the horizontal Sobel calculation result image corresponding to the target image and the vertical Sobel calculation result image corresponding to the predicted alignment image and the vertical Sobel calculation result image corresponding to the target image. For example, the mean of the two L1 losses, that is, the sum of the two L1 losses divided by 2, can be used as the edge feature loss between the predicted alignment image and the target image.

[0071] In some embodiments, based on edge feature information of a predicted aligned image related to a training image and edge feature information of a target image related to the training image, calculating the edge feature loss between the predicted aligned image and the target image includes: calculating a first L1 loss between a horizontal Sobel calculation result image corresponding to the predicted aligned image and a horizontal Sobel calculation result image corresponding to the target image; calculating a second L1 loss between a vertical Sobel calculation result image corresponding to the predicted aligned image and a vertical Sobel calculation result image corresponding to the target image; and taking the sum of the first L1 loss and the second L1 loss as the edge feature loss between the predicted aligned image and the target image.

[0072] In the present application, each time the image alignment network is trained, for a predicted alignment image related to the training image and a target image related to the training image, the L1 norm loss function can be used to calculate the first L1 loss between the horizontal Sobel calculation result image corresponding to the predicted alignment image and the horizontal Sobel calculation result image corresponding to the target image. At the same time, the L1 norm loss function can be used to calculate the second L1 loss between the vertical Sobel calculation result image corresponding to the predicted alignment image and the vertical Sobel calculation result image corresponding to the target image. The sum of the first L1 loss and the second L1 loss is used as the edge feature loss between the predicted alignment image and the target image. Based on the edge feature loss between the aligned image and the target image, the parameter value of the image alignment network is updated.

[0073] In some embodiments, based on edge feature information of a predicted aligned image related to a training image and edge feature information of a target image related to the training image, before calculating the edge feature loss between the predicted aligned image and the target image, it also includes: setting the pixel values ​​of pixels whose pixel values ​​in the horizontal direction Sobel calculation result image corresponding to the predicted aligned image are less than a first horizontal direction threshold corresponding to the predicted aligned image to 0, and setting the pixel values ​​of pixels whose pixel values ​​in the horizontal direction Sobel calculation result image corresponding to the target image are less than a second horizontal direction threshold corresponding to the target image to 0; setting the pixel values ​​of pixels whose pixel values ​​in the vertical direction Sobel calculation result image corresponding to the predicted aligned image are less than the first vertical direction threshold corresponding to the predicted aligned image to 0, and setting the pixel values ​​of pixels whose pixel values ​​in the vertical direction Sobel calculation result image corresponding to the target image are less than the second vertical direction threshold corresponding to the target image to 0.

[0074] In the present application, each predicted alignment image associated with a training image may correspond to the same first horizontal threshold. Each target image associated with a training image may correspond to the same second horizontal threshold. Each predicted alignment image associated with a training image may correspond to the same first vertical threshold. Each target image associated with a training image may correspond to the same second vertical threshold.

[0075] In the present application, each time the image alignment network is trained, for a predicted alignment image related to the training image and a target image related to the training image, before calculating the edge feature loss between the predicted alignment image and the target image based on the edge feature information of the predicted alignment image related to the training image and the edge feature information of the target image related to the training image, the pixels with smaller pixel values ​​in the Sobel calculation result image can be further filtered to avoid the interference of the pixel values ​​of the pixels with smaller pixel values ​​in the Sobel calculation result image on the training of the image alignment network. The pixels with smaller pixel values ​​in the Sobel calculation result image usually appear in the weak texture area in the Sobel calculation result image.

[0076] The pixel values ​​of pixels in the Sobel calculation result image in the horizontal direction corresponding to the predicted aligned image, whose pixel values ​​are less than the first horizontal direction threshold value corresponding to the predicted aligned image, may be set to 0.

[0077] In other words, the pixel values ​​of the pixels whose pixel values ​​in the horizontal Sobel calculation result image corresponding to the predicted aligned image are less than the first horizontal threshold value are changed from the original Sobel calculation results to 0.

[0078] The pixel values ​​of pixels in the Sobel calculation result image corresponding to the target image in the horizontal direction, whose pixel values ​​are less than the second horizontal direction threshold value corresponding to the target image, may be set to 0.

[0079] In other words, the pixel values ​​of the pixels in the Sobel calculation result image corresponding to the target image in the horizontal direction, whose pixel values ​​are less than the second horizontal threshold value corresponding to the target image, are changed from the original Sobel calculation results to 0.

[0080] The pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image, whose pixel values ​​are less than the first vertical threshold value corresponding to the predicted aligned image, may be set to 0.

[0081] In other words, the pixel values ​​of the pixels whose pixel values ​​in the vertical Sobel calculation result image corresponding to the predicted aligned image are less than the first vertical threshold value are changed from the original Sobel calculation results to 0.

[0082] The pixel values ​​of pixels in the Sobel calculation result image corresponding to the target image in the vertical direction, whose pixel values ​​are less than the second vertical direction threshold value corresponding to the target image, may be set to 0.

[0083] In other words, the pixel values ​​of the pixels in the vertical Sobel calculation result image corresponding to the target image, whose pixel values ​​are less than the second vertical threshold value, are changed from the original Sobel calculation results to 0.

[0084] In the present application, each time the image alignment network is trained, for a predicted aligned image related to the training image and a target image related to the training image, after filtering pixels with smaller pixel values ​​in the Sobel calculation result image, the edge feature loss between the predicted aligned image and the target image can be calculated based on the horizontal Sobel calculation result image corresponding to the predicted aligned image, the horizontal Sobel calculation result image corresponding to the target image, the vertical Sobel calculation result image corresponding to the predicted aligned image, and the vertical Sobel calculation result image corresponding to the target image.

[0085] The pixel values ​​of pixels in the horizontal Sobel calculation result image corresponding to the predicted aligned image whose pixel values ​​are less than the first horizontal threshold are set to 0 and do not participate in the calculation of the edge feature loss between the predicted aligned image and the target image.

[0086] The pixel values ​​of pixels in the horizontal Sobel calculation result image corresponding to the target image whose pixel values ​​are less than the second horizontal threshold value corresponding to the target image are set to 0 and do not participate in the calculation of the edge feature loss between the predicted aligned image and the target image.

[0087] The pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image whose pixel values ​​are less than the first vertical threshold are set to 0 and do not participate in the calculation of the edge feature loss between the predicted aligned image and the target image.

[0088] The pixel values ​​of the pixels in the vertical Sobel calculation result image corresponding to the target image whose pixel values ​​are less than the second vertical threshold are set to 0 and do not participate in the calculation of the edge feature loss between the predicted aligned image and the target image.

[0089] In some embodiments, a first horizontal threshold corresponding to a predicted aligned image associated with a training image is determined based on the average value of pixel values ​​of pixels in a horizontal Sobel calculation result image corresponding to the predicted aligned image associated with the training image, a second horizontal threshold corresponding to a target image associated with the training image is determined based on the average value of pixel values ​​of pixels in a horizontal Sobel calculation result image corresponding to the target image associated with the training image, a first vertical threshold corresponding to the predicted aligned image associated with the training image is determined based on the average value of pixel values ​​of pixels in a vertical Sobel calculation result image corresponding to the predicted aligned image associated with the training image, and a second vertical threshold corresponding to the target image associated with the training image is determined based on the average value of pixel values ​​of pixels in a vertical Sobel calculation result image corresponding to the target image associated with the training image.

[0090] For each predicted aligned image related to the training image, the average value of the pixel values ​​of the pixels in the horizontal Sobel calculation result image corresponding to the predicted aligned image related to the training image can be determined as the first horizontal threshold corresponding to the predicted aligned image related to the training image.

[0091] For each target image related to the training image, the average value of pixel values ​​of pixels in the horizontal Sobel calculation result image corresponding to the target image related to the training image may be determined as the second horizontal threshold corresponding to the target image related to the training image.

[0092] For each predicted aligned image related to the training image, the average value of the pixel values ​​of the pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image related to the training image can be determined as the first vertical threshold corresponding to the predicted aligned image related to the training image.

[0093] For each target image related to the training image, the average value of pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the target image related to the training image may be determined as the second vertical threshold corresponding to the target image related to the training image.

[0094] In some embodiments, the first horizontal direction threshold value corresponding to the predicted aligned image related to the training image is the product of the average value of the pixel values ​​of the pixels in the horizontal direction Sobel calculation result image corresponding to the predicted aligned image related to the training image and a preset weight, the second horizontal direction threshold value corresponding to the target image related to the training image is the product of the average value of the pixel values ​​of the pixels in the horizontal direction Sobel calculation result image corresponding to the target image related to the training image and a preset weight, the first vertical direction threshold value corresponding to the predicted aligned image related to the training image is the product of the average value of the pixel values ​​of the pixels in the vertical direction Sobel calculation result image corresponding to the predicted aligned image related to the training image and a preset weight, and the second vertical direction threshold value corresponding to the target image related to the training image is the product of the average value of the pixel values ​​of the pixels in the vertical direction Sobel calculation result image corresponding to the target image related to the training image and a preset weight.

[0095] For example, the preset weight is 0.8. For each predicted aligned image associated with the training image, the first horizontal direction threshold corresponding to the predicted aligned image associated with the training image is the product of the average value of the pixel values ​​of the pixels in the horizontal direction Sobel calculation result image corresponding to the predicted aligned image associated with the training image and 0.8.

[0096] For each target image associated with the training image, the second horizontal threshold corresponding to the target image associated with the training image is the product of an average value of pixel values ​​of pixels in the horizontal Sobel calculation result image corresponding to the target image associated with the training image and 0.8.

[0097] For each predicted aligned image associated with the training image, the first vertical threshold corresponding to the predicted aligned image associated with the training image is the product of the average value of the pixel values ​​of the pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image associated with the training image and 0.8.

[0098] For each target image associated with the training image, the second vertical direction threshold corresponding to the target image associated with the training image is the product of the average value of the pixel values ​​of the pixels in the vertical direction Sobel calculation result image corresponding to the target image associated with the training image and 0.8.

[0099] Step 102: input the image to be processed into an image alignment network to obtain an aligned image output by the image alignment network.

[0100] In the present application, after the image alignment network is trained, the image alignment network can be used to perform alignment processing on the image to be processed. The image to be processed is input into the image alignment network to obtain the aligned image output by the image alignment network.

[0101] Please refer to Figure 2 , which shows a structural block diagram of an image alignment device provided in an embodiment of the present application. The device comprises: a training unit 201 and an alignment unit 202.

[0102] The training unit 201 is configured to repeatedly perform a training operation until the training of the image alignment network is completed, and the training operation includes: inputting a training image into the image alignment network to obtain a predicted alignment image related to the training image output by the image alignment network; obtaining edge feature information of the predicted alignment image and edge feature information of a target image related to the training image; based on the edge feature information of the predicted alignment image and the edge feature information of the target image, calculating an edge feature loss between the predicted alignment image and the target image, wherein the edge feature loss indicates a difference in edge features between the predicted alignment image and the target image; and updating a parameter value of the image alignment network based on the edge feature loss;

[0103] The alignment unit 202 is configured to input the image to be processed into the image alignment network to obtain the aligned image output by the image alignment network.

[0104] In some embodiments, the edge feature information of the predicted aligned image includes: the horizontal Sobel calculation result image corresponding to the predicted aligned image, and the vertical Sobel calculation result image corresponding to the predicted aligned image; the edge feature information of the target image includes: the horizontal Sobel calculation result image corresponding to the target image, and the vertical Sobel calculation result image corresponding to the target image.

[0105] In some embodiments, the image alignment device further comprises:

[0106] The filtering unit is configured to, before calculating the edge feature loss between the predicted aligned image and the target image based on the edge feature information of the predicted aligned image and the edge feature information of the target image, set the pixel values ​​of the pixels whose pixel values ​​in the horizontal direction Sobel calculation result image corresponding to the predicted aligned image are less than the first horizontal direction threshold corresponding to the predicted aligned image to 0, and set the pixel values ​​of the pixels whose pixel values ​​in the horizontal direction Sobel calculation result image corresponding to the target image are less than the second horizontal direction threshold corresponding to the target image to 0; set the pixel values ​​of the pixels whose pixel values ​​in the vertical direction Sobel calculation result image corresponding to the predicted aligned image are less than the first vertical direction threshold corresponding to the predicted aligned image to 0, and set the pixel values ​​of the pixels whose pixel values ​​in the vertical direction Sobel calculation result image corresponding to the target image are less than the second vertical direction threshold corresponding to the target image to 0.

[0107] In some embodiments, the first horizontal threshold is determined based on the average pixel values ​​of pixels in the horizontal Sobel calculation result image corresponding to the predicted aligned image, the second horizontal threshold is determined based on the average pixel values ​​of pixels in the horizontal Sobel calculation result image corresponding to the target image, the first vertical threshold is determined based on the average pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image, and the second vertical threshold is determined based on the average pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the target image.

[0108] In some embodiments, the first horizontal threshold is the product of the average pixel values ​​of the pixels in the horizontal Sobel calculation result image corresponding to the predicted aligned image and a preset weight, the second horizontal threshold is the product of the average pixel values ​​of the pixels in the horizontal Sobel calculation result image corresponding to the target image and a preset weight, the first vertical threshold is the product of the average pixel values ​​of the pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image and a preset weight, and the second vertical threshold is the product of the average pixel values ​​of the pixels in the vertical Sobel calculation result image corresponding to the target image and a preset weight.

[0109] In some embodiments, the training unit includes:

[0110] The edge feature loss calculation subunit is configured to calculate a first L1 loss between a horizontal Sobel calculation result image corresponding to the predicted aligned image and a horizontal Sobel calculation result image corresponding to the target image; calculate a second L1 loss between a vertical Sobel calculation result image corresponding to the predicted aligned image and a vertical Sobel calculation result image corresponding to the target image; and use the sum of the first L1 loss and the second L1 loss as the edge feature loss between the predicted aligned image and the target image.

[0111] Figure 3 3 is a block diagram of an electronic device provided by the present application. The electronic device includes a processing component 322, which further includes one or more processors, and a memory resource represented by a memory 332 for storing instructions executable by the processing component 322, such as an application. The application stored in the memory 332 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 322 is configured to execute instructions to perform the above method.

[0112] The electronic device may also include a power supply component 326 configured to perform power management of the electronic device, a wired or wireless network interface 350 configured to connect the electronic device to a network, and an input / output (I / O) interface 358. The electronic device may operate based on an operating system stored in the memory 432, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.

[0113] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by an electronic device to complete the above method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0114] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0115] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image alignment method, characterized in that: The method comprises: The training operation is repeatedly performed until the training of the image alignment network is completed, and the training operation includes: inputting the training image into the image alignment network to obtain the predicted alignment image related to the training image output by the image alignment network; obtaining edge feature information of the predicted alignment image and edge feature information of the target image related to the training image; based on the edge feature information of the predicted alignment image and the edge feature information of the target image, calculating the edge feature loss between the predicted alignment image and the target image, wherein the edge feature information of the predicted alignment image includes: a Sobel calculation result image in the horizontal direction corresponding to the predicted alignment image and a Sobel calculation result image in the vertical direction corresponding to the predicted alignment image, and the edge feature information of the target image includes: a Sobel calculation result image in the horizontal direction corresponding to the target image and a Sobel calculation result image in the vertical direction corresponding to the target image, and the edge feature loss indicates the difference in edge features between the predicted alignment image and the target image; based on the edge feature loss, updating the parameter value of the image alignment network; Input the image to be processed into the image alignment network to obtain the aligned image output by the image alignment network; The step of calculating the edge feature loss between the predicted aligned image and the target image based on the edge feature information of the predicted aligned image and the edge feature information of the target image comprises: Calculating a first L1 loss between a Sobel calculation result image in the horizontal direction corresponding to the predicted aligned image and a Sobel calculation result image in the horizontal direction corresponding to the target image; Calculating a second L1 loss between a Sobel calculation result image in the vertical direction corresponding to the predicted aligned image and a Sobel calculation result image in the vertical direction corresponding to the target image; The sum of the first L1 loss and the second L1 loss is taken as the edge feature loss between the predicted aligned image and the target image.

2. The method according to claim 1, characterized in that Before calculating the edge feature loss between the predicted aligned image and the target image based on the edge feature information of the predicted aligned image and the edge feature information of the target image, the method further includes: The pixel values ​​of the pixels in the Sobel calculation result image in the horizontal direction corresponding to the predicted aligned image, whose pixel values ​​are less than the first horizontal direction threshold value corresponding to the predicted aligned image, are set to 0, and the pixel values ​​of the pixels in the Sobel calculation result image in the horizontal direction corresponding to the target image, whose pixel values ​​are less than the second horizontal direction threshold value corresponding to the target image, are set to 0; The pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image, whose pixel values ​​are less than the first vertical threshold corresponding to the predicted aligned image, are set to 0, and the pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the target image, whose pixel values ​​are less than the second vertical threshold corresponding to the target image, are set to 0.

3. The method according to claim 2, characterized in that The first horizontal threshold is determined based on the average pixel values ​​of pixels in the horizontal Sobel calculation result image corresponding to the predicted aligned image, the second horizontal threshold is determined based on the average pixel values ​​of pixels in the horizontal Sobel calculation result image corresponding to the target image, the first vertical threshold is determined based on the average pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image, and the second vertical threshold is determined based on the average pixel values ​​of pixels in the vertical Sobel calculation result image corresponding to the target image.

4. The method according to claim 3, characterized in that The first horizontal threshold is the product of the average pixel values ​​of the pixels in the horizontal Sobel calculation result image corresponding to the predicted aligned image and the preset weight, the second horizontal threshold is the product of the average pixel values ​​of the pixels in the horizontal Sobel calculation result image corresponding to the target image and the preset weight, the first vertical threshold is the product of the average pixel values ​​of the pixels in the vertical Sobel calculation result image corresponding to the predicted aligned image and the preset weight, and the second vertical threshold is the product of the average pixel values ​​of the pixels in the vertical Sobel calculation result image corresponding to the target image and the preset weight.

5. An image alignment device, characterized in that: The device comprises: A training unit is configured to repeatedly perform a training operation until the training of the image alignment network is completed, the training operation comprising: inputting a training image into the image alignment network to obtain a predicted alignment image related to the training image output by the image alignment network; obtaining edge feature information of the predicted alignment image and edge feature information of a target image related to the training image; based on the edge feature information of the predicted alignment image and the edge feature information of the target image, calculating an edge feature loss between the predicted alignment image and the target image, wherein the edge feature information of the predicted alignment image comprises: a Sobel calculation result image in the horizontal direction corresponding to the predicted alignment image and a Sobel calculation result image in the vertical direction corresponding to the predicted alignment image, the edge feature information of the target image comprises: a Sobel calculation result image in the horizontal direction corresponding to the target image and a Sobel calculation result image in the vertical direction corresponding to the target image, and the edge feature loss indicates the difference in edge features between the predicted alignment image and the target image; based on the edge feature loss, updating a parameter value of the image alignment network; An alignment unit is configured to input the image to be processed into the image alignment network to obtain an aligned image output by the image alignment network; The training unit is specifically used to calculate a first L1 loss between a horizontal Sobel calculation result image corresponding to the predicted aligned image and a horizontal Sobel calculation result image corresponding to the target image; calculate a second L1 loss between a vertical Sobel calculation result image corresponding to the predicted aligned image and a vertical Sobel calculation result image corresponding to the target image; and use the sum of the first L1 loss and the second L1 loss as the edge feature loss between the predicted aligned image and the target image.

6. An electronic device comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 4. 7 . A storage medium, when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in claim 1 .

Images