Image alignment method, device, equipment and storage medium

Abstract

The embodiment of the present application provides an image alignment method, device, equipment and storage medium, the method comprising: dividing a current image into multiple current image blocks, dividing a reference image into multiple reference image blocks; determining the initial displacement of each current image block relative to the reference image; determining a matching image block corresponding to each reference image block in the multiple current image blocks according to the initial displacement of each current image block relative to the reference image; determining a local motion vector of the current image relative to the reference image according to the coordinate difference between the matching image block corresponding to each reference image block and the reference image block; aligning the current image with the reference image according to the local motion vector. The accuracy of image alignment is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image alignment method, device, equipment and storage medium. Background Art

[0002] Image alignment refers to extracting the feature points of each of the two images, matching the feature points of the two images, and determining the transformation parameters between the images based on the matched feature points. The transformation parameters can make the two images have the same spatial layout.

[0003] In related technologies, the Gaussian pyramid method is usually used to align images. Specifically, the Gaussian pyramid algorithm is used to downsample the image to obtain multiple levels of images. The resolution of each level is different. The bottom layer image is the original image with the highest resolution, and the top layer image is the image with the lowest resolution. Starting from the top layer of the Gaussian pyramid, the two images are aligned, and the alignment result is used as the initial value of the next layer until the bottom layer image is aligned. If the resolution of the top layer image is poor, the key feature points cannot be identified, resulting in low accuracy of image alignment. Summary of the invention

[0004] The present application relates to an image alignment method, device, equipment and storage medium, which improve the accuracy of image alignment.

[0005] In a first aspect, an embodiment of the present application provides an image alignment method, comprising:

[0006] Dividing the current image into a plurality of current image blocks, and dividing the reference image into a plurality of reference image blocks;

[0007] Determining an initial displacement of each current image block relative to the reference image;

[0008] Determining, according to an initial displacement of each current image block relative to the reference image, a matching image block corresponding to each reference image block among the multiple current image blocks;

[0009] Determining a local motion vector of the current image relative to the reference image according to a coordinate difference between a matching image block corresponding to each reference image block and the reference image block;

[0010] The current image and the reference image are aligned according to the local motion vector.

[0011] In a possible implementation, for any reference image block, determining a matching image block corresponding to the reference image block from among the multiple current image blocks according to an initial displacement of the current image block relative to the reference image includes:

[0012] Determining, from among the multiple current image blocks, an initial image block corresponding to the reference image block according to an initial displacement of the current image block relative to the reference image;

[0013] Determine, according to the position of the initial image block in the current image, a plurality of to-be-selected image blocks from the plurality of current image blocks, wherein a coordinate difference between the to-be-selected image blocks and the initial image block is less than or equal to a preset threshold;

[0014] A matching image block corresponding to the reference image block is determined from the multiple image blocks to be selected.

[0015] In a possible implementation, determining a plurality of image blocks to be selected from the plurality of current image blocks according to the position of the initial image block in the current image includes:

[0016] Determine a rectangular area in the initial image with the location of the initial image block as the center;

[0017] The image blocks within the rectangular area are determined as the multiple image blocks to be selected.

[0018] In a possible implementation manner, determining a matching image block corresponding to the reference image block from among the multiple image blocks to be selected includes:

[0019] Obtaining a pixel difference between the reference image block and each image block to be selected;

[0020] The candidate image block with the smallest pixel difference is determined as the matching image block corresponding to the reference image block.

[0021] In a possible implementation, determining an initial displacement of each current image block relative to the reference image includes:

[0022] Determine a plurality of current image units in the current image, and determine a plurality of reference image units in the reference image, wherein the current image unit includes a plurality of current image blocks, and the reference image unit includes a plurality of reference image blocks;

[0023] Determine an initial motion vector between each current image unit and the corresponding reference image unit;

[0024] According to the initial motion vector between each current image unit and the corresponding reference image unit, the initial displacement of each current image block relative to the reference image is determined.

[0025] In a possible implementation manner, determining a plurality of current image units in the current image and determining a plurality of reference image units in the reference image includes:

[0026] Determine a region of interest (ROI) in the current image and the reference image respectively;

[0027] Determine the plurality of current image units in the ROI region of the current image;

[0028] The plurality of reference image units are determined in the ROI region of the reference image.

[0029] In a possible implementation, for any current image block, determining an initial displacement of the current image block relative to the reference image according to an initial motion vector between each current image unit and a corresponding reference image unit includes:

[0030] Determine whether there is a target image unit where the current image block is located in the multiple current image units;

[0031] If yes, determining the initial displacement of the current image block relative to the reference image according to the initial motion vector between the target image unit and the corresponding reference image unit;

[0032] If not, the initial displacement of the current image block relative to the reference image is determined to be 0, or a first image unit that is closest to the current image block is determined among the multiple current image units, and the initial displacement of the current image block relative to the reference image is determined based on an initial motion vector between the first image unit and the corresponding reference image unit.

[0033] In a second aspect, an embodiment of the present application provides an image alignment device, including a division module, a first determination module, a second determination module, a third determination module and an alignment module, wherein:

[0034] The division module is used to divide the current image into a plurality of current image blocks and divide the reference image into a plurality of reference image blocks;

[0035] The first determination module is used to determine an initial displacement of each current image block relative to the reference image;

[0036] The second determination module is used to determine, according to the initial displacement of each current image block relative to the reference image, a matching image block corresponding to each reference image block in the multiple current image blocks;

[0037] The third determination module is used to determine the local motion vector of the current image relative to the reference image according to the coordinate difference between the matching image block corresponding to each reference image block and the reference image block;

[0038] The alignment module is used to align the current image with the reference image according to the local motion vector.

[0039] In a possible implementation manner, for any reference image block, the second determining module is specifically configured to:

[0040] Determining, from among the multiple current image blocks, an initial image block corresponding to the reference image block according to an initial displacement of the current image block relative to the reference image;

[0041] Determine, according to the position of the initial image block in the current image, a plurality of to-be-selected image blocks from the plurality of current image blocks, wherein a coordinate difference between the to-be-selected image blocks and the initial image block is less than or equal to a preset threshold;

[0042] A matching image block corresponding to the reference image block is determined from the multiple image blocks to be selected.

[0043] In a possible implementation manner, the second determining module is specifically configured to:

[0044] Determine a rectangular area in the initial image with the location of the initial image block as the center;

[0045] The image blocks within the rectangular area are determined as the multiple image blocks to be selected.

[0046] In a possible implementation manner, the second determining module is specifically configured to:

[0047] Obtaining a pixel difference between the reference image block and each image block to be selected;

[0048] The candidate image block with the smallest pixel difference is determined as the matching image block corresponding to the reference image block.

[0049] In a possible implementation manner, the first determining module is specifically configured to:

[0050] Determine a plurality of current image units in the current image, and determine a plurality of reference image units in the reference image, wherein the current image unit includes a plurality of current image blocks, and the reference image unit includes a plurality of reference image blocks;

[0051] Determine an initial motion vector between each current image unit and the corresponding reference image unit;

[0052] According to the initial motion vector between each current image unit and the corresponding reference image unit, the initial displacement of each current image block relative to the reference image is determined.

[0053] In a possible implementation manner, the first determining module is specifically configured to:

[0054] Determine a region of interest (ROI) in the current image and the reference image respectively;

[0055] Determine the plurality of current image units in the ROI region of the current image;

[0056] The plurality of reference image units are determined in the ROI region of the reference image.

[0057] In a possible implementation manner, for any current image block, the first determining module is specifically configured to:

[0058] Determine whether there is a target image unit where the current image block is located in the multiple current image units;

[0059] If yes, determining the initial displacement of the current image block relative to the reference image according to the initial motion vector between the target image unit and the corresponding reference image unit;

[0060] If not, the initial displacement of the current image block relative to the reference image is determined to be 0, or a first image unit that is closest to the current image block is determined among the multiple current image units, and the initial displacement of the current image block relative to the reference image is determined based on an initial motion vector between the first image unit and the corresponding reference image unit.

[0061] In a third aspect, an embodiment of the present application provides an image alignment device, including: a processor and a memory;

[0062] The memory stores computer-executable instructions;

[0063] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image alignment method as described in the first aspect.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the image alignment method described in the first aspect.

[0065] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the image alignment method described in the first aspect.

[0066] The embodiment of the present application provides an image alignment method, device, equipment and storage medium, which divides the current image into multiple current image blocks and divides the reference image into multiple reference image blocks; determines the initial displacement of each current image block relative to the reference image; determines the matching image block corresponding to each reference image block in multiple current image blocks according to the initial displacement of each current image block relative to the reference image; determines the local motion vector of the current image relative to the reference image according to the coordinate difference between the matching image block corresponding to each reference image block and the reference image block; and aligns the current image and the reference image according to the local motion vector. The initial displacement is obtained from the original image, and its accuracy is higher than the initial value obtained from the downsampled image. Local alignment of the image using the initial displacement as a reference value can improve the accuracy of image alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0068] Figure 2 is a schematic diagram of an image alignment method in the related art;

[0069] Figure 3 A schematic diagram of a flow chart of an image alignment method provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a flow chart of another image alignment method provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of a reference image and a ROI region in a current image provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of the correspondence between image units and image blocks in the current image provided by an embodiment of the present application;

[0073] Figure 7 A schematic diagram of an initial image block and multiple image blocks to be selected provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of the structure of an image alignment device provided in an embodiment of the present application;

[0075] Fig. 9 A schematic diagram of the structure of an image alignment device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] For ease of understanding, the following Figure 1 , the application scenarios to which the embodiments of the present application are applicable are described.

[0078] Figure 1 This is a schematic diagram of an application scenario provided by the embodiment of this application. Figure 1 The terminal device may be provided with an image fusion module. When the terminal device needs to perform fusion processing on multiple frames of images, the terminal device may fuse the multiple frames of images through the image fusion module to improve the image quality and finally obtain a fused image.

[0079] When fusing multiple frames of images, the images need to be aligned first. In related technologies, the Gaussian pyramid method is usually used to align images. Figure 2 , the image alignment method in the related art is explained.

[0080] Figure 2 is a schematic diagram of an image alignment method in related art. Figure 2 , including two images, Figure 1 and image A. Image 1 is downsampled by the Gaussian pyramid algorithm to obtain three images, image 2, image 3 and image 4, where the image resolution is image 1> image 2> image 3> image 4; image A is downsampled by the Gaussian pyramid algorithm to obtain three images, image B, image C and image D, where the image resolution is image A> image B> image C> image D. When aligning images, image 4 and image D are first aligned to obtain an alignment result of X; image 3 and image C are aligned with the alignment result X as the initial value to obtain an alignment result Y; image 2 and image B are aligned with the alignment result Y as the initial value to obtain an alignment result Z; image 1 and image A are aligned with the alignment result Z as the initial value.

[0081] If the resolution of image D and image 4 is poor, the key feature points in the image cannot be identified, which will lead to inaccurate alignment result X, and thus result in low accuracy of the final image alignment.

[0082] In order to solve the above technical problems, the present application uses the initial displacement of the global alignment result of the original image as a reference value to perform local alignment on the image. Since the accuracy of the initial displacement of the global alignment result of the original image is higher than the initial value obtained from the downsampled image, the accuracy of image alignment can be improved by using the initial displacement of the global alignment result of the original image as a reference value for image alignment.

[0083] The technical solution shown in the present application is described in detail below through specific embodiments. It should be noted that the following embodiments can exist independently or in combination with each other, and the same or displayed content will not be repeatedly described in different embodiments.

[0084] Figure 3 A flowchart of an image alignment method provided in an embodiment of the present application. Figure 3 , the method comprising:

[0085] S301. Divide a current image into a plurality of current image blocks, and divide a reference image into a plurality of reference image blocks.

[0086] The execution subject of the embodiment of the present application can be a terminal device such as a mobile phone or a camera, or it can be an image alignment device set in the terminal device. The image alignment device can be implemented by software or by a combination of software and hardware.

[0087] The reference image may have better image quality than the current image.

[0088] The size of the image block can be determined according to the resolution of the image. The larger the image resolution, the larger the image block can be.

[0089] The image block may be square, and the side length may be 16 to 64 pixels.

[0090] S302: Determine an initial displacement of each current image block relative to the reference image.

[0091] The initial displacement of each current image block relative to the reference image can be determined as follows:

[0092] An initial motion vector between the current image and the reference image is determined, and an initial displacement of each current image block relative to the reference image is determined based on the initial motion vector between the current image and the reference image.

[0093] For example, the current image and the reference image have the same feature A, the coordinates of feature A in the current image are (5,5), and the coordinates of feature A in the reference image are (15,15). Then, based on feature A, the initial motion vector of the current image relative to the reference image can be determined to be (10,10), and further, the initial displacement of each current image block relative to the reference image can be determined to be (10,10).

[0094] In order to make the initial displacement of the current image block relative to the reference image more accurate, the current image may be divided into a plurality of current image units. Specifically, the initial displacement of each current image block relative to the reference image may be determined in the following manner:

[0095] Determine a plurality of current image units in the current image, and determine a plurality of reference image units in the reference image, wherein the current image unit includes a plurality of current image blocks, and the reference image unit includes a plurality of reference image blocks; determine an initial motion vector between each current image unit and a corresponding reference image unit; and determine an initial displacement of each current image block relative to the reference image based on the initial motion vector between each current image unit and the corresponding reference image unit.

[0096] For example, the current image is divided into four current image units A, B, C, and D, and the reference image is divided into four current image units A', B', C', and D', and the initial motion vector of A relative to A' is determined to be (a, b), the initial motion vector of B relative to B' is determined to be (c, d), the initial motion vector of C relative to C' is determined to be (e, f), and the initial motion vector of D relative to D' is determined to be (g, h). The current image is then divided into sixteen image blocks, 1, 2, 3, ..., 16; among them, image blocks 1 to 4 belong to image unit A, image blocks 5 to 8 belong to image unit B, image blocks 9 to 12 belong to image unit C, and image blocks 13 to 16 belong to image unit D; the initial displacement of image blocks 1 to 4 is determined to be (a, b), the initial displacement of image blocks 5 to 8 is determined to be (c, d), the initial displacement of image blocks 9 to 12 is determined to be (e, f), and the initial displacement of image blocks 13 to 16 is determined to be (g, h).

[0097] S303 . Determine, from a plurality of current image blocks, a matching image block corresponding to each reference image block according to an initial displacement of each current image block relative to the reference image.

[0098] The matching image block corresponding to each reference image block may be determined in multiple current image blocks according to the initial displacement of the current image block relative to the reference image in the following manner:

[0099] According to the initial displacement of the current image block relative to the reference image, an initial image block corresponding to the reference image block is determined among the multiple current image blocks; according to the position of the initial image block in the current image, a plurality of candidate image blocks are determined among the multiple current image blocks, and the coordinate difference between the candidate image blocks and the initial image block is less than or equal to a preset threshold; and a matching image block corresponding to the reference image block is determined among the multiple candidate image blocks.

[0100] For example, if the initial displacement of the current image block relative to the reference image is (10, 10), and the coordinates of the original image block in the reference image are (3, 3), then the coordinates of the initial image block corresponding to the original image block in the current image are (13, 13). According to the coordinates of the initial image block in the current image, an image block whose coordinate difference is less than or equal to a preset threshold is determined as a candidate image block, and then a matching image block corresponding to the original image block in the reference image is determined from multiple candidate image blocks.

[0101] S304 : Determine a local motion vector of the current image relative to the reference image according to a coordinate difference between a matching image block corresponding to each reference image block and the reference image block.

[0102] The number of local motion vectors of the current image relative to the reference image is the same as the number of image blocks in the current image.

[0103] For example, if the current image has 1000 image blocks, the number of local motion vectors of the current image relative to the reference image is 1000.

[0104] The local motion vectors of different current image blocks in the current image relative to the reference image block may be the same or different.

[0105] For example, there are sixteen image blocks, 1, 2, 3, ..., 16, in the current image. Among them, the local motion vectors of image blocks 1 to 4 relative to reference image blocks 1 to 4 are the same, all (a, b); the local motion vectors of image blocks 5 to 8 are the same, all (c, d); the local motion vectors of image blocks 9 to 12 are the same, all (e, f); the local motion vectors of image blocks 13 to 16 are the same, all (g, h).

[0106] S305 . Align the current image and the reference image according to the local motion vector.

[0107] The current image and the reference image can be aligned based on the local motion vector in the following way:

[0108] According to the local motion vector, each current image block is aligned with the corresponding reference image block until all current image blocks are aligned with the corresponding reference image blocks.

[0109] For example, if the current image has sixteen image blocks, 1, 2, 3, ..., 16, and the reference image has corresponding sixteen image blocks, 1, 2, 3, ..., 16, the current image block 1 and the reference image block 1 are aligned according to the local motion vector 1, the current image block 2 and the reference image block 2 are aligned according to the local motion vector 2, the current image block 3 and the reference image block 3 are aligned according to the local motion vector 3, and so on, until all sixteen image blocks are aligned.

[0110] exist Figure 3 In the embodiment shown, the current image is first divided into a plurality of current image blocks, and the reference image is divided into a plurality of reference image blocks; then the initial displacement of each current image block relative to the reference image is determined; according to the initial displacement of each current image block relative to the reference image, the matching image block corresponding to each reference image block is determined in the plurality of current image blocks; according to the coordinate difference between the matching image block corresponding to each reference image block and the reference image block, the local motion vector of the current image relative to the reference image is determined; according to the local motion vector, the current image and the reference image are aligned. The initial displacement is obtained from the original image, and its accuracy is higher than the initial value obtained from the downsampled image. Local alignment of the image using the initial displacement as a reference value can improve the accuracy of image alignment. In addition, dividing the image into small image blocks for alignment can increase the feature points used for alignment, further improving the accuracy of image alignment.

[0111] Based on any of the above embodiments, Figure 4 The embodiment shown is used to explain the above-mentioned image alignment method in detail.

[0112] Figure 4 This is a flowchart of another image alignment method provided in an embodiment of the present application. Figure 4 , the method may include:

[0113] S401. Divide a current image into a plurality of current image blocks, and divide a reference image into a plurality of reference image blocks.

[0114] It should be noted that the execution process of S401 can refer to the execution process of S301, which will not be described in detail here.

[0115] S402: Determine a plurality of current image units in the current image, and determine a plurality of reference image units in the reference image, wherein the current image unit includes a plurality of current image blocks, and the reference image unit includes a plurality of reference image blocks.

[0116] A plurality of current image units may be determined in the current image, and a plurality of reference image units may be determined in the reference image in the following manner:

[0117] Determine a region of interest (ROI) in the current image and the reference image respectively; determine the multiple current image units in the ROI region of the current image; and determine the multiple reference image units in the ROI region of the reference image.

[0118] The ROI area can be the same as the original size of the image or smaller than the original size of the image.

[0119] The ROI region of the current image has the same coordinates as the ROI region of the reference image.

[0120] For ease of understanding, below, combined Figure 5 Describe the ROI region in the reference image and the current image.

[0121] Figure 5 Schematic diagram of the reference image and the ROI area in the current image provided in the embodiment of the present application. Figure 5 , including the current image 501 and the reference image 502. If the upper left corner of the current image 501 and the reference image 502 are both the coordinate origin, then the ROI region in the current image 501 and the ROI region in the reference image 502 have the same coordinates on the y-axis and the same coordinates on the x-axis.

[0122] The size of the image unit can be determined according to the resolution of the image. The larger the image resolution, the larger the image unit can be. For example, the size of the image unit can be 128 to 512.

[0123] S403: Determine an initial motion vector between each current image unit and the corresponding reference image unit.

[0124] For example, the current image unit and the corresponding reference image unit have the same feature A, the coordinates of feature A in the current image unit are (3,5), and the coordinates of feature A in the reference image unit are (15,15). Then, based on feature A, the initial motion vector between the current image unit and the corresponding reference image can be determined to be (12,10).

[0125] S404: Determine an initial displacement of each current image block relative to the reference image according to an initial motion vector between each current image unit and a corresponding reference image unit.

[0126] For any current image block, the initial displacement of each current image block relative to the reference image can be determined according to the initial motion vector between each current image unit and the corresponding reference image unit in the following manner:

[0127] Determine whether there is a target image unit where the current image block is located among the multiple current image units; if so, determine the initial displacement of the current image block relative to the reference image according to the initial motion vector between the target image unit and the corresponding reference image unit; if not, determine the initial displacement of the current image block relative to the reference image to be 0, or determine the first image unit closest to the current image block among the multiple current image units, and determine the initial displacement of the current image block relative to the reference image according to the initial motion vector between the first image unit and the corresponding reference image unit.

[0128] For ease of understanding, the following Figure 6 , determining the initial displacement of the current image block relative to the reference image is described in detail.

[0129] Figure 6 This is a schematic diagram of the correspondence between image units and image blocks in the current image provided by the embodiment of the present application. Figure 6 , including image 601 and image 602. Image 601 and image 602 are both current images, except that the sizes of the image blocks divided into the current images are different. Image 601 is to first determine the ROI area in the current image, and then divide the ROI area into four image units A, B, C and D; Image 602 is to directly divide the current image into 1, 2, 3, ..., 63, 64, a total of sixty-four image blocks. Among them, image blocks 10, 11, 12, 18, 19, 20, 26, 27, 28 correspond to image unit A, image blocks 13, 14, 15, 21, 22, 23, 29, 30, 31 correspond to image unit B, image blocks 34, 35, 36, 42, 43, 44, 50, 51, 52 correspond to image unit C, and image blocks 37, 38, 39, 45, 46, 47, 53, 54, 55 correspond to image unit D.

[0130] The initial displacement of image unit A relative to the corresponding reference image is (a, b), the initial displacement of image B relative to the corresponding reference image is (c, d), the initial displacement of image unit C relative to the corresponding reference image is (e, f), and the initial displacement of image D relative to the corresponding reference image is (g, h). (a, b) is determined as the initial displacement of image blocks 10, 11, 12, 18, 19, 20, 26, 27, 28 relative to the reference image, (c, d) is determined as the initial displacement of image blocks 13, 14, 15, 21, 22, 23, 29, 30, 31 relative to the reference image, (e, f) is determined as the initial displacement of image blocks 34, 35, 36, 42, 43, 44, 50, 51, 52 relative to the reference image, and (g, h) is determined as image blocks 34, 35, 36, 42, 43, 44, 50, 51, 52. For other image blocks that do not have corresponding target image units, their initial displacement relative to the reference image can be determined as 0; or, (a, b) is determined as the initial displacement of image blocks 1, 2, 3, 4, 9, 17, and 25 relative to the reference image, (c, d) is determined as the initial displacement of image blocks 5, 6, 7, 8, 16, 24, and 32 relative to the reference image, (e, f) is determined as the initial displacement of image blocks 33, 41, 49, 57, 58, 59, and 60 relative to the reference image, and (g, h) is determined as the initial displacement of image blocks 40, 48, 56, 61, 62, 63, and 64 relative to the reference image.

[0131] If there are multiple target image units in the image block at the same time, the initial displacement of the current image block relative to the reference image is determined based on the initial motion vector between any target image unit and the corresponding reference image unit; or, the average value of the initial motion vectors of all target image units relative to the corresponding reference image units is determined as the initial displacement of the current image block relative to the reference image.

[0132] For example, the initial displacement of image unit A relative to the corresponding reference image is (a, b), the initial displacement of image B relative to the corresponding reference image is (c, d), the initial displacement of image unit C relative to the corresponding reference image is (e, f), and the initial displacement of image D relative to the corresponding reference image is (g, h). If the image block is located in the middle of image units A, B, C, and D, (a, b) can be determined as the initial displacement of the image block relative to the reference image, (c, d) can be determined as the initial displacement of the image block relative to the reference image, (e, f) can be determined as the initial displacement of the image block relative to the reference image, and (g, h) can be determined as the initial displacement of the image block relative to the reference image. Alternatively, (a, b), (c, d), (e, f), and (g, h) are averaged to obtain (x, y), and (x, y) is determined as the initial displacement of the image block relative to the reference image.

[0133] S405 . Determine, from among multiple current image blocks, an initial image block corresponding to the reference image block according to an initial displacement of the current image block relative to the reference image.

[0134] The initial image block corresponding to the reference image block may be determined from among multiple current image blocks according to the initial displacement of the current image block relative to the reference image in the following manner:

[0135] Determine the coordinates of any reference image block in the reference image, determine the first image block in the current image corresponding to the coordinates of the reference image block, and determine the initial image block in the current image substantially corresponding to the reference image block based on the coordinates of the first image block and the initial displacement of the first image block.

[0136] For example, the coordinates of the original image block in the reference image are (3,3), and the coordinates of the first image block in the current image corresponding to the coordinates of the original image block are also (3,3). If the initial displacement of the first image block relative to the reference image is (10,10), then the coordinates of the initial image block in the current image that substantially corresponds to the original image block are (13,13).

[0137] S406: Determine a plurality of image blocks to be selected from a plurality of current image blocks according to the position of the initial image block in the current image.

[0138] The coordinate difference between the selected image block and the initial image block is less than or equal to a preset threshold.

[0139] A plurality of candidate image blocks may be determined from a plurality of current image blocks in the following manner:

[0140] A rectangular area is determined in the initial image with the location of the initial image block as the center; and image blocks within the rectangular area are determined as the multiple image blocks to be selected.

[0141] For ease of understanding, below, combined Figure 7 The process of determining multiple candidate image blocks is described in detail.

[0142] Figure 7 Schematic diagram of an initial image block and multiple image blocks to be selected provided in an embodiment of the present application. Figure 7 , the preset threshold is 10. The coordinates of the initial image block are (20, 20), then a rectangular area whose horizontal coordinates differ from the initial image block by 10 and whose vertical coordinates differ from the initial image block by 10 is first determined, and the 9 image blocks in the rectangular area are all image blocks to be selected.

[0143] S407: Obtain a pixel difference between the reference image block and each of the candidate image blocks.

[0144] The pixel difference between the reference image block and each candidate image block can be obtained in the following way:

[0145] The pixel difference between each pixel in the reference image block and the corresponding pixel in each image block to be selected is obtained, and the pixel difference between all pixels in the reference image block and all corresponding pixels in each image block to be selected is used as the pixel difference between the reference image block and each image block to be selected.

[0146] For example, if there are 256 pixels in the reference image block, obtaining the pixel difference between the reference image block and each image block to be selected may be obtaining the pixel difference between 256 pixels in the reference image block and corresponding 256 pixels in each image block to be selected.

[0147] S408: Determine the candidate image block with the smallest pixel difference as the matching image block corresponding to the reference image block.

[0148] The matching image block may be the same as the initial image block or may be different from the initial image block.

[0149] S409 : Determine a local motion vector of the current image relative to the reference image according to a coordinate difference between the matching image block corresponding to each reference image block and the reference image block.

[0150] It should be noted that the execution process of S409 can refer to the execution process of S304, which will not be repeated here.

[0151] S410: Align the current image and the reference image according to the local motion vector.

[0152] It should be noted that the execution process of S410 can refer to the execution process of S410, which will not be repeated here.

[0153] exist Figure 4In the illustrated embodiment, the current image is first divided into a plurality of current image blocks, and the reference image is divided into a plurality of reference image blocks; then a plurality of current image units are determined in the current image, and a plurality of reference image units are determined in the reference image, wherein the current image unit includes a plurality of current image blocks, and the reference image unit includes a plurality of reference image blocks; an initial motion vector between each current image unit and the corresponding reference image unit is determined; an initial displacement of each current image block relative to the reference image is determined according to the initial motion vector between each current image unit and the corresponding reference image unit; an initial image block corresponding to the reference image block is determined in a plurality of current image blocks according to the initial displacement of the current image block relative to the reference image; a plurality of image blocks to be selected are determined in a plurality of current image blocks according to the position of the initial image block in the current image; a pixel difference between the reference image block and each image block to be selected is obtained; a image block to be selected with the smallest pixel difference is determined as a matching image block corresponding to the reference image block; a local motion vector of the current image relative to the reference image is determined according to the coordinate difference between the matching image block corresponding to each reference image block and the reference image block; and the current image and the reference image are aligned according to the local motion vector. The initial displacement is obtained from the original image, and its accuracy is higher than the initial value obtained from the downsampled image. Using the initial displacement as a reference value to align the image locally can improve the accuracy of image alignment. And dividing the image into small image blocks for alignment can further improve the accuracy of image alignment.

[0154] Figure 8 This is a schematic diagram of the structure of an image alignment device provided in an embodiment of the present application. Figure 8 The image alignment device 10 may include a division module 11, a first determination module 12, a second determination module 13, a third determination module 14 and an alignment module 15, wherein:

[0155] The division module 11 is used to divide the current image into a plurality of current image blocks and divide the reference image into a plurality of reference image blocks;

[0156] The first determination module 12 is used to determine the initial displacement of each current image block relative to the reference image;

[0157] The second determination module 13 is used to determine, according to the initial displacement of each current image block relative to the reference image, a matching image block corresponding to each reference image block in the multiple current image blocks;

[0158] The third determination module 14 is used to determine the local motion vector of the current image relative to the reference image according to the coordinate difference between the matching image block corresponding to each reference image block and the reference image block;

[0159] The alignment module 15 is used to align the current image with the reference image according to the local motion vector.

[0160] In a possible implementation manner, for any reference image block, the second determining module 13 is specifically configured to:

[0161] Determining, from among the multiple current image blocks, an initial image block corresponding to the reference image block according to an initial displacement of the current image block relative to the reference image;

[0162] Determine, according to the position of the initial image block in the current image, a plurality of to-be-selected image blocks from the plurality of current image blocks, wherein a coordinate difference between the to-be-selected image blocks and the initial image block is less than or equal to a preset threshold;

[0163] A matching image block corresponding to the reference image block is determined from the multiple image blocks to be selected.

[0164] In a possible implementation manner, the second determining module 13 is specifically configured to:

[0165] Determine a rectangular area in the initial image with the location of the initial image block as the center;

[0166] The image blocks within the rectangular area are determined as the multiple image blocks to be selected.

[0167] In a possible implementation manner, the second determining module 13 is specifically configured to:

[0168] Obtaining a pixel difference between the reference image block and each image block to be selected;

[0169] The candidate image block with the smallest pixel difference is determined as the matching image block corresponding to the reference image block.

[0170] In a possible implementation manner, the first determining module 12 is specifically configured to:

[0171] Determine a plurality of current image units in the current image, and determine a plurality of reference image units in the reference image, wherein the current image unit includes a plurality of current image blocks, and the reference image unit includes a plurality of reference image blocks;

[0172] Determine an initial motion vector between each current image unit and the corresponding reference image unit;

[0173] According to the initial motion vector between each current image unit and the corresponding reference image unit, the initial displacement of each current image block relative to the reference image is determined.

[0174] In a possible implementation manner, the first determining module 12 is specifically configured to:

[0175] Determine a region of interest (ROI) in the current image and the reference image respectively;

[0176] Determine the plurality of current image units in the ROI region of the current image;

[0177] The plurality of reference image units are determined in the ROI region of the reference image.

[0178] In a possible implementation manner, for any current image block, the first determining module 12 is specifically configured to:

[0179] Determine whether there is a target image unit where the current image block is located in the multiple current image units;

[0180] If yes, determining the initial displacement of the current image block relative to the reference image according to the initial motion vector between the target image unit and the corresponding reference image unit;

[0181] If not, the initial displacement of the current image block relative to the reference image is determined to be 0, or a first image unit that is closest to the current image block is determined among the multiple current image units, and the initial displacement of the current image block relative to the reference image is determined based on an initial motion vector between the first image unit and the corresponding reference image unit.

[0182] The image alignment device 10 provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0183] Fig. 9 This is a schematic diagram of the structure of an image alignment device provided in an embodiment of the present application. Fig. 9 The image alignment device 20 may include: a memory 21 and a processor 22. Exemplarily, the memory 21 and the processor 22 are connected to each other via a bus 23.

[0184] The memory 21 is used to store program instructions;

[0185] The processor 22 is used to execute the program instructions stored in the memory, so as to enable the image alignment device 20 to perform the above-mentioned image alignment method.

[0186] Fig. 9 The image alignment device shown in the embodiment can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0187] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above-mentioned image alignment method.

[0188] The embodiment of the present application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the above-mentioned image alignment method.

[0189] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

[0190] In addition, although each operation is described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or to be performed in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0191] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image alignment method, characterized in that: include: Dividing the current image into a plurality of current image blocks, and dividing the reference image into a plurality of reference image blocks; Determining an initial displacement of each current image block relative to the reference image; Determining, according to an initial displacement of each current image block relative to the reference image, a matching image block corresponding to each reference image block among the multiple current image blocks; Determining a local motion vector of the current image relative to the reference image according to a coordinate difference between a matching image block corresponding to each reference image block and the reference image block; aligning the current image with the reference image according to the local motion vector; For any reference image block; determining a matching image block corresponding to the reference image block among the multiple current image blocks according to an initial displacement of the current image block relative to the reference image, comprising: Determining, from among the multiple current image blocks, an initial image block corresponding to the reference image block according to an initial displacement of the current image block relative to the reference image; Determine, according to the position of the initial image block in the current image, a plurality of image blocks to be selected from the plurality of current image blocks, wherein a coordinate difference between the image blocks to be selected and the initial image block is less than or equal to a preset threshold; A matching image block corresponding to the reference image block is determined from the multiple image blocks to be selected.

2. The method according to claim 1, characterized in that Determining a plurality of image blocks to be selected from the plurality of current image blocks according to the position of the initial image block in the current image comprises: Determine a rectangular area in the initial image with the location of the initial image block as the center; The image blocks within the rectangular area are determined as the multiple image blocks to be selected.

3. The method according to claim 2, characterized in that Determining a matching image block corresponding to the reference image block from among the multiple image blocks to be selected includes: Obtaining a pixel difference between the reference image block and each image block to be selected; The candidate image block with the smallest pixel difference is determined as the matching image block corresponding to the reference image block.

4. The method according to any one of claims 1 to 3, characterized in that: Determining an initial displacement of each current image block relative to the reference image includes: Determine a plurality of current image units in the current image, and determine a plurality of reference image units in the reference image, wherein the current image unit includes a plurality of current image blocks, and the reference image unit includes a plurality of reference image blocks; Determine an initial motion vector between each current image unit and the corresponding reference image unit; According to the initial motion vector between each current image unit and the corresponding reference image unit, the initial displacement of each current image block relative to the reference image is determined.

5. The method according to claim 4, characterized in that Determining a plurality of current image units in the current image and determining a plurality of reference image units in the reference image comprises: Determine a region of interest (ROI) in the current image and the reference image respectively; Determine the plurality of current image units in the ROI region of the current image; The plurality of reference image units are determined in the ROI region of the reference image.

6. The method according to claim 5, characterized in that For any current image block, determining an initial displacement of the current image block relative to the reference image according to an initial motion vector between each current image unit and a corresponding reference image unit includes: Determine whether there is a target image unit where the current image block is located in the multiple current image units; If yes, determining the initial displacement of the current image block relative to the reference image according to the initial motion vector between the target image unit and the corresponding reference image unit; If not, the initial displacement of the current image block relative to the reference image is determined to be 0, or a first image unit that is closest to the current image block is determined among the multiple current image units, and the initial displacement of the current image block relative to the reference image is determined based on an initial motion vector between the first image unit and the corresponding reference image unit.

7. An image alignment device, characterized in that: It includes a division module, a first determination module, a second determination module, a third determination module and an alignment module, wherein: The division module is used to divide the current image into a plurality of current image blocks and divide the reference image into a plurality of reference image blocks; The first determination module is used to determine an initial displacement of each current image block relative to the reference image; The second determination module is used to determine, according to the initial displacement of each current image block relative to the reference image, a matching image block corresponding to each reference image block in the multiple current image blocks; The third determination module is used to determine the local motion vector of the current image relative to the reference image according to the coordinate difference between the matching image block corresponding to each reference image block and the reference image block; The alignment module is used to align the current image with the reference image according to the local motion vector; For any reference image block; the second determination module is used to determine, according to the initial displacement of the current image block relative to the reference image, an initial image block corresponding to the reference image block among the multiple current image blocks; Determine, according to the position of the initial image block in the current image, a plurality of image blocks to be selected from the plurality of current image blocks, wherein a coordinate difference between the image blocks to be selected and the initial image block is less than or equal to a preset threshold; A matching image block corresponding to the reference image block is determined from the multiple image blocks to be selected.

8. An image alignment device, characterized in that: include: Processor, memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image alignment method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the image alignment method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the image alignment method according to any one of claims 1 to 6 when executed by a processor.

Images