Image alignment method and device, computer device and storage medium

Abstract

The application relates to an image alignment method and device, computer equipment, a storage medium and a computer program product. According to texture feature information of a reference frame image, a plurality of first block images with balanced texture complexity are divided in the reference frame image, target second block images with the maximum similarity to each first block image in the reference frame image are determined in second block images of other frames to be aligned, a moving vector of the target second block image is determined according to the position of the target second block in the other frames to be aligned and the position of the first block image corresponding to the target second block in the reference frame image, and image alignment is performed based on the moving vector. Compared with a traditional whole image alignment mode, the present application divides an image into block images with balanced texture complexity, determines a moving vector of each block image based on the similarity between the block images, and performs image alignment based on the moving vector, thereby achieving the effect of improving image alignment accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image alignment method and device, a computer device, a storage medium, and a computer program product. BACKGROUND

[0002] With the development of computer technology, people can take pictures through mobile devices such as mobile phones and cameras. There is a way to synthesize multiple images to enhance the quality of the image. Because of the time difference or hand-held vibration, there is a shift in multiple images. Therefore, the multiple images need to be aligned before being synthesized. The current way of aligning multiple images is usually to align the entire image. However, there is local motion and object deformation in the actual image, which makes the image unable to be accurately matched.

[0003] Therefore, the current image alignment method has the defect of low alignment accuracy. SUMMARY

[0004] Therefore, it is necessary to provide an image alignment method, device, computer device, computer readable storage medium, and computer program product that can improve alignment accuracy.

[0005] In a first aspect, the present application provides an image alignment method, which comprises:

[0006] obtaining at least two frames of images to be aligned, and determining a reference frame image from the at least two frames of images to be aligned;

[0007] dividing a plurality of first block images in the reference frame image according to the texture feature information of the reference frame image; the texture complexity of the plurality of first block images is within a preset range;

[0008] determining at least one second block image from other frames of images to be aligned for each first block image, and obtaining the similarity between the first block image and each second block image in the at least one second block image, and determining a target second block image with the maximum similarity; the size of the second block image corresponds to the size of the first block image;

[0009] determining a movement vector of the target second block image according to the position of the target second block image in the other frames of images to be aligned and the position of the first block image in the reference frame image;

[0010] aligning the target second block image and the first block image corresponding to the target second block image according to the movement vector.

[0011] In one of the embodiments, the dividing a plurality of first block images in the reference frame image according to the texture feature information of the reference frame image comprises:

[0012] taking the reference frame image as a to-be-divided image, and dividing the to-be-divided image into a plurality of candidate block images of the same size;

[0013] obtaining the texture complexity and the edge length of each candidate block image;

[0014] if the texture complexity is greater than a first preset complexity threshold, the edge length is greater than or equal to a preset edge length threshold, and the number of divisions is less than a preset number threshold, taking the candidate block image as a new to-be-divided image, and returning to the step of dividing the to-be-divided image into a plurality of candidate block images of the same size; otherwise, determining the region corresponding to the candidate block image as a target candidate block image in the reference frame image;

[0015] determining a plurality of first block images in the reference frame image according to the determined target candidate block images.

[0016] In one of the embodiments, the determining a plurality of first block images in the reference frame image according to the determined target candidate block images comprises:

[0017] obtaining the texture complexity of the target candidate block images;

[0018] merging the target candidate block images with texture complexity less than or equal to a second preset complexity threshold and adjacent to each other to obtain merged target candidate block images; the second preset complexity threshold is less than the first preset complexity threshold;

[0019] obtaining a plurality of first block images in the reference frame image according to the unmerged target candidate block images and the merged target candidate block images.

[0020] In one of the embodiments, the obtaining the texture complexity of the candidate block image comprises:

[0021] obtaining a gray gradient value corresponding to the candidate block image according to the gray difference between each pixel in the candidate block image and its adjacent pixel;

[0022] determining a standard deviation value corresponding to the candidate block image according to the distance between each pixel in the candidate block image and a preset position in the candidate block image;

[0023] determining an entropy value corresponding to the candidate block image according to each gray level existing in the candidate block image;

[0024] determine the texture complexity of the candidate block image according to a weighted sum of the gray scale gradient value, the standard deviation value and the entropy value.

[0025] In one of the embodiments, the acquiring the similarity between the first block image and each of the at least one second block image comprises:

[0026] if the size of the first block image is greater than or equal to the preset region size, determining the similarity between the first block image and each of the at least one second block image based on a first similarity comparison strategy;

[0027] if the size of the first block image is smaller than the preset region size, and the luminance difference between the second block image and the first block image is smaller than a preset luminance difference threshold, determining the similarity between the first block image and each of the at least one second block image based on a second similarity comparison strategy;

[0028] if the size of the first block image is smaller than the preset region size, and the luminance difference between the second block image and the first block image is greater than or equal to the preset luminance difference threshold, determining the similarity between the first block image and each of the at least one second block image based on a third similarity comparison strategy;

[0029] wherein the first similarity comparison strategy, the second similarity comparison strategy and the third similarity comparison strategy are different from each other.

[0030] In one of the embodiments, the determining the similarity between the first block image and each of the at least one second block image based on the first similarity comparison strategy comprises:

[0031] constructing a homography matrix according to the texture feature information in the first block image and the same texture feature information in each of the second block images, adjusting each of the second block images to match the first block image according to the homography matrix, and determining the similarity between the first block image and each of the second block images according to an error value of the adjusted second block image and the first block image; or

[0032] the determining the similarity between the first block image and each of the at least one second block image based on the second similarity comparison strategy comprises:

[0033] acquiring a search region greater than the first block image size and containing the first block image in other frames to be aligned images, matching the first block image with a second block image in the search region, and determining the similarity of the first block image and the second block image in the search region according to the error value of the second block image in the search region and the first block image; or

[0034] The third similarity comparison strategy includes:

[0035] The Hamming distance between the first block image and the second block image is acquired, and the similarity of the first block image and each of the at least one second block image is obtained according to the Hamming distance.

[0036] In one embodiment, the Hamming distance between the first block image and the second block image is acquired, which includes:

[0037] The gray values of the center pixel point of the first block image and other pixel points adjacent to the center pixel point are acquired, the other pixel points with a gray value greater than that of the center pixel point are recorded as a first value, and the other pixel points with a gray value less than or equal to that of the center pixel point are recorded as a second value; the first value and the second value are different.

[0038] According to the values corresponding to the other pixel points, a first value string corresponding to the first block image is obtained.

[0039] The gray values of the center pixel point of the second block image and other pixel points adjacent to the center pixel point are acquired, the other pixel points with a gray value greater than that of the center pixel point are recorded as a first value, and the other pixel points with a gray value less than or equal to that of the center pixel point are recorded as a second value.

[0040] According to the values corresponding to the other pixel points, a second value string corresponding to the second block image is obtained.

[0041] The first value string and the second value string are subjected to exclusive or operation to obtain the Hamming distance between the first block image and the second block image.

[0042] In one embodiment, the moving vector of the target second block image is determined according to the position of the target second block image in the other frames to be aligned images and the position of the first block image in the reference frame image, which includes:

[0043] determine an initial moving vector of the object corresponding to the target second block image according to a distance and a direction between a position of the target second block image in other frames to be aligned images and a position of the first block image corresponding to the target second block image in the reference frame image, wherein the first block image has the maximum similarity with the target second block image;

[0044] obtain a confidence degree of the initial moving vector;

[0045] if the confidence degree is less than a preset confidence threshold, adjust the first initial moving vector of the object corresponding to the target second block image according to the initial moving vector of the object corresponding to the target second block image and the confidence degree, to obtain a moving vector of the target second block image, wherein the target second block image is adjacent to the target second block image and has a confidence degree greater than or equal to the preset confidence threshold.

[0046] In one embodiment, the confidence degree of the initial moving vector is obtained by:

[0047] obtain a preset normalization parameter corresponding to a similarity comparison strategy of the target second block image and the first block image;

[0048] obtain the confidence degree of the target second block image according to a ratio of the similarity between the target second block image and the first block image corresponding to the maximum similarity and the preset normalization parameter.

[0049] In one embodiment, the reference frame image is determined from the at least two frames to be aligned images by:

[0050] obtain a definition of each frame to be aligned image in the at least two frames to be aligned images;

[0051] take the frame to be aligned image with the maximum definition as the reference frame image.

[0052] In a second aspect, the present application provides an image alignment device, the device comprising:

[0053] an obtaining module, configured to obtain at least two frames to be aligned images, and determine a reference frame image from the at least two frames to be aligned images;

[0054] a dividing module, configured to divide a plurality of first block images in the reference frame image according to texture feature information of the reference frame image, wherein texture complexity of the plurality of first block images is within a preset range;

[0055] The first determining module is used to determine at least one second block image from other frames of images to be aligned for each first block image, and to obtain the similarity between the first block image and each second block image in the at least one second block image, and to determine the target second block image with the highest similarity; the size of the second block image corresponds to that of the first block image;

[0056] The second determining module is used to determine the movement vector of the target second block image based on the position of the target second block image in the other frame images to be aligned, and the position of the first block image in the reference frame image.

[0057] An alignment module is used to align the target second block image and its corresponding first block image based on the movement vector.

[0058] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0059] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0060] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0061] The aforementioned image alignment method, apparatus, computer device, storage medium, and computer program product determine a reference frame image from at least two frames of images to be aligned. Based on the texture feature information of the reference frame image, they divide the reference frame image into multiple first block images with balanced texture complexity. They then determine a target second block image among the second block images of other frames to be aligned, which has the highest similarity to each of the first block images in the reference frame image. Based on the position of the target second block in the other frames to be aligned and the position of its corresponding first block image with the highest similarity in the reference frame image, they determine the movement vector of the target second block image and perform image alignment based on the movement vector. Compared to traditional methods that align the entire image, this scheme improves image alignment accuracy by dividing the image into blocks with balanced texture complexity, determining the movement vector of each block image based on the similarity between blocks, and performing image alignment based on the movement vector. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating an image alignment method in one embodiment;

[0063] Figure 2 A flowchart of a block image division step in an embodiment is shown in FIG. 2A.

[0064] Figure 3 A flowchart of an error determination step in an embodiment is shown in FIG. 3A.

[0065] Figure 4 A flowchart of a vector correction step in an embodiment is shown in FIG. 4A.

[0066] Figure 5 A block diagram of an image alignment device in an embodiment is shown in FIG. 5A.

[0067] Figure 6 An internal structure diagram of a computer device in an embodiment is shown in FIG. 6A. DETAILED DESCRIPTION

[0068] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0069] In an embodiment, as shown in FIG. 1A, an image alignment method is provided, and the embodiment is exemplified by the method applied to a terminal. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction of the terminal and the server, and includes the following steps. Figure 1

[0070] In step S202, at least two frames of to-be-aligned images are obtained, and a reference frame image is determined from the at least two frames of to-be-aligned images.

[0071] The to-be-aligned images can be images captured by a camera or other device of a mobile phone. There can be multiple frames of to-be-aligned images, and the synthesis of multiple frames of images can achieve noise reduction and HDR (High dynamic range) effects on the images. However, due to time differences or handheld shaking, etc., there will be a shift between images when multiple frames of images are synthesized, so it is necessary to align the multiple frames of images. When aligning the images, the terminal can obtain at least two frames of to-be-aligned images, which can be images continuously captured within a preset time period, or multiple images having the same image content. The terminal can determine a reference frame image from the at least two frames of to-be-aligned images.

[0072] ​The reference frame image can be one representative image from the at least two to-be-aligned images. The terminal can determine the reference frame image according to the definition of each frame image. For example, in one embodiment, the reference frame image is determined from the at least two to-be-aligned images, including: obtaining the definition of each frame image from the at least two to-be-aligned images; and taking the frame image with the largest definition as the reference frame image. In this embodiment, the terminal can obtain the definition of each frame image from the at least two to-be-aligned images when determining the reference frame image. The terminal can take the frame image with the largest definition as the reference frame image. That is, the reference frame image can be the most clear and stable image from the at least two to-be-aligned images, and the other to-be-aligned images can be compared and aligned based on the reference frame image.

[0073] In step S204, a plurality of first block images are divided in the reference frame image according to the texture feature information of the reference frame image; and the texture complexity of each of the plurality of first block images is within a preset range.

[0074] The texture feature information can be the texture in the reference frame image. The texture feature information can be the feature information formed by the spatial position of the gray scale distribution. The texture feature information reflects the information of the image in the direction, interval and change amplitude by calculating the correlation between the pixels in the image. The terminal can obtain the texture feature information in the reference frame image, and divide a plurality of first block images in the reference frame image according to the texture feature information of the reference frame image, wherein the texture complexity of each of the plurality of first block images is within a preset range. That is, the terminal can adaptively divide the reference frame image into a plurality of blocks with balanced texture complexity. The size of each block in the reference frame image can be the same or different, and the size of each block needs to be determined according to the texture complexity of each block. For example, the terminal can divide the block with high texture complexity into multiple parts, and divide the block with low texture complexity into fewer parts or combine it with other blocks. That is, the terminal can divide the reference frame image into multiple parts to obtain a plurality of block images with texture complexity within a preset range.

[0075] The terminal can determine the texture complexity of the block image in various ways, for example, the terminal can determine the texture complexity of each block image by gradient value, standard deviation and entropy value, so as to determine whether the block needs to be further divided or combined.

[0076] In step S206, at least one second block image is determined from the other to-be-aligned images for each first block image, the similarity between the first block image and each of the at least one second block image is obtained, and the target second block image with the largest similarity is determined; and the size of the second block image corresponds to the size of the first block image.

[0077] The terminal can divide the reference frame image into block images to obtain a plurality of first block images in the reference frame image. In addition to the reference frame image, there are other frame images to be aligned in the at least two frame images to be aligned. The terminal can determine a second block image in the other frame images to be aligned according to the size of each first block image in the reference frame image. For example, for each first block image, the terminal can determine a second block image with the same size as the first block image in the other frame images to be aligned. The terminal can determine a plurality of second block images with the same size as the first block image in the other frame images to be aligned, that is, there can be one or more second block images in each frame of the other frame images to be aligned.

[0078] After obtaining each second block image in the other frame images to be aligned, the terminal can obtain the similarity between each first block image in the reference frame image and the second block image in the other frame images to be aligned. For example, for each first block image, a plurality of second block images are determined in one frame of the other frame images to be aligned, and each second block image is compared with the first block image for similarity, so that the terminal can obtain the similarity between each second block image and the first block image. Further, the terminal can determine a target second block image with the maximum similarity from the plurality of second block images in the frame of the other frame images to be aligned, and the target second block image is the most similar part to the first block image in the frame of the other frame images to be aligned. Since there can be a plurality of first block images in the reference frame image, after the terminal performs the similarity comparison for each first block image, the terminal can obtain a target second block image most similar to each first block image in the frame of the other frame images to be aligned. In the frame of the other frame images to be aligned, there can be a plurality of second block images most similar to the first block image, and the terminal can determine which is the target second block image by calculating the matching error between the first block image and the second block image with the maximum similarity. The matching error can be determined by the distance information between the first block image and the second block image.

[0079] When there are a plurality of other frame images to be aligned, the terminal can determine at least one second block image in each frame of the other frame images to be aligned, and determine the target second block image in each frame of the other frame images to be aligned according to the above method.

[0080] In step S208, the movement vector of the target second block image is determined according to the position of the target second block image in the other frame images to be aligned and the position of the first block image in the reference frame image.

[0081] The target second block image can be a second block image in the other frames to-be-aligned images that has the largest similarity with the corresponding first block image. That is, there is a first block image in the reference frame image that has the largest similarity with the target second block image. However, due to time difference or handheld shaking, the position of the target second block image in the to-be-aligned image can be different from the position of the corresponding first block image in the reference frame image. Therefore, the terminal needs to move the target second block image. The terminal can determine the movement vector of the target second block image according to the position of the target second block image in the other frames to-be-aligned images and the position of the corresponding first block image in the reference frame that has the largest similarity. The movement vector can be a vector for determining the movement direction and distance of the target second block image, so that the target second block image can be aligned with the corresponding first block image.

[0082] In step S210, the images of the target second block image and the corresponding first block image are aligned based on the movement vector.

[0083] Each target second block image has a corresponding movement vector. The terminal can move the target second block image based on the determined movement vector, so that the target second block image is aligned with the corresponding first block image that has the largest similarity. The terminal can move each target second block image in each frame to-be-aligned image based on the movement vector, so that each target second block image is aligned with the corresponding first block image in the reference frame. After the terminal aligns each target second block image in each frame to-be-aligned image with the corresponding first block image, the aligned image can be obtained. Thus, the terminal can realize the alignment of multiple frames of images.

[0084] In the image alignment method, a reference frame image is determined from at least two frames to-be-aligned images. According to the texture feature information of the reference frame image, multiple first block images with balanced texture complexity are divided in the reference frame image. The target second block image in the second block image of the other frames to-be-aligned images that has the largest similarity with each first block image in the reference frame image is determined. The movement vector of the target second block image is determined according to the position of the target second block in the other frames to-be-aligned images and the position of the corresponding first block image in the reference frame image that has the largest similarity. The image is aligned based on the movement vector. Compared with the traditional alignment method of the whole image, the present application divides the image into block images with balanced texture complexity, determines the movement vector of each block image based on the similarity between the block images, and aligns the image based on the movement vector, thereby improving the accuracy of image alignment.

[0085] In one embodiment, dividing a reference frame image into multiple first block images based on the texture feature information of the reference frame image includes: using the reference frame image as the image to be divided, dividing the image to be divided into multiple candidate block images of the same size; for each candidate block image, obtaining the texture complexity and side length of the candidate block image; if the texture complexity is greater than a first preset complexity threshold, the side length is greater than or equal to a preset side length threshold, and the number of divisions is less than a preset number of divisions threshold, then using the candidate block image as a new image to be divided, and returning to the step of dividing the image to be divided into multiple candidate block images of the same size; otherwise, determining the region corresponding to the candidate block image as a target candidate block image in the reference frame image; and determining multiple first block images in the reference frame image based on the determined multiple target candidate block images.

[0086] In this embodiment, the terminal can divide the reference frame image into multiple first block images with balanced texture complexity based on the texture feature information in the reference frame image. The terminal can first use the reference frame image as the image to be divided, and then divide it into multiple candidate block images of the same size. The terminal can analyze the image within each candidate block, such as analyzing the texture complexity of the image within the block, to determine whether further segmentation or merging is needed. There can be multiple candidate block images. For each candidate block image, the terminal can obtain its texture complexity and side length, thereby determining whether further segmentation or merging of the block is needed based on the texture complexity and the block's side length.

[0087] like Figure 2 As shown, the blocks in image 301 can be the image after dividing the reference frame image into multiple candidate block images of the same size, such as the four square blocks in 301. The terminal can perform texture complexity and side length detection for each candidate block image. The terminal can determine the texture complexity of the block image in various ways. For example, in one embodiment, obtaining the texture complexity of the candidate block image includes: obtaining the grayscale gradient value corresponding to the candidate block image based on the grayscale difference between each pixel in the candidate block image and its neighboring pixels; determining the standard deviation value corresponding to the candidate block image based on the distance between each pixel in the candidate block image and a preset position in the candidate block image; determining the entropy value corresponding to the candidate block image based on each grayscale level present in the candidate block image; and determining the texture complexity of the candidate block image based on the weighted sum of the grayscale gradient value, standard deviation value, and entropy value.

[0088] In this embodiment, the terminal can determine the texture complexity of the candidate block image in a manner of gradient value of gray scale, variance and standard deviation, entropy value, and the like. The image gradient is commonly used for edge detection and image complexity analysis, the direction of the gradient points to the direction of the fastest change, the pixel value changes most at the edge, and the gradient value is larger. The variance and standard deviation can describe the dispersion degree between pixels. The entropy value can be a measure for describing the amount of information possessed in the image, indicating the complexity of the image. The higher the complexity of the image, the greater the entropy value, and vice versa. There can be multiple candidate block images. For each candidate block image, the terminal can obtain the gray scale difference value between each pixel and its adjacent pixel in the candidate block image, to obtain the gray scale gradient value corresponding to the candidate block image. The terminal can determine the standard deviation value corresponding to the candidate block image according to the distance between each pixel in the candidate block image and the preset position in the candidate block image. Moreover, the terminal can also determine the entropy value corresponding to the candidate block image according to each gray scale level existing in the candidate block image.

[0089] Specifically, for the gray scale gradient value, the calculation formula can be as follows: Grad(x, y) = dx(i, j) + dy(i, j); dx(i, j) = p(i, j) - p(i-1, j); dy(i, j) = p(i, j) - p(i, j-1); where Grad(x, y) can be the gray scale gradient value of the candidate block image, dx(i, j) is the gray scale change value in the x-axis direction, dy(i, j) is the gray scale change value in the y-axis direction, p(i, j) is the gray scale value of a pixel, p(i, j) - p(i-1, j) represents the gray scale change value of two adjacent pixels in the x-axis direction, and p(i, j) - p(i, j-1) represents the gray scale change value of two adjacent pixels in the y-axis direction. The above variance can also be referred to as the variance, and the formula of the variance and the standard deviation can be as follows:

[0090]

[0091] where Xi represents the position of the i-th pixel in the candidate block image, μ is the preset position in the candidate block image, and N is the total number of pixels in the candidate block image. The above entropy formula can be as follows:

[0092]

[0093] H(X) is an entropy value of a candidate block image, pi can be a probability of each gray level appearing in the candidate block image, the probability can be determined by determining all gray levels appearing in the candidate block image, and then determining pi based on the number of each gray level accounting for the total number of gray levels, and n can be the total number of gray levels in the candidate block image. After the terminal obtains the above gray gradient value, standard deviation value and entropy value, the texture complexity of the candidate block image can be determined according to the weighted sum of the gray gradient value, the standard deviation value and the entropy value, wherein the weights corresponding to the gray gradient value, the standard deviation value and the entropy value in the above weighted sum can be set according to actual conditions.

[0094] After the terminal obtains the texture complexity of each candidate block image and the side length, it can be determined whether the texture complexity is greater than a first preset complexity threshold, whether the side length is greater than or equal to a preset side length threshold, and whether the division times are less than a preset times threshold. If the terminal detects that the texture complexity is greater than the first preset complexity threshold, the side length is greater than or equal to the preset side length threshold, and the division times are less than the preset times threshold, the terminal can determine that the texture of the candidate block image is complex enough and needs to be further divided. The terminal takes the candidate block image as a new image to be divided, and returns to the step of dividing the image to be divided into a plurality of candidate block images of the same size for the next division. Specifically, as shown in 302 in FIG. 3, after calculating the texture complexity of each candidate block image, the terminal determines that the texture of the candidate block image is complex enough through the above determination, and then divides the candidate block image again. Figure 2

[0095] When the terminal detects that the texture complexity of the candidate block image is less than or equal to the first preset complexity threshold, or the side length is less than the preset side length threshold, or the division times are greater than or equal to the preset times threshold, the terminal can determine that the division of the block image needs to be stopped, and the region corresponding to the candidate block image is determined as a target candidate block image in the reference frame image. There can be multiple target candidate block images, and finally the terminal can obtain the division result as shown in 304 and 305 in FIG. 3. That is, each block in 305 can be a target candidate block image, and the terminal can determine a plurality of first block images in the reference frame image according to the plurality of target candidate block images determined. Figure 2

[0096] Through the above embodiments, the terminal can determine the texture complexity of the block image based on multiple parameters, and divide the block with too high texture complexity, so as to obtain multiple block images with balanced texture complexity in the reference frame. Therefore, the terminal can perform image alignment based on the plurality of first block images in the reference frame image, and the accuracy of image alignment is improved.

[0097] ​​In an embodiment, the terminal determines the first block images in the reference frame image according to the determined plurality of target candidate block images, including: obtaining the texture complexity of the plurality of target candidate block images; merging the target candidate block images adjacent to each other and having the texture complexity less than or equal to a second preset complexity threshold to obtain the merged target candidate block images; the second preset complexity threshold is less than the first preset complexity threshold; and determining the first block images in the reference frame image according to the target candidate block images not merged and the merged target candidate block images.

[0098] In the embodiment, after the terminal divides the plurality of candidate block images, the terminal can obtain the plurality of target candidate block images having low texture complexity. The target candidate block images can include the block images having too low texture complexity and too small size. Therefore, the terminal can merge the blocks. After the terminal divides the plurality of target candidate block images, the terminal can obtain the texture complexity of the plurality of target candidate block images, and obtain the target candidate block images having the texture complexity less than or equal to a second preset complexity threshold. The terminal can merge the target candidate block images adjacent to each other and having the texture complexity less than or equal to the second preset complexity threshold to obtain the merged target candidate block images. The second preset complexity threshold is less than the first preset complexity threshold. Therefore, the terminal can keep the texture complexity of the merged target candidate block images within a certain complexity range. Specifically, as shown in 303 of FIG. 3, the terminal can merge the target candidate block images adjacent to each other and having too low texture complexity. Therefore, the terminal can determine the first block images in the reference frame image according to the target candidate block images not merged and the merged target candidate block images. Figure 2

[0099] Through the embodiment, the terminal can merge the blocks adjacent to each other and having too low texture complexity. Therefore, the terminal can obtain the block images in the reference frame image having the texture complexity within a preset complexity range, and perform image alignment based on the block images. The accuracy of image alignment is improved.

[0100] ​In one embodiment, the similarity between the first block image and each of the at least one second block image is obtained by: if the size of the first block image is greater than or equal to a preset region size, determining the similarity between the first block image and each of the at least one second block image based on a first similarity comparison strategy; if the size of the first block image is less than the preset region size, and the brightness difference between the first block image and each of the at least one second block image is less than a preset brightness difference threshold, determining the similarity between the first block image and each of the at least one second block image based on a second similarity comparison strategy; if the size of the first block image is less than the preset region size, and the brightness difference between the first block image and each of the at least one second block image is greater than or equal to the preset brightness difference threshold, determining the similarity between the first block image and each of the at least one second block image based on a third similarity comparison strategy; wherein the first similarity comparison strategy, the second similarity comparison strategy and the third similarity comparison strategy are different from each other.

[0101] In the embodiment, there can be multiple first block images in the reference frame image. For each first block image, the terminal can determine a second block image of the same size in other frame images to be aligned based on the size of the first block image, wherein there can be multiple second block images in each of the other frame images to be aligned, and the terminal can determine the similarity between the second block image in the other frame image to be aligned and the first block image in multiple ways. For example, the terminal can detect the size of the first block image. If the size of the first block image is greater than or equal to a preset region size, the terminal can determine the similarity between the first block image and each of the at least one second block image based on a first similarity comparison strategy. If the terminal detects that the size of the first block image is less than the preset region size, the terminal can detect the brightness difference between the first block image and each of the second block images. If the terminal detects that the size of the first block image is less than the preset region size, and the brightness difference between the first block image and each of the second block images is less than a preset brightness difference threshold, it indicates that the brightness change between the two blocks is balanced, and the terminal can determine the similarity between the first block image and each of the at least one second block image based on a second similarity comparison strategy. If the terminal detects that the size of the first block image is less than the preset region size, and the brightness difference between the first block image and each of the second block images is greater than or equal to the preset brightness difference threshold, it indicates that the brightness change between the two blocks is large, and the terminal can determine the similarity between the first block image and the second block image based on a third similarity comparison strategy.

[0102] The first similarity comparison strategy, the second similarity comparison strategy and the third similarity comparison strategy can be different from each other. The first similarity comparison strategy can be a comparison strategy based on a matrix. The second similarity comparison strategy can be a comparison strategy based on a local block matching manner. The third similarity comparison strategy can be a comparison strategy based on a stereo matching manner. For example, in an embodiment, the similarity of the first block image to each of the at least one second block image is determined based on the first similarity comparison strategy, including: constructing a homography matrix according to texture feature information in the first block image and the same texture feature information in each of the second block images, adjusting each of the second block images to match the first block image according to the homography matrix, and determining the similarity of the first block image to each of the at least one second block image according to error values of the adjusted second block images and the first block image. The similarity of the first block image to each of the at least one second block image is determined based on the second similarity comparison strategy, including: obtaining a search region larger than the first block image and containing the first block image in other frames of images to be aligned, matching the first block image to a second block image in the search region, and determining the similarity of the first block image to the second block image in the search region according to error values of the second block image in the search region and the first block image. The similarity of the first block image to each of the at least one second block image is determined based on the third similarity comparison strategy, including: obtaining a Hamming distance of the first block image and the second block image, and obtaining the similarity of the first block image to each of the at least one second block image according to the Hamming distance.

[0103] In the embodiment, the terminal can determine the similarity between the blocks based on the matching cost, which is also referred to as an error value. The terminal can determine the matching cost based on the size of the block image and the brightness difference between the block images. For a case where the size of the first block image is greater than or equal to a preset region size, the terminal can construct a homography matrix according to texture feature information in the first block image and the same texture feature information in each of the second block images, adjust each of the second block images to match the first block image according to the homography matrix, and determine the similarity of the first block image to each of the at least one second block image based on error values of the adjusted second block images and the first block image. Specifically, when the first block image is large, for example, greater than the preset region size, it indicates that the first block belongs to a flat region or a region with weak texture. The terminal can detect feature points, match the block image to a second block image in other frames of images to be aligned through a homography matrix, and obtain a matching error value E 1_ini and an initial movement vector V 1_ini, where H is a 3*3 homography matrix containing rotation, scaling, translation and other information, and the second block image in the other frame to be aligned image can be transformed by the homography matrix to match the first block image. The terminal obtains the matching error value E 1_ini Then, the similarity between the first block image and each of the at least one second block image can be determined based on the error value, for example, the smaller the error value, the greater the similarity.

[0104] For the case where the size of the first block image is smaller than the preset region size and the luminance difference between the second block image and the first block image is smaller than the preset luminance difference threshold, the terminal can obtain a search region larger than the size of the first block image and containing the first block image in the other frame to be aligned image according to the size of the first block image, match the first block image with the second block image in the search region, and determine the similarity between the first block image and the second block image in the search region according to the error value between the second block image in the search region and the first block image. Specifically, the search region can be a search window determined by the terminal in the other frame to be aligned image, and the terminal determines the second block image in the search region and searches for the second block image most similar to the first block image. When the size of the first block image is small, the complexity of the first block image can be high, and at this time, if the luminance changes of the first block image and the second block image are uniform, the terminal can estimate the matching cost by local block matching, for example, the terminal can search for the most similar second block image based on the first block image in the search region containing the first block image in the other frame to be aligned image. Thus, the matching error value E 2_ini between the first block image and the second block image is obtained, and the initial movement vector V 2_ini between the second block image and the first block image is obtained.

[0105]

[0106] where D p (u,v) represents the matching error value; p represents the norm, usually 1 or 2; (u,v) represents the initial movement vector; I(x+u+u0,y+v+v0) represents the search region; and T(x,y) represents the region of the block image.

[0107] For a case that the size of the first block image is less than the preset region size and the luminance difference between the first block image and the second block image is greater than or equal to the preset luminance difference threshold, the terminal can obtain the Hamming distance between the first block image and the second block image, and obtain the similarity between the first block image and each of the at least one second block image according to the Hamming distance. The Hamming distance can be determined by stereo matching.

[0108] For example, in one embodiment, obtaining the Hamming distance between the first block image and the second block image includes: obtaining the gray values of the center pixel point of the first block image and other pixel points adjacent to the center pixel point, recording other pixel points with a gray value greater than the gray value of the center pixel point as a first value, and recording other pixel points with a gray value less than or equal to the gray value of the center pixel point as a second value; the first value and the second value are different; obtaining a first value string corresponding to the first block image according to the values corresponding to the other pixel points; obtaining the gray values of the center pixel point of the second block image and other pixel points adjacent to the center pixel point, recording other pixel points with a gray value greater than the gray value of the center pixel point as a first value, and recording other pixel points with a gray value less than or equal to the gray value of the center pixel point as a second value; obtaining a second value string corresponding to the second block image according to the values corresponding to the other pixel points; and performing an exclusive or operation on the first value string and the second value string to obtain the Hamming distance between the first block image and the second block image.

[0109] In this embodiment, before obtaining the Hamming distance, the terminal can first convert the pixel values ​​in the first block image and the second block image. The terminal can obtain the grayscale values ​​of the center pixel and other pixels adjacent to it in the first block image, and record the pixels with grayscale values ​​greater than the center pixel as first values, the pixels with grayscale values ​​less than or equal to the center pixel as second values, and the pixels with grayscale values ​​less than or equal to the center pixel as second values. The first and second values ​​are different. Thus, the terminal can obtain a first value string corresponding to the first block image based on the values ​​corresponding to the other pixels. Additionally, the terminal can also obtain the grayscale values ​​of the center pixel and other pixels adjacent to it in the second block image, recording the pixels with grayscale values ​​greater than the center pixel as first values, and the pixels with grayscale values ​​less than or equal to the center pixel as second values. Thus, the terminal can obtain a second value string corresponding to the second block image based on the values ​​corresponding to the other pixels. After obtaining the aforementioned numerical strings, the terminal can perform a mutually exclusive OR (XOR) operation on the first and second numerical strings to obtain the Hamming distance between the first and second blocks. The terminal can also obtain the initial motion vector between the first and second block images. Therefore, the terminal can determine the similarity between the first and second block images based on this Hamming distance.

[0110] Specifically, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating the error determination step in one embodiment. When the texture complexity of the first block image is high and the brightness difference between the first block image and the second block image is large, the terminal can first encode the difference between the center point of the first block image and the second block image and its neighboring pixels using stereo matching, resulting in two strings, such as {11010101} corresponding to the first block image and {10100111} corresponding to the second block image in the figure. Specifically, the terminal can select any point in the block image and draw a rectangle, for example, 3×3, centered on that point. Every point in the rectangle except the center point is compared with the center point. If the gray value is less than that of the center point, it is recorded as 1, and if the gray value is greater than that of the center point, it is recorded as 0, thereby obtaining the value of each pixel. The terminal extracts the values ​​in a certain order to obtain the aforementioned first and second value strings. The terminal can perform a mutually exclusive OR operation on the two numerical strings to obtain the Hamming distance. For example, for {11010101} and {10100111}, the Hamming distance is 4, indicating that the matching error E between the first block image and the second block image is 4. 3_ini The value is 4, and the terminal can also obtain the initial motion vector V based on the positional difference between the first block image and the second block image. 3_iniThus, the terminal can determine the similarity based on the matching error.

[0111] Through the above embodiments, the terminal can determine different similarity comparison strategies between the blocks based on the size of the block images and the luminance change values between the block images, thereby obtaining the similarity between the blocks and the initial movement vector. The terminal aligns the blocks based on the similarity and the initial movement vector, thereby improving the accuracy of image alignment.

[0112] In one embodiment, the terminal determines the movement vector of the target second block image based on the position of the target second block image in the other frame image to be aligned and the position of the first block image in the reference frame image, including: determining the initial movement vector of the object corresponding to the target second block image based on the distance and direction between the position of the target second block image in the other frame image to be aligned and the position of the first block image corresponding to the maximum similarity in the reference frame image; obtaining the confidence of the initial movement vector; if the confidence is less than a preset confidence threshold, adjusting the initial movement vector of the object corresponding to the target second block image based on the initial movement vector of the object corresponding to the target second block image and the confidence thereof, to obtain the movement vector of the target second block image; wherein the target second block image is associated with the target second block image adjacent thereto and having a confidence greater than or equal to the preset confidence threshold.

[0113] In this embodiment, the terminal can determine the movement vector of the target second block image based on the position of each target second block image in the image to be aligned and the position of the first block image corresponding to the maximum similarity in the reference frame image. The terminal can first determine the initial movement vector. The terminal can determine the initial movement vector of the object corresponding to the target second block image based on the distance and direction between the position of the target second block image in the other frame image to be aligned and the position of the first block image corresponding to the maximum similarity in the reference frame image. After the terminal determines the confidence of each initial movement vector, the terminal can obtain the confidence of the initial movement vector and detect the confidence.

[0114] The confidence can be based on the matching error, but the matching error can be obtained according to different similarity comparison strategies, so the terminal can determine the confidence by normalizing each matching error. For example, in an embodiment, obtaining the confidence of the initial movement vector includes: obtaining a preset normalization parameter corresponding to the similarity comparison strategy of the target second block image and the first block image; obtaining the confidence of the target second block image according to the ratio of the similarity of the target second block image and the first block image corresponding to the maximum similarity to the preset normalization parameter. In this embodiment, when the terminal normalizes, it can obtain a preset normalization parameter corresponding to the similarity comparison strategy of the target second block image and the first block image, and obtain the confidence of the target second block image according to the ratio of the similarity of the target second block image and the first block image corresponding to the maximum similarity to the preset normalization parameter. Specifically, since the sizes of the above-mentioned blocks are different, different similarity comparison strategies are used, so the terminal can first determine the preset normalization parameter N of each similarity comparison strategy, for example, the preset normalization parameter has three N1, N2 and N3 for the three similarity comparison strategies. The terminal can obtain the corresponding confidence under each similarity comparison strategy by the formula C n = E n / N n , n = 1, 2, 3. Wherein, E n represents the matching error obtained by the nth similarity comparison strategy, C n represents the confidence calculated based on the matching error of the nth similarity comparison strategy. Thus, the terminal can use the above calculation method to make the error values fall on the same reference for subsequent comparison.

[0115] The terminal can obtain the first initial movement vector of the object corresponding to the target second block image whose confidence is less than the preset confidence threshold, and obtain the second initial movement vector of the object corresponding to the target second block image whose confidence is greater than or equal to the preset confidence threshold. Wherein, the target second block image can have multiple, for each target second block image whose confidence is less than the preset confidence threshold, the terminal can obtain the second initial movement vector of the object corresponding to the target second block image adjacent to the target second block image and whose confidence is greater than or equal to the preset confidence threshold, and adjust the first initial movement vector according to the second initial movement vector and the confidence to obtain the movement vector of the target second block image.

[0116] Wherein, the adjustment can be adjustment in the direction of the vector and the representative movement distance. Specifically, as shown in Figure 4 Figure 4 ​This is a flowchart illustrating the vector correction steps in one embodiment. After the similarity comparison described above, each second block image contains a corresponding confidence score and an initial movement vector. The terminal can adjust the confidence score if it is lower than a preset confidence threshold. c Vector correction is performed on blocks that are below a preset confidence threshold (Th). c Blocks are adjacent and have a confidence level higher than the threshold Th c The blocks are corrected using a weighted sum to obtain the final movement vector V. d The calculation formula is as follows: V d =(∑ i V i ·C i ) / (∑ i C i ), C i >Th c Where i represents the number of adjacent blocks. For example... Figure 4 As shown, dashed arrows represent image blocks with confidence levels below a preset confidence threshold, indicating that these blocks require correction. Solid arrows represent high-confidence image blocks, which can be used to correct low-confidence image blocks. Figure 4 The middle part can be a correction of the vector of the intermediate block. By comparing the before and after correction, it can be seen that by correcting the movement vector, the terminal can transform the block image into a high-confidence block image.

[0117] Through the above embodiments, the terminal can first normalize the confidence level, make a confidence level judgment based on the confidence level under the same benchmark, and correct the block with low confidence level based on its adjacent high confidence level block. Thus, the terminal can move the object in the block image based on the movement vector of the corrected block, thereby improving the accuracy of image alignment.

[0118] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, the embodiment of the present application further provides an image alignment device for implementing the image alignment method as mentioned above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme as recorded in the above method, so the specific definition in one or more image alignment device embodiments provided below can refer to the definition of the image alignment method in the above, which will not be repeated here.

[0120] In one embodiment, as shown in Figure 5 An image alignment device is provided, comprising: an acquisition module 500, a division module 502, a first determination module 504, a second determination module 506, and an alignment module 508, wherein:

[0121] The acquisition module 500 is configured to acquire at least two frames of to-be-aligned images, and determine a reference frame image from the at least two frames of to-be-aligned images.

[0122] The division module 502 is configured to divide a plurality of first block images in the reference frame image according to texture feature information of the reference frame image; the texture complexity of the plurality of first block images is within a preset range.

[0123] The first determination module 504 is configured to determine at least one second block image from other frames of to-be-aligned images for each first block image, and acquire the similarity between each first block image and each second block image in the at least one second block image in the reference frame image, to determine a target second block image with the maximum similarity; the size of the second block image corresponds to the size of the first block image.

[0124] The second determination module 506 is configured to determine a movement vector of the target second block image according to the position of the target second block image in the other frames of to-be-aligned images and the position of the first block image in the reference frame image.

[0125] The alignment module 508 is configured to align the image of the target second block image and the first block image corresponding to the target second block image based on the movement vector.

[0126] In one embodiment, the acquisition module 500 is specifically configured to divide a reference frame image as a to-be-divided image into a plurality of candidate block images of the same size; for each candidate block image, acquire a texture complexity and an edge length of the candidate block image; if the texture complexity is greater than a first preset complexity threshold, the edge length is greater than or equal to a preset edge length threshold, and a division frequency is less than a preset frequency threshold, take the candidate block image as a new to-be-divided image, and return to the step of dividing the to-be-divided image into a plurality of candidate block images of the same size; otherwise, determine a region corresponding to the candidate block image as a target candidate block image in the reference frame image; and determine a plurality of first block images in the reference frame image according to the plurality of target candidate block images determined.

[0127] In one embodiment, the acquisition module 500 is specifically configured to acquire texture complexities of a plurality of target candidate block images; merge target candidate block images with adjacent target candidate block images and with a texture complexity less than or equal to a second preset complexity threshold to obtain merged target candidate block images; the second preset complexity threshold is less than the first preset complexity threshold; and determine a plurality of first block images in the reference frame image according to the unmerged target candidate block images and the merged target candidate block images.

[0128] In one embodiment, the acquisition module 500 is specifically configured to acquire a gray scale gradient value of a candidate block image according to a gray scale difference between each pixel in the candidate block image and an adjacent pixel; determine a standard deviation value of the candidate block image according to a distance between each pixel in the candidate block image and a preset position in the candidate block image; determine an entropy value of the candidate block image according to each gray scale level present in the candidate block image; and determine a texture complexity of the candidate block image according to a weighted sum of the gray scale gradient value, the standard deviation value, and the entropy value.

[0129] In one embodiment, the first determination module 504 is specifically configured to, if a size of a first block image is greater than or equal to a preset region size, determine a similarity between the first block image and each second block image in at least one second block image based on a first similarity comparison strategy; if the size of the first block image is less than the preset region size, and a luminance difference between the second block image and the first block image is less than a preset luminance difference threshold, determine the similarity between the first block image and each second block image in the at least one second block image based on a second similarity comparison strategy; if the size of the first block image is less than the preset region size, and the luminance difference between the second block image and the first block image is greater than or equal to the preset luminance difference threshold, determine the similarity between the first block image and each second block image in the at least one second block image based on a third similarity comparison strategy; and wherein the first similarity comparison strategy, the second similarity comparison strategy, and the third similarity comparison strategy are different from each other.

[0130] In one embodiment, the first determining module 504 is specifically configured to construct a homography matrix according to the texture feature information in the first block image and the same texture feature information in each second block image, adjust matching of each second block image with the first block image according to the homography matrix, and determine the similarity of the first block image with each second block image in the at least one second block image according to an error value of the adjusted second block image and the first block image.

[0131] In one embodiment, the first determining module 504 is specifically configured to obtain a search region greater than the size of the first block image and containing the first block image in the other frames to-be-aligned images, match the first block image with a second block image in the search region, and determine the similarity of the first block image with the second block image in the search region according to an error value of the second block image in the search region and the first block image.

[0132] In one embodiment, the first determining module 504 is specifically configured to obtain a Hamming distance of the first block image and the second block image, and obtain the similarity of the first block image with each second block image in the at least one second block image according to the Hamming distance.

[0133] In one embodiment, the first determining module 504 is specifically configured to obtain the gray values of a center pixel point of the first block image and other pixel points adjacent to the center pixel point, record other pixel points with a gray value greater than that of the center pixel point as a first value, and record other pixel points with a gray value less than or equal to that of the center pixel point as a second value; the first value and the second value are different; obtain a first value string corresponding to the first block image according to the values corresponding to the other pixel points; obtain the gray values of a center pixel point of the second block image and other pixel points adjacent to the center pixel point, record other pixel points with a gray value greater than that of the center pixel point as a first value, and record other pixel points with a gray value less than or equal to that of the center pixel point as a second value; obtain a second value string corresponding to the second block image according to the values corresponding to the other pixel points; and perform an exclusive or operation on the first value string and the second value string to obtain the Hamming distance of the first block image and the second block image.

[0134] In one embodiment, the second determining module 506 is specifically configured to determine an initial moving vector of the object corresponding to the target second block image according to a distance and a direction between a position of the target second block image in the other frame image to be aligned and a position of the first block image corresponding to the target second block image in the reference frame image, in which the first block image has the maximum similarity; obtain a confidence degree of the initial moving vector; if the confidence degree is less than a preset confidence threshold, adjust the initial moving vector of the object corresponding to the target second block image according to the initial moving vector of the object corresponding to the target second block image and the confidence degree, to obtain a moving vector of the target second block image; and the target second block image is associated with the object corresponding to the target second block image and the object corresponding to the target second block image is adjacent and has a confidence degree greater than or equal to the preset confidence threshold.

[0135] In one embodiment, the second determining module 506 is specifically configured to obtain a preset normalization parameter corresponding to a similarity comparison strategy of the target second block image and the first block image; and obtain a confidence degree of the target second block image according to a ratio of the similarity of the target second block image and the first block image corresponding to the target second block image to the preset normalization parameter.

[0136] In one embodiment, the obtaining module 500 is specifically configured to obtain a definition of each frame of the at least two frames of images to be aligned; and take a frame of the images to be aligned with the maximum definition as a reference frame image.

[0137] The above modules in the image alignment device can be realized by software, hardware, and a combination thereof, in whole or in part. The above modules can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to be called and executed by a processor to perform operations corresponding to the above modules.

[0138] In one embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in FIG. 8. Figure 6As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement an image alignment method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0139] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0140] In one embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the image alignment method described above.

[0141] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the image alignment method described above.

[0142] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the image alignment method described above.

[0143] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0145] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0146] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method of image alignment, characterized by, The method comprises: acquiring at least two frames of to-be-aligned images, and determining a reference frame image from the at least two frames of to-be-aligned images; dividing a plurality of first block images in the reference frame image according to texture feature information of the reference frame image; texture complexities of the plurality of first block images are within a preset range; for each first block image, determining at least one second block image from other frames of to-be-aligned images, and acquiring a similarity between the first block image and each second block image in the at least one second block image, to determine a target second block image with the maximum similarity; a size of the target second block image corresponds to a size of the first block image; determining a movement vector of the target second block image according to a position of the target second block image in the other frames of to-be-aligned images and a position of the first block image in the reference frame image; aligning images of the target second block image and the first block image corresponding to the target second block image according to the movement vector; the acquiring of the similarity between the first block image and each second block image in the at least one second block image comprises: if the size of the first block image is greater than or equal to a preset region size, determining the similarity between the first block image and each second block image in the at least one second block image based on a first similarity comparison strategy; if the size of the first block image is less than the preset region size and a luminance difference between the second block image and the first block image is less than a preset luminance difference threshold, determining the similarity between the first block image and each second block image in the at least one second block image based on a second similarity comparison strategy; if the size of the first block image is less than the preset region size and the luminance difference between the second block image and the first block image is greater than or equal to the preset luminance difference threshold, determining the similarity between the first block image and each second block image in the at least one second block image based on a third similarity comparison strategy; wherein the first similarity comparison strategy, the second similarity comparison strategy and the third similarity comparison strategy are different from each other.

2. The method of claim 1, wherein, the dividing of the plurality of first block images in the reference frame image according to the texture feature information of the reference frame image comprises: taking the reference frame image as a to-be-divided image, and dividing the to-be-divided image into a plurality of candidate block images with the same size; for each candidate block image, acquiring a texture complexity and an edge length of the candidate block image; if the texture complexity is greater than a first preset complexity threshold, the edge length is greater than or equal to a preset edge length threshold, and a division frequency is less than a preset frequency threshold, taking the candidate block image as a new to-be-divided image, and returning to the step of dividing the to-be-divided image into a plurality of candidate block images with the same size; otherwise, determining a region corresponding to the candidate block image as a target candidate block image in the reference frame image; determining the plurality of first block images in the reference frame image according to the plurality of determined target candidate block images.

3. The method of claim 2, wherein, The method further includes: obtaining texture complexity of the candidate block image; merging target candidate block images with texture complexity less than or equal to a second preset complexity threshold and adjacent to each other to obtain merged target candidate block images, the second preset complexity threshold being less than the first preset complexity threshold; obtaining the first block image in the reference frame image according to the unmerged target candidate block image and the merged target candidate block image.

4. The method of claim 2, wherein, The method further includes: obtaining a gray gradient value corresponding to the candidate block image according to a gray difference value between each pixel in the candidate block image and a neighboring pixel thereof; determining a standard deviation value corresponding to the candidate block image according to a distance between each pixel in the candidate block image and a preset position in the candidate block image; determining an entropy value corresponding to the candidate block image according to each gray level existing in the candidate block image; determining the texture complexity of the candidate block image according to a weighted sum of the gray gradient value, the standard deviation value and the entropy value.

5. The method of claim 1, wherein, The method further includes: determining the similarity between the first block image and each second block image in the at least one second block image according to a homography matrix constructed according to texture feature information in the first block image and the same texture feature information in each second block image, adjusting each second block image to match the first block image according to the homography matrix, and determining the similarity between the first block image and each second block image in the at least one second block image according to an error value between the adjusted second block image and the first block image; or The method further includes: obtaining a search region larger than the first block image and containing the first block image in other frames of images to be aligned, matching the first block image with a second block image in the search region, and determining the similarity between the first block image and the second block image in the search region according to an error value between the second block image in the search region and the first block image; or The method further includes: obtaining a Hamming distance between the first block image and the second block image, and determining the similarity between the first block image and each second block image in the at least one second block image according to the Hamming distance.

6. The method of claim 5, wherein, The method further includes: The gray scale values of the center pixel point and other pixel points adjacent to the center pixel point of the first block image are obtained, other pixel points with a gray scale value greater than that of the center pixel point are recorded as first values, and other pixel points with a gray scale value less than or equal to that of the center pixel point are recorded as second values; the first values and the second values are different; A first value string corresponding to the first block image is obtained according to the values corresponding to the other pixel points; The gray scale values of the center pixel point and other pixel points adjacent to the center pixel point of the second block image are obtained, other pixel points with a gray scale value greater than that of the center pixel point are recorded as first values, and other pixel points with a gray scale value less than or equal to that of the center pixel point are recorded as second values; A second value string corresponding to the second block image is obtained according to the values corresponding to the other pixel points; The first value string and the second value string are subjected to exclusive or operation to obtain the Hamming distance between the first block image and the second block image.

7. The method of claim 1, wherein, The determination of the movement vector of the target second block image according to the position of the target second block image in the other frame image to be aligned and the position of the first block image in the reference frame image comprises: An initial movement vector of an object corresponding to the target second block image is determined according to the distance and direction of the position of the target second block image in the other frame image to be aligned and the position of the first block image corresponding to the maximum similarity in the reference frame image; The confidence of the initial movement vector is obtained; If the confidence is less than a preset confidence threshold, the first initial movement vector of the object corresponding to the target second block image is adjusted according to the initial movement vector of the object corresponding to the target second block image and the confidence thereof, to obtain the movement vector of the target second block image; wherein the target second block image is adjacent to the associated target second block image and has a confidence greater than or equal to the preset confidence threshold.

8. The method of claim 7, wherein, The confidence of the initial movement vector is obtained, comprising: A preset normalization parameter corresponding to the similarity comparison strategy of the target second block image and the first block image is obtained; The confidence of the target second block image is obtained according to the ratio of the similarity of the target second block image and the first block image corresponding to the maximum similarity to the preset normalization parameter.

9. The method according to any one of claims 1 to 8, characterized in that, The determination of a reference frame image from the at least two frame images to be aligned comprises: The clarity of each frame image to be aligned in the at least two frame images to be aligned is obtained; The frame image to be aligned with the maximum clarity is taken as the reference frame image.

10. An image alignment apparatus characterized by comprising: The device comprises: An acquisition module is configured to acquire at least two frame images to be aligned, and determine a reference frame image from the at least two frame images to be aligned; A division module is configured to divide a plurality of first block images in the reference frame image according to the texture feature information of the reference frame image; the texture complexity of the plurality of first block images is within a preset range. The first determining module is configured to determine at least one second block image from other frame images to be aligned for each first block image, and to obtain a similarity between the first block image and each second block image in the at least one second block image, and to determine a target second block image with the maximum similarity; the second block image corresponds in size to the first block image; if the size of the first block image is greater than or equal to a preset region size, a first similarity comparison strategy is used to determine the similarity between the first block image and each second block image in the at least one second block image; if the size of the first block image is less than the preset region size, and a brightness difference between the second block image and the first block image is less than a preset brightness difference threshold, a second similarity comparison strategy is used to determine the similarity between the first block image and each second block image in the at least one second block image; if the size of the first block image is less than the preset region size, and the brightness difference between the second block image and the first block image is greater than or equal to the preset brightness difference threshold, a third similarity comparison strategy is used to determine the similarity between the first block image and each second block image in the at least one second block image; the first similarity comparison strategy, the second similarity comparison strategy, and the third similarity comparison strategy are different from each other. The second determining module is configured to determine a movement vector of the target second block image according to a position of the target second block image in the other frame images to be aligned and a position of the first block image in the reference frame image. The aligning module is configured to align the target second block image and the first block image corresponding to the target second block image according to the movement vector. 11.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-10. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 9.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 9.

Images