Image alignment method and device based on feature point matching and template matching

Abstract

The present invention relates to the field of image processing technology, and provides an image alignment method and device based on feature point matching and template matching. The purpose is to solve the problem of image alignment difficulties in scenes with repeated textures or little or no texture. The main solution includes: removing the top and bottom camera information areas to prevent alignment interference, and then aligning the two images based on feature point matching and template matching based on the center area of ​​one image against the other to obtain the center point offset of the two images. After alignment using the two methods, the center areas of the two images are compared to obtain the standard correlation coefficient, and the selection of the final aligned image is determined according to the size of the two correlation coefficients and a threshold.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and provides an image alignment method and device based on feature point matching and template matching. Background Art

[0002] Existing technical solutions:

[0003] Detect the feature points of the two images separately (such as SIFT, SURF, etc.), then match them according to the feature point description information, select a certain number of better feature point pairs based on the feature distance, and then calculate the perspective matrix according to the RANSAC (random sampling consensus) algorithm, and finally align them according to the perspective matrix.

[0004] Disadvantages of existing technical solutions:

[0005] It depends on the feature quality of the image. At the same time, it is difficult to align scenes with repeated textures (such as large areas of nets, etc.) and scenes with little or no texture (large areas of walls, sky, etc.). Summary of the Invention

[0006] The purpose of the present invention is to solve the problem of difficulty in image alignment in scenes with repeated textures (such as large areas of nets, etc.) and with little or no texture (large areas of walls, sky, etc.).

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical means:

[0008] The present invention provides an image alignment method based on feature point matching and template matching, comprising the following steps:

[0009] Step 1: Get the image to be aligned imageA and the target image imageB;

[0010] Step 2: Remove the camera watermark information from imageA and imageB:

[0011] Set the pixel values ​​outside the ROI1 area of ​​imageA to zero to obtain image imageA_mask, and set the pixel values ​​outside the ROI1 area of ​​imageB to zero to obtain image imageB_mask;

[0012] Step 3: Match the feature points of imageA_mask and imageB_mask, and obtain the perspective matrix H1 from imageA_mask to imageB_mask;

[0013] Step 4: According to the perspective matrix H1, perform perspective transformation on imageA toward imageB to obtain image imageA_F aligned with imageB and with the same size as imageB.

[0014] Step 5: Cut out ROI2 (roi2_x, roi2_y, roi2_w, roi2_h) from imageA and use it as a template. roi2_x, roi2_y, roi2_w, roi2_h represent the coordinates of the upper left corner of the rectangular area, as well as its width and height.

[0015] Step 6: Use imageA_roi2 as a template to perform template matching on imageB. Specifically, use standard correlation coefficient matching to obtain the best matching position (dst_x, dst_y) and the maximum standard correlation coefficient Vco_T.

[0016] Step 7: Calculate the offset from imageA to imageB using the following formula:

[0017] dx=dst_x-roi2_x

[0018] dy=dst_y-roi2_y

[0019] Step 8: Then construct the translation matrix from imageA to imageB as the perspective matrix H2:

[0020] H2=[[1.,0.,1.0*dx],[0.,1.,1.0*dy],[0.,0.,1.]]

[0021] Step 9: Perform perspective transformation on imageA toward imageB according to the perspective matrix H2 to obtain image imageA_T aligned with imageB, with the same size as imageB.

[0022] Step 10: Crop the image imageA_F_roi2 of ROI2 from imageA_F; crop the image imageB_roi2 of ROI2 from imageB;

[0023] Step 11: Calculate the standard correlation coefficient Vco_F of imageA_F_roi2 and imageB_roi2. The formula for solving the standard correlation coefficient of the two images is as follows:

[0024]

[0025]

[0026]

[0027] Where T(x, y) and I(x, y) represent the pixel values ​​of the two images;

[0028] Step 12: Set the allowable threshold th. If Vco_T>Vco_F+th, use imageA_T as the image for aligning mageA to mageB. Otherwise, use imageA_F as the image for aligning mageA to mageB.

[0029] In the above technical solution, the feature point matching methods include FAST, SIFT, SURF, Harris, and BRIEF.

[0030] In the above technical solution, the perspective matrix implementation includes LMEDS and RANSAC methods.

[0031] The present invention also provides an image alignment device based on feature point matching and template matching, comprising the following modules:

[0032] Image acquisition module: obtains the image to be aligned imageA and the target image imageB;

[0033] Edge trimming module: removes the camera watermark information from imageA and imageB:

[0034] The pixel values ​​outside the ROI1 area of ​​imageA are set to zero to obtain the image imageA_mask, and the pixel values ​​outside the ROI1 area of ​​imageB are set to zero to obtain the image imageB_nask;

[0035] The first perspective matrix module: match the feature points of imageA_mask and imageB_mask, and obtain the perspective matrix H1 from imageA_mask to imageB_mask;

[0036] First alignment module: Based on the perspective matrix H1, imageA is transformed into imageB to obtain an image imageA_F aligned with imageB, and its size is the same as imageB;

[0037] Template module: crop ROI2 (roi2_x, roi2_y, roi2_w, roi2_h) from imageA as the template, roi2_x, roi2_y, roi2_w, roi2_h represent the coordinates of the upper left corner x, y, width and height of the rectangular area respectively;

[0038] Correlation module: Use imageA_roi2 as a template to perform template matching on imageB. Specifically, it uses standard correlation coefficient matching to obtain the best matching position (dst_x, dst_y) and the maximum standard correlation coefficient Vco_T;

[0039] Offset module: Calculate the offset from imageA to imageB according to the following formula:

[0040] dx=dst_x-roi2_x

[0041] dy=dst_y-roi2_y

[0042] Second perspective matrix module: Then construct the translation matrix from imageA to imageB as the perspective matrix H2:

[0043] H2=[[1.,0.,1.0*dx],[0.,1.,1.0*dy],[0.,0.,1.]]

[0044] The second alignment module: According to the perspective matrix H2, imageA is transformed into imageB to obtain an image imageA_T aligned with imageB, and its size is the same as imageB;

[0045] Cropping module: Crops the image imageA_F_roi2 of ROI2 from imageA_F; crops the image imageB_roi2 of ROI2 from imageB;

[0046] Correlation coefficient module: Calculates the standard correlation coefficient Vco_F of imageA_F_roi2 and imageB_roi2. The formula for solving the standard correlation coefficient of the two images is as follows:

[0047]

[0048]

[0049]

[0050] Where T(x, y) and I(x, y) represent the pixel values ​​of the two images;

[0051] Judgment module: Set the allowable threshold th. If Vco_T>Vco_F+th, use imageA_T as the image for aligning mageA to mageB; otherwise, use imageA_F as the image for aligning mageA to mageB.

[0052] In the above technical solution, the feature point matching methods include FAST, SIFT, SURF, Harris, and BRIEF.

[0053] In the above technical solution, the perspective matrix implementation includes LMEDS and RANSAC methods.

[0054] The present invention also provides a storage medium. When a processor executes a program in the storage medium, the processor implements the image alignment method based on feature point matching and template matching as described above.

[0055] Because the present invention adopts the above-mentioned technical means, it has the following beneficial effects: while retaining the advantages of feature point matching, the present invention uses template matching to solve the scenes with a lack of feature points, and can solve the image alignment of repetitive textures with less severe camera distortion (such as large areas of nets, etc.) and scenes with little or no texture (large areas of walls, sky, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of the process of the present invention;

[0057] Figure 2 、 Figure 3 This is a diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with certain specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions of the present invention are intended to fall within the scope of the claims of the present invention.

[0059] In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0060] The present invention proposes an image alignment method based on feature point matching and template matching, which effectively realizes the alignment of image A to image B.

[0061] Step 1: Get the image to be aligned imagcA and the target image imageB;

[0062] Step 2: Set the pixel values ​​outside the ROI1 area of ​​imageA to zero to obtain image imageA_mask, and set the pixel values ​​outside the ROI1 area of ​​imageB to zero to obtain image imageB_mask. In this example, the ROI1 area is (0, 0.2*H, W, 0.6*H). The main purpose is to remove the camera watermark information;

[0063] Step 3: Match the feature points of imageA_mask and imageB_mask (any method: FAST, SIFT, SURF, Harris, BRIEF, etc., SIFT is used in this example), and obtain the perspective matrix H1 from imageA_mask to imageB_mask (any method: LMEDS, RANSAC, etc., RANSAC is used in this example);

[0064] Step 4: According to the perspective matrix H1, perform perspective transformation on imageA toward imageB to obtain image imageA_F aligned with imageB and with the same size as imageB.

[0065] Step 5: Cut out ROI2 (roi2_x, roi2_y, roi2_w, roi2_h) from imageA and use it as a template. In this example, the ROI2 area is (roi2_x, roi2_y, roi2_w, roi2_h) = (0.25*W, 0.25*H, 0.5*W, 0.5*H), where the numerical values ​​represent the coordinates of the upper left corner x, y, width, and height of the rectangular area respectively.

[0066] Step 6: Use imageA_roi2 as a template to perform template matching on imageB (using "standard correlation coefficient matching" or CCOEFF_NORMED) to obtain the best matching position (dst_x, dst_y) and the maximum standard correlation coefficient Vco_T;

[0067] Step 7: Calculate the offset from imageA to imageB using the following formula:

[0068] dx=dst_x-roi2_x

[0069] dy=dst_y-roi2_y

[0070] Step 8: Then construct the translation matrix from imageA to imageB as the perspective matrix H2:

[0071] H2=[[1.,0.,1.0*dx],[0.,1.,1.0*dy],[0.,0.,1.]]

[0072] Step 9: Perform perspective transformation on imageA toward imageB according to the perspective matrix H2 to obtain image imageA_T aligned with imageB, with the same size as imageB.

[0073] Step 10: Crop the image imageA_F_roi2 of ROI2 from imageA_F; crop the image imageB_roi2 of ROI2 from imageB;

[0074] Step 11: Calculate the standard correlation coefficient Vco_F of imageA_F_roi2 and imageB_roi2. The formula for solving the standard correlation coefficient of the two images is as follows:

[0075]

[0076]

[0077]

[0078] Among them, T(x, y) and I(x, y) represent the pixel values ​​of two images.

[0079] Step 12: Set the allowable threshold th (empirical value, 0.03 in the example). If Vco_T>Vco_F+th, use imageA_T as the image for aligning mageA to mageB; otherwise, use imageA_F as the image for aligning mageA to mageB.

[0080] In order to enable those skilled in the art to better understand the technical solution of the present invention, template matching is selected as the final output material case:

[0081] Figure 2 Middle left upper image: imageA Upper right image: imageB;

[0082] Figure 2 Lower left image: Grayscale image of imageA aligned to imageB using feature point matching method;

[0083] Figure 2 Lower right center image: Template matching method, grayscale image of imageA aligned to imageB;

[0084] This example shows that template matching performs better than feature point matching, and template matching will be selected as the final output based on the judgment condition (step 12).

[0085] Example of selecting feature point matching as the final output material:

[0086] Figure 3 Upper left image: imageA Upper right image: imageB

[0087] Figure 3 Lower left image: Grayscale image of imageA aligned to imageB using feature point matching method

[0088] Figure 3 Lower right: Template matching method, grayscale image of imageA aligned to imageB

[0089] This example shows that feature point matching performs better than template matching, and feature point matching will be selected as the final output based on the judgment criteria (step 12).

Claims

1. An image alignment method based on feature point matching and template matching, characterized in that: The following steps are involved: Step 1: Get the image to be aligned imageA and the target image imageB; Step 2: Remove the camera watermark information from imageA and imageB: Set the pixel values ​​outside the ROI1 area of ​​imageA to zero to obtain image imageA_mask, and set the pixel values ​​outside the ROI1 area of ​​imageB to zero to obtain image imageB_mask; Step 3: Match the feature points of imageA_mask and imageB_mask, and obtain the perspective matrix H1 from imageA_mask to imageB_mask; Step 4: According to the perspective matrix H1, perform perspective transformation on imageA toward imageB to obtain image imageA_F aligned with imageB and with the same size as imageB. Step 5: Cut out ROI2 (roi2_x, roi2_y, roi2_w, roi2_h) from imageA and use it as a template. roi2_x, roi2_y, roi2_w, roi2_h represent the coordinates of the upper left corner of the rectangular area, as well as its width and height. Step 6: Use imageA_roi2 as a template to perform template matching on imageB. Specifically, use standard correlation coefficient matching to obtain the best matching position (dst_x, dst_y) and the maximum standard correlation coefficient Vco_T. Step 7: Calculate the offset from imageA to imageB using the following formula: dx=dst_x-roi2_x dy=dst_y-roi2_y Step 8: Then construct the translation matrix from imageA to imageB as the perspective matrix H2: H2=[[1.,0.,1.0*dx],[0.,1.,1.0*dy],[0.,0.,1.]] Step 9: Perform perspective transformation on imageA toward imageB according to the perspective matrix H2 to obtain image imageA_T aligned with imageB, with the same size as imageB. Step 10: Crop the image imageA_F_roi2 of ROI2 from imageA_F; crop the image imageB_roi2 of ROI2 from imageB; Step 11: Calculate the standard correlation coefficient Vco_F of imageA_F_roi2 and imageB_roi2. The formula for solving the standard correlation coefficient of the two images is as follows: Where T(x, y) and I(x, y) represent the pixel values ​​of the two images; Step 12: Set the tolerance threshold th. If Vco_T>Vco_F+th, use imageA_T as the image to align mageA to mageB. Otherwise, use imageA_F as the image to align imageA to imageB.

2. The image alignment method based on feature point matching and template matching according to claim 1, characterized in that: Feature point matching methods include FAST, SIFT, SURF, Harris, and BRIEF.

3. The image alignment method based on feature point matching and template matching according to claim 1, characterized in that: Perspective matrix implementation includes LMEDS and RANSAC methods.

4. An image alignment device based on feature point matching and template matching, characterized in that: Includes the following modules: Image acquisition module: obtains the image to be aligned imageA and the target image imageB; Edge trimming module: removes the camera watermark information from imageA and imageB: Set the pixel values ​​outside the ROI1 area of ​​imageA to zero to obtain image imageA_mask, and set the pixel values ​​outside the ROI1 area of ​​imageB to zero to obtain image imageB_mask; The first perspective matrix module: match the feature points of imageA_mask and imageB_mask, and obtain the perspective matrix H1 from imageA_mask to imageB_mask; First alignment module: Based on the perspective matrix H1, imageA is transformed into imageB to obtain an image imageA_F aligned with imageB, and its size is the same as imageB; Template module: crop ROI2 (roi2_x, roi2_y, roi2_w, roi2_h) from imageA as the template, roi2_x, roi2_y, roi2_w, roi2_h represent the coordinates of the upper left corner x, y, width and height of the rectangular area respectively; Correlation module: Use imageA_roi2 as a template to perform template matching on imageB. Specifically, it uses standard correlation coefficient matching to obtain the best matching position (dst_x, dst_y) and the maximum standard correlation coefficient Vco_T; Offset module: Calculate the offset from imageA to imageB according to the following formula: dx=dst_x-roi2_x dy=dst_y-roi2_y Second perspective matrix module: Then construct the translation matrix from imageA to imageB as the perspective matrix H2: H2=[[1.,0.,1.0*dx],[0.,1.,1.0*dy],[0.,0.,1.]] The second alignment module: According to the perspective matrix H2, imageA is transformed into imageB to obtain an image imageA_T aligned with imageB, and its size is the same as imageB; Cropping module: Crops the image imageA_F_roi2 of ROI2 from imageA_F; crops the image imageB_roi2 of ROI2 from imageB; Correlation coefficient module: Calculates the standard correlation coefficient Vco_F of imageA_F_roi2 and imageB_roi2. The formula for solving the standard correlation coefficient of the two images is as follows: Where T(x, y) and I(x, y) represent the pixel values ​​of the two images; Judgment module: Set the allowable threshold th. If Vco_T>Vco_F+th, use imageA_T as the image for aligning mageA to mageB; otherwise, use imageA_F as the image for aligning imageA to imageB.

5. The image alignment device based on feature point matching and template matching according to claim 1, characterized in that: Feature point matching methods include FAST, SIFT, SURF, Harris, and BRIEF.

6. The image alignment device based on feature point matching and template matching according to claim 1, characterized in that: Perspective matrix implementation includes LMEDS and RANSAC methods.

7. A storage medium, characterized in that: When the processor executes the program in the storage medium, it implements the image alignment method based on feature point matching and template matching as described in any one of claims 1 to 3.

Images