A fast image deocclusion method based on SURF feature matching

Abstract

The invention discloses a method for fast image deocclusion based on SURF feature matching, which belongs to the field of computer vision. The method comprises the following steps: inputting an occluded image; constructing a Hessian matrix; constructing a Gaussian pyramid scale space; preliminarily determining the position of feature points; accurately locating extreme points; determining the main direction of feature points; constructing a SURF feature point description operator; image registration; and image fusion. The invention can effectively reduce the computation time of image deocclusion, and provides a method for image deocclusion that is both accurate and efficient for the application of computer vision, thereby achieving the expected effect.

Description

Technical Field

[0001] The invention belongs to the field of computer vision, and in particular refers to a fast image de-occlusion method based on SURF feature matching. Background Art

[0002] In recent years, with the continuous development of computer vision technology, image occlusion removal is a very important task in the field of computer vision and has become the basis of many computer vision tasks.

[0003] However, most of the existing image de-occlusion algorithms are based on deep learning methods, which require a lot of preliminary work, such as image preprocessing and model training. In addition, the algorithm has high requirements for the device and cannot meet the real-time requirements. Therefore, it is not applicable to small and lightweight devices, and cannot perform tasks with high real-time requirements. Summary of the invention

[0004] In order to solve the problems in the prior art, the present invention proposes a method for fast image deocclusion based on SURF feature matching. The method can realize fast deocclusion of two-dimensional images.

[0005] The technical solution adopted by the present invention is:

[0006] A fast image deocclusion method based on SURF feature matching, comprising the following steps:

[0007] Step 1: Establish an image library, which includes complete images of all objects that may be blocked;

[0008] Step 2, for the image I containing the occluded object, extract the SURF features in the image I;

[0009] Step 3, matching the SURF features of image I with the SURF features of each image in the image library to obtain a matching image of image I;

[0010] Step 4: transform image I and its matching image into the same coordinate system to complete the registration between the two images;

[0011] Step 5: fuse the two images according to the image registration result, and output the image I after removing the occlusion.

[0012] Furthermore, the specific method of step 2 is:

[0013] Step 201, constructing the discriminant of the Hessian matrix;

[0014] For image I(x,y), its Hessian matrix is:

[0015]

[0016] x, y represent the pixel points in the image;

[0017] The discriminant of the Hessian matrix is:

[0018]

[0019] Gaussian filtering is performed on the image, and the Hessian matrix after filtering is:

[0020]

[0021] Where (x, y) is the pixel position, σ is the scale factor, and L(x, y, σ) = G(σ)*I(x, y) represents the Gaussian scale space of the image;

[0022] The box filter D is used to replace the Gaussian filter L. xy Multiplying by a weighting factor of 0.9 to balance the error caused by the box filter approximation, the discriminant of the Hessian matrix is:

[0023] det(H)=D xx D yy -(0.9×D xy ) 2 ;

[0024] Step 202, constructing a Gaussian pyramid scale space;

[0025] When constructing the pyramid, the image size is guaranteed to be unchanged. The sizes of images in different groups are consistent. The template sizes of the box filters used in different groups gradually increase. The same size of filters is used for different layers of images in the same group, but the scale space factor of the filters gradually increases.

[0026] Step 203, preliminarily determine the position of the feature point by using non-maximum suppression;

[0027] Compare the approximate response value of each pixel processed by the Hessian matrix with the 26 points in its three-dimensional neighborhood. If the approximate response value of the pixel is the maximum or minimum value among the 26 points, it is retained as a preliminary feature point.

[0028] Step 204, accurately locate the extreme point;

[0029] The three-dimensional linear interpolation method is used to obtain sub-pixel feature points, and the points with approximate response values ​​less than the threshold are removed;

[0030] Step 205, using Harr wavelet transform to determine the main direction of the feature point;

[0031] The Harr wavelet features in the neighborhood of the feature point are counted, and a circular area with the feature point as the center and a radius of 6σ is selected as the neighborhood. The circular area is further divided, and all response values ​​in the fan-shaped area with the feature point as the center and an opening angle of 60° are considered. The fan-shaped area is rotated with an angle of 0.2, and the main direction is determined by counting the Harr wavelet response values ​​in the fan-shaped area;

[0032] Step 206, constructing a SURF feature point descriptor;

[0033] 4×4 rectangular area blocks are extracted around the feature points, and the direction of the obtained rectangular area blocks is along the main direction of the feature points; each sub-area in the rectangular area block counts the Haar wavelet features of 25 pixels in the horizontal and vertical directions, and the Haar wavelet features are the sum of the horizontal direction values, the sum of the vertical direction values, the sum of the absolute values ​​of the horizontal direction values, and the sum of the absolute values ​​of the vertical direction. Here, the horizontal and vertical directions are relative to the main direction; the values ​​of the four directions are used as the feature vectors of each sub-area, and a 4×4×4=64-dimensional vector is obtained as the descriptor of the SURF feature.

[0034] Furthermore, in step 3, the matching degree is determined by calculating the Euclidean distance of the feature vectors between the two feature points. The shorter the Euclidean distance, the better the matching degree of the two feature points. In addition, the judgment of the Hessian matrix trace is added. If the matrix traces of the two feature points have the same positive and negative signs, it means that the two feature points have contrast changes in the same direction. If they are different, it means that the contrast directions of the two feature points are opposite. Even if the Euclidean distance is 0, they are directly eliminated.

[0035] Furthermore, in step 4, the two images are transformed into the same coordinate system through the transformation matrix T to complete the registration between the two images. The conversion method is as follows:

[0036]

[0037] The transformation matrix T is calculated by SURF matching point pairs.

[0038] The beneficial effects of the present invention are:

[0039] 1. Based on the extraction and matching of SURF (Speeded up Robust Features) feature points, the present invention realizes a method for fast deocclusion of two-dimensional images. Through SURF feature matching, image registration and image fusion, the calculation time of image deocclusion is reduced, providing an accurate and efficient image deocclusion method for computer vision applications.

[0040] 2. The present invention can solve the problem of quickly restoring the blocked part of a two-dimensional image in indoor and outdoor environments, and has good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Flow chart of the method of the present invention.

[0042] Figure 2 The image comparison is before and after removing the occlusion, where Figure a) is the image before removing the occlusion, and Figure b) is the image after removing the occlusion. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] A fast image deocclusion method based on SURF feature matching. The method first establishes an image library including all complete images of objects that may be occluded, and then performs the following steps: Figure 1 The following steps are shown:

[0045] Step 1: Construct the Hessian matrix;

[0046] The SURF algorithm uses the Hessian matrix determinant of the pixel point to approximate the original pixel value of the image, thereby generating a feature response image of the circular image for the Hessian matrix, and then further locates the feature points in the feature response image. For an image I(x,y), its Hessian matrix is ​​shown as follows:

[0047]

[0048] The determinant of the Hessian matrix is ​​shown below:

[0049]

[0050] Before constructing the Hessian matrix, the image needs to be Gaussian filtered. The formula is the expression of the Hessian matrix after filtering.

[0051]

[0052] Among them, (x, y) is the pixel position, σ is the scale factor, L(x, y, σ) = G(σ)*I(x, y) represents the Gaussian scale space of the image, which is obtained by convolving the image with different Gaussians.

[0053] L(x, y, σ) in the Hessian matrix discriminant is the Gaussian convolution of the original image. Since the Gaussian kernel obeys the normal distribution, the coefficients become smaller and smaller from the center point to the outside. In order to improve the operation speed, the SURF algorithm uses a box filter D to replace the Gaussian filter L, so in L xy A weighting factor of 0.9 is added to balance the error caused by the box filter approximation. The H matrix discriminant can be expressed as:

[0054] det(H)=D xx D yy -(0.9×D xy ) 2

[0055] Among them, 0.9 is the weight coefficient, which is used to balance the error caused by approximation.

[0056] Step 2: Construct Gaussian pyramid scale space;

[0057] The SURF algorithm ensures that the image size remains unchanged when constructing the pyramid, and processes the image indirectly by changing the scale of the Gaussian filter. The size of the images in different groups is consistent, but the template size of the box filter used in different groups gradually increases. The same size of filters is used for different layers of the same group of images, but the scale space factor of the filter gradually increases.

[0058] Step 3: Use non-maximum suppression to preliminarily determine the location of feature points;

[0059] The approximate response value of each pixel processed by the Hessian matrix is ​​compared with the 26 points in its three-dimensional neighborhood. If it is the maximum or minimum value among the 26 points, it is retained as a preliminary feature point.

[0060] Step 4: Accurately locate the extreme points;

[0061] Three-dimensional linear interpolation is used to obtain sub-pixel feature points, and points with approximate response values ​​less than a certain threshold are removed.

[0062] Step 5: Use Harr wavelet transform to determine the main direction of the feature points;

[0063] In SURF, the Harr wavelet features in the neighborhood of the feature point are counted. A circular area with the feature point as the center and a radius of 6σ is selected as the neighborhood. In order to obtain the main direction information, the circular area needs to be further divided, considering all the response values ​​in the fan-shaped area with the feature point as the center and an angle of 60°, and rotating the fan-shaped area with an angle of 0.2. The main direction is determined by counting the Harr wavelet response values ​​in the fan-shaped area.

[0064] Step 6: Construct SURF feature point description operator.

[0065] In the SURF algorithm, 4×4 rectangular blocks are extracted around the feature points, and the direction of the obtained rectangular blocks is along the main direction of the feature points. Each sub-region counts the Harr wavelet features of 25 pixels in the horizontal and vertical directions. The horizontal and vertical directions here are relative to the main direction. The Harr wavelet features are the sum of the horizontal direction values, the sum of the vertical direction values, the sum of the absolute values ​​of the horizontal direction values, and the absolute sum of the vertical direction values. These 4 values ​​are used as the feature vectors of each sub-block area, so there are a total of 4×4×4=64 dimensional vectors as descriptors of SURF features.

[0066] Step 7: SURF feature matching;

[0067] The matching degree is determined by calculating the Euclidean distance of the eigenvectors between two feature points. The shorter the Euclidean distance, the better the matching degree of the two feature points. SURF adds the judgment of the Hessian matrix trace (the sum of the matrix eigenvalues). If the matrix traces of two feature points have the same sign, it means that the two feature points have contrast changes in the same direction. If they are different, it means that the contrast directions of the two feature points are opposite. Even if the Euclidean distance is 0, they are directly eliminated.

[0068] Step 8: Image registration;

[0069] Image registration is to transform two images into the same coordinate system through the transformation matrix T, as shown in the following formula:

[0070]

[0071] The transformation matrix T can be calculated by matching the SURF point pairs. Then the two images can be registered and converted to the same image coordinate system.

[0072] Step 9: Image fusion;

[0073] The two images are fused according to the image registration results, and a two-dimensional image is output after removing the occlusion.

[0074] This method can effectively process images acquired in indoor and outdoor environments and complete the function of rapid image de-occlusion. Figure 2 The middle figure b) is the image after de-occlusion. It can be seen that this method can effectively remove occluded objects in the image.

[0075] The average operation time of this method is 0.27s. It can be seen that this method can effectively reduce the operation time of image de-occlusion, and provide an accurate and efficient image de-occlusion method for computer vision applications, achieving the expected effect.

[0076] In summary, the present invention is mainly aimed at the scene where part of the scenery of the two-dimensional image is blocked. It determines the position and main direction of the feature point by constructing the Hessian matrix and the Gaussian pyramid scale space, and then constructs the SURF feature point description operator, performs feature matching according to the feature point and the description operator, and finally performs image registration and image fusion according to the feature matching point pairs to complete image de-blocking. The present invention can reduce the time consumption of image de-blocking and has high real-time performance.

[0077] The above is a detailed introduction to the image fast de-occlusion method based on SURF feature matching proposed in the present invention. In this article, specific examples are used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A fast image deocclusion method based on SURF feature matching, characterized in that: The following steps are involved: Step 1: Establish an image library, which includes complete images of all objects that may be blocked; Step 2: For an image I containing an occluded object, extract the SURF features in the image I; the specific method is as follows: Step 201, constructing the discriminant of the Hessian matrix; For image I(x,y), its Hessian matrix is: x, y represent the pixel points in the image; The discriminant of the Hessian matrix is: Gaussian filtering is performed on the image, and the Hessian matrix after filtering is: Where (x, y) is the pixel position, σ is the scale factor, and L(x, y, σ) = G(σ)*I(x, y) represents the Gaussian scale space of the image; The box filter D is used to replace the Gaussian filter L. xy Multiplying by a weighting factor of 0.9 to balance the error caused by the box filter approximation, the discriminant of the Hessian matrix is: it(H)=D xx D yy -(0.9×D xy ) 2 ; Step 202, constructing a Gaussian pyramid scale space; When constructing the pyramid, the image size is guaranteed to be unchanged. The sizes of images in different groups are consistent. The template sizes of the box filters used in different groups gradually increase. The same size of filters is used for different layers of images in the same group, but the scale space factor of the filters gradually increases. Step 203, preliminarily determine the position of the feature point by using non-maximum suppression; Compare the approximate response value of each pixel processed by the Hessian matrix with the 26 points in its three-dimensional neighborhood. If the approximate response value of the pixel is the maximum or minimum value among the 26 points, it is retained as a preliminary feature point. Step 204, accurately locate the extreme point; The three-dimensional linear interpolation method is used to obtain sub-pixel feature points, and the points with approximate response values ​​less than the threshold are removed; Step 205, using Harr wavelet transform to determine the main direction of the feature point; The Harr wavelet features in the neighborhood of the feature point are counted, and a circular area with the feature point as the center and a radius of 6σ is selected as the neighborhood. The circular area is further divided, and all response values ​​in the fan-shaped area with the feature point as the center and an opening angle of 60° are considered. The fan-shaped area is rotated with an angle of 0.2, and the main direction is determined by counting the Harr wavelet response values ​​in the fan-shaped area; Step 206, constructing a SURF feature point descriptor; Extract 4×4 rectangular blocks around the feature point. The direction of the obtained rectangular blocks is along the main direction of the feature point. In each sub-region of the rectangular block, count the Haar wavelet features of 25 pixels in the horizontal and vertical directions. The Haar wavelet features are the sum of the horizontal direction value, the sum of the vertical direction value, the sum of the absolute value of the horizontal direction value, and the sum of the absolute value of the vertical direction. Here, the horizontal and vertical directions are relative to the main direction. Take the values ​​of the four directions as the feature vector of each sub-region, and get a 4×4×4=64-dimensional vector as the descriptor of the SURF feature. Step 3, matching the SURF features of image I with the SURF features of each image in the image library to obtain a matching image of image I; Step 4: transform image I and its matching image into the same coordinate system to complete the registration between the two images; Step 5: fuse the two images according to the image registration result, and output the image I after removing the occlusion.

2. The method for fast image deocclusion based on SURF feature matching according to claim 1, characterized in that: In step 3, the matching degree is determined by calculating the Euclidean distance of the feature vectors between the two feature points. The shorter the Euclidean distance, the better the matching degree of the two feature points. In addition, the judgment of the Hessian matrix trace is added. If the matrix traces of the two feature points have the same sign, it means that the two feature points have contrast changes in the same direction. If they are different, it means that the contrast directions of the two feature points are opposite. Even if the Euclidean distance is 0, they are directly eliminated.

3. The method for fast image deocclusion based on SURF feature matching according to claim 1, characterized in that: In step 4, the two images are transformed into the same coordinate system through the transformation matrix T to complete the registration between the two images. The conversion method is as follows: The transformation matrix T is calculated by SURF matching point pairs.

Images