Method for identifying and positioning micro-operation object under deflection and shielding

Abstract

The invention discloses a method for identifying and positioning a micro-operation object under deflection and shielding, and belongs to the technical field of micro-operation object identification and positioning under microscopic vision. The problem that when the micro-operation object is deflected and shielded, an existing method is low in positioning precision of the micro-operation object is solved. The method specifically comprises the steps of performing graying and denoising processing on an original image where a micro-operation object is located to obtain a denoised image; performing wavelet decomposition on the denoised image to obtain a down-sampling image; respectively extracting feature points in the downsampling image and the microoperation object template image, and describing the feature points; feature points in the downsampling image and the microoperation object template image are matched to obtain rough matching feature point pairs, the feature point pairs obtained through rough matching are screened, and feature point pairs obtained through fine matching are obtained; and obtaining an operation point coordinate of the micro-operation object in the original image according to a fine matching result. The method can be applied to identification and positioning of the micro-operation object.

Description

Technical Field

[0001] The invention belongs to the technical field of micro-operation object recognition and positioning under microscopic vision, and specifically relates to a method for recognizing and positioning a micro-operation object under deflection and occlusion. Background Art

[0002] As micro-operation tasks become more complex, the requirements for micro-operation precision and accuracy are getting higher and higher. Simple manual operation can no longer meet the needs of the operation process. With the development of target detection and positioning technology based on image processing and microscopic visual control theory, the level of automation of micro-operation has been greatly improved. In the picking and releasing of micro-operation, the identification and positioning of the end of the probe and the operation object are the prerequisites for realizing micro-operation contact picking and releasing. The deflection and occlusion of the micro-operation object will have a significant impact on the identification and positioning of the micro-operation object, seriously reducing the recognition accuracy and positioning accuracy of the micro-operation object.

[0003] Target recognition and positioning based on microscopic vision are mainly achieved through different types of recognition technologies. The main methods are feature-based matching, grayscale-based template matching, and deep learning-based methods. The target recognition algorithm based on feature matching is realized through features such as lines, points, shapes, and textures in image sequences with known geometric information. It has many studies and applications in target recognition and positioning in microscopic vision. For the matching method based on feature points, the efficiency of feature point extraction and the accuracy of matching point screening directly affect the matching results. Therefore, domestic and foreign researchers are making improvements in improving efficiency and accuracy. When the micro-operation object to be identified is occluded or the background interference is large, feature-based matching often cannot complete accurate recognition and positioning. The template matching method based on grayscale directly operates on pixel values, does not require any form of specific target mode, and has strong robustness to image noise due to the involvement of highly redundant information. Therefore, it is widely used in microscopic vision target recognition and positioning. However, the high computational cost of the template matching method based on grayscale limits its application in microscopic target recognition and positioning. Therefore, many scholars have made improvements on the efficiency of template matching. Deep learning-based methods are widely used in the recognition of micro-manipulation objects. The application of deep learning algorithms to image target recognition requires a large data set, while data sets for specific micro-manipulation objects are difficult to construct. In addition, deep learning algorithms are generally used for qualitative recognition and classification within the microscale range, and have low accuracy in specific positioning.

[0004] In summary, although domestic and foreign scholars have conducted a lot of research on the recognition and positioning of micro-operation objects, the existing microscopic visual target positioning and recognition algorithms still have the problem of low micro-operation object positioning accuracy when the micro-operation object is deflected and occluded. Therefore, studying a high-precision and high-robust micro-operation object recognition and positioning algorithm is of great significance to realize the automation of micro-operation. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of low positioning accuracy of micro-operation objects by existing methods when the micro-operation objects are deflected and blocked, and to propose a method for identifying and positioning micro-operation objects under deflection and blockage.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a method for identifying and locating a micro-operation object under deflection and occlusion, the method specifically comprising the following steps:

[0007] Step 1: grayscale the original image where the micro-operation object is located to obtain a grayscale image, and then denoise the grayscale image to obtain a denoised image;

[0008] Step 2, initialize the wavelet decomposition times n=1, perform the first wavelet decomposition on the denoised image, and obtain the downsampled image after the first wavelet decomposition;

[0009] Step 3, let n=n+1, perform the nth wavelet decomposition on the downsampled image after the n-1th wavelet decomposition, and obtain the downsampled image after the nth wavelet decomposition;

[0010] Step 4: determine whether n=N is satisfied, where N is the maximum number of wavelet decompositions;

[0011] If n=N, then step 5 is performed on the downsampled image after the Nth wavelet decomposition;

[0012] If n=N is not satisfied, return to step 3;

[0013] Step 5: Using the BRISK algorithm to extract the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image respectively, and then using the SURF algorithm to describe the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image;

[0014] Step 6: According to the feature point description result in step 5, a brute force matching algorithm is used to match the feature points in the downsampled image after the Nth wavelet decomposition with the feature points in the micro-operation object template image to obtain a rough matching feature point pair;

[0015] Then, the feature point pairs obtained by the rough matching are screened, and the feature point pairs that are incorrectly matched are eliminated from the feature point pairs obtained by the rough matching to obtain the feature point pairs obtained by the precise matching;

[0016] Step 7: Based on the precisely matched feature point pairs, the homography matrix of the micro-operation object template image and the original image is constructed. The coordinates of the four vertices of the template image in the original image are obtained according to the coordinate information of the template image and the homography matrix. The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu coordinate.

[0017] The beneficial effects of the present invention are:

[0018] 1. Use the improved bilateral filtering algorithm to remove high-frequency noise in the image while retaining the high-frequency edge information in the image; construct an image pyramid through wavelet transform and add high-frequency information, which can retain and highlight the high-frequency information in the image while downsampling, and the extreme point features in the image are more obvious.

[0019] 2. The BRISK algorithm is used to extract feature points, and the SURF algorithm is used to describe the feature points. Compared with traditional algorithms, it achieves high-efficiency feature extraction while ensuring high matching accuracy.

[0020] 3. Compared with traditional algorithms, the feature point pair matching and screening algorithm of the present invention does not require iteration, which improves the efficiency of the algorithm. After screening the matching point pairs in the image, the mismatched point pairs are completely removed, ensuring the accuracy of subsequent positioning. Moreover, the screening method of the present invention improves the screening accuracy compared with the existing method.

[0021] Experimental results show that when the micro-operation object is occluded and deflected, the positioning error of the traditional method is greater than 5 pixels, and the time consumption is more than 400 ms, which is inefficient. The method of the present invention can achieve accurate positioning when the micro-operation object is deflected and occluded, with a positioning error of less than 1 pixel and an average time consumption of about 98 ms. Compared with the traditional method, the method of the present invention has obvious advantages in positioning accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a method for identifying and locating a micro-operation object under deflection and occlusion of the present invention;

[0023] Figure 2 It is a flow chart of the matching point screening algorithm based on size and structure similarity. DETAILED DESCRIPTION

[0024] Specific implementation method 1: Combination Figure 1This embodiment describes a method for identifying and locating a micro-operation object under deflection and occlusion, and the method is specifically as follows:

[0025] Step 1: grayscale the original image where the micro-operation object is located to obtain a grayscale image, and then denoise the grayscale image to obtain a denoised image;

[0026] Step 2, initialize the wavelet decomposition times n=1, perform the first wavelet decomposition on the denoised image, and obtain the downsampled image after the first wavelet decomposition;

[0027] Step 3, let n=n+1, perform the nth wavelet decomposition on the downsampled image after the n-1th wavelet decomposition, and obtain the downsampled image after the nth wavelet decomposition;

[0028] Step 4: determine whether n=N is satisfied, where N is the maximum number of wavelet decompositions (which can be set according to actual conditions, and is set to 3 in the present invention);

[0029] If n=N, then step 5 is performed on the downsampled image after the Nth wavelet decomposition;

[0030] If n=N is not satisfied, return to step 3;

[0031] Step 5: Using the BRISK algorithm to extract the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image respectively, and then using the SURF algorithm to describe the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image;

[0032] Step 6: According to the feature point description result in step 5, the Brute Force (BF) algorithm is used to match the feature points in the downsampled image after the Nth wavelet decomposition with the feature points in the micro-operation object template image to obtain a coarse matching feature point pair;

[0033] Then, the feature point pairs obtained by the rough matching are screened, and the feature point pairs that are incorrectly matched are eliminated from the feature point pairs obtained by the rough matching to obtain the feature point pairs obtained by the precise matching;

[0034] Step 7: Based on the precisely matched feature point pairs, the homography matrix of the micro-operation object template image and the original image is constructed. The coordinates of the four vertices of the template image in the original image are obtained according to the coordinate information of the template image and the homography matrix. The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu Coordinates (i.e., the coordinates of the geometric center of the operation object in the template image in the original image).

[0035] Since the metal micro-components of the operating objects in the image will have an angle offset relative to the template image, the feature point matching algorithm can use the feature information of the image feature points for matching. By obtaining the positional relationship between the feature points corresponding to the template image and the downsampled image after the Nth wavelet decomposition, the angle offset and relative position of the operating target in the original image and the target on the template image can be determined, thereby completing the positioning of the target object. Therefore, the template matching algorithm based on feature points is selected to identify the operating object and locate the operating point.

[0036] Common algorithms for feature point extraction and description include Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Binary Robust Invariant Scalable Keypoints (BRISK). Compared with SIFT, SURF optimizes the description algorithm, ensures matching accuracy with fewer description features, and improves matching efficiency. The BRISK algorithm simplifies the feature extraction method and feature description steps, and is more efficient than the first two. However, due to the simplification of the description algorithm, the similarity of the description features is higher, resulting in poor matching accuracy.

[0037] In order to achieve high-efficiency and high-precision feature point matching, the present invention proposes a feature point matching algorithm based on the combination of BRISK and SURF, using the BRISK algorithm to extract feature points and the SURF algorithm to describe feature points. By matching feature points of 50 micro-operation objects and comparing them with other algorithms, the comparison results show that the algorithm proposed in the present invention achieves high-efficiency feature extraction while ensuring high matching accuracy, which has obvious advantages over the original algorithm.

[0038] Specific implementation method 2: This implementation method is different from the specific implementation method 1 in that the grayscale processing of the original image where the micro-operation object is located adopts a weighted average grayscale algorithm.

[0039] The other steps and parameters are the same as those in the first embodiment.

[0040] The implementation process of the weighted average grayscale algorithm is as follows:

[0041] gray=α1R+α2G+α3B

[0042] Among them, R, G, B are the values ​​of the pixels in the original image in the red, green, and blue color channels, α1, α2, and α3 are weighting coefficients, and gray is the calculated pixel grayscale value;

[0043] Substitute the R, G, and B values ​​of each pixel in the original image into the above formula to obtain the grayscale value corresponding to each pixel;

[0044] Specific implementation method three: This implementation method is different from specific implementation method one or two in that the grayscale image denoising process adopts an improved bilateral filtering algorithm, which is specifically:

[0045]

[0046] Among them, f(x,y) is the gray value of the pixel (x,y) in the gray image, f Bilateral (x0, y0) is the grayscale value of the pixel (x0, y0) in the denoised image, M(x0, y0) is the set of pixels in the convolution kernel (the convolution kernel size used in the present invention is 3×3) centered at the pixel (x0, y0) in the grayscale image, and ω(x, y) is the bilateral filtering weight function of the pixel (x, y);

[0047] ω(x,y)=ω d (x,y)·ω s (x,y)

[0048] Among them, ω d (x,y) is the spatial distance domain weighting coefficient of the pixel point (x,y), ω s (x,y) is the grayscale domain weighting coefficient of the pixel (x,y);

[0049]

[0050] Among them, σ d is the spatial domain standard deviation (for each pixel in the set M(x0,y0), it is calculated Then calculate the corresponding The calculated standard deviation is used as the spatial domain standard deviation), σ s is the standard deviation in the grayscale domain (the grayscale value of each pixel in the set M(x0,y0) is subtracted from the grayscale value of the pixel (x,y), and then the standard deviation of all the differences is calculated), f(x0,y0) is the grayscale value of the pixel (x0,y0) in the grayscale image, |·| indicates the calculation of the absolute value, and k is an intermediate variable.

[0051] The other steps and parameters are the same as those in the first or second embodiment.

[0052] The present invention adopts an improved bilateral filtering algorithm to remove high-frequency noise in an image while retaining high-frequency edge information in the image.

[0053] Specific implementation method 4: This implementation method is different from one of the specific implementation methods 1 to 3 in that the calculation method of the intermediate variable k is:

[0054] Establish the Robert operator ▽f in the x direction x and the Robert operator ▽f in the y direction y They are:

[0055]

[0056] Use ▽f x and ▽f y Construct a gradient image of the grayscale image, then calculate the average grayscale value of all pixels in the gradient image, and use the calculated average grayscale value as k.

[0057] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0058] Specific implementation method 5: This implementation method is different from any one of specific implementation methods 1 to 4 in that the denoised image is subjected to the first wavelet decomposition to obtain a downsampled image after the first wavelet decomposition; specifically:

[0059] The Haar wavelet decomposition convolution kernel H is expressed as:

[0060]

[0061] Perform Haar wavelet decomposition on the denoised image based on H:

[0062]

[0063] Among them, H T represents the transpose of H, G represents the image after denoising, G w represents the image after Haar wavelet decomposition, G 11 Represents G w The low-frequency sub-image in G (the low-frequency sub-image contains the basic information in image G), G 12 , G 21 and G 22 Represents G w The high-frequency sub-images in G (all three sub-images contain a certain degree of high-frequency information); and G 11 , G 12 , G 21 , G 22 The size of is one quarter of G;

[0064] The three high-frequency sub-images G 12 , G 21 and G 22An addition operation is performed, and the grayscale of each pixel in the image obtained by the addition operation is normalized, and the image obtained by the normalization operation is used as the down-sampled image after the first wavelet decomposition.

[0065] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0066] In this embodiment, the wavelet transform can retain and highlight the high-frequency information in the image while downsampling, making the extreme point features in the image more obvious. The problem of low processing efficiency caused by a large number of pixels in the original image can be solved by downsampling.

[0067] Specific implementation method six: Combination Figure 2 This embodiment is different from any one of the specific embodiments 1 to 5 in that the feature point pairs obtained by rough matching are screened, and the feature point pairs obtained by rough matching are eliminated from the feature point pairs obtained by rough matching to obtain the feature point pairs obtained by precise matching; specifically:

[0068] Step 61: record the set of feature point pairs obtained by rough matching as set A, select three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3) at random from set A, and remove the selected feature point pairs from set A;

[0069] Among them, P1, P2 and P3 are the feature points in the downsampled image after the Nth wavelet decomposition, and p1, p2 and p3 are the feature points in the template image of the micro-operation object;

[0070] Step 62: The distance between feature point P1 and feature point P2 is recorded as D1, the distance between feature point P1 and feature point P3 is recorded as D2, the distance between feature point P2 and feature point P3 is recorded as D3, the distance between feature point p1 and feature point p2 is recorded as d1, the distance between feature point p1 and feature point p3 is recorded as d2, and the distance between feature point P2 and feature point P3 is recorded as d3;

[0071] Then calculate the intermediate variables s1, s2 and s3:

[0072]

[0073] Step 6.3. Calculate the variance of s1, s2 and s3;

[0074] If the calculated variance is less than the set distance variance threshold t1, continue to execute step 64;

[0075] If the calculated variance is greater than or equal to the set threshold t1, then a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A; the selected feature point pair is used to replace any one of the three pairs of feature point pairs corresponding to the variance, to obtain a group of feature point pairs consisting of three pairs of feature point pairs, and then the obtained group of feature point pairs is returned to execute step 62;

[0076] Step six four, calculate the angle α1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle α2 between the line connecting the feature point P1 and the feature point P2 and the feature point P3, calculate the angle α3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3, calculate the angle β1 between the line connecting the feature point p1 and the feature point p2 and the line connecting the feature point p1 and the feature point p3, calculate the angle β2 between the line connecting the feature point p1 and the feature point p2 and the line connecting the feature point p2 and the feature point p3, calculate the angle β3 between the line connecting the feature point p1 and the feature point p3 and the line connecting the feature point p2 and the feature point p3;

[0077] The angle between vectors is calculated as follows:

[0078]

[0079] Calculate the angle deviations |α1-β1|, |α2-β2| and |α3-β3|;

[0080] If |α1-β1|, |α2-β2| and |α3-β3| are all less than the set angle variance threshold t2, then the three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3) are all correctly matched feature point pairs, and step six-five is performed for the three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3);

[0081] Otherwise, a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A, and any one of the three feature point pairs (referring to the three feature point pairs currently involved in the operation) is replaced with the selected feature point pair to obtain a set of feature point pairs consisting of three feature point pairs, and then the obtained set of feature point pairs is returned to execute step 62;

[0082] Step 65: Select any two pairs from the three pairs of feature point pairs as base points, and use the remaining feature point pairs in the selected base point pair set A to perform verification in turn. The verification method is:

[0083] For any remaining feature point pair in set A, use the feature point pair and the base point to form a set of feature point pairs. If the set of feature point pairs satisfies both the distance variance threshold condition and the angle variance threshold condition (i.e., the variance calculated based on the set of feature point pairs is less than t1, and the three angle deviations corresponding to the set of feature point pairs are all less than t2), then the feature point pair is a correctly matched feature point pair, otherwise, the feature point pair is a mismatched feature point pair;

[0084] After verifying the remaining feature point pairs in set A respectively, all correctly matched feature point pairs are obtained, that is, the precisely matched feature point pairs are obtained.

[0085] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0086] Specific implementation method seven: This implementation method is different from any one of specific implementation methods one to six in that the value of the distance variance threshold t1 is 0.04.

[0087] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0088] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the value of the angle variance threshold t2 is 0.06.

[0089] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0090] By setting appropriate distance variance threshold t1 and angle variance threshold t2, the correct matching point pairs can be retained while all wrong matching point pairs can be filtered out, and the screening effect is the best. Moreover, the matching point pair screening algorithm of the present invention does not require iteration compared to the traditional algorithm, which improves the efficiency of the algorithm. After screening the matching point pairs in the image, the wrong matching point pairs are completely removed, which improves the accuracy of the screening.

[0091] Specific implementation method 9: This implementation method is different from any one of the specific implementation methods 1 to 8 in that the operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu Coordinates, specifically:

[0092]

[0093] Among them, P cen-Cu _x is the operating point P cen-Cu The horizontal axis, P cen-Cu _y is the operating point P cen-Cu, P′1_x, P′2_x, P′3_x and P′4_x are the corresponding horizontal coordinates of the four vertices of the template image in the original image, and P′1_y, P′2_y, P′3_y and P′4_y are the corresponding vertical coordinates of the four vertices of the template image in the original image.

[0094] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0095] Example

[0096] This embodiment proposes a method for identifying and locating a micro-operation object under deflection and occlusion, and the method is specifically as follows:

[0097] Step 1: grayscale the original image where the micro-operation object is located using a weighted average grayscale algorithm to obtain a grayscale image, and then use an improved bilateral filtering algorithm to denoise the grayscale image to obtain a denoised image;

[0098] The improved bilateral filtering algorithm is specifically as follows:

[0099]

[0100] Among them, f(x,y) is the gray value of the pixel (x,y) in the gray image, f Bilateral (x0, y0) is the grayscale value of the pixel (x0, y0) in the denoised image, M(x0, y0) is the set of pixels in the convolution kernel (the convolution kernel size used in the present invention is 3×3) centered at the pixel (x0, y0) in the grayscale image, and ω(x, y) is the bilateral filtering weight function of the pixel (x, y);

[0101] ω(x,y)=ω d (x,y)·ω s (x,y)

[0102] Among them, ω d (x,y) is the spatial distance domain weighting coefficient of the pixel point (x,y), ω s (x,y) is the grayscale domain weighting coefficient of the pixel (x,y);

[0103]

[0104] Among them, σ d is the spatial domain standard deviation (for each pixel in the set M(x0,y0), it is calculated Then calculate the corresponding The calculated standard deviation is used as the spatial domain standard deviation), σ sis the standard deviation in the grayscale domain (the grayscale value of each pixel in the set M(x0,y0) is subtracted from the grayscale value of the pixel (x,y), and then the standard deviation of all the differences is calculated), f(x0,y0) is the grayscale value of the pixel (x0,y0) in the grayscale image, |·| indicates the calculation of the absolute value, and k is an intermediate variable.

[0105] The calculation method of the intermediate variable k is:

[0106] Establish the Robert operator ▽f in the x direction x and the Robert operator ▽f in the y direction y They are:

[0107]

[0108] Use ▽f x and ▽f y Construct a gradient image of the grayscale image, then calculate the average grayscale value of all pixels in the gradient image, and use the calculated average grayscale value as k.

[0109] Step 2: Initialize the wavelet decomposition times n=1, perform the first wavelet decomposition on the denoised image, and obtain the downsampled image after the first wavelet decomposition; specifically:

[0110] The Haar wavelet decomposition convolution kernel H is expressed as:

[0111]

[0112] Perform Haar wavelet decomposition on the denoised image based on H:

[0113]

[0114] Among them, H T represents the transpose of H, G represents the image after denoising, G w represents the image after Haar wavelet decomposition, G 11 Represents G w The low-frequency sub-image in G (the low-frequency sub-image contains the basic information in image G), G 12 , G 21 and G 22 Represents G w The high-frequency sub-images in G (all three sub-images contain a certain degree of high-frequency information); and G 11 , G 12 , G 21 , G 22 The size of is one quarter of G;

[0115] The three high-frequency sub-images G 12 , G21 and G 22 An addition operation is performed, and the grayscale of each pixel in the image obtained by the addition operation is normalized, and the image obtained by the normalization operation is used as the down-sampled image after the first wavelet decomposition.

[0116] Step 3, let n=n+1, perform the nth wavelet decomposition on the downsampled image after the n-1th wavelet decomposition, and obtain the downsampled image after the nth wavelet decomposition;

[0117] Step 4: determine whether n=N is satisfied, where N is the maximum number of wavelet decompositions (which can be set according to actual conditions, and is set to 3 in the present invention);

[0118] If n=N, then step 5 is performed on the downsampled image after the Nth wavelet decomposition;

[0119] If n=N is not satisfied, return to step 3;

[0120] Step 5: Using the BRISK algorithm to extract the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image respectively, and then using the SURF algorithm to describe the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image;

[0121] Step 6: According to the feature point description result in step 5, the Brute Force (BF) algorithm is used to match the feature points in the downsampled image after the Nth wavelet decomposition with the feature points in the micro-operation object template image to obtain a coarse matching feature point pair;

[0122] Then, the feature point pairs obtained by rough matching are screened, and the feature point pairs with incorrect matching are eliminated from the feature point pairs obtained by rough matching to obtain the feature point pairs obtained by precise matching; specifically:

[0123] Step 61: record the set of feature point pairs obtained by rough matching as set A, select three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3) at random from set A, and remove the selected feature point pairs from set A;

[0124] Among them, P1, P2 and P3 are the feature points in the downsampled image after the Nth wavelet decomposition, and p1, p2 and p3 are the feature points in the template image of the micro-operation object;

[0125] Step 62: The distance between feature point P1 and feature point P2 is recorded as D1, the distance between feature point P1 and feature point P3 is recorded as D2, the distance between feature point P2 and feature point P3 is recorded as D3, the distance between feature point p1 and feature point p2 is recorded as d1, the distance between feature point p1 and feature point p3 is recorded as d2, and the distance between feature point P2 and feature point P3 is recorded as d3;

[0126] Then calculate the intermediate variables s1, s2 and s3:

[0127]

[0128] Step 6.3. Calculate the variance of s1, s2 and s3;

[0129] If the calculated variance is less than the set distance variance threshold t1 (in the present invention, the value of t1 is 0.04), continue to execute step 64;

[0130] If the calculated variance is greater than or equal to the set threshold t1, then a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A; the selected feature point pair is used to replace any one of the three pairs of feature point pairs corresponding to the variance, to obtain a group of feature point pairs consisting of three pairs of feature point pairs, and then the obtained group of feature point pairs is returned to execute step 62;

[0131] Step six four, calculate the angle α1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle α2 between the line connecting the feature point P1 and the feature point P2 and the feature point P3, calculate the angle α3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3, calculate the angle β1 between the line connecting the feature point p1 and the feature point p2 and the line connecting the feature point p1 and the feature point p3, calculate the angle β2 between the line connecting the feature point p1 and the feature point p2 and the line connecting the feature point p2 and the feature point p3, calculate the angle β3 between the line connecting the feature point p1 and the feature point p3 and the line connecting the feature point p2 and the feature point p3;

[0132] The angle between vectors is calculated as follows:

[0133]

[0134] Calculate the angle deviations |α1-β1|, |α2-β2| and |α3-β3|;

[0135] If |α1-β1|, |α2-β2| and |α3-β3| are all less than the set angle variance threshold t2 (in the present invention, the value of t2 is 0.06), then the three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3) are all correctly matched feature point pairs, and step six-five is performed on the three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3);

[0136] Otherwise, a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A, and any one of the three feature point pairs (referring to the three feature point pairs currently involved in the operation) is replaced with the selected feature point pair to obtain a set of feature point pairs consisting of three feature point pairs, and then the obtained set of feature point pairs is returned to execute step 62;

[0137] Step 65: Select any two pairs from the three pairs of feature point pairs as base points, and use the remaining feature point pairs in the selected base point pair set A to perform verification in turn. The verification method is:

[0138] For any remaining feature point pair in set A, use the feature point pair and the base point to form a set of feature point pairs. If the set of feature point pairs satisfies both the distance variance threshold condition and the angle variance threshold condition (i.e., the variance calculated based on the set of feature point pairs is less than t1, and the three angle deviations corresponding to the set of feature point pairs are all less than t2), then the feature point pair is a correctly matched feature point pair, otherwise, the feature point pair is a mismatched feature point pair;

[0139] After verifying the remaining feature point pairs in set A respectively, all correctly matched feature point pairs are obtained, that is, the precisely matched feature point pairs are obtained.

[0140] Step 7: Based on the precisely matched feature point pairs, the homography matrix of the micro-operation object template image and the original image is constructed. The coordinates of the four vertices of the template image in the original image are obtained according to the coordinate information of the template image and the homography matrix. The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu Coordinates (i.e. the coordinates of the geometric center of the operation object in the template image in the original image):

[0141]

[0142] Among them, P cen-Cu _x is the operating point P cen-Cu The horizontal axis, P cen-Cu _y is the operating point P cen-Cu’s vertical coordinate, P1′_x, P2′_x, P3′_x and P4′_x are the horizontal coordinates of the four vertices of the template image in the original image, and P1′_y, P2′_y, P3′_y and P4′_y are the vertical coordinates of the four vertices of the template image in the original image.

[0143] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for identifying and locating a micro-operation object under deflection and occlusion, characterized in that: The method specifically comprises the following steps: Step 1: grayscale the original image where the micro-operation object is located to obtain a grayscale image, and then denoise the grayscale image to obtain a denoised image; Step 2, initialize the wavelet decomposition times n=1, perform the first wavelet decomposition on the denoised image, and obtain the downsampled image after the first wavelet decomposition; Step 3, let n=n+1, perform the nth wavelet decomposition on the downsampled image after the n-1th wavelet decomposition, and obtain the downsampled image after the nth wavelet decomposition; Step 4: determine whether n=N is satisfied, where N is the maximum number of wavelet decompositions; If n=N, then step 5 is performed on the downsampled image after the Nth wavelet decomposition; If n=N is not satisfied, return to step 3; Step 5: Using the BRISK algorithm to extract the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image respectively, and then using the SURF algorithm to describe the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image; Step 6: According to the feature point description result in step 5, a brute force matching algorithm is used to match the feature points in the downsampled image after the Nth wavelet decomposition with the feature points in the micro-operation object template image to obtain a rough matching feature point pair; Then, the feature point pairs obtained by the rough matching are screened, and the feature point pairs that are incorrectly matched are eliminated from the feature point pairs obtained by the rough matching to obtain the feature point pairs obtained by the precise matching; Step 7: Based on the precisely matched feature point pairs, the homography matrix of the micro-operation object template image and the original image is constructed. The coordinates of the four vertices of the template image in the original image are obtained according to the coordinate information of the template image and the homography matrix. The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu coordinate.

2. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 1, characterized in that: The grayscale processing of the original image where the micro-operation object is located adopts a weighted average grayscale algorithm.

3. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 2, characterized in that: The grayscale image denoising process adopts an improved bilateral filtering algorithm, which is specifically: Among them, f(x,y) is the gray value of the pixel (x,y) in the gray image, f Bilateral (x0, y0) is the gray value of the pixel (x0, y0) in the denoised image, M(x0, y0) is the set of pixels in the convolution kernel centered on the pixel (x0, y0) in the gray image, and ω(x, y) is the bilateral filtering weight function of the pixel (x, y); ω(x,y)=ω d (x,y)·ω s (x,y) Among them, ω d (x,y) is the spatial distance domain weighting coefficient of the pixel point (x,y), ω s (x,y) is the grayscale domain weighting coefficient of the pixel (x,y); Among them, σ d is the spatial domain standard deviation, σ s is the standard deviation in the grayscale domain, f(x0,y0) is the grayscale value of the pixel (x0,y0) in the grayscale image, |·| represents the calculation of the absolute value, and k is an intermediate variable.

4. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 3, characterized in that: The calculation method of the intermediate variable k is: Establish the Robert operator in the x direction and the Robert operator in the y direction They are: use and Construct a gradient image of the grayscale image, then calculate the average grayscale value of all pixels in the gradient image, and use the calculated average grayscale value as k.

5. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 4, characterized in that: The first wavelet decomposition is performed on the denoised image to obtain a downsampled image after the first wavelet decomposition; specifically: The Haar wavelet decomposition convolution kernel H is expressed as: Perform Haar wavelet decomposition on the denoised image based on H: Among them, H T represents the transpose of H, G represents the image after denoising, G w represents the image after Haar wavelet decomposition, G 11 Represents G w The low-frequency sub-image in G 12 , G 21 and G 22 Represents G w High-frequency sub-images in ; The three high-frequency sub-images G 12 , G 21 and G 22 An addition operation is performed, and the grayscale of each pixel in the image obtained by the addition operation is normalized, and the image obtained by the normalization operation is used as the down-sampled image after the first wavelet decomposition.

6. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 5, characterized in that: The feature point pairs obtained by rough matching are screened, and the feature point pairs obtained by rough matching are eliminated from the feature point pairs obtained by rough matching to obtain the feature point pairs obtained by precise matching; specifically: Step 61: record the set of feature point pairs obtained by rough matching as set A, select three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3) at random from set A, and remove the selected feature point pairs from set A; Among them, P1, P2 and P3 are the feature points in the downsampled image after the Nth wavelet decomposition, and p1, p2 and p3 are the feature points in the template image of the micro-operation object; Step 62: The distance between feature point P1 and feature point P2 is recorded as D1, the distance between feature point P1 and feature point P3 is recorded as D2, the distance between feature point P2 and feature point P3 is recorded as D3, the distance between feature point p1 and feature point p2 is recorded as d1, the distance between feature point p1 and feature point p3 is recorded as d2, and the distance between feature point P2 and feature point P3 is recorded as d3; Then calculate the intermediate variables s1, s2 and s3: Step 6.

3. Calculate the variance of s1, s2 and s3; If the calculated variance is less than the set distance variance threshold t1, continue to execute step 64; If the calculated variance is greater than or equal to the set threshold t1, then a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A; the selected feature point pair is used to replace any one of the three pairs of feature point pairs corresponding to the variance, to obtain a group of feature point pairs consisting of three pairs of feature point pairs, and then the obtained group of feature point pairs is returned to execute step 62; Step six four, calculate the angle α1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle α2 between the line connecting the feature point P1 and the feature point P2 and the feature point P3, calculate the angle α3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3, calculate the angle β1 between the line connecting the feature point p1 and the feature point p2 and the line connecting the feature point p1 and the feature point p3, calculate the angle β2 between the line connecting the feature point p1 and the feature point p2 and the line connecting the feature point p2 and the feature point p3, calculate the angle β3 between the line connecting the feature point p1 and the feature point p3 and the line connecting the feature point p2 and the feature point p3; Calculate the angle deviations |α1-β1|, |α2-β2| and |α3-β3|; If |α1-β1|, |α2-β2| and |α3-β3| are all less than the set angle variance threshold t2, then the three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3) are all correctly matched feature point pairs, and step six-five is performed for the three pairs of feature point pairs (P1, p1), (P2, p2), (P3, p3); Otherwise, a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A, and any one of the three feature point pairs is replaced with the selected feature point pair to obtain a set of feature point pairs consisting of three feature point pairs, and then the obtained set of feature point pairs is returned to execute step 62; Step 65: Select any two pairs from the three pairs of feature point pairs as base points, and use the remaining feature point pairs in the selected base point pair set A to perform verification in turn. The verification method is: For any remaining feature point pair in set A, use the feature point pair and the base point to form a set of feature point pairs. If the set of feature point pairs satisfies both the distance variance threshold condition and the angle variance threshold condition, then the feature point pair is a correctly matched feature point pair. Otherwise, the feature point pair is a mismatched feature point pair. After verifying the remaining feature point pairs in set A respectively, all correctly matched feature point pairs are obtained, that is, the precisely matched feature point pairs are obtained.

7. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 6, characterized in that: The value of the distance variance threshold t1 is 0.

04.

8. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 7, characterized in that: The value of the angle variance threshold t2 is 0.

06.

9. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 8, characterized in that: The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu Coordinates, specifically: Among them, P cen-Cu _x is the operating point P cen-Cu The horizontal axis, P cen-Cu _y is the operating point P cen-Cu ’s vertical coordinate, P1′_x, P2′_x, P3′_x and P4′_x are the horizontal coordinates of the four vertices of the template image in the original image, and P1′_y, P2′_y, P3′_y and P4′_y are the vertical coordinates of the four vertices of the template image in the original image.

Images