A method for identifying and locating micro-manipulated objects under deflection and occlusion

Abstract

A method for identifying and locating a micro-operation object under deflection and occlusion belongs to the technical field of micro-operation object identification and positioning under microscopic vision. The present invention solves the problem of low positioning accuracy of micro-operation objects by existing methods when the micro-operation object is deflected and occluded. The present invention specifically comprises the following steps: graying and denoising the original image where the micro-operation object is located to obtain a denoised image; performing wavelet decomposition on the denoised image to obtain a downsampled image; extracting feature points from the downsampled image and the micro-operation object template image respectively, and describing the feature points; matching the feature points in the downsampled image and the micro-operation object template image to obtain a coarse matching feature point pair, and then screening the feature point pairs obtained by the coarse matching to obtain a fine matching feature point pair; and obtaining the operation point coordinates of the micro-operation object in the original image based on the fine matching result. The method of the present invention can be applied to the identification and positioning of micro-operation objects.

Description

Technical Field

[0001] The present invention belongs to the technical field of micro-operation object recognition and positioning under microscopic vision, and particularly relates to a method for recognizing and positioning a micro-operation object under deflection and occlusion. Background Art

[0002] As micromanipulation tasks become increasingly complex, the requirements for micromanipulation precision and accuracy are becoming increasingly stringent. Simple manual operation can no longer meet the demands of the operation process. The development of image processing-based target detection and positioning technology and microscopic vision control theory has greatly improved the level of automation in micromanipulation. During micromanipulation pick-up and release, the identification and positioning of the probe tip and the manipulated object are prerequisites for achieving contact pick-up and release. Deflection and occlusion of the micromanipulation object can significantly affect its identification and positioning, severely reducing its recognition accuracy and positioning precision.

[0003] Microscopic object recognition and localization are primarily achieved through various recognition techniques, including feature-based matching, grayscale-based template matching, and deep learning-based methods. Feature-based object recognition algorithms utilize features such as lines, points, shapes, and textures in image sequences with known geometric information. They have been extensively studied and applied in microscopic object recognition and localization. For feature-based matching, the efficiency of feature point extraction and the accuracy of matching point screening directly impact the matching results. Therefore, researchers both domestically and internationally are working to improve these efficiency and accuracy. However, feature-based matching often fails to accurately identify and localize microscopic objects when they are obscured or when background noise is significant. Grayscale-based template matching, on the other hand, operates directly on pixel values, does not require any specific target pattern, and, due to its high level of redundant information, is robust to image noise. Therefore, it is widely used in microscopic object recognition and localization. However, the high computational cost of grayscale-based template matching limits its application in microscopic object recognition and localization. Consequently, numerous researchers have sought to improve the efficiency of template matching. Deep learning-based methods are widely used in the recognition of micro-operation objects. The application of deep learning algorithms to image target recognition requires a large data set, but data sets for specific micro-operation objects are difficult to construct. In addition, deep learning algorithms are generally used for qualitative recognition and classification within the micro-scale range, and have low accuracy in specific positioning.

[0004] In summary, although scholars at home and abroad have conducted extensive research on the recognition and positioning of micro-manipulation objects, existing micro-vision target positioning and recognition algorithms still suffer from low micro-manipulation object positioning accuracy when the micro-manipulation object is deflected or occluded. Therefore, the development of a high-precision and highly robust micro-manipulation object recognition and positioning algorithm is of great significance for the automation of micro-manipulation. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem of low positioning accuracy of micro-operation objects by existing methods when the micro-operation objects are deflected and occluded, and to propose a method for identifying and positioning micro-operation objects under deflection and occlusion.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a method for identifying and locating a micro-operation object under deflection and occlusion, the method specifically comprising the following steps:

[0007] Step 1: grayscale the original image where the micro-operation object is located to obtain a grayscale image, and then perform denoising on the grayscale image to obtain a denoised image;

[0008] Step 2: Initialize the wavelet decomposition times n=1, perform the first wavelet decomposition on the denoised image, and obtain the downsampled image after the first wavelet decomposition;

[0009] Step 3: let n=n+1, perform the nth wavelet decomposition on the downsampled image after the n-1th wavelet decomposition to obtain the downsampled image after the nth wavelet decomposition;

[0010] Step 4: Determine whether n=N is satisfied, where N is the maximum number of wavelet decompositions;

[0011] If n=N, then execute step 5 on the downsampled image after the Nth wavelet decomposition;

[0012] If n=N is not satisfied, return to step 3;

[0013] Step 5: Using the BRISK algorithm to extract the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image respectively, and then using the SURF algorithm to describe the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image;

[0014] Step 6: Based on the feature point description results in step 5, a brute force matching algorithm is used to match the feature points in the downsampled image after the Nth wavelet decomposition with the feature points in the micro-operation object template image to obtain a coarse matching feature point pair;

[0015] Then, the feature point pairs obtained by rough matching are screened, and the feature point pairs with incorrect matching are eliminated from the feature point pairs obtained by rough matching to obtain the feature point pairs obtained by precise matching;

[0016] Step 7: Based on the precisely matched feature point pairs, the homography matrix of the micro-operation object template image and the original image is constructed. The coordinates of the four vertices of the template image in the original image are obtained according to the coordinate information of the template image and the homography matrix. The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu coordinate.

[0017] The beneficial effects of the present invention are:

[0018] 1. Use the improved bilateral filtering algorithm to remove high-frequency noise in the image while retaining the high-frequency edge information in the image; construct an image pyramid through wavelet transform and add high-frequency information, which can retain and highlight the high-frequency information in the image while downsampling, and the extreme point features in the image are more obvious.

[0019] 2. The BRISK algorithm is used to extract feature points, and the SURF algorithm is used to describe the feature points. Compared with traditional algorithms, it achieves high-efficiency feature extraction while ensuring high matching accuracy.

[0020] 3. Compared with traditional algorithms, the feature point pair matching and screening algorithm of the present invention does not require iteration, which improves the efficiency of the algorithm. After screening the matching point pairs in the image, the mismatched point pairs are completely removed, ensuring the accuracy of subsequent positioning. Moreover, the screening method of the present invention improves the screening accuracy compared with the existing method.

[0021] Experimental results show that when the micro-operation object is occluded and deflected, the positioning error of the traditional method is greater than 5 pixels, and the time consumption is more than 400ms, which is inefficient. The method of the present invention can achieve accurate positioning when the micro-operation object is deflected and occluded, with a positioning error of less than 1 pixel and an average time consumption of about 98ms. Compared with the traditional method, the method of the present invention has obvious advantages in positioning accuracy and speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a method for identifying and locating a micro-operation object under deflection and occlusion according to the present invention;

[0023] Figure 2 This is a flowchart of the matching point screening algorithm based on size and structure similarity. DETAILED DESCRIPTION

[0024] Specific implementation method 1: Combination Figure 1This embodiment describes a method for identifying and locating a micro-operation object under deflection and occlusion, and the method is specifically as follows:

[0025] Step 1: grayscale the original image where the micro-operation object is located to obtain a grayscale image, and then perform denoising on the grayscale image to obtain a denoised image;

[0026] Step 2: Initialize the wavelet decomposition times n=1, perform the first wavelet decomposition on the denoised image, and obtain the downsampled image after the first wavelet decomposition;

[0027] Step 3: let n=n+1, perform the nth wavelet decomposition on the downsampled image after the n-1th wavelet decomposition to obtain the downsampled image after the nth wavelet decomposition;

[0028] Step 4: determine whether n=N is satisfied, where N is the maximum number of wavelet decompositions (which can be set according to actual conditions and is set to 3 in the present invention);

[0029] If n=N, then execute step 5 on the downsampled image after the Nth wavelet decomposition;

[0030] If n=N is not satisfied, return to step 3;

[0031] Step 5: Using the BRISK algorithm to extract the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image respectively, and then using the SURF algorithm to describe the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image;

[0032] Step 6: Based on the feature point description results in step 5, a Brute Force (BF) algorithm is used to match the feature points in the downsampled image after the Nth wavelet decomposition with the feature points in the micro-operation object template image to obtain a coarse matching feature point pair;

[0033] Then, the feature point pairs obtained by rough matching are screened, and the feature point pairs with incorrect matching are eliminated from the feature point pairs obtained by rough matching to obtain the feature point pairs obtained by precise matching;

[0034] Step 7: Based on the precisely matched feature point pairs, the homography matrix of the micro-operation object template image and the original image is constructed. The coordinates of the four vertices of the template image in the original image are obtained according to the coordinate information of the template image and the homography matrix. The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu Coordinates (i.e., the coordinates of the geometric center of the operation object in the template image in the original image).

[0035] Since the metal micro-components of the operating objects in the image will have an angular offset relative to the template image, the feature point matching algorithm can use the feature information of the image feature points for matching. By obtaining the positional relationship between the feature points corresponding to the template image and the downsampled image after the Nth wavelet decomposition, the angular offset and relative position of the operating target in the original image and the target on the template image can be determined, thereby completing the positioning of the target object. Therefore, the template matching algorithm based on feature points is selected to identify the operating object and locate the operating point.

[0036] Common algorithms for feature point extraction and description include Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Binary Robust Invariant Scalable Keypoints (BRISK). Compared to SIFT, SURF optimizes the description algorithm, ensuring matching accuracy with fewer descriptive features and improving matching efficiency. The BRISK algorithm simplifies the feature extraction and description steps, making it more efficient than the previous two. However, this simplified description algorithm results in a higher similarity of descriptive features, resulting in lower matching accuracy.

[0037] To achieve efficient and high-precision feature point matching, this paper proposes a feature point matching algorithm based on a combination of BRISK and SURF. BRISK is used to extract feature points, while SURF is used to describe them. Feature point matching on 50 micro-operation objects was performed and compared with other algorithms. The results show that the proposed algorithm achieves efficient feature extraction while maintaining high matching accuracy, offering significant advantages over existing algorithms.

[0038] Specific embodiment 2: This embodiment differs from specific embodiment 1 in that the grayscale processing performed on the original image where the micro-operation object is located adopts a weighted average grayscale algorithm.

[0039] Other steps and parameters are the same as those in the first embodiment.

[0040] The implementation process of the weighted average grayscale algorithm is as follows:

[0041] gray=α1R+α2G+α3B

[0042] Among them, R, G, B are the values ​​of the pixel in the red, green, and blue color channels in the original image, α1, α2, and α3 are weighting coefficients, and gray is the calculated pixel grayscale value;

[0043] Substitute the R, G, and B values ​​of each pixel in the original image into the above formula to obtain the grayscale value corresponding to each pixel;

[0044] Specific embodiment three: This embodiment differs from specific embodiment one or two in that the grayscale image denoising process uses an improved bilateral filtering algorithm, which is specifically:

[0045]

[0046] Among them, f(x,y) is the grayscale value of the pixel (x,y) in the grayscale image, f Bilateral (x0, y0) is the grayscale value of the pixel (x0, y0) in the denoised image, M(x0, y0) is the set of pixels within the convolution kernel (the convolution kernel size used in this invention is 3×3) centered at the pixel (x0, y0) in the grayscale image, and ω(x, y) is the bilateral filtering weight function for the pixel (x, y);

[0047] ω(x,y)=ω d (x,y)·ω s (x,y)

[0048] Among them, ω d (x,y) is the spatial distance domain weighting coefficient of the pixel point (x,y), ω s (x,y) is the grayscale domain weighting coefficient of the pixel (x,y);

[0049]

[0050] Among them, σ d is the spatial domain standard deviation (for each pixel in the set M(x0,y0), it is calculated Then calculate the corresponding The calculated standard deviation is used as the spatial domain standard deviation), σ s is the grayscale domain standard deviation (the grayscale value of each pixel in the set M(x0,y0) is subtracted from the grayscale value of the pixel (x,y), and then the standard deviation of all the differences is calculated), f(x0,y0) is the grayscale value of the pixel (x0,y0) in the grayscale image, |·| represents the calculated absolute value, and k is an intermediate variable.

[0051] Other steps and parameters are the same as those in the first or second embodiment.

[0052] The present invention adopts an improved bilateral filtering algorithm to remove high-frequency noise in an image while retaining high-frequency edge information in the image.

[0053] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the calculation method of the intermediate variable k is:

[0054] Establish the Robert operator ▽f in the x direction x and the Robert operator ▽f in the y direction y They are:

[0055]

[0056] Use ▽f x and ▽f y Construct a gradient image of the grayscale image, then calculate the average grayscale value of all pixels in the gradient image, and use the calculated average grayscale value as k.

[0057] The other steps and parameters are the same as those in the first to third embodiments.

[0058] Specific embodiment 5: This embodiment differs from any one of specific embodiments 1 to 4 in that the denoised image is subjected to a first wavelet decomposition to obtain a downsampled image after the first wavelet decomposition; specifically:

[0059] The Haar wavelet decomposition convolution kernel H is expressed as:

[0060]

[0061] Perform Haar wavelet decomposition on the denoised image based on H:

[0062]

[0063] Among them, H T represents the transpose of H, G represents the image after denoising, G w Represents the image after Haar wavelet decomposition, G 11 Represents G w The low-frequency sub-image in G (the low-frequency sub-image contains the basic information in image G), G 12 , G 21 and G 22 Represents G w The high-frequency sub-images in G (all three sub-images contain a certain degree of high-frequency information); and G 11 , G 12 , G 21 , G 22 The size of each is one quarter of G;

[0064] The three high-frequency sub-images G 12 , G 21 and G 22An addition operation is performed, and the grayscale of each pixel in the image obtained by the addition operation is normalized, and the image obtained by the normalization operation is used as the down-sampled image after the first wavelet decomposition.

[0065] The other steps and parameters are the same as those in the first to fourth embodiments.

[0066] In this embodiment, the wavelet transform can retain and highlight the high-frequency information in the image while downsampling, making the extreme point features in the image more obvious. The downsampling process can solve the problem of low processing efficiency caused by the large number of pixels in the original image.

[0067] Specific implementation method six: combination Figure 2 This embodiment differs from any one of the first to fifth embodiments in that the feature point pairs obtained by rough matching are screened, and the feature point pairs obtained by rough matching are eliminated from the feature point pairs obtained by rough matching to obtain the feature point pairs obtained by precise matching; specifically:

[0068] Step 6.1: The set of feature point pairs obtained by rough matching is recorded as set A. Three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3) are randomly selected from set A, and the selected feature point pairs are removed from set A.

[0069] Among them, P1, P2 and P3 are the feature points in the downsampled image after the Nth wavelet decomposition, and p1, p2 and p3 are the feature points in the template image of the micro-operation object;

[0070] Step 62: record the distance between feature point P1 and feature point P2 as D1, the distance between feature point P1 and feature point P3 as D2, the distance between feature point P2 and feature point P3 as D3, the distance between feature point P1 and feature point P2 as d1, the distance between feature point P1 and feature point P3 as d2, and the distance between feature point P2 and feature point P3 as d3;

[0071] Then calculate the intermediate variables s1, s2 and s3:

[0072]

[0073] Step 6.3. Calculate the variance of s1, s2 and s3;

[0074] If the calculated variance is less than the set distance variance threshold t1, then continue to step 64;

[0075] If the calculated variance is greater than or equal to the set threshold t1, then a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A; the selected feature point pair is used to replace any one of the three feature point pairs corresponding to the variance, to obtain a set of feature point pairs consisting of three feature point pairs, and then the step 62 is returned to the obtained set of feature point pairs;

[0076] Step 64, calculate the angle α1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle α2 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P2 and the feature point P3, calculate the angle α3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3, calculate the angle β1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle β2 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P2 and the feature point P3, and calculate the angle β3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3;

[0077] The angle between vectors is calculated as follows:

[0078]

[0079] Calculate the angle deviations |α1-β1|, |α2-β2|, and |α3-β3|;

[0080] If |α1-β1|, |α2-β2|, and |α3-β3| are all less than the set angle variance threshold t2, then the three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3) are all correctly matched feature point pairs, and step 65 is performed on the three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3);

[0081] Otherwise, randomly select a feature point pair from the remaining feature point pairs in set A, remove the selected feature point pair from set A, and use the selected feature point pair to replace any one of the three feature point pairs (referring to the three feature point pairs currently participating in the calculation) to obtain a new set of feature point pairs consisting of three feature point pairs, and then return to step 62 for this set of feature point pairs;

[0082] Step 65: Select any two pairs from the three pairs of feature point pairs as base points, and use the remaining feature point pairs in the selected base point pair set A to perform verification in sequence. The verification method is:

[0083] For any remaining feature point pair in set A, use this feature point pair and the base point to form a set of feature point pairs. If this set of feature point pairs satisfies both the distance variance threshold condition and the angle variance threshold condition (i.e., the variance calculated based on this set of feature point pairs is less than t1, and the three angle deviations corresponding to this set of feature point pairs are all less than t2), then this feature point pair is a correctly matched feature point pair. Otherwise, this feature point pair is a mismatched feature point pair.

[0084] After verifying the remaining feature point pairs in set A respectively, all correctly matched feature point pairs are obtained, that is, the precisely matched feature point pairs are obtained.

[0085] The other steps and parameters are the same as those in the first to fifth embodiments.

[0086] Specific embodiment seven: This embodiment differs from any one of specific embodiments one to six in that the value of the distance variance threshold t1 is 0.04.

[0087] The other steps and parameters are the same as those in the first to sixth embodiments.

[0088] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that the value of the angle variance threshold t2 is 0.06.

[0089] The other steps and parameters are the same as those in the first to seventh embodiments.

[0090] By setting appropriate distance variance thresholds t1 and angle variance thresholds t2, correct matching point pairs can be retained while all incorrect matching point pairs are filtered out, achieving optimal screening results. Furthermore, compared to traditional algorithms, the matching point pair screening algorithm of the present invention does not require iteration, which improves algorithm efficiency. After screening matching point pairs in the image, incorrect matching point pairs are completely removed, improving screening accuracy.

[0091] Specific embodiment 9: This embodiment differs from any one of specific embodiments 1 to 8 in that the operation point P of the micro-operation object is calculated based on the obtained coordinates. cen-Cu Coordinates, specifically:

[0092]

[0093] Among them, P cen-Cu _x is the operating point P cen-Cu The horizontal axis, P cen-Cu _y is the operating point P cen-CuThe vertical coordinates of the four vertices of the template image are P′1_x, P′2_x, P′3_x and P′4_x, respectively. The horizontal coordinates of the four vertices of the template image in the original image are P′1_y, P′2_y, P′3_y and P′4_y, respectively.

[0094] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0095] Example

[0096] This embodiment proposes a method for identifying and locating a micro-operation object under deflection and occlusion, and the method is specifically as follows:

[0097] Step 1: grayscale the original image where the micro-operation object is located using a weighted average grayscale algorithm to obtain a grayscale image, and then use an improved bilateral filtering algorithm to denoise the grayscale image to obtain a denoised image;

[0098] The improved bilateral filtering algorithm is specifically as follows:

[0099]

[0100] Among them, f(x,y) is the grayscale value of the pixel (x,y) in the grayscale image, f Bilateral (x0, y0) is the grayscale value of the pixel (x0, y0) in the denoised image, M(x0, y0) is the set of pixels within the convolution kernel (the convolution kernel size used in this invention is 3×3) centered at the pixel (x0, y0) in the grayscale image, and ω(x, y) is the bilateral filtering weight function for the pixel (x, y);

[0101] ω(x,y)=ω d (x,y)·ω s (x,y)

[0102] Among them, ω d (x,y) is the spatial distance domain weighting coefficient of the pixel point (x,y), ω s (x,y) is the grayscale domain weighting coefficient of the pixel (x,y);

[0103]

[0104] Among them, σ d is the spatial domain standard deviation (for each pixel in the set M(x0,y0), it is calculated Then calculate the corresponding The calculated standard deviation is used as the spatial domain standard deviation), σ sis the grayscale domain standard deviation (the grayscale value of each pixel in the set M(x0,y0) is subtracted from the grayscale value of the pixel (x,y), and then the standard deviation of all the differences is calculated), f(x0,y0) is the grayscale value of the pixel (x0,y0) in the grayscale image, |·| represents the calculated absolute value, and k is an intermediate variable.

[0105] The calculation method of the intermediate variable k is:

[0106] Establish the Robert operator ▽f in the x direction x and the Robert operator ▽f in the y direction y They are:

[0107]

[0108] Use ▽f x and ▽f y Construct a gradient image of the grayscale image, then calculate the average grayscale value of all pixels in the gradient image, and use the calculated average grayscale value as k.

[0109] Step 2: Initialize the wavelet decomposition times n=1, perform the first wavelet decomposition on the denoised image, and obtain the downsampled image after the first wavelet decomposition; specifically:

[0110] The Haar wavelet decomposition convolution kernel H is expressed as:

[0111]

[0112] Perform Haar wavelet decomposition on the denoised image based on H:

[0113]

[0114] Among them, H T represents the transpose of H, G represents the image after denoising, G w Represents the image after Haar wavelet decomposition, G 11 Represents G w The low-frequency sub-image in G (the low-frequency sub-image contains the basic information in image G), G 12 , G 21 and G 22 Represents G w The high-frequency sub-images in G (all three sub-images contain a certain degree of high-frequency information); and G 11 , G 12 , G 21 , G 22 The size of each is one quarter of G;

[0115] The three high-frequency sub-images G 12 , G21 and G 22 An addition operation is performed, and the grayscale of each pixel in the image obtained by the addition operation is normalized, and the image obtained by the normalization operation is used as the down-sampled image after the first wavelet decomposition.

[0116] Step 3: let n=n+1, perform the nth wavelet decomposition on the downsampled image after the n-1th wavelet decomposition to obtain the downsampled image after the nth wavelet decomposition;

[0117] Step 4: determine whether n=N is satisfied, where N is the maximum number of wavelet decompositions (which can be set according to actual conditions and is set to 3 in the present invention);

[0118] If n=N, then execute step 5 on the downsampled image after the Nth wavelet decomposition;

[0119] If n=N is not satisfied, return to step 3;

[0120] Step 5: Using the BRISK algorithm to extract the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image respectively, and then using the SURF algorithm to describe the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image;

[0121] Step 6: Based on the feature point description results in step 5, a Brute Force (BF) algorithm is used to match the feature points in the downsampled image after the Nth wavelet decomposition with the feature points in the micro-operation object template image to obtain a coarse matching feature point pair;

[0122] Then, the feature point pairs obtained by rough matching are screened, and the feature point pairs with incorrect matching are eliminated from the feature point pairs obtained by rough matching to obtain the feature point pairs obtained by precise matching; specifically:

[0123] Step 6.1: The set of feature point pairs obtained by rough matching is recorded as set A. Three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3) are randomly selected from set A, and the selected feature point pairs are removed from set A.

[0124] Among them, P1, P2 and P3 are the feature points in the downsampled image after the Nth wavelet decomposition, and p1, p2 and p3 are the feature points in the template image of the micro-operation object;

[0125] Step 62: record the distance between feature point P1 and feature point P2 as D1, the distance between feature point P1 and feature point P3 as D2, the distance between feature point P2 and feature point P3 as D3, the distance between feature point P1 and feature point P2 as d1, the distance between feature point P1 and feature point P3 as d2, and the distance between feature point P2 and feature point P3 as d3;

[0126] Then calculate the intermediate variables s1, s2 and s3:

[0127]

[0128] Step 6.3. Calculate the variance of s1, s2 and s3;

[0129] If the calculated variance is less than the set distance variance threshold t1 (in the present invention, the value of t1 is 0.04), then continue to step 64;

[0130] If the calculated variance is greater than or equal to the set threshold t1, then a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A; the selected feature point pair is used to replace any one of the three feature point pairs corresponding to the variance, to obtain a set of feature point pairs consisting of three feature point pairs, and then the step 62 is returned to the obtained set of feature point pairs;

[0131] Step 64, calculate the angle α1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle α2 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P2 and the feature point P3, calculate the angle α3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3, calculate the angle β1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle β2 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P2 and the feature point P3, and calculate the angle β3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3;

[0132] The angle between vectors is calculated as follows:

[0133]

[0134] Calculate the angle deviations |α1-β1|, |α2-β2|, and |α3-β3|;

[0135] If |α1-β1|, |α2-β2|, and |α3-β3| are all less than the set angle variance threshold t2 (in the present invention, the value of t2 is 0.06), then the three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3) are all correctly matched feature point pairs, and step 65 is performed on the three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3);

[0136] Otherwise, randomly select a feature point pair from the remaining feature point pairs in set A, remove the selected feature point pair from set A, and use the selected feature point pair to replace any one of the three feature point pairs (referring to the three feature point pairs currently participating in the calculation) to obtain a new set of feature point pairs consisting of three feature point pairs, and then return to step 62 for this set of feature point pairs;

[0137] Step 65: Select any two pairs from the three pairs of feature point pairs as base points, and use the remaining feature point pairs in the selected base point pair set A to perform verification in sequence. The verification method is:

[0138] For any remaining feature point pair in set A, use this feature point pair and the base point to form a set of feature point pairs. If this set of feature point pairs satisfies both the distance variance threshold condition and the angle variance threshold condition (i.e., the variance calculated based on this set of feature point pairs is less than t1, and the three angle deviations corresponding to this set of feature point pairs are all less than t2), then this feature point pair is a correctly matched feature point pair. Otherwise, this feature point pair is a mismatched feature point pair.

[0139] After verifying the remaining feature point pairs in set A respectively, all correctly matched feature point pairs are obtained, that is, the precisely matched feature point pairs are obtained.

[0140] Step 7: Based on the precisely matched feature point pairs, the homography matrix of the micro-operation object template image and the original image is constructed. The coordinates of the four vertices of the template image in the original image are obtained according to the coordinate information of the template image and the homography matrix. The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu Coordinates (i.e., the coordinates of the geometric center of the operation object in the template image in the original image):

[0141]

[0142] Among them, P cen-Cu _x is the operating point P cen-Cu The horizontal axis, P cen-Cu _y is the operating point P cen-CuThe vertical coordinates of the four vertices of the template image are P1′_x, P2′_x, P3′_x and P4′_x, respectively. The horizontal coordinates of the four vertices of the template image in the original image are P1′_y, P2′_y, P3′_y and P4′_y are the vertical coordinates of the four vertices of the template image in the original image.

[0143] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for identifying and locating a micro-operation object under deflection and occlusion, characterized in that: The method specifically comprises the following steps: Step 1: grayscale the original image where the micro-operation object is located to obtain a grayscale image, and then perform denoising on the grayscale image to obtain a denoised image; Step 2: Initialize the wavelet decomposition times n=1, perform the first wavelet decomposition on the denoised image, and obtain the downsampled image after the first wavelet decomposition; Step 3: let n=n+1, perform the nth wavelet decomposition on the downsampled image after the n-1th wavelet decomposition to obtain the downsampled image after the nth wavelet decomposition; Step 4: Determine whether n=N is satisfied, where N is the maximum number of wavelet decompositions; If n=N, then execute step 5 on the downsampled image after the Nth wavelet decomposition; If n=N is not satisfied, return to step 3; Step 5: Using the BRISK algorithm to extract the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image respectively, and then using the SURF algorithm to describe the feature points in the downsampled image after the Nth wavelet decomposition and the micro-operation object template image; Step 6: Based on the feature point description results in step 5, a brute force matching algorithm is used to match the feature points in the downsampled image after the Nth wavelet decomposition with the feature points in the micro-operation object template image to obtain a coarse matching feature point pair; Then, the feature point pairs obtained by rough matching are screened, and the feature point pairs with incorrect matching are eliminated from the feature point pairs obtained by rough matching to obtain the feature point pairs obtained by precise matching; Step 7: Based on the precisely matched feature point pairs, the homography matrix of the micro-operation object template image and the original image is constructed. The coordinates of the four vertices of the template image in the original image are obtained according to the coordinate information of the template image and the homography matrix. The operation point P of the micro-operation object is calculated according to the obtained coordinates. cen-Cu coordinate.

2. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 1, wherein: The grayscale processing performed on the original image where the micro-operation object is located adopts a weighted average grayscale algorithm.

3. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 2, wherein: The grayscale image denoising process adopts an improved bilateral filtering algorithm, which is specifically as follows: Among them, f(x,y) is the grayscale value of the pixel (x,y) in the grayscale image, f Bilateral (x0, y0) is the grayscale value of the pixel (x0, y0) in the denoised image, M(x0, y0) is the set of pixels in the convolution kernel centered at the pixel (x0, y0) in the grayscale image, and ω(x, y) is the bilateral filtering weight function for the pixel (x, y); ω(x,y)=ω d (x,y)·ω s (x,y) Among them, ω d (x,y) is the spatial distance domain weighting coefficient of the pixel point (x,y), ω s (x,y) is the grayscale domain weighting coefficient of the pixel (x,y); Among them, σ d is the spatial domain standard deviation, σ s is the standard deviation of the grayscale domain, f(x0,y0) is the grayscale value of the pixel (x0,y0) in the grayscale image, |·| represents the calculation of the absolute value, and k is an intermediate variable.

4. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 3, wherein: The calculation method of the intermediate variable k is: Establish the Robert operator in the x direction and the Robert operator in the y direction They are: use and Construct a gradient image of the grayscale image, then calculate the average grayscale value of all pixels in the gradient image, and use the calculated average grayscale value as k.

5. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 4, wherein: The first wavelet decomposition is performed on the denoised image to obtain a downsampled image after the first wavelet decomposition; specifically: The Haar wavelet decomposition convolution kernel H is expressed as: Perform Haar wavelet decomposition on the denoised image based on H: Among them, H T represents the transpose of H, G represents the image after denoising, G w Represents the image after Haar wavelet decomposition, G 11 Represents G w The low-frequency sub-image in G 12 , G 21 and G 22 Represents G w High-frequency sub-images in ; The three high-frequency sub-images G 12 , G 21 and G 22 An addition operation is performed, and the grayscale of each pixel in the image obtained by the addition operation is normalized, and the image obtained by the normalization operation is used as the down-sampled image after the first wavelet decomposition.

6. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 5, characterized in that: The feature point pairs obtained by the rough matching are screened, and the feature point pairs obtained by the rough matching are eliminated from the feature point pairs obtained by the rough matching to obtain the feature point pairs obtained by the precise matching; specifically: Step 6.1: The set of feature point pairs obtained by rough matching is recorded as set A. Three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3) are randomly selected from set A, and the selected feature point pairs are removed from set A. Among them, P1, P2 and P3 are the feature points in the downsampled image after the Nth wavelet decomposition, and p1, p2 and p3 are the feature points in the template image of the micro-operation object; Step 62: record the distance between feature point P1 and feature point P2 as D1, the distance between feature point P1 and feature point P3 as D2, the distance between feature point P2 and feature point P3 as D3, the distance between feature point P1 and feature point P2 as d1, the distance between feature point P1 and feature point P3 as d2, and the distance between feature point P2 and feature point P3 as d3; Then calculate the intermediate variables s1, s2 and s3: Step 6.

3. Calculate the variance of s1, s2 and s3; If the calculated variance is less than the set distance variance threshold t1, then continue to step 64; If the calculated variance is greater than or equal to the set threshold t1, then a feature point pair is randomly selected from the remaining feature point pairs in set A, and the selected feature point pair is removed from set A; the selected feature point pair is used to replace any one of the three feature point pairs corresponding to the variance, to obtain a set of feature point pairs consisting of three feature point pairs, and then the step 62 is returned to the obtained set of feature point pairs; Step 64, calculate the angle α1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle α2 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P2 and the feature point P3, calculate the angle α3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3, calculate the angle β1 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P1 and the feature point P3, calculate the angle β2 between the line connecting the feature point P1 and the feature point P2 and the line connecting the feature point P2 and the feature point P3, and calculate the angle β3 between the line connecting the feature point P1 and the feature point P3 and the line connecting the feature point P2 and the feature point P3; Calculate the angle deviations |α1-β1|, |α2-β2|, and |α3-β3|; If |α1-β1|, |α2-β2|, and |α3-β3| are all less than the set angle variance threshold t2, then the three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3) are all correctly matched feature point pairs, and step 65 is performed on the three pairs of feature point pairs (P1, p1), (P2, p2), and (P3, p3); Otherwise, randomly select a feature point pair from the remaining feature point pairs in set A, remove the selected feature point pair from set A, and use the selected feature point pair to replace any one of the three feature point pairs to obtain a new set of feature point pairs consisting of three feature point pairs, and then return to step 62 for this set of feature point pairs; Step 65: Select any two pairs from the three pairs of feature point pairs as base points, and use the remaining feature point pairs in the selected base point pair set A to perform verification in sequence. The verification method is: For any remaining feature point pair in set A, use this feature point pair and the base point to form a set of feature point pairs. If this set of feature point pairs satisfies both the distance variance threshold condition and the angle variance threshold condition, then this feature point pair is a correctly matched feature point pair. Otherwise, this feature point pair is a mismatched feature point pair. After verifying the remaining feature point pairs in set A respectively, all correctly matched feature point pairs are obtained, that is, the precisely matched feature point pairs are obtained.

7. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 6, wherein: The value of the distance variance threshold t1 is 0.

04.

8. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 7, wherein: The value of the angle variance threshold t2 is 0.

06.

9. The method for identifying and locating a micro-operation object under deflection and occlusion according to claim 8, wherein: The operation point P of the micro-operation object is calculated according to the obtained coordinates cen-Cu Coordinates, specifically: Among them, P cen-Cu _x is the operating point P cen-Cu The horizontal axis, P cen-Cu _y is the operating point P cen-Cu The vertical coordinates of the four vertices of the template image are P1′_x, P2′_x, P3′_x and P4′_x, respectively. The horizontal coordinates of the four vertices of the template image in the original image are P1′_y, P2′_y, P3′_y and P4′_y are the vertical coordinates of the four vertices of the template image in the original image.

Images