Drilling positioning method based on computer vision

Abstract

The application discloses a drilling positioning method based on computer vision and relates to the technical field of drilling positioning, and comprises the following steps: obtaining a target pretreatment image and a template pretreatment image; generating a target hole rim constraint image, a target key point set, a target descriptor set, a template key point set and a template descriptor set; forming a geometric perception descriptor set; inputting the geometric perception descriptor set into an improved LightGlue feature matching network to obtain a first round feature matching pair set and a corresponding matching confidence average; outputting a hole rim geometric consistent feature matching pair set if a preset threshold is met; if the preset threshold is not met, returning to the improved LightGlue feature matching network input for continuous iteration; and obtaining a spatial positioning coordinate of a drilling center in a machining equipment coordinate system. The application can reduce the hole rim mismatch probability under the conditions of repeated hole positions, repeated textures or complex metal texture backgrounds.

Description

Technical Field

[0001] This invention relates to the field of borehole positioning technology, and in particular to a borehole positioning method based on computer vision. Background Technology

[0002] With the development of intelligent manufacturing, flexible processing and robotic automated assembly, the requirements for drilling center positioning accuracy and real-time performance in industrial sites are constantly increasing. In the process of CNC drilling, automatic hole reaming, flange assembly and aerospace structural parts processing, it is necessary to use industrial vision systems to identify and center the pre-made holes, process holes or positioning holes on the workpiece surface.

[0003] Existing technologies often employ Canny edge detection, Hough circle detection, template matching, or manual feature extraction methods such as SIFT, SURF, and ORB for hole location identification and center fitting. However, these methods are sensitive to changes in lighting and surface texture interference. When there are bright reflective areas or dense scratches on the workpiece surface, non-hole edge areas are easily misidentified as drilling boundaries, resulting in discontinuous extracted edge contours or a large number of false edges. This can cause the drilling center to shift or even fail to locate. At the same time, they lack the semantic understanding of the true physical structure of the hole, making it difficult to stably distinguish between the true hole edge and the background texture in complex backgrounds.

[0004] With the development of deep learning feature matching technology, some existing technologies have begun to use SuperGlue and LightGlue attention feature matching networks for high-precision image matching to improve hole location accuracy. However, existing LightGlue feature matching networks are mainly designed for natural scene images, and usually perform feature search and cross-image attention calculation directly on global high-resolution industrial images. This is not only computationally intensive, but also prone to generating incorrect matching relationships when industrial workpieces have a large number of repetitive textures, repetitive hole positions, or similar processing structures. In addition, the position encoding of existing LightGlue feature matching networks usually only considers two-dimensional pixel coordinate information, lacking the ability to constrain the geometric features of the hole edge, resulting in hole edge matching drift and incorrect convergence problems in complex industrial backgrounds. At the same time, the early stopping mechanisms of existing feature matching networks mostly rely only on matching confidence, lacking constraints on the overall circular or elliptical geometric consistency of the hole edge, making it difficult to guarantee that the final matching result meets the geometric characteristics of the real drill hole boundary. Summary of the Invention

[0005] One objective of this invention is to propose a computer vision-based drilling location method. Under conditions of repeated hole positions, repeated textures, or complex metal texture backgrounds, this invention uses radial geometric constraints to ensure that key point matching must meet geometric consistency relative to the hole edge boundary, thereby reducing the probability of hole edge mismatch.

[0006] A drilling positioning method based on computer vision according to an embodiment of the present invention includes: Acquire the target drilling image and the standard drilling template image of the workpiece to be processed, and preprocess them respectively to obtain the target preprocessed image and the template preprocessed image; The preprocessed target image is input into the SAM2 aperture edge segmentation model based on the zero-shot semantic segmentation mechanism to generate the target aperture edge constraint image; The template preprocessed image is input into the SAM2 hole edge segmentation model to obtain the template hole edge prior mask and generate the template hole edge constraint image. Feature extraction is performed on the target hole edge constraint image and the template hole edge constraint image respectively to obtain the target key point set, the target descriptor set, the template key point set and the template descriptor set. For each key point in the target key point set and the template key point set, calculate the minimum radial geometric distance to the corresponding prior mask boundary curve of the hole edge, and inject it into the target descriptor subset and the template descriptor subset to form a geometrically aware descriptor subset; Input the geometrically aware descriptor set into the improved LightGlue feature matching network to obtain the first round of feature matching pair set and the corresponding average matching confidence score; The first-round feature matching pair set is fitted with a circle or ellipse, the fitting residual is calculated, and a circle / ellipse consistency convergence index is constructed based on the average matching confidence and the fitting residual. When the circle / ellipse consistency convergence index meets the preset threshold, adaptive early stopping is triggered, subsequent attention iterations are stopped, and the set of feature matching pairs with geometric consistency of the hole edge is output. If the preset threshold is not met, the improved LightGlue feature matching network is returned as input to continue the iteration. Image mapping is established based on the set of geometrically consistent feature matching pairs of borehole edges. After removing abnormal matches, sub-pixel borehole edges are reconstructed and fitted to obtain the pixel coordinates of the borehole center in the target preprocessed image.

[0007] The pixel coordinates are mapped to the coordinate system of the machining equipment to obtain the spatial positioning coordinates of the drilling center in the coordinate system of the machining equipment.

[0008] Optionally, the preprocessing includes camera distortion correction, scale normalization, and brightness equalization.

[0009] Optionally, inputting the preprocessed target image into the SAM2 aperture edge segmentation model based on a zero-shot semantic segmentation mechanism includes: For the target preprocessed image, establish the pixel coordinate domain; The preprocessed target image is input into the SAM2 pore edge segmentation model based on zero-shot semantic segmentation mechanism to generate a target pore edge probability map; Construct a priori mask for the target aperture edge based on the target aperture edge probability map; Based on the prior mask of the target aperture edge and the pixel coordinate domain, generate a set of pixel coordinates of the boundary of the prior mask of the target aperture edge; Based on the set of pixel coordinates of the prior mask boundary of the target aperture, construct the target aperture boundary extension region; Based on the prior mask of the target aperture edge and the expansion region of the target aperture edge boundary, the effective constraint region of the target aperture edge is determined; Based on the effective constraint region of the target aperture edge, determine the set of pixel coordinates outside the mask; Based on the effective constraint region of the target aperture edge and the set of pixel coordinates outside the mask, the target preprocessed image is subjected to mask constraint cropping to generate the target aperture edge constraint image.

[0010] If the current pixel coordinates belong to the set of pixel coordinates outside the mask, then the pixel values ​​of the same pixel coordinates in the target hole edge constraint image are set to zero pixel values ​​consistent with the channel dimension of the target preprocessed image.

[0011] Optionally, the SAM2 aperture edge segmentation model based on the zero-shot semantic segmentation mechanism includes: The target preprocessed image is semantically encoded using an image encoder to obtain the target image semantic feature map; Based on the semantic feature map of the target image, a set of candidate clues for the edge of the target hole is generated in the pixel coordinate domain of the preprocessed target image; The target aperture edge cue embedding feature set is obtained by encoding the cue features of the candidate cue set of the target aperture edge by the cue encoder; By performing joint mask decoding on the semantic feature map of the target image and the embedded feature set of the target aperture edge cue, a set of candidate aperture edge masks for the target is obtained. The candidate mask filter calculates the confidence score of each target candidate aperture edge mask and the semantic feature map of the target image, and the target candidate aperture edge mask with the highest confidence score is determined as the target aperture edge probability map.

[0012] Optionally, inputting the template preprocessed image into the SAM2 aperture edge segmentation model includes: For the template preprocessed image, establish the template pixel coordinate domain; Input the preprocessed template image into the SAM2 aperture edge segmentation model to generate a template aperture edge probability map; Construct a priori mask for template aperture edges based on the template aperture edge probability map; Based on the template aperture prior mask and the template pixel coordinate domain, generate a set of template aperture prior mask boundary pixel coordinates; Based on the set of pixel coordinates of the prior mask boundary of the template aperture, the extended region of the template aperture boundary is constructed; Based on the prior mask of the template hole edge and the extension region of the template hole edge boundary, the effective constraint region of the template hole edge is determined; Based on the effective constraint area of ​​the template hole edge, determine the set of pixel coordinates outside the template mask; Based on the effective constraint region of the template hole edge and the set of pixel coordinates outside the template mask, the template preprocessed image is mask-constrained and cropped to generate the template hole edge constraint image; The same feature extraction network is used to extract features from the target hole edge constraint image and the template hole edge constraint image respectively, resulting in a target key point set, a target descriptor set, a template key point set, and a template descriptor set.

[0013] Optionally, calculating the minimum radial geometric distance from each key point in the target key point set and the template key point set to the corresponding prior mask boundary curve of the hole edge includes: Based on the set of pixel coordinates of the prior mask boundary of the target aperture, calculate the minimum radial geometric distance of the target corresponding to each target key point in the target key point set; Based on the set of pixel coordinates of the prior mask boundary at the template aperture edge, calculate the minimum radial geometric distance of the template corresponding to each template key point in the template key point set; The two-dimensional pixel coordinates of each target key point in the target hole edge constraint image are combined with the corresponding target minimum radial geometric distance to form the target three-dimensional geometric position vector corresponding to each target key point; The two-dimensional pixel coordinates of each template key point in the template hole edge constraint image are combined with the corresponding minimum radial geometric distance of the template to form the template three-dimensional geometric position vector corresponding to each template key point. Input the target's 3D geometric position vector and the template's 3D geometric position vector into the 3D position encoding function to generate a target geometric position encoding vector and a template geometric position encoding vector that are consistent with the dimensions of the target descriptor and the template descriptor, respectively. The target geometric position encoding vector is injected into the target descriptor, and the template geometric position encoding vector is injected into the template descriptor to generate the target geometric perception descriptor and the template geometric perception descriptor, respectively. All target geometry-aware descriptors are combined to form a target geometry-aware descriptor set, and all template geometry-aware descriptors are combined to form a template geometry-aware descriptor set. The target geometry-aware descriptor set and the template geometry-aware descriptor set together constitute the geometry-aware descriptor set.

[0014] Optionally, inputting the geometry-aware descriptor subset into the improved LightGlue feature matching network includes: The geometrically aware descriptor set is input into the improved LightGlue feature matching network, and the target geometrically aware descriptor set and the template geometrically aware descriptor set are expressed in matrix form to obtain the first round target input feature matrix and the first round template input feature matrix. Self-attention calculations are performed on the first-round target input feature matrix and the first-round template input feature matrix respectively to obtain the first-round target self-attention feature matrix and the first-round template self-attention feature matrix; Cross-attention calculation is performed based on the first round target self-attention feature matrix and the first round template self-attention feature matrix to obtain the first round target cross-attention feature matrix and the first round template cross-attention feature matrix; Based on the first round target cross-attention feature matrix and the first round template cross-attention feature matrix, calculate the first round matching similarity between target key points and template key points; Perform matching confidence prediction on the first round of matching similarity to obtain the first round of matching confidence matrix, and determine the first round of candidate matching relationships based on the first round of matching confidence matrix; Based on the first round of candidate matching relationships, bidirectional mutual selection and confidence threshold determination are performed on any target key point and any template key point to generate the first round of feature matching pair set; Calculate the average matching confidence score based on the first-round matching confidence score of each first-round feature matching pair in the first-round feature matching pair set.

[0015] Optionally, the step of performing circular or elliptical fitting on the first round of feature matching pairs, calculating the fitting residuals, and constructing a circular / elliptical consistency convergence index based on the average matching confidence and the fitting residuals includes: The first round of feature matching pairs is used as the current round of feature matching pairs, and a round representation of the current round of feature matching pairs is established. Based on the current set of feature matching pairs, extract the set of coordinates of key matching points on the target hole edge; Perform circular fitting on the set of key point coordinates for matching the target hole edge to obtain the current wheel's circular fitting parameters and the current wheel's circular fitting residual; Elliptical fitting is performed on the set of coordinates of key points for matching the target hole edge to obtain the elliptical fitting parameters and residuals of the current wheel. Determine the fitting residual of the current wheel based on the circular fitting residual or the elliptical fitting residual of the current wheel; Calculate the average matching confidence of the current round based on the matching confidence of each feature matching pair in the current round feature matching pair set; The weighted sum of the average confidence score of the current round and the fitting residual of the current round is used to obtain the circle / ellipse consistency convergence index of the current round. When the current wheel circle / ellipse consistency convergence index is less than or equal to the preset threshold, it is determined that the current wheel feature matching pair set satisfies the hole edge geometric consistency convergence condition, triggering adaptive early stopping, stopping attention iteration, and outputting the current wheel feature matching pair set as the hole edge geometric consistency feature matching pair set; When the convergence index of the current wheel circle / ellipse is greater than the preset threshold, it is determined that the current wheel feature matching pair set does not meet the convergence condition of hole edge geometry consistency. The target cross attention feature matrix and the template cross attention feature matrix of the current wheel are then input into the next attention layer of the improved LightGlue feature matching network to continue iterating.

[0016] Optionally, the step of establishing an image mapping based on the set of geometrically consistent feature matching pairs of aperture edges, reconstructing sub-pixel aperture edges after removing anomalous matches, and fitting the image includes: Based on the set of matching pairs of geometrically consistent hole edge features, the coordinate pairs of matching key points between the target hole edge constraint image and the template hole edge constraint image are extracted; Based on the matching key point coordinate pairs, the geometric mapping relationship between the target hole edge constraint image and the template hole edge constraint image is established, and the hole edge homography geometric mapping matrix is ​​obtained by solving. Based on the aperture edge geometry mapping matrix, calculate the mapping reprojection error corresponding to each aperture edge geometry consistent feature matching pair; Based on the mapping reprojection error, feature matching pairs that do not meet geometric consistency are eliminated to obtain the final set of valid feature matching pairs. Based on the final set of effective feature matching pairs, extract the set of coordinates of effective target hole edge key points in the target hole edge constraint image; Based on the effective set of key point coordinates of the target borehole edge, sub-pixel level contour reconstruction is performed on the borehole edge in the target borehole edge constraint image to obtain the sub-pixel contour point set of the target borehole. Perform circular fitting based on the set of sub-pixel contour points of the target borehole to obtain the center pixel coordinates of the circular borehole; Elliptical fitting is performed based on the set of sub-pixel contour points of the target borehole to obtain the center pixel coordinates of the elliptical borehole; Based on the pixel coordinates of the center of a circular or elliptical borehole, the pixel coordinates of the borehole center in the target preprocessed image are obtained.

[0017] Optionally, the step of mapping pixel coordinates to the machining equipment coordinate system to obtain the spatial positioning coordinates of the drilling center in the machining equipment coordinate system includes: Based on the pixel coordinates of the borehole center in the target preprocessed image, construct the homogeneous pixel coordinates of the borehole center; Based on the camera intrinsic parameter matrix, the homogeneous pixel coordinates of the borehole center are converted into camera normalized imaging rays. Calculate the ray scale factor of the borehole center reaching the target machining plane along the camera's normalized imaging ray. Multiply the ray scale factor of the borehole center reaching the target processing plane along the normalized imaging ray of the camera by the normalized imaging ray of the borehole center in the camera coordinate system to obtain the spatial positioning coordinates of the borehole center in the camera coordinate system. Map the spatial positioning coordinates of the drilling center in the camera coordinate system to the machining equipment coordinate system to obtain the spatial positioning coordinates of the drilling center in the machining equipment coordinate system; Based on the spatial positioning coordinates of the drilling center in the coordinate system of the machining equipment, the feed positioning target point of the CNC machine tool spindle, the end effector of the industrial robot, or the automated drilling machine is generated. Based on the type of machining object, the feed positioning command is generated based on the feed positioning target point.

[0018] The beneficial effects of this invention are: This invention fuses the SAM2 aperture edge segmentation prior with an improved LightGlue feature matching network. The SAM2 aperture edge segmentation model performs zero-shot semantic segmentation on the target preprocessed image and the template preprocessed image, generating a target aperture edge prior mask and a template aperture edge prior mask. Based on the aperture edge prior mask, mask constraints are applied to highly reflective areas, scratch areas, oil stain areas, and repetitive texture background areas, ensuring that subsequent LightGlue feature matching is performed only within the actual aperture edge region. Since SAM2 has semantic-level recognition capabilities for the physical boundaries of holes, it can effectively suppress the interference of bright pseudo-edges and repetitive textures in complex industrial backgrounds on the matching results, fundamentally reducing the number of incorrect matching points. Simultaneously, the improved LightGlue feature matching network performs cross-attention calculations only within the aperture edge constraint region, compressing the feature search range from the global high-resolution image to the local aperture edge region, significantly reducing the attention computation load.

[0019] This invention addresses the problem of existing LightGlue feature matching networks using only two-dimensional positional encoding and lacking constraints on aperture edge structure. It introduces a minimum radial geometric distance encoding mechanism based on the prior mask boundary of the aperture edge. The minimum radial geometric distance between the target keypoint and the template keypoint relative to the corresponding aperture edge boundary, along with the two-dimensional pixel coordinates, is input into a unified, learnable three-dimensional positional encoding function to form target geometric positional encoding vectors and template geometric positional encoding vectors. These vectors are then injected into the target descriptor and template descriptor, respectively. This allows the descriptor to simultaneously possess local aperture edge texture information, two-dimensional spatial position information, and aperture edge radial geometric structure information. This enhances the network's geometric perception ability for circular and elliptical aperture edge structures. Under conditions of repeated aperture locations, repeated textures, or complex metal texture backgrounds, radial geometric constraints ensure that keypoint matching must satisfy geometric consistency relative to the aperture edge boundary, reducing the probability of aperture edge mismatch.

[0020] This invention proposes an adaptive early stopping mechanism based on circle / ellipse consistency. After each round of attention iteration, a circular or elliptical fitting is performed on the current round's feature matching pair set, and the circular or elliptical fitting residual is calculated. Then, a circle / ellipse consistency convergence index is constructed by combining the average matching confidence. When the current round's feature matching pair set simultaneously meets the conditions of high matching confidence and low aperture edge fitting residual, adaptive early stopping is triggered, and subsequent attention iterations are stopped. If the aperture edge geometric consistency condition is not met, subsequent attention matching continues. This avoids the problem of premature convergence in traditional LightGlue when the local texture confidence is high but the overall aperture edge structure is incorrect. This invention can dynamically control the attention iteration depth, reducing the amount of invalid computation while ensuring aperture edge geometric consistency. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a drilling positioning method based on computer vision proposed in this invention; Figure 2 This is a network structure diagram of the improved LightGlue feature matching network in a computer vision-based borehole localization method proposed in this invention. Figure 3 This is the interface of the drilling machine control system in the drilling positioning method based on computer vision proposed in this invention. Detailed Implementation

[0022] Example 1: Reference Figures 1-2 A computer vision-based borehole positioning method includes: Acquire the target drilling image and the standard drilling template image of the workpiece to be processed. Perform camera distortion correction, scale normalization and brightness equalization on the target drilling image and the standard drilling template image respectively to obtain the target preprocessed image and the template preprocessed image. The target preprocessed image is input into the SAM2 hole edge segmentation model based on the zero-shot semantic segmentation mechanism to obtain the target hole edge prior mask. The target preprocessed image is then subjected to mask constraint cropping based on the target hole edge prior mask to remove strong reflective areas, scratch areas, oil stain areas and repetitive texture background areas outside the mask, thereby generating the target hole edge constraint image. The template preprocessed image is input into the SAM2 hole edge segmentation model to obtain the template hole edge prior mask, and the template hole edge constraint image is generated based on the template hole edge prior mask. The same feature extraction network is used to extract features from the target hole edge constraint image and the template hole edge constraint image respectively to obtain the target key point set, the target descriptor set, the template key point set, and the template descriptor set. For each key point in the target key point set and the template key point set, calculate the minimum radial geometric distance to the corresponding prior mask boundary curve of the hole edge. Then, inject the minimum radial geometric distance and the two-dimensional coordinates of the key point into the target descriptor subset and the template descriptor subset through a unified learnable three-dimensional position encoding function to form a geometrically aware descriptor subset. The geometrically aware descriptor set is input into the improved LightGlue feature matching network, and self-attention calculation, cross-attention calculation and matching confidence prediction are performed in sequence to obtain the first round of feature matching pair set and the corresponding average matching confidence. The first-round feature matching pair set is fitted with a circle or ellipse, the fitting residual is calculated, and a circle / ellipse consistency convergence index is constructed based on the average matching confidence and the fitting residual. When the circle / ellipse consistency convergence index meets the preset threshold, adaptive early stopping is triggered, subsequent attention iterations are stopped, and the set of feature matching pairs with geometric consistency of the hole edge is output. If the preset threshold is not met, the improved LightGlue feature matching network is returned as input to continue the iteration. Based on the set of geometrically consistent feature matching pairs, a geometric mapping relationship is established between the target hole edge constraint image and the template hole edge constraint image. Feature matching pairs that do not meet the geometric consistency are eliminated to obtain the final effective feature matching pair set. Based on the final effective feature matching pair set, sub-pixel level contour reconstruction is performed on the borehole edge in the target hole edge constraint image, and circular or elliptical fitting is performed to obtain the pixel coordinates of the borehole center in the target preprocessed image. Using the pre-calibrated camera calibration parameters and hand-eye calibration parameters, the pixel coordinates are mapped to the coordinate system of the processing equipment to obtain the spatial positioning coordinates of the drilling center in the coordinate system of the processing equipment. Based on the spatial positioning coordinates, feed positioning commands are generated for the CNC machine tool spindle, the end effector of the industrial robot, or the automated drilling machine to achieve drilling center positioning.

[0023] In this embodiment, the target preprocessed image is input into the SAM2 aperture edge segmentation model based on the zero-shot semantic segmentation mechanism, including: establishing a pixel coordinate domain for the target preprocessed image; In Example 1, the pixel coordinate domain of the target preprocessed image consists of all pixel coordinates in the target preprocessed image whose horizontal pixel coordinates are within the pixel width range of the target preprocessed image and whose vertical pixel coordinates are within the pixel height range of the target preprocessed image. The preprocessed target image is input into the SAM2 pore edge segmentation model based on zero-shot semantic segmentation mechanism to generate a target pore edge probability map; Construct a priori mask for the target aperture edge based on the target aperture edge probability map; In Example 1, the target aperture edge probability value corresponding to each pixel coordinate in the target aperture edge probability map is compared with the segmentation threshold: When the probability value of the target hole edge is greater than or equal to the segmentation threshold, the pixel coordinates in the target preprocessed image are retained as the target hole edge region; when the probability value of the target hole edge is less than the segmentation threshold, the pixel coordinates in the target preprocessed image are defined as the region outside the mask, thus obtaining the target hole edge prior mask.

[0024] Based on the prior mask of the target aperture edge and the pixel coordinate domain, generate a set of pixel coordinates of the boundary of the prior mask of the target aperture edge; In Example 1, neighborhood determination is performed on each pixel coordinate in the pixel coordinate domain of the target preprocessed image to obtain the set of pixel coordinates of the target hole edge prior mask boundary.

[0025] The neighborhood determination specifically includes: determining whether the current pixel coordinates are retained as the target aperture edge region in the target aperture edge prior mask, and determining whether there are any neighboring pixel coordinates that are designated as the area outside the mask in the set of eight neighboring pixel coordinates of the current pixel coordinates; when the current pixel coordinates are retained as the target aperture edge region in the target aperture edge prior mask and there is at least one neighboring pixel coordinate that is designated as the area outside the mask in the set of eight neighboring pixel coordinates of the current pixel coordinates, the current pixel coordinates are determined as the boundary pixel coordinates of the target aperture edge region, and all the extracted boundary pixel coordinates of the target aperture edge region are combined to generate the set of boundary pixel coordinates of the target aperture edge prior mask.

[0026] Based on the set of pixel coordinates of the prior mask boundary of the target aperture, construct the target aperture boundary extension region; In Example 1, the Euclidean distance from the pixel coordinates to each boundary pixel coordinate in the set of pixel coordinates of the target aperture prior mask boundary is calculated, and the smallest Euclidean distance is selected as the nearest boundary distance value. When the nearest boundary distance value is less than or equal to the pixel outward expansion of the target aperture boundary extension region relative to the set of pixel coordinates of the target aperture prior mask boundary, the corresponding pixel coordinates are assigned to the target aperture boundary extension region. The target aperture boundary extension region represents the aperture edge neighborhood range formed around the set of pixel coordinates of the target aperture prior mask boundary, including the inner edge of the aperture and the outer metal surface of the aperture.

[0027] Based on the prior mask of the target aperture edge and the expansion region of the target aperture edge boundary, the effective constraint region of the target aperture edge is determined; In Example 1, it is determined whether the current pixel coordinates belong to the target hole edge boundary extension region. If the current pixel coordinates belong to the target hole edge boundary extension region, the current pixel coordinates are included in the target hole edge effective constraint region.

[0028] Based on the effective constraint region of the target aperture edge, determine the set of pixel coordinates outside the mask; In Example 1, it is determined whether the pixel coordinates belong to the effective constraint area of ​​the target hole edge. If the pixel coordinates do not belong to the effective constraint area of ​​the target hole edge, the pixel coordinates are included in the set of pixel coordinates outside the mask.

[0029] The set of pixel coordinates outside the mask represents the set of pixel coordinates in the target preprocessed image that are excluded by the effective constraint area of ​​the target hole edge. The pixel coordinates in the set of pixel coordinates outside the mask correspond to at least one of the strong reflective area, scratch area, oil stain area and repetitive texture background area. Based on the effective constraint region of the target aperture edge and the set of pixel coordinates outside the mask, the target preprocessed image is cropped by mask constraint to generate the target aperture edge constraint image; In Example 1, if the current pixel coordinates belong to the effective constraint area of ​​the target hole edge, the pixel value of the current pixel coordinates in the target preprocessed image is retained as the pixel value of the same pixel coordinates in the target hole edge constraint image.

[0030] If the current pixel coordinates belong to the set of pixel coordinates outside the mask, then the pixel values ​​of the same pixel coordinates in the target hole edge constraint image are set to zero pixel values ​​consistent with the channel dimension of the target preprocessed image.

[0031] In this embodiment, the SAM2 aperture edge segmentation model based on the zero-sample semantic segmentation mechanism includes: The target preprocessed image is semantically encoded using an image encoder to obtain the target image semantic feature map; In Example 1, the target preprocessed image is divided into several target image blocks according to a preset image block size. Linear embedding processing is performed on each target image block to obtain target image block embedding features. The target image block embedding features are fused with the corresponding two-dimensional position embedding features to obtain target image encoded input features.

[0032] The target image is encoded into a multi-layer self-attention encoding layer in the feature input image encoder. The global contextual association between target image blocks is calculated layer by layer to obtain the global semantic features of the target image. Feature rearrangement and spatial scale restoration are performed on the global semantic features of the target image to obtain the semantic feature map of the target image. The semantic feature map of the target image represents the semantic differences between the hole edge region, strong reflective region, scratch region, oil stain region and repeated texture background region in the target preprocessed image.

[0033] Based on the semantic feature map of the target image, a set of candidate clues for the edge of the target hole is generated in the pixel coordinate domain of the preprocessed target image; In Example 1, the semantic feature map of the target image is mapped to the pixel coordinate domain of the target preprocessed image to obtain the semantic response map of the target aperture edge. The response value of the semantic response map of the target aperture edge is filtered, and the pixel coordinates that meet the preset aperture edge response threshold are retained to obtain the set of candidate pixel coordinates of the target aperture edge.

[0034] The set of candidate pixel coordinates for the target aperture edge is divided into connected regions to obtain several candidate connected regions for the target aperture edge. For each candidate connected region for the target aperture edge, the corresponding minimum bounding rectangle is determined and used as the candidate bounding box for the candidate connected region for the target aperture edge. For each candidate connected region for the target aperture edge, the corresponding geometric center is determined and used as the candidate center point for the candidate connected region for the target aperture edge. The candidate bounding box and candidate center point for each candidate connected region for the target aperture edge are combined into a candidate cue for the target aperture edge. All candidate cue for the target aperture edge constitute the set of candidate cue for the target aperture edge.

[0035] The target aperture edge cue embedding feature set is obtained by encoding the cue features of the candidate cue set of the target aperture edge by the cue encoder; By performing joint mask decoding on the semantic feature map of the target image and the embedded feature set of the target aperture edge cue, a set of candidate aperture edge masks for the target is obtained. In Example 1, each target aperture edge hint embedding feature in the target aperture edge hint embedding feature set is subjected to cross-feature attention interaction with the semantic feature map of the target image to obtain the aperture edge response feature map corresponding to each target aperture edge candidate hint. Feature channel compression and spatial resolution restoration are performed on each aperture edge response feature map to obtain the initial aperture edge response map corresponding to each target aperture edge candidate hint. Pixel-by-pixel aperture edge response prediction is performed on each initial aperture edge response map to obtain the target candidate aperture edge mask corresponding to each target aperture edge candidate hint.

[0036] The candidate mask filter calculates the confidence score of each target candidate aperture edge mask and the semantic feature map of the target image, and the target candidate aperture edge mask with the highest confidence score is determined as the target aperture edge probability map.

[0037] In this embodiment, inputting the template preprocessed image into the SAM2 hole edge segmentation model includes: establishing the template pixel coordinate domain for the template preprocessed image; In Example 1, the template pixel coordinate domain consists of all template pixel coordinates in the template preprocessed image whose horizontal pixel coordinates are within the pixel width range of the template preprocessed image and whose vertical pixel coordinates are within the pixel height range of the template preprocessed image.

[0038] Input the preprocessed template image into the SAM2 aperture edge segmentation model to generate a template aperture edge probability map; In Example 1, the template aperture probability map is obtained by zero-sample semantic segmentation of the template preprocessed image using the SAM2 aperture segmentation model. Each template pixel coordinate in the template aperture probability map corresponds to a template aperture probability value, which is used to represent the probability that the corresponding template pixel coordinate in the template preprocessed image belongs to the template aperture region.

[0039] Construct a priori mask for template aperture edges based on the template aperture edge probability map; In Example 1, the template hole edge probability value corresponding to each template pixel coordinate in the template hole edge probability map is compared with the segmentation threshold. When the template hole edge probability value is greater than or equal to the segmentation threshold, the template pixel coordinates in the template preprocessing image are retained as the template hole edge region. When the template hole edge probability value is less than the segmentation threshold, the template pixel coordinates in the template preprocessing image are defined as the region outside the template mask, thus obtaining the template hole edge prior mask.

[0040] Based on the template aperture prior mask and the template pixel coordinate domain, generate a set of template aperture prior mask boundary pixel coordinates; In Example 1, it is determined whether the current template pixel coordinates are retained as the template hole edge region in the template hole edge prior mask, and it is determined whether there are any neighboring template pixel coordinates that are defined as the region outside the template mask in the set of eight neighboring pixel coordinates of the current template pixel coordinates.

[0041] When the current template pixel coordinates are preserved as the template hole edge region in the template hole edge prior mask and there is at least one neighboring template pixel coordinate in the set of eight neighboring pixel coordinates of the current template pixel coordinates that is defined as the region outside the template mask, the current template pixel coordinates are determined as the template hole edge region boundary pixel coordinates. All extracted template hole edge region boundary pixel coordinates are combined to generate the template hole edge prior mask boundary pixel coordinate set.

[0042] Based on the set of pixel coordinates of the prior mask boundary of the template aperture, the extended region of the template aperture boundary is constructed; Based on the prior mask of the template hole edge and the extension region of the template hole edge boundary, the effective constraint region of the template hole edge is determined; In Example 1, it is determined whether the current template pixel coordinates belong to the template hole edge boundary extension region. If the current template pixel coordinates belong to the template hole edge boundary extension region, the current template pixel coordinates are included in the template hole edge effective constraint region. The template hole edge effective constraint region represents the range of template pixel coordinates in the template preprocessing image that are allowed to participate in the generation of the template hole edge constraint image and subsequent feature extraction.

[0043] Based on the effective constraint area of ​​the template hole edge, determine the set of pixel coordinates outside the template mask; In Example 1, it is determined whether the template pixel coordinates belong to the effective constraint area of ​​the template hole edge. If the template pixel coordinates do not belong to the effective constraint area of ​​the template hole edge, the template pixel coordinates are included in the set of pixel coordinates outside the template mask.

[0044] Based on the effective constraint region of the template hole edge and the set of pixel coordinates outside the template mask, the template preprocessed image is mask-constrained and cropped to generate the template hole edge constraint image; The same feature extraction network is used to extract features from the target hole edge constraint image and the template hole edge constraint image respectively, resulting in a target key point set, a target descriptor set, a template key point set, and a template descriptor set; In Example 1, convolutional feature response calculation is performed on the target hole edge constraint image to generate a target hole edge feature response map. Pixel coordinates that meet the key point response threshold are selected from the target hole edge feature response map as target key points. Target descriptors corresponding one-to-one with the target key points are generated according to the local neighborhood features of the target key points in the target hole edge constraint image. All target key points are combined to form a target key point set, and all target descriptors are combined to form a target descriptor set.

[0045] Convolutional feature response calculation is performed on the template hole edge constraint image to generate a template hole edge feature response map. Pixel coordinates that meet the key point response threshold are selected from the template hole edge feature response map as template key points. Template descriptors corresponding one-to-one with the template key points are generated according to the local neighborhood features of the template key points in the template hole edge constraint image. All template key points are combined to form a template key point set, and all template descriptors are combined to form a template descriptor set.

[0046] In this embodiment, the minimum radial geometric distance from each key point in the target key point set and the template key point set to the corresponding hole edge prior mask boundary curve is calculated, including: based on the target hole edge prior mask boundary pixel coordinate set, the minimum radial geometric distance of the target corresponding to each target key point in the target key point set is calculated; Based on the set of pixel coordinates of the prior mask boundary at the template aperture edge, calculate the minimum radial geometric distance of the template corresponding to each template key point in the template key point set; The two-dimensional pixel coordinates of each target key point in the target hole edge constraint image are combined with the corresponding target minimum radial geometric distance to form the target three-dimensional geometric position vector corresponding to each target key point; In Example 1, the target three-dimensional geometric position vector is composed of the horizontal pixel coordinates of the target key point in the target hole edge constraint image, the vertical pixel coordinates of the target key point in the target hole edge constraint image, and the target minimum radial geometric distance corresponding to the target key point. It is used to represent the planar position of the target key point in the target hole edge constraint image and the radial geometric position relative to the target hole edge boundary.

[0047] The two-dimensional pixel coordinates of each template key point in the template hole edge constraint image are combined with the corresponding minimum radial geometric distance of the template to form the template three-dimensional geometric position vector corresponding to each template key point. Input the target's 3D geometric position vector and the template's 3D geometric position vector into the 3D position encoding function to generate a target geometric position encoding vector and a template geometric position encoding vector that are consistent with the dimensions of the target descriptor and the template descriptor, respectively. In Example 1, the unified learnable 3D position coding function is composed of a 3D geometry input layer, a normalized mapping layer, a learnable linear projection layer, a nonlinear activation layer, and a descriptor dimension alignment layer connected in sequence.

[0048] The 3D geometry input layer receives the horizontal pixel coordinates, vertical pixel coordinates, and minimum radial geometric distance of the key points. The normalization mapping layer maps the horizontal pixel coordinates, vertical pixel coordinates, and minimum radial geometric distance to a uniform numerical scale. The learnable linear projection layer maps the normalized 3D geometry input into intermediate position features. The nonlinear activation layer enhances the ability of the intermediate position features to express the radial geometric changes of the hole edge. The descriptor dimension alignment layer converts the intermediate position features into a position encoding vector consistent with the dimensions of the target descriptor and the template descriptor.

[0049] During the borehole center localization inference stage, the unified learnable 3D position encoding function after training is invoked to convert each target 3D geometric position vector into the corresponding target geometric position encoding vector, and each template 3D geometric position vector into the corresponding template geometric position encoding vector.

[0050] The target geometric position encoding vector and the target descriptor have the same vector dimension, and the template geometric position encoding vector and the template descriptor have the same vector dimension.

[0051] The target geometric position encoding vector is injected into the target descriptor, and the template geometric position encoding vector is injected into the template descriptor to generate the target geometric perception descriptor and the template geometric perception descriptor, respectively. In Example 1, the target geometric position encoding vector and the corresponding target descriptor are fused in the same dimension to obtain the target geometric perception descriptor, which is used to simultaneously represent the local hole edge texture features, two-dimensional pixel coordinate features, and radial geometric features relative to the target hole edge boundary of the corresponding target key point.

[0052] The template geometric position encoding vector is fused with the corresponding template descriptor using vectors of the same dimension to obtain the template geometric perception descriptor, which is used to simultaneously represent the local hole edge texture features, two-dimensional pixel coordinate features, and radial geometric features relative to the template hole edge boundary of the corresponding template key point.

[0053] All target geometry-aware descriptors are combined to form a target geometry-aware descriptor set, and all template geometry-aware descriptors are combined to form a template geometry-aware descriptor set. The target geometry-aware descriptor set and the template geometry-aware descriptor set together constitute the geometry-aware descriptor set.

[0054] In this embodiment, the geometric perception descriptor set is input into the improved LightGlue feature matching network, including: inputting the geometric perception descriptor set into the improved LightGlue feature matching network, and performing matrix representation on the target geometric perception descriptor set and the template geometric perception descriptor set to obtain the first round target input feature matrix and the first round template input feature matrix; In Example 1, all target geometric perception descriptors are arranged row by row according to the order of target key points to obtain the first round target input feature matrix, and all template geometric perception descriptors are arranged row by row according to the order of template key points to obtain the first round template input feature matrix.

[0055] Self-attention calculations are performed on the first-round target input feature matrix and the first-round template input feature matrix respectively to obtain the first-round target self-attention feature matrix and the first-round template self-attention feature matrix; In Example 1, the first-round target input feature matrix is ​​processed through self-attention query projection, self-attention key projection, and self-attention value projection to obtain the target self-attention query matrix, target self-attention key matrix, and target self-attention value matrix. The intra-keypoint correlation is calculated on the target self-attention query matrix and the target self-attention key matrix, and the results of the intra-keypoint correlation calculation are normalized to obtain the target self-attention weight matrix. The target self-attention weight matrix is ​​applied to the target self-attention value matrix, and residual fusion is performed with the first-round target input feature matrix to obtain the first-round target self-attention feature matrix.

[0056] The calculation method for the first round template self-attention feature matrix is ​​the same as that for the first round target self-attention feature matrix.

[0057] Cross-attention calculation is performed based on the first round target self-attention feature matrix and the first round template self-attention feature matrix to obtain the first round target cross-attention feature matrix and the first round template cross-attention feature matrix; In Example 1, the first-round target self-attention feature matrix is ​​subjected to cross-attention query projection, cross-attention key projection, and cross-attention value projection to obtain the target cross-attention query matrix, target cross-attention key matrix, and target cross-attention value matrix. Simultaneously, the first-round template self-attention feature matrix is ​​subjected to cross-attention query projection, cross-attention key projection, and cross-attention value projection to obtain the template cross-attention query matrix, template cross-attention key matrix, and template cross-attention value matrix. Cross-image keypoint correlation calculation is performed on the target cross-attention query matrix and the template cross-attention key matrix, and the cross-image keypoint correlation calculation results are normalized to obtain the cross-attention weight matrix of template side features to target side features. The cross-attention weight matrix of template side features to target side features is applied to the template cross-attention value matrix and residual fusion is performed with the first-round target self-attention feature matrix to obtain the first-round target cross-attention feature matrix.

[0058] The first round of template cross-attention feature matrix is ​​calculated by exchanging the query, key, and value matrices of the target side and the template side and calculating them symmetrically according to the above logic.

[0059] Based on the first round target cross-attention feature matrix and the first round template cross-attention feature matrix, calculate the first round matching similarity between target key points and template key points; In Example 1, the target cross-attention feature vector corresponding to each target key point is extracted from the first round target cross-attention feature matrix, and the template cross-attention feature vector corresponding to each template key point is extracted from the first round template cross-attention feature matrix. Vector similarity is calculated for any target cross-attention feature vector and any template cross-attention feature vector to obtain the first round matching similarity between the corresponding target key point and the corresponding template key point. The first round matching similarity is used to measure the degree of matching between the target key point and the template key point in the hole edge geometric perception description features.

[0060] Perform matching confidence prediction on the first round of matching similarity to obtain the first round of matching confidence matrix, and determine the first round of candidate matching relationships based on the first round of matching confidence matrix; In Example 1, each first-round matching similarity is input into the matching confidence prediction layer. Through similarity scaling, confidence biasing, and bidirectional normalization, the first-round matching confidence between the corresponding target key point and the corresponding template key point is obtained. All first-round matching confidences are arranged according to the order of the target key points and the order of the template key points to obtain the first-round matching confidence matrix.

[0061] For each target keypoint, select the template keypoint corresponding to the target keypoint and with the highest confidence in the first round of matching from the first round of matching confidence matrix. This template keypoint is selected as the first round candidate template keypoint corresponding to the target keypoint in the template keypoint set. Similarly, select the target keypoint corresponding to the template keypoint and with the highest confidence in the first round of matching from the first round of matching confidence matrix. This target keypoint is selected as the first round candidate target keypoint corresponding to the template keypoint in the target keypoint set. The first round candidate matching relationship is determined by all the first round candidate template keypoints and all the first round candidate target keypoints.

[0062] Based on the first round of candidate matching relationships, bidirectional mutual selection and confidence threshold determination are performed on any target key point and any template key point to generate the first round of feature matching pair set; In Example 1, a two-way mutual selection and confidence threshold determination are performed on any target key point and any template key point: Determine whether the corresponding template key point is a first-round candidate template key point of the target key point, and determine whether the target key point is a first-round candidate target key point of the template key point.

[0063] The confidence threshold determination includes: determining whether the confidence of the first-round matching between the target key point and the template key point is greater than or equal to the matching confidence threshold.

[0064] When both the target keypoint and the template keypoint satisfy the two-way mutual selection judgment and the confidence threshold judgment, the target keypoint, the template keypoint, and the corresponding first-round matching confidence are combined into the first-round feature matching pair.

[0065] The matching confidence threshold is determined based on the first-round matching confidence distribution of correct and incorrect matching pairs in the validation sample set. The quantile that optimizes the combined correct match recall rate and incorrect match rejection rate is selected as the matching confidence threshold. In Example 1, the matching confidence threshold is set to 0.75.

[0066] Calculate the average matching confidence score based on the first-round matching confidence score of each first-round feature matching pair in the first-round feature matching pair set.

[0067] In Example 1, when the first round feature matching pair set is not empty, the first round matching confidence scores corresponding to all first round feature matching pairs in the first round feature matching pair set are summed, and the summation result is divided by the number of matching pairs in the first round feature matching pair set to obtain the average matching confidence score. When the first round feature matching pair set is empty, the average matching confidence score is set to zero.

[0068] In this embodiment, the first round of feature matching pairs is fitted with a circle or ellipse, the fitting residual is calculated, and a circle / ellipse consistency convergence index is constructed based on the average matching confidence and the fitting residual. This includes: taking the first round of feature matching pairs as the current round of feature matching pairs and establishing the round expression of the current round of feature matching pairs. In Example 1, the current round feature matching pair set consists of multiple feature matching pairs. Each feature matching pair includes a target key point, a template key point, and a corresponding matching confidence. The number of matching pairs in the current round feature matching pair set represents the number of hole edge matching points participating in the circular or elliptical fitting of the current round.

[0069] Based on the current set of feature matching pairs, extract the set of coordinates of key matching points on the target hole edge; In Example 1, the horizontal and vertical pixel coordinates of the corresponding target key points in the target hole edge constraint image are extracted from each feature matching pair in the current wheel feature matching pair set. All extracted horizontal and vertical pixel coordinates are combined to obtain the target hole edge matching key point coordinate set, which represents the hole edge spatial distribution of the current wheel feature matching pair set in the target hole edge constraint image.

[0070] Perform circular fitting on the set of key point coordinates for matching the target hole edge to obtain the current wheel's circular fitting parameters and the current wheel's circular fitting residual; In Example 1, the horizontal and vertical pixel coordinates of each target key point are extracted from the target hole edge matching key point coordinate set. The horizontal pixel coordinates of the circle center, the vertical pixel coordinates of the circle center, and the circle radius are used as variables to be solved to establish the distance deviation relationship between the target hole edge matching key point coordinate set and the circle to be fitted.

[0071] For each target key point, calculate the pixel distance from the target key point to the center of the circle to be fitted, calculate the difference between the pixel distance and the radius of the circle to be fitted, and obtain the circle distance deviation value corresponding to the target key point. Sum the squares of the circle distance deviation values ​​corresponding to all target key points in the target hole edge matching key point coordinate set to obtain the total circle fitting deviation value.

[0072] Adjust the horizontal pixel coordinates of the circle's center, the vertical pixel coordinates of the circle's center, and the circle's radius to minimize the total circle fitting deviation. Then, determine the horizontal pixel coordinates of the circle's center at this point as the horizontal pixel coordinates of the current round's circle fitting center, determine the vertical pixel coordinates of the circle's center at this point as the vertical pixel coordinates of the current round's circle fitting center, and determine the circle's radius at this point as the current round's circle fitting radius. The horizontal pixel coordinates of the current round's circle fitting center, the vertical pixel coordinates of the current round's circle fitting center, and the current round's circle fitting radius together constitute the current round's circle fitting parameters.

[0073] After obtaining the current wheel's circular fitting parameters, calculate the current wheel's circular fitting residual, which measures the deviation of the target key points in the current wheel's feature matching pair set from the circular hole edge structure. ; in, Indicates the first The circular fitting residual is used to measure the first... The degree of normalized deviation of the target key points in the wheel feature matching set relative to the circular hole edge structure. Indicates the first The first round of feature matching pairs in the set The horizontal pixel coordinates of each target key point in the target hole edge constraint image. Indicates the first The first round of feature matching pairs in the set The vertical pixel coordinates of each target key point in the target hole edge constraint image. Indicates the first The horizontal pixel coordinates of the fitted center of the circular shape. Indicates the first The vertical pixel coordinates of the fitted center of the circular shape. Indicates the first The fitted radius of the wheel circle. This represents the pixel scale constant used to avoid a denominator of zero.

[0074] Elliptical fitting is performed on the set of coordinates of key points for matching the target hole edge to obtain the elliptical fitting parameters and residuals of the current wheel. In Example 1, the horizontal pixel coordinates, vertical pixel coordinates, major axis length, minor axis length, and principal axis rotation angle of the current wheel ellipse fitting center are used as the parameters to be solved. The coordinate rotation and translation parameter equations for the transformation of each target key point in the target hole edge matching key point coordinate set to the ellipse principal axis coordinate system are established.

[0075] Substitute each target key point in the set of target hole edge matching key point coordinates into the coordinate rotation and translation parameter equation to obtain the horizontal and vertical rotation coordinates of the target key points with respect to the parameters to be solved.

[0076] With the objective of minimizing the sum of squared algebraic deviations of each target key point in the target hole edge matching key point coordinate set relative to the ellipse to be fitted, the following parameters are obtained by jointly solving for the horizontal pixel coordinates of the current wheel ellipse fitting center, the vertical pixel coordinates of the current wheel ellipse fitting center, the length of the major semi-axis of the current wheel ellipse fitting, the length of the minor semi-axis of the current wheel ellipse fitting, and the rotation angle of the principal axis of the current wheel ellipse fitting: ; in, Indicates the first Ellipse fitting parameters, Indicates the current feature matching iteration round. This represents the parameter calculation operation that minimizes the subsequent objective function. Indicates the first The horizontal pixel coordinates of the fitting center of the wheel ellipse. Indicates the first The vertical pixel coordinates of the fitting center of the wheel ellipse. Indicates the first The length of the major semi-axis of the wheel is fitted to an ellipse. Indicates the first The wheel ellipse fits the length of the shorter semi-axis. Indicates the first The wheel ellipse is fitted with the principal axis rotation angle. Indicates the first The target key point number in the set of key point coordinates for matching the wheel target hole edge. Indicates the first The number of matching pairs in the round feature matching pair set. Indicates the first Round The lateral rotation coordinates of each target key point relative to the principal axis of the ellipse. Indicates the first Round The longitudinal rotation coordinates of each target key point relative to the principal axis of the ellipse.

[0077] Based on the current wheel ellipse fitting parameters obtained from the solution, the target key points are transformed to the ellipse principal axis coordinate system to obtain the actual horizontal rotation coordinates and actual vertical rotation coordinates corresponding to the target key points.

[0078] Based on the actual lateral rotation coordinates, actual longitudinal rotation coordinates, the length of the major semi-axis of the current wheel's elliptical fitting, and the length of the minor semi-axis of the current wheel's elliptical fitting, the elliptical normalized deviation value corresponding to the target key point is calculated. The absolute values ​​of the elliptical normalized deviation values ​​corresponding to all target key points are averaged to obtain the current wheel's elliptical fitting residual, which is used to measure the degree of deviation of the target key points in the current wheel's feature matching pair set from the elliptical hole edge structure. ; in, This represents the residual of the elliptical fit.

[0079] Determine the fitting residual of the current wheel based on the circular fitting residual or the elliptical fitting residual of the current wheel; In Example 1, when the target hole edge is fitted with a circular shape, the circular fitting residual of the current wheel is determined as the fitting residual of the current wheel; when the target hole edge is fitted with an elliptical shape, the elliptical fitting residual of the current wheel is determined as the fitting residual of the current wheel.

[0080] S66. Calculate the average matching confidence of the current round based on the matching confidence of each feature matching pair in the current round feature matching pair set; In Example 1, when the current round feature matching pair set is not empty, the matching confidence scores of all feature matching pairs in the current round feature matching pair set are summed, and the summation result is divided by the number of matching pairs in the current round feature matching pair set to obtain the average matching confidence score of the current round; when the current round feature matching pair set is empty, the average matching confidence score of the current round is set to zero.

[0081] The weighted sum of the average confidence score of the current round and the fitting residual of the current round is used to obtain the circle / ellipse consistency convergence index of the current round. In Example 1, the average confidence level of the current round is subtracted from the constant 1 to obtain the complementary term of the average confidence level of the current round. The complementary term of the average confidence level of the current round is then weighted and fused with the fitting residual of the current round to obtain the circle / ellipse consistency convergence index of the current round.

[0082] When the current wheel circle / ellipse consistency convergence index is less than or equal to the preset threshold, it is determined that the current wheel feature matching pair set satisfies the hole edge geometric consistency convergence condition, triggering adaptive early stopping, stopping attention iteration, and outputting the current wheel feature matching pair set as the hole edge geometric consistency feature matching pair set; When the convergence index of the current wheel circle / ellipse is greater than the preset threshold, it is determined that the current wheel feature matching pair set does not meet the convergence condition of hole edge geometry consistency. The target cross attention feature matrix and the template cross attention feature matrix of the current wheel are then input into the next attention layer of the improved LightGlue feature matching network to continue iterating.

[0083] In Example 1, the preset threshold is determined based on the circular / elliptical consistency convergence index distribution of correct and incorrect hole edge matching samples in the verification sample set. The quantile that optimizes the retention rate of correct hole edge matching and the rejection rate of incorrect hole edge matching is selected as the preset threshold. In Example 1, the preset threshold is set to 0.18.

[0084] The improved LightGlue feature matching network in this embodiment is defined as a aperture-edge constrained feature matching network that takes a target geometry-aware descriptor set and a template geometry-aware descriptor set as inputs. The inputs include a first-round target input feature matrix and a first-round template input feature matrix. The dimension of the first-round target input feature matrix is... The dimension of the first round template input feature matrix is ,in Indicates the number of key points of the target. Indicates the number of key points in the template. The descriptor dimension is represented; the improved LightGlue feature matching network structure includes a 3D position encoding input layer, a self-attention layer, a cross-attention layer, a matching confidence prediction layer, a bidirectional mutual selection decision layer, and a circle / ellipse consistency adaptive early stopping layer. The self-attention layer and the cross-attention layer are stacked alternately in the LightGlue manner to form 9 attention matching layers. Each attention matching layer sequentially completes the self-attention feature enhancement of the target side and the template side, the cross-attention information interaction between the target side and the template side, and the matching confidence prediction, outputting the first round of feature matching pair set and the corresponding average matching confidence, and outputting the hole edge geometric consistency feature matching pair set in subsequent iterations.

[0085] Compared to existing LightGlue feature matching networks, this method inputs the minimum radial geometric distance between the target keypoints and template keypoints and the corresponding prior mask boundary at the aperture edge, along with the 2D pixel coordinates, into a unified learnable 3D position encoding function to generate target geometric position encoding vectors and template geometric position encoding vectors. These vectors are then injected into the target descriptor and template descriptor, respectively, so that the descriptor simultaneously contains local texture features, 2D spatial position features, and aperture edge radial geometric features. The average matching confidence score, together with the circular or elliptical fitting residuals, constitutes a circular / elliptical consistency convergence index, which is used to replace early stopping decisions that rely solely on matching confidence score. The training method is as follows: a training sample set containing target hole edge constraint images, template hole edge constraint images, and correctly labeled matching relationships is used to jointly train a unified learnable 3D position encoding function, self-attention projection parameters, cross-attention projection parameters, and matching confidence prediction parameters. The optimization objectives are to increase the confidence of correct matching pairs, decrease the confidence of incorrect matching pairs, and reduce the fitting residual of circular or elliptical hole edges. This enables the network to focus matching calculations on the real hole edge area even under backgrounds of strong reflection, scratches, oil stains, and repetitive textures, thereby improving the stability, anti-interference ability, and sub-pixel fitting accuracy of borehole center positioning.

[0086] In this embodiment, an image mapping is established based on the set of matching pairs of geometrically consistent hole edge features. After removing abnormal matches, the sub-pixel hole edge is reconstructed and fitted. This includes: extracting the coordinate pairs of matching key points between the target hole edge constraint image and the template hole edge constraint image based on the set of matching pairs of geometrically consistent hole edge features. In Example 1, the horizontal and vertical pixel coordinates of the target key points in the target hole edge constraint image are extracted from each hole edge geometric consistency feature matching pair, and the horizontal and vertical pixel coordinates of the template key points in the template hole edge constraint image are extracted. The horizontal and vertical pixel coordinates of the target key points in the target hole edge constraint image, the horizontal and vertical pixel coordinates of the template key points in the template hole edge constraint image, and the corresponding matching confidence are combined to obtain the matching key point coordinate pairs.

[0087] Based on the matching key point coordinate pairs, the geometric mapping relationship between the target hole edge constraint image and the template hole edge constraint image is established, and the hole edge homography geometric mapping matrix is ​​obtained by solving. In Example 1, the horizontal and vertical pixel coordinates of the template key point in each matching key point coordinate pair in the template hole edge constraint image are used as the coordinates before mapping, and the horizontal and vertical pixel coordinates of the target key point in the same matching key point coordinate pair in the target hole edge constraint image are used as the coordinates after mapping. The planar perspective coordinate correspondence between the template hole edge constraint image and the target hole edge constraint image is established by the coordinates before mapping and the coordinates after mapping.

[0088] For each pair of matching key point coordinates, establish a set of projection constraints on the positions of the template key points that should fall into the corresponding target key point positions after transformation by the homography geometric mapping matrix to be solved. Combine the projection constraints corresponding to all pairs of matching key point coordinates to form a set of hole edge geometric mapping equations.

[0089] When the number of matching keypoint coordinate pairs meets the minimum number required to solve the homography geometric mapping matrix (minimum number is 4), the hole edge geometric mapping equations are solved by matrix calculation to obtain the initial hole edge homography geometric mapping matrix. When the number of matching keypoint coordinate pairs exceeds the minimum number required to solve the homography geometric mapping matrix, the initial hole edge homography geometric mapping matrix is ​​optimized by least squares with the objective of minimizing the sum of the squared pixel distances between the mapped pixel coordinates of all template keypoints after projection onto the initial hole edge homography geometric mapping matrix and the horizontal and vertical pixel coordinates of the corresponding target keypoints in the target hole edge constraint image.

[0090] Based on the aperture edge geometry mapping matrix, calculate the mapping reprojection error corresponding to each aperture edge geometry consistent feature matching pair; In Example 1, the template key points in each hole edge geometric consistency feature matching pair are projected onto the target hole edge constraint image according to the hole edge geometric mapping matrix to obtain the mapped horizontal pixel coordinates and mapped vertical pixel coordinates of the template key points in the target hole edge constraint image.

[0091] The pixel distance between the mapped horizontal and vertical pixel coordinates of the template key points in the target hole edge constraint image and the corresponding horizontal and vertical pixel coordinates of the target key points in the target hole edge constraint image is calculated to obtain the mapping reprojection error corresponding to the hole edge geometric consistency feature matching pair.

[0092] Based on the mapping reprojection error, feature matching pairs that do not meet geometric consistency are eliminated to obtain the final set of valid feature matching pairs. In Example 1, the mapping reprojection error corresponding to each pair of geometrically consistent feature matching at the aperture edge is compared with a reprojection error threshold. When the mapping reprojection error is less than or equal to the reprojection error threshold, the corresponding pair of geometrically consistent feature matching at the aperture edge is retained as the final valid feature matching pair. When the mapping reprojection error is greater than the reprojection error threshold, the corresponding pair of geometrically consistent feature matching at the aperture edge is removed from the set of geometrically consistent feature matching pairs at the aperture edge. All the retained final valid feature matching pairs are combined to obtain the final valid feature matching pair set.

[0093] The reprojection error threshold is determined based on the mapping reprojection error distribution of correct and incorrect hole edge matching pairs in the verification sample set. The quantile that optimizes the retention rate of correct hole edge matching and the rejection rate of incorrect hole edge matching is selected as the reprojection error threshold. In Example 1, the reprojection error threshold is set to 1.5 pixels.

[0094] Based on the final set of effective feature matching pairs, extract the set of coordinates of effective target hole edge key points in the target hole edge constraint image; In Example 1, the horizontal and vertical pixel coordinates of the target key points in the target hole edge constraint image are extracted from each of the final effective feature matching pairs in the final effective feature matching pair set. The extracted horizontal and vertical pixel coordinates of the target key points in the target hole edge constraint image are determined as the effective target hole edge key point coordinates. All effective target hole edge key point coordinates are combined to obtain the effective target hole edge key point coordinate set.

[0095] Based on the effective set of key point coordinates of the target borehole edge, sub-pixel level contour reconstruction is performed on the borehole edge in the target borehole edge constraint image to obtain the sub-pixel contour point set of the target borehole. In Example 1, a local hole edge neighborhood of a preset size is selected in the target hole edge constraint image with the effective target hole edge key point coordinates as the center, and a local hole edge neighborhood coordinate set is obtained. For each local neighbor pixel coordinate in the local hole edge neighborhood coordinate set, its pixel gray value in the target hole edge constraint image is extracted, and the local horizontal relative coordinate and local vertical relative coordinate of the local neighbor pixel coordinate relative to the effective target hole edge key point coordinate are calculated.

[0096] Based on the pixel grayscale values ​​in the local aperture edge neighborhood coordinate set, the image gradient operator is used to calculate the grayscale change rate of each local neighbor pixel coordinate in the horizontal and vertical directions, obtaining the local horizontal gradient value and the local vertical gradient value. The square root of the square of the local horizontal gradient value and the local vertical gradient value is calculated to obtain the aperture edge grayscale response value corresponding to each local neighbor pixel coordinate. The local horizontal relative coordinates in the local aperture edge neighborhood coordinate set are weighted and averaged according to the corresponding aperture edge grayscale response value to obtain the horizontal sub-pixel offset corresponding to the effective target aperture edge key point coordinate. The local vertical relative coordinates in the local aperture edge neighborhood coordinate set are weighted and averaged according to the corresponding aperture edge grayscale response value to obtain the vertical sub-pixel offset corresponding to the effective target aperture edge key point coordinate.

[0097] The horizontal subpixel offset is superimposed on the horizontal pixel coordinates of the effective target hole edge key point coordinates, and the vertical subpixel offset is superimposed on the vertical pixel coordinates of the effective target hole edge key point coordinates to obtain the target borehole subpixel contour points corresponding to the effective target hole edge key point coordinates. Multiple target borehole subpixel contour points are calculated for all effective target hole edge key point coordinates. All target borehole subpixel contour points are combined to obtain the target borehole subpixel contour point set.

[0098] Perform circular fitting based on the set of sub-pixel contour points of the target borehole to obtain the center pixel coordinates of the circular borehole; In Example 1, when the set of sub-pixel contour points of the target borehole corresponds to a circular borehole edge structure, the horizontal pixel coordinates of the center of the circle, the vertical pixel coordinates of the center of the circle, and the radius of the circle are used as variables to be solved. The pixel distance from each sub-pixel contour point of the target borehole in the set of sub-pixel contour points to the center of the circle to be fitted is calculated, and the difference between the pixel distance and the radius of the circle to be fitted is calculated to obtain the circle fitting deviation value.

[0099] The total circular fitting deviation value is obtained by summing the squares of the circular fitting deviation values ​​corresponding to all sub-pixel contour points of the target borehole in the set of target borehole sub-pixel contour points. The horizontal pixel coordinates of the center of the circle, the vertical pixel coordinates of the center of the circle, and the radius of the circle are adjusted to minimize the total circular fitting deviation value. The horizontal pixel coordinates of the center of the circle and the vertical pixel coordinates of the center of the circle at this time are determined as the center pixel coordinates of the circular borehole.

[0100] Elliptical fitting is performed based on the set of sub-pixel contour points of the target borehole to obtain the center pixel coordinates of the elliptical borehole; In Example 1, when the target borehole sub-pixel contour point set corresponds to an elliptical borehole edge structure, the coordinate rotation and translation parameter equations for the transformation of the target borehole sub-pixel contour points to the elliptical principal axis coordinate system are established using the lateral pixel coordinates of the elliptical center, the longitudinal pixel coordinates of the elliptical center, the length of the major semi-axis of the ellipse, the length of the minor semi-axis of the ellipse, and the rotation angle of the elliptical principal axis as parameters to be solved.

[0101] Substituting each target borehole subpixel contour point in the target borehole subpixel contour point set into the coordinate rotation and translation parameter equation, we obtain the algebraic expressions of the horizontal rotation coordinates and the vertical rotation coordinates of the target borehole subpixel contour points with respect to the parameters to be solved.

[0102] With the goal of minimizing the sum of squared algebraic deviations of each sub-pixel contour point of the target borehole relative to the ellipse to be fitted, the horizontal pixel coordinates of the ellipse center, the vertical pixel coordinates of the ellipse center, the length of the major semi-axis of the ellipse, the length of the minor semi-axis of the ellipse, and the rotation angle of the principal axis of the ellipse are jointly solved. The horizontal pixel coordinates and vertical pixel coordinates of the ellipse center obtained by solving are determined as the pixel coordinates of the ellipse borehole center.

[0103] Based on the pixel coordinates of the center of a circular or elliptical borehole, the pixel coordinates of the borehole center in the target preprocessed image are obtained.

[0104] In Example 1, when the target borehole subpixel contour point set is fitted with a circle, the pixel coordinates of the center of the circular borehole are determined as the pixel coordinates of the borehole center in the target preprocessed image.

[0105] When the target borehole subpixel contour point set is fitted with an ellipse, the pixel coordinates of the center of the elliptical borehole are determined as the pixel coordinates of the borehole center in the target preprocessed image.

[0106] The pixel coordinates of the borehole center in the target preprocessed image include the horizontal pixel coordinates and the vertical pixel coordinates of the borehole center in the target preprocessed image.

[0107] In this embodiment, mapping pixel coordinates to the processing equipment coordinate system to obtain the spatial positioning coordinates of the drilling center in the processing equipment coordinate system includes: constructing homogeneous pixel coordinates of the drilling center based on the pixel coordinates of the drilling center in the target preprocessed image. In Example 1, the pixel coordinates of the borehole center in the target preprocessed image include the horizontal pixel coordinates and the vertical pixel coordinates of the borehole center in the target preprocessed image. The horizontal pixel coordinates and the vertical pixel coordinates of the borehole center in the target preprocessed image are homogenized to obtain the homogenous pixel coordinates of the borehole center, which are used to represent the homogenous pixel position of the borehole center in the target preprocessed image.

[0108] Based on the camera intrinsic parameter matrix, the homogeneous pixel coordinates of the borehole center are converted into camera normalized imaging rays. In Example 1, the camera intrinsic parameter matrix is ​​inverted to obtain the inverse matrix of the camera intrinsic parameter matrix. The inverse matrix of the camera intrinsic parameter matrix is ​​multiplied by the homogeneous pixel coordinates of the borehole center to obtain the normalized imaging ray of the borehole center in the camera coordinate system.

[0109] Calculate the ray scale factor of the borehole center reaching the target machining plane along the camera's normalized imaging ray. In Example 1, the camera optical center is used as the origin of the camera coordinate system. The spatial candidate point of the borehole center in the camera coordinate system is represented as a spatial point formed by extending the normalized imaging ray of the borehole center in the camera coordinate system from the camera optical center to the target processing plane. The spatial candidate point is obtained by multiplying the normalized imaging ray of the borehole center in the camera coordinate system with the ray scale factor to be solved. The spatial candidate point is substituted into the target processing plane relationship determined by the target processing plane normal vector in the camera coordinate system and the target processing plane offset in the camera coordinate system so that the spatial candidate point satisfies the target processing plane constraint.

[0110] Calculate the inner product between the normal vector of the target processing plane in the camera coordinate system and the normalized imaging ray of the borehole center in the camera coordinate system to obtain the projection value of the ray plane direction. Take the opposite of the offset of the target processing plane in the camera coordinate system and divide it by the projection value of the ray plane direction to obtain the ray scale factor, which is used to represent the scale multiple when the normalized imaging ray from the camera optical center along the borehole center in the camera coordinate system extends to the target processing plane.

[0111] Multiply the ray scale factor of the borehole center reaching the target processing plane along the normalized imaging ray of the camera by the normalized imaging ray of the borehole center in the camera coordinate system to obtain the spatial positioning coordinates of the borehole center in the camera coordinate system. In Example 1, the ray scale factor of the borehole center reaching the target processing plane along the normalized imaging ray of the camera is multiplied with the normalized imaging ray of the borehole center in the camera coordinate system to obtain the spatial positioning coordinates of the borehole center in the camera coordinate system.

[0112] The spatial positioning coordinates of the borehole center in the camera coordinate system are three-dimensional spatial coordinates. The spatial positioning coordinates of the borehole center in the camera coordinate system include the coordinate components of the borehole center in the direction of the first coordinate axis of the camera coordinate system, the coordinate components of the borehole center in the direction of the second coordinate axis of the camera coordinate system, and the coordinate components of the borehole center in the direction of the third coordinate axis of the camera coordinate system.

[0113] Map the spatial positioning coordinates of the drilling center in the camera coordinate system to the machining equipment coordinate system to obtain the spatial positioning coordinates of the drilling center in the machining equipment coordinate system; In Example 1, based on the pre-calibrated hand-eye calibration parameters, a rigid transformation matrix from the camera coordinate system to the processing equipment coordinate system is constructed. The rigid transformation matrix from the camera coordinate system to the processing equipment coordinate system consists of a rotation matrix and a translation vector. The spatial positioning coordinates of the borehole center in the camera coordinate system are converted into the homogeneous spatial positioning coordinates of the borehole center in the camera coordinate system. The rigid transformation matrix from the camera coordinate system to the processing equipment coordinate system is multiplied by the homogeneous spatial positioning coordinates of the borehole center in the camera coordinate system to obtain the homogeneous spatial positioning coordinates of the borehole center in the processing equipment coordinate system.

[0114] The spatial positioning coordinates of the drilling center in the machining equipment coordinate system are extracted from the homogeneous spatial positioning coordinates of the drilling center in the machining equipment coordinate system. The spatial positioning coordinates of the drilling center in the machining equipment coordinate system are three-dimensional spatial coordinates, including the coordinate components of the drilling center in the first coordinate axis direction, the coordinate components of the drilling center in the second coordinate axis direction, and the coordinate components of the drilling center in the third coordinate axis direction.

[0115] Based on the spatial positioning coordinates of the drilling center in the coordinate system of the machining equipment, the feed positioning target point of the CNC machine tool spindle, the end effector of the industrial robot, or the automated drilling machine is generated. Based on the type of machining object, the feed positioning command is generated based on the feed positioning target point.

[0116] In Example 1, the output rules for the feed positioning command include: When the object to be processed is a CNC machine tool spindle, a CNC machine tool spindle feed positioning command is generated. The CNC machine tool spindle feed positioning command includes the first spatial coordinates of the CNC machine tool spindle feed target, the second spatial coordinates of the CNC machine tool spindle feed target, and the third spatial coordinates of the CNC machine tool spindle feed target. When the processing target is an industrial robot end effector, an industrial robot end effector feed positioning command is generated. The industrial robot end effector feed positioning command includes the first spatial coordinates of the industrial robot end effector feed target, the second spatial coordinates of the industrial robot end effector feed target, and the third spatial coordinates of the industrial robot end effector feed target. When the processing target is an automated drilling machine, an automated drilling machine feed positioning command is generated. The automated drilling machine feed positioning command includes the first spatial coordinates of the automated drilling machine feed target, the second spatial coordinates of the automated drilling machine feed target, and the third spatial coordinates of the automated drilling machine feed target.

[0117] Example 2: In an automated drilling and positioning production line, the operator places a batch of milled metal connecting plates into a vision positioning station. All connecting plates have pre-drilled holes, with varying degrees of reflection, scratches, and oil stains near the hole openings. The system acquires one target drilling image and one standard drilling template image. The target drilling image has a resolution of 4096×3072, and the standard drilling template image also has a resolution of 4096×3072. After camera distortion correction, scale normalization, and brightness equalization, the average grayscale of the target pre-processed image is adjusted from 117.6 to 128.4, and the grayscale standard deviation is reduced from 61.2 to 43.7. The average grayscale of the template pre-processed image is 127.9, and the grayscale standard deviation is 41.8.

[0118] In the preprocessed image of the target, there is a reflective area of ​​cutting fluid on the upper right side of the hole, with a maximum grayscale value of 248. Three machining scratches are present on the left side of the hole, with one scratch having an edge gradient peak of 82.5, higher than the average local gradient of 75.3 at the actual hole edge. Traditional edge detection methods extracted 428 edge fragments from this preprocessed image, of which only 93 were located near the actual hole edge. The remaining edge fragments mainly came from the reflective area and the scratch area. When directly performing circle detection, the candidate circle center deviated from the manually measured hole center by 2.86 pixels.

[0119] When using the method of this invention, the system inputs the preprocessed target image into the SAM2 aperture edge segmentation model to obtain a target aperture edge probability map. In the target aperture edge probability map, the average aperture edge probability value for the true aperture edge region is 0.91, the average aperture edge probability value for the boundary of the dark area within the aperture is 0.87, the average aperture edge probability value for the reflective area is 0.18, the average aperture edge probability value for the scratched area is 0.23, and the average aperture edge probability value for the oil stain area is 0.16. The system constructs a priori mask for the target aperture edge using a segmentation threshold of 0.62. The number of pixels retained as the target aperture edge region in the priori mask is 18,642, and the number of pixels defined as the region outside the mask is 12,564,570. The system then extracts the set of pixel coordinates of the boundary pixels of the priori mask, obtaining a total of 2,187 pixel coordinates of the target aperture edge region boundaries.

[0120] The system constructs the target aperture edge boundary extension region based on the prior mask boundary pixel coordinate set of the target aperture edge, with an outward pixel expansion of 18 pixels. After boundary nearest distance calculation, the number of pixels within the effective constraint region of the target aperture edge is 74,215, accounting for only 0.59% of the total number of pixels in the target preprocessed image. In the reflective area, 96.4% of the pixels are assigned to the off-mask pixel coordinate set; in the scratch area, 92.1% of the pixels are assigned to the off-mask pixel coordinate set; and in the oil stain area, 97.8% of the pixels are assigned to the off-mask pixel coordinate set. After generating the target aperture edge constraint image, the effective aperture edge region is preserved, while the background region outside the aperture edge is set to zero pixel value.

[0121] The system performs the same hole edge segmentation and mask constraint clipping on the standard borehole template image. In the template hole edge probability map, the average probability value of the template hole edge region is 0.94, and the average probability value of the region outside the template mask is 0.09. After constructing the template hole edge prior mask, the system obtains 2254 boundary pixel coordinates of the template hole edge region, and generates the effective constraint region of the template hole edge with an outward expansion of 18 pixels. The final template hole edge constraint image contains 75106 effective pixels.

[0122] The system employs the same feature extraction network to extract features from both the target hole edge constraint image and the template hole edge constraint image. A target hole edge feature response map is generated from the target hole edge constraint image, with a keypoint response threshold set to 0.72, resulting in 486 target keypoints and 486 target descriptors. Similarly, 472 template keypoints and 472 template descriptors are obtained from the template hole edge constraint image. In comparison, the traditional full-image LightGlue input stage extracts 5318 target keypoints in the target preprocessed image and 4967 template keypoints in the template preprocessed image, with approximately 41% of the target keypoints originating from reflective, scratch, and repetitive texture regions.

[0123] Before improving the LightGlue input, the system calculated the minimum radial geometric distance to the target aperture edge prior mask boundary pixel coordinate set for each of the 486 target keypoints. For example, the 2D pixel coordinates of the 37th target keypoint are (2064.31, 1531.78), and its distance to the nearest point on the target aperture edge boundary is 2.42 pixels; the 2D pixel coordinates of the 118th target keypoint are (2091.64, 1506.22), and its minimum radial geometric distance is 5.18 pixels. The system performed the same calculation on the 472 template keypoints and input the horizontal pixel coordinates, vertical pixel coordinates, and minimum radial geometric distance of each keypoint into the 3D position encoding function to obtain a geometric position encoding vector consistent with the descriptor dimension. After injecting the geometric position encoding, the target descriptor and template descriptor simultaneously contain aperture edge texture features, 2D pixel position features, and aperture edge radial geometric features.

[0124] The system inputs the target geometrically perceived descriptor set and the template geometrically perceived descriptor set into the improved LightGlue feature matching network. After the first round of self-attention calculation, the mean correlation within keypoints on the target side increased from 0.42 to 0.68, and the mean correlation within keypoints on the template side increased from 0.45 to 0.70. After the first round of cross-attention calculation, the system obtains the first-round matching confidence matrix and performs bidirectional mutual selection and confidence threshold determination. The matching confidence threshold is set to 0.75, and the first round outputs 164 feature matching pairs with an average matching confidence of 0.83. Traditional LightGlue full-image matching outputs 618 matching pairs on the same image, but 137 of these pairs, after manual verification, come from reflective or scratched areas, resulting in a false matching rate of 22.2%. In the first round of matching pairs of this invention, there are 14 false matches, resulting in a false matching rate of 8.5%.

[0125] The system uses the first round of feature matching pairs as the current round's feature matching pair set and extracts the coordinate set of key points for matching the target hole edge. Since the pre-made hole has an elliptical projection at the current shooting angle, the system performs both circular and elliptical fitting simultaneously. The residual for the first round's circular fitting is 0.134, the residual for the elliptical fitting is 0.091, and the average confidence score for the current round is 0.83. The system weightedly fuses the complementary term of the average confidence score with the elliptical fitting residual to obtain the current round's circular / elliptical consistency convergence index of 0.131, which is lower than the preset threshold of 0.18, triggering adaptive early stopping. If traditional LightGlue uses a fixed 9-layer computation, subsequent attention layers still need to be executed for the same image; this invention stops iteration after the 3rd attention layer, reducing the time for a single matching calculation from 126 milliseconds for traditional full-image LightGlue to 71 milliseconds.

[0126] The system outputs the current set of feature matching pairs as a set of hole edge geometrically consistent feature matching pairs, and establishes a hole edge homography geometric mapping matrix from the template hole edge constraint image to the target hole edge constraint image. The system calculates the mapping reprojection error for each hole edge geometrically consistent feature matching pair, with a reprojection error threshold of 1.5 pixels. Of the 164 sets of hole edge geometrically consistent feature matching pairs, 142 sets have a mapping reprojection error less than or equal to 1.5 pixels and are retained as final valid feature matching pairs; 22 sets have a mapping reprojection error greater than 1.5 pixels and are discarded. The average mapping reprojection error of the retained matching pairs is 0.62 pixels, and the maximum mapping reprojection error is 1.43 pixels.

[0127] The system extracts the coordinate set of valid target borehole edge key points from 142 final valid feature matching pairs and performs sub-pixel level contour reconstruction on the borehole edges in the target borehole edge constrained image. For each valid target borehole edge key point, the system selects a 9×9 local borehole edge neighborhood, calculates the local lateral gradient value, local longitudinal gradient value, and borehole edge grayscale response value, and calculates the lateral and longitudinal sub-pixel offsets accordingly. Statistically, the lateral sub-pixel offset distribution ranges from -0.41 pixels to 0.38 pixels, and the longitudinal sub-pixel offset distribution ranges from -0.36 pixels to 0.44 pixels. The system ultimately obtains 142 target borehole sub-pixel contour points.

[0128] Since the target hole edge is projected as an ellipse in the image, the system performs elliptical fitting. The fitted result shows that the lateral pixel coordinates of the elliptical borehole center are 2058.47, the lateral pixel coordinates are 1536.29, the major semi-axis length is 132.62 pixels, the minor semi-axis length is 126.08 pixels, and the principal axis rotation angle is 0.083 radians. The pixel coordinates of the reference hole center obtained by manual microscopic measurement are (2058.58, 1536.17). The lateral error of the method of this invention is 0.11 pixels, the lateral error is 0.12 pixels, and the overall error is 0.16 pixels. The center coordinates obtained by the traditional Canny+Hough method are (2061.34, 1534.91), with an overall error of 3.05 pixels; the center coordinates obtained by the traditional full-image LightGlue method are (2059.02, 1535.79), with an overall error of 0.61 pixels.

[0129] In the same batch of 1200 test images, there were 320 highly reflective samples, 285 severely scratched samples, 196 oil-stained interference samples, and 399 samples with repeated hole textures. The traditional Canny+Hough method had an average localization error of 2.14 pixels and a failure rate of 18.7%; the traditional SIFT+RANSAC method had an average localization error of 1.36 pixels and a failure rate of 9.8%; the traditional full-image LightGlue method had an average localization error of 0.64 pixels and a failure rate of 4.6%; and the method of this invention had an average localization error of 0.19 pixels and a failure rate of 0.8%. In the highly reflective samples, the method of this invention had an average error of 0.21 pixels, while the traditional full-image LightGlue method had an average error of 0.77 pixels. In the severely scratched samples, the method of this invention had an average error of 0.23 pixels, while the traditional Canny+Hough method had an average error of 2.68 pixels. In the samples with repeated hole textures, the method of this invention had a mismatch rate of 2.9%, while the traditional full-image LightGlue method had a mismatch rate of 13.6%.

[0130] On the edge industrial control computer, the traditional full-image LightGlue takes an average of 126 milliseconds for single positioning and has a peak memory usage of 4.9GB; the method of this invention takes an average of 71 milliseconds for single positioning and has a peak memory usage of 1.8GB. The system then converts the drill center pixel coordinates into homogeneous drill center pixel coordinates, and combines these with the camera intrinsic parameter matrix, target machining plane constraints, and hand-eye calibration parameters to obtain the spatial positioning coordinates of the drill center in the machining equipment coordinate system. Compared with the contact measurement reference, these spatial positioning coordinates have an average planar error of 0.012 mm and a maximum error of 0.031 mm, meeting the requirements for automated drilling feed positioning.

[0131] In Example 1, please refer to Figure 3 On the interface of the drilling machine control system, an automatic reverse function indicator light can be set as needed; it is off by default when the machine is turned on. When the light is on and this function is enabled, the motor automatically reverses when the spindle moves downward to the set depth, for tapping purposes.

[0132] Press and hold the forward / reverse button to turn this function on or off. When the function is on, the maximum spindle speed is limited to less than 600 RPM. Press the forward / reverse button and the zero button simultaneously. The indicator light and depth display will flash and show the absolute depth. Use the encoder knob to adjust the depth value for automatic reversal. Then, briefly press the forward / reverse button again. The relative depth will return to zero and be displayed normally. The indicator light will remain on, completing the depth setting.

[0133] In addition, an overload protection indicator light can be set to flash when the motor load is too high (over 85%), and remain lit when the machine is stopped due to overload. It can be reset by powering off and restarting or by pressing the spindle stop / rotate button.

[0134] A zeroing button can be set; a short press resets the relative spindle depth value to zero. The absolute and relative spindle depth values ​​are reset to zero by default after power-off and power-on. The absolute value is not displayed; it is only used to set the depth position when the automatic reverse function is activated. During normal operation, a digital spindle speed display (0.4-inch 5-digit blue LED display, dimensions 50x16x7, character height 10.2mm, illuminates upon power-on) is also available. The spindle speed is displayed by reducing the motor speed by 1:1.6, with a resolution of 50 RPM. A "-" sign precedes the reverse digit. When the spindle is not rotating, rotating the encoder briefly displays the preset speed, then displays the actual speed after 1.5 seconds. The default speed after power-off and power-on is 1000 RPM. A long press switches the spindle depth to millimeters / inches; in imperial units, the indicator light after "in" illuminates.

[0135] It also features a digital display for the spindle feed depth (0.4-inch 5-digit blue LED display, which lights up when the machine is powered on), with a minimum resolution of 0.1mm and a minimum imperial resolution of 0.005in. The spindle displacement is detected by a photoelectric encoder through gear rotation, displaying a positive value upwards when the spindle is at the zero position and a negative value downwards.

[0136] An automatic start indicator light can also be set. It is off by default upon power-on. When the light is on, the function is activated, the spindle moves, and the motor automatically starts when the absolute depth exceeds 5mm and lasts for more than 0.5 seconds; it automatically stops when the depth is less than 5mm. Additionally, a long press of the "Spindle Start / Stop" button starts and stops the spindle machining function. A forward / reverse button can also be set. A short press switches between forward and reverse; in automatic reverse mode, a short press has no function. A long press toggles automatic reverse; pressing it simultaneously with the zeroing button sets the automatic reverse position.

[0137] It also features an encoder knob, a 24-bit pulse encoder, which is used by default to adjust the speed, with the speed adjusted in 50 RPM pulses. Simultaneously pressing the forward / reverse + zero button sets the depth value for automatic reverse, in 1mm increments per pulse (0.05in per pulse in imperial units).

[0138] An LED button is located at the bottom of the display interface to control the 12V, 5W LED light, requiring a DC 12V-0.5A power supply from the board. A laser button is also included to control two laser markers (5V, 40mA), requiring a DC 5V-0.1A power supply from the board. Preferably, a spindle start / stop button can also be provided; a short press toggles spindle start / stop, while a long press activates / deactivates the auto-start function.

[0139] Finally, it's important to note that an emergency stop switch must be installed on the drilling machine to cut off the main power supply. This switch is not on the control panel and needs to be installed separately. Alternatively, a protective cover interlock microswitch can be installed on the external protective cover of the machine, as needed. When the microswitch is off, the spindle stops rotating, and the "Spindle Start / Stop" button on the control panel will also prevent the spindle from rotating.

[0140] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A drilling positioning method based on computer vision, characterized in that, include: Acquire the target drilling image and the standard drilling template image of the workpiece to be processed, and preprocess them respectively to obtain the target preprocessed image and the template preprocessed image; The preprocessed target image is input into the SAM2 aperture edge segmentation model based on the zero-shot semantic segmentation mechanism to generate the target aperture edge constraint image; The template preprocessed image is input into the SAM2 hole edge segmentation model to obtain the template hole edge prior mask and generate the template hole edge constraint image. Feature extraction is performed on the target hole edge constraint image and the template hole edge constraint image respectively to obtain the target key point set, the target descriptor set, the template key point set and the template descriptor set. For each key point in the target key point set and the template key point set, calculate the minimum radial geometric distance to the corresponding prior mask boundary curve of the hole edge, and inject it into the target descriptor subset and the template descriptor subset to form a geometrically aware descriptor subset; Input the geometrically aware descriptor set into the improved LightGlue feature matching network to obtain the first round of feature matching pair set and the corresponding average matching confidence score; The first-round feature matching pair set is fitted with a circle or ellipse, the fitting residual is calculated, and a circle / ellipse consistency convergence index is constructed based on the average matching confidence and the fitting residual. When the circle / ellipse consistency convergence index meets the preset threshold, adaptive early stopping is triggered, subsequent attention iterations are stopped, and a set of geometrically consistent feature matching pairs of the hole edge is output. If the preset threshold is not met, return the improved LightGlue feature matching network input to continue the iteration; Based on the set of matching pairs of geometrically consistent borehole edges, an image mapping is established. After removing abnormal matches, the sub-pixel borehole edges are reconstructed and fitted to obtain the pixel coordinates of the borehole center in the target preprocessed image. The pixel coordinates are mapped to the coordinate system of the machining equipment to obtain the spatial positioning coordinates of the drilling center in the coordinate system of the machining equipment.

2. The drilling positioning method based on computer vision according to claim 1, characterized in that, The preprocessing includes camera distortion correction, scale normalization, and brightness equalization.

3. The drilling positioning method based on computer vision according to claim 1, characterized in that, The step of inputting the preprocessed target image into the SAM2 aperture edge segmentation model based on a zero-shot semantic segmentation mechanism includes: For the target preprocessed image, establish the pixel coordinate domain; The preprocessed target image is input into the SAM2 pore edge segmentation model based on the zero-shot semantic segmentation mechanism to generate a target pore edge probability map; Construct a priori mask for the target aperture edge based on the target aperture edge probability map; Based on the prior mask of the target aperture edge and the pixel coordinate domain, generate a set of pixel coordinates of the boundary of the prior mask of the target aperture edge; Based on the set of pixel coordinates of the prior mask boundary of the target aperture, construct the target aperture boundary extension region; Based on the prior mask of the target aperture edge and the expansion region of the target aperture edge boundary, the effective constraint region of the target aperture edge is determined; Based on the effective constraint region of the target aperture edge, determine the set of pixel coordinates outside the mask; Based on the effective constraint region of the target aperture edge and the set of pixel coordinates outside the mask, the target preprocessed image is subjected to mask constraint cropping to generate the target aperture edge constraint image.

4. The drilling positioning method based on computer vision according to claim 3, characterized in that, The SAM2 aperture edge segmentation model based on the zero-shot semantic segmentation mechanism includes: The target preprocessed image is semantically encoded using an image encoder to obtain the target image semantic feature map; Based on the semantic feature map of the target image, a set of candidate clues for the edge of the target hole is generated in the pixel coordinate domain of the preprocessed target image; The target aperture edge cue embedding feature set is obtained by encoding the cue features of the candidate cue set of the target aperture edge by the cue encoder; By performing joint mask decoding on the semantic feature map of the target image and the embedded feature set of the target aperture edge cue, a set of candidate aperture edge masks for the target is obtained. The candidate mask filter calculates the confidence score of each target candidate aperture edge mask and the semantic feature map of the target image, and the target candidate aperture edge mask with the highest confidence score is determined as the target aperture edge probability map.

5. The drilling positioning method based on computer vision according to claim 1, characterized in that, The step of inputting the template preprocessed image into the SAM2 aperture edge segmentation model includes: For the template preprocessed image, establish the template pixel coordinate domain; Input the preprocessed template image into the SAM2 aperture edge segmentation model to generate a template aperture edge probability map; Construct a priori mask for template aperture edges based on the template aperture edge probability map; Based on the template aperture prior mask and the template pixel coordinate domain, generate a set of template aperture prior mask boundary pixel coordinates; Based on the set of pixel coordinates of the prior mask boundary of the template aperture, the extended region of the template aperture boundary is constructed; Based on the prior mask of the template hole edge and the extension region of the template hole edge boundary, the effective constraint region of the template hole edge is determined; Based on the effective constraint area of ​​the template hole edge, determine the set of pixel coordinates outside the template mask; Based on the effective constraint region of the template hole edge and the set of pixel coordinates outside the template mask, the template preprocessed image is mask-constrained and cropped to generate the template hole edge constraint image; The same feature extraction network is used to extract features from the target hole edge constraint image and the template hole edge constraint image respectively, resulting in a target key point set, a target descriptor set, a template key point set, and a template descriptor set.

6. The drilling positioning method based on computer vision according to claim 1, characterized in that, The calculation of the minimum radial geometric distance from each key point in the target key point set and the template key point set to the corresponding prior mask boundary curve of the hole edge includes: Based on the set of pixel coordinates of the prior mask boundary of the target aperture, calculate the minimum radial geometric distance of the target corresponding to each target key point in the target key point set; Based on the set of prior mask boundary pixel coordinates of template hole edge, calculate the minimum radial geometric distance of the template corresponding to each template key point in the template key point set; The two-dimensional pixel coordinates of each target key point in the target hole edge constraint image are combined with the corresponding target minimum radial geometric distance to form the target three-dimensional geometric position vector corresponding to each target key point; The two-dimensional pixel coordinates of each template key point in the template hole edge constraint image are combined with the corresponding minimum radial geometric distance of the template to form the template three-dimensional geometric position vector corresponding to each template key point. Input the target's 3D geometric position vector and the template's 3D geometric position vector into the 3D position encoding function to generate a target geometric position encoding vector and a template geometric position encoding vector that are consistent with the dimensions of the target descriptor and the template descriptor, respectively. The target geometric position encoding vector is injected into the target descriptor, and the template geometric position encoding vector is injected into the template descriptor to generate the target geometric perception descriptor and the template geometric perception descriptor, respectively. All target geometry-aware descriptors are combined to form a target geometry-aware descriptor set, and all template geometry-aware descriptors are combined to form a template geometry-aware descriptor set. The target geometry-aware descriptor set and the template geometry-aware descriptor set together constitute the geometry-aware descriptor set.

7. The drilling positioning method based on computer vision according to claim 1, characterized in that, The input of the geometry-aware descriptor subset into the improved LightGlue feature matching network includes: The geometrically aware descriptor set is input into the improved LightGlue feature matching network, and the target geometrically aware descriptor set and the template geometrically aware descriptor set are expressed in matrix form to obtain the first round target input feature matrix and the first round template input feature matrix. Self-attention calculations are performed on the first-round target input feature matrix and the first-round template input feature matrix respectively to obtain the first-round target self-attention feature matrix and the first-round template self-attention feature matrix; Cross-attention calculation is performed based on the first round target self-attention feature matrix and the first round template self-attention feature matrix to obtain the first round target cross-attention feature matrix and the first round template cross-attention feature matrix; Based on the first-round target cross-attention feature matrix and the first-round template cross-attention feature matrix, calculate the first-round matching similarity between the target key points and the template key points; Perform matching confidence prediction on the first round of matching similarity to obtain the first round of matching confidence matrix, and determine the first round of candidate matching relationships based on the first round of matching confidence matrix; Based on the first round of candidate matching relationships, bidirectional mutual selection and confidence threshold determination are performed on any target key point and any template key point to generate the first round of feature matching pair set; Calculate the average matching confidence score based on the first-round matching confidence score of each first-round feature matching pair in the first-round feature matching pair set.

8. The drilling positioning method based on computer vision according to claim 7, characterized in that, The process of fitting the first-round feature matching pair set into a circular or elliptical shape, calculating the fitting residuals, and constructing a circular / elliptical consistency convergence index based on the average matching confidence and the fitting residuals includes: The first round of feature matching pairs is used as the current round of feature matching pairs, and a round representation of the current round of feature matching pairs is established. Based on the current set of feature matching pairs, extract the set of coordinates of key matching points on the target hole edge; Perform circular fitting on the set of key point coordinates for matching the target hole edge to obtain the current wheel's circular fitting parameters and the current wheel's circular fitting residual; Elliptical fitting is performed on the set of coordinates of key points for matching the target hole edge to obtain the elliptical fitting parameters and residuals of the current wheel. Determine the fitting residual of the current wheel based on the circular fitting residual or the elliptical fitting residual of the current wheel; Calculate the average matching confidence of the current round based on the matching confidence of each feature matching pair in the current round feature matching pair set; The weighted sum of the average confidence score of the current round and the fitting residual of the current round is used to obtain the circle / ellipse consistency convergence index of the current round. When the current wheel circle / ellipse consistency convergence index is less than or equal to the preset threshold, it is determined that the current wheel feature matching pair set satisfies the hole edge geometric consistency convergence condition, triggering adaptive early stopping, stopping attention iteration, and outputting the current wheel feature matching pair set as the hole edge geometric consistency feature matching pair set; When the convergence index of the current wheel circle / ellipse is greater than the preset threshold, it is determined that the current wheel feature matching pair set does not meet the convergence condition of hole edge geometry consistency. The target cross attention feature matrix and the template cross attention feature matrix of the current wheel are then input into the next attention layer of the improved LightGlue feature matching network to continue iterating.

9. The drilling positioning method based on computer vision according to claim 1, characterized in that, The process of establishing an image mapping based on a set of geometrically consistent feature matching pairs of aperture edges, reconstructing sub-pixel aperture edges after removing anomalous matches, and fitting the image includes: Based on the set of matching pairs of geometrically consistent hole edge features, the coordinate pairs of matching key points between the target hole edge constraint image and the template hole edge constraint image are extracted. Based on the matching key point coordinate pairs, the geometric mapping relationship between the target hole edge constraint image and the template hole edge constraint image is established, and the hole edge homography geometric mapping matrix is ​​obtained by solving. Based on the aperture edge geometry mapping matrix, calculate the mapping reprojection error corresponding to each aperture edge geometry consistent feature matching pair; Based on the mapping reprojection error, feature matching pairs that do not meet geometric consistency are eliminated to obtain the final set of valid feature matching pairs. Based on the final set of effective feature matching pairs, extract the set of coordinates of effective target hole edge key points in the target hole edge constraint image; Based on the effective set of key point coordinates of the target borehole edge, sub-pixel level contour reconstruction is performed on the borehole edge in the target borehole edge constraint image to obtain the sub-pixel contour point set of the target borehole. Perform circular fitting based on the set of sub-pixel contour points of the target borehole to obtain the center pixel coordinates of the circular borehole; Elliptical fitting is performed based on the set of sub-pixel contour points of the target borehole to obtain the center pixel coordinates of the elliptical borehole; Based on the pixel coordinates of the center of a circular or elliptical borehole, the pixel coordinates of the borehole center in the target preprocessed image are obtained.

10. The drilling positioning method based on computer vision according to claim 1, characterized in that, The step of mapping pixel coordinates to the machining equipment coordinate system to obtain the spatial positioning coordinates of the drilling center in the machining equipment coordinate system includes: Based on the pixel coordinates of the borehole center in the target preprocessed image, construct the homogeneous pixel coordinates of the borehole center; Based on the camera intrinsic parameter matrix, the homogeneous pixel coordinates of the borehole center are converted into camera normalized imaging rays. Calculate the ray scale factor of the borehole center reaching the target machining plane along the camera's normalized imaging ray. Multiply the ray scale factor of the borehole center reaching the target processing plane along the normalized imaging ray of the camera by the normalized imaging ray of the borehole center in the camera coordinate system to obtain the spatial positioning coordinates of the borehole center in the camera coordinate system. Map the spatial positioning coordinates of the drilling center in the camera coordinate system to the machining equipment coordinate system to obtain the spatial positioning coordinates of the drilling center in the machining equipment coordinate system; Based on the spatial positioning coordinates of the drilling center in the coordinate system of the machining equipment, the feed positioning target point of the CNC machine tool spindle, the end effector of the industrial robot, or the automated drilling machine is generated. Based on the type of machining object, the feed positioning command is generated based on the feed positioning target point.

Images