An industrial umbrella tool hole array multi-view three-dimensional hole positioning method based on point cloud fingerprint matching and related equipment

Abstract

The application provides an industrial umbrella tool hole array multi-view three-dimensional hole positioning method based on point cloud fingerprint matching and related equipment, and belongs to the technical field of industrial automation. The method comprises the following steps: collecting multi-view images, and labeling camera intrinsic parameters for each image; obtaining a dense three-dimensional point cloud and camera extrinsic parameters; obtaining a two-dimensional center of each hole, elliptical parameters and a hole candidate area; projecting the dense three-dimensional point cloud to each view image plane by using the camera intrinsic parameters and the camera extrinsic parameters, and recording the three-dimensional points falling into the hole candidate area as the point cloud fingerprints of the holes; obtaining the corresponding relationship of the same-named holes across views; obtaining the three-dimensional center of the holes by re-projection residual optimization; obtaining the hole radius by weighted fusion according to the elliptical parameters of each view and the observation angle, and outputting the three-dimensional center of the holes, the unit normal, the radius and the hole state marker. The application can realize high-precision positioning of each lens placement hole on the industrial umbrella tool hole array in a three-dimensional space.

Description

Technical Field

[0001] This application relates to the field of industrial automation technology, and in particular to a method and related equipment for multi-view three-dimensional hole positioning of industrial umbrella hole array based on point cloud fingerprint matching. Background Technology

[0002] In related technologies, single-camera two-dimensional visual positioning can only obtain the position of holes in the image plane, and it is difficult to directly obtain the hole depth and local normal, which cannot fully support the three-dimensional pose required for lens placement.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a multi-view three-dimensional hole positioning method and related equipment for industrial umbrella hole arrays based on point cloud fingerprint matching, which can achieve high-precision positioning of each lens placement hole on the industrial umbrella hole array in three-dimensional space.

[0005] To achieve the above objectives, one aspect of this application proposes a multi-view three-dimensional hole location method for industrial umbrella hole arrays based on point cloud fingerprint matching, the method comprising the following steps: Collect multi-view images from multiple cameras positioned in different directions, and label the camera intrinsic parameters for each image; Sparse and dense reconstructions are performed on the multi-view images with the camera intrinsic parameters to obtain dense 3D point clouds and extrinsic parameters of each camera. Edge extraction and ellipse fitting are performed on the candidate regions output by the detection model in the images from different viewpoints to obtain the two-dimensional center, ellipse parameters and candidate regions of each hole; The dense 3D point cloud is projected onto image planes of various viewpoints using the camera intrinsic and extrinsic parameters, and the 3D point numbers falling into the hole candidate region are recorded as the point cloud fingerprints of the holes. The point cloud fingerprints of each hole under different viewpoints are compared, the number of identical 3D point numbers among the point cloud fingerprints is counted and a matching cost matrix is ​​constructed, and the correspondence between holes with the same name across viewpoints is obtained by solving the problem. Based on the correspondence, multiple two-dimensional centers of the same-named holes are combined with the camera extrinsic parameters to establish a spatial line of sight. The least squares intersection point is solved to obtain the initial value of the three-dimensional center. The three-dimensional center of the hole is obtained by reprojection residual optimization. The unit normal is obtained by searching the neighboring point cloud in the dense three-dimensional point cloud based on the three-dimensional center of the hole. The radius of the hole is obtained by weighted fusion based on the ellipse parameters and observation angles from each viewpoint. The three-dimensional center of the hole, the unit normal, the radius, and the hole status marker are output.

[0006] In some embodiments, acquiring multi-view images output by multiple cameras arranged in different directions and labeling camera intrinsic parameters for each image includes: Multi-directional cameras are arranged around the hole array component of the industrial umbrella. The multi-directional cameras are pre-calibrated and their focal length, principal point, radial distortion coefficient, and tangential distortion coefficient are stored as original camera intrinsic parameters. The multi-directional cameras include a center camera, a left camera, a right camera, and a top camera. During acquisition, the multi-angle cameras simultaneously capture images. The camera corresponding to each image is identified based on the image file name or camera number, and the corresponding original camera intrinsic parameters are written into the image's metadata to obtain multi-view images.

[0007] In some embodiments, performing sparse and dense reconstruction based on the multi-view images with the camera intrinsic parameters to obtain a dense 3D point cloud and the extrinsic parameters of each camera includes: Feature extraction and feature matching are performed on the multi-view images with the camera intrinsic parameters. Sparse reconstruction is performed based on the matching results to obtain the camera extrinsic parameters of each camera and obtain a sparse 3D model. Based on the sparse 3D model, the multi-view image is densely reconstructed to obtain a dense 3D point cloud; the camera intrinsic parameters and the camera extrinsic parameters are used to project each 3D point in the dense 3D point cloud onto the image plane of each view.

[0008] In some embodiments, the step of performing edge extraction and ellipse fitting on the candidate regions output by the detection model in the images from each viewpoint to obtain the two-dimensional center, ellipse parameters, and candidate regions of each hole includes: The YOLO object detection model is used to detect holes in images from various viewpoints, and the candidate regions or masks for each hole are output. Each candidate region is expanded outward by a preset pixel boundary according to the actual size of each hole and then cropped. Local histogram equalization is performed on the cropped region to reduce the grayscale difference caused by metal reflection to a preset range, thus obtaining the enhanced region. The hole edge points are extracted in the enhanced region and sub-pixel edge localization is performed using Zernike moments to obtain a sub-pixel level edge point set. Ellipse fitting is performed on the sub-pixel level edge point set to obtain the two-dimensional center, major axis, minor axis and rotation angle of each hole as ellipse parameters; The candidate region, mask, or elliptical neighborhood constructed based on the elliptical parameters is used as the hole candidate region.

[0009] In some embodiments, the step of projecting the dense 3D point cloud onto image planes at various viewpoints using the camera intrinsic and extrinsic parameters, and recording the 3D point numbers falling within the candidate hole region as the point cloud fingerprint of the hole, includes: For each 3D point in the dense 3D point cloud, the 3D point is projected onto the image plane of each viewpoint according to the camera intrinsic parameters and the camera extrinsic parameters. It is determined whether the projected point falls into the candidate area of ​​the hole of the preset hole in the preset viewpoint image. If it falls into the hole, the number of the 3D point is added to the set of 3D point numbers corresponding to the hole. After traversing all the three-dimensional points in the dense three-dimensional point cloud and all the holes in all viewpoint images, the set of three-dimensional point numbers corresponding to each hole is used as the point cloud fingerprint of the hole.

[0010] In some embodiments, comparing the point cloud fingerprints of each hole from different viewpoints, counting the number of identical 3D point numbers among the point cloud fingerprints and constructing a matching cost matrix to solve for the cross-view correspondence of holes with the same name includes: A hole pair is formed by taking one hole from any two different perspectives. The intersection of the sets of 3D point numbers corresponding to the two holes is calculated, and the number of the same 3D point numbers in the intersection is taken as the number of common points. Calculate the proportion of the number of common points to the minimum total number of points in the two point cloud fingerprints; If the number of common points exceeds a preset threshold, the hole pair is marked as a candidate hole pair with the same name, and the number of common points or the ratio is used as the matching cost of the candidate hole pair with the same name. After traversing all hole combinations across all viewpoints, a cross-viewpoint matching cost matrix is ​​constructed with each hole as a row and each hole as a column; each element in the matching cost matrix is ​​the matching cost value or null value for the corresponding hole pair. The Hungarian matching algorithm is used to solve the matching cost matrix globally to obtain the one-to-one correspondence between holes from each viewpoint as the cross-viewpoint correspondence of holes with the same name.

[0011] In some embodiments, the step of establishing a spatial line of sight by combining the multiple two-dimensional centers of the same-named holes with the camera extrinsic parameters according to the correspondence, solving for the least squares intersection points to obtain the initial value of the three-dimensional center, and obtaining the three-dimensional center of the hole through reprojection residual optimization includes: For the two-dimensional center of multiple viewpoints that have been matched as the same real hole, a spatial line of sight is established from each camera center through the two-dimensional center based on the camera extrinsic parameters and camera intrinsic parameters of each viewpoint. Find the least-squares intersection point of multiple spatial lines of sight in the same coordinate system, and use it as the initial value of the three-dimensional center of the hole; The initial value of the three-dimensional center is reprojected back to the image plane of each viewpoint according to the camera intrinsic and extrinsic parameters of each viewpoint, and the reprojection residual between the projection point and the two-dimensional center of the corresponding viewpoint is calculated. If the reprojection residual of a certain view exceeds a preset threshold, the view is determined to be an abnormal observation. The view is then discarded, and the least squares intersection point is re-solved using the remaining view to obtain the optimized three-dimensional center of the hole.

[0012] In some embodiments, the step of searching for and fitting a neighboring point cloud in the dense 3D point cloud based on the 3D center of the hole to obtain the unit normal, weighting and fusing the ellipse parameters and viewing angles from each perspective to obtain the hole radius, and outputting the 3D center of the hole, the unit normal, the radius, and the hole status marker includes: Using the three-dimensional center of the hole as a reference, search for neighboring point clouds within a preset radius in the dense three-dimensional point cloud; Points that are greater than the radius of the hole and less than a preset outer diameter threshold are selected from the neighborhood point cloud to form a ring-shaped point cloud; the ring-shaped point cloud covers the metal surface near the edge of the hole and avoids the hollow area inside the hole. The annular point cloud is fitted with a plane, and the normal of the fitted plane is used as the unit normal of the local metal surface where the hole is located. For the ellipse parameters of the hole in the image from each viewpoint, the pixel radius of each viewpoint is extracted, and the pixel radius is converted into a physical radius estimate of each viewpoint by combining the focal length of the camera at each viewpoint and the three-dimensional center depth of the hole. The angle between the direction vector from the optical center of each camera to the three-dimensional center of the hole and the unit normal is calculated as the observation angle. The estimated physical radius values ​​of each viewpoint are weighted and fused in such a way that the smaller the observation angle, the higher the weight, to obtain the fused hole radius; The three-dimensional center of the hole, the unit normal, the fused hole radius, and the hole status marker indicating whether a lens has been placed in the hole are collectively output as three-dimensional hole position information.

[0013] To achieve the above objectives, another aspect of this application proposes a multi-view three-dimensional hole positioning system for industrial umbrella hole arrays based on point cloud fingerprint matching, used to implement the method described above. The system includes: The image acquisition module is used to acquire multi-view images output by multiple cameras arranged in different directions, and to annotate the camera intrinsic parameters for each image; The 3D reconstruction module is used to perform sparse reconstruction and dense reconstruction based on the multi-view image with the camera intrinsic parameters to obtain a dense 3D point cloud and the extrinsic parameters of each camera. The two-dimensional detection module is used to extract edges and fit ellipses to candidate regions output by the detection model in images from various perspectives, so as to obtain the two-dimensional center, ellipse parameters and candidate regions of each hole. The point cloud fingerprint construction module is used to project the dense three-dimensional point cloud onto the image plane of each view using the camera intrinsic parameters and the camera extrinsic parameters, and record the number of the three-dimensional points falling into the hole candidate area as the point cloud fingerprint of the hole. The cross-view matching module is used to compare the point cloud fingerprints of each hole under different views, count the number of identical 3D point numbers among the point cloud fingerprints and construct a matching cost matrix to solve for the correspondence between holes with the same name across views. The triangulation positioning module is used to establish a spatial line of sight by combining the multiple two-dimensional centers of the same hole with the camera extrinsic parameters according to the correspondence, solve the least squares intersection point to obtain the initial value of the three-dimensional center, and obtain the three-dimensional center of the hole through reprojection residual optimization. The pose estimation module is used to search for the neighboring point cloud in the dense three-dimensional point cloud based on the three-dimensional center of the hole to obtain the unit normal, and to obtain the hole radius by weighted fusion based on the ellipse parameters and observation angles of each viewpoint. The output module is used to output the three-dimensional center, unit normal, radius, and hole status marker of the hole.

[0014] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0016] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.

[0017] The embodiments of this application include at least the following beneficial effects: This application provides a method, system, electronic device, storage medium, and program product for multi-view three-dimensional hole positioning of industrial umbrella hole arrays based on point cloud fingerprint matching. The solution includes: acquiring multi-view images and labeling each image with camera intrinsic parameters; obtaining a dense three-dimensional point cloud and extrinsic parameters of each camera; obtaining the two-dimensional center, elliptic parameters, and candidate regions of each hole; projecting the dense three-dimensional point cloud onto the image plane of each view using the camera intrinsic and extrinsic parameters, and recording the three-dimensional points falling into the candidate regions of the holes as point cloud fingerprints of the holes; obtaining the correspondence between holes with the same name across viewpoints; obtaining the three-dimensional center of the holes through reprojection residual optimization; obtaining the hole radius by weighted fusion based on the elliptic parameters of each viewpoint and the observation angle; and outputting the three-dimensional center, unit normal, radius, and hole status marker of the holes. This application can achieve high-precision positioning of lens placement holes on industrial umbrella hole arrays in three-dimensional space. Attached Figure Description

[0018] Figure 1 This is a flowchart of a multi-view three-dimensional hole location method for industrial umbrella hole arrays based on point cloud fingerprint matching provided in an embodiment of this application; Figure 2 This is an overall flowchart provided in the embodiments of this application; Figure 3 This is a top camera view provided in an embodiment of this application; Figure 4 This is a left-side camera view provided in an embodiment of this application; Figure 5 This is a view of the center camera provided in an embodiment of this application; Figure 6 This is a right-side camera view provided in an embodiment of this application; Figure 7(a) is a first-angle overall device diagram provided in an embodiment of this application; Figure 7(b) is a second-angle overall device diagram provided in an embodiment of this application; Figure 8 This is an intermediate result image of two-dimensional hole detection and ellipse fitting provided in the embodiments of this application; Figure 9(a) is a first real dense point cloud projection map provided in an embodiment of this application; Figure 9(b) is a second realistic dense point cloud projection map provided in an embodiment of this application; Figure 9(c) is a third realistic dense point cloud projection map provided in the embodiments of this application; Figure 10 This is a cross-view matching diagram of holes with the same name provided in the embodiments of this application; Figure 11 This is a schematic diagram of the actual three-dimensional output (center point, local normal, hole radius) provided in the embodiments of this application; Figure 12 This is the overall three-dimensional normal map provided in the embodiments of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0022] 1. Industrial Umbrellas / Umbrella-shaped Metal Hole Arrays: refers to metal workpieces or tooling fixtures with an arc-shaped, fan-shaped, or umbrella-shaped outer contour, whose surface is distributed with multiple circular or near-circular through holes. The industrial umbrellas referred to in this invention primarily emphasize the umbrella-shaped or arc-shaped spatial structure of such metal hole arrays, and are not limited to everyday fabric umbrellas.

[0023] 2. Lens mounting hole: This refers to the hole located on the hole array component of the industrial umbrella for mounting, positioning, or placing lenses. This hole may appear circular, elliptical, or approximately elliptical due to perspective in different camera images.

[0024] 3. Hole Array: This refers to a collection of holes formed on the surface of a metal umbrella by multiple lens placement holes. The holes can be arranged in rows, columns, arcs, or local regular arrays. Due to the large number of holes and their similar appearance, cross-view matching can easily lead to confusion between adjacent holes.

[0025] 4. Holes to be Placed and Holes Already Placed: Holes to be placed refer to holes that have not yet been fitted with lenses and require subsequent lens placement; holes already placed refer to holes that already contain lenses or transparent components. Placed holes may appear as highlights, transparent edges, reflections, or partial occlusion in the image. This invention can mark or filter placed holes.

[0026] 5. Lens placement target pose: refers to the target information used to control the robot, robotic arm, nozzle or other actuator to place the lens, including at least the center position of the hole, and preferably also the local normal of the hole and the radius of the hole.

[0027] 6. Multi-view image: refers to an image obtained by at least two cameras from different directions capturing the same industrial umbrella aperture array component. In one embodiment, a combination of multiple cameras, such as a center camera, a left camera, a right camera, and a top camera, may be used.

[0028] 7. Dense point cloud: refers to the set of three-dimensional points on the surface of a metal hole array obtained through multi-view three-dimensional reconstruction methods. Each point contains three-dimensional coordinates and is used to characterize the spatial morphology of the hole and the surrounding metal surface.

[0029] 8. COLMAP: A 3D reconstruction tool used to estimate camera pose and generate sparse 3D models from multi-view images. In this paper, it can be understood as the multi-view geometric reconstruction module.

[0030] 9. OpenMVS: A multi-view stereo dense reconstruction tool for generating dense 3D point clouds based on camera pose and images. In this article, it can be understood as the dense point cloud generation module.

[0031] 10. YOLO: An object detection or instance segmentation model used to detect hole regions in images and output hole candidate regions, masks, or boundary information.

[0032] 11. Zernike edge detection: refers to a method that uses Zernike moments to detect sub-pixel edges based on local grayscale changes in an image, and is used to improve the accuracy of hole edge extraction.

[0033] 12. Point cloud fingerprint: This refers to the set of 3D points or the set of 3D point numbers associated with each 2D hole region after a dense 3D point cloud is projected onto images from various cameras. Different 2D detection results from different perspectives belonging to the same real hole will share many identical or adjacent 3D points; therefore, the features of this point cloud set can be used as a basis for cross-view matching.

[0034] 13. Triangulation: This refers to determining the center position of the hole in three-dimensional space based on the two-dimensional center point of the same hole from multiple camera perspectives and the camera pose.

[0035] 14. Hungarian Matching Algorithm: A combinatorial optimization algorithm for solving task assignment problems, it can find the optimal matching scheme that minimizes the total cost or maximizes the total benefit in polynomial time. In this scheme, it is used to solve the cross-view hole matching cost matrix, so that the one-to-one correspondence between holes in each viewpoint is globally optimal.

[0036] 15. Global Matching Algorithm: This algorithm considers the overall cost of all candidate matching relationships and obtains the globally optimal matching scheme through global optimization (such as the Hungarian algorithm, graph cut, maximum flow, etc.). In this context, it refers to globally optimal solution methods similar to the Hungarian algorithm.

[0037] 16. Detection Model: A deep learning model or traditional computer vision method used to detect and locate target objects in an image. In this scheme, it specifically refers to the model used to detect hole regions. YOLO can be used, as well as Mask R-CNN, U-Net, instance segmentation networks, traditional image thresholding segmentation, or edge detection methods.

[0038] 17. Sparse 3D Model: A 3D model composed of a sparse feature point cloud and camera pose. Unlike dense point clouds (where each pixel corresponds to a 3D point), sparse models only contain 3D points corresponding to salient feature points in the image (such as corner points and edge points). They have a smaller number of points but higher computational efficiency. In this scheme, they serve as a pre-step for dense reconstruction to estimate camera extrinsic parameters. Sparse 3D models can be generated using COLMAP.

[0039] 18. Hole Detection Model: A model used to detect and locate hole regions in images. In this approach, the model outputs a candidate region or mask for each hole, which is then refined through edge extraction and ellipse fitting to determine the hole center position. YOLO, Mask R-CNN, U-Net, instance segmentation networks, etc., can be used.

[0040] This application takes into account that during the lens assembly process of industrial umbrellas or umbrella-shaped metal hole array components, the metal substrate typically has a large number of circular holes for mounting or placing lenses. To achieve automated assembly, the robot needs to obtain information such as the three-dimensional center position of each hole to be placed, the normal direction of the local metal surface where the hole is located, and the hole size. For process scenarios where the placement error requirement is no greater than 0.5 mm, relying solely on manual teaching or two-dimensional vision positioning is insufficient to reliably meet the requirements.

[0041] These workpieces differ from ordinary planar perforated plates: they have an overall arc-shaped, fan-shaped, or umbrella-shaped spatial structure, and the holes will produce obvious perspective distortion under different camera angles; the metal surface has strong reflective properties, which easily forms bright spots, dark edges, or local saturation areas near the hole edges; some holes may have lenses placed inside, which will produce transparent boundaries, high-light reflections, or local occlusions, making hole edge detection more difficult. Furthermore, these holes are usually densely distributed in an array or near-array pattern, with adjacent holes having similar appearances. If the cross-viewpoint correspondence is inaccurate, different holes can easily be mistakenly matched as the same hole, leading to incorrect 3D positioning results.

[0042] Furthermore, for densely packed hole arrays, the number of holes from different viewpoints is large, their appearance is similar, and their arrangement rules are close. Matching based solely on two-dimensional center distance, detection box size, major and minor axes of the ellipse, or adjacent relationships in the image can easily lead to misidentifying adjacent holes as the same hole. Once this type of mismatch occurs, even if triangulation or reprojection optimization is subsequently applied, incorrect 3D hole positions will still be obtained. Therefore, reliably establishing matching relationships for identical holes across different viewpoints is a key issue in multi-view 3D hole location.

[0043] Existing solutions typically include manual teaching, single-camera 2D vision positioning, structured light camera or depth camera measurement, and conventional multi-view 3D reconstruction. Manual teaching requires operators to teach the hole positions point by point, which is inefficient. When the workpiece's installation position, batch shape, or fixture status changes, re-teaching is often necessary. Single-camera 2D vision positioning can only obtain the hole positions in the image plane, making it difficult to directly obtain the hole depth and local normal, and cannot fully support the 3D pose required for lens placement.

[0044] Structured light cameras or depth cameras can directly acquire depth information within a certain range, and theoretically can be used for 3D measurement of holes. However, limitations still exist in scenarios involving metal umbrella-shaped hole arrays. On the one hand, high-precision structured light equipment is expensive and has high requirements for installation space, measurement distance, field of view, and on-site calibration conditions. On the other hand, strong reflections from metal surfaces, dark cavities inside holes, and transparent reflections from placed lenses can all cause localized loss of depth data, unstable depth jumps at hole edges, or incomplete edge contours. For lens placement tasks that require outputting the center, normal, and radius of each lens in a large number of holes, relying solely on structured light or a single depth measurement is still insufficient to reliably obtain hole poses that meet high-precision placement requirements.

[0045] While conventional multi-view 3D reconstruction can recover the point cloud of a workpiece surface based on multi-camera images, it is still prone to mismatch between adjacent holes if it lacks a two-dimensional precision detection mechanism for dense hole arrays and a cross-view matching mechanism for corresponding holes. If the point cloud of hole edges obtained from a single reconstruction is sparse or has a lot of noise, it will also affect the estimation accuracy of hole center, normal and radius.

[0046] Therefore, a 3D localization method is needed that can combine multi-view images, dense point clouds, 2D hole precision detection, and cross-view matching mechanism to stably output the hole center, normal, and radius required for lens placement, providing a reliable foundation for subsequent high-precision robot placement.

[0047] The relevant technologies have at least the following shortcomings: 1. Manual teaching is inefficient and has poor adaptability.

[0048] Manual teaching requires establishing placement points for each hole or type of workpiece. When workpiece batches, fixture positions, hole array postures, or assembly states change, the placement points need to be re-taught or corrected, making it difficult to meet the batch and automated assembly requirements of industrial umbrella hole array components.

[0049] 2. Single-view two-dimensional positioning cannot obtain reliable three-dimensional hole positions.

[0050] Single-camera images can only provide two-dimensional planar information and cannot directly obtain the depth of holes and the normal of local metal surfaces. For holes in arc-shaped, fan-shaped, or umbrella-shaped spatial structures, relying solely on the two-dimensional center point cannot meet the requirements for the three-dimensional center position and placement orientation of the lens.

[0051] 3. Structured light camera or depth camera solutions have shortcomings in cost and on-site adaptability.

[0052] High-precision structured light cameras are typically expensive and have stringent requirements regarding installation distance, field of view, projection angle, and calibration conditions. In scenarios with strong metallic reflections, deep cavities with holes, or reflections from transparent lenses, structured light or depth measurements are prone to problems such as missing depth at hole edges, increased noise, or discontinuous contours, making it difficult to stably extract the center, normal, and radius of each hole in a dense array of holes.

[0053] 4. In dense aperture array scenarios, cross-view mismatch is prone to occur.

[0054] Metal umbrellas often have numerous holes with similar appearances. Matching holes based solely on 2D image position, detection box size, or circular shape can easily lead to misidentifying adjacent holes as the same hole. Once a cross-view matching error occurs, the subsequent 3D hole positions obtained through triangulation will show significant deviations.

[0055] 5. Metallic reflections can interfere with the detection of hole edges.

[0056] Highlights, dark edges, localized saturation, and reflective textures on metal surfaces can lead to partial missing edges, false detections, or center misalignment of holes. Directly using the center of the detection frame or a rough segmentation result as the hole center is insufficient to meet the requirements for high-precision lens placement.

[0057] 6. Single-time dense reconstruction of point clouds contains random noise and local missing data.

[0058] The edge region of the hole is usually narrow. A single dense reconstruction may result in sparse, discontinuous or noisy edge point clouds due to image matching errors, reflections, occlusions or insufficient viewpoints, which will affect the estimation of the hole center, normal and radius.

[0059] 7. Existing solutions often only output point coordinates and lack hole pose information for placement facing the lens.

[0060] Lens placement requires not only the coordinates of the hole's center, but also the normal direction of the local metal surface where the hole is located and the hole's dimensions. Outputting only a single 3D point is insufficient to fully support robot placement posture planning and subsequent high-precision compensation.

[0061] In view of this, this application provides a method and related equipment for multi-view three-dimensional hole location of industrial umbrella hole array based on point cloud fingerprint matching. In cases of metal reflection, dense holes, similar hole appearance, and incomplete hole edges, this solution does not rely on simple two-dimensional position or hole number for cross-view matching. Instead, it uses dense point cloud projection to form hole point cloud fingerprints and determines the matching relationship of holes with the same name based on a common three-dimensional point set. This outputs the three-dimensional center, local normal, radius, and hole state of the hole, providing initial three-dimensional hole positions for subsequent precise hole location and robot coordinate compensation.

[0062] Figure 1 This is an optional flowchart of a multi-view three-dimensional hole location method for industrial umbrella hole arrays based on point cloud fingerprint matching provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0063] Step S101: Acquire multi-view images output by multiple cameras arranged in different directions, and label the camera intrinsic parameters for each image; Step S102: Perform sparse reconstruction and dense reconstruction based on multi-view images with camera intrinsic parameters to obtain dense 3D point clouds and extrinsic parameters of each camera. Step S103: Edge extraction and ellipse fitting are performed on the candidate regions output by the detection model in the images from each viewpoint to obtain the two-dimensional center, ellipse parameters and candidate regions of each hole. Step S104: Project the dense 3D point cloud onto the image plane of each view using camera intrinsic and extrinsic parameters, and record the 3D point numbers that fall into the candidate area of ​​the hole as the point cloud fingerprint of the hole. Step S105: Compare the point cloud fingerprints of each hole from different perspectives, count the number of identical 3D point numbers between the point cloud fingerprints and construct a matching cost matrix to solve for the cross-view corresponding hole relationship. Step S106: Based on the correspondence, establish a spatial line of sight by combining multiple two-dimensional centers of the same-named holes with camera extrinsic parameters, solve the least squares intersection point to obtain the initial value of the three-dimensional center, and obtain the three-dimensional center of the hole through reprojection residual optimization. Step S107: Based on the three-dimensional center of the hole, search the neighboring point cloud in the dense three-dimensional point cloud to obtain the unit normal. Based on the ellipse parameters of each viewpoint and the observation angle, weighted fuse to obtain the radius of the hole. Output the three-dimensional center of the hole, the unit normal, the radius, and the hole status label.

[0064] Steps S101 to S107 as illustrated in this embodiment ensure that each image has an independent projection reference through multi-view image acquisition and camera intrinsic parameter annotation. Dense 3D point clouds with spatial coordinates and camera extrinsic parameters are generated through sparse and dense reconstruction to lay the data foundation for subsequent 3D spatial calculations. Subpixel-level 2D centers and candidate regions of each hole are obtained through 2D edge extraction and ellipse fitting to improve 2D observation accuracy. The dense 3D point cloud is projected onto the image planes of each viewpoint, and the 3D point numbers falling into the candidate regions are recorded as point cloud fingerprints. The 2D detection results are correlated with the 3D spatial regions to provide spatial constraints for cross-view matching. A matching cost matrix is ​​constructed and solved by statistically analyzing the number of shared 3D point numbers among the point cloud fingerprints. The same spatial region is projected from the same set of 3D points under different viewpoints. This method achieves accurate correspondence of holes with the same name across different viewpoints, avoiding the mismatch problem of adjacent similar holes that easily occurs when matching based solely on two-dimensional position or hole order. By triangulating multiple two-dimensional centers of holes with the same name in conjunction with camera extrinsic parameters to obtain initial three-dimensional center values, and then optimizing through reprojection residuals and removing abnormal viewpoints, multi-view redundant observation is used to suppress single-view detection errors, thereby obtaining stable and reliable three-dimensional centers of holes. By searching the neighborhood point cloud around the three-dimensional center of the hole and fitting a plane to obtain the unit normal to meet the pose requirements of lens placement, and by weighting and fusing the pixel radii in the ellipse parameters of each viewpoint to obtain the accurate hole radius, the three-dimensional center, unit normal, radius, and status marker of each hole are finally output as the target pose for lens placement, realizing high-precision three-dimensional positioning of each lens placement hole on a dense hole array of industrial umbrellas.

[0065] In some embodiments, step S101 may include, but is not limited to, steps S111 to S112: Step S111: Arrange multi-directional cameras around the industrial umbrella hole array component, and use the multi-directional cameras to pre-calibrate and store their respective focal length, principal point, radial distortion coefficient and tangential distortion coefficient as the original camera intrinsic parameters; the multi-directional cameras include a center camera, a left camera, a right camera and a top camera. In step S112, during acquisition, multiple cameras simultaneously capture images. The camera corresponding to each image is identified based on the image file name or camera number, and the corresponding original camera intrinsic parameters are written into the image's metadata to obtain multi-view images.

[0066] In steps S111 to S112 of some embodiments, four cameras—center, left, right, and top—are arranged around the umbrella. Each camera has its focal length, principal point, and distortion coefficient pre-calibrated as original camera intrinsic parameters. During acquisition, the camera is identified according to the file name or number, and the corresponding intrinsic parameters are written into the image metadata. In this embodiment, each image carries its own independent camera intrinsic parameters instead of sharing the same model, avoiding the accumulation of projection errors caused by lens differences among multiple cameras. This ensures that feature point mapping in subsequent sparse reconstruction, pixel-level stereo matching in dense reconstruction, and the conversion of 3D points to pixel coordinates in point cloud projection are all based on accurate camera models, providing a fundamental guarantee for overall positioning accuracy.

[0067] In some embodiments, step S102 may include, but is not limited to, steps S201 to S202: Step S201: Based on the multi-view images with camera intrinsic parameters, feature extraction and feature matching are performed, and sparse reconstruction is performed according to the matching results to obtain the camera extrinsic parameters of each camera and obtain a sparse 3D model. Step S202: Dense reconstruction of multi-view images based on sparse 3D model to obtain dense 3D point cloud; camera intrinsic and camera extrinsic parameters are used to project each 3D point in the dense 3D point cloud onto the image plane of each view.

[0068] In steps S201 to S202 of some embodiments, feature extraction and matching are performed on multi-view images based on camera intrinsic parameters. Sparse reconstruction is then performed to obtain extrinsic parameters of each camera and a sparse 3D model. Dense reconstruction is then performed on the multi-view images based on the sparse model to obtain a dense 3D point cloud. By first estimating the camera pose and sparse point cloud structure through sparse reconstruction, and then using these as constraints for dense reconstruction, both reconstruction efficiency and point cloud density are ensured. The generated dense 3D point cloud has spatial coordinates, which, together with the camera intrinsic and extrinsic parameters, are used to subsequently project each 3D point onto the image plane of each view to construct a point cloud fingerprint, providing spatial geometric constraints for cross-view matching.

[0069] In some embodiments, step S103 may include, but is not limited to, steps S301 to S305: Step S301: Use the YOLO target detection model to detect holes in images from various viewpoints and output candidate regions or masks for each hole. Step S302: Expand the preset pixel boundary of each candidate region outward according to the actual size of each hole and crop it. Perform local histogram equalization on the cropped region to reduce the grayscale difference caused by metal reflection to a preset range, and obtain the enhanced region. Step S303: Extract the hole edge points in the enhanced region and use Zernike moments for sub-pixel edge localization to obtain a sub-pixel level edge point set; Step S304: Perform ellipse fitting on the sub-pixel level edge point set to obtain the two-dimensional center, major axis, minor axis and rotation angle of each hole as ellipse parameters. Step S305: Select the candidate region, mask, or elliptical neighborhood constructed based on elliptical parameters as the hole candidate region.

[0070] In steps S301 to S305 of some embodiments, YOLO is used to detect hole output candidate regions or masks. After expanding the boundaries and cropping the candidate regions, local histogram equalization is performed to reduce grayscale differences caused by metal reflection. Edge points are extracted in the enhanced regions and Zernike moments are used for subpixel edge localization. Ellipse fitting is performed on the subpixel edge point set to obtain the two-dimensional center, major and minor axes, and rotation angle. The candidate regions, masks, or elliptical neighborhoods are used as hole candidate regions. In this embodiment, YOLO is used to quickly obtain the approximate location of the hole, and then the center accuracy is improved to the subpixel level through subpixel edge localization and ellipse fitting. This avoids the systematic error introduced by simply relying on the center of the detection box as the hole center. At the same time, grayscale enhancement reduces the interference of metal reflection on edge extraction, improving the robustness and accuracy of two-dimensional hole detection and providing more accurate two-dimensional observations for subsequent triangulation localization and radius estimation.

[0071] In some embodiments, step S104 may include, but is not limited to, steps S401 to S402: Step S401: For each 3D point in the dense 3D point cloud, project the 3D point onto the image plane of each viewpoint according to the camera intrinsic and extrinsic parameters, and determine whether the projected point falls into the candidate area of ​​the preset hole in the preset viewpoint image. If it falls into the candidate area, add the number of the 3D point to the set of 3D point numbers corresponding to the hole. Step S402: After traversing all 3D points in the dense 3D point cloud and all holes in all viewpoint images, the set of 3D point numbers corresponding to each hole is used as the point cloud fingerprint of the hole.

[0072] In steps S401 to S402 of some embodiments, the hole detection results in the two-dimensional image are mapped to the real area in the three-dimensional space through projection association. The same real hole shares a large number of the same three-dimensional point numbers under different viewpoints because it is projected from the same spatial area. The number of shared numbers between different holes is significantly smaller. This point cloud fingerprint provides a discrimination basis for subsequent cross-view matching that does not depend on the two-dimensional position or hole order.

[0073] In some embodiments, step S105 may include, but is not limited to, steps S501 to S505: Step S501: Form a hole pair by taking one hole from each of any two different perspectives, perform an intersection operation on the three-dimensional point number sets corresponding to the two holes, and count the number of identical three-dimensional point numbers in the intersection as the number of common points. Step S502: Calculate the ratio of the number of common points to the minimum total number of points in the two point cloud fingerprints; Step S503: If the number of common points exceeds a preset threshold, the hole pair is marked as a candidate hole pair with the same name, and the number or proportion of common points is used as the matching cost of the candidate hole pair with the same name. Step S504: After traversing all hole combinations between all viewpoints, construct a cross-viewpoint matching cost matrix with each hole as a row and each hole as a column; each element in the matching cost matrix is ​​the matching cost value or null value of the corresponding hole pair. Step S505: The Hungarian matching algorithm is used to solve the matching cost matrix globally to obtain the one-to-one correspondence between holes from different viewpoints as the cross-viewpoint correspondence of holes with the same name.

[0074] In steps S501 to S505 of some embodiments, obviously mismatched hole pairs are filtered out by setting a threshold for the number of common points. The matching problem between multiple perspectives is transformed into a globally optimal allocation problem by using the matching cost matrix. The Hungarian algorithm is used to ensure the consistency of the matching relationship between each perspective, avoiding the local optimum problem that may be introduced by greedy matching.

[0075] In some embodiments, step S106 may include, but is not limited to, steps S601 to S604: Step S601: For the two-dimensional centers of multiple viewpoints that have been matched as the same real hole, establish a spatial line of sight from each camera center through the two-dimensional center based on the camera extrinsic and intrinsic parameters of each viewpoint. Step S602: Solve for the least squares intersection of multiple spatial lines of sight in the same coordinate system, and use it as the initial value of the three-dimensional center of the hole; Step S603: Reproject the initial value of the three-dimensional center back to the image plane of each viewpoint according to the camera intrinsic and extrinsic parameters of each viewpoint, and calculate the reprojection residual between the projection point and the corresponding two-dimensional center of the viewpoint. Step S604: If the reprojection residual of a certain view exceeds the preset threshold, the view is determined to be an abnormal observation. The view is then removed, and the least squares intersection point is re-solved using the remaining view to obtain the optimized three-dimensional center of the hole.

[0076] In steps S601 to S604 of some embodiments, the statistically optimal initial value of the three-dimensional center is obtained by using the observation values ​​of all perspectives through least squares solution. Abnormal perspectives with large detection errors are identified and eliminated by reprojection residual calculation, thereby avoiding the influence of single-view hole detection offset or lens reflection interference on the positioning results, thus obtaining a stable and reliable three-dimensional center.

[0077] In some embodiments, step S107 may include, but is not limited to, steps S701 to S707: Step S701: Using the three-dimensional center of the hole as a reference, search for the neighboring point cloud within a preset radius in the dense three-dimensional point cloud; Step S702: Select points from the neighborhood point cloud that are more than the radius of the hole but less than the preset outer diameter threshold and whose distance from the three-dimensional center of the hole is greater than the radius of the hole. Form a ring-shaped point cloud. The ring-shaped point cloud covers the metal surface near the edge of the hole and avoids the hollow area inside the hole. Step S703: Perform plane fitting on the annular point cloud, and use the normal of the fitted plane as the unit normal of the local metal surface where the hole is located. Step S704: Extract the pixel radius of the hole in each viewpoint image from the ellipse parameters of each viewpoint image, and convert the pixel radius into the physical radius estimate of each viewpoint by combining the focal length of the camera in each viewpoint and the three-dimensional center depth of the hole. Step S705: Calculate the angle between the direction vector from the optical center of each camera to the three-dimensional center of the hole and the unit normal as the observation angle; Step S706: Weighted fusion of the physical radius estimates from each viewpoint is performed, with higher weight given to smaller viewing angles, to obtain the fused hole radius; Step S707: Output the three-dimensional hole position information by taking the three-dimensional center of the hole, the unit normal, the fused hole radius, and the hole status mark indicating whether a lens has been placed in the hole.

[0078] In steps S701 to S707 of some embodiments, the ring-shaped point cloud screening ensures that the points participating in the plane fitting are concentrated on the metal surface near the edge of the hole. The weights are set in reverse order of distance so that the points closer to the edge of the hole contribute more, thus improving the accuracy of the normal estimation. The weighted fusion of the radius makes the observation value of the hole normal more weighted the more direct the viewpoint, making full use of the high-quality information in multi-view observation. The status label enables the downstream system to distinguish between the hole to be placed and the hole that has already been placed, avoiding duplicate placement or collision.

[0079] This application also provides an industrial umbrella hole array multi-view three-dimensional hole position positioning system based on point cloud fingerprint matching, used to implement the method described above. The system includes: The image acquisition module is used to acquire multi-view images output by multiple cameras arranged in different directions, and to annotate the camera intrinsic parameters for each image; The 3D reconstruction module is used to perform sparse and dense reconstruction based on multi-view images with camera intrinsic parameters to obtain dense 3D point clouds and extrinsic parameters of each camera. The two-dimensional detection module is used to extract edges and fit ellipses to candidate regions output by the detection model in images from various perspectives, so as to obtain the two-dimensional center, ellipse parameters and candidate regions of each hole. The point cloud fingerprint construction module is used to project dense 3D point clouds onto image planes from various viewpoints using camera intrinsic and extrinsic parameters, and record the 3D point numbers that fall into the candidate area of ​​the hole as the point cloud fingerprint of the hole. The cross-view matching module is used to compare the point cloud fingerprints of each hole from different viewpoints, count the number of identical 3D point numbers between point cloud fingerprints and construct a matching cost matrix to solve for the correspondence between holes with the same name across viewpoints. The triangulation positioning module is used to establish a spatial line of sight by combining multiple two-dimensional centers of the same hole with camera extrinsic parameters according to the correspondence, solve the least squares intersection point to obtain the initial value of the three-dimensional center, and obtain the three-dimensional center of the hole through reprojection residual optimization. The pose estimation module is used to search for the neighboring point cloud in the dense 3D point cloud based on the 3D center of the hole to obtain the unit normal, and to obtain the hole radius by weighted fusion based on the ellipse parameters and observation angle of each viewpoint. The output module is used to output the three-dimensional center, unit normal, radius, and status marker of the hole.

[0080] As an optional implementation, this invention provides a multi-view 3D hole location method for industrial umbrella hole arrays based on point cloud fingerprint matching. This method is based on multi-view images and dense 3D point clouds, with point cloud fingerprint construction and cross-view matching of corresponding holes as core steps, outputting initial 3D hole positions for subsequent precise hole location and robot placement compensation. (Reference) Figure 2 The method includes the following steps: Among them, the point cloud fingerprint construction in S4 and the cross-view same-name hole matching in S5 are the core steps that distinguish this invention from ordinary multi-view 3D reconstruction, ordinary 2D hole matching and simple triangulation positioning; S1-S3 are used to obtain multi-view observation, dense point cloud and 2D hole region, and S6-S8 are used to output the initial 3D hole position based on the matching result.

[0081] S1: Acquire multi-view images of the hole array components of industrial umbrellas, and inject corresponding camera intrinsic parameters into different camera images according to camera number or image name; S2, based on multi-view images, performs sparse and dense reconstruction to obtain dense three-dimensional point clouds on the surface of the metal hole array and the corresponding camera extrinsic parameters; S3 performs two-dimensional detection on lens placement holes in images from various perspectives, obtaining candidate regions, masks, edges, two-dimensional centers, and elliptical parameters for each hole, and can mark holes to be placed, holes already placed, or abnormal holes. S4 projects the dense 3D point cloud onto the image plane of each viewpoint, and associates the corresponding set of 3D points within each 2D hole region to form the point cloud fingerprint of the hole; S5. Based on the point cloud fingerprint similarity between two-dimensional holes from different perspectives, determine the matching relationship of the same-named holes across different perspectives. S6, triangulation is performed based on the two-dimensional center of the hole in multiple viewpoints and the camera pose, and the three-dimensional center of the hole is obtained by optimizing the reprojection error. S7. The normal of the local metal surface where the hole is located is estimated based on the point cloud of the hole's neighborhood, and the radius of the hole is estimated by combining the multi-view elliptical scale. S8 outputs initial 3D hole position information, which includes hole number, 3D center, unit normal, radius, hole status marker and related information, and is used for subsequent hole position fine positioning, robot coordinate transformation and placement compensation.

[0082] S1: In one embodiment, multiple industrial cameras, such as a center camera, a left-side camera, a right-side camera, and a top camera, are arranged around the industrial umbrella-shaped hole array component. The multiple cameras acquire workpiece images synchronously or nearly synchronously from different directions. Each camera is pre-calibrated to obtain focal length, principal point, radial distortion, and tangential distortion parameters.

[0083] The system identifies the image source based on image name, camera number, or camera role, and injects the corresponding camera intrinsic parameters into the 3D reconstruction module. This avoids reconstruction errors caused by multiple cameras sharing the same camera model and improves the stability of multi-view reconstruction. The number of cameras is not limited to four; it can also be two, three, five, or more, as long as it can cover the area of ​​the aperture array to be located from multiple perspectives.

[0084] refer to Figure 3 Top camera view, Figure 4 Left-side camera view Figure 5 Center camera view Figure 6 The camera view on the right, and the overall device diagrams from different angles in Figures 7(a) and 7(b).

[0085] In a preferred embodiment, the multi-camera images may include a center view, a left view, a right view, and a top view; wherein the center, left, and right views are used for two-dimensional hole detection and cross-view matching, and the top view can participate in three-dimensional reconstruction or provide additional spatial constraints.

[0086] S2: The system first performs feature extraction and feature matching based on multi-view images, estimates the extrinsic parameters between different cameras, and obtains a sparse 3D model through bundle adjustment. Subsequently, based on the sparse model, the images are distorted and reconstructed to obtain a dense point cloud on the surface of the industrial umbrella hole array component.

[0087] To reduce random noise and local point cloud gaps in a single dense reconstruction, this invention can perform multiple rounds of dense reconstruction. Multiple rounds of reconstruction can employ any of the following methods: (1) Fix the sparse reconstruction results and camera pose, perform dense reconstruction multiple times, keep the dense point cloud in the same coordinate system for stitching, and complete the fusion by downsampling.

[0088] (2) In each round, complete sparse reconstruction and dense reconstruction are re-executed, and the results of multiple rounds are merged in the same coordinate system.

[0089] The above methods can obtain a more stable and denser point cloud on the surface of the metal hole array, which is especially helpful in enhancing the point cloud coverage in the edge area of ​​the hole.

[0090] The aforementioned multi-round dense point cloud fusion is mainly used as a pre-enhancement step for point cloud fingerprint construction. Its purpose is to improve the point cloud coverage in the edge region of the hole, make the set of three-dimensional points associated with each two-dimensional hole more stable, and thus improve the reliability of subsequent cross-view matching of the same hole.

[0091] S3: For each viewpoint image, the system uses a hole detection model to obtain candidate hole regions or hole masks. To improve the accuracy of the 2D hole center, this invention does not directly use the center of the detection box as the hole center, but further performs image enhancement, edge extraction, and ellipse fitting within the candidate region. (Reference) Figure 8 Intermediate results of two-dimensional hole detection and ellipse fitting.

[0092] Specifically, it may include: 1. Trim the candidate area for holes and expand it outwards by a certain boundary according to the size of the holes; 2. Enhance the grayscale of the candidate region, for example, through local histogram equalization, to reduce the grayscale differences caused by metallic reflections and dark hole areas; 3. Extract hole edge points based on mask, edge response, and bright / dark region distribution; 4. Improve edge point localization accuracy by employing Zernike edge detection or other sub-pixel edge detection methods; 5. Perform ellipse fitting on the edge points to obtain the ellipse center, major axis, minor axis, rotation angle, and fitting error; 6. Abnormal detection results are eliminated based on hole area, ellipse major-minor axis ratio, normalized fitting error, boundary distance, specular area, and lens detection results. Holes to be placed and holes already placed can be marked.

[0093] For holes where lenses have already been placed, the system can output a hole status flag, allowing subsequent placement processes to skip the hole or avoid it; for holes to be placed, the system continues to perform 3D positioning. This yields more stable 2D center and scale information of the holes from various viewpoints.

[0094] The aforementioned two-dimensional hole detection results are mainly used to define the associated area of ​​the point cloud projection, that is, to determine which three-dimensional points belong to a certain two-dimensional hole area. This invention does not use a single two-dimensional detection center as the final three-dimensional positioning basis, but combines the two-dimensional detection results with the dense point cloud projection relationship. Refer to Figures 9(a), 9(b), and 9(c) for the actual dense point cloud projection diagrams.

[0095] S4: One of the key points of this invention is that it does not rely solely on two-dimensional image features for cross-view hole matching, but instead utilizes the projection relationship of dense three-dimensional point clouds in different viewpoints to construct a three-dimensional point cloud fingerprint for each two-dimensional hole.

[0096] Specifically, for each 3D point in the dense point cloud, it is projected onto the respective camera image planes based on camera intrinsic and extrinsic parameters. If the projected point falls within a candidate region, mask region, or elliptical neighborhood of a 2D hole, then the 3D point is associated with that 2D hole. For each 2D hole, the system records the set of 3D points associated with it; this set of 3D points constitutes the point cloud fingerprint of that 2D hole. (Reference) Figure 10 A cross-view matching diagram of holes with the same name.

[0097] Since the same real-world hole appears as the same spatial region from different camera perspectives, its corresponding 2D detection results will be associated with many identical or adjacent 3D points. Conversely, the 3D point sets between different real-world holes overlap less. Therefore, point cloud fingerprints can be used for cross-view matching of holes with the same name.

[0098] S5: After obtaining the point cloud fingerprint of each hole in each viewpoint, the system calculates the matching cost between holes from different viewpoints. The matching cost can be determined based on one or more of the following metrics: 1. The number of common points in the set of three-dimensional points corresponding to two two-dimensional holes; 2. The proportion of common points to the size of their respective point sets; 3. The center distance between two point sets in three-dimensional space; 4. Neighborhood consistency between two candidate holes in the spatial reconstruction model; In one implementation, if the number of common 3D points associated with two candidate holes exceeds a preset threshold, they are considered to likely belong to the same real hole. The system can construct a cross-view hole matching cost matrix and use the Hungarian matching method or other global matching methods to obtain a one-to-one correspondence between different viewpoints.

[0099] This method can reduce the risk of mismatches even if multiple holes are densely arranged and have similar appearances in a two-dimensional image, using three-dimensional point cloud fingerprints.

[0100] Therefore, the cross-view matching of the present invention does not rely solely on hole number, left-right order in the image, or two-dimensional center distance, but instead utilizes the feature that the same real spatial region shares a set of three-dimensional points under different viewpoints to establish a matching relationship for holes with the same name.

[0101] S6: For the multi-view 2D centers that have been matched to the same real hole, the system establishes spatial lines of sight originating from each camera center based on the camera pose and camera intrinsic parameters. By solving the least-squares intersection of multiple lines of sight, the initial value of the 3D center of the hole is obtained.

[0102] To improve positioning accuracy, the system further projects the 3D center back onto the image planes of each camera, calculates the reprojection error between the projected point and the observed 2D center, and employs a robust optimization method to refine the 3D center. Robust optimization can use a soft-threshold loss function to reduce the impact of abnormal observations on the results.

[0103] If the reprojection residual of a certain viewpoint is significantly greater than that of other viewpoints, the system can remove that viewpoint as an anomaly and then re-optimize. This method allows for the acquisition of a stable 3D center even when there are issues such as aperture detection offsets, mirror reflection interference, or localized occlusion at individual viewpoints.

[0104] S7: In order to meet the orientation requirements of lens placement, the present invention further estimates the normal of the local metal surface where the hole is located.

[0105] Specifically, a dense point cloud neighborhood within a certain range is searched near the 3D center of the hole, and a ring-shaped point cloud around the hole is selected. The ring-shaped point cloud preferentially covers the metal surface near the edge of the hole, while avoiding the void area inside the hole and noise points far away from the hole as much as possible. The system can fit a local plane or quadratic surface based on the spatial distribution of the neighborhood point cloud to obtain the normal of the local surface where the hole is located.

[0106] For normal estimation, methods such as weighted singular value decomposition, random sample consensus plane fitting, or weighted quadratic surface fitting can be used. The weights can be determined based on the distance from the point cloud to the hole center, so that points closer to the hole edge contribute more to the normal estimation.

[0107] The radius of the hole can be estimated based on the pixel radius of the multi-view ellipse, the camera focal length, the hole depth, and the viewing angle, and the radius estimates from multiple viewpoints are weighted and fused. The closer the viewpoint is to the normal direction of the hole, the higher the weight of its radius estimate.

[0108] Finally, the system outputs the initial three-dimensional hole position for each hole, including the three-dimensional center coordinates, unit normal vector, hole radius, and hole state. This is used for three-dimensional hole position positioning before lens placement and can also serve as input for subsequent local slicing fine positioning and robot coordinate compensation.

[0109] S8: The data output by this invention mainly includes: hole number; initial three-dimensional coordinates of the hole; local normal vector of the hole; hole radius; status markers indicating whether a lens has been detected or whether the hole can be placed.

[0110] Boundary Description: This invention primarily addresses the problem of initial 3D hole location under multi-view image conditions, outputting the 3D center, local normal, radius, and hole state of the hole. Subsequent local slice precision localization, arc-missing diagnosis, multi-candidate center gating, robot coordinate residual compensation, functional reference compensation, and row Z-homogenization after obtaining the initial 3D hole location can be used as processing steps in the subsequent lens placement system, but are not considered core protected content of this invention. References Figure 11 A true 3D output (center point, local normal, hole radius), for reference. Figure 12 The overall three-dimensional normal diagram.

[0111] As an optional implementation, in this embodiment, four industrial cameras are arranged above the umbrella and on its left, center, and right sides, respectively designated as the center camera, left-side camera, right-side camera, and top camera. Each camera has 20 megapixels and its internal parameters have been pre-calibrated.

[0112] The system acquires four images of the umbrella. First, it identifies the camera number corresponding to each image and writes the focal length, principal point, and distortion parameters for each different camera image. Then, it uses a multi-view reconstruction module to complete feature extraction, feature matching, sparse reconstruction, and dense point cloud generation.

[0113] To improve the point cloud density at the edge of the hole, the system performs multiple rounds of dense reconstruction while keeping the sparse reconstruction results constant. The multiple rounds of dense point cloud coordinates are then stitched together and fused using a voxel mesh to obtain the final dense point cloud.

[0114] Next, the system uses YOLO technology to perform 2D hole segmentation and detection on the images from the center camera, left camera, and right camera. After the detection model outputs a hole mask, the system performs edge enhancement, Zernike edge extraction, and ellipse fitting on the candidate hole regions. Candidate holes with excessive fitting errors, small areas, large areas, or close to the image boundary are discarded by the system.

[0115] The system then projects the fused dense point cloud onto three camera image planes used for hole detection. For each 2D hole, the system counts the 3D points falling into its elliptical neighborhood or mask region to form the point cloud fingerprint of that hole.

[0116] The system calculates the number of common 3D points among the point cloud fingerprints of holes from different viewpoints and constructs a matching cost matrix. A global matching algorithm is used to determine the detection results of holes belonging to the same real hole from each viewpoint. For successfully matched holes, the system triangulates the 2D centers from multiple viewpoints to obtain initial 3D center values.

[0117] The system then projects the 3D center back to each viewpoint, calculates the reprojection residuals, and refines the 3D center using robust least squares optimization. If the residual of a certain viewpoint exceeds a preset threshold, the observations from that viewpoint are removed and the system is re-optimized.

[0118] Finally, the system extracts the annular point cloud within the neighborhood of the hole center, fits the local surface normal of the hole, and estimates the hole radius using a multi-view elliptical scale. The system outputs the hole number, initial 3D center, normal, and radius. This output can be used for subsequent precise hole positioning, robot coordinate transformation, and placement compensation, but it is not directly equivalent to the final robot placement coordinates.

[0119] The beneficial effects of this invention are: 1. Improve cross-view matching accuracy in dense hole array scenarios. This invention constructs a point cloud fingerprint for each two-dimensional hole by projecting dense point cloud and determines the matching relationship of holes with the same name based on a common three-dimensional point set. Compared with methods that only use two-dimensional position or detection box features, this method can better avoid mismatch of adjacent holes.

[0120] 2. Improve the accuracy of 2D hole center detection in metallic reflective scenes. This invention further refines the mask, extracts edges, and fits ellipses based on the output of the detection model, so that the hole center no longer depends on the coarse detection box center, which is beneficial to improving the accuracy of subsequent 3D triangulation.

[0121] 3. Improve the stability of 3D reconstructed point clouds. This invention can enhance the coverage of point clouds at the edges of holes through multiple rounds of dense reconstruction and voxel fusion, reducing the impact of random noise and local defects in a single reconstruction on hole location.

[0122] 4. It can output complete hole pose information required for lens placement. This invention not only outputs the three-dimensional center of the hole, but also the local normal and radius of the hole, providing a basis for the center position, orientation, and dimensions of the lens for subsequent placement.

[0123] 5. Improved robustness to placed lenses, reflections, and localized detection anomalies. This invention utilizes multi-view observation, point cloud fingerprint matching, reprojection optimization, and abnormal viewpoint removal to prevent individual viewpoint detection errors from directly causing overall positioning failure.

[0124] 6. Applicable to lens placement processes with a diameter of no more than 0.5 mm. This invention provides stable three-dimensional hole position data, which can provide input for subsequent robot coordinate transformation and compensation, thereby serving high-precision lens placement.

[0125] Key points and protection points of this invention: In the three-patent layout, this invention is positioned as "front-end initial 3D hole position positioning". Its core protection points should prioritize point cloud fingerprint construction, matching of common 3D point sets, and output of initial 3D hole positions based on the matching results; subsequent hole position refinement and robot coordinate compensation are protected in separate cases.

[0126] 1. A method for constructing a point cloud fingerprint of a hole based on dense point cloud projection. The three-dimensional point cloud is projected onto the image plane of each viewpoint, and each two-dimensional hole is associated with a set of three-dimensional points or a set of three-dimensional point numbers. This set of three-dimensional points represents the real spatial region where the hole is located.

[0127] 2. A cross-view matching method for holes with the same name based on a common 3D point set. Holes from different viewpoints are not matched simply based on 2D image distance, hole order, or detection box size, but rather based on point cloud fingerprint similarity, the number of common 3D points, or the proportion of common points to determine the relationship between holes with the same name.

[0128] 3. A 3D center triangulation and reprojection optimization method driven by point cloud fingerprint matching results. After identifying holes with the same name through point cloud fingerprint matching, triangulation is performed using the 2D center from multiple viewpoints and the camera pose. The 3D center of the hole is then obtained through reprojection error optimization and outlier viewpoint removal.

[0129] 4. A multi-round dense point cloud fusion method for metal umbrella-shaped hole arrays. By fixing the sparse reconstruction results and performing multiple rounds of dense reconstruction, or by aligning the complete reconstruction results of multiple rounds through a unified coordinate system, a more stable point cloud of the metal hole array is obtained through fusion, providing more sufficient hole edge point cloud for point cloud fingerprint construction.

[0130] 5. Method for associating 2D hole detection results with point cloud projection regions. By defining the 3D point projection association range through hole candidate regions, masks, edges, or elliptical neighborhoods, the 2D detection results become regional constraints for point cloud fingerprint construction, rather than being used only as the final 2D center.

[0131] 6. Initial 3D aperture position output method for lens placement. The output includes not only the 3D center of the aperture, but also the local normal, aperture radius, and aperture state, making it a basis for subsequent precise aperture positioning and robot coordinate compensation.

[0132] 7. Low-hardware-cost multi-view 3D hole location solution based on industrial-grade RGB cameras. The system uses multiple industrial-grade RGB cameras to acquire multi-view images of metal umbrella-shaped hole array components. Through camera intrinsic parameter injection, sparse reconstruction, dense reconstruction, multi-round point cloud fusion, 2D hole precision detection, point cloud fingerprint construction, and cross-view matching, the initial location output of the 3D center, normal, and radius of the hole is achieved.

[0133] Other alternatives to the embodiments of this application: 1. Two-dimensional hole detection models are not limited to YOLO. Mask R-CNN, U-Net, instance segmentation networks, traditional image thresholding segmentation, or edge detection methods can also be used, as long as they can obtain hole candidate regions or hole edges.

[0134] 2. Edge localization methods are not limited to Zernike edges; Canny edges, Sobel edges, subpixel contour extraction, active contour models, or other edge detection methods can also be used.

[0135] 3. Ellipse fitting methods are not limited to least squares ellipse fitting. Random sampling consistency ellipse fitting, robust ellipse fitting, or ellipse parameter estimation methods based on mask moments can also be used.

[0136] 4. The 3D reconstruction method is not limited to COLMAP and OpenMVS. Other structured light, binocular vision, multi-view vision, depth camera or neural reconstruction methods can also be used, as long as the 3D point cloud of the umbrella surface and the camera pose can be obtained.

[0137] 5. Point cloud fingerprints can be derived from a three-dimensional set of point numbers, a histogram of point cloud spatial distribution, local point cloud centers, point cloud density, point cloud normal distribution, or a combination of the above features.

[0138] 6. Cross-perspective matching methods are not limited to Hungarian matching; greedy matching, graph matching, maximum flow matching, cluster matching, or learning-based matching methods can also be used.

[0139] 7. Multi-round point cloud fusion can employ voxel downsampling, statistical filtering, radius filtering, weighted averaging, confidence fusion, or other point cloud fusion methods.

[0140] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0141] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0142] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0143] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0144] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0145] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] This application provides a method, system, electronic device, storage medium, and program product for multi-view 3D hole location of industrial umbrella hole arrays based on point cloud fingerprint matching. It acquires multi-view images of industrial umbrella hole array components using multiple cameras arranged in different directions. Based on the multi-view images, it performs sparse and dense reconstruction to obtain dense 3D point clouds and extrinsic parameters of each camera. It performs 2D detection on holes in each view image to obtain the 2D center and candidate regions of each hole. It projects the dense 3D point cloud onto the image plane of each view and records the 3D point numbers falling into each candidate region as the point cloud fingerprint of that hole. It compares the point cloud fingerprints of each hole under different views and counts the number of shared 3D point numbers between fingerprints to construct a matching cost matrix to solve the correspondence relationship of holes with the same name across views. Based on this correspondence, it triangulates and optimizes the multiple 2D centers of holes with the same name in conjunction with camera extrinsic parameters to obtain the 3D center of the hole. Finally, it outputs the lens placement target pose of each hole.

[0147] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0148] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0149] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for multi-view three-dimensional hole location of industrial umbrella hole array based on point cloud fingerprint matching, characterized in that, The method includes the following steps: Collect multi-view images from multiple cameras positioned in different directions, and label the camera intrinsic parameters for each image; Sparse and dense reconstructions are performed on the multi-view images with the camera intrinsic parameters to obtain dense 3D point clouds and extrinsic parameters of each camera. Edge extraction and ellipse fitting are performed on the candidate regions output by the detection model in the images from different viewpoints to obtain the two-dimensional center, ellipse parameters and candidate regions of each hole; The dense 3D point cloud is projected onto image planes of various viewpoints using the camera intrinsic and extrinsic parameters, and the 3D point numbers falling into the hole candidate region are recorded as the point cloud fingerprints of the holes. The point cloud fingerprints of each hole under different viewpoints are compared, the number of identical 3D point numbers among the point cloud fingerprints is counted and a matching cost matrix is ​​constructed, and the correspondence between holes with the same name across viewpoints is obtained by solving the problem. Based on the correspondence, multiple two-dimensional centers of the same-named holes are combined with the camera extrinsic parameters to establish a spatial line of sight. The least squares intersection point is solved to obtain the initial value of the three-dimensional center. The three-dimensional center of the hole is obtained by reprojection residual optimization. The unit normal is obtained by searching the neighboring point cloud in the dense three-dimensional point cloud based on the three-dimensional center of the hole. The radius of the hole is obtained by weighted fusion based on the ellipse parameters and observation angles from each viewpoint. The three-dimensional center of the hole, the unit normal, the radius, and the hole status marker are output.

2. The method according to claim 1, characterized in that, The acquisition process collects multi-view images output by multiple cameras positioned in different directions, and labels each image with camera intrinsic parameters, including: Multi-directional cameras are arranged around the hole array component of the industrial umbrella. The multi-directional cameras are pre-calibrated and their focal length, principal point, radial distortion coefficient, and tangential distortion coefficient are stored as original camera intrinsic parameters. The multi-directional cameras include a center camera, a left camera, a right camera, and a top camera. During acquisition, the multi-angle cameras simultaneously capture images. The camera corresponding to each image is identified based on the image file name or camera number, and the corresponding original camera intrinsic parameters are written into the image's metadata to obtain multi-view images.

3. The method according to claim 1, characterized in that, The process of performing sparse and dense reconstruction based on the multi-view images with the camera intrinsic parameters to obtain a dense 3D point cloud and the extrinsic parameters of each camera includes: Feature extraction and feature matching are performed on the multi-view images with the camera intrinsic parameters. Sparse reconstruction is performed based on the matching results to obtain the camera extrinsic parameters of each camera and obtain a sparse 3D model. Based on the sparse 3D model, the multi-view image is densely reconstructed to obtain a dense 3D point cloud; the camera intrinsic parameters and the camera extrinsic parameters are used to project each 3D point in the dense 3D point cloud onto the image plane of each view.

4. The method according to claim 1, characterized in that, The process of edge extraction and ellipse fitting of candidate regions output by the detection model in images from various viewpoints to obtain the two-dimensional center, ellipse parameters, and candidate regions of each hole includes: The YOLO object detection model is used to detect holes in images from various viewpoints, and the candidate regions or masks for each hole are output. Each candidate region is expanded outward by a preset pixel boundary according to the actual size of each hole and then cropped. Local histogram equalization is performed on the cropped region to reduce the grayscale difference caused by metal reflection to a preset range, thus obtaining the enhanced region. The hole edge points are extracted in the enhanced region and sub-pixel edge localization is performed using Zernike moments to obtain a sub-pixel level edge point set. Ellipse fitting is performed on the sub-pixel level edge point set to obtain the two-dimensional center, major axis, minor axis and rotation angle of each hole as ellipse parameters; The candidate region, mask, or elliptical neighborhood constructed based on the elliptical parameters is used as the hole candidate region.

5. The method according to claim 1, characterized in that, The process of projecting the dense 3D point cloud onto image planes at various viewpoints using the camera intrinsic and extrinsic parameters, and recording the 3D points falling within the candidate hole region as the point cloud fingerprint of the hole, includes: For each 3D point in the dense 3D point cloud, the 3D point is projected onto the image plane of each viewpoint according to the camera intrinsic parameters and the camera extrinsic parameters. It is determined whether the projected point falls into the candidate area of ​​the hole of the preset hole in the preset viewpoint image. If it falls into the hole, the number of the 3D point is added to the set of 3D point numbers corresponding to the hole. After traversing all the three-dimensional points in the dense three-dimensional point cloud and all the holes in all viewpoint images, the set of three-dimensional point numbers corresponding to each hole is used as the point cloud fingerprint of the hole.

6. The method according to claim 1, characterized in that, The process involves comparing the point cloud fingerprints of each hole from different viewpoints, counting the number of identical 3D point numbers among the point cloud fingerprints, constructing a matching cost matrix, and solving for the cross-view correspondence of holes with the same name, including: A hole pair is formed by taking one hole from any two different perspectives. The intersection of the sets of 3D point numbers corresponding to the two holes is calculated, and the number of the same 3D point numbers in the intersection is taken as the number of common points. Calculate the proportion of the number of common points to the minimum total number of points in the two point cloud fingerprints; If the number of common points exceeds a preset threshold, the hole pair is marked as a candidate hole pair with the same name, and the number of common points or the ratio is used as the matching cost of the candidate hole pair with the same name. After traversing all hole combinations across all viewpoints, a cross-viewpoint matching cost matrix is ​​constructed with each hole as a row and each hole as a column; each element in the matching cost matrix is ​​the matching cost value or null value for the corresponding hole pair. The Hungarian matching algorithm is used to solve the matching cost matrix globally to obtain the one-to-one correspondence between holes from each viewpoint as the cross-viewpoint correspondence of holes with the same name.

7. The method according to claim 1, characterized in that, The process of establishing a spatial line of sight by combining multiple two-dimensional centers of the same-named holes with the camera extrinsic parameters according to the correspondence, solving for the least squares intersection points to obtain the initial value of the three-dimensional center, and optimizing the three-dimensional center of the hole through reprojection residuals includes: For the two-dimensional center of multiple viewpoints that have been matched as the same real hole, a spatial line of sight is established from each camera center through the two-dimensional center based on the camera extrinsic parameters and camera intrinsic parameters of each viewpoint. Find the least-squares intersection point of multiple spatial lines of sight in the same coordinate system, and use it as the initial value of the three-dimensional center of the hole; The initial value of the three-dimensional center is reprojected back to the image plane of each viewpoint according to the camera intrinsic and extrinsic parameters of each viewpoint, and the reprojection residual between the projection point and the two-dimensional center of the corresponding viewpoint is calculated. If the reprojection residual of a certain view exceeds a preset threshold, the view is determined to be an abnormal observation. The view is then discarded, and the least squares intersection point is re-solved using the remaining view to obtain the optimized three-dimensional center of the hole.

8. The method according to claim 1, characterized in that, The process involves searching for and fitting a neighboring point cloud in the dense 3D point cloud based on the 3D center of the hole to obtain the unit normal, weighting and fusing the ellipse parameters and viewing angles from each perspective to obtain the hole radius, and outputting the 3D center, unit normal, radius, and hole status marker, including: Using the three-dimensional center of the hole as a reference, search for neighboring point clouds within a preset radius in the dense three-dimensional point cloud; Points that are greater than the radius of the hole and less than a preset outer diameter threshold are selected from the neighborhood point cloud to form a ring-shaped point cloud; the ring-shaped point cloud covers the metal surface near the edge of the hole and avoids the hollow area inside the hole. The annular point cloud is fitted with a plane, and the normal of the fitted plane is used as the unit normal of the local metal surface where the hole is located. For the ellipse parameters of the hole in the image from each viewpoint, the pixel radius of each viewpoint is extracted, and the pixel radius is converted into a physical radius estimate of each viewpoint by combining the focal length of the camera at each viewpoint and the three-dimensional center depth of the hole. The angle between the direction vector from the optical center of each camera to the three-dimensional center of the hole and the unit normal is calculated as the observation angle. The estimated physical radius values ​​of each viewpoint are weighted and fused in such a way that the smaller the observation angle, the higher the weight, to obtain the fused hole radius; The three-dimensional center of the hole, the unit normal, the fused hole radius, and the hole status marker indicating whether a lens has been placed in the hole are collectively output as three-dimensional hole position information.

9. A multi-view three-dimensional hole positioning system for industrial umbrella hole arrays based on point cloud fingerprint matching, used to implement the method as described in any one of claims 1 to 8, characterized in that, The system includes: The image acquisition module is used to acquire multi-view images output by multiple cameras arranged in different directions, and to annotate the camera intrinsic parameters for each image; The 3D reconstruction module is used to perform sparse reconstruction and dense reconstruction based on the multi-view image with the camera intrinsic parameters to obtain a dense 3D point cloud and the extrinsic parameters of each camera. The two-dimensional detection module is used to extract edges and fit ellipses to candidate regions output by the detection model in images from various perspectives, so as to obtain the two-dimensional center, ellipse parameters and candidate regions of each hole. The point cloud fingerprint construction module is used to project the dense three-dimensional point cloud onto the image plane of each view using the camera intrinsic parameters and the camera extrinsic parameters, and record the number of the three-dimensional points falling into the hole candidate area as the point cloud fingerprint of the hole. The cross-view matching module is used to compare the point cloud fingerprints of each hole under different views, count the number of identical 3D point numbers among the point cloud fingerprints and construct a matching cost matrix to solve for the correspondence between holes with the same name across views. The triangulation positioning module is used to establish a spatial line of sight by combining the multiple two-dimensional centers of the same hole with the camera extrinsic parameters according to the correspondence, solve the least squares intersection point to obtain the initial value of the three-dimensional center, and obtain the three-dimensional center of the hole through reprojection residual optimization. The pose estimation module is used to search for the neighboring point cloud in the dense three-dimensional point cloud based on the three-dimensional center of the hole to obtain the unit normal, and to obtain the hole radius by weighted fusion based on the ellipse parameters and observation angles of each viewpoint. The output module is used to output the three-dimensional center, unit normal, radius, and hole status marker of the hole.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 8.

Images