A multi-type chip pin positioning method based on binocular vision

Abstract

The application relates to a multi-type chip pin positioning method based on binocular vision and relates to the technical field of machine vision image processing, which comprises a set visual acquisition time point; the application is characterized in that a two-level sub-network architecture is constructed, a first-level sub-network rapidly positions a chip body and identifies a type, a second-level sub-network searches for a pin in a chip region based on chip type prior knowledge, an image matching complexity index of a pin candidate region of a left view and a right view is calculated, an area with inconsistent textures of the left view and the right view is accurately identified, fast local matching is adopted to ensure efficiency, robust global matching is adopted in a high-complexity area to ensure precision, an optimal balance between matching speed and positioning precision is achieved, a pin topology relation graph is constructed, a physical prior of equal-interval arrangement of chip pins is used as a constraint condition, a linear interpolation method is adopted to complete missing nodes according to the equal-interval principle, and a complete pin three-dimensional coordinate set is ensured to be output, so that more accurate three-dimensional coordinates are output during positioning.

Description

Technical Field

[0001] This invention relates to the field of machine vision image technology, and in particular to a method for locating pins of multiple types of chips based on binocular vision. Background Technology

[0002] Precise pin positioning is a key technology in semiconductor packaging, automated wire bonding, surface mount technology, and chip inspection. As the electronics manufacturing industry moves towards higher density and miniaturization, chip packaging types are becoming increasingly diverse. Common types include quad flat packages, small outline packages, ball grid array packages, and square flat leadless packages. Different package types exhibit significant differences in pin layout, pin spacing, and pin shape, which places higher demands on the adaptability, accuracy, and robustness of automated positioning systems. In existing visual positioning technologies, monocular vision methods extract pin features through two-dimensional image processing. While this enables rapid detection of some chip types, the lack of depth information makes it difficult to handle precision assembly scenarios involving chip tilt, inconsistent heights, or the need for three-dimensional spatial coordinates. Binocular vision technology, by simulating the stereoscopic imaging principle of the human eye, can recover the three-dimensional information of an object from the parallax of the left and right views, making spatial pin positioning possible. Existing chip pin localization methods based on binocular vision still have the following shortcomings: Most methods are designed for a single package type. When the production line needs to switch to different types of chips, the algorithm parameters often need to be readjusted, lacking a unified and adaptive processing framework. The pins are usually made of metal with a smooth surface. Under industrial light sources, they are prone to producing specular highlights, which can cause the pin edges to become blurred and lose their features. At the same time, the pin area may have low texture or repetitive texture issues, which can cause traditional stereo matching algorithms to produce mismatches and matching failures when searching for corresponding points in the left and right views, thus affecting accuracy. To address the aforementioned technical shortcomings, a solution is proposed. Summary of the Invention

[0003] This invention adopts the following technical solution: a method for locating pins of multiple types of chips based on binocular vision, comprising the following steps: Step 1: Based on the set visual acquisition time points, acquire left and right view image data of the chip using a binocular camera, and integrate and construct the left and right view images according to the time series to obtain the left and right images of the chip. Step 2: Construct a network framework for chip identification and localization. The network framework includes a first-level sub-network that performs chip body recognition on the left and right images of the chip to output the bounding box of the chip body and the chip type. The second-level sub-network locates the chip based on the distance between the chip pin positions and the overall chip area, extracts pin features to generate pin candidate regions for the left and right images. Step 3: Calculate the image matching complexity index of the pin candidate regions in the left and right views respectively to obtain the image comparison complexity; Step 4: Select a matching strategy based on the image contrast complexity, generate initial disparity maps for the left and right views, and convert the disparity into a preliminary set of three-dimensional coordinates of the pins by combining the binocular camera calibration parameters. Step 5: Construct a pin topology graph, using the preliminary 3D coordinates of each pin as nodes, the spatial distance and direction vector between pins as edge attributes, and the equal spacing constraint between pins as a constraint condition. Identify and remove abnormal nodes with confidence scores below the threshold, process missing nodes according to the geometric relationship between adjacent nodes, and output an optimized set of precise 3D coordinates.

[0004] Furthermore, based on the set visual acquisition time points, left and right view image data of the chip are acquired using a binocular camera. The left and right view images are then integrated and constructed according to the time series to obtain the left and right images of the chip. The specific process is as follows: Based on the accuracy requirements of chip detection and the ambient lighting conditions, the resolution and exposure time parameters of the binocular camera are preset, and hardware triggering or software synchronization mode is configured to ensure that the left and right cameras start acquiring data at the same time. At the preset visual acquisition time point, a acquisition trigger command is sent to the binocular camera. The left and right cameras simultaneously acquire the left and right views of the chip in the current field of view, obtain the original image data, and store it in the image sequence library in chronological order to form the original image dataset. The original image is preprocessed, and the processed left and right views are paired according to the timestamp to generate left and right images of the chip.

[0005] Furthermore, a network framework for chip identification and localization is constructed. The network framework includes a first-level sub-network that performs chip body recognition on the left and right images of the chip to output the bounding box of the chip body and the chip type. The specific process is as follows: Collect left and right view images containing different types of chips, and label the chip body in each image. The labeling content includes the coordinates of the minimum bounding rectangle of the chip and the chip type label, and construct a chip body recognition dataset. The left and right images of the chip are input into the first-level sub-network to extract image features and generate corresponding left and right image features. For the features of the left and right images, the detection results with high matching degree are selected as output through region and classification recognition, and the bounding box coordinates and type probability distribution of each chip are output.

[0006] Furthermore, the second-level sub-network locates the chip pins based on the distance between the pin positions and the overall chip area, extracts pin features to generate candidate pin regions for the left and right images, and the specific process is as follows: Obtain the chip body bounding box and chip type label output by the first-level sub-network, and retrieve the corresponding pin layout information from the preset knowledge base according to the chip type, including the number of pins, standard spacing, arrangement, and the distance of the pins relative to the chip body and the chip edge. Calculate the distance from the center of each window to the edge of the chip. If the distance is within the a priori allowable range, it will be marked as a potential pin candidate region. During the transmission process, the pin candidate region is formed based on the number of pins and the bounding box coordinates of each pin, resulting in the pin candidate region output of the second-level sub-network.

[0007] Furthermore, the image matching complexity index of the pin candidate regions in the left and right views is calculated separately to obtain the image comparison complexity. The specific process is as follows: Based on the pin candidate areas in the left and right views, grayscale images are extracted and normalized. The grayscale variance and mean within each area are calculated. After comprehensive analysis, the complexity index of the left view is obtained. Complexity index of the right view ; Compare the complexity metrics of the corresponding pin candidate regions in the left and right views, and calculate the difference between the two as the overall image comparison complexity of the pins. The complexity metrics of all candidate pin regions are summarized.

[0008] Furthermore, a matching strategy is selected based on the image contrast complexity to generate initial disparity maps for the left and right views. These disparities are then converted into a preliminary set of three-dimensional coordinates for the pins using the binocular camera calibration parameters. The specific process is as follows: The overall image comparison complexity is compared with a preset threshold. If it is less than the preset threshold, a fast local matching strategy is selected; if it is greater than the preset threshold, it indicates that there is a large texture difference between the left and right views, and a robust global matching strategy is selected. The fast local matching strategy uses a template matching method. Taking the center point of the candidate region of the left view as the center, a fixed-size image block is taken as the template. The best matching position is searched on the corresponding epipolar line in the right view. The correlation value is calculated, and the position corresponding to the maximum value is taken as the matching point. The difference between the pixel column coordinates of the matching point and the center point of the left view is the disparity value. The robust global matching strategy converts the pixel grayscale values ​​of the candidate regions of the left and right view pins into binary bit strings, calculates the distance on the right view epipolar line, selects the position with the smallest distance as the matching point, and calculates the disparity value. Based on the intrinsic parameter matrix, baseline distance, and focal length obtained from the binocular camera calibration, the disparity value of the center point of each pin is substituted into the triangulation formula to obtain the preliminary three-dimensional coordinates of the pin, thus forming a preliminary set of three-dimensional coordinates.

[0009] Furthermore, outlier nodes with confidence levels below a threshold are identified and removed. Missing nodes are then processed based on the geometric relationships between adjacent nodes, resulting in an optimized set of precise 3D coordinates. The specific process is as follows: Based on the chip type information, each pin in the initial three-dimensional coordinate set is taken as a node, and the connection relationship between the nodes is established according to the preset arrangement order. The spatial distance and direction vector of adjacent pin nodes are calculated as the attributes of the edge to form a pin topology graph. By leveraging the geometric constraints of the chip pins being typically arranged at equal intervals, the deviation between the actual and theoretical coordinates is calculated based on the left and right adjacent nodes. Combined with the confidence score during the pin matching process, a comprehensive confidence score is defined. If the confidence score is lower than the preset confidence threshold, the node is marked as an abnormal node and removed. For missing locations resulting from the removal of abnormal nodes, if the missing node is located between two retained nodes, linear interpolation is performed according to the principle of equal spacing. Bilinear interpolation is then used to reorder all retained nodes and complete nodes, and the optimized set of accurate 3D coordinates is output.

[0010] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: This binocular vision-based method for locating pins of various chip types employs simultaneous acquisition by left and right cameras. The original images are preprocessed and timestamp-matched to generate aligned left and right images. A two-level sub-network architecture is constructed. The first-level sub-network quickly locates the chip body and identifies its type, providing precise spatial constraints for the second-level sub-network. The second-level sub-network searches for pins within the chip region based on prior knowledge of the chip type, significantly reducing computational complexity and false detection rate. By calculating the image matching complexity index of candidate pin regions in the left and right views, the texture richness and matchability of each region are evaluated. A complexity difference calculation formula is introduced to further enhance the accuracy of the method. Accurately identify areas with inconsistent textures in the left and right views. Dynamically select matching strategies for low-complexity areas based on complexity metrics. Fast local matching ensures efficiency, while robust global matching ensures accuracy in high-complexity areas, achieving an optimal balance between matching speed and positioning accuracy. Construct a pin topology diagram, using the physical prior of equally spaced chip pins as a constraint. Identify and remove abnormal nodes through comprehensive confidence assessment. For missing nodes, use linear interpolation to complete them according to the equally spaced principle, ensuring a complete set of pin 3D coordinates is output. This improves the efficiency of anomaly handling, resulting in more accurate 3D coordinates for positioning. Attached Figure Description

[0011] Figure 1A schematic diagram of the overall steps of the method of the present invention is shown. Detailed Implementation

[0012] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0013] Example 1: like Figure 1 As shown, a method for locating pins of multiple types of chips based on binocular vision includes the following steps: Step 1: Based on the set visual acquisition time points, acquire left and right view image data of the chip using a binocular camera, and integrate and construct the left and right view images according to the time series to obtain the left and right images of the chip. Step 2: Construct a network framework for chip identification and localization. The network framework includes a first-level sub-network that performs chip body recognition on the left and right images of the chip to output the bounding box of the chip body and the chip type. The second-level sub-network locates the chip based on the distance between the chip pin positions and the overall chip area, extracts pin features to generate pin candidate regions for the left and right images. Step 3: Calculate the image matching complexity index of the pin candidate regions in the left and right views respectively to obtain the image comparison complexity; Step 4: Select a matching strategy based on the image contrast complexity, generate initial disparity maps for the left and right views, and convert the disparity into a preliminary set of three-dimensional coordinates of the pins by combining the binocular camera calibration parameters. Step 5: Construct a pin topology graph, using the preliminary 3D coordinates of each pin as nodes, the spatial distance and direction vector between pins as edge attributes, and the equal spacing constraint between pins as a constraint condition. Identify and remove abnormal nodes with confidence scores below the threshold, process missing nodes according to the geometric relationship between adjacent nodes, and output an optimized set of precise 3D coordinates.

[0014] In this embodiment, based on the minimum pin spacing and detection accuracy requirements of the chip, the resolution of the binocular camera is preset, the exposure time is 10ms, the gain is 1, and the hardware trigger mode is configured. The trigger signals of the left and right cameras are connected to the same signal source. The industrial control computer sends trigger pulses to the binocular camera at preset time intervals. The left and right cameras simultaneously acquire the current field of view image. Each acquisition yields a pair of original images, a left view and a right view. After each pair of images is acquired, a unified timestamp is added, and the images are stored in the original image database of the solid-state drive in chronological order. Distortion correction and image filtering are performed on the original images. The left and right views with the same timestamp are combined into image pairs and arranged in chronological order to generate the left and right images of the chip for subsequent processing. A total of 1000 pairs of left and right views containing 20 different types of chips were collected. An annotation tool was used to annotate the chip itself in each image. The annotation information included: the bounding box coordinates were the coordinates of the top-left corner of the chip's smallest bounding rectangle. and the coordinates of the bottom right corner ; Chip type label: Used The coding system comprises 20 categories. Network architecture design: The first-level sub-network adopts a convolutional neural network architecture, and the network output includes: Chip bounding box coordinates ,in ( ) represents the coordinates of the center point. For width and height; Chip type probability distribution ; Training process: The labeled dataset was divided into training, validation and test sets in a ratio of 7:2:1. The Adam optimizer was used with an initial learning rate of 0.001, a batch size of 32, and 200 training rounds. The loss function includes localization loss, confidence loss, and classification loss, with weight coefficients of 0.05, 1.0, and 0.5, respectively. The left and right images of the chip are input into the first-level sub-network to obtain the detection results for each image. High-confidence detection boxes are selected through non-maximum suppression. The intersection of the detection results from the left and right views is taken as the final output, including the chip bounding box. and chip type ; For each chip type, a pin layout knowledge base is pre-established, storing the total number of pins, the number of pin rows and pins per row, the standard pin spacing, and the distance range of pins relative to the chip edge. Based on the chip type output by the first-level sub-network, the corresponding pin layout information is retrieved from the knowledge base. Within the bounding box of the chip body, multiple initial search windows are defined according to the pin arrangement. Taking a dual-in-line package (DIP) chip as an example, rectangular search areas are defined on the left and right sides of the chip. Within each search area, a sliding window method is used to generate candidate windows. For each candidate window, the vertical distance from the center to the chip edge is calculated. If the vertical distance is within the range of the prior knowledge base... If the window is within the specified area, it will be reserved as a potential pin candidate region. The retained potential pin candidate region image patches are input into a lightweight convolutional neural network, which outputs the confidence score of each image patch belonging to the pin and the pin center coordinate offset. Set confidence threshold ,reserve For each candidate region, the precise pin center pixel coordinates are calculated based on the offset: ; The detected pin centers are sorted according to the number of pin rows and the number of pins per row to ensure that the order is consistent with the actual pin arrangement of the chip. Finally, a set of pin candidate regions for the left and right images is output. In the above formula, Let be the average grayscale value of the i-th pin region in the left view. Based on the pin candidate areas in the left and right views, grayscale images are extracted and normalized. The grayscale variance and mean within each area are calculated. After comprehensive analysis, the complexity index of the left view is obtained. Complexity index of the right view ; Compare the complexity metrics of the corresponding pin candidate regions in the left and right views, and calculate the difference between the two as the overall image comparison complexity of the pins. ; The overall image comparison complexity is compared with a preset threshold. If it is less than the preset threshold, a fast local matching strategy is selected; if it is greater than the preset threshold, it indicates that there is a large texture difference between the left and right views, and a robust global matching strategy is selected. The fast local matching strategy uses a template matching method. Taking the center point of the candidate region of the left view as the center, a fixed-size image block is taken as the template. The best matching position is searched on the corresponding epipolar line in the right view. The correlation value is calculated, and the position corresponding to the maximum value is taken as the matching point. The difference between the pixel column coordinates of the matching point and the center point of the left view is the disparity value. The robust global matching strategy converts the pixel grayscale values ​​of the candidate regions of the left and right view pins into binary bit strings, calculates the distance on the right view epipolar line, selects the position with the smallest distance as the matching point, and calculates the disparity value. Based on the intrinsic parameter matrix and baseline distance obtained from the binocular camera calibration and focal length Substitute the disparity value of the center point of each pin into the triangulation formula:

[0015] The initial three-dimensional coordinates of the pins are obtained, forming a preliminary set of three-dimensional coordinates: In the above formula, X, Y, and Z are the three-dimensional spatial coordinates of the pin. Based on the chip type information, each pin in the initial three-dimensional coordinate set is taken as a node, and the connection relationship between the nodes is established according to the preset arrangement order. The spatial distance and direction vector of adjacent pin nodes are calculated as the attributes of the edge to form a pin topology graph. By leveraging the geometric constraints of the chip pins being typically arranged at equal intervals, the deviation between the actual and theoretical coordinates is calculated based on the left and right adjacent nodes. Combined with the confidence score during the pin matching process, a comprehensive confidence score is defined. If the confidence score is lower than the preset confidence threshold, the node is marked as an abnormal node and removed. For missing locations resulting from the removal of abnormal nodes, if the missing node is located between two retained nodes, linear interpolation is performed according to the principle of equal spacing. Bilinear interpolation is then used to reorder all retained nodes and complete nodes, and the optimized set of accurate 3D coordinates is output.

[0016] The size of the interval and threshold is set to facilitate comparison. The size of the threshold depends on the amount of sample data and the number of bases set by those skilled in the art for each set of sample data; as long as it does not affect the ratio between the parameter and the quantized value.

[0017] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation. In the two embodiments provided in this application, it should be understood that the disclosed apparatus and system can be implemented in other ways; for example, the apparatus embodiments described above are merely illustrative, and the division of modules is merely a logical functional division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed; furthermore, the coupling or direct coupling or communication connection between the shown or discussed mutuals can be through some interfaces, and the indirect coupling or communication connection between the apparatus or modules can be electrical, mechanical or other forms. The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for locating pins of multiple types of chips based on binocular vision, characterized in that, Includes the following steps: Step 1: Based on the set visual acquisition time points, acquire left and right view image data of the chip using a binocular camera, and integrate and construct the left and right view images according to the time series to obtain the left and right images of the chip. Step 2: Construct a network framework for chip identification and localization. The network framework includes a first-level sub-network that performs chip body recognition on the left and right images of the chip to output the bounding box of the chip body and the chip type. The second-level sub-network locates the chip based on the distance between the chip pin positions and the overall chip area, extracts pin features to generate pin candidate regions for the left and right images. Step 3: Calculate the image matching complexity index of the pin candidate regions in the left and right views respectively to obtain the image comparison complexity; Step 4: Select a matching strategy based on the image contrast complexity, generate initial disparity maps for the left and right views, and convert the disparity into a preliminary set of three-dimensional coordinates of the pins by combining the binocular camera calibration parameters. Step 5: Construct a pin topology graph, using the preliminary 3D coordinates of each pin as nodes, the spatial distance and direction vector between pins as edge attributes, and the equal spacing constraint between pins as a constraint condition. Identify and remove abnormal nodes with confidence levels below the threshold, process missing nodes according to the geometric relationship between adjacent nodes, and output an optimized set of precise 3D coordinates.

2. The multi-type chip pin positioning method based on binocular vision according to claim 1, characterized in that, Based on the set visual acquisition time points, left and right view image data of the chip are acquired using a binocular camera. The left and right view images are then integrated and constructed according to the time series to obtain the left and right images of the chip. The specific process is as follows: Based on the accuracy requirements of chip detection and the ambient lighting conditions, the resolution and exposure time parameters of the binocular camera are preset, and hardware triggering or software synchronization mode is configured to ensure that the left and right cameras start acquiring data at the same time. At the preset visual acquisition time point, a acquisition trigger command is sent to the binocular camera. The left and right cameras simultaneously acquire the left and right views of the chip in the current field of view, obtain the original image data, and store it in the image sequence library in chronological order to form the original image dataset. The original image is preprocessed, and the processed left and right views are paired according to the timestamp to generate left and right images of the chip.

3. The multi-type chip pin positioning method based on binocular vision according to claim 1, characterized in that, A network framework for chip identification and localization is constructed. The network framework includes a first-level sub-network that performs chip body recognition on the left and right images of the chip to output the bounding box of the chip body and the chip type. The specific process is as follows: Collect left and right view images containing different types of chips, and label the chip body in each image. The labeling content includes the coordinates of the minimum bounding rectangle of the chip and the chip type label, and construct a chip body recognition dataset. The left and right images of the chip are input into the first-level sub-network to extract image features and generate corresponding left and right image features. For the features of the left and right images, the detection results with high matching degree are selected as output through region and classification recognition, and the bounding box coordinates and type probability distribution of each chip are output.

4. The multi-type chip pin positioning method based on binocular vision according to claim 1, characterized in that, The second-level sub-network locates the chip pins based on the distance between the pin positions and the overall chip area, extracts pin features to generate candidate pin regions for the left and right images, and the specific process is as follows: Obtain the chip body bounding box and chip type label output by the first-level sub-network, and retrieve the corresponding pin layout information from the preset knowledge base according to the chip type, including the number of pins, standard spacing, arrangement, and the distance of the pins relative to the chip body and the chip edge. Calculate the distance from the center of each window to the edge of the chip. If the distance is within the a priori allowable range, it will be marked as a potential pin candidate region. During the transmission process, the pin candidate region is formed based on the number of pins and the bounding box coordinates of each pin, resulting in the pin candidate region output of the second-level sub-network.

5. The method for locating multi-type chip pins based on binocular vision according to claim 1, characterized in that, For the pin candidate regions in the left and right images, calculate the image matching complexity index for the pin candidate regions in the left and right views respectively, and obtain the image comparison complexity. The specific process is as follows: Based on the pin candidate areas in the left and right views, grayscale images are extracted and normalized. The grayscale variance and mean within each area are calculated. After comprehensive analysis, the complexity index of the left view is obtained. Complexity index of the right view ; Compare the complexity metrics of the corresponding pin candidate regions in the left and right views, and calculate the difference between the two as the overall image comparison complexity of the pins. The complexity metrics of all candidate pin regions are summarized.

6. The multi-type chip pin positioning method based on binocular vision according to claim 1, characterized in that, A matching strategy is selected based on the image contrast complexity to generate initial disparity maps for the left and right views. Then, combined with the binocular camera calibration parameters, the disparity is converted into a preliminary set of three-dimensional coordinates for the pins. The specific process is as follows: The overall image comparison complexity is compared with a preset threshold. If it is less than the preset threshold, a fast local matching strategy is selected; if it is greater than the preset threshold, it indicates that there is a large texture difference between the left and right views, and a robust global matching strategy is selected. The fast local matching strategy uses a template matching method. Taking the center point of the candidate region of the left view as the center, a fixed-size image block is taken as the template. The best matching position is searched on the corresponding epipolar line in the right view. The correlation value is calculated, and the position corresponding to the maximum value is taken as the matching point. The difference between the pixel column coordinates of the matching point and the center point of the left view is the disparity value. The robust global matching strategy converts the pixel grayscale values ​​of the candidate regions of the left and right view pins into binary bit strings, calculates the distance on the epipolar line of the right view, selects the position with the smallest distance as the matching point, and calculates the disparity value. Based on the intrinsic parameter matrix, baseline distance, and focal length obtained from the binocular camera calibration, the disparity value of the center point of each pin is substituted into the triangulation formula to obtain the preliminary three-dimensional coordinates of the pin, thus forming a preliminary set of three-dimensional coordinates.

7. The method for locating multi-type chip pins based on binocular vision according to claim 1, characterized in that, Identify and remove outlier nodes with confidence scores below a threshold. Process missing nodes based on the geometric relationships between adjacent nodes, and output an optimized set of precise 3D coordinates. The specific process is as follows: Based on the chip type information, each pin in the initial three-dimensional coordinate set is taken as a node, and the connection relationship between the nodes is established according to the preset arrangement order. The spatial distance and direction vector of adjacent pin nodes are calculated as the attributes of the edge to form a pin topology graph. Using the geometric constraints of the chip pins being arranged at equal intervals, the deviation between the actual coordinates and the theoretical coordinates is calculated based on the left and right adjacent nodes. Combined with the confidence score in the pin matching process, a comprehensive confidence score is defined. If the confidence score is lower than the preset confidence threshold, the node is marked as an abnormal node and removed. For missing locations resulting from the removal of abnormal nodes, if the missing node is located between two retained nodes, linear interpolation is performed according to the principle of equal spacing. Bilinear interpolation is then used to reorder all retained nodes and complete nodes, and the optimized set of accurate 3D coordinates is output.

Images