A deep learning-based component pin defect visual online detection system

Abstract

The application provides a kind of based on deep learning's component pin defect vision online detection system, it is related to machine vision detection technical field, including: acquisition module, for through industrial camera synchronous acquisition after being pasted and welded, enter the printed circuit board image before reflow soldering process, the original pin image of complete chip pin is obtained by intercepting;Original pin image is carried out image enhancement processing, and the pin image after enhancement is obtained;Enhancement module is used to input the pin image after enhancement into the deep learning segmentation network that is trained in advance, and the pin mask image is obtained;Pin mask image is carried out morphological processing, and the pin connected domain image is obtained.The application realizes the high-precision online detection of pin defect.

Description

Technical Field

[0001] This invention relates to the field of machine vision inspection technology, and in particular to an online visual inspection system for component pin defects based on deep learning. Background Technology

[0002] On surface mount technology (SMT) production lines, machine vision is commonly used to detect pin defects for square, flat packaged chips. In a real-world production line case, the equipment uses a fixed-angle industrial camera to capture images of the printed circuit board after mounting and before reflow soldering. Traditional threshold segmentation and edge detection algorithms are used to extract the pin contours, and template comparison is then used to determine whether there are any offset or deformation defects in the pins. However, this technology has its limitations. After the chip is mounted, it is prone to slight tilting, rotation, and other spatial orientation deviations from the printed circuit board substrate. In addition, the uneven reflection of the metal pin area caused by flux residue makes it difficult for traditional algorithms to detect pin segmentation breakage and noise interference. Furthermore, spatial orientation distortion can lead to inaccurate contour fitting and deviation calculation, making it impossible to accurately identify minute pin offset defects, resulting in false positives and false negatives. This makes it difficult to meet the requirements of high-precision online inspection. Summary of the Invention

[0003] This invention provides a deep learning-based online visual inspection system for component pin defects, enabling high-precision online inspection of pin defects.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0005] The first aspect is a deep learning-based online visual inspection system for component pin defects, including:

[0006] The acquisition module is used to synchronously acquire images of the printed circuit board after surface mount soldering and before entering the reflow soldering process using an industrial camera, and to capture the original pin images of the complete chip pins; the original pin images are then subjected to image enhancement processing to obtain enhanced pin images;

[0007] The enhancement module is used to input the enhanced pin image into a pre-trained deep learning segmentation network to obtain a pin mask image; and to perform morphological processing on the pin mask image to obtain a pin connected component image.

[0008] The calculation module is used to construct a spatial reference plane in the pin connected domain image using the first and second corner feature points of the chip body; discretize the spatial reference plane into a grid region composed of control nodes; apply virtual boundary conditions to the control nodes based on the relative displacement constraints between the chip body and the printed circuit board; obtain the displacement field distribution of the control nodes in the spatial coordinate system through shape function interpolation and stiffness matrix integration of the grid region, and obtain the spatial attitude correction parameters based on the displacement field distribution.

[0009] The correction module is used to perform projection transformation on the pin connected component image according to the spatial attitude correction parameters to obtain the corrected pin connected component image; extract the profile of a single pin from the corrected pin connected component image; and obtain the theoretical edge line of each pin by extracting the edge profile point set of each pin.

[0010] The processing module is used to calculate the deviation between the edge contour point set of each pin and the corresponding theoretical edge line to obtain the detection result.

[0011] Furthermore, an industrial camera is used to simultaneously capture images of the printed circuit board after surface mount soldering and before it enters the reflow soldering process, obtaining raw pin images of the complete chip pins; image enhancement processing is then performed on the raw pin images to obtain enhanced pin images, including:

[0012] The printed circuit board image is synchronously acquired by an industrial camera under the control of a trigger signal. The chip area is located in the printed circuit board image according to the template matching result of the square flat package chip. The image area containing all pins is cropped based on the outline of the chip body to obtain the original pin image.

[0013] Adaptive histogram equalization is performed on the original pin image. The contrast stretching mapping curve is calculated based on the local gray-level distribution of the image. A homomorphic filtering algorithm based on illumination component estimation is used to suppress the reflection difference between the flux residue area and the metal pin surface, thereby enhancing the gray-level gradient between the pin area and the background area, resulting in an enhanced pin image.

[0014] Furthermore, the enhanced pin image is input into a pre-trained deep learning segmentation network to obtain a pin mask image; morphological processing is then performed on the pin mask image to obtain a pin connected component image, including:

[0015] The enhanced pin image is input into a pre-trained encoder-decoder deep learning segmentation network. The encoder extracts multi-scale semantic features of the enhanced pin image through multi-layer convolution and pooling operations. The decoder restores the multi-scale semantic features to the original image resolution through upsampling and skip connections, and outputs a pin category probability map pixel by pixel. After thresholding, a binarized pin mask image is obtained. In the pin mask image, the pin region is assigned a first gray value and the background region is assigned a second gray value.

[0016] The pin mask image is subjected to a morphological opening operation of erosion followed by dilation to filter out isolated noise points caused by flux residue. Then, a morphological closing operation of dilation followed by erosion is performed to fill the tiny holes inside the pin area caused by edge blurring and to connect the gaps formed by the segmentation and breakage between adjacent pins, resulting in a pin connected region image in which each pin area is separated from each other and the edges are continuously closed.

[0017] Furthermore, a spatial reference plane is constructed in the pin connected component image using the first and second corner feature points of the chip body; the spatial reference plane is discretized into a grid region composed of control nodes, including:

[0018] In the pin connected domain image, the first and second corner feature points of the chip body are located by the sub-pixel edge extraction algorithm. The first and second corner feature points are located at the two ends of the diagonal of the chip body. At the same time, the preset positioning mark points on the printed circuit board substrate are extracted. The first corner feature point, the second corner feature point and the positioning mark points are used to establish a spatial reference plane. The spatial reference plane represents the relative spatial attitude between the chip body and the printed circuit board.

[0019] The spatial reference plane is discretized into a quadrilateral grid region composed of control nodes according to the isoparametric element division method. The control nodes are arranged in a regular array in the plane. Each control node has an initial coordinate value in the spatial coordinate system. The spatial position relationship between adjacent control nodes is established through shape functions.

[0020] Furthermore, virtual boundary conditions based on the relative displacement constraints between the chip body and the printed circuit board are applied to the control node. By performing shape function interpolation and stiffness matrix integration on the mesh region, the displacement field distribution of the control node in the spatial coordinate system is obtained through iterative calculation. Based on the displacement field distribution, spatial attitude correction parameters are obtained, including:

[0021] The control nodes corresponding to the first and second corner feature points of the chip body are used as fixed constraint boundaries, and their displacement is set to zero. The control nodes corresponding to the positioning mark points on the printed circuit board substrate are used as loading boundaries, and virtual displacement loads are applied along each axis of the spatial coordinate system to form virtual boundary conditions based on the relative displacement constraints between the chip body and the printed circuit board.

[0022] Based on the virtual boundary conditions, construct the element stiffness matrix of each mesh element, and assemble all element stiffness matrices according to the global number of the control nodes to form the overall stiffness matrix;

[0023] The nonlinear equations consisting of the overall stiffness matrix and the load vector are solved by the Newton-Raphson iterative method. The displacement increment of the control node is updated after each iteration until the norm of the displacement increment between two adjacent iterations is less than a preset threshold, thus obtaining the displacement field distribution of each control node in the spatial coordinate system.

[0024] The out-of-plane displacement components of each control node along the imaging optical axis are extracted from the displacement field distribution, and the spatial attitude correction parameters are obtained by fitting the correspondence between the out-of-plane displacement components and the spatial coordinates.

[0025] Furthermore, the pin connected component image is projected and transformed according to the spatial attitude correction parameters to obtain the corrected pin connected component image, including:

[0026] Obtain spatial attitude correction parameters, which include tilt correction and rotation correction. Construct the tilt correction and rotation correction into a perspective projection transformation matrix from three-dimensional space to a two-dimensional imaging plane.

[0027] Each pixel in the pin connected component image is used as the transformation source point. The mapping coordinates of each pixel in the corrected image space are calculated according to the perspective projection transformation matrix. By resampling the gray values ​​at the mapping coordinates, the corrected pin connected component image with spatial pose distortion eliminated is obtained.

[0028] Furthermore, the contour of a single pin is extracted from the corrected pin connected component image. By extracting the edge contour point set of each pin, the theoretical edge line of each pin is fitted, including:

[0029] A connected component labeling algorithm is performed on the corrected pin connected component image. Each pin region is independently labeled according to the eight-neighbor connectivity criterion to obtain the independent connected component corresponding to each pin.

[0030] For each independent connected component, extract the sequence of closed contour points pixel by pixel along the boundary of the connected component to obtain the contour point set of a single pin;

[0031] For each pin, fit the outline point set to the straight lines on both sides. Based on the pin extension direction, group the points in the outline point set that are located on both sides of the pin. By calculating the slope and intercept parameters of the straight lines on both sides, obtain the theoretical left edge line and the theoretical right edge line of each pin.

[0032] Furthermore, the deviation between the edge contour point set of each pin and the corresponding theoretical edge line is calculated to obtain the detection results, including:

[0033] Obtain the left and right theoretical edge lines of each pin, and obtain the subsets of contour points located at the left and right edges in the contour point set of each pin; calculate the Euclidean distance from each contour point in the left contour point subset to the corresponding left theoretical edge line to obtain the left edge deviation sequence, and calculate the Euclidean distance from each contour point in the right contour point subset to the corresponding right theoretical edge line to obtain the right edge deviation sequence.

[0034] The left edge deviation sequence and the right edge deviation sequence of each pin are fused together to calculate the cumulative deviation of each pin in the width direction. The cumulative deviation of all pins is then used to construct a deviation distribution curve according to the pin arrangement order. The deviation distribution curve is smoothed by a sliding window filtering method, and continuous peak segments exceeding the preset deviation threshold are extracted from the deviation distribution curve.

[0035] Based on the position index of the continuous peak segment in the pin arrangement sequence, the pin number with the offset defect is determined. Combined with the peak deviation and deviation change rate of the continuous peak segment, the severity of the offset defect is classified, and the detection result including the defective pin number, offset direction, offset amount and severity classification is obtained.

[0036] In a second aspect, a computing device includes:

[0037] One or more processors;

[0038] A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to execute the system.

[0039] Thirdly, a computer-readable storage medium storing a program that, when executed by a processor, performs the system.

[0040] The above-described solution of the present invention has at least the following beneficial effects:

[0041] This technique overcomes the technical challenges of existing detection technologies. It involves acquiring images of printed circuit board (PCB) pins before reflow soldering using an industrial camera and performing image enhancement processing. A pre-trained deep learning segmentation network is used to extract pin masks, and morphological processing is applied to obtain pin connected component images. A spatial reference plane is constructed using chip body corner feature points, and spatial attitude correction parameters are obtained through mesh discretization, virtual boundary constraints, and iterative calculations. The pin images are then subjected to projection transformation correction to extract pin contours, fit theoretical edge lines, and calculate deviations. This overcomes the technical problems of existing detection technologies, such as pin segmentation breakage due to uneven flux residue reflection, noise interference, and distortion in contour fitting and deviation calculation caused by spatial attitude deviations after chip mounting. It also addresses the inaccurate identification of minute pin offset defects, the susceptibility to false positives and false negatives, and the difficulty in meeting the requirements of high-precision online detection. Ultimately, this technique achieves high-precision, robust online detection of component pin defects, improves detection accuracy and stability, reduces false positives and false negatives, adapts to the real-time detection needs of surface mount technology (SMT) production lines, ensures the quality of PCB chip pin soldering, and improves production efficiency and product qualification rate. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of an online visual inspection system for component pin defects based on deep learning, provided by an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the process of a deep learning-based online visual inspection system for component pin defects, which calculates the deviation between the edge contour point set of each pin and the corresponding theoretical edge line to obtain the inspection result.

[0044] Figure 3 This is a statistical diagram illustrating the accuracy of pin defect detection for different chip types.

[0045] Figure 4 This is a schematic diagram illustrating the changing trends of training loss and accuracy in deep learning models.

[0046] Figure 5 This is a statistical diagram showing the detection rate and false detection rate of different defect types.

[0047] Figure 6 This is a diagram showing the comparison of detection efficiency and accuracy before and after system implementation. Detailed Implementation

[0048] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art.

[0049] like Figure 1 As shown, an embodiment of the present invention proposes a deep learning-based online visual inspection system for component pin defects, comprising:

[0050] The acquisition module is used to synchronously acquire images of the printed circuit board after surface mount soldering and before entering the reflow soldering process using an industrial camera, and to capture the original pin images of the complete chip pins; the original pin images are then subjected to image enhancement processing to obtain enhanced pin images;

[0051] The enhancement module is used to input the enhanced pin image into a pre-trained deep learning segmentation network to obtain a pin mask image; and to perform morphological processing on the pin mask image to obtain a pin connected component image.

[0052] The calculation module is used to construct a spatial reference plane in the pin connected domain image using the first and second corner feature points of the chip body; discretize the spatial reference plane into a grid region composed of control nodes; apply virtual boundary conditions to the control nodes based on the relative displacement constraints between the chip body and the printed circuit board; obtain the displacement field distribution of the control nodes in the spatial coordinate system through shape function interpolation and stiffness matrix integration of the grid region, and obtain the spatial attitude correction parameters based on the displacement field distribution.

[0053] The correction module is used to perform projection transformation on the pin connected component image according to the spatial attitude correction parameters to obtain the corrected pin connected component image; extract the profile of a single pin from the corrected pin connected component image; and obtain the theoretical edge line of each pin by extracting the edge profile point set of each pin.

[0054] The processing module is used to calculate the deviation between the edge contour point set of each pin and the corresponding theoretical edge line to obtain the detection result.

[0055] In this embodiment of the invention, the printed circuit board image before reflow soldering is acquired, the original image of the chip pins is cropped and enhanced, and the enhanced image is input into a pre-trained deep learning segmentation network to obtain a pin mask image. Morphological processing is then performed to obtain a pin connected component image. A spatial reference plane is constructed using the chip's corner feature points. After mesh discretization of the plane, virtual boundary constraints are applied. Spatial attitude correction parameters are obtained through shape function interpolation, stiffness matrix integration, and iterative calculation. Based on these correction parameters, a projection transformation correction is performed on the pin connected component image. The pin contour is extracted and fitted with a theoretical edge line. Finally, the detection result is obtained by calculating the deviation between the contour points and the theoretical edge line. The resulting technical approach overcomes the technical problems of existing detection technologies, such as pin breakage caused by uneven reflection of flux residue, noise interference, distortion of contour fitting and deviation calculation caused by spatial posture deviation after chip mounting, inaccurate identification of minute pin offset defects, easy false detection and missed detection, and difficulty in meeting the requirements of high-precision online detection. It achieves high-precision and robust online detection of component pin defects, improves detection accuracy and stability, effectively reduces false detection and missed detection rates, adapts to the real-time detection requirements of surface mount technology chip mounting production lines, ensures the soldering quality of printed circuit board chip pins, and helps improve production efficiency and product qualification rate.

[0056] In a preferred embodiment of the present invention, an industrial camera synchronously acquires images of the printed circuit board after surface mount soldering and before it enters the reflow soldering process, capturing the original pin images of the complete chip pins; the original pin images are then subjected to image enhancement processing to obtain enhanced pin images, including:

[0057] The system synchronously acquires printed circuit board (PCB) images using an industrial camera under trigger signal control. Based on template matching results for the square flat packaged chip, the system locates the chip's region within the PCB image. Using the chip's outline as a reference, an image region containing all pins is cropped to obtain the original pin image. Specifically, in the surface mount production line, when the square flat packaged chip completes surface mount soldering and is about to enter the reflow soldering process, the system sends a synchronous trigger signal to the industrial camera. Under the control of this trigger signal, the industrial camera acquires images of the PCB at the current workstation, obtaining a complete PCB image including the chip body, all pins, and the PCB substrate. After acquiring the PCB image, the system calls a preset square flat packaged chip standard template for template matching. It traverses all areas to be detected in the PCB image, calculates the grayscale similarity between each area and the standard template, and identifies the area with the highest grayscale similarity value as the chip's location, thus completing the precise chip positioning. After the chip is located, the pre-defined pixel range is extended outward along the outline of the identified chip body as a reference to form a cut-off area that completely covers all the pins of the chip. The printed circuit board image is then cropped according to this cut-off area to remove irrelevant background areas, and finally the original pin image containing only the complete chip pins is obtained.

[0058] Adaptive histogram equalization is performed on the original pin image. A contrast stretching mapping curve is calculated based on the local gray-level distribution of the image. A homomorphic filtering algorithm based on illumination component estimation is used to suppress the reflection difference between the flux residue area and the metal pin surface, thereby enhancing the gray-level gradient between the pin area and the background area, resulting in an enhanced pin image. Specifically, after obtaining the original pin image, adaptive histogram equalization is first performed to improve the local contrast. The original pin image is divided into several local sub-regions of equal size. The number of pixels corresponding to each gray level in each local sub-region is counted. The gray-level probability density of that gray level is obtained by dividing the number of pixels at a single gray level by the total number of pixels in the corresponding local sub-region. The gray-level probability densities of all gray levels in a single local sub-region are then accumulated sequentially in ascending order of gray value. The cumulative distribution values ​​of each gray level are obtained, and then the cumulative distribution value is multiplied by the maximum gray level value of the image to calculate the target gray value after mapping. This is used to construct the contrast stretching mapping curve of each local sub-region. For the pixels at the splicing position of adjacent local sub-regions, bilinear interpolation is used for gray-level transition. The distance from the boundary pixel to the center of the two adjacent sub-regions is used as a weighting coefficient. The mapped gray values ​​of the two sub-regions are weighted and summed to eliminate the gray-level abrupt change caused by the splicing of sub-regions and complete the adaptive histogram equalization processing. A homomorphic filtering algorithm based on illumination component estimation is used to suppress the difference between flux residue and the reflection of the metal pin surface. For the image after equalization processing, the gray value of each pixel is incremented by one and the natural logarithm is taken to convert the image gray-level information to the logarithmic domain, realizing the separation of illumination component and reflection component in the image. The image data in the logarithmic domain is multiplied pixel by pixel with the high-pass filter coefficients to weaken the low-frequency components representing uneven illumination in the image, while retaining the high-frequency components representing the pin edges and contour features. After the filtering operation is completed, the filtered logarithmic domain values ​​are restored by exponential operation, converting the data from the logarithmic domain back to the ordinary grayscale domain, thus eliminating the problem of uneven reflection caused by flux residue. After the above processing, the grayscale gradient between the pin area and the background area is enhanced, and finally an enhanced pin image with clear edges and low noise interference is obtained.

[0059] In this embodiment of the invention, an industrial camera is controlled by a trigger signal to synchronously acquire images of the printed circuit board. A square flat packaged chip is located by template matching and the complete pin image is cropped based on the outline of the chip body. At the same time, the original pin image is processed by adaptive histogram equalization combined with a homomorphic filtering algorithm based on illumination component estimation. Therefore, the technical problems of inaccurate chip positioning, incomplete pin image cropping, large difference between flux residue and metal pin surface reflection, and insufficient grayscale contrast between pin area and background area are overcome. Thus, the complete original pin image is accurately acquired, reflection interference is effectively suppressed, and the grayscale gradient between pin and background is enhanced.

[0060] In a preferred embodiment of the present invention, the enhanced pin image is input into a pre-trained deep learning segmentation network to obtain a pin mask image; morphological processing is performed on the pin mask image to obtain a pin connected component image, including:

[0061] The enhanced pin image is input into a pre-trained encoder-decoder deep learning segmentation network. The encoder extracts multi-scale semantic features from the enhanced pin image through multi-layer convolution and pooling operations. The decoder restores the multi-scale semantic features to the original image resolution through upsampling and skip connections, and outputs a pin category probability map pixel by pixel. After thresholding, a binarized pin mask image is obtained. In the pin mask image, the pin region is assigned a first gray value, and the background region is assigned a second gray value. Specifically, the enhanced pin image is input into a pre-trained encoder-decoder deep learning segmentation network using a large number of chip pin samples. The encoder part of the network performs multi-layer convolution and pooling operations on the enhanced pin image. During the convolution process, a convolution kernel of a set size slides across the image region by region. The convolution operation uses the following formula: ,in, The convolutional feature value output at the current position. For the input image in coordinates Pixel value at that location, For the convolution kernel in coordinates The weight value at that point, , The coordinates are relative within the convolution kernel window. The pooling operation uses max pooling, which divides the convolution feature map into regions evenly according to a fixed-size window. The maximum value among all feature values ​​in each window is taken as the feature value after pooling. By continuously downsampling and fusing features, multi-scale semantic features that fully represent the overall structure of the pin are finally extracted. The decoder upsamples the multi-scale semantic features and gradually enlarges the low-resolution feature map to the same resolution as the original enhanced pin image through linear interpolation. At the same time, a skip connection method is used to add and fuse the feature map obtained by the decoder upsampled with the high-resolution detail feature map output by the corresponding level of the encoder pixel by pixel, so as to preserve the complete edge detail information of the pin. After the decoder completes feature recovery, it outputs a pin category probability map pixel by pixel. Each pixel corresponds to a probability value representing that the pixel belongs to the pin region. The probability map is thresholded by comparing the probability value of each pixel with a preset segmentation threshold. Pixels with a probability value greater than or equal to the segmentation threshold are determined to be pin regions and assigned a first gray value. Pixels with a probability value less than the segmentation threshold are determined to be background regions and assigned a second gray value. In this way, a binarized pin mask image is generated.

[0062] The pin mask image is subjected to a morphological opening operation (erosion followed by dilation) to filter out isolated noise points caused by flux residue. Then, a morphological closing operation (dilation followed by erosion) is performed to fill the tiny holes inside the pin regions caused by edge blurring and to connect the gaps formed by segmentation between adjacent pins. This results in a pin connected region image where each pin region is separated from the others and has continuously closed edges. Specifically, this includes: performing morphological opening and closing operations sequentially on the obtained binarized pin mask image, followed by a morphological opening operation (erosion followed by dilation). ,in, The result of the erosion calculation. For the input pin mask image, As a structural element, The coordinates within the structuring element. To obtain the minimum grayscale value within the area covered by the structuring element, the erosion process uses a structuring element of a set size to traverse the pin mask image position by position. The minimum grayscale value among all pixels covered by the structuring element is used to replace the grayscale value of the pixel at the center of the structuring element. This operation filters out small, isolated noise points caused by flux residue in the image. Immediately after erosion, dilation is performed. The same structuring element of the same size is used to traverse the image, and the maximum grayscale value among all pixels covered by the structuring element is used to replace the grayscale value of the pixel at the center of the structuring element. This restores the original contour size of the pin area and prevents the pin contour from shrinking due to the erosion operation. After the opening operation is completed, a morphological closing operation of dilation followed by erosion is performed. ,in, The result of the expansion operation. To obtain the maximum gray value within the area covered by the structuring element, the tiny holes inside the pin area caused by the blurring of the image edges are first filled by dilation processing, while the gaps formed by the discontinuity of the segmentation between adjacent pins are connected. Then, the expanded pin outline is restored to its true size by erosion processing to avoid excessive expansion and deformation of the outline. After the above two morphological processing steps, the pin connected region image with independent and separated pin areas, continuous edges and complete closure is finally obtained.

[0063] In this embodiment of the invention, the enhanced pin image is input into a pre-trained encoder-decoder deep learning segmentation network. The encoder extracts multi-scale semantic features, the decoder restores the image resolution and thresholds it to obtain a binarized pin mask image. Then, the mask image is subjected to morphological opening operations of erosion followed by dilation and morphological closing operations of dilation followed by erosion in sequence. Therefore, it overcomes the technical problems of traditional segmentation algorithms, such as difficulty in accurately extracting pin regions, interference from isolated noise points caused by flux residue, micro-holes in the pin region due to blurred edges, and gaps formed by the segmentation of adjacent pins. Thus, it achieves pixel-level accurate segmentation of the pin region, effectively filters out noise, fills holes, and connects broken gaps, resulting in a pin connected region image in which each pin region is separated and the edges are continuously closed.

[0064] In a preferred embodiment of the present invention, a spatial reference plane is constructed in the pin connected component image using first and second corner feature points of the chip body; the spatial reference plane is discretized into a grid region composed of control nodes, including:

[0065] In the pin connected component image, a sub-pixel edge extraction algorithm is used to locate the first and second corner feature points of the chip body. These first and second corner feature points are located at opposite ends of the chip body's diagonal. Simultaneously, preset positioning markers on the printed circuit board substrate are extracted. A spatial reference plane is established using the first and second corner feature points and the positioning markers. This spatial reference plane represents the relative spatial orientation between the chip body and the printed circuit board. Specifically, this includes: first, performing Gaussian smoothing denoising on the pin connected component image to reduce image noise interference with edge extraction; during smoothing, taking a neighborhood window of a preset size around each pixel, calculating the sum of gray values ​​of all pixels within the neighborhood window, and then dividing the sum of gray values ​​by the total number of pixels within the neighborhood window to obtain the smoothed gray value of that pixel. This completes the smoothing and denoising of the entire image. The Sobel operator calculates the gradient magnitude and gradient direction of an image. The gradient magnitude is the square root of the sum of the squares of the gradient values ​​in the x and y directions of each pixel. The gradient direction is the angle obtained by taking the arctangent of the ratio of the gradient values ​​in the y and x directions. Pixels with gradient magnitudes greater than a preset gradient threshold are marked as coarse edge points. Subpixel thinning is then performed on these coarse edge points. The gray-level moments in the neighborhood of each coarse edge point are calculated. The gray-level moments of all pixels in the three-by-three neighborhood of the coarse edge point are multiplied by their corresponding x-coordinates, summed, and then divided by the sum of the gray-level moments of all pixels in the neighborhood to obtain the subpixel-level x-coordinate. Similarly, the gray-level moments of all pixels in the neighborhood are multiplied by their corresponding y-coordinates, summed, and then divided by the sum of the gray-level moments of all pixels in the neighborhood to obtain the subpixel-level y-coordinate. This achieves subpixel-level edge localization.

[0066] Based on the extracted sub-pixel edges of the chip body, the first and second corner feature points of the chip body are located. The chip body has a square structure, and the corner feature points are the intersections of the contour edges. The first and second corner feature points are located at the two ends of the diagonal of the chip body, respectively. The corner feature points are filtered by calculating the curvature of the edge points. A fitted straight line is constructed by taking a point on the edge and a preset number of edge points before and after it. The distance from the point to the fitted straight line is calculated. The larger the distance, the greater the curvature of the point. The edge intersection with the largest curvature is the corner feature point of the chip body. By traversing all edge intersections, the two corner feature points located at the two ends of the diagonal are selected and marked as the first corner feature point and the second corner feature point, respectively.

[0067] Pre-defined positioning markers are extracted from the printed circuit board substrate. These positioning markers are pre-designed feature points on the substrate, possessing obvious grayscale differences and fixed shapes. Using a shape matching algorithm, regions conforming to the preset shapes are traversed in the pin connected component image. The similarity between the contour features of this region and the contour features of the preset positioning markers is calculated. The number of overlapping pixels between the two shape contours is divided by the total number of pixels in the contour of the preset positioning markers to obtain the shape similarity value. The center coordinates of regions with similarity greater than a preset similarity threshold are used as the coordinates of the positioning markers, thus completing the extraction of the positioning markers.

[0068] Based on three non-collinear points—the first corner feature point, the second corner feature point, and the positioning marker—a spatial reference plane is established. The spatial coordinates of these three points are determined as follows: first corner feature point coordinates, second corner feature point coordinates, and positioning marker coordinates. First, the vectors pointing from the first corner feature point to the second corner feature point and from the first corner feature point to the positioning marker are calculated. Then, the cross product of these two vectors yields the normal vector of the plane. The first component of the normal vector is the difference in y-coordinates between the second and first corner feature points multiplied by the difference in z-coordinates between the positioning marker and the first corner feature point, minus the difference in z-coordinates between the second and first corner feature points. The first component of the normal vector is the difference in z-coordinates between the positioning marker point and the first corner feature point, multiplied by the difference in y-coordinates. The second component of the normal vector is the difference in z-coordinates between the second and first corner feature points multiplied by the difference in x-coordinates between the positioning marker point and the first corner feature point, minus the difference in x-coordinates between the second and first corner feature points multiplied by the difference in z-coordinates between the positioning marker point and the first corner feature point. The third component of the normal vector is the difference in x-coordinates between the second and first corner feature points multiplied by the difference in y-coordinates between the positioning marker point and the first corner feature point, minus the difference in y-coordinates between the second and first corner feature points multiplied by the difference in x-coordinates between the positioning marker point and the first corner feature point. Substituting the coordinates of the first corner feature point into the general equation of the plane, the constant term of the equation is calculated. The constant term is the sum of the product of the three components of the normal vector with the corresponding coordinates of the first corner feature point, and then the negative value is taken. Finally, the complete equation of the spatial reference plane is determined, which can accurately represent the relative spatial attitude between the chip body and the printed circuit board.

[0069] The spatial reference plane is discretized into a quadrilateral grid region composed of control nodes using an isoparametric element division method. The control nodes are arranged in a regular array in the plane, and each control node has an initial coordinate value in the spatial coordinate system. The spatial position relationship between adjacent control nodes is established through shape functions. Specifically, the parameters for grid division are determined. Based on the size of the chip body, the distribution density of the pins, and the accuracy requirements of subsequent detection, the number of rows and columns of the grid are set. The number of rows and columns are both preset positive integers. The grid covers the entire spatial reference plane, and the grid density can accurately characterize the spatial attitude change of the plane. Then, the size of each quadrilateral grid cell is calculated. Taking the direction from the first corner feature point to the second corner feature point as the x-axis direction, the spatial distance between the two points in this direction is first calculated. The distance is the square root of the sum of the squares of the differences in the x-coordinates, the y-coordinates, and the z-coordinates of the two points. This distance is then divided by the number of grid columns minus one to obtain the side length of a single grid cell in the x-axis direction. Taking the direction perpendicular to the x-axis as the y-axis direction, two edge points on the chip body perpendicular to the x-axis direction are selected, and the spatial distance between the two points is calculated. This distance is then divided by the number of grid rows minus one to obtain the side length of a single grid cell in the y-axis direction.

[0070] The initial spatial coordinates of each control node are determined. The control nodes are arranged in a regular array in the plane. The first corner feature point is used as the origin of the first row and first column of the grid. The coordinates of each control node are calculated sequentially according to the set grid size. The x-coordinate of the control node in the i-th row and j-th column is the product of the x-coordinate of the first corner feature point plus the value of j minus one and the grid side length in the x-axis direction. The y-coordinate is the product of the y-coordinate of the first corner feature point plus the value of i minus one and the grid side length in the y-axis direction. The z-coordinate is calculated through the equation of the spatial reference plane. After substituting the x-coordinate and y-coordinate of the control node into the plane equation and deforming it, the z-coordinate is calculated as the product of the first component of the normal vector and the x-coordinate of the node, the product of the second component of the normal vector and the y-coordinate of the node, and the constant term of the plane equation. The negative number is then taken and divided by the third component of the normal vector. This is how the initial spatial coordinates of each control node are obtained, ensuring that all control nodes fall accurately on the spatial reference plane.

[0071] Spatial relationships are established between adjacent control nodes through shape functions. Each quadrilateral mesh element consists of four adjacent control nodes. The quadrilateral mesh element is mapped to a standard coordinate system with coordinates ranging from zero to one. In the standard coordinate system, the four vertices correspond to the four control nodes of the quadrilateral element. The shape function values ​​corresponding to the four control nodes are as follows: ; ; ; ; This is the shape function corresponding to the first control node; This is the shape function corresponding to the second control node; The shape function corresponding to the third control node; This is the shape function corresponding to the fourth control node; , These are dimensionless natural coordinates in the standard coordinate system.

[0072] The three-dimensional spatial coordinates of any point within the unit are calculated by weighted summation of the coordinates of the four control nodes and the corresponding shape function. The interpolation formula is as follows: ; ; ;

[0073] In the formula: , , , These are the lateral spatial coordinates of the four control nodes; , , , These are the longitudinal spatial coordinates of the four control nodes; , , , These are the vertical spatial coordinates of the four control nodes.

[0074] In this embodiment of the invention, a sub-pixel edge extraction algorithm is used to locate the corner feature points at both ends of the diagonal of the chip body, and a spatial reference plane is constructed by combining the preset positioning markers on the printed circuit board substrate. At the same time, the plane is discretized into quadrilateral grid control nodes with regular array arrangement, initial coordinates and spatial position association between adjacent nodes established by shape functions by the isoparametric unit division method. Therefore, the technical problems of difficulty in accurately representing the relative spatial attitude of the chip and the printed circuit board, irregular discretization of the spatial reference plane and lack of spatial position association of control nodes are overcome, thereby realizing accurate quantitative representation of the relative spatial attitude of the chip and the printed circuit board.

[0075] In a preferred embodiment of the present invention, virtual boundary conditions based on the relative displacement constraint between the chip body and the printed circuit board are applied to the control node. By performing shape function interpolation and stiffness matrix integration on the mesh region, the displacement field distribution of the control node in the spatial coordinate system is obtained through iterative calculation. Spatial attitude correction parameters are obtained based on the displacement field distribution, including:

[0076] Using the control nodes corresponding to the first and second corner feature points of the chip body as fixed constraint boundaries, their displacement is set to zero. Using the control nodes corresponding to the positioning markers on the printed circuit board substrate as loading boundaries, virtual displacement loads are applied along each axis of the spatial coordinate system to form virtual boundary conditions based on the relative displacement constraints between the chip body and the printed circuit board. Specifically, this includes: clarifying the basis for applying the virtual boundary conditions; using the first and second corner feature points of the chip body and the positioning markers on the printed circuit board substrate as references, finding the control nodes in the discretized grid region that coincide with the positions of these three feature points, respectively serving as fixed constraint boundaries and loading boundaries, and applying corresponding constraints and loads to form virtual boundary conditions.

[0077] First, the correspondence between feature points and control nodes is matched. By calculating the distance between the spatial coordinates of each feature point and the initial coordinates of all control nodes, the control node with the smallest distance is determined as the control node corresponding to that feature point. That is, the first corner feature point corresponds to one control node, the second corner feature point corresponds to one control node, and the positioning marker point corresponds to one control node. The control nodes corresponding to the two chip body corner feature points are set as fixed constraint boundaries, ensuring that the displacement of these two control nodes in the three axes of the spatial coordinate system is zero, thus ensuring that the relative position of the chip body is fixed and simulating the constraint state related to its own contour after chip mounting. The control node corresponding to the positioning marker point of the printed circuit board substrate is set as the loading boundary, and a virtual displacement load along the three axes of the spatial coordinate system is applied to this control node. The magnitude of the virtual displacement load is preset according to the range of small posture deviations that may occur during chip mounting, and the load direction covers the three axes, thereby simulating the possible relative displacement relationship between the chip and the printed circuit board. Through the application of the above fixed constraints and loading loads, virtual boundary conditions based on the relative displacement constraints between the chip body and the printed circuit board are formed.

[0078] Based on the virtual boundary conditions, the element stiffness matrix of each grid cell is constructed, and all element stiffness matrices are assembled according to the global number of the control nodes to form an overall stiffness matrix. Specifically, based on the applied virtual boundary conditions, for each quadrilateral grid cell obtained by discretization, an element stiffness matrix is ​​constructed separately, and all element stiffness matrices are assembled according to the global number of the control nodes to form an overall stiffness matrix covering the entire grid region, ensuring that the matrix reflects the overall mechanical properties of the grid region.

[0079] The element stiffness matrix of a single mesh element is constructed. Each quadrilateral mesh element consists of four adjacent control nodes. The construction of the element stiffness matrix is ​​based on the shape function, material properties, and element geometry of the mesh element. When calculating the element stiffness matrix, firstly, based on the established shape function, the partial derivatives of the shape function with respect to two coordinates in the standard coordinate system are calculated. Then, the partial derivatives, virtual material parameters such as elastic modulus and Poisson's ratio, and the area of ​​the mesh element are calculated using the initial coordinates of the four control nodes of the element. The area is half the product of the diagonals of the quadrilateral. These are substituted into the stiffness matrix calculation formula, and the value of the element stiffness matrix is ​​calculated element by element. Each element of the element stiffness matrix is ​​a mechanical correlation coefficient between control nodes, reflecting the force and displacement transmission relationship between control nodes. The size of the element stiffness matrix corresponds to the degrees of freedom of the element control nodes. Each control node has three axial degrees of freedom, and four control nodes correspond to twelve degrees of freedom. Therefore, the element stiffness matrix is ​​twelve rows and twelve columns.

[0080] After the element stiffness matrix of a single mesh element is constructed, the assembly of the overall stiffness matrix begins. First, all control nodes are globally numbered. Following the order of the mesh array, starting with the control node in the first row and first column, a unique global number is assigned to each control node, ensuring that each control node's number is unique and complete. Following the principle that elements of the element stiffness matrix correspond to the global numbers of the control nodes, each element of the element stiffness matrix is ​​mapped one by one to the corresponding position in the overall stiffness matrix. If multiple element stiffness matrices correspond to the same global node combination, the elements at the corresponding positions are added together. This process completes the assembly of all element stiffness matrices. The assembled overall stiffness matrix has both rows and columns equal to the total number of degrees of freedom of all control nodes. The overall stiffness matrix comprehensively reflects the mechanical relationships between all control nodes throughout the entire mesh region.

[0081] The nonlinear equations consisting of the global stiffness matrix and load vectors are solved using the Newton-Raphson iterative method. After each iteration, the displacement increments of the control nodes are updated until the norm of the displacement increments between two adjacent iterations is less than a preset threshold. This yields the displacement field distribution of each control node in the spatial coordinate system. Specifically, this involves: initializing the displacement vectors of the control nodes by setting the initial displacements of all control nodes to zero; constructing load vectors where each element corresponds one-to-one with the global degrees of freedom of the control nodes; and setting the load vector elements of the corresponding degrees of freedom of only the loading boundary control nodes to which virtual displacement loads are applied to have the preset virtual displacement load magnitude, while the load vector elements of the corresponding degrees of freedom of the other control nodes are all zero. The iterative calculation then begins, and the current iteration number is calculated. The residual vector is calculated by subtracting the product of the current global stiffness matrix and the current displacement vector from the load vector. Each element of the residual vector equals the load value for the corresponding degree of freedom minus the sum of the products of the corresponding row of the global stiffness matrix and the displacement vector. The displacement increment vector is calculated by multiplying the inverse of the current global stiffness matrix and the residual vector. During the calculation, the global stiffness matrix is ​​inverted, and the inverse matrix is ​​multiplied element-wise by the residual vector and summed to obtain the displacement increments for each control node along the three axes. The displacement increment norm is then calculated to determine if the iteration has converged. The displacement increment norm is calculated by summing the squares of all elements in the displacement increment vector and then taking the square root of the sum.

[0082] The calculated displacement increment norm is compared with a preset threshold. If the displacement increment norm is greater than or equal to the preset threshold, it indicates that the iteration has not converged. The current displacement vector is added to the displacement increment vector to obtain the updated displacement vector. At the same time, the overall stiffness matrix is ​​recalculated based on the updated displacement vector, and the above steps of residual calculation, displacement increment calculation, and convergence judgment are repeated. If the displacement increment norm is less than the preset threshold, it indicates that the iteration has converged, and the iteration is stopped. The displacement vector at this time is the displacement field distribution of all control nodes in the spatial coordinate system. This displacement field distribution accurately reflects the spatial displacement of each control node under virtual boundary conditions, and indirectly reflects the relative spatial attitude deviation between the chip body and the printed circuit board.

[0083] The out-of-plane displacement components of each control node along the imaging optical axis are extracted from the displacement field distribution. Spatial attitude correction parameters are obtained by fitting the correspondence between the out-of-plane displacement components and spatial coordinates. Specifically, this includes: defining the out-of-plane displacement component, which refers to the displacement of the control node along the imaging optical axis, which is perpendicular to the printed circuit board substrate and corresponds to the vertical direction of the spatial coordinate system. Therefore, the vertical displacement value in the displacement field distribution of each control node is extracted as the out-of-plane displacement component of that control node. At the same time, the initial spatial coordinates of each control node are recorded to form a set of correspondences between the spatial coordinates of the control node and the out-of-plane displacement components.

[0084] Spatial attitude correction parameters are obtained by fitting the above-mentioned set of correspondences using the least squares method. These parameters include tilt and rotation corrections. The fitting process is based on a plane fitting model, assuming a linear relationship between the out-of-plane displacement components and the lateral and longitudinal coordinates of the control nodes, and a fitting equation is constructed. The calculation process of the fitting equation is as follows: , Represents the actual out-of-plane displacement component of the control node; This represents the tilt correction amount corresponding to the horizontal coordinate; This represents the tilt correction amount corresponding to the longitudinal coordinate; Represents the rotation correction amount; Represents the horizontal spatial coordinates of the control node; The longitudinal spatial coordinates of the control nodes are represented. The lateral, longitudinal, and off-plane displacement components of all control nodes are substituted into the fitting equation to calculate the fitting error for each control node. The fitting error is the difference between the actual off-plane displacement component and the off-plane displacement component calculated by the fitting equation. By adjusting the tilt and rotation correction values ​​in the fitting equation, the sum of squares of the fitting errors for all control nodes is minimized. These tilt and rotation correction values ​​are the required spatial attitude correction parameters, which precisely quantify the minute tilt and rotation deviations of the chip body relative to the printed circuit board.

[0085] In this embodiment of the invention, virtual boundary conditions are constructed by setting the control nodes corresponding to the edge feature points of the chip body as fixed constraint boundaries and the control nodes corresponding to the positioning marker points of the printed circuit board as loading boundaries. The unit stiffness matrix is ​​integrated to form an overall stiffness matrix. The displacement field distribution is obtained by solving the nonlinear equation system through the Newton-Raphson iterative method, and the out-of-plane displacement components are extracted to fit the spatial attitude correction parameters. Therefore, the technical problems of insufficient accuracy in solving the spatial attitude correction parameters under the relative displacement constraints between the chip body and the printed circuit board, poor convergence of nonlinear displacement calculation, and inaccurate solution of the displacement field of the mesh nodes are overcome. Thus, the accurate solution and stable iterative convergence of the spatial displacement field of the control nodes are realized, and high-precision spatial attitude correction parameters are obtained efficiently.

[0086] In a preferred embodiment of the present invention, a projection transformation is performed on the pin connected component image according to spatial attitude correction parameters to obtain a corrected pin connected component image, including:

[0087] The spatial attitude correction parameters are obtained, including tilt correction and rotation correction. The tilt correction and rotation correction are then used to construct a perspective projection transformation matrix from three-dimensional space to a two-dimensional imaging plane. Specifically, this involves: obtaining the fitted spatial attitude correction parameters, which include two sets of tilt correction and one set of rotation correction. The two sets of tilt correction correspond to the tilt deviation of the chip relative to the printed circuit board in the horizontal and vertical directions, respectively, while the rotation correction corresponds to the rotation deviation of the chip relative to the printed circuit board. Based on these three sets of correction, a perspective projection transformation matrix is ​​constructed to map the three-dimensional spatial attitude deviation to the two-dimensional imaging plane, thereby achieving reverse correction of image distortion.

[0088] The structure of the perspective projection transformation matrix is ​​defined. This matrix is ​​a 4x4 matrix, and its elements are determined by spatial attitude correction parameters, including tilt correction and rotation correction, industrial camera intrinsic parameters, including the preset camera focal length, and the center coordinates of the imaging plane. Each element of the matrix corresponds to a mapping relationship from three-dimensional spatial coordinates to two-dimensional imaging plane coordinates, accurately compensating for image distortion caused by chip spatial attitude deviations. The values ​​of each element of the matrix are calculated. First, the rotation correlation coefficient is calculated based on the rotation correction, which is the rotation angle of the chip. The cosine and sine values ​​of this angle are calculated. The cosine value is the length of the adjacent side corresponding to the rotation angle divided by the length of the hypotenuse, and the sine value is the length of the opposite side corresponding to the rotation angle divided by the length of the hypotenuse. Then, combined with the two sets of tilt correction values, the lateral tilt coefficient and the longitudinal tilt coefficient are calculated respectively. The lateral tilt coefficient is the lateral tilt correction multiplied by the camera focal length, and the longitudinal tilt coefficient is the longitudinal tilt correction multiplied by the camera focal length.

[0089] Each element of the four-row, four-column perspective projection transformation matrix is ​​determined as follows: the element in the first row and first column is the cosine of the rotation angle; the element in the first row and second column is the negative sine of the rotation angle; the element in the first row and third column is the lateral tilt coefficient; the element in the first row and fourth column is the product of the image plane center x-coordinate and the lateral tilt coefficient, minus the product of the camera focal length and the lateral offset. The lateral offset is a preset fixed value used to compensate for camera installation deviations. The element in the second row and first column is the sine of the rotation angle; the element in the second row and second column is the cosine of the rotation angle; the element in the second row and third column is the longitudinal tilt coefficient; the element in the second row and fourth column is the lateral tilt coefficient; the element in the second row and fourth column is the product of the image plane center x-coordinate and the lateral tilt coefficient, minus the product of the camera focal length and the lateral offset. The lateral offset is a preset fixed value used to compensate for camera installation deviations. The element in the fourth column of the matrix is ​​the product of the ordinate of the imaging plane center and the longitudinal tilt coefficient, minus the product of the camera focal length and the longitudinal offset, where the longitudinal offset is also a preset fixed value; the elements in the first and second columns of the third row of the matrix are all zero, the element in the third column of the third row is the camera focal length, and the element in the fourth column of the third row is zero; the elements in the first and second columns of the fourth row of the matrix are all zero, the element in the third column of the fourth row is the tilt compensation coefficient, which is obtained by weighted summation of two sets of tilt corrections, with the weight being a preset proportional coefficient, and the element in the fourth column of the fourth row is one; after all elements are calculated, they are integrated to form a complete perspective projection transformation matrix.

[0090] Each pixel in the pin connected component image is used as a transformation source point. The mapping coordinates of each pixel in the corrected image space are calculated according to the perspective projection transformation matrix. By resampling the gray values ​​at the mapping coordinates, the corrected pin connected component image with spatial pose distortion eliminated is obtained. Specifically, after completing the construction of the perspective projection transformation matrix, the obtained pin connected component image is used as the processing object. Each pixel in the image is used as a transformation source point. The mapping coordinates of the pixel in the corrected image space are calculated according to the perspective projection transformation matrix. Then, the gray values ​​at the mapping coordinates are resampled to fill the gray value gaps caused by the coordinate deviation. Finally, the corrected pin connected component image with spatial pose distortion eliminated and edges regularized is obtained.

[0091] Traverse all pixels in the pin connected component image, determining the source coordinates of each pixel. The source coordinates are the horizontal and vertical coordinates of the pixel in the original pin connected component image. Simultaneously, add a vertical coordinate to each source coordinate to form the 3D source coordinates of each pixel. Calculate the mapped coordinates of each pixel by performing matrix multiplication on the constructed perspective projection transformation matrix. The operation is as follows: multiply the horizontal coordinate of the 3D source coordinate by the element in the first row and first column of the matrix, add the vertical coordinate of the 3D source coordinate multiplied by the element in the first row and second column of the matrix, add the vertical coordinate of the 3D source coordinate multiplied by the element in the first row and third column of the matrix, and add the element in the first row and fourth column of the matrix to obtain the horizontal numerator of the mapped coordinates. Similarly, multiply the horizontal coordinate of the 3D source coordinate by the matrix... The vertical numerator of the mapped coordinates is calculated by multiplying the vertical coordinate of the 3D source coordinates by the element in the second row and first column of the matrix, multiplying the vertical coordinate of the 3D source coordinates by the element in the second row and third column of the matrix, and then adding the element in the second row and fourth column of the matrix. The horizontal coordinate of the 3D source coordinates is multiplied by the element in the fourth row and first column of the matrix, multiplied by the vertical coordinate of the 3D source coordinates by the element in the fourth row and second column of the matrix, multiplied by the vertical coordinate of the 3D source coordinates by the element in the fourth row and third column of the matrix, and then added to the element in the fourth row and fourth column of the matrix. This yields the denominator of the mapped coordinates. The horizontal numerator is divided by the denominator to obtain the horizontal coordinate of the mapped coordinates. The vertical numerator is divided by the denominator to obtain the vertical coordinate of the mapped coordinates. This completes the calculation of the mapped coordinates for a single pixel.

[0092] The third step is to determine the validity of the mapped coordinates. If both the horizontal and vertical coordinates of the mapped coordinates are within the preset range of the corrected image size, the mapped coordinates are retained and subsequent grayscale resampling is performed. If the mapped coordinates exceed the image boundary, the pixel is determined to be an invalid pixel and is not resampled; it is subsequently filled with background grayscale values. Grayscale resampling is then performed. Since the calculated mapped coordinates may be non-integer and cannot directly correspond to the pixels in the corrected image, bilinear interpolation is used for resampling. The specific calculation process is as follows: find the four nearest integer coordinate pixels around the mapped coordinates and determine the grayscale values ​​of these four pixels; calculate the horizontal and vertical distances between the mapped coordinates and these four integer coordinates, using the reciprocal of the distance as the weight, with a larger weight for closer distances; multiply the grayscale value of each integer coordinate pixel by its corresponding weight, and then add the four products together to obtain the grayscale value at the mapped coordinates. This grayscale value is the grayscale value of the corresponding pixel in the corrected image.

[0093] Repeat the above steps to traverse all pixels in the original pin connected domain image, complete the mapping coordinate calculation and grayscale value resampling for each pixel, and finally integrate them to form a complete corrected pin connected domain image. This image has eliminated the distortion caused by chip space orientation tilt and rotation, and the pin area edges are continuous and regular.

[0094] In this embodiment of the invention, a perspective projection transformation matrix from three-dimensional space to two-dimensional imaging plane is constructed based on spatial attitude correction parameters including tilt correction and rotation correction. The technical means of calculating the mapping coordinates and resampling the gray values ​​of each pixel in the pin connected domain image are used. Therefore, the technical problems of distortion, inaccurate pixel coordinate mapping, and loss of gray information in the corrected image caused by spatial attitude distortion of the pin connected domain image are overcome. Thus, the spatial attitude distortion of the pin connected domain image is accurately eliminated, and a regular and clear corrected image is obtained.

[0095] In a preferred embodiment of the present invention, the contour of a single pin is extracted from the corrected pin connected component image, and the theoretical edge line of each pin is obtained by fitting the edge contour point set of each pin, including:

[0096] A connected component labeling algorithm is performed on the corrected pin connected component image. Each pin region is independently labeled according to the eight-neighbor connectivity criterion to obtain the independent connected component corresponding to each pin. Specifically, the corrected pin connected component image is preprocessed and binarized again to be consistent with the obtained binarized mask image. The pin region is the first gray value and the background region is the second gray value to ensure that the gray value difference between the pin region and the background region is clear and to eliminate the interference of slight gray value noise on the connected component labeling. The labeling parameters are initialized, the label number is set to start from 1, and a labeling image with the same size as the corrected pin connected component image is created. In the initial state, the label value of all pixels is 0.

[0097] Following a left-to-right, top-to-bottom order, the corrected pin-connected region image is traversed pixel by pixel. It is determined whether the current pixel is a pin pixel (i.e., its grayscale value is the first grayscale value and its marker value is 0). If the condition is met, the pixel is determined to be the starting pixel of a new connected region, and its marker value is set as the current marker number. Based on the eight-neighbor connectivity criterion, all adjacent pin pixels of the starting pixel are searched. An eight-neighbor region refers to the eight adjacent pixels in the four directions (up, down, left, right) and the four diagonal directions (upper left, upper right, lower left, lower right). Each of these eight adjacent pixels is checked to see if it is a pin pixel and is unmarked. If the condition is met, its marker value is set as the current marker number. Simultaneously, using these adjacent pixels as new starting points, the eight-neighbor search process is repeated until all pin pixels within the connected region are marked.

[0098] After marking a connected component, increment the mark number by 1 and continue traversing the image to find the next unmarked pin pixel until all pixels in the image have been traversed and all pin regions have been marked. Finally, a marked image is obtained, where each mark number corresponds to an independent pin connected component, and each independent connected component is the region of a single pin, thus achieving accurate separation and calibration of a single pin.

[0099] For each independent connected component, extract the closed contour point sequence pixel by pixel along the boundary of the connected component to obtain the contour point set of a single pin. Specifically, this includes: traversing the obtained marked image, sequentially selecting the independent connected component corresponding to each mark number, and for a single independent connected component, first determining the boundary pixel judgment criteria of the connected component: if a pixel belongs to the connected component with the current mark number, and at least one pixel in its eight neighboring regions does not belong to the connected component with the mark value of 0, i.e., the background pixel, then the pixel is the boundary pixel of the connected component.

[0100] Find the starting boundary pixel of the connected component. Start traversing from the top-left pixel of the connected component and find the first pixel that meets the boundary pixel judgment criteria. Take this pixel as the starting boundary pixel and record its horizontal and vertical coordinates in the image. Extract contour points pixel by pixel along the boundary pixels. Starting from the starting boundary pixel, search for adjacent boundary pixels in a clockwise direction. The search order is as follows: first, determine whether the pixel to the right of the current pixel is a boundary pixel. If not, determine the pixel to the bottom right. Then search in the order of right, bottom right, bottom, bottom left, left, top left, top, top right to ensure that each time an adjacent boundary pixel is found, avoiding skipping boundaries or repeated extraction.

[0101] For each boundary pixel found, its horizontal and vertical coordinates are recorded and added to the contour point sequence. Simultaneously, the pixel is marked as extracted to avoid duplicate recording. The fifth step continues to traverse the boundary pixels until the found boundary pixel completely overlaps with the initial boundary pixel. At this point, the contour point sequence forms a closed loop, and extraction stops. This closed contour point sequence is the contour point set for the current single pin. The above operations are repeated to extract the boundary contour for each independent connected component, obtaining the contour point set corresponding to each single pin. Each contour point set contains the pixel coordinates of all pixels on the pin's boundary, fully reflecting the pin's boundary shape.

[0102] For each pin's contour point set, two edge lines are fitted. Using the pin's extension direction as a reference, points located on both sides of the pin's boundary are grouped. By calculating the slope and intercept parameters of the two edge lines, the theoretical left and right edge lines for each pin are obtained. Specifically, this includes: determining the extension direction of each pin, which is the direction extending outward from the chip body; analyzing the coordinate distribution of a single pin's contour point set; calculating the average horizontal and vertical coordinates of all pixels in the contour point set; and determining the direction of the pin's central axis, which is the pin's extension direction. For example, if the vertical coordinate variation range of the contour point set is much larger than the horizontal coordinate variation range, the pin's extension direction is vertical; if the horizontal coordinate variation range is larger, the extension direction is horizontal.

[0103] The contour point set is grouped into a left boundary point group and a right boundary point group. Taking the pin extension direction as the vertical direction as an example, the lateral distance from each contour point to the central axis is calculated. The lateral coordinate of the central axis is the average of the lateral coordinates of all contour points. The lateral coordinate of a contour point is subtracted from the lateral coordinate of the central axis. If the result is negative, the point is a left boundary point and is added to the left boundary point group; if the result is positive, the point is a right boundary point and is added to the right boundary point group; if the result is zero, the point is a point on the central axis and is not included in either group. For pins with a lateral extension direction, the same method is used to determine the left and right boundary points based on the vertical distance, thus completing the grouping.

[0104] To fit the theoretical left edge line, the least squares method is used. The equation of the theoretical left edge line is assumed to be the vertical coordinate equal to the slope multiplied by the horizontal coordinate plus the intercept. First, the relevant parameters of all points in the left boundary point group are calculated: the total number of left boundary points is counted, and the sum of the horizontal coordinates, the sum of the vertical coordinates, the sum of the products of the horizontal and vertical coordinates, and the sum of the squares of the horizontal coordinates are calculated for all points.

[0105] The slope is calculated as follows: multiply the total number of points on the left boundary by the sum of the products of the horizontal and vertical coordinates of all points, and subtract the sum of the horizontal coordinates of all points multiplied by the sum of the vertical coordinates of all points to obtain the numerator; multiply the total number of points on the left boundary by the sum of the squares of the horizontal coordinates of all points, and subtract the square of the sum of the horizontal coordinates of all points to obtain the denominator; divide the numerator by the denominator to obtain the slope of the theoretical left edge line.

[0106] The intercept is calculated as follows: subtract the slope multiplied by the sum of the horizontal coordinates of all points from the sum of the vertical coordinates of all points to obtain the numerator; divide the numerator by the total number of points on the left boundary to obtain the intercept of the theoretical left edge line.

[0107] Using the same fitting method as the left theoretical edge line, the right boundary point group is fitted, and the slope and intercept of the right theoretical edge line are calculated. The above operation is repeated to group and fit the contour point set of each individual pin, and finally the left theoretical edge line and right theoretical edge line corresponding to each pin are obtained.

[0108] By performing a connected component labeling algorithm on the corrected pin connected component image and independently labeling each pin region using the eight-neighbor connectivity criterion, extracting the closed contour point sequence of a single pin pixel by pixel, and then fitting the theoretical edge lines on both sides of the boundary points in groups according to the pin extension direction, the technical problems of pin regions interfering with each other and being difficult to segment independently, incomplete contour extraction, and large deviations in line fitting caused by disordered edge points are overcome. Thus, accurate segmentation and complete contour extraction of a single pin are achieved, and high-precision theoretical edge lines on both sides of the pin are fitted.

[0109] like Figure 2 As shown, in another preferred embodiment of the present invention, the deviation between the edge contour point set of each pin and the corresponding theoretical edge line is calculated to obtain the detection result, including:

[0110] Obtain the theoretical left and right edge lines for each pin, and obtain subsets of contour points located at the left and right edges of the contour point set for each pin. Calculate the Euclidean distance from each contour point in the left contour point subset to the corresponding theoretical left edge line to obtain the left edge deviation sequence, and the Euclidean distance from each contour point in the right contour point subset to the corresponding theoretical right edge line to obtain the right edge deviation sequence. Specifically, this includes: obtaining the core data obtained from the previous processing, including the theoretical left and right edge lines for each pin, and the contour point set for each pin; simultaneously extracting the left and right contour point subsets from the contour point set. These subsets completely correspond to the grouped left and right boundary point groups, with all left contour point subsets located at the left edge of the pin and all right contour point subsets located at the right edge of the pin, ensuring the specificity of the deviation calculation.

[0111] Calculate the Euclidean distance from each contour point in the left contour point subset to the theoretical left edge line, forming a left edge deviation sequence. For each contour point in the left contour point subset, obtain its horizontal and vertical coordinates. Substitute these coordinates into the Euclidean distance calculation formula. The calculation process is as follows: multiply the slope of the contour point by its horizontal coordinate, subtract its vertical coordinate, and add the intercept in the standard form of the theoretical left edge line. Take the absolute value of the result as the numerator. Add one to the square of the slope, and take the square root of the result as the denominator. Divide the numerator by the denominator. The resulting value is the Euclidean distance from the contour point to the theoretical left edge line, which is the deviation value of a single left contour point. Calculate the deviation values ​​of all contour points in the left contour point subset sequentially. Arrange these deviation values ​​according to the contour point extraction order to form the left edge deviation sequence of the pin. Each value in the sequence corresponds to the deviation of a left contour point.

[0112] Using the same calculation method as the left edge deviation sequence, the Euclidean distance from each contour point in the right contour point subset to the right theoretical edge line is calculated to form the right edge deviation sequence; that is, the right theoretical edge line is first arranged into a standard form, and then the Euclidean distance from each right contour point to the line is calculated one by one. All deviation values ​​are arranged in the extraction order to obtain the right edge deviation sequence of the pin.

[0113] The left and right edge deviation sequences of each pin are fused to calculate the cumulative deviation of each pin in the width direction. The cumulative deviations of all pins are then used to construct a deviation distribution curve according to their pin arrangement. A sliding window filtering method is used to smooth the deviation distribution curve, extracting continuous peak segments exceeding a preset deviation threshold. Specifically, this involves: fusing the left and right edge deviation sequences of each pin to calculate the cumulative deviation of a single pin in the width direction; the fusion method is as follows: first, the sum of all deviation values ​​in the left edge deviation sequence is calculated to obtain the left cumulative deviation; then, the sum of all deviation values ​​in the right edge deviation sequence is calculated to obtain the right cumulative deviation; the left and right cumulative deviations are added together to obtain the cumulative deviation of the pin in the width direction. The larger the cumulative deviation, the more significant the offset of the pin. Simultaneously, to avoid interference from individual abnormal deviation points, extreme deviation values ​​exceeding a reasonable range in each deviation sequence are removed before calculating the cumulative deviation to ensure the accuracy of the cumulative deviation.

[0114] Construct a deviation distribution curve by arranging the cumulative deviation of all pins in the actual pin arrangement order on the chip. Plot the cumulative deviation of each pin on the horizontal axis and the cumulative deviation of each pin on the vertical axis to form a deviation distribution curve. This curve visually reflects the deviation distribution of all pins. The cumulative deviation of normal pins is small and within a preset reasonable range, while the cumulative deviation of pins with offset defects will be significantly higher, forming a peak on the curve.

[0115] A sliding window filtering method is used to smooth the deviation distribution curve. The purpose is to eliminate noise interference in the curve, make the peak sections clearer, and avoid misjudgments caused by noise. The specific filtering process is as follows: The size of the sliding window is preset. The window size is set according to the total number of pins, usually an interval corresponding to 3 to 5 pins. The sliding window is moved point by point along the horizontal axis of the deviation distribution curve. At each position, the average value of the cumulative deviation of all pins within the window is calculated. This average value is used to replace the cumulative deviation of the pin at the center of the window. The window is moved in sequence until it covers the entire deviation distribution curve, resulting in a smoothed deviation distribution curve. The smoothed curve effectively filters out local noise and highlights the true abnormal deviation peaks.

[0116] Extract continuous peak segments from the deviation distribution curve that exceed a preset deviation threshold. First, set a preset deviation threshold, which is based on the allowable offset range of the chip pins and is the core standard for determining whether a pin has an offset defect. The threshold value needs to be determined based on actual production line testing requirements and chip specifications. Traverse the smoothed deviation distribution curve and determine whether the cumulative deviation of each pin is greater than the preset deviation threshold. If the cumulative deviation of a pin is greater than the threshold, and the cumulative deviations of one or more adjacent pins are also greater than the threshold, then the curve segments corresponding to these consecutive pins are continuous peak segments. Each continuous peak segment corresponds to a group of pins with offset defects. The length of the segment corresponds to the number of defective pins, and the peak deviation of the segment corresponds to the maximum offset degree of the defective pins.

[0117] Based on the position index of the continuous peak segment in the pin arrangement sequence, the pin number with the offset defect is determined. Combining the peak deviation and deviation change rate of the continuous peak segment, the severity of the offset defect is graded, resulting in a detection result including the defective pin number, offset direction, offset amount, and severity grade. Specifically, this includes determining the pin number with the offset defect. The horizontal axis of the deviation distribution curve represents the pin arrangement sequence, with each horizontal axis coordinate corresponding to a unique pin number. By mapping the start and end position indices of the continuous peak segment on the horizontal axis to the pin arrangement sequence, the pin number with the offset defect can be obtained. For example, if the start position index of the continuous peak segment is 5 and the end position index is 7, then pins numbered 5, 6, and 7 are all pins with offset defects. If the continuous peak segment has a single position index, then the single pin corresponding to that index is the defective pin, ensuring accurate location of the defective pin number.

[0118] Determine the offset direction of each defective pin, and combine the obtained left edge deviation sequence and right edge deviation sequence to compare the magnitude of the cumulative deviation on the left and the cumulative deviation on the right: if the cumulative deviation on the left is greater than the cumulative deviation on the right, it means that the left contour of the pin deviates from the theoretical edge to a greater extent, and the offset direction of the pin is determined to be to the left; if the cumulative deviation on the right is greater than the cumulative deviation on the left, it means that the right contour of the pin deviates from the theoretical edge to a greater extent, and the offset direction of the pin is determined to be to the right; if the cumulative deviations on both sides are small, the pin is determined to be bidirectionally offset, and the offset direction is to both sides.

[0119] The offset of each defective pin is determined, with the peak deviation of the continuous peak segment as the core basis. The peak deviation is the largest cumulative deviation in the continuous peak segment. For a single defective pin, its offset is its own cumulative deviation. For a segment composed of multiple consecutive defective pins, the offset of each defective pin is its own cumulative deviation. At the same time, the peak deviation of the segment is recorded as a reference for the maximum offset of the group of defective pins.

[0120] By combining the peak deviation and the deviation change rate, the severity of the offset defect is classified. First, the deviation change rate is calculated as follows: Select the cumulative deviation of the starting pin and the cumulative deviation of the peak pin in the continuous peak segment. Subtract the cumulative deviation of the starting pin from the cumulative deviation of the peak pin to obtain the deviation increment. Subtract the number of the starting pin from the number of the peak pin to obtain the pin number increment. Divide the deviation increment by the pin number increment to obtain the deviation change rate, which reflects the rate of change of the defect pin offset.

[0121] Based on preset grading standards, the system is classified into three levels, based on peak deviation and deviation change rate: Level 1 (Mild Deviation): The peak deviation is greater than the preset deviation threshold but less than the threshold separating mild and moderate deviations, and the deviation change rate is less than the preset change rate threshold. In this case, the pin deviation is small and does not affect subsequent soldering quality. Level 2 (Moderate Deviation): The peak deviation is greater than or equal to the threshold separating mild and moderate deviations but less than the threshold separating moderate and severe deviations, and the deviation change rate is greater than or equal to the preset change rate threshold but less than the severe change rate threshold. In this case, the pin deviation is moderate and requires subsequent manual review. Level 3 (Severe Deviation): The peak deviation is greater than or equal to the threshold separating moderate and severe deviations, and the deviation change rate is greater than or equal to the severe change rate threshold. In this case, the pin deviation is large and will affect soldering quality, requiring immediate shutdown for processing.

[0122] After completing all the above operations, the integrated results are obtained, which include the defective pin number, offset direction, offset amount, and severity level, accurately and comprehensively reflecting the specific situation of the pin offset defect.

[0123] In this embodiment of the invention, the Euclidean distances from the contour points on both sides of the pin to the corresponding theoretical edge lines are calculated to obtain the edge deviation sequence. The cumulative deviation of the width is calculated and a deviation distribution curve is constructed. The curve is smoothed by sliding window filtering, and the continuous peak segments exceeding the threshold are extracted. Then, the defective pin number is determined and the severity of the offset defect is classified. Therefore, this method overcomes the technical problems of single pin edge deviation evaluation, deviation distribution being easily affected by noise, inaccurate positioning of defective pins, and inability to quantify and classify the severity of offset. It achieves accurate quantitative analysis of pin offset deviation and rapid and accurate positioning of defective pins, completes intelligent classification and judgment of offset defects, and outputs complete detection results including pin number, offset direction, offset amount, and severity, thereby improving the accuracy and reliability of pin offset defect detection.

[0124] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the system as described above. All implementations in the above system embodiments are applicable to this embodiment and can achieve the same technical effects.

[0125] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the system as described above. All implementations in the above system embodiments are applicable to this embodiment and can achieve the same technical effects.

[0126] Experimental example:

[0127] I. Experimental Background

[0128] This experimental example validates a deep learning-based online visual inspection system for component pin defects. The experimental scenario is a surface mount production line in an electronics manufacturing company. The system is deployed at the inspection station after surface mount soldering and before reflow soldering, and is used to detect pin defects in various types of chips, including prismatic flat packages, thin quad flat packages, and ball grid array packages. The experiment aims to verify the system's pin segmentation accuracy, spatial orientation correction effect, and defect detection accuracy.

[0129] II. Experimental Equipment and Parameters

[0130] The experimental equipment includes: a 5-megapixel industrial camera (equipped with a telecentric lens and a magnification of 0.5x), a ring LED light source (wavelength 520 nm), an AI inference workstation (equipped with an NVIDIA RTX 4090 graphics card), and a deep learning segmentation network with an encoder-decoder structure.

[0131] Experimental parameter settings: image acquisition resolution 2448 x 2048 pixels, exposure time 5 ms, image enhancement using adaptive histogram equalization (CLAHE, clipLimit2, tileSize8×8) and homomorphic filtering (high-pass filter cutoff frequency 30 Hz); deep learning segmentation network using ResNet-50 encoder and FPN decoder, input image size 512 x 512 pixels; defect judgment threshold set to cumulative deviation exceeding 15 micrometers.

[0132] III. Experimental Procedure

[0133] Figure 3This study demonstrates the accuracy verification of pin defect detection for different chip types, testing the pin defect detection accuracy of five common chip types. The detection accuracy is 96.2% for a 64-pin rectangular flat package (KP), 94.8% for a 128-pin KP, 93.5% for a 256-pin KP, 95.3% for a 100-pin thin quad flat package (TBP), and 91.7% for a ball grid array (BGA) package. While the detection accuracy decreases slightly with increasing pin count, it remains above 90% for all types.

[0134] Figure 4 This paper demonstrates the analysis of the deep learning model training process and monitors the training process of the deep learning segmentation network. In the early training phase (0-20 epochs), the training loss rapidly decreased from 0.85 to 0.45, and the validation accuracy improved from 72.5% to 83.6%. In the middle training phase (20-60 epochs), the loss continued to decrease to 0.18, and the accuracy improved to 92.3%. In the later training phase (60-100 epochs), the loss converged to 0.075, and the accuracy stabilized at 95.4%. The model essentially converged after 80 epochs.

[0135] Figure 5 The report presents statistics on the detection rates and false positive rates for different defect types, and includes specific testing for five common pin defect types. The detection rate for pin offset defects was 97.5%, with a false positive rate of 2.1%; the detection rate for missing pin defects was 99.2%, with a false positive rate of 0.8%; the detection rate for pin adhesion defects was 98.8%, with a false positive rate of 1.2%; the detection rate for pin warping defects was 96.3%, with a false positive rate of 3.2%; and the detection rate for flux residue defects was 94.5%, with a false positive rate of 4.8%. The detection results for missing pins and adhesion defects were the best.

[0136] Figure 6 This paper demonstrates a comparison of inspection efficiency before and after system implementation, highlighting the differences in inspection efficiency indicators. Before implementation, manual visual inspection was used, with an inspection speed of 120 pieces per hour, an accuracy rate of 87.5%, a false negative rate of 8.5%, a single-piece inspection time of 30 seconds, and a manual review rate of 25%. After implementing the system of this invention, the inspection speed increased to 450 pieces per hour (an improvement of 275%), the inspection accuracy increased to 95.4%, the false negative rate decreased to 2.1%, the single-piece inspection time was shortened to 8 seconds, and the manual review rate decreased to 5%.

[0137] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based online visual inspection system for component pin defects, characterized in that, include: The acquisition module is used to synchronously acquire images of the printed circuit board after surface mount soldering and before entering the reflow soldering process using an industrial camera, and to capture the original pin images of the complete chip pins. The original pin image is enhanced to obtain the enhanced pin image. The enhancement module is used to input the enhanced pin image into a pre-trained deep learning segmentation network to obtain a pin mask image; and to perform morphological processing on the pin mask image to obtain a pin connected component image. The calculation module is used to construct a spatial reference plane in the pin connected domain image using the first and second corner feature points of the chip body; discretize the spatial reference plane into a grid region composed of control nodes; apply virtual boundary conditions to the control nodes based on the relative displacement constraints between the chip body and the printed circuit board; obtain the displacement field distribution of the control nodes in the spatial coordinate system through shape function interpolation and stiffness matrix integration of the grid region, and obtain the spatial attitude correction parameters based on the displacement field distribution. The correction module is used to perform projection transformation on the pin connected component image according to the spatial attitude correction parameters to obtain the corrected pin connected component image; extract the profile of a single pin from the corrected pin connected component image; and obtain the theoretical edge line of each pin by extracting the edge profile point set of each pin. The processing module is used to calculate the deviation between the edge contour point set of each pin and the corresponding theoretical edge line to obtain the detection result.

2. The deep learning-based online visual inspection system for component pin defects according to claim 1, characterized in that, The original pin images of the complete chip pins are captured by simultaneously acquiring images of the printed circuit board after surface mount soldering and before entering the reflow soldering process using an industrial camera. The original pin image is enhanced to obtain the enhanced pin image, including: The printed circuit board image is synchronously acquired by an industrial camera under the control of a trigger signal. The chip area is located in the printed circuit board image according to the template matching result of the square flat package chip. The image area containing all pins is cropped based on the outline of the chip body to obtain the original pin image. Adaptive histogram equalization is performed on the original pin image. The contrast stretching mapping curve is calculated based on the local gray-level distribution of the image. A homomorphic filtering algorithm based on illumination component estimation is used to suppress the reflection difference between the flux residue area and the metal pin surface, thereby enhancing the gray-level gradient between the pin area and the background area, resulting in an enhanced pin image.

3. The deep learning-based online visual inspection system for component pin defects according to claim 2, characterized in that, The enhanced pin image is input into a pre-trained deep learning segmentation network to obtain a pin mask image; morphological processing is then performed on the pin mask image to obtain a pin connected component image, including: The enhanced pin image is input into a pre-trained encoder-decoder deep learning segmentation network. The encoder extracts multi-scale semantic features of the enhanced pin image through multi-layer convolution and pooling operations. The decoder restores the multi-scale semantic features to the original image resolution through upsampling and skip connections, and outputs a pin category probability map pixel by pixel. After thresholding, a binarized pin mask image is obtained. In the pin mask image, the pin region is assigned a first gray value and the background region is assigned a second gray value. The pin mask image is subjected to a morphological opening operation of erosion followed by dilation to filter out isolated noise points caused by flux residue. Then, a morphological closing operation of dilation followed by erosion is performed to fill the tiny holes inside the pin area caused by edge blurring and to connect the gaps formed by the segmentation and breakage between adjacent pins, resulting in a pin connected region image in which each pin area is separated from each other and the edges are continuously closed.

4. The deep learning-based online visual inspection system for component pin defects according to claim 3, characterized in that, In the pin connected domain image, a spatial reference plane is constructed using the first and second corner feature points of the chip body; The spatial reference plane is discretized into a grid region composed of control nodes, including: In the pin connected domain image, the first and second corner feature points of the chip body are located by the sub-pixel edge extraction algorithm. The first and second corner feature points are located at the two ends of the diagonal of the chip body. At the same time, the preset positioning mark points on the printed circuit board substrate are extracted. The first corner feature point, the second corner feature point and the positioning mark points are used to establish a spatial reference plane. The spatial reference plane represents the relative spatial attitude between the chip body and the printed circuit board. The spatial reference plane is discretized into a quadrilateral grid region composed of control nodes according to the isoparametric element division method. The control nodes are arranged in a regular array in the plane. Each control node has an initial coordinate value in the spatial coordinate system. The spatial position relationship between adjacent control nodes is established through shape functions.

5. The deep learning-based online visual inspection system for component pin defects according to claim 4, characterized in that, Virtual boundary conditions based on the relative displacement constraints between the chip body and the printed circuit board are applied to the control node. By performing shape function interpolation and stiffness matrix integration on the mesh region, the displacement field distribution of the control node in the spatial coordinate system is obtained through iterative calculation. Spatial attitude correction parameters are then derived from the displacement field distribution, including: The control nodes corresponding to the first and second corner feature points of the chip body are used as fixed constraint boundaries, and their displacement is set to zero. The control nodes corresponding to the positioning mark points on the printed circuit board substrate are used as loading boundaries, and virtual displacement loads are applied along each axis of the spatial coordinate system to form virtual boundary conditions based on the relative displacement constraints between the chip body and the printed circuit board. Based on the virtual boundary conditions, construct the element stiffness matrix of each mesh element, and assemble all element stiffness matrices according to the global number of the control nodes to form the overall stiffness matrix; The nonlinear equations consisting of the overall stiffness matrix and the load vector are solved by the Newton-Raphson iterative method. The displacement increment of the control node is updated after each iteration until the norm of the displacement increment between two adjacent iterations is less than a preset threshold, thus obtaining the displacement field distribution of each control node in the spatial coordinate system. The out-of-plane displacement components of each control node along the imaging optical axis are extracted from the displacement field distribution, and the spatial attitude correction parameters are obtained by fitting the correspondence between the out-of-plane displacement components and the spatial coordinates.

6. The deep learning-based online visual inspection system for component pin defects according to claim 5, characterized in that, The pin connected component image is projected and transformed according to the spatial attitude correction parameters to obtain the corrected pin connected component image, including: Obtain spatial attitude correction parameters, which include tilt correction and rotation correction. Construct the tilt correction and rotation correction into a perspective projection transformation matrix from three-dimensional space to a two-dimensional imaging plane. Each pixel in the pin connected component image is used as the transformation source point. The mapping coordinates of each pixel in the corrected image space are calculated according to the perspective projection transformation matrix. By resampling the gray values ​​at the mapping coordinates, the corrected pin connected component image with spatial pose distortion eliminated is obtained.

7. The deep learning-based online visual inspection system for component pin defects according to claim 6, characterized in that, Individual pin profiles are extracted from the corrected pin connected component image. The theoretical edge line for each pin is then fitted by extracting the edge profile point set for each pin, including: A connected component labeling algorithm is performed on the corrected pin connected component image. Each pin region is independently labeled according to the eight-neighbor connectivity criterion to obtain the independent connected component corresponding to each pin. For each independent connected component, extract the sequence of closed contour points pixel by pixel along the boundary of the connected component to obtain the contour point set of a single pin; For each pin, fit the outline point set to the straight lines on both sides. Based on the pin extension direction, group the points in the outline point set that are located on both sides of the pin. By calculating the slope and intercept parameters of the straight lines on both sides, obtain the theoretical left edge line and the theoretical right edge line of each pin.

8. The deep learning-based online visual inspection system for component pin defects according to claim 7, characterized in that, The deviation between the edge contour point set of each pin and the corresponding theoretical edge line is calculated to obtain the detection results, including: Obtain the left and right theoretical edge lines of each pin, and obtain the subsets of contour points located at the left and right edges in the contour point set of each pin; calculate the Euclidean distance from each contour point in the left contour point subset to the corresponding left theoretical edge line to obtain the left edge deviation sequence, and calculate the Euclidean distance from each contour point in the right contour point subset to the corresponding right theoretical edge line to obtain the right edge deviation sequence. The left edge deviation sequence and the right edge deviation sequence of each pin are fused together to calculate the cumulative deviation of each pin in the width direction. The cumulative deviation of all pins is then used to construct a deviation distribution curve according to the pin arrangement order. The deviation distribution curve is smoothed by a sliding window filtering method, and continuous peak segments exceeding the preset deviation threshold are extracted from the deviation distribution curve. Based on the position index of the continuous peak segment in the pin arrangement sequence, the pin number with the offset defect is determined. Combined with the peak deviation and deviation change rate of the continuous peak segment, the severity of the offset defect is classified, and the detection result including the defective pin number, offset direction, offset amount and severity classification is obtained.

9. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the system as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, performs the system as described in any one of claims 1 to 8.

Images