Solder ball non-qualified product identification and rejection system based on online monitoring

Abstract

The invention relates to the technical field of surface optical defect detection, in particular to a tin ball unqualified product identification and rejection system based on online monitoring, which constructs a structure expression map based on linkage analysis of a contour change area, an edge response intensity matrix and a direction gradient mapping matrix and in combination with position difference and brightness fall between connection points. High-sensitivity capture of local details on the surface of the solder ball is realized, and the recognition precision and boundary judgment capability of irregular pit and spot areas are remarkably improved by calculating coordinate offset and boundary length change between connection points and combining a graph neural network. Through a multi-factor association adjustment mode of determining confidence, label disturbance frequency and historical deviation records, adaptive convergence of identification classification parameters and progressive optimization of an output matrix structure are realized, the identification precision of a flaw area is improved, and the real-time performance and stability of subsequent action response are optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of surface optical defect detection, and in particular to a system for identifying and rejecting unqualified solder balls based on online monitoring. Background Art

[0002] Surface optical defect detection technology aims to use optical imaging to scan, capture images, and perform feature analysis on the surface of target objects, and detect physical defects such as scratches, pits, spots, and protrusions on the surface based on optical information. Optical defect detection has the technical advantages of being non-contact, real-time, and highly precise, and is suitable for high-throughput assembly line operations and surface quality assessment of tiny targets.

[0003] The purpose of the unqualified solder ball identification and rejection system based on online monitoring is to identify defective solder balls with pits and spots on the surface that affect reliability, and remove them from the transmission path after identification. The system improves the automation level and detection accuracy of solder ball quality inspection, reduces traditional inspection errors and missed judgments, improves the assembly yield and welding reliability of electronic devices, and thus ensures the downstream packaging quality and electrical performance.

[0004] The existing technology uses general optical image acquisition and analysis methods to identify surface defects, relying on static thresholds of image contours and grayscale intensity. It is difficult to effectively characterize the local curvature mutations and morphological deviations caused by micro-deformations on the surface of the solder ball, and it is easy to miss the detection of slight pits and low-contrast spots. Most existing technologies only rely on single image results to trigger rejection instructions, without considering the delay, offset and execution stability differences between the identification label and the actual action. It is difficult to establish a refined response control and feedback tuning mechanism. The existing technology fails to dynamically adjust parameters according to the real-time status of the production line, and it is difficult to adapt to the recognition fluctuations caused by multiple factors such as light disturbances and component changes in the detection environment. It is very easy to misjudge and miss small spot defects, which in turn affects the rejection accuracy and downstream device welding reliability, resulting in uncontrollable accumulation of quality defects. Summary of the Invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a system for identifying and rejecting unqualified solder balls based on online monitoring.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solution: A system for identifying and rejecting unqualified solder balls based on online monitoring includes:

[0007] Structure Generation Module: Based on the contour change area, edge response intensity matrix, and directional gradient mapping matrix in the solder ball array image, it calculates the grayscale change degree and delineates the curvature jump area, extracts the relative position and brightness difference between connection points, and establishes a structure expression map.

[0008] Topology comparison module: Based on the structural expression graph, the coordinate offset and boundary length change between connection point pairs are calculated, and a graph neural network is used to perform node embedding and edge weight propagation. The point set with continuous direction difference and low edge length deviation is selected to construct a path, and a topology offset identification set is obtained.

[0009] Defect removal module: extracts connected areas based on the topological offset recognition set, uses a convolutional neural network to extract spatial structure and grayscale joint feature maps, calculates node coordinate density, regional area and grayscale amplitude mean to construct a spatial distribution map, and generates a removal classification label map group;

[0010] Action tracking module: Based on the above-mentioned eliminated classification label graph group, after binding the frame number, the response delay is measured and the position offset is eliminated, the state-contradictory frames are filtered, the spatial error and time lag between the marked points and the action points are extracted, the deviation records are summarized and the direction consistency is calculated to obtain the execution difference distribution list;

[0011] Weight update module: Based on the execution difference distribution list, measure the model output label confidence and action feedback deviation value, extract the corresponding label point set in the classification sequence, update the probability amplitude and weight correction factor and adjust the output parameter weight to generate a judgment output adjustment matrix.

[0012] As a further solution of the present invention, the structure generation module includes:

[0013] Contour extraction submodule: Based on the contour change area, edge response intensity matrix and directional gradient mapping matrix in the solder ball array image, the grayscale value difference of each row of pixels in the image grid is calculated, and the boundary positions where the jump amplitude exceeds the set grayscale threshold are extracted. The local directional gradient of each boundary position is determined and the angle fluctuation range is calculated. The density of directional mutation points within the fluctuation range is counted and the mutation point distribution area is delineated. The boundary point set that meets the joint conditions of grayscale mutation amplitude and directional change frequency is obtained, and the boundary directional jump feature group is generated.

[0014] Connection construction submodule: Based on the boundary direction jump feature group, the horizontal and vertical coordinate differences and brightness values ​​between each pair of points are measured and the point groups whose boundary angles are lower than the valley structure change amplitude are screened out. It is determined whether a continuous directional gradient trend is formed between the point groups and a line segment connection operation is performed. The starting and ending points of the connected paths are marked and a closed boundary structure is constructed. The number of line segments and the number of intersections of the closed boundary are extracted and the structural integrity is confirmed to establish a structural expression map.

[0015] As a further solution of the present invention, the topology comparison module includes:

[0016] Position offset submodule: Based on the structural expression graph, the relative difference of the connection point pairs in the row and column pixel coordinates is measured, and the point pairs within the average offset range are screened. The node embedding information propagation method of the graph neural network is used to model the spatial topological relationship between nodes and update the edge connection status. The connection stability score is calculated through the edge weight interaction between nodes. The point groups with connection scores below the stability threshold are eliminated, and the connection status and start and end point coordinate numbers of the retained point groups are recorded to establish a set of boundary stable point pairs.

[0017] Contour construction submodule: Based on the set of stable boundary point pairs, the difference in the angle between the direction vectors of each pair of points is calculated, and a continuous point sequence whose angle falls into the stable interval is extracted. The direction of the connecting line segments in the adjacent point sequences is determined to be continuously maintained and form a closed structure segment. The path segments are connected in sequence according to the starting and ending coordinate numbers to establish a complete continuous closed line combination and obtain a set of direction-continuous paths;

[0018] Node comparison submodule: Based on the set of continuous paths in the direction, the convergence difference between the starting point and the end point of each path in the pixel grid is measured, and it is determined whether the node is repeated when the path is closed. The node coordinate matching difference is calculated, and the structural units with error values ​​greater than the average node spacing are extracted. The spatial coverage position of the structural units in the graph is compared with the overlap of the dense edge segments, and the structural fragments with an offset degree exceeding the threshold are screened out to obtain the topological offset identification set.

[0019] As a further solution of the present invention, the graph neural network is according to the formula: in: For the

[0020] The updated embedding vector of i nodes, is the current embedding vector of the jth neighboring node, w ij is the spatial edge weight between nodes i and j, α ij is the dynamic weight adjustment factor, β i is the structural response coefficient of the current node, Δ ij is the pixel offset difference between node i and neighboring node j, γ i is the grayscale coupling weight of the i-th node, G i is the grayscale mean response of the area where node i is located.

[0021] As a further solution of the present invention, the extraction of a continuous point sequence whose angle falls into a stable interval is specifically to construct a connecting line segment in the coordinate order in the obtained boundary stable point pair set, calculate the direction vector angle between two adjacent line segments, the direction vector is taken as a directed vector composed of the start and end coordinates of the adjacent line segments, and the angle calculation adopts the arc cosine function to act on the unit vector dot product of the adjacent direction vectors. The unit of the obtained angle value is angle, and an angle stability interval θstable is set, indicating that the directions of the two adjacent line segments remain consistent. The point sequence that satisfies the adjacent angle values ​​falling into the stable interval is extracted as a group of continuous point sequences. If a certain angle value exceeds the stable interval threshold, it is considered that the direction has changed suddenly, the point sequence is interrupted, and the judgment is restarted. The output is a set of multiple point sequences that meet the direction continuity.

[0022] As a further solution of the present invention, the defect removal module includes:

[0023] Density assessment submodule: extracts connected areas based on the topological offset identification set, accumulates the number of horizontal and vertical coordinates of nodes in each area, uses a convolutional neural network to extract a joint feature map of node distribution and grayscale change in the area, calculates the total number of nodes in the unit area grid, extracts the area boundary coordinate point set to calculate the circumscribed area valley value, groups and counts the difference between the internal pixel grayscale valley peak value, constructs a joint array of node number, area and grayscale change, and generates a node distribution feature map;

[0024] Intensity screening submodule: Based on the node distribution feature map, extract pixel units whose grayscale values ​​are all greater than the average grayscale response and perform a continuous adjacent position aggregation operation, measure the number of pixels in the aggregated area, calculate the corresponding proportion of the total area of ​​the map, compare the rate of change of the region with the average grayscale response of the four neighbors, retain the areas with fluctuations greater than the response mean, perform a contour closure check and confirm the area, and obtain a set of high-response closed areas;

[0025] Label generation submodule: Based on the high-response closed area set, fit the outer contour boundary of each area to a fitting rectangle, extract the boundary range of the four-point coordinates of the rectangle, judge the intersection of the positions between the areas, perform the horizontal and vertical overlap rates and screen out areas with high overlap ratios, extract the center point coordinates and category numbers of the remaining areas, construct a set of independently numbered block areas, and generate a group of images with excluded classification labels.

[0026] As a further solution of the present invention, the convolutional neural network is based on the formula: Where: U is the joint characteristic response map value,

[0027] μ p is the central response value output by the pth convolution channel, is the average value of the center response of all channels, ρ p is the pixel density coefficient of the pth region, χp is the texture granularity measure of the pth region, δ p is the weighted value of the intensity variation of the pth region, λ is the response difference index, q is the total number of channels, and p is the number of the pth channel.

[0028] As a further solution of the present invention, the motion tracking module includes:

[0029] Number matching submodule: Based on the elimination classification label image group, the image frame number in each label image is extracted, and then the corresponding time number in the elimination device action log is matched. After the matching frame number is extracted, it is sorted in chronological order. The trigger time of each group of frame records and the execution time interval in the action record are calculated and it is determined whether the interval value is greater than the preset difference threshold. The number, offset second value and label status identifier of the delay exceeding limit frame are recorded to generate a label action timing mapping set;

[0030] Error extraction submodule: Based on the label action timing mapping set, the center coordinates of the marked area in each frame image and the horizontal and vertical differences of the execution position coordinates in the action record are calculated, the error vector is extracted and the direction angle measurement and quantity are counted, the error samples whose direction angles fall into the continuous concentrated interval are classified and marked, and the direction consistency ratio is counted. Combined with the time delay information in the corresponding frame, a joint record table is formed, and sample frames with large errors and a high proportion of concentrated directions are screened out to obtain a list of execution difference distributions.

[0031] As a further solution of the present invention, the weight updating module includes:

[0032] Confidence determination submodule: Based on the execution difference distribution list, extract the confidence value of each classification label in the predicted output of the corresponding image frame in each group of records, compare the classification label and the action feedback label number to see if they are consistent and calculate the numerical offset, extract the frame sample number whose absolute value of the difference is greater than the confidence threshold range, and mark the direction classification code to generate the output deviation confidence set;

[0033] Label extraction submodule: Based on the output deviation confidence set, extract the classification sequence index corresponding to the predicted label in each abnormal frame number, scan the entire classification probability sequence to locate the corresponding index position, count the number of times it is ranked at the end and archive the label numbers, identify the category numbers with a frequency higher than the lower limit of the proportion, aggregate them into a clustered label set, and obtain a high-frequency label interference set;

[0034] Parameter adjustment submodule: Based on the high-frequency label interference set, the historical prediction offset rate and interference frequency ratio corresponding to each interference label number are calculated, the parameters of the column positions of each type of label in the classification matrix are replaced, the probability amplitude ratio of the label is redistributed and the label strength numerical distribution structure is adjusted, the updated parameter matrix is ​​established, and the judgment output adjustment matrix is ​​generated.

[0035] As a further solution of the present invention, when acquiring the solder ball array image, an inclined plate structure with an inclined angle is provided so that the solder balls roll along the surface of the inclined plate under the action of gravity to form a single-line channel. A high-frame rate area scanning camera is fixedly installed above the inclined plate. The camera continuously collects multiple frames of images of the solder balls during the rolling process at a speed of not less than 300 frames per second. The images are spatially aligned by combining the frame timestamp and the solder ball displacement information. The images of the same solder ball at different instantaneous angles during the rolling process are spliced ​​into a set of complete perspective frame sequences to construct a two-dimensional solder ball array image. The solder ball array image can realize the acquisition of multi-angle states of the solder ball surface.

[0036] Compared with the prior art, the advantages and positive effects of the present invention are:

[0037] By analyzing the contour change area, edge response intensity matrix, and directional gradient mapping matrix, we can quantify the grayscale change and curvature jump characteristics. We also combine the position differences and brightness differences between connection points to construct a structural expression map, which can achieve highly sensitive capture of local details on the solder ball surface.

[0038] By calculating the coordinate offset and boundary length changes between connection points, combined with graph neural network embedding and edge weight propagation mechanisms, we further explore high-confidence paths with continuous direction differences and low edge length deviations between nodes. This allows us to accurately identify locally distorted areas of the solder ball morphology, significantly improving the recognition accuracy and boundary judgment capabilities of irregular pits and spots.

[0039] By adjusting the multi-factor correlation of judgment confidence, label disturbance frequency and historical deviation records, the adaptive convergence of recognition and classification parameters and the progressive optimization of the output matrix structure can be achieved, which not only improves the recognition accuracy of defective areas, but also optimizes the real-time and stability of subsequent action responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The present invention will be further described below with reference to the accompanying drawings.

[0041] Figure 1 is a system flow chart of the present invention;

[0042] Figure 2 Schematic diagram of the system framework of the present invention. DETAILED DESCRIPTION

[0043] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] The embodiment of the present invention provides a system for identifying and rejecting unqualified solder balls based on online monitoring. Figure 1 , Figure 1 This is a flow chart of a system for identifying and rejecting defective solder balls based on online monitoring provided by an embodiment of the present invention. The system includes:

[0045] Structure Generation Module: Based on the contour change area, edge response intensity matrix, and directional gradient mapping matrix in the solder ball array image, it calculates the grayscale change degree and delineates the curvature jump area, extracts the relative position and brightness difference between connection points, and establishes a structure expression map.

[0046] Topology comparison module: Based on the structural expression graph, it calculates the coordinate offset and boundary length change between connection point pairs, uses graph neural networks for node embedding and edge weight propagation, selects point sets with continuous direction differences and low edge length deviations to construct paths, and obtains the topology offset identification set;

[0047] Defect Removal Module: This module extracts connected areas based on a topological offset recognition set, uses a convolutional neural network to extract a joint feature map of spatial structure and grayscale, calculates the density of node coordinates, the area of ​​the region, and the mean of the grayscale amplitude to construct a spatial distribution map, and generates a group of rejection classification labels.

[0048] Motion tracking module: Based on the elimination of classification label graphs, after binding frame numbers, it measures response delay and eliminates position offsets, filters out frames with inconsistent states, extracts spatial errors and time lags between markers and action points, summarizes deviation records and calculates directional consistency, and obtains a list of execution difference distributions;

[0049] Weight update module: Based on the execution difference distribution list, measure the model output label confidence and action feedback deviation value, extract the corresponding label point set in the classification sequence, update the probability amplitude and weight correction factor, adjust the output parameter weight, and generate the judgment output adjustment matrix.

[0050] See also Figure 2 , the structure generation module includes:

[0051] Contour extraction submodule: Based on the contour change area, edge response intensity matrix and directional gradient mapping matrix in the solder ball array image, the grayscale value difference of each row of pixels in the image grid is calculated, and the boundary positions where the jump amplitude exceeds the set grayscale threshold are extracted. The local directional gradient of each boundary position is determined and the angle fluctuation range is calculated. The density of directional mutation points within the fluctuation range is counted and the mutation point distribution area is delineated. The boundary point set that meets the joint conditions of grayscale mutation amplitude and directional change frequency is obtained, and the boundary directional jump feature group is generated.

[0052] Connection construction submodule: Based on the boundary direction jump feature group, the horizontal and vertical coordinate differences and brightness values ​​between each pair of points are measured and point groups with boundary angles lower than the valley structure change amplitude are screened out. It is determined whether a continuous directional gradient trend is formed between point groups and a line segment connection operation is performed. The starting and ending points of the connected paths are marked and a closed boundary structure is constructed. The number of line segments and the number of intersections of the closed boundary are extracted and the structural integrity is confirmed to establish a structural expression map.

[0053] Contour extraction submodule: Based on the contour change area, edge response intensity matrix and directional gradient mapping matrix in the solder ball array image, the image grayscale difference processing method is adopted. The input image is scanned line by line to obtain pixel row vectors. A one-dimensional difference operation is performed to extract horizontal grayscale changes. In the difference result, the grayscale jump threshold is set according to the high percentile value of the grayscale gradient statistical distribution. The pixel positions with grayscale mutation amplitude greater than the threshold are screened, and the corresponding boundary points are extracted and the spatial coordinates are recorded. The horizontal and vertical gradient responses are calculated based on the adjacent areas of the boundary points. The direction vector sequence is constructed respectively. The angle between each direction vector in the local area is measured. The fluctuation range of the angle change is counted and the local density of the mutation point is calculated. The mutation judgment threshold is set according to the density. The area with frequent angle mutation is marked as unstable direction area. The boundary points that meet both the grayscale mutation amplitude exceeding the grayscale threshold and the directional mutation density exceeding the statistical standard are formed into a set. This is used as the point set to express the joint grayscale and directional jump features in contour analysis, and the boundary directional jump feature group is generated.

[0054] Connection construction submodule: Based on the boundary direction jump feature group, the spatial geometric relationship analysis method is adopted to measure the horizontal and vertical pixel coordinate differences between the points respectively, and calculate the brightness value difference. The boundary direction angles between the point pairs are compared, and the point groups with angles less than the structural change threshold are screened out. The remaining point groups are established in position order. A fixed-length sliding window is used to continuously detect the directional gradient changes in the sequence to determine whether a trend path with consistent direction is formed. The connected path establishment operation is performed on the point sequence that meets the stable direction change conditions. The starting and end points of the path are marked in order according to the spatial coordinate sequence, and the path set is used as the boundary frame. The constructed path set is detected in turn to see if it forms a closed structure. The total number of line segments in each path is counted and the number of path intersections is marked. If the number of line segments is not less than the set threshold and the number of intersections exceeds the intersection judgment benchmark, the path combination is marked as a structural closed area, and a closed area structure set is constructed and numbered and archived to generate a structural expression map.

[0055] See also Figure 2 , the topology comparison module includes:

[0056] Position offset submodule: Based on the structural expression graph, the relative difference of connection point pairs in row and column pixel coordinates is measured, and point pairs within the average offset range are selected. The node embedding information propagation method of the graph neural network is used to model the spatial topological relationship between nodes and update the edge connection status. The connection stability score is calculated through the interaction of edge weights between nodes. Point groups with connection scores below the stability threshold are eliminated. The connection status and start and end point coordinate numbers of the retained point groups are recorded to establish a set of boundary stable point pairs.

[0057] Contour construction submodule: Based on a set of stable point pairs on the boundary, the difference in the angle between the direction vectors of each pair of points is calculated, and a series of continuous points whose angles fall into the stable interval is extracted. The direction of the connecting line segments in the adjacent point series is determined to be continuously maintained and form a closed structure segment. Each path segment is connected in sequence according to the starting and ending coordinate numbers to establish a complete continuous closed line combination and obtain a set of directionally continuous paths.

[0058] Node comparison submodule: Based on a set of directional continuous paths, the convergence difference between the starting point and the end point of each path in the pixel grid is measured to determine whether the nodes are repeated when the path is closed. The node coordinate matching difference is calculated, and the structural units with an error value greater than the average node spacing are extracted. The spatial coverage position of the structural units in the graph is compared with the overlap of dense edge segments, and structural segments with an offset degree exceeding the threshold are screened out to obtain a topological offset identification set.

[0059] Position offset submodule: Based on the structural expression map, the pixel coordinate difference calculation method is used to measure the coordinate difference of the connection points in the map in turn, record the horizontal difference and vertical difference values ​​respectively, normalize the difference and construct a two-dimensional coordinate difference matrix, and calculate the mean and standard deviation of all elements in the matrix. The positive and negative standard deviation intervals of the mean are used as the offset range screening criteria, and the point pairs within the range are used as candidate point groups. The graph neural network embedding propagation method is used to set the network structure as a feedforward structure with three layers of graph convolution units. The embedding dimensions of each layer of nodes are set to 64, 128, and 64 respectively, and the mean aggregation method is used to transmit the adjacent node information. , the node embedding vector is initialized as the concatenation vector composed of the pixel position and grayscale value of each node in the structure graph, and the initial edge weight matrix is ​​constructed for each edge according to the spatial relative position difference between the connected nodes. The edge weight is weighted updated in each round of propagation, and the node information is updated using the weighted summation of the embedding vector. The node vector output in each round is scored using the Euclidean distance similarity, and the stability score of the connection between nodes is calculated based on the score. The edge connection rows with scores lower than the set stability threshold are eliminated. The stability threshold is determined by the lower quartile value of the edge weight distribution. The connection status of the remaining point groups is marked and the coordinate numbers of the starting and ending points of the path are recorded to generate a set of boundary stable point pairs;

[0060] Contour construction submodule: Based on the boundary stable point pair set, the direction vector angle difference method is used to construct the line segment structure of each pair of points in the set in the coordinate order, calculate the direction vector of the line segment, combine the direction vectors of two adjacent line segments in a spatial sequence and measure the angle difference. The preset stable interval in the angle difference statistics is the range of angle difference within 15 degrees. The point series whose continuous line segment direction angle differences all fall into the stable interval are extracted as continuous point series. Directional stability analysis is performed on all continuous point series. A sliding window with a length of 5 is used to count the rate of change of the direction vector angle. Point series segments with local angle mutations exceeding 30 degrees are screened out. Structural segment closure detection is performed on point series that meet the continuous direction trend conditions to determine whether the point series meets the closed-loop feature of the relative position of the first and last points. The closable point series is constructed as a structural segment. The path segments are connected in sequence according to the start and end coordinate numbering order. The path index mapping table is constructed according to the connection order and sorted into a line segment set. A complete continuous closed line combination is established to obtain a directional continuous path set.

[0061] Node comparison submodule: Based on the set of directional continuous paths, the path start and end position comparison method is used to extract the position coordinates of the starting point and end point of each path in the path set in the pixel grid, calculate the convergence difference and record it in the path difference record table, and determine whether there is any node position overlap in the closed path. The judgment logic is that the spatial distance between the node and other nodes in the path is less than 3 pixel units for duplication. The coordinates of each path are matched, the spatial error value between the node in the path and the corresponding coordinate in the atlas is calculated, the error mean and the deviation difference of each node are calculated, and the structural units with the deviation difference greater than the average node spacing in the path are extracted. The spatial position coverage analysis method is used to calculate the overlap of the structural units with the high-density edge segment area in the structural expression atlas, the pixel coverage rate of the overlapping area is counted, and the units with a coverage rate higher than 50% are marked as offset structural fragments. The offset fragment numbers and position indexes are sorted to generate a topological offset identification set.

[0062] The graph neural network follows the formula:

[0063] in: is the updated embedding vector of the i-th node, is the current embedding vector of the jth neighboring node, w ij is the spatial edge weight between nodes i and j, α ij is the dynamic weight adjustment factor, β i is the structural response coefficient of the current node, Δ ij is the relationship between node i and its neighboring nodes

[0064] The pixel offset difference between i is the grayscale coupling weight of the i-th node, G i is the grayscale mean response of the area where node i is located;

[0065] Execution process: First, the system obtains the row and column pixel coordinates of all connection point pairs based on the structural expression map, calculates the coordinate offset difference between each node, and obtains the initial edge weight coefficient w based on the grayscale change of adjacent points. ij , extract the node direction vector angle and calculate the angle difference, and map it to the direction consistency adjustment factor α through normalization ij , calculate the mean grayscale value G of each node in the local area i And grayscale variance, get the grayscale coupling weight γ i , combined with the grayscale fluctuation range to reflect the stability of local content, the structural response coefficient β is calculated by combining the boundary length change rate of the area corresponding to the node position with the local grayscale variance i , which represents the dynamic sensitivity of the structural unit where the node is located. The system substitutes the above parameters into the graph neural network embedding propagation update formula to realize the dynamic propagation and weight adjustment of the node feature state in the spatial structure, and then eliminates the point pair sets with insufficient connection stability through the similarity and connection strength score between the embedded vectors after propagation.

[0066] The method of extracting a continuous point sequence whose angle falls into a stable interval is as follows: in the obtained set of boundary stable point pairs, connecting line segments are constructed in coordinate order, and the angle between the direction vectors of two adjacent line segments is calculated. The direction vector is a directed vector composed of the start and end coordinates of the adjacent line segments. The angle calculation adopts the arc cosine function to act on the unit vector dot product of the adjacent direction vectors. The obtained angle value is in radians and degrees. An angle stability interval θstable is set, indicating that the directions of the two adjacent line segments remain consistent. The point sequence that satisfies the adjacent angle values ​​falling into the stable interval is extracted as a set of continuous point sequences. If a certain angle value exceeds the stable interval threshold, it is considered that the direction has suddenly changed, the point sequence is interrupted, and the judgment is restarted. The output is a set of multiple point sequences that meet the direction continuity.

[0067] See also Figure 2 , the defect removal module includes:

[0068] Density assessment submodule: Extracts connected areas based on the topological offset recognition set, accumulates the number of horizontal and vertical coordinates of nodes in each area, uses a convolutional neural network to extract a joint feature map of node distribution and grayscale change within the area, calculates the total number of nodes within the unit area grid, extracts the area boundary coordinate point set to calculate the circumscribed area valley value, groups and counts the difference between the internal pixel grayscale valley peak value, constructs a joint array of node number, area and grayscale change, and generates a node distribution feature map;

[0069] Intensity screening submodule: Based on the node distribution feature map, it extracts pixel units with grayscale values ​​greater than the average grayscale response and performs continuous adjacent position aggregation operations. The number of pixels in the aggregated area is measured, and the corresponding proportion of the total area of ​​the map is calculated. The rate of change of the region and the average grayscale response of the four neighbors are compared. The areas with fluctuations greater than the response mean are retained. After performing a contour closure check, the area is confirmed and a set of high-response closed areas is obtained.

[0070] Label generation submodule: Based on the set of highly responsive closed regions, the outer contours of each region are fitted into a rectangle, the coordinate boundary range of the four points of the rectangle is extracted, the intersection between regions is determined, the horizontal and vertical overlap ratios are calculated, and regions with high overlap ratios are screened out. The center point coordinates and category numbers of the remaining regions are extracted, and a set of independently numbered tile regions is constructed to generate a group of images with excluded classification labels.

[0071] Density assessment submodule: Extract connected areas based on the topological offset recognition set, use coordinate statistics to deduplicate the horizontal and vertical coordinates of the nodes in each area, obtain the node number distribution, and use a convolutional neural network to construct a three-layer convolution and two-layer pooling structure. The convolution kernel size is set to 3×3, the step size is 1, the activation function is ReLU, and the input is a dual-channel tensor of the node grayscale image and the spatial position image. The spatial grayscale joint feature map is extracted block by block for the regional map. The region is divided into a 20×20 grid and the number of unit grid nodes is counted. The boundary points of the region are extracted and the circumscribed area is calculated using the minimum rectangle method. After counting the area values ​​of each region, the valley value is extracted according to the quintile. The internal grayscale is calculated using the maximum and minimum difference values. The node number, area and grayscale difference arrays are constructed to generate the node distribution feature map.

[0072] Intensity screening submodule: Based on the node distribution feature map, it extracts pixel units with grayscale values ​​greater than the image average grayscale, performs eight-neighborhood aggregation operations to form connected blocks, calculates the area ratio based on the number of pixels in the block, and calculates the ratio with the total area of ​​the image. It measures the average grayscale difference between each area and the four neighboring areas, retains areas where the difference exceeds the image grayscale standard deviation, performs contour tracing to determine closure, records areas that meet the closure criteria, and generates a set of high-response closed areas.

[0073] Label generation submodule: Based on the set of high-response closed areas, a boundary fitting algorithm is used to perform outer boundary fitting rectangle processing on each area, and the minimum circumscribed rectangle method is used to calculate the enclosing rectangle of the area. The coordinates of the rectangle vertices are obtained by projecting the boundary points and the coordinate positions of the four points are recorded respectively. After extracting the boundary range of each rectangle, overlap analysis is performed between regions. The overlap ratio is determined by the overlap ratio of the horizontal projection and the vertical projection. The overlap ratio is calculated by dividing the overlap length by the minimum rectangle length in the corresponding direction. Areas with both horizontal and vertical overlap ratios greater than the preset threshold of 0.5 are eliminated. The center point coordinates of the retained areas after elimination are extracted and the category labels are automatically assigned based on the processing sequence number. The number generation uses the area sorting position plus a fixed prefix to construct a number identifier. Each set of numbered areas is packaged into an independent numbered tile data structure to generate a eliminated classification label map group.

[0074] Convolutional neural network follows the formula:

[0075] Where: U is the joint feature response map value, μ p is the central response value output by the pth convolution channel, is the average value of the center response of all channels, ρ p is the pixel density coefficient of the pth region, χ p is the texture granularity measure of the pth region, δ p is the weighted value of the intensity variation of the pth region, λ is the response difference index, q is the total number of channels, and p is the number of the pth channel;

[0076] Execution process: First, all connected areas in the topological offset recognition set are input into the convolutional network to obtain the response feature maps under multiple channels, and the central response value μ is extracted in each channel map. p , calculate the central response mean of each channel The system calculates the node density per unit area for each region and constructs the node density coefficient ρ p , based on the particle size level of the texture changes in the region, the particle size parameter χ is obtained. p The system measures the intensity variation range in the grayscale distribution and normalizes it to generate the regional grayscale variation weight δ p In order to regulate the nonlinear differences between the responses of different channels, the response difference index λ is introduced, and the difference between the central response value and the mean is exponentially transformed. The system uses the above formula to jointly weight each channel to generate a joint feature response map U of node distribution and grayscale change.

[0077] See also Figure 2 , the motion tracking module includes:

[0078] Number matching submodule: Based on the elimination classification label map group, the image frame number is extracted from each label map, and then matched with the time number in the elimination device action log. The frame numbers that are successfully matched are extracted and sorted in chronological order. The trigger time of each group of frame records and the execution time interval in the action record are calculated and the interval value is determined to be greater than the preset difference threshold. The number, offset value in seconds and label status identifier of the frame with excessive delay are recorded to generate a label action timing mapping set.

[0079] Error extraction submodule: Based on the label-action temporal mapping set, it calculates the center coordinates of the marked area in each frame image and the horizontal and vertical differences of the execution position coordinates in the action record. It extracts the error vector and counts the direction angle measurement and quantity. It classifies and marks the error samples whose direction angles fall into the continuous concentrated interval and calculates the direction consistency ratio. It combines the time delay information in the corresponding frame to form a joint record table, screens out sample frames with large errors and a high proportion of concentrated directions, and obtains a list of execution difference distributions.

[0080] Number matching submodule: Based on the elimination classification label graph group, the frame number parsing method is used to extract the frame number of the label graph, and the image data management interface is called to read the timestamp field and frame index value in the image metadata field in sequence, and the extracted image frame number is matched with the time number recorded in the elimination device action log. The matching method adopts a two-way time window calibration strategy, and the image frame timestamp and the action log recording time are calculated. The time alignment tolerance threshold is set to 0.05 seconds, and the matching frame number pairs within the time range are screened. The set of frame numbers with successful matching is recorded, and the frame sequence time table is constructed by sorting in ascending order of timestamps. The time interval between the image frame trigger time and the corresponding action time in the table is calculated. The preset interval difference threshold is 0.1 seconds. The frame number with an interval value greater than the threshold is marked as an abnormal frame. The number, interval second value and the associated elimination label identifier are recorded, and the results are summarized to generate a label action time sequence mapping set;

[0081] The error extraction submodule uses a spatial coordinate difference vector calculation method based on the label-action temporal mapping set to extract the center coordinates of the marked area in each frame image. The image analysis interface is called to parse the marked area outline from the label image and calculate the geometric center. The action coordinate values ​​of the corresponding frame extraction and rejection device are recorded in the action log. The difference between the horizontal and vertical coordinates is calculated to form an error vector set. The angle between the execution directions of the error vector set is counted. The angle values ​​of adjacent error vector directions are calculated using the angle cosine method and a direction sequence is constructed. Sample vectors whose direction angles fall within a set continuous concentration interval are clustered. The angle concentration threshold is set to 15 degrees. Sample segments with a continuous length of at least 5 frames are marked as concentrated areas. The time delay records corresponding to the sample frame number index are matched. A joint record table is constructed containing four fields: frame number, position deviation, direction identifier, and delay time. Frame entries in the joint table with error magnitudes greater than the pixel distance threshold and belonging to the concentrated direction mark are filtered. The filtering threshold is set to the mean error plus 1 standard deviation. Frames that meet the conditions are extracted to construct an index set and generate an execution difference distribution list.

[0082] See also Figure 2 , the weight update module includes:

[0083] Confidence determination submodule: Based on the execution difference distribution list, it extracts the confidence value of each classification label in the predicted output of the corresponding image frame in each group of records, compares the classification label with the action feedback label number to see if they are consistent and calculates the numerical offset. It extracts the frame sample number whose absolute value of the difference is greater than the confidence threshold range and annotates the direction classification code to generate the output deviation confidence set;

[0084] Label extraction submodule: Based on the output deviation confidence set, extract the classification sequence index corresponding to the predicted label in each abnormal frame number, scan the entire classification probability sequence to locate the corresponding index position, count the number of times it is ranked at the end and archive the label numbers, identify the category numbers with a frequency higher than the lower limit of the proportion, aggregate them into a clustered label set, and obtain the high-frequency label interference set;

[0085] Parameter adjustment submodule: Based on the high-frequency label interference set, the historical prediction offset rate and interference frequency ratio corresponding to each interference label number are calculated, the parameters of the column positions of each type of label in the classification matrix are replaced, the probability amplitude ratio of the label is redistributed, and the distribution structure of the label strength value is adjusted. The updated parameter matrix is ​​established and the judgment output adjustment matrix is ​​generated;

[0086] Confidence determination submodule: Based on the execution difference distribution list, the confidence difference judgment method is used to perform prediction result extraction operations on the image frame samples in each group of records. The model output interface is called to read the classification label sequence and confidence value corresponding to each frame sample. The confidence data structure is a one-dimensional floating-point array with a length equal to the number of categories. Each element represents the predicted probability of the corresponding label. The label corresponding to the maximum probability is extracted as the predicted label number. At the same time, the label number in the action feedback record is read, and the number consistency comparison is performed. The confidence value and the action feedback label probability value under the same label are differenced. The confidence threshold range is set to 0.15. The frame sample number whose absolute difference exceeds the threshold is extracted as an abnormal sample. The direction classification code is added to the extracted sample as an identification field. The frame number, confidence difference, and direction code are arranged into a triple structure to generate the output deviation confidence set.

[0087] Label extraction submodule: Based on the output deviation confidence set, the sequence index positioning method is used to perform target position identification on the label sequence of each abnormal frame number. The complete classification probability sequence output by the model is read according to the frame number, the sequence is sorted from high to low, and the index position of the actual predicted label in the abnormal frame in the sequence is extracted. The number of occurrences of each label in the last three digits after sorting is accumulated to form a label sorting statistics table. The relative frequency of the label number is calculated according to the number of occurrences, and the lower limit of the proportion is set to 5%. The label numbers with a frequency exceeding the lower limit are filtered and filed as label category groups. The label category groups are numbered and merged to generate a cluster set. The high-frequency perturbation label numbers are summarized to obtain the high-frequency label interference set.

[0088] Parameter adjustment submodule: Based on the high-frequency label interference set, the matrix weight rearrangement method is used to modify the classification model output parameter matrix. The offset rate and interference frequency ratio corresponding to each interference label number in the historical prediction record are counted. The offset rate is defined as the mean of the difference between the predicted value and the feedback value. The interference frequency is the number of times the label appears in the abnormal sample divided by the total number of abnormal frames. The statistical results are mapped to weight adjustment factors, and the corresponding column positions in the original classification parameter matrix are located according to the label number. The matrix column is numerically adjusted, and the original probability amplitude value is subtracted by the adjustment factor while keeping the total probability normalized. The amplitude ratio of the output value corresponding to each label is redistributed and the output tensor is reconstructed. The intensity processing of the normalized updated label probability is reorganized and the distribution structure is reorganized to construct the corrected output classification parameter matrix and generate the judgment output adjustment matrix.

[0089] When acquiring array images, an inclined plate structure with an inclination angle (30°) is set up to allow the solder balls to roll along the surface of the inclined plate under the action of gravity to form a single-line channel. A high-frame-rate area scan camera is fixedly installed above the inclined plate. The camera continuously captures multiple frames of images of the solder balls during the rolling process at a speed of not less than 300 frames per second. The images are spatially aligned by combining the frame timestamp and the solder ball displacement information. The images of the same solder ball at different instantaneous angles during the rolling process are spliced ​​into a set of complete perspective frame sequences to construct a two-dimensional solder ball array image. The solder ball array image can realize the acquisition of the multi-angle state of the solder ball surface.

[0090] The above is a detailed description of an embodiment of the present invention. However, the content described is only a preferred embodiment of the present invention and should not be considered to limit the scope of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A solder ball non-conforming product identification and rejection system based on online monitoring, characterized in that: The system comprises: Structure Generation Module: Based on the contour change area, edge response intensity matrix, and directional gradient mapping matrix in the solder ball array image, it calculates the grayscale change degree and delineates the curvature jump area, extracts the relative position and brightness difference between connection points, and establishes a structure expression map. Topology comparison module: Based on the structural expression graph, the coordinate offset and boundary length change between connection point pairs are calculated, and a graph neural network is used to perform node embedding and edge weight propagation. The point set with continuous direction difference and low edge length deviation is selected to construct a path, and a topology offset identification set is obtained. Defect removal module: extracts connected areas based on the topological offset recognition set, uses a convolutional neural network to extract spatial structure and grayscale joint feature maps, calculates node coordinate density, regional area and grayscale amplitude mean to construct a spatial distribution map, and generates a removal classification label map group; Action tracking module: Based on the above-mentioned eliminated classification label graph group, after binding the frame number, the response delay is measured and the position offset is eliminated, the state-contradictory frames are filtered, the spatial error and time lag between the marked points and the action points are extracted, the deviation records are summarized and the direction consistency is calculated to obtain the execution difference distribution list; Weight update module: Based on the execution difference distribution list, measure the model output label confidence and action feedback deviation value, extract the corresponding label point set in the classification sequence, update the probability amplitude and weight correction factor and adjust the output parameter weight to generate a judgment output adjustment matrix.

2. The unqualified solder ball identification and rejection system based on online monitoring according to claim 1 is characterized in that: The structure generation module includes: Contour extraction submodule: Based on the contour change area, edge response intensity matrix and directional gradient mapping matrix in the solder ball array image, the grayscale value difference of each row of pixels in the image grid is calculated, and the boundary positions where the jump amplitude exceeds the set grayscale threshold are extracted. The local directional gradient of each boundary position is determined and the angle fluctuation range is calculated. The density of directional mutation points within the fluctuation range is counted and the mutation point distribution area is delineated. The boundary point set that meets the joint conditions of grayscale mutation amplitude and directional change frequency is obtained, and the boundary directional jump feature group is generated. Connection construction submodule: Based on the boundary direction jump feature group, the horizontal and vertical coordinate differences and brightness values ​​between each pair of points are measured and the point groups whose boundary angles are lower than the valley structure change amplitude are screened out. It is determined whether a continuous directional gradient trend is formed between the point groups and a line segment connection operation is performed. The starting and ending points of the connected paths are marked and a closed boundary structure is constructed. The number of line segments and the number of intersections of the closed boundary are extracted and the structural integrity is confirmed to establish a structural expression map.

3. The unqualified solder ball identification and rejection system based on online monitoring according to claim 1 is characterized in that: The topology comparison module includes: Position offset submodule: Based on the structural expression graph, the relative difference of the connection point pairs in the row and column pixel coordinates is measured, and the point pairs within the average offset range are screened. The node embedding information propagation method of the graph neural network is used to model the spatial topological relationship between nodes and update the edge connection status. The connection stability score is calculated through the edge weight interaction between nodes. The point groups with connection scores below the stability threshold are eliminated, and the connection status and start and end point coordinate numbers of the retained point groups are recorded to establish a set of boundary stable point pairs. Contour construction submodule: Based on the set of stable boundary point pairs, the difference in the angle between the direction vectors of each point pair is calculated, and a continuous point sequence whose angle falls into the stable interval is extracted. The direction of the connecting line segments in the adjacent point sequences is determined to be continuously maintained and form a closed structure segment. The path segments are connected in sequence according to the starting and ending coordinate numbers to establish a complete continuous closed line combination and obtain a set of direction-continuous paths. Node comparison submodule: Based on the set of continuous paths in the direction, the convergence difference between the starting point and the end point of each path in the pixel grid is measured, and it is determined whether the node is repeated when the path is closed. The node coordinate matching difference is calculated, and the structural units with error values ​​greater than the average node spacing are extracted. The spatial coverage position of the structural units in the graph is compared with the overlap of the dense edge segments, and the structural fragments with an offset degree exceeding the threshold are screened out to obtain the topological offset identification set.

4. The unqualified solder ball identification and rejection system based on online monitoring according to claim 1 is characterized in that: The graph neural network is based on the formula: in: is the updated embedding vector of the i-th node, is the current embedding vector of the jth neighboring node, w ij is the spatial edge weight between nodes i and j, α ij is the dynamic weight adjustment factor, β i is the structural response coefficient of the current node, Δ ij is the pixel offset difference between node i and neighboring node j, γ i is the grayscale coupling weight of the i-th node, G i is the grayscale mean response of the area where node i is located.

5. The unqualified solder ball identification and rejection system based on online monitoring according to claim 3 is characterized in that: The method of extracting a continuous point sequence whose angle falls into a stable interval is specifically to construct a connecting line segment in the coordinate order in the obtained boundary stable point pair set, calculate the angle between the direction vectors of two adjacent line segments, the direction vector is taken as a directed vector composed of the start and end coordinates of the adjacent line segments, the angle is calculated by applying the arc cosine function to the unit vector dot product of the adjacent direction vectors, and the obtained angle value is in radians and degrees, and an angle stability interval θstable is set, indicating that the directions of the two adjacent line segments remain consistent, and extract the point sequence that satisfies the adjacent angle values ​​falling into the stable interval as a group of continuous point sequences, and if a certain angle value exceeds the stable interval threshold, it is considered that the direction suddenly changes, the point sequence is interrupted, and the judgment is restarted, and the output is a set of multiple point sequences that meet the direction continuity.

6. The unqualified solder ball identification and rejection system based on online monitoring according to claim 1 is characterized in that: The defect removal module includes: Density assessment submodule: extracts connected areas based on the topological offset identification set, accumulates the number of horizontal and vertical coordinates of nodes in each area, uses a convolutional neural network to extract a joint feature map of node distribution and grayscale change in the area, calculates the total number of nodes in the unit area grid, extracts the area boundary coordinate point set to calculate the circumscribed area valley value, groups and counts the difference between the internal pixel grayscale valley peak value, constructs a joint array of node number, area and grayscale change, and generates a node distribution feature map; Intensity screening submodule: Based on the node distribution feature map, extract pixel units whose grayscale values ​​are all greater than the average grayscale response and perform a continuous adjacent position aggregation operation, measure the number of pixels in the aggregated area, calculate the corresponding proportion of the total area of ​​the map, compare the rate of change of the region with the average grayscale response of the four neighbors, retain the areas with fluctuations greater than the response mean, perform a contour closure check and confirm the area, and obtain a set of high-response closed areas; Label generation submodule: Based on the high-response closed area set, fit the outer contour boundary of each area to a fitting rectangle, extract the boundary range of the four-point coordinates of the rectangle, judge the intersection of the positions between the areas, perform the horizontal and vertical overlap rates and screen out areas with high overlap ratios, extract the center point coordinates and category numbers of the remaining areas, construct a set of independently numbered block areas, and generate a group of images with excluded classification labels.

7. The unqualified solder ball identification and rejection system based on online monitoring according to claim 1 is characterized in that: The convolutional neural network is based on the formula: Where: U is the joint feature response map value, μ p is the central response value output by the pth convolution channel, is the average value of the center response of all channels, ρ p is the pixel density coefficient of the pth region, x p is the texture granularity measure of the pth region, δ p is the weighted value of the intensity variation of the pth region, λ is the response difference index, q is the total number of channels, and p is the number of the pth channel.

8. The unqualified solder ball identification and rejection system based on online monitoring according to claim 1 is characterized in that: The motion tracking module includes: Number matching submodule: Based on the elimination classification label image group, the image frame number in each label image is extracted, and then the corresponding time number in the elimination device action log is matched. After the matching frame number is extracted, it is sorted in chronological order. The trigger time of each group of frame records and the execution time interval in the action record are calculated and it is determined whether the interval value is greater than the preset difference threshold. The number, offset second value and label status identifier of the delay exceeding limit frame are recorded to generate a label action timing mapping set; Error extraction submodule: Based on the label action timing mapping set, the center coordinates of the marked area in each frame image and the horizontal and vertical differences of the execution position coordinates in the action record are calculated, the error vector is extracted and the direction angle measurement and quantity are counted, the error samples whose direction angles fall into the continuous concentrated interval are classified and marked, and the direction consistency ratio is counted. Combined with the time delay information in the corresponding frame, a joint record table is formed, and sample frames with large errors and a high proportion of concentrated directions are screened out to obtain a list of execution difference distributions.

9. The unqualified solder ball identification and rejection system based on online monitoring according to claim 1 is characterized in that: The weight updating module includes: Confidence determination submodule: Based on the execution difference distribution list, extract the confidence value of each classification label in the predicted output of the corresponding image frame in each group of records, compare the classification label and the action feedback label number to see if they are consistent and calculate the numerical offset, extract the frame sample number whose absolute value of the difference is greater than the confidence threshold range, and mark the direction classification code to generate the output deviation confidence set; Label extraction submodule: Based on the output deviation confidence set, extract the classification sequence index corresponding to the predicted label in each abnormal frame number, scan the entire classification probability sequence to locate the corresponding index position, count the number of times it is ranked at the end and archive the label numbers, identify the category numbers with a frequency higher than the lower limit of the proportion, aggregate them into a clustered label set, and obtain a high-frequency label interference set; Parameter adjustment submodule: Based on the high-frequency label interference set, the historical prediction offset rate and interference frequency ratio corresponding to each interference label number are calculated, the parameters of the column positions of each type of label in the classification matrix are replaced, the probability amplitude ratio of the label is redistributed and the label strength numerical distribution structure is adjusted, the updated parameter matrix is ​​established, and the judgment output adjustment matrix is ​​generated.

10. The unqualified solder ball identification and rejection system based on online monitoring according to claim 1 is characterized in that: When acquiring the solder ball array image, an inclined plate structure with an inclined angle is set up so that the solder balls roll along the surface of the inclined plate under the action of gravity to form a single-line channel. A high-frame rate area scanning camera is fixedly installed above the inclined plate. The camera continuously collects multiple frames of images of the solder balls during the rolling process at a speed of not less than 300 frames per second. The images are spatially aligned by combining the frame timestamp and the solder ball displacement information. The images of the same solder ball at different instantaneous angles during the rolling process are spliced ​​into a set of complete perspective frame sequences to construct a two-dimensional solder ball array image. The solder ball array image can realize the acquisition of multi-angle status of the solder ball surface.

Images