A Computer Vision-Based Method and System for Defect Diagnosis in Chip Packaging

Abstract

This application provides a chip packaging defect diagnosis method and system based on computer vision, belonging to the field of computer vision technology. First, it acquires dynamic multi-source image sequences of key processes in chip packaging, including solder melting, encapsulation adhesive curing, and pin shaping. Then, it performs defect feature cell encoding processing on the dynamic multi-source image sequences to form a defect feature cell encoding set. Next, it constructs a cellular automaton evolution model based on this defect feature cell encoding set. Then, it performs multi-source association verification processing on the cellular automaton evolution model, matching the cell evolution trajectory with a defect evolution rule base to generate verification results. Finally, it analyzes the verification results to reconstruct the complete evolution path and source association chain of the defect cells, thereby accurately determining the type of defect, the specific process and location where it occurred, and outputting a comprehensive diagnostic result. This invention achieves accurate source tracing and diagnosis of the dynamic evolution process of packaging defects through cross-process and multi-source fusion.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method and system for diagnosing chip packaging defects based on computer vision. Background Technology

[0002] Real-time diagnosis and tracing of defects in chip packaging manufacturing are crucial for ensuring the quality of the final product. Existing defect detection technologies largely rely on analyzing static images of the finished product after a single process is completed. For example, visible light or X-rays are used to photograph the packaged chip, and image processing algorithms are then used to identify static defects such as cracks, bubbles, and pin deformation. These methods have significant limitations: First, they are "post-hoc inspections," unable to capture the generation and propagation of defects during the dynamic evolution of key packaging processes, making it difficult to pinpoint the precise process and root cause of the defect. Second, relying on image data from a single source (such as visible light only) provides limited information and is insensitive to internal or minute defects that only manifest under specific physical conditions (such as temperature changes), leading to a high false negative rate. Third, existing technologies typically treat the detection of each process as an independent event, lacking effective modeling of the correlation and evolution mechanisms between different processes, resulting in isolated and one-sided diagnostic results that fail to form a complete tracing chain from defect initiation to final manifestation. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a chip packaging defect diagnosis method and system based on computer vision.

[0004] According to a first aspect of this application, a computer vision-based method for diagnosing chip packaging defects is provided, the method comprising:

[0005] Collect multi-source evolution image data of chip packaging. The multi-source evolution image data of chip packaging includes dynamic images of key processes of chip packaging such as solder melting, encapsulation glue curing, and pin shaping, as well as multi-source image sequences of infrared thermal imaging and visible light imaging for each process. The multi-source image sequence is accompanied by process time stamps, source identifiers and equipment parameters.

[0006] Defect feature cell encoding processing is performed on the multi-source evolution image data of chip packaging. The features are split into traceable cell units and encoded to obtain a set of defect feature cell codes.

[0007] A cellular automaton evolution model is constructed based on the defect feature cell encoding set. The cellular automaton evolution model includes the inter-process transition rules, source association rules, and mutation triggering conditions of the cell state.

[0008] Multi-source correlation verification is performed on the cellular automata evolution model, matching the cellular evolution trajectory with the defect evolution rule base to generate cellular verification results;

[0009] The cell verification results are analyzed to reconstruct the evolution path and source association chain of defective cells, determine the defect type, occurrence process and location, and output chip packaging defect diagnosis results containing defective cell information, evolution path, type and location.

[0010] According to a second aspect of this application, a computer vision-based chip packaging defect diagnosis system is provided. The computer vision-based chip packaging defect diagnosis system includes a processor and a readable storage medium. The readable storage medium stores a program that, when executed by the processor, implements the aforementioned computer vision-based chip packaging defect diagnosis method.

[0011] Based on any of the above aspects, by collecting dynamic multi-source image sequences covering key processes such as solder melting, encapsulation adhesive curing, and pin shaping, along with precise process context information, and then using defect feature cellular encoding processing, complex image features are decomposed into independently traceable basic units, achieving refined and structured representation of defect features. On this basis, a cellular automata evolution model is constructed to cleverly simulate the state transitions of defect features between processes, the correlation between multi-source information, and the dynamic process of anomaly triggering, connecting isolated defect feature points into an evolutionary trajectory with inherent logic. Through multi-source correlation verification, this evolutionary trajectory is matched and verified against a preset defect evolution rule base, significantly improving the accuracy of defect identification and achieving a deep reconstruction of the root causes and propagation paths of defects. Ultimately, the diagnostic results output by this invention are no longer isolated defect type judgments, but rather in-depth analysis reports containing complete evolutionary paths and source correlation chains, achieving a fundamental improvement in chip packaging defect identification from static identification to dynamic tracing, and from single-point diagnosis to full-chain insight, greatly enhancing the predictability and accuracy of quality control. Attached Figure Description

[0012] Figure 1 A flowchart illustrating the chip packaging defect diagnosis method based on computer vision provided in an embodiment of this application is shown.

[0013] Figure 2 This illustration shows a schematic diagram of the component structure of a computer vision-based chip packaging defect diagnosis system provided in an embodiment of this application. Detailed Implementation

[0014] Figure 1 A flowchart illustrating the chip packaging defect diagnosis method based on computer vision provided in this application embodiment is shown, and the detailed steps are described below.

[0015] Step S110: Collect multi-source evolution image data of chip packaging. The multi-source evolution image data of chip packaging includes dynamic images of key processes of chip packaging such as solder melting, encapsulation glue curing, and pin shaping, as well as multi-source image sequences of infrared thermal imaging and visible light imaging for each process. The multi-source image sequence is accompanied by process time stamps, source identifiers and equipment parameters.

[0016] In this embodiment, image acquisition devices consisting of visible light cameras and infrared thermal imagers can be deployed at each key process station of the chip packaging production line. The device for the solder melting process station is installed above the solder furnace outlet, the device for the encapsulating adhesive curing process station is installed on the side of the curing furnace conveyor belt, and the device for the lead shaping process station is installed directly in front of the shaping mold.

[0017] As the chip enters the process area via the conveyor belt, an image acquisition device is triggered to synchronously capture images at a preset frame rate. A visible light camera captures the appearance of the package structure, while an infrared thermal imager captures the temperature field distribution image of the corresponding area. Each image is automatically appended with a process timestamp, formatted as "process code-year-month-day-hour-minute-second-millisecond," where solder melting is "HX," encapsulation adhesive curing is "FG," and pin shaping is "YZ." Source identification uses "VIS" for visible light images and "IR" for infrared thermal imaging. Equipment parameters are acquired in real-time from the production equipment control system via an industrial bus, including the solder furnace's set temperature, conveyor belt speed, curing oven's heating curve, and shaping mold pressure parameters, and are stored in association with the corresponding process image sequence via process timestamps.

[0018] Step S120: Perform defect feature cell encoding processing on the chip packaging multi-source evolution image data, split the features into traceable cell units and encode them to obtain a defect feature cell encoding set.

[0019] Step S121: Classify the multi-source evolution image data of chip packaging according to the process time stamp, group the infrared thermal imaging and visible light imaging at the same process time into one group to obtain the process source image group. Each process source image group is accompanied by the process time stamp, source identifier and equipment parameters.

[0020] In this embodiment, stored multi-source evolution image data is read, and the process time stamp field of each image is parsed. The time stamp prefix is ​​extracted according to "process code-year-month-day-hour-minute-second". Visible light images and infrared thermal images with the same time stamp prefix are merged into a process source image group. For example, a visible light image with the time stamp "HX-2024-05-20-10-30-25-123" and an infrared thermal image with the time stamp "HX-2024-05-20-10-30-25-125" are determined to be images from the same process time and grouped together because the first 19 digits of the time stamp prefix are the same. An index file is generated for each process source image group, recording the number of images contained, the source identifier of each image, the complete process time stamp, and the corresponding equipment parameter snapshot.

[0021] Step S122: Perform collaborative feature extraction on the multi-source images in each process source image group to obtain collaborative extracted features. Visible light images extract the contour features, pin arrangement features, interface bonding features, and material texture features of the packaging structure. Infrared thermal imaging extracts temperature distribution features, temperature gradient features, and heat conduction uniformity features. Contour features are obtained by scanning the image boundary using an edge tracking algorithm, recording the coordinates of pixels with gray-level abrupt changes, and fitting them to a closed curve. The curvature change, line segment length, and corner angle of the curve are extracted. Pin arrangement features are obtained by identifying individual pins through connected component segmentation, calculating the geometric center coordinates of each pin, the distance between pins, and the angle between adjacent pins. Interface bonding features are obtained by scanning the gray-level interface region, recording the gray-level abrupt change range, abrupt change amplitude, and continuous segment length. Material texture features are obtained by statistically analyzing the gray-level change period in different directions using a gray-level co-occurrence matrix, calculating texture density, uniformity, and directional consistency. Temperature distribution features are obtained by statistically analyzing the temperature values ​​of thermal imaging pixels to obtain a temperature distribution histogram. Temperature gradient features are obtained by calculating the gradient distribution matrix using the temperature difference between adjacent pixels. Heat conduction uniformity features are obtained by calculating the temperature standard deviation and coefficient of variation.

[0022] Step S1221: Perform preprocessing on the visible light image in the process source image group: use Gaussian filtering algorithm to remove image noise, adjust image grayscale distribution through histogram equalization, and use image registration algorithm to align the visible light image with the preset chip packaging standard template to obtain the registered visible light image.

[0023] Visible light images are read from the source image group of the process, converted to grayscale, and then input into the preprocessing flow. The Gaussian filtering module uses a 5×5 filter kernel, with kernel element values ​​calculated using a two-dimensional Gaussian function, to perform convolution operations on the image to eliminate high-frequency noise. The histogram equalization module statistically analyzes the grayscale histogram of the image, calculates the cumulative distribution function, and maps the original grayscale values ​​to a new grayscale range, enhancing image contrast. Image registration employs a feature-point-based registration method, extracting the corresponding corner points of the package edge in the image and the standard template. The affine transformation matrix is ​​calculated using the least squares method, and the image is rotated, translated, and scaled to ensure that key features of the package structure (such as pin arrays and package body boundaries) coincide with the standard template, resulting in a registered visible light image.

[0024] Step S1222: Perform contour feature extraction on the registered visible light image: Scan the image using an edge detection operator, set high and low thresholds to filter edge pixels, and dynamically adjust the high and low thresholds according to the average gray value of the image; perform connectivity analysis on the edge pixels, delete isolated edge points, and connect adjacent edge pixels to form edge segments; perform curve fitting on the edge segments to obtain the closed contour curve of the package body, pins, and interface; extract the geometric parameters of the contour curve, including the curvature value of each segment of the curve, the length of the segment, the turning angle of adjacent segments, and the roundness and rectangularity of the contour.

[0025] After registration, the visible light image is input into the contour feature extraction module. First, the Canny edge detection operator is used for edge pixel detection. The high threshold of the Canny edge detection operator is typically set to 1.5 times the average grayscale value of the image, and the low threshold is set to 0.4 times the high threshold. Strong and weak edge pixels are filtered out using this dual-threshold method. Eight-neighborhood connectivity analysis is performed on the edge pixels, and isolated edge point sets with fewer than 5 pixels are deleted. The remaining edge pixels are connected into continuous edge segments using a chain code-based edge tracing algorithm. The Douglas-Peucker algorithm is used to approximate the edge segments into polygons, retaining key turning points, and fitting closed curves such as the rectangular contour of the package body, the trapezoidal contour of the pins, and the annular contour of the package interface. For each closed contour, curvature values ​​are calculated by sampling points at 10-pixel intervals, and the distribution of curvature extrema is statistically analyzed. The length of the line segment between adjacent turning points is measured, and the angle between the line segments is calculated. The roundness is calculated by the ratio of the contour area to the area of ​​the smallest circumscribed circle, and the rectangularity is calculated by the ratio to the area of ​​the smallest circumscribed rectangle.

[0026] Step S1223: Extract pin arrangement features from the registered visible light image: Use an adaptive threshold segmentation algorithm to segment the pin region and the background region, where the threshold is set according to the grayscale difference between the pin and the background; Perform connected component analysis on the pin region to mark the pixel range of each independent pin; Calculate the minimum bounding rectangle of each pin region and determine the geometric center coordinates of the rectangle as the pin center coordinates; Statistically analyze the distribution pattern of all pin center coordinates, and calculate the pixel distance between adjacent pin centers, the angle between adjacent pins, and the standard deviation of the distribution of all pin center coordinates.

[0027] For the pin regions of the registered visible light image, the NiBlack adaptive thresholding algorithm is used, with a local window of 15×15 pixels. The threshold calculation formula within the window is T=mk×s, where m is the mean gray level of the window, s is the standard deviation of the gray level of the window, and k is set to 0.2. After obtaining the binarized image, an 8-neighborhood connected region labeling algorithm is used to identify individual pin regions, and the set of pixel coordinates for each connected region is recorded. For each pin region, the minimum bounding rectangle algorithm is used to calculate its bounding box. The geometric center coordinates (cx, cy) of the rectangle are calculated by the mean of the coordinates of the top left and bottom right corners of the rectangle. All pin center coordinates are sorted according to the x-axis coordinate, and the Euclidean distance between adjacent pin centers is calculated. The angle between the line connecting adjacent pin centers and the x-axis is calculated using the arctangent function. The standard deviations of the distribution in the x and y directions of all pin center coordinates are calculated to evaluate the neatness of the pin arrangement.

[0028] Step S1224: Extract the bonding features of the encapsulation interface from the registered visible light image: Locate the interface area between the encapsulation adhesive and the chip body, set scan lines along the interface direction, and set the scan line spacing according to the interface width; collect the pixel grayscale values ​​on each scan line, and record the position and amplitude of grayscale value abrupt changes; count the continuous length of the abrupt change position, and calculate the proportion of the continuous abrupt change segment to the total length of the interface as the bonding degree; record the maximum, minimum and average values ​​of the abrupt change amplitude as interface bonding quality parameters.

[0029] Based on the standard chip packaging template, the annular interface area between the encapsulant and the chip body is located. This annular interface area is typically 5-10 pixels wide. A radial scan line is set every 1 degree along the circumference of the interface, covering the interface area and a 5-pixel range above and below it. For each scan line, pixel grayscale values ​​are collected from the inside to the outside of the interface. When the grayscale change of three consecutive pixels exceeds 20, it is considered a grayscale abrupt change. The start and end positions of the abrupt change, as well as the abrupt change amplitude (maximum grayscale difference), are recorded. The ratio of the total length of the consecutive abrupt change segments to the total perimeter of the interface represents the bonding degree; a higher ratio indicates a tighter interface bonding. Simultaneously, the maximum, minimum, and arithmetic mean of all abrupt change amplitudes are recorded to form a set of interface bonding quality parameters.

[0030] Step S1225: Perform preprocessing on the infrared thermal image in the process source image group: use the median filtering algorithm to remove thermal imaging noise, and set the filtering window size according to the thermal imaging resolution; convert the thermal imaging pixel values ​​into actual temperature values ​​through the temperature calibration algorithm, and retrieve the calibration parameters from the equipment parameters; align the thermal image with the registered visible light image to obtain the registered infrared image.

[0031] During infrared thermal imaging preprocessing, a filtering window is selected based on the image resolution. A 3×3 median filtering window is used for 640×512 resolution images, and a 5×5 window is used for 1280×1024 resolution images. The median value of the neighboring pixels of each pixel is sorted and used to replace the original value. Temperature calibration is achieved by finding the "temperature-grayscale" conversion curve in the device parameters. This curve is generated from the thermal imager's factory calibration data and maps grayscale values ​​to Celsius temperature values. Image registration uses a contour feature-based registration method. The edge contour of the encapsulated body in the infrared thermal image is extracted and rigidly transformed (rotated and translated) with the corresponding contour of the registered visible light image to make the contour centers of the two coincide and the contour dimensions consistent, thus obtaining the registered infrared image.

[0032] Step S1226: Perform temperature feature extraction on the registered infrared image: count the temperature values ​​of all pixels in the image to obtain a temperature distribution histogram; use the gradient operator to calculate the temperature difference between adjacent pixels to obtain a gradient distribution matrix; calculate the standard deviation and coefficient of variation of the temperature values ​​as features of heat conduction uniformity; analyze the areas where the temperature values ​​are higher than the preset normal range, and record the area, center coordinates and mean temperature of the area as features of thermal anomaly areas.

[0033] In the temperature feature extraction of the registered infrared image, the number of bins in the temperature distribution histogram was set to 256, and the number of pixels in each temperature range was counted. The temperature gradient was calculated using the Sobel operator, with gradient values ​​Gx and Gy calculated in the x and y directions respectively. The gradient magnitude was calculated using sqrt(Gx^2+Gy^2) to obtain the gradient distribution matrix. The heat conduction uniformity feature was achieved by calculating the standard deviation (σ) and coefficient of variation (σ / μ, where μ is the temperature mean) of the temperature values ​​across the entire image. The preset normal temperature range was set according to the characteristics of the encapsulation material: 220-240℃ for soldering and 120-140℃ for encapsulation adhesive curing. Areas exceeding this range were extracted using connected component analysis, and their pixel area, geometric center coordinates, and the mean temperature within the region were recorded.

[0034] Step S123: Perform cell splitting on the collaboratively extracted features, dividing the chip packaging image into equal-sized cell grids, with each cell grid corresponding to a cell unit. Distribute the extracted features to each cell unit according to the cell grid range. Each cell unit contains all source feature parameters of the corresponding region. Define the state of the feature parameters of the cell unit, mapping the parameters of contour integrity, pin position deviation, interface fit, texture uniformity, temperature deviation, and gradient change rate to cell states. The states are divided into normal state, suspected state, and variant state.

[0035] The registered visible light image and the registered infrared image are uniformly divided into 10×10 pixel square cell grids, with areas of less than 10 pixels at the image edge retained according to the actual size. Each cell is assigned a unique grid coordinate (i, j), where i is the row index and j is the column index. Various collaboratively extracted features are assigned to corresponding cell units according to their spatial location. For example, if a curve in the contour feature passes through cell (3, 5), the curvature value, line length, and other parameters of that curve segment are stored in that cell; the pixel temperature value in the temperature feature is assigned to its corresponding cell. Cell state definition adopts multi-parameter fusion judgment. Contour integrity is evaluated by the proportion of missing pixels in the contour, pin position deviation is the Euclidean distance between the actual center and the standard center, interface fit is the ratio calculated in step S1224, texture uniformity is measured by the energy value of the gray-level co-occurrence matrix, temperature deviation is the difference between the cell temperature mean and the standard temperature, and gradient change rate is the mean of the gradient magnitude within the cell. Each parameter is set to a normal threshold range. When all parameters are within the threshold, it is considered normal. When 1-2 parameters exceed the threshold, it is considered suspected. When 3 or more parameters exceed the threshold or a single parameter exceeds the threshold by more than twice, it is considered abnormal.

[0036] Step S124: Assign a unique cell identifier to each cell unit. The cell identifier includes process code, source code, grid coordinate code and status code. The process code corresponds to the process time stamp, the source code distinguishes between infrared and visible light, the grid coordinate code corresponds to the row and column position of the cell in the grid, and the status code corresponds to normal state, suspected state and variant state.

[0037] Cell identifiers are encoded using a 24-bit string in the format "CC-SS-III-JJJ-ST", where CC is a 2-bit process code (HX / FG / YZ), SS is a 2-bit source code (VIS / IR), III is a 3-bit row index (000-999), JJJ is a 3-bit column index (000-999), and ST is a 2-bit state code (00 = normal state, 01 = suspected state, 10 = variant state). For example, the suspected state cell identifier of the (3,5) grid in the visible light image of the soldering process is "HX-VIS-003-005-01".

[0038] Step S125: Perform state encoding conversion on each cell unit, converting the cell state and corresponding feature parameters into a fixed-length binary code. The first part is the state code, the middle part is the visible light feature parameter code, and the last part is the infrared feature parameter code. The visible light feature parameter code and the infrared feature parameter code are obtained by normalizing their respective feature parameters. Calculate the check code for the encoded binary value and add it to the end of the code as a check bit.

[0039] The cell state encoding uses 16 bits of binary code. The first two bits are the state code (00 = normal state, 01 = suspected state, 10 = abnormal state), the middle six bits are the visible light characteristic parameter code, the last six bits are the infrared characteristic parameter code, and the last two bits are the check code. Visible light characteristic parameters include contour integrity, pin position deviation, interface fit, and texture uniformity. Each parameter is normalized to the [0, 15] interval and converted to 4 bits of binary code, which are then combined into a 6-bit code through a bitwise OR operation. Infrared characteristic parameters include temperature deviation and gradient change rate. After normalization to the [0, 31] interval, they are converted to 5 bits of binary code and combined into a 6-bit code. Even parity is used for the check code. The number of 1s in the first 14 bits is calculated; if the number is odd, the check bit is 1; otherwise, it is 0.

[0040] Step S126: Connect the cell codes under the same source in the order of the process time stamp to form the process source code sequence; integrate all process source code sequences and the mapping table of cell identifier and code, and the state parameter mapping rules to obtain the defect feature cell code set.

[0041] Cell codes from the same source (visible light or infrared) are arranged in ascending order by process timestamp, forming a process source code sequence. The sequence header includes metadata such as process code, source identifier, cell grid size, and number of feature parameters. A mapping table between cell identifiers and codes is stored in JSON format, recording the binary code and original value of each feature parameter corresponding to each cell identifier. State parameter mapping rules define the normalization method, threshold range, and state determination logic for each feature parameter. The above data is integrated into a defect feature cell code set, stored in ZIP compression format. The compressed package contains the code sequence file, mapping table file, and rule description file.

[0042] Step S130: Construct a cellular automaton evolution model based on the defect feature cell encoding set. The cellular automaton evolution model includes inter-process transition rules, source association rules, and mutation triggering conditions for cell states.

[0043] Step S131: Decode the process source encoding sequence in the defect feature cell encoding set, restore the process time, source type, grid coordinates, state and feature parameters of each cell unit through the mapping table, obtain the cell state dataset, and sort it according to the process time stamp to form the cell state time sequence.

[0044] The cellular automata model building module reads the defect feature cell encoding set, decompresses it, and decodes the process source encoding sequence through a mapping table. The decoding process first verifies the check bits and removes invalid codes, then parses the cell state and the original values ​​of the feature parameters based on the binary encoding. The decoded cell units are arranged in ascending order by process time stamp, and cells at the same process time are sorted by grid coordinates (i, j), forming a cell state time sequence. Each element in the sequence contains the process time stamp, source type, grid coordinates, cell state, and a complete set of feature parameters.

[0045] Step S132: Define the four-neighbor and eight-neighbor neighbors of each cell, record the neighbor list of each cell and the distance parameters between neighbors; calculate the similarity of the same type of feature parameters based on the multi-source feature parameters, and weight and fuse the similarities to obtain the comprehensive similarity. Cells with a comprehensive similarity higher than a preset threshold are marked as related cells and added to the neighbor list.

[0046] Cellular neighbors are defined using a dual neighborhood system. A four-neighbor neighborhood includes four cells: top (i-1, j), bottom (i+1, j), left (i, j-1), and right (i, j+1). An eight-neighbor neighborhood adds four more cells to the four-neighbor system: top-left (i-1, j-1), top-right (i-1, j+1), bottom-left (i+1, j-1), and bottom-right (i+1, j+1). The distance between neighbors is calculated using Manhattan distance, with a distance of 1 for four-neighbor neighborhoods and a distance of [missing value] for eight-neighbor neighborhoods. In feature similarity calculation, cosine similarity is used for visible light features, and Euclidean distance similarity is used for infrared features, with weights set to 0.6 and 0.4 respectively. The comprehensive similarity calculation formula is S = 0.6 × S_vis + 0.4 × (1 - S_ir / S_ir_max), where S_vis is the cosine similarity of visible light features, S_ir is the Euclidean distance of infrared features, and S_ir_max is the maximum possible distance. The preset comprehensive similarity threshold is 0.7; cells with a value higher than this are marked as related cells and added to the neighbor list.

[0047] Step S133: Analyze the cell state change data of normal and defective samples to determine the basic thresholds for the proportion of neighboring states, the deviation range of feature parameters, and the duration of time sequence. Based on the basic thresholds, construct the execution conditions for the conversion of normal cells to suspected states. The execution conditions are as follows: when a cell meets the condition that the number of mutated cells among its neighbors reaches a set proportion and its own feature parameters deviate from the normal range, the conversion operation from normal state to suspected state is executed. Construct the execution conditions for the conversion of suspected state cells to mutated states. The execution conditions are as follows: when a suspected state cell meets the condition that its feature parameters continue to deviate and its infrared temperature parameter becomes abnormal in a subsequent process, the conversion operation from suspected state to mutated state is executed. Construct the execution conditions for the conversion of mutated state cells to suspected states. The execution conditions are as follows: when a mutated state cell meets the condition that its feature parameters return to normal in a subsequent process and all its neighboring cells are in the normal state, the conversion operation from mutated state to suspected state is executed. Record the feature parameter thresholds, the proportion of neighboring states, and the duration of time sequence corresponding to the three conversion execution conditions to form a state conversion parameter table.

[0048] Cell state change data of 1000 normal samples and 500 defective samples were collected. Statistical analysis was used to determine the following basic thresholds: the proportion of abnormal cells among neighbors was set to 0.3 (i.e., more than 30% of neighbors were abnormal), the deviation range of feature parameters was 1.5 times the normal threshold, and the duration was two process cycles. The conditions for transitioning from normal to suspected state were: the proportion of abnormal cells in the four-neighbor area ≥ 0.3, and any one feature parameter exceeding the normal threshold. The conditions for transitioning from suspected to abnormal state were: the deviation range of feature parameters ≥ 1.5 times the normal threshold within two consecutive process cycles, and the infrared temperature parameter exceeding the normal range. The conditions for transitioning from abnormal to suspected state were: all feature parameters returning to the normal threshold within two consecutive process cycles, and all eight neighboring cells being normal. The above conditions and corresponding parameters were recorded in a state transition parameter table in CSV format, including fields such as transition type, feature parameter name, threshold, neighbor proportion, and duration.

[0049] Among them, step S1331: collect normal sample image data and defective sample image data of each process of chip packaging, and the sample data covers production scenarios under different equipment parameters and different environmental conditions; perform defect feature cell encoding processing on the collected normal sample image data and defective sample image data respectively, and obtain normal sample cell state set and defective sample cell state set through encoding processing.

[0050] When collecting sample data, 1000 normal samples and 800 defective samples were collected for each of the three key processes: solder melting, encapsulation adhesive curing, and lead shaping. Normal samples needed to cover different combinations of equipment parameters, such as solder temperature (in increments of 5°C within the 220-240°C range), conveyor belt speed (in increments of 0.2 m / min within the 0.5-1.5 m / min range), curing temperature (in increments of 5°C within the 120-140°C range), curing time (in increments of 5 min within the 20-40 min range), and shaping pressure (in increments of 20 N within the 200-400 N range) and die clearance (in increments of 0.02 mm within the 0.1-0.3 mm range). Environmental conditions included workshop temperature (in increments of 2°C within the 20-30°C range) and humidity (in increments of 5% within the 40%-60% range). The defect samples cover seven typical defect categories, including cracks, missing corners, pin deformation, encapsulation bubbles, encapsulation overflow, interface peeling, and thermal deformation. There are no fewer than 100 sets of samples for each category, and each set of defect samples is labeled with a defect level (minor, moderate, severe). The defect feature cell encoding processing described in step S120 is performed on all sample image data to obtain the normal sample cell state set (containing the normal state feature parameters of each process and each cell) and the defect sample cell state set (containing the abnormal state feature parameters of each process, each defect type, each cell, and state transition process data).

[0051] Step S1332: Analyze the normal sample cell state set, classify and statistically analyze the characteristic parameter range of the normal state of the cells according to process type and modal type, and determine the normal threshold of different characteristic parameters based on the statistical results; calculate the similarity of adjacent cells in the normal sample, statistically analyze the similarity distribution, and determine the similarity threshold of related cells based on the distribution. The different characteristic parameters include contour curvature, pin spacing, interface fit, and temperature.

[0052] The normal sample cell state set is classified according to process type (soldering, encapsulant curing, pin shaping) and modal type (visible light, infrared). For characteristic parameters such as contour curvature, pin spacing, interface fit, and temperature in each data category, the normal threshold range is calculated using the 3σ criterion: lower limit of normal threshold = sample mean - 3 × sample standard deviation, upper limit = sample mean + 3 × sample standard deviation. For example, the sample mean of contour curvature in the visible light mode of the soldering process is 0.1 / mm, and the standard deviation is 0.02 / mm, so the normal threshold range is 0.04-0.16 / mm. The comprehensive similarity of all adjacent cells (four-neighborhood and eight-neighborhood) in the normal sample is calculated (calculation method is the same as step S132), and the frequency histogram of the similarity distribution is statistically analyzed. The similarity value with a cumulative frequency of 95% is taken as the similarity threshold of the associated cells, which is usually between 0.65 and 0.75.

[0053] Step S1333: Analyze the cell state set of defect samples, track the cell state changes of each defect sample from initial formation to diffusion process, record the transition time from normal state to suspected state, and from suspected state to variant state in each sample, the change in characteristic parameters, and the state of the corresponding neighboring cells; classify and statistically analyze the state transition rules according to defect type, and determine the rules that crack defects are manifested as gradual diffusion of cells with abnormal contour curvature, and encapsulated bubbles are manifested as the synergistic occurrence of abnormal interface adhesion and temperature.

[0054] For each defect sample, starting from the initial defect formation process, the process is traced to all subsequent processes, recording the specific process timestamps (accurate to the second) for each cell as it transitions from the normal state to the suspected state, and from the suspected state to the mutated state. Simultaneously, the changes in feature parameters during state transitions are recorded, i.e., the difference in feature parameters before and after the transition (e.g., change in contour curvature = curvature after transition - curvature before transition), as well as the distribution of the number of neighboring cells (four-neighborhood and eight-neighborhood) in their states (normal state, suspected state, mutated state) at the transition time. The above data is then classified and statistically analyzed according to defect type (7 categories including cracks and missing corners), and a temporal pattern mining algorithm (such as PrefixSpan) is used to extract state transition patterns. For example, the pattern of crack defects is as follows: initially, abnormal contour curvature appears in the cells at the edge of the package (more than twice the normal threshold). Subsequently, adjacent cells along the contour direction change from the normal state to the suspected state and then to the abnormal state in 1-2 process cycles, forming a linear diffusion trajectory. The pattern of bubble defects is as follows: the cells in the package interface area first show abnormal interface adhesion (less than 0.5 times the normal threshold), and at the same time, the corresponding infrared mode cells show abnormal temperature (more than 1.5 times the normal threshold). The two abnormal features overlap by more than 80% in spatial location, and the time difference between their simultaneous occurrence is less than 5 seconds.

[0055] Step S1334: Construct the first state transition logic for the transformation from normal state cell to suspected state: Set the execution conditions as follows: any feature parameter of the cell exceeds the normal threshold of the corresponding process, there are a set number of suspected or variant state cells among its four neighbors, and the parameter deviates from the state for a set process duration; when the cell meets the execution conditions, perform the transformation operation from normal state to suspected state; when only the feature parameter exceeds the threshold but all neighbors are normal state, or the parameter deviates from the state for a set process duration that is less than the set duration, perform the operation of maintaining the normal state and mark it as a warning cell.

[0056] The execution conditions of the first state transition logic include three parallel conditions: (1) At least one of the characteristic parameters of the cell, such as the contour curvature, pin spacing, interface fit, and temperature, exceeds the normal threshold range of the process; (2) The total number of suspected and abnormal cells in its four neighboring cells is greater than or equal to the set number, which is determined according to the process type. The setting number is 2 for soldering and encapsulation glue curing processes and 1 for pin shaping processes; (3) The duration of the parameter deviation from the state is greater than or equal to 1 process cycle (the process cycle is determined according to the production cycle, such as 30 seconds / cycle for soldering processes). When all three conditions are met, the cell state changes from the normal state to the suspected state. If only condition (1) is met, but condition (2) is not met (all neighbors are in the normal state), or condition (3) is not met (the duration of parameter deviation is less than 1 process cycle), the cell remains in the normal state but is marked as a warning cell, and its state change is closely monitored in subsequent processes.

[0057] Step S1335: Construct the second state transition logic for converting suspected state cells to mutated state cells: Set the execution conditions as follows: in subsequent processes, both modal characteristic parameters of suspected state cells exceed the normal threshold, the number of mutated and suspected state cells in the eight-neighborhood is greater than the number of normal state cells, and the temperature value of the corresponding area in infrared thermal imaging is higher than the normal threshold and the excess reaches the set standard; when the execution conditions are met, execute the conversion operation from suspected state to mutated state; when only a single modal parameter is abnormal or the number of abnormal neighbors is less than the set number, execute the operation of maintaining suspected state; when both modal characteristic parameters return to normal and all neighbors are normal, execute the operation of converting suspected state back to normal state.

[0058] The execution conditions of the second state transition logic include three parallel conditions: (1) In the subsequent two consecutive process cycles, the characteristic parameters of the visible light mode and the infrared mode of the suspected state cell both exceed their respective normal threshold ranges (e.g., visible light profile curvature > 0.16 / mm and infrared temperature > 240℃); (2) Among its eight neighboring cells, the number of variant state cells + the number of suspected state cells > the number of normal state cells; (3) The temperature value of the corresponding area of ​​infrared thermal imaging is higher than the normal threshold of infrared mode temperature in this process, and the excess is ≥ 10% of the upper limit of the normal threshold (e.g., if the upper limit of the normal threshold is 240℃, then the excess is ≥ 24℃, i.e., the temperature is ≥ 264℃). When all three conditions are met, the cell state changes from suspected state to variant state. If only the single modal parameter abnormality in condition (1) is met, or the number of abnormal neighbors in condition (2) is ≤ the number of normal states, then the cell remains in suspected state. If, within the next process cycle, the dual-modal characteristic parameters of the suspected cell recover to the normal threshold range and all eight neighboring cells are in the normal state, then the cell state changes from the suspected state back to the normal state.

[0059] Step S1336: Construct the third state transition logic for the transformation from a variant cell to a suspected state: Set the execution conditions as follows: in subsequent processes, both modal feature parameters of the variant cell recover to the normal threshold range, the proportion of normal cells in the eight neighboring regions reaches a set ratio, and the state is maintained at each of the consecutive set processes; when the execution conditions are met, the transformation operation from variant to suspected state is executed; when only a single modal parameter recovers or the proportion of normal neighboring cells is less than the set ratio, the operation of maintaining the variant state is executed; when the feature parameters continue to be abnormal and spread to new neighboring cells, the operation of maintaining the variant state is executed and marked as a spread state.

[0060] The execution conditions of the third state transition logic include three parallel conditions: (1) the bimodal characteristic parameters of the mutated cell are restored to the normal threshold range in the next two consecutive process cycles; (2) the proportion of normal cells in its eight neighboring cells is greater than or equal to the set ratio (the set ratio is determined according to the process type, set to 80% for soldering and encapsulation glue curing processes, and 90% for pin shaping processes); (3) the above state (normal bimodal parameters and normal neighboring cell proportion) is maintained for two consecutive process times. When all three conditions are met, the cell state is changed from the mutated state to the suspected state. If only the single-modal parameter recovery in condition (1) is met, or the proportion of normal neighboring cells in condition (2) is less than the set ratio, the cell remains in the mutated state. If the characteristic parameters of the mutated cell continue to be abnormal (exceeding the normal threshold) in subsequent processes, and a new cell in its eight neighboring cells is changed from the normal state / suspected state to the mutated state (i.e., defect diffusion), the cell remains in the mutated state and is additionally marked as the diffusion state to distinguish it from the non-diffusion mutated cell.

[0061] Step S1337: Select a verification sample set to test the accuracy of the first state transition logic, the second state transition logic, and the third state transition logic, and record the false positives and false negatives during the test; adjust the key parameters for the false positives and false negatives, and re-execute the test; when the accuracy of the transition logic in distinguishing between normal and abnormal state transitions reaches the set standard, determine the specific parameters of the state transition logic, including the feature parameter threshold and the proportion of neighbor numbers.

[0062] 30% of the data from the defective sample cell state set is randomly selected as the validation sample set, and the remaining 70% is used as the training sample set (already used for the initial construction of the aforementioned state transition logic). The validation sample set is used to test the accuracy of the first, second, and third state transition logics. Accuracy = (number of cases that transitioned correctly + number of cases that correctly maintained the original state) / total number of test cases. False positives (e.g., a normal state mistakenly transitioning to a suspected state) and false negatives (e.g., a state that should have transitioned to a suspected state but remained normal) are recorded during the testing process. For false positives and false negatives, the causes are analyzed and key parameters are adjusted. For example, if a false positive is caused by a feature parameter threshold that is too low, the threshold is appropriately increased; if a false negative is caused by a high proportion of neighbors, the proportion is appropriately decreased. After adjustment, the validation sample set is used again for testing. This process is repeated until the accuracy of the state transition logic in distinguishing between normal and abnormal state transitions reaches the set standard (e.g., accuracy ≥ 95%). The feature parameter threshold and the proportion of neighbors determined at this point are the final specific parameters of the state transition logic.

[0063] Step S134: Establish the cell grid coordinate mapping relationship between the visible light image and the infrared image at the same process time to ensure that cells at the same physical location in the dual-modal image correspond to the same grid coordinates; when a suspected or variant cell is detected in the visible light image, locate the cell with the corresponding grid coordinate in the infrared image according to the coordinate mapping relationship; check whether the located infrared cell has a temperature anomaly. If a temperature anomaly exists, complete the dual-modal anomaly matching and mark the suspected or variant cell in the visible light as a valid state cell; if only an abnormal cell is detected in the single-modal image and no anomaly is detected at the corresponding coordinate in the other modality, mark the abnormal cell as a cell to be confirmed; calculate the degree of matching of the dual-modal feature parameters of the valid state cell, and record the calculation result as the dual-modal cell state matching degree.

[0064] The transformation matrix obtained through image registration establishes a coordinate mapping relationship between visible light and infrared image cell grids, ensuring that regions with the same physical location correspond to the same (i, j) grid coordinates. When a suspected or aberrant cell is detected in the visible light image, its grid coordinates (i, j) are extracted, and cells with the same coordinates are searched in the infrared image at the same processing time. The temperature deviation of the infrared cells is checked to see if it exceeds the normal range. If it does, it is determined to be a bimodal abnormal match, and the visible light cell is marked as a valid state cell. If the infrared cell temperature is normal, the visible light abnormal cell is marked as a cell to be confirmed. The degree of matching of bimodal feature parameters is achieved by calculating the correlation coefficient between visible light abnormal features and infrared abnormal features. For example, the spatial correlation between contour discontinuities in visible light and temperature gradient anomalies in infrared is considered. A correlation coefficient absolute value ≥ 0.5 is considered a high match, denoted as 1.0; 0.3-0.5 is denoted as 0.7; and < 0.3 is denoted as 0.3.

[0065] Step S135: Extract the initial characteristic parameters of each process that triggers cell mutation in the historical defect samples, and classify them according to soldering, encapsulant curing, and pin shaping processes; for the soldering process, analyze the characteristic parameters in the samples that simultaneously deviate from the normal range when triggering mutation, determine the temperature gradient characteristic parameter and the pin arrangement deviation parameter as joint triggering parameters, and set the mutation triggering condition for the soldering process when both deviate from the normal range; for the encapsulant curing process, determine the temperature distribution dispersion parameter and the interface adhesion parameter as joint triggering parameters, and set the mutation triggering condition for the encapsulant curing process when both deviate from the normal range; for the pin shaping process, determine the pin position deviation parameter and the contour curvature parameter as joint triggering parameters, and set the mutation triggering condition for the pin shaping process when both deviate from the normal range; associate and store the mutation triggering conditions of each process with the corresponding process.

[0066] Five hundred defect samples were extracted from the historical defect database and divided into three groups according to process type, with no fewer than 100 samples in each group. For soldering process samples, Pearson correlation coefficient analysis was used to analyze the correlation between characteristic parameters. The correlation coefficient between temperature gradient and pin arrangement deviation was 0.68, which was determined to be a strong correlation and set as a joint triggering parameter. Variation was triggered when the temperature gradient > 5℃ / mm and the pin arrangement deviation > 50μm. In the encapsulation adhesive curing samples, the correlation coefficient between temperature distribution standard deviation and interface adhesion was -0.72 (negative correlation). Variation was triggered when the temperature distribution standard deviation > 8℃ and the interface adhesion < 0.6. In the pin shaping samples, the correlation coefficient between pin position deviation and contour curvature was 0.65. Variation was triggered when the position deviation > 30μm and the rate of change of curvature > 0.1 / mm. The joint triggering parameters and thresholds for each process were stored in an XML configuration file. The file structure included process nodes, parameter nodes, threshold nodes, and logical relationship nodes.

[0067] Step S136: Construct a cellular automaton evolution model with cellular units as nodes, neighbor relationships as edges, state transition logic and modal association logic as evolution logic, and mutation triggering logic as the initial triggering mechanism. In the cellular automaton evolution model, each node is marked with cell identifier, state, feature parameters and process time, and each edge is marked with neighbor type and similarity. The evolution logic is associated with process type and modal type.

[0068] The cellular automaton evolutionary model employs a directed graph data structure. Nodes store attributes such as cell identifier, current state, feature parameter vector, and process time stamp. Edges represent neighbor relationships, categorized as four-neighbor edges and eight-neighbor edges, with edge attributes including neighbor type (four-neighbor / eight-neighbor) and a comprehensive similarity value. State transition logic is implemented through a state transition parameter table, invoking the corresponding transition rule based on the current process type. Modal association logic is controlled by bimodal matching degree; cross-modal state influence is activated when the matching degree is ≥0.7. Mutation triggering logic reads the joint triggering parameters of each process, forcibly transforming the cell state to a mutated state when the conditions are met. The model is implemented using Python's NetworkX library, with node and edge attributes stored in a dictionary. The evolution process is achieved by iteratively updating the cell state at each process time.

[0069] Step S140: Perform multi-source correlation verification on the cellular automata evolution model, match the cellular evolution trajectory with the defect evolution rule base, and generate cellular verification results.

[0070] Step S141: Call the preset defect evolution rule library. The defect evolution rule library contains standard data for defects such as cracks, missing corners, pin deformation, encapsulation bubbles, encapsulation overflow, interface peeling, and thermal deformation. Each defect corresponds to a standard cell evolution trajectory, a standard multi-source association mode, a standard state transition sequence, and a standard mutation triggering parameter. The standard cell evolution trajectory is the cell state change path of the defect from triggering to diffusion. The standard multi-source association mode is the feature matching law of the defect under multiple sources. The standard state transition sequence is the cell state change order corresponding to the defect. The standard mutation triggering parameter is the feature parameter threshold when the defect initially occurs.

[0071] The defect evolution rule base is stored in a relational database, including a defect type table, a standard trajectory table, a multi-source association table, a state transition table, and a trigger parameter table. The defect type table records the defect ID, name, associated process, hazard level, etc.; the standard trajectory table stores the cell coordinate sequence and time sequence relationship of the defect from triggering to propagation; the multi-source association table defines the matching rules between visible light and infrared features, such as crack defects exhibiting "discontinuous visible light profile - localized high infrared temperature"; the state transition table specifies the transition order and duration from normal state to suspected state to variant state; and the trigger parameter table records the characteristic parameter thresholds at the initial occurrence of each defect, such as a 40μm threshold for pin deformation position deviation.

[0072] Step S1411: Collect physical samples of defects such as cracks, missing corners, pin deformation, packaging bubbles, packaging overflow, interface peeling, and thermal deformation that occur in chip packaging production. For each type of defect, collect sufficient samples covering different production scenarios, different equipment parameters, and different environmental conditions. Each sample is accompanied by detailed production records, including process parameters, equipment model, ambient temperature, defect location, and cause analysis.

[0073] In collaboration with the quality inspection department of the chip packaging production workshop, we collected physical samples of various defects that occurred during production over the past three years. For each defect type (cracks, missing corners, lead deformation, encapsulation bubbles, encapsulation overflow, interface peeling, thermal deformation), at least 200 physical samples were collected to ensure coverage of different production scenarios (e.g., different production lines, different shifts), different equipment parameters (e.g., different solder temperatures, encapsulation adhesive viscosity, shaping pressure, etc., with parameter ranges consistent with the sample collection range in step S1331), and different environmental conditions (e.g., different workshop temperatures and humidity, with ranges consistent with step S1331). Each physical sample was accompanied by detailed production records exported from the MES system, including specific process parameters for each step (e.g., solder furnace temperature in each zone, encapsulation adhesive coating amount, lead shaping mold model), production equipment model and serial number, ambient temperature and humidity during production, the specific location of the defect on the chip (accurate to millimeter coordinates), and a preliminary analysis of the defect cause by the quality engineer (e.g., "uneven solder temperature," "air bubbles mixed in with the encapsulation adhesive").

[0074] Step S1412: Perform full-process multi-source image acquisition for each defective sample. At the soldering, encapsulation glue coating, curing, and pin shaping process nodes, simultaneously acquire visible light images and infrared thermal imaging. The image acquisition parameters are consistent with the acquisition parameters in actual production. Add process time stamp, source identifier, equipment parameters, and sample number to each image.

[0075] A dedicated image acquisition system with parameters identical to the actual production line was constructed, including a visible light camera (1920×1080 resolution, 30fps frame rate, 16mm lens focal length) and an infrared thermal imager (640×512 resolution, 15fps frame rate, temperature range -20-500℃). For each defective physical sample, simulating the actual production process, visible light and infrared thermal images were simultaneously captured using the acquisition system at four key process nodes: solder melting, encapsulant coating, encapsulant curing, and pin shaping. The shooting parameters were set as follows: visible light image exposure time 1 / 100s, gain 1dB; infrared thermal imaging integration time 8ms, emissivity 0.95. Add the process time stamp (format same as step S110), source identifier ("VK" represents visible light, "HW" represents infrared), equipment parameters (camera / thermal imager model, focal length, exposure time, etc.) and sample number (format is "defect type code-sample number", such as "LW-001" represents crack sample number 1) to each acquired image.

[0076] Step S1413: Perform defect feature cell encoding processing and cellular automata evolution model construction on the collected sample image data to obtain the cell state time sequence, cell evolution trajectory, multi-source association data and mutation triggering parameters for each defect sample; the cell evolution trajectory includes all cell state change nodes from initial triggering to final formation of the defect, the multi-source association data includes the matching degree of multi-source cell states in each process, and the mutation triggering parameters include the feature parameters of the initial mutated cell and the triggering process.

[0077] For each defect sample, the multi-source image data collected from the entire process is processed strictly according to the defect feature cell encoding process described in step S120. This includes process source image group classification, collaborative feature extraction (contour, pin arrangement, interface bonding, material texture, temperature distribution, temperature gradient, heat conduction uniformity, etc.), cell splitting, cell identifier assignment, state encoding transformation, and encoding set generation. Subsequently, based on the encoded defect feature cell encoding set, a cellular automaton evolution model is constructed according to the method described in step S130, including cell state decoding, neighbor definition, state transition rule construction, source association rule establishment, and mutation trigger condition setting. The model outputs a cell state time sequence for each defect sample, sorted by process time stamps, recording the state and characteristic parameters of all cells in each process. The cell evolution trajectory clearly marks the complete node path from the initial trigger cell, through the transformation from normal state to suspected state to mutated state, and how the mutated cells affect neighboring cells and gradually spread to finally form the defect morphology. The multi-source correlation data covers the matching degree values ​​of visible light and infrared cell states in each process, i.e., the degree of consistency of dual-modal characteristic parameters. The mutation trigger parameters specifically include the characteristic parameter values ​​of the initial mutated cell when the mutation is triggered (such as contour curvature value, pin position deviation value, temperature deviation value, etc.) as well as the process type and specific time stamp of the mutation trigger.

[0078] Step S1414: Perform commonality analysis on multiple sample data of the same defect type, use a clustering algorithm to cluster the cell evolution trajectories of multiple samples, and obtain the standard evolution trajectory of the defect type through clustering; statistically analyze the mean distribution of multi-source matching degree, and determine the standard multi-source association pattern through statistics; analyze the common order of state transition of multiple samples, and form a standard state transition sequence through analysis; calculate the mean and standard deviation of mutation triggering parameters, and set the range of standard mutation triggering parameters based on the calculation results.

[0079] For all sample data of the same defect type (e.g., crack), the cell evolution trajectory of each sample is first extracted. Each trajectory is represented as a multi-dimensional vector containing cell coordinates, state change time, and feature parameter changes. The K-means clustering algorithm is used to cluster these vectors. During clustering, the optimal number of clusters K is determined using the silhouette coefficient method; typically, K is between 3 and 5. The cluster center trajectory is selected as the standard evolution trajectory for this defect type, representing the most typical diffusion path and state change pattern of this type of defect. For multi-source associated data, the multi-source matching degree values ​​of all samples in each process are statistically analyzed, and their mean and standard deviation are calculated. The range of mean ± 1 standard deviation is defined as the standard multi-source association pattern of this defect type in each process, i.e., the normal bimodal feature matching law. The state transition process of multiple samples is analyzed to find the common state change sequence of all samples, such as "normal state—suspected state—variant state—diffusion state," and this common sequence is defined as the standard state transition sequence. For the mutation triggering parameter, the mean and standard deviation of each characteristic parameter of the initial mutant cell of all samples are calculated, and the standard mutation triggering parameter range is set to mean ± 2 times standard deviation to cover the initial triggering situation of the vast majority of samples.

[0080] Step S1415: Create a standard rule entry for each defect type. The entry includes the defect type name, standard cell evolution trajectory, standard multi-source association pattern, standard state transition sequence, standard mutation triggering parameters, typical characteristic parameter range, and defect description. The defect description includes the defect formation mechanism, common occurrence processes, and degree of harm.

[0081] Create a standard rule entry for each defect type, such as cracks, missing corners, and pin deformation. Each entry records in detail: defect type name (e.g., "encapsulation bubble"); standard cell evolution trajectory (i.e., the center trajectory data obtained from clustering in step S1414); standard multi-source association pattern (the mean distribution range of multi-source matching degree for each process); standard state transition sequence (e.g., "normal state - suspected state - variant state"); standard mutation trigger parameter range (mean of each characteristic parameter ± 2 times the standard deviation); typical characteristic parameter range (the typical value range of characteristic parameters of this defect in the mature stage, such as the interface fit of encapsulation bubbles is usually <0.3, and the temperature deviation is >15℃); the defect description section elaborates on the formation mechanism of the defect (e.g., encapsulation bubbles are formed due to air mixed in with the encapsulation adhesive or volatiles not being discharged in time during the curing process, accumulating at the interface), common occurrence processes (e.g., encapsulation adhesive curing process), and the degree of harm (e.g., slight bubbles may affect heat dissipation, while severe bubbles lead to a decrease in encapsulation strength or even chip failure).

[0082] Step S1416: Construct a rule base index system. The index items of the rule base index system include defect type, occurrence process, core feature parameter category, and mutation trigger parameter category. Integrate all standard rule entries with the rule base index system to form a preset defect evolution rule base.

[0083] Design a rule base index system to enable rapid querying and matching of defect rules. Index items mainly include: defect type (e.g., cracks, pin deformation, etc., as primary indexes); occurrence process (e.g., soldering, encapsulation adhesive curing, etc., as secondary indexes, associated with the defect type); core feature parameter categories (e.g., contour features, temperature features, etc., indicating which features primarily manifest the defect); and variation triggering parameter categories (e.g., temperature gradient parameters, pin arrangement deviation parameters, etc., indicating which parameters primarily trigger the defect). All standard rule entries created in step S1415 are classified and labeled according to the above index items. Then, these entries are integrated with the index system and stored in a relational database to form a pre-defined defect evolution rule base. Each rule entry can be quickly retrieved using index items, supporting multi-dimensional queries by defect type, occurrence process, etc.

[0084] Step S142: Traverse all mutant cells in the cellular automaton evolution model, extract the process type, feature parameters and triggering conditions when each mutant cell is triggered by mutation, compare the extracted process type, feature parameters and triggering conditions with the standard mutation triggering parameters of each defect in the defect evolution rule base, and calculate the parameter matching rate; when the parameter matching rate is higher than the preset threshold, mark the mutant cell as a suspected defect cell and record the candidate defect type corresponding to the suspected defect cell.

[0085] Traverse all nodes of the cellular automaton model and filter cells in the mutant state. For each mutant cell, extract the process type (HX / FG / YZ), characteristic parameter value (e.g., pin position deviation = 55μm), and trigger condition (whether the joint trigger parameters are met) at the time of mutation triggering. Compare with the standard trigger parameters of each defect in the rule base and calculate the ratio of the number of matching parameters to the total number of parameters (parameter matching rate). The preset parameter matching rate threshold is 0.8. When the parameter matching rate of a certain defect type is ≥0.8, the cell is marked as a suspected defect cell, and the corresponding candidate defect type ID and name are recorded.

[0086] Step S143: For each suspected defective cell, extract the cell evolution trajectory of the suspected defective cell in the cellular automata evolution model. The cell evolution trajectory includes the state changes of all associated cells, process time, neighbor state influence and source matching degree.

[0087] The evolution trajectory of suspected defective cells is realized by tracing back the temporal sequence of cell states. Starting from the current mutated cell, it traces backward to the initial normal state, recording the state change at each step, the state of neighboring cells that triggered the change, and the bimodal matching degree. The trajectory data is in list format, with each element containing the step time stamp, cell state, list of neighboring cells influencing the change, and source matching degree value. Related cells include direct neighboring cells and non-neighboring related cells with a comprehensive similarity ≥ 0.7.

[0088] Step S144: Compare the extracted cell evolution trajectory with the standard cell evolution trajectory of the corresponding candidate defect type, and calculate the trajectory overlap, state transition sequence matching rate, and neighbor influence rule fit. The trajectory overlap is the proportion of cell nodes that overlap between the actual trajectory and the standard trajectory. The state transition sequence matching rate is the degree of consistency between the actual and standard state change order. The neighbor influence rule fit is the consistency between the influence of the neighbor states on cell evolution in the actual and standard trajectories.

[0089] The trajectory overlap rate is calculated as the proportion of cells in the actual evolution trajectory that overlap with the standard trajectory out of the total number of nodes in the standard trajectory. The state transition sequence matching rate is calculated using the Dynamic Time Warping (DTW) algorithm to determine the similarity between the actual and standard state sequences; a smaller DTW distance results in a higher matching rate. The neighbor influence rule consistency analysis examines the consistency between the actual neighbor state changes and the standard rule, such as the standard rule that "variant cells preferentially influence four neighboring cells," and counts the proportion of actual trajectories that conform to this rule. All three indicators are normalized to the [0, 1] interval, with weights of 0.4, 0.3, and 0.3, respectively.

[0090] Step S145: Extract multi-source association data from the source image group of the process where the suspected defective cell is located. The multi-source association data includes the state matching degree of multi-source cells, the degree of coincidence of feature parameters, and the overlap range of abnormal areas. Compare the extracted multi-source association data with the standard multi-source association pattern of the corresponding candidate defect type, and calculate the average source matching degree and the overlap rate of abnormal areas. When multiple sources have abnormalities and the average matching degree is higher than the threshold, mark it as a source verification passed. When there is only a single source abnormality or the matching degree is insufficient, retrieve the supplementary data of the cell to be confirmed in the cellular automata evolution model, and perform the verification again using the supplementary data.

[0091] Multi-source correlation data is extracted from the metadata of the source image group of the process, including the bimodal matching degree sequence, the feature parameter correlation coefficient matrix, and the coordinates of the bounding rectangle of the abnormal region. When comparing with the standard multi-source correlation pattern, the mean of the bimodal matching degree is calculated, and the overlap rate of the abnormal region is achieved by calculating the intersection-over-union (IoU) ratio between the actual and standard abnormal regions. The source matching degree mean threshold is set to 0.6, and the IoU threshold is set to 0.5; both are satisfied when the source verification passes. If it fails, supplementary feature parameter data of the cell to be verified is retrieved, the correlation coefficient matrix is ​​recalculated, and the verification is performed again.

[0092] Step S146: Track the evolution and propagation path of suspected defective cells in subsequent processes, and check whether the cell state changes in the evolution and propagation path conform to the standard state transition sequence of the corresponding candidate defect type; calculate the consistency coefficient of cross-process evolution by comprehensively calculating the degree of conformity between the state changes of each process and the standard sequence; when the consistency coefficient is higher than the consistency threshold, confirm the defect type; when the consistency coefficient is not higher than the consistency threshold, change the candidate defect type and re-execute the verification using the new candidate defect type.

[0093] The evolutionary propagation path is tracked through the iterative process of a cellular automata model, recording the state changes of suspected defective cells in subsequent processes and their impact on neighboring cells. Standard state transition sequences are retrieved from a defect evolution rule base; for example, the standard sequence for crack defects is "normal state—suspected state—variant state—diffusion state". The consistency coefficient of cross-process evolution is calculated by comparing the duration and transition order of each state in the actual sequence with the standard sequence, using a weighted summation method, with the weight increasing as the process progresses. The consistency threshold is set to 0.75; if it is higher than this value, the defect type is confirmed; otherwise, a different candidate type is tried for re-verification, with a maximum of three candidate types attempted.

[0094] Step S147: Integrate the parameter matching rate, trajectory overlap, source matching mean, and consistency coefficient. After normalizing each indicator to the same dimension, calculate the comprehensive verification score by weighting according to the set weights. When the comprehensive verification score is higher than the preset qualified threshold, generate the cell verification result, which includes the defect type, comprehensive verification score, suspected defect cell information, and evolution trajectory. When the comprehensive verification score is lower than the preset qualified threshold, mark the defect as an unknown defect and record the cell evolution data of the unknown defect for updating the defect evolution rule base.

[0095] Each indicator was normalized using the min-max method, mapped to the [0, 1] interval. The parameter matching rate was weighted at 0.25, trajectory overlap at 0.25, the mean source matching degree at 0.2, and the consistency coefficient at 0.3. These weighted sums yielded the comprehensive verification score. A pass threshold of 0.7 was set. Values ​​above this threshold generated cell verification results, including the defect type name, comprehensive score, grid coordinates of suspected defective cells, and evolution trajectory data. Values ​​below the threshold stored the cell identifiers, feature parameters, and evolution trajectories of unknown defects in the pending rule base folder for subsequent manual analysis and rule base updates.

[0096] Step S150: Analyze the cell verification results, reconstruct the evolution path and source association chain of the defective cells, determine the defect type, occurrence process and location, and output the chip packaging defect diagnosis results containing defective cell information, evolution path, type and location.

[0097] Step S151: Extract the defect type, comprehensive verification score, suspected defect cell information and evolution trajectory from the cell verification results, and retrieve the corresponding standard cell evolution trajectory, standard multi-source association pattern and standard feature parameter range from the defect evolution rule base according to the defect type.

[0098] The defect type code is parsed from the cell verification results, and the standard cell evolution trajectory (including standard grid coordinate sequence and state transition time interval), standard multi-source correlation mode (correspondence between visible light and infrared features), and standard feature parameter range (normal threshold and abnormal threshold of each parameter) are obtained by querying the defect evolution rule base. For example, the standard feature parameter range for crack defects includes profile curvature change rate > 0.2 / mm, temperature gradient > 8℃ / mm, etc.

[0099] Step S152: Extract complete evolution data of suspected defective cells from the cellular automata evolution model, including cell state, feature parameters, neighbor state, source matching degree and state transition reason at each process time; sort by process time stamp and reconstruct the time evolution path of defective cells, in which the cell state change and triggering factors of each process are marked.

[0100] By identifying suspected defective cells, the corresponding nodes in the cellular automaton model are located, and the state records of these nodes at all process times are extracted, including state values, feature parameter vectors, neighbor state matrices, bimodal matching degree, and state transition reasons (such as "neighbor variant state triggering" or "joint parameter triggering"). These are arranged in ascending order by process timestamp to form a time evolution path. Each node in the path is labeled with the process name, state value, and triggering factor (e.g., in process HX at 10:30:25, the state changes from normal to suspected state, and the triggering factor is "two variant cells in the four neighboring regions").

[0101] Step S153: Trace the first mutant cell in the time evolution path, extract the process time stamp of the first mutant cell, take the process corresponding to the process time stamp as the initial process of the defect occurrence, and record the equipment parameters, source identifier and environmental parameters of the initial process of the defect occurrence; when there are multiple initial mutant cells, determine the earliest cell that triggers the mutation based on the neighbor relationship and evolution propagation direction analysis, and determine the cell as the root cell.

[0102] In the time evolution path, find the record where the state first changes to a variant state, extract its process timestamp, and parse it to obtain the corresponding process type (e.g., "HX" corresponds to the soldering process). Retrieve the equipment operating parameters (e.g., soldering temperature, conveyor belt speed), source identifier (VIS / IR), and environmental parameters (workshop temperature, humidity) at that moment from the equipment parameter database. If there are multiple initial variant cells at the same moment in the path, determine the direction of variant state propagation (e.g., from left to right) by analyzing their neighbor relationships. The earliest appearing cell is identified as the root cell, and its grid coordinates are used as the defect starting position.

[0103] For example, step S1531: Extract the process time stamps of all variant cells from the time evolution path of the reconstructed defect cells, sort the extracted process time stamps in ascending order, and obtain the variant cell time series list by sorting; the variant cell time series list contains the cell identifier, process time, state transition reason and characteristic parameters of each variant cell.

[0104] Traverse the temporal evolution path of defective cells (which records the cell's state change history in chronological order of process timestamps) and filter out all cell records in the mutated state. Extract the process timestamp of each mutated cell from these records and sort the timestamps in ascending order of time. Based on the sorted timestamps, construct a mutated cell time series table. Each record in this mutated cell time series table contains the following information: the unique cell identifier of the mutated cell (e.g., HX202405201030_VK_0305_10); the specific process timestamp at which the cell entered the mutated state; the reason for the state transition to the mutated state (e.g., "joint trigger parameters satisfied" or "influence of neighboring mutated cells"); and all characteristic parameter values ​​at the time of entering the mutated state (e.g., profile curvature, pin spacing, temperature, etc.).

[0105] Step S1532: Select the variant cell with the smallest process time stamp from the variant cell time series table, and use this variant cell as a candidate initial variant cell. Extract the cell identifier, grid coordinates, feature parameters, neighbor status and source matching degree of the candidate initial variant cell. Retrieve the original image data and cell encoding data of the process in which the candidate initial variant cell is located, and use the retrieved original image data and cell encoding data to verify the validity of the variant state of the candidate initial variant cell.

[0106] In the time series table of variant cells, the record with the smallest process time stamp value is located and selected. The corresponding variant cell is the candidate initial variant cell, which is the earliest variant cell to appear in the entire time evolution path. From this record, the cell identifier, its row and column coordinates in the cell grid (obtained by grid coordinate encoding parsing), the feature parameters when entering the variant state, the state of its neighboring cells at that time (the number and distribution of normal, suspected, and variant states), and the bimodal source matching degree are extracted. Subsequently, based on the process time stamp and source identifier in the cell identifier, the original visible light and infrared image data of the process in which the candidate initial variant cell is located are retrieved from the database, as well as the corresponding defect feature cell encoding data (including feature extraction data before encoding, cell splitting process data, etc.). The validity of the candidate initial variant cell variant state determination is verified by comparing the actual features of the region corresponding to the cell in the original image (such as whether there are obvious signs of physical defects) with the feature parameters in the cell encoding data. For example, if the region in the original image does have severe pin deformation and the pin position deviation parameter in the cell encoding is far beyond the normal threshold, then the verification is successful.

[0107] Step S1533: Check whether the state transition reason of the candidate initial mutant cell conforms to the set state transition logic, check whether the multi-source matching degree of the candidate initial mutant cell is higher than the matching degree threshold, and confirm that the candidate initial mutant cell is a valid state cell through checking and verification; extract the feature parameters of the candidate initial mutant cell, compare the extracted feature parameters with the standard mutation trigger parameters of the corresponding defect type, and confirm whether the feature parameters are within the standard range.

[0108] First, review the state transition logic (first, second, and third state transition logic) set in step S133, and check whether the state transition reason of the candidate initial mutant cell (such as "joint trigger parameters are satisfied") meets the execution conditions of one of the transition logics. For example, if the transition reason is joint triggering, it is necessary to confirm whether it meets the joint triggering parameter conditions of the corresponding process (such as the temperature gradient and pin arrangement deviation of the soldering process both deviating from the normal range). At the same time, check whether the multi-source matching degree (dual-modal feature parameter consistency) of the candidate initial mutant cell is higher than the preset matching degree threshold (such as 0.7). If it is higher, it indicates that the dual-modal feature indicates the same mutation state. If all the above checks and verifications are met, the candidate initial mutant cell is confirmed as a valid state cell. Next, extract the feature parameters of the valid state cell and compare them one by one with the standard mutation triggering parameter range of the corresponding defect type retrieved from the defect evolution rule base in step S151 to confirm whether each feature parameter falls within the standard mutation triggering parameter range, thereby further verifying its rationality as an initial mutant cell.

[0109] Step S1534: When the candidate initial mutant cell passes the verification, extract the process timestamp of the candidate initial mutant cell, take the process corresponding to the process timestamp as the initial process of the defect occurrence, and record the name, equipment parameters, environmental parameters and production operator information of the initial process of the defect occurrence; retrieve the production log of the initial process of the defect occurrence, and extract the process parameter details when the defect occurred from the production log. The process parameter details include solder temperature, encapsulation adhesive viscosity and pin shaping pressure.

[0110] Once the candidate initial mutant cell passes all validations, the process timestamp is extracted from its cell identifier or corresponding record. Based on the process code in the process timestamp (e.g., HX corresponds to the solder melting process, FG to the encapsulation adhesive curing process), the specific process name corresponding to that timestamp is determined; this process is the initial process where the defect occurred. The database is consulted to retrieve and record the equipment parameters (e.g., solder oven model and serial number), environmental parameters (workshop temperature and humidity), and the production operator information (employee ID, with privacy protection measures applied, such as only recording the first and last two digits, with * in the middle). Simultaneously, based on the process timestamp and equipment information, the production log of the initial process where the defect occurred is retrieved. Detailed process parameters at the time of the defect are extracted from the log, such as the solder temperature setpoints and actual measurements for each zone in the soldering process, and the conveyor belt speed; the encapsulation adhesive viscosity, coating amount, and temperature curves for each zone of the curing oven in the encapsulation adhesive curing process; and key process parameters such as the shaping pressure setpoint, mold opening and closing speed, and holding time in the lead shaping process.

[0111] Step S1535: If the candidate initial mutant cell fails the verification, select the mutant cell corresponding to the next process time stamp from the mutant cell time sequence table, use the mutant cell as the new candidate initial mutant cell, and repeat the verification process for the new candidate initial mutant cell.

[0112] If a candidate initial mutant cell fails validation (e.g., the state transition reason is illogical, the multi-source matching degree is below the threshold, or the feature parameters are not within the standard mutation triggering range), then the mutant cell with the second smallest process time stamp is selected from the mutant cell time series table and used as a new candidate initial mutant cell. Then, the validation process from steps S1532 to S1534 is repeated: relevant information of the new candidate cell is extracted, the original data is retrieved to verify its mutant validity, the state transition reason and multi-source matching degree are checked, and the feature parameters are compared with the standard mutation triggering parameter range. This process is repeated until a candidate initial mutant cell that passes all validations is found.

[0113] Step S1536: When there are multiple candidate initial mutant cells, calculate the matching degree between each characteristic parameter of each candidate initial mutant cell and the corresponding standard mutation triggering parameter, and calculate the comprehensive matching degree by weighting each matching degree. Select the candidate initial mutant cell with the highest comprehensive matching degree as the root cell. When multiple candidate initial mutant cells have the same matching degree, analyze the evolutionary propagation direction between the candidate initial mutant cells, and determine the cell that first triggers the mutation of other cells through analysis. Select this cell as the root cell.

[0114] When multiple candidate initial mutant cells with the same or very close process timestamps (e.g., within the same process cycle) exist in the mutant cell time sequence table, and all have passed the aforementioned verification process, it is necessary to further determine the root cell. First, for each candidate initial mutant cell, the matching degree between its characteristic parameters and the corresponding standard mutation triggering parameters is calculated. The matching degree calculation method is as follows: for each characteristic parameter, if the actual value is within the standard range, the matching degree is 1; if it exceeds the range, a matching degree value between 0 and 1 is calculated based on the degree of exceeding (exceeding proportion) (the greater the exceedance, the lower the matching degree). Then, based on the importance of each characteristic parameter in mutation triggering (e.g., the weight of the joint triggering parameter is higher than other parameters), a corresponding weight is assigned to the matching degree of each characteristic parameter, and the weighted sum is used to obtain the comprehensive matching degree of the candidate cell. The candidate initial mutant cell with the highest comprehensive matching degree is selected as the root cell. If the comprehensive matching degrees of multiple candidate cells are the same or very close (difference < 0.05), the evolutionary propagation direction between these candidate cells is analyzed. By examining their cellular evolution trajectories in subsequent processes, we can determine which cells appeared first and influenced the state mutations of other candidate cells through their neighbor relationships. The candidate cell that first triggered the mutations of other cells was identified as the root cell.

[0115] Step S1537: Extract all cell state data of the process where the root cell is located, analyze the cell state distribution around the root cell, and count the proportion of normal, suspected, and variant cells; record the diffusion direction of the root cell, which is the direction in which variant cells spread from the root cell to other directions in subsequent processes.

[0116] After identifying the root cell, extract the state data of all cells in that process based on its process timestamp. Focus on analyzing the cell state distribution within a certain range (e.g., a 3×3 or 5×5 grid) around the root cell, counting the number of normal, suspected, and mutated cells within this range, and calculating their proportion of the total number of cells to understand the local state environment at the initial occurrence of the defect. Subsequently, examine the cell evolution trajectory of the root cell in subsequent processes (i.e., processes after the initial one), observing how mutated cells spread from the root cell to its neighboring cells and further cells. Record the main diffusion direction of mutated cells, such as horizontal, vertical, diagonal, or radial diffusion; this diffusion direction reflects the propagation path characteristics of the defect in the chip packaging structure.

[0117] Step S154: Connect the multi-source cell data of the same defect in each process. The multi-source cell data includes the contour, arrangement, and texture feature parameters in the visible light image and the temperature and gradient feature parameters in the infrared image. Mark the matching points and differences of the multi-source features. Record the defect feature changes of each source in the process sequence and form a defect source association chain through the recording.

[0118] Multi-source cell data corresponding to the same defect are extracted according to the process sequence (soldering—encapsulant curing—lead shaping). Visible light data includes parameters such as contour curvature, lead position deviation, and interface fit of each process, while infrared data includes parameters such as temperature deviation, gradient amplitude, and thermal uniformity. Dual-modal parameters are compared to mark matching points (e.g., visible light contour anomalies and infrared temperature anomalies appear simultaneously) and difference points (e.g., defects are detected by visible light but not by infrared). Changes in multi-source features are recorded as defect source association chains in chronological order. Each node in the chain includes the process name, source type, change in feature parameters, and matching status (match / difference).

[0119] Step S155: Map the grid coordinates of the defect cells to the pixel coordinates of the original image, and calculate the corresponding region of each defect cell in the original image by combining the cell size parameter; when the defect involves multiple cells, merge the corresponding regions of each cell, and obtain the complete defect region by merging, and mark the boundary coordinates, center coordinates and coverage area of ​​the defect.

[0120] The formula for converting the grid coordinates (i, j) of a defect cell to pixel coordinates is: x = j × cell_size + cell_size / 2, y = i × cell_size + cell_size / 2, where cell_size is the cell side length (10 pixels). Each cell corresponds to a rectangular region in the original image of [j × cell_size, (j+1) × cell_size) × [i × cell_size, (i+1) × cell_size). Regions of multiple defect cells are merged using a contour extraction algorithm to obtain the minimum bounding rectangle. The boundary coordinates are the top-left corner (xmin, ymin) and bottom-right corner (xmax, ymax) of the rectangle, and the center coordinates are ((xmin + xmax) / 2, (ymin + ymax) / 2). The covered area is (xmax - xmin) × (ymax - ymin) pixels.

[0121] Step S156: Integrate the defect type, initial process, time evolution path, source association chain, and actual location range to form a chip packaging defect diagnosis result containing defect cell information, evolution path, type, process node, and location, along with a comprehensive verification score and multi-source feature data.

[0122] The diagnostic results are organized in XML format. The root node contains attributes such as defect ID, comprehensive verification score, and diagnosis time. Child nodes include: defect type (name, code, hazard level), occurrence process (process name, timestamp, equipment parameters), evolution path (process sequence, state change list), source association chain (multi-source feature comparison table), and location information (boundary coordinates, center coordinates, coverage area). Attached is multi-source feature data as a Base64 encoded binary stream, containing the original feature parameter matrix and image data.

[0123] Step S210: Extract the defect type, initial process, cell evolution trajectory, multi-source feature data, defect location and comprehensive verification score from the chip packaging defect diagnosis results, and integrate the extracted data to form a diagnosis result data package; add a unique identifier to the diagnosis result data package, the unique identifier including the chip production batch number, chip unique code, diagnosis time and diagnosis equipment number.

[0124] Key data fields were extracted from the chip packaging defect diagnosis result XML file: defect type name, code and timestamp of the initial process, JSON array of cell evolution trajectory, Base64 stream of multi-source feature data, boundary coordinates of the defect location, and comprehensive verification score. This data was packaged into a diagnosis result data package in JSON format, with a unique identifier added to the header in the format "batch number-chip ID-diagnosis time-device ID". The batch number is an 8-digit number, the chip ID is a 12-digit alphanumeric combination, the diagnosis time is "YYYYMMDDHHMMSS", and the device ID is a 6-digit device number.

[0125] Step S220: Associate the diagnostic result data package with the chip's manufacturing process data, which includes equipment parameters, process parameters, operator information, environmental parameters, and quality inspection records for each process. Analyze the correlation between defect types and manufacturing process data. For example, pin deformation defects are related to the pressure parameters of the pin shaping process, and encapsulation bubbles are related to the curing temperature of the encapsulation adhesive.

[0126] The corresponding production process data is retrieved from the MES system database using the chip's unique code. This includes equipment operating parameters for each process (such as the temperature of each zone in the soldering furnace and the pressure of the forming mold), process parameters (such as the viscosity of the encapsulating adhesive and the pin spacing setting), operator ID, environmental parameters (temperature, humidity, cleanliness), and quality inspection records from previous processes (such as AOI inspection results). Association rule mining algorithms (such as Apriori) are used to analyze the correlation between defect types and process parameters. With a minimum support of 0.2 and a minimum confidence of 0.7, association rules such as "pin deformation—forming pressure > 300N" and "encapsulation bubble—curing temperature > 150℃" are identified.

[0127] Step S230: Retrieve the physical inspection results of the defective chip, compare the defect type and location in the diagnostic results with the physical inspection results, and check the consistency between the two; calculate the diagnostic accuracy rate, which is the proportion of diagnostic cases in which the diagnostic results and physical inspection results are consistent in both defect type and location to the total number of cases; when the diagnostic results and physical inspection results are inconsistent, record the differences and reasons, including cell encoding errors and inaccurate defect evolution rule base parameters.

[0128] The physical inspection report of the chip is retrieved from the quality inspection system. This report includes information such as the defect type, location coordinates, and defect size identified during manual re-inspection. The defect type in the diagnostic results is compared with the type detected in the physical chip. The location is compared using the Euclidean distance calculated from the center coordinates (a distance < 50 pixels is considered consistent). The diagnostic accuracy is calculated using the formula: "Accuracy = Number of consistent cases / Total number of cases × 100%". Discrepancies are recorded, such as "diagnosed as pin deformation, actual chip has a missing corner." The cause analysis is determined by checking the cell encoding process (e.g., errors in feature parameter extraction) and rule base parameters (e.g., excessively high threshold settings).

[0129] Step S240: Adjust system parameters based on verification results and correlation analysis: When the diagnostic accuracy is lower than the preset standard, adjust the parameter range of defect feature cell encoding, the state transition threshold of the cellular automata evolution model, or the standard parameters of the defect evolution rule base; when a new defect type or a defect that cannot be matched by existing rules is detected, organize the diagnostic data of the defect into new rule entries and add the new rule entries to the defect evolution rule base.

[0130] The preset diagnostic accuracy standard is 95%. When the actual accuracy falls below this value, the system parameter adjustment process is initiated. If the error is due to cell encoding error, the normalization range of the feature parameters is recalibrated (e.g., adjusting the temperature deviation threshold from ±10℃ to ±8℃). If the error is due to an unreasonable state transition threshold, the neighbor ratio threshold in the state transition parameter table is modified (e.g., from 0.3 to 0.25). If the error is due to rule base parameter issues, the standard trigger parameters are updated (e.g., reducing the pin deformation position deviation threshold from 40μm to 35μm). For new defect types, engineers analyze their cell evolution trajectory and feature parameters, manually create new rule entries, including standard trajectories, association patterns, trigger parameters, etc., and import them into the defect evolution rule base through the rule base management tool.

[0131] Step S250: Classify and statistically analyze the diagnostic results according to defect type and occurrence process, and generate a production quality analysis report through classification and statistics; the production quality analysis report includes the occurrence frequency of each defect type, the defect occurrence rate of each process, the causes of major defects and improvement suggestions; for defects caused by abnormal temperature gradient in the soldering process, it is recommended to adjust the temperature distribution parameters of the soldering furnace; for bubbles caused by abnormal curing temperature of the encapsulating glue, it is recommended to optimize the temperature control curve of the curing furnace.

[0132] The diagnostic results are categorized and statistically analyzed monthly. The frequency and occurrence of defects are calculated by type (frequency = number of occurrences / total number of inspections), and the defect incidence rate is calculated by process (incidence rate = number of defects in the process / total output of the process). Pareto charts are used to analyze the main defect types, focusing on the top 20% of high-frequency defects. Combined with correlation analysis results, the causes of major defects are determined, such as "abnormal solder temperature gradient" and "excessively fast curing rate of encapsulating adhesive." Improvement suggestions are based on process parameter optimization, such as adjusting the solder oven temperature distribution parameters to "zone 1 230℃, zone 2 225℃, zone 3 220℃," and optimizing the curing oven temperature control curve to "heating rate 5℃ / min, 130℃ holding time 30min."

[0133] Step S260: Store the optimized system parameters, the updated defect evolution rule base, and the production quality analysis report into the database.

[0134] Optimized system parameters (such as feature parameter thresholds and state transition thresholds) are stored in the `system_params` table of the configuration database, categorized by parameter ID, parameter name, parameter value, and update time. Updated defect evolution rule bases are written to the rule base table in batches via database transactions, including newly added rule entries and modified parameter values. Production quality analysis reports are generated as PDF files and stored on a file server. The database records metadata such as the report's storage path, generation time, and report type, indexed by chip batch number.

[0135] During data acquisition, sensitive data involving operator information is handled using a Role-Based Access Control (RBAC) policy. Only authorized administrators can view the complete information, while access by ordinary users is automatically anonymized (e.g., the employee ID is replaced with "***"). Image data is encrypted using the TLS 1.3 protocol during transmission and the original image files are encrypted using the AES-256 encryption algorithm during storage. The key is managed through a Hardware Security Module (HSM). Cellular encoded data is encrypted at the field level in the database. Sensitive parameters (such as device passwords) are stored as encrypted ciphertext, and the decryption key is generated using a dynamic password.

[0136] Furthermore, Figure 2 A schematic diagram of the hardware structure of a computer vision-based chip packaging defect diagnosis system 100 for implementing the methods provided in the embodiments of this application is shown. Figure 2 As shown, the computer vision-based chip packaging defect diagnosis system 100 may include at least one processor 102 (the processor 102 may be, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, a transmission device 106 for communication functions, and a controller 108. Those skilled in the art will understand that... Figure 2The structure shown is for illustrative purposes only and does not limit the structure of the computer vision-based chip packaging defect diagnosis system 100. For example, the computer vision-based chip packaging defect diagnosis system 100 may also include components that are more complex than those shown in the diagram. Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0137] The memory 104 can be used to store software programs and modules for application software, such as the program instructions corresponding to the method embodiments described above in this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-described method for diagnosing chip packaging defects based on computer vision. The transmission device 106 is used to acquire or send data via a network.

[0138] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

Claims

1. A computer vision-based chip package defect diagnosis method, characterized by, The method comprises: Collecting chip package multi-source evolution image data, which contains dynamic images of key processes of chip package in solder melting, package glue curing, and pin shaping, and multi-source image sequences of infrared thermal imaging and visible light imaging of each process, the multi-source image sequences being attached with process time stamps, source identifiers, and equipment parameters; Performing defect feature cell coding processing on the chip package multi-source evolution image data, splitting features into traceable cell units and coding to obtain a defect feature cell coding set; Constructing a cellular automaton evolution model based on the defect feature cell coding set, the cellular automaton evolution model containing inter-process transfer rules of cell states, source correlation rules, and variation trigger conditions; Performing multi-source correlation verification processing on the cellular automaton evolution model, matching cell evolution trajectories with a defect evolution rule library to generate a cell verification result; Analyzing the cell verification result, reconstructing the evolution path and source correlation chain of the defect cell, determining the defect type, occurrence process, and location, and outputting a chip package defect diagnosis result containing defect cell information, evolution path, type, and location. 2.The computer vision-based chip package defect diagnosis method of claim 1, wherein, The defect feature cell coding processing on the chip package multi-source evolution image data, splitting features into traceable cell units and coding to obtain a defect feature cell coding set comprises: Classifying the chip package multi-source evolution image data according to process time stamps, grouping infrared thermal imaging and visible light imaging of the same process time together to obtain process source image groups, each process source image group being attached with a process time stamp, a source identifier, and equipment parameters; Performing cooperative feature extraction on the multi-source images in each process source image group to obtain cooperative extraction features, including the following: visible light images extract contour features, pin arrangement features, package interface fitting features, and material texture features; infrared thermal imaging extracts temperature distribution features, temperature gradient features, and thermal conductivity uniformity features; contour features are scanned by an edge tracking algorithm to scan the image boundary, record the gray scale mutation pixel coordinates, and fit them into a closed curve to extract the curvature change, line segment length, and corner angle; pin arrangement features are identified by connected region segmentation to identify individual pins, calculate the geometric center coordinates of each pin, the distance between pins, and the angle between adjacent pins; package interface fitting features are scanned by interface region gray scale to record the gray scale mutation range, mutation amplitude, and continuous segment length; material texture features are calculated by gray level co-occurrence matrix to calculate the texture density, uniformity, and direction consistency; temperature distribution features are calculated by thermal imaging pixel temperature value statistics to obtain a temperature distribution histogram; temperature gradient features are calculated by adjacent pixel temperature difference to obtain a gradient distribution matrix; thermal conductivity uniformity features are calculated by temperature standard deviation and coefficient of variation; The cooperative extraction features are subjected to cell splitting, and the chip packaging image is divided into cell grids of equal size, each cell grid corresponds to a cell unit, the extracted features are distributed to each cell unit according to the cell grid range, and each cell unit contains all source feature parameters of the corresponding region; the feature parameters of the cell unit are subjected to state definition, and the profile integrity, pin position deviation, interface fit, texture uniformity, temperature deviation and gradient change rate parameters are mapped to the cell state, and the state is divided into normal state, suspected state and variation state; A unique cell identifier is assigned to each cell unit, which includes process code, source code, grid coordinate code and state code, the process code corresponds to the process timestamp, the source code distinguishes between infrared and visible light, the grid coordinate code corresponds to the row and column position of the cell in the grid, and the state code corresponds to the normal state, suspected state and variation state; State code conversion is performed on each cell unit to convert the cell state and corresponding feature parameters into fixed-length binary codes, the front part is the state code, the middle part is the visible light feature parameter code, and the rear part is the infrared feature parameter code; the visible light feature parameter code and the infrared feature parameter code are respectively converted after the respective feature parameters are normalized; the check code of the coded binary value is calculated and added to the end of the code as a check bit; The cell codes under the same source are connected in sequence according to the process timestamp to form a process source code sequence; the mapping table of cell identifier and code and the state parameter mapping rule are integrated to obtain a defect feature cell code set. 3.The computer vision-based chip package defect diagnosis method of claim 1, wherein, The defect feature cell code set is used to construct a cellular automaton evolution model, which includes: The process source code sequence in the defect feature cell code set is decoded, and the process time, source type, grid coordinate, state and feature parameter of each cell unit are restored through the mapping table to obtain a cell state data set, and the cell state data set is sorted according to the process timestamp to form a cell state time sequence; The four-neighbor neighbors and eight-neighbor neighbors of each cell unit are defined, and the neighbor list of each cell and the distance parameter between the neighbors are recorded; the similarity of the same type of feature parameters is calculated based on the multi-source feature parameters, and the comprehensive similarity is obtained by weighted fusion of each similarity; the cell with a comprehensive similarity higher than a preset threshold is marked as a related cell and supplemented to the neighbor list; The cell state change data of the normal sample and the defective sample is analyzed to determine the basic threshold of the neighbor state quantity ratio, the characteristic parameter deviation range, and the time sequence duration length; the execution condition for the conversion of the normal state cell to the suspected state is constructed based on the basic threshold, and the execution condition is that when the cell satisfies the condition that the number of the variant state cells in the neighbors reaches the set ratio and the characteristic parameter of the cell deviates from the normal range, the conversion operation from the normal state to the suspected state is performed; the execution condition for the conversion of the suspected state cell to the variant state is constructed, and the execution condition is that when the suspected state cell satisfies the condition that the characteristic parameter continuously deviates in the subsequent process and the infrared temperature parameter is abnormal, the conversion operation from the suspected state to the variant state is performed; the execution condition for the conversion of the variant state cell to the suspected state is constructed, and the execution condition is that when the variant state cell satisfies the condition that the characteristic parameter returns to normal and all the neighbor cells are in the normal state in the subsequent process, the conversion operation from the variant state to the suspected state is performed; the characteristic parameter threshold, the neighbor state quantity ratio, and the time sequence duration length corresponding to the three conversion execution conditions are recorded to form a state conversion parameter table; The cell grid coordinate mapping relationship between the visible light image and the infrared image at the same process time is established, so that the cells at the same physical position in the dual-mode images correspond to the same grid coordinates; when the suspected state or the variant state cell appears in the visible light image is detected, the cell at the corresponding grid coordinate in the infrared image is located according to the coordinate mapping relationship; whether the located infrared cell has temperature abnormality is checked, and if there is temperature abnormality, the dual-mode abnormality matching is completed, and the suspected state or the variant state cell in the visible light is marked as an effective state cell; if only the abnormal cell in the single-mode image is detected, and no abnormality is detected in the other mode corresponding coordinate, the abnormal cell is marked as a to-be-confirmed cell; the degree of agreement of the dual-mode characteristic parameters of the effective state cell is calculated, and the calculation result is taken as the dual-mode cell state matching degree and is recorded; The initial characteristic parameters of the cells triggering variation in the historical defective sample are extracted, and are classified and arranged according to the soldering, encapsulation glue curing, and pin shaping processes; for the soldering process, the characteristic parameters deviating from the normal range at the same time in the sample triggering variation are analyzed, the temperature gradient characteristic parameter and the pin arrangement deviation parameter are determined as the joint triggering parameters, and the variation triggering condition of the soldering process is set as the condition that both the parameters deviate from the normal range; for the encapsulation glue curing process, the temperature distribution dispersion degree parameter and the interface adhesion degree parameter are determined as the joint triggering parameters, and the variation triggering condition of the encapsulation glue curing process is set as the condition that both the parameters deviate from the normal range; for the pin shaping process, the pin position deviation parameter and the contour curvature parameter are determined as the joint triggering parameters, and the variation triggering condition of the pin shaping process is set as the condition that both the parameters deviate from the normal range; the variation triggering conditions of the processes are stored in association with the corresponding processes; The cell automaton evolution model is constructed by taking the cell unit as a node, the neighbor relationship as an edge, the state conversion logic and the mode association logic as evolution logic, and the variation triggering logic as an initial triggering mechanism; each node in the cell automaton evolution model is marked with a cell identifier, a state, characteristic parameters, and a process time, each edge is marked with a neighbor type and a similarity, and the evolution logic is associated with the process type and the mode type. 4.The computer vision-based chip package defect diagnosis method of claim 1, wherein, The multi-source correlation verification processing is performed on the cellular automaton evolution model, a cellular evolution track is matched with a defect evolution rule library, and a cellular verification result is generated, including: A preset defect evolution rule library is called, the defect evolution rule library contains standard data of cracks, missing corners, pin deformation, packaging bubbles, packaging overflow, interface peeling, and thermal deformation defects, each defect corresponds to a standard cellular evolution track, a standard multi-source correlation mode, a standard state transition sequence, and a standard variation trigger parameter; the standard cellular evolution track is a cellular state change path of the defect from triggering to diffusion, the standard multi-source correlation mode is a characteristic matching rule of the defect under multi-source, the standard state transition sequence is a cellular state change order corresponding to the defect, and the standard variation trigger parameter is a characteristic parameter threshold when the defect initially occurs; All variation state cells in the cellular automaton evolution model are traversed, the process type, characteristic parameter, and trigger condition of each variation state cell when the variation is triggered are extracted, the extracted process type, characteristic parameter, and trigger condition are compared with the standard variation trigger parameters of each defect in the defect evolution rule library, and a parameter matching rate is calculated; when the parameter matching rate is higher than a preset threshold, the variation state cell is marked as a suspected defect cell, and a candidate defect type corresponding to the suspected defect cell is recorded; For each suspected defect cell, a cellular evolution track of the suspected defect cell in the cellular automaton evolution model is extracted, and the cellular evolution track includes state changes, process times, neighbor state influences, and source matching degrees of all associated cells; The extracted cellular evolution track is compared with a standard cellular evolution track of the corresponding candidate defect type, a track coincidence degree, a state transition sequence matching rate, and a neighbor influence rule fitting degree are calculated; the track coincidence degree is a cellular node coincidence proportion of the actual track and the standard track, the state transition sequence matching rate is an agreement degree of the actual and standard state change sequences, and the neighbor influence rule fitting degree is an influence consistency of the actual and standard neighbor states on the cellular evolution; Multi-source correlation data of a process source image group in which the suspected defect cell is located are extracted, and the multi-source correlation data include multi-source cellular state matching degrees, characteristic parameter agreement degrees, and abnormal region overlapping ranges; The extracted multi-source correlation data are compared with a standard multi-source correlation mode of the corresponding candidate defect type, a source matching degree average and an abnormal region overlapping rate are calculated; when all the sources are abnormal and the source matching degree average is higher than a threshold, the source verification is marked as passed; when only a single source is abnormal or the matching degree is insufficient, supplementary data of the to-be-confirmed cell in the cellular automaton evolution model are re-called, and the verification is performed again using the supplementary data; An evolution propagation path of the suspected defect cell in subsequent processes is tracked, and it is checked whether the cellular state changes in the evolution propagation path conform to a standard state transition sequence of the corresponding candidate defect type; a consistency coefficient of cross-process evolution is calculated by comprehensively calculating the agreement degrees of the state changes of each process and the standard sequence; when the consistency coefficient is higher than a consistency threshold, the defect type is confirmed; when the consistency coefficient is not higher than the consistency threshold, the candidate defect type is replaced, and the verification is performed again using the new candidate defect type; The matching rate of integration parameters, the trajectory coincidence degree, the mean of source matching degree, and the consistency coefficient are normalized to the same dimension, and a comprehensive verification score is obtained by weighting calculation according to the set weight; when the comprehensive verification score is higher than a preset qualified threshold, a cell verification result is generated, the cell verification result includes a defect type, a comprehensive verification score, suspected defect cell information, and an evolution trajectory; and when the comprehensive verification score is lower than the preset qualified threshold, the defect is marked as an unknown defect, and cell evolution data of the unknown defect is recorded for updating of a defect evolution rule library. 5.The computer vision-based chip package defect diagnosis method of claim 1, wherein, The parsed cell verification result is used to reconstruct an evolution path and a source association chain of the defect cell, to determine a defect type, a process of occurrence, and a position, and to output a chip package defect diagnosis result including defect cell information, an evolution path, a type, and a position, including: The defect type, the comprehensive verification score, the suspected defect cell information, and the evolution trajectory in the cell verification result are extracted, and a standard cell evolution trajectory, a standard multi-source association mode, and a standard feature parameter range corresponding to the defect type are retrieved from the defect evolution rule library; Complete evolution data of the suspected defect cell is extracted from the cellular automaton evolution model, including cell states, feature parameters, neighbor states, source matching degrees, and state transition reasons at each process time; the time evolution path of the defect cell is reconstructed by sorting according to the process time stamp, and the time evolution path marks the cell state change and the triggering factor of each process; The first variant cell in the time evolution path is traced back, the process time stamp of the first variant cell is extracted, the process corresponding to the process time stamp is taken as the initial process of defect occurrence, and the device parameters, the source identifier, and the environmental parameters of the initial process of defect occurrence are recorded; when there are multiple initial variant cells, the cell that triggers the variation earliest is determined as the root cell according to neighbor relationship and evolution propagation direction analysis; The multi-source cell data of the same defect in each process are connected in series, the multi-source cell data including contour, arrangement, and texture feature parameters in a visible light image and temperature and gradient feature parameters in an infrared image, and the matching points and difference points of multi-source features are marked; the defect feature change of each source is recorded according to the process order, and a defect source association chain is formed by recording; The grid coordinates of the defect cell are mapped to the pixel coordinates of the original image, and the corresponding area of each defect cell in the original image is calculated by combining the size parameters of the cell unit; when the defect involves multiple cells, the corresponding areas of the cells are merged to obtain a complete defect area, and the boundary coordinates, the center coordinates, and the coverage area of the defect are marked; The defect type, the initial process of occurrence, the time evolution path, the source association chain, and the actual position range are integrated to form a chip package defect diagnosis result including defect cell information, an evolution path, a type, a process node, and a position, with the comprehensive verification score and the multi-source feature data. 6.The computer vision-based chip package defect diagnosis method of claim 2, wherein, The cooperative feature extraction is performed on the multi-source images in each process source image group to obtain cooperative extraction features, including: Performing preprocessing on the visible light image in the process source image group: using a Gaussian filter algorithm to remove image noise, and adjusting the image gray scale distribution through histogram equalization, and using an image registration algorithm to align the visible light image with the preset chip packaging standard template to obtain a registered visible light image; Performing contour feature extraction on the registered visible light image: using an edge detection operator to scan the image, setting high and low threshold values to screen edge pixels, and dynamically adjusting the high and low threshold values according to the image gray scale mean value; performing connectivity analysis on the edge pixels, deleting isolated edge points, and connecting adjacent edge pixels to form edge line segments; performing curve fitting on the edge line segments to obtain closed contour curves of the packaging body, pins and interfaces; extracting the geometric parameters of the contour curves, including the curvature values of each segment of the curve, the lengths of the line segments, the corner angles of adjacent line segments, and the circularity and rectangularity of the contour; Performing pin arrangement feature extraction on the registered visible light image: using an adaptive threshold segmentation algorithm to segment the pin region and the background region, wherein the threshold is set according to the gray scale difference between the pin and the background; performing connected region analysis on the pin region to mark the pixel range of each independent pin; calculating the minimum circumscribed rectangle of each pin region to determine the geometric center coordinates of the rectangle as the pin center coordinates; counting the distribution rule of all pin center coordinates, calculating the pixel distance between adjacent pin centers, the included angle between adjacent pins, and the distribution standard deviation of all pin center coordinates; Performing packaging interface fitting feature extraction on the registered visible light image: positioning the interface region between the packaging glue and the chip body, setting scanning lines along the interface direction, and setting the scanning line spacing according to the interface width; collecting the pixel gray scale values on each scanning line and recording the positions and amplitudes of the gray scale mutations; counting the continuous length of the mutation positions and calculating the proportion of the continuous mutation segment in the total length of the interface as the fitting degree; recording the maximum value, minimum value and mean value of the mutation amplitude as the interface fitting quality parameters; Performing preprocessing on the infrared thermal image in the process source image group: using a median filter algorithm to remove thermal imaging noise, and setting the filter window size according to the thermal imaging resolution; converting the thermal imaging pixel value to the actual temperature value through a temperature calibration algorithm, and calling the calibration parameters from the device parameters; aligning the thermal imaging with the registered visible light image to obtain a registered infrared image; Performing temperature feature extraction on the registered infrared image: counting the temperature values of all pixels in the image to obtain a temperature distribution histogram; calculating the temperature difference between adjacent pixels using a gradient operator to obtain a temperature gradient distribution matrix; calculating the standard deviation and coefficient of variation of the temperature values as the thermal conduction uniformity parameters; analyzing the area, center coordinates and temperature mean value of the region with a temperature value higher than the preset normal range as the thermal abnormal region feature. 7.The computer vision-based chip package defect diagnosis method of claim 3, wherein, The triggering logic setting step of the conversion of the normal state cell to the suspected state, the suspected state cell to the variation state, and the variation state cell to the suspected state, comprises: Collect normal sample image data and defect sample image data of each process of the chip packaging, and the sample data covers different equipment parameters and different environmental conditions; perform defect feature cell encoding processing on the collected normal sample image data and defect sample image data respectively, and obtain normal sample cell state set and defect sample cell state set through the encoding processing; Analyze the normal sample cell state set, classify and count the feature parameter range of the normal state of the cell according to the process type and the modal type, determine the normal threshold of different feature parameters according to the statistical result, calculate the similarity of adjacent cells in the normal sample, count the similarity distribution, and determine the similarity threshold of the associated cell according to the distribution; the different feature parameters include profile curvature, pin pitch, interface fit degree and temperature; Analyze the defect sample cell state set, track the cell state change of the defect from the initial formation to the diffusion process in each defect sample, record the conversion time, feature parameter change and corresponding neighbor cell state when the normal state is converted to the suspected state and the suspected state is converted to the variant state in each sample, classify and count the state conversion law according to the defect type, and determine the law that the crack defect shows gradual diffusion of profile curvature abnormal cell and the packaging bubble shows the coordinated appearance of interface fit abnormality and temperature abnormality; A first state conversion logic for converting the normal state cell to the suspected state is constructed: the execution condition is set as that any feature parameter of the cell exceeds the corresponding process normal threshold, there are a set number of suspected state or variant state cells in its four neighbor neighbors, and the parameter deviation state lasts for a set process time; when the cell meets the execution condition, the conversion operation from the normal state to the suspected state is performed; when only the feature parameter exceeds the threshold but all the neighbors are normal state, or the parameter deviation state lasts for less than the set time, the operation of keeping the normal state is performed and the cell is marked as a warning cell; A second state conversion logic for converting the suspected state cell to the variant state is constructed: the execution condition is set as that the suspected state cell has both modal feature parameters exceeding the normal threshold in the subsequent process, the number of variant state and suspected state cells in the eight neighbor neighbors is more than that of normal state cells, the temperature value in the corresponding area of the infrared thermal imaging is higher than the normal threshold and exceeds the set standard; when the execution condition is met, the conversion operation from the suspected state to the variant state is performed; when only the single modal parameter is abnormal or the number of abnormal neighbors is less than the set number, the operation of keeping the suspected state is performed; when both modal feature parameters return to normal and all neighbors are normal state, the operation of converting the suspected state back to the normal state is performed; A third state conversion logic for converting the variant state cell to the suspected state is constructed: the execution condition is set as that the variant state cell has both modal feature parameters returning to the normal threshold range in the subsequent process, the proportion of normal state cells in the eight neighbor neighbors reaches a set proportion, and the state is kept for a continuous set process time; when the execution condition is met, the conversion operation from the variant state to the suspected state is performed; when only the single modal parameter returns to normal or the proportion of normal neighbors is less than the set proportion, the operation of keeping the variant state is performed; when the feature parameter continues to be abnormal and spreads to new neighbor cells, the operation of keeping the variant state is performed and the cell is marked as a diffusion state. The accuracy of the first state transition logic, the second state transition logic and the third state transition logic is tested by selecting a verification sample set, and misjudgment and missed judgment cases in the testing process are recorded; the key parameters are adjusted for the misjudgment and missed judgment cases, and the testing is re-executed; when the accuracy of the transition logic in distinguishing between normal and abnormal state transitions reaches a set standard, the specific parameters of the state transition logic are determined, and the key parameters include a feature parameter threshold and a neighbor number ratio. 8.The computer vision-based chip package defect diagnosis method of claim 4, wherein, The construction process of the preset defect evolution rule library includes: Collecting crack, corner missing, pin deformation, packaging bubble, packaging overflow, interface peeling, and thermal deformation defect physical samples in chip packaging production, collecting sufficient samples of each defect type covering different production scenarios, different equipment parameters, and different environmental conditions, and each sample being attached with detailed production records including process parameters, equipment model, environmental temperature, defect occurrence position, and cause analysis; Performing full-process multi-source image collection on each defect sample, synchronously collecting visible light images and infrared thermal images at the soldering, packaging glue coating, curing, and pin shaping process nodes, and the image shooting parameters being consistent with the collection parameters in actual production; adding process time stamp, source identification, equipment parameters, and sample number to each image; Performing defect feature cell encoding processing and cell automaton evolution model construction steps on the collected sample image data to obtain cell state time sequence, cell evolution trajectory, multi-source correlation data, and variation trigger parameters of each defect sample; the cell evolution trajectory includes all cell state change nodes from the initial trigger to the final formation of the defect, the multi-source correlation data includes the matching degree of the multi-source cell states of each process, and the variation trigger parameters include the feature parameters and trigger process of the initial variation cell; Performing commonality analysis on multiple sample data of the same defect type, clustering the cell evolution trajectories of multiple samples by using a clustering algorithm, obtaining a standard evolution trajectory of the defect type through clustering; statistically distributing the mean value of the multi-source matching degree, determining a standard multi-source correlation mode through statistics; analyzing the common order of state transition of multiple samples, forming a standard state transition sequence through analysis; calculating the mean value and standard deviation of the variation trigger parameters, and setting a standard variation trigger parameter range according to the calculation result; Creating a standard rule entry for each defect type, and the entry includes defect type name, standard cell evolution trajectory, standard multi-source correlation mode, standard state transition sequence, standard variation trigger parameter, typical feature parameter range, and defect description; the defect description includes the formation mechanism of the defect, the common occurrence process, and the damage degree; Building a rule library index system, and the index items of the rule library index system include defect type, occurrence process, core feature parameter category, and variation trigger parameter category; integrating all standard rule entries with the rule library index system, and forming the preset defect evolution rule library through integration. 9.The computer vision-based chip package defect diagnosis method of claim 1, wherein, After the step of analyzing the cell check result, reconstructing the evolution path and source association chain of the defective cell, determining the defect type, occurrence process and location, and outputting the chip packaging defect diagnosis result containing the defect cell information, evolution path, type and location, the method further comprises: Extracting the defect type, occurrence initial process, cell evolution track, multi-source feature data, defect location and comprehensive check score in the chip packaging defect diagnosis result, integrating the extracted data to form a diagnosis result data packet, and adding a unique identifier to the diagnosis result data packet, wherein the unique identifier contains the chip production batch number, chip unique code, diagnosis time and diagnosis equipment number; Associating the diagnosis result data packet with the production process data of the chip, wherein the production process data includes equipment parameters, process parameters, operator information, environmental parameters and quality detection records of each process; analyzing the association between the defect type and the production process data, wherein the pin deformation defect is related to the pressure parameter of the pin shaping process, and the packaging bubble is related to the curing temperature of the packaging glue; Retrieving the physical detection result of the defective chip, comparing the defect type and location in the diagnosis result with the physical detection result, and checking the consistency of the two; calculating the diagnosis accuracy rate, wherein the diagnosis accuracy rate is the proportion of the number of diagnosis cases in which the diagnosis result and the physical detection result are consistent in defect type and location to the total number of cases; when the diagnosis result and the physical detection result are inconsistent, recording the difference points and reasons, wherein the difference points and reasons include cell code error, inaccurate parameters of the defect evolution rule library; Based on the verification result and the association analysis, adjusting the system parameters: when the diagnosis accuracy rate is lower than the preset standard, adjusting the parameter interval of the defect feature cell code, the state transition threshold of the cellular automaton evolution model or the standard parameters of the defect evolution rule library; when a new defect type is detected or the existing rule cannot match the defect, organizing the diagnosis data of the defect into a new rule item, and adding the new rule item to the defect evolution rule library; Classifying and counting the diagnosis results according to the defect type and occurrence process, and generating a production quality analysis report through the classification and counting; the production quality analysis report contains the occurrence frequency of each defect type, the defect occurrence rate of each process, the formation cause of the main defect and the improvement suggestion; for the defects caused by abnormal temperature gradient of the soldering process, it is suggested to adjust the temperature distribution parameters of the soldering furnace; for the bubbles caused by abnormal curing temperature of the packaging glue, it is suggested to optimize the temperature control curve of the curing furnace; Storing the optimized system parameters, updated defect evolution rule library and production quality analysis report to the database.

10. A computer vision-based chip package defect diagnosis system, characterized by, A chip packaging defect diagnosis method based on computer vision is provided, which comprises a processor and a readable storage medium, wherein the readable storage medium stores a program which is executed by the processor to realize the chip packaging defect diagnosis method based on computer vision according to any one of claims 1-9.

Images