An AI agent-based AOI defect automatic rejudgment method and system

Abstract

This invention discloses an AOI (Automated Optical Inspection) defect automatic review method and system based on an AI agent, relating to the field of SMT (Surface Mount Technology) manufacturing technology. This invention automatically acquires defect information and extracts visual features through an AI agent, combining a multimodal large model with an SMT defect knowledge base for multi-step reasoning, significantly improving the efficiency and accuracy of defect review and reducing the false positive and false negative rates of manual review. It automatically executes defect judgment operations, simulating manual clicking and input of notes, reducing the workload of review operators, thereby lowering labor costs and improving the automation level of the production line. Furthermore, by incorporating industry standards such as IPC-A-610, the system ensures the objectivity and consistency of judgment standards, avoiding inconsistencies in judgment criteria caused by differences in experience among different operators, and providing manufacturing enterprises with a powerful quality control tool.

Description

Technical Field

[0001] This invention relates to the field of SMT manufacturing technology, specifically to an automatic AOI defect re-judgment method and system based on AI intelligent agents. Background Technology

[0002] As the Electronic Manufacturing Services (EMS) industry moves towards higher density, higher precision, and miniaturization, Surface Mount Technology (SMT) has become a core process in PCB assembly. In SMT production lines, Automated Optical Inspection (AOI) equipment is widely used to scan PCBs in real time to identify anomalies such as soldering defects, missing components, and reversed polarity. However, due to the complexity of the SMT process and the influence of factors such as lighting, solder paste reflection, and PCB silkscreen interference on optical imaging, AOI equipment generally suffers from a high "over-kill" rate, i.e., a false alarm rate. Industry statistics show that the false alarm rate in the initial alarm of AOI equipment is often as high as 60%-80%, causing a large number of good products to be misjudged as defects.

[0003] To identify genuine defects, the current industry standard process involves setting up a "manual re-verification" station. The re-verification operator needs to review each defect image on the AOI re-verification software, compare it with the confidence level given by the AOI, and conduct a second confirmation in accordance with industry standards such as IPC-A-610. The operator then manually clicks the "OK" or "NG" button and enters remarks.

[0004] However, the existing manual review method has the following significant technical shortcomings: 1. Efficiency bottlenecks and high labor costs: A typical SMT production line may generate thousands of defect alarms per day. Inspectors need to complete image browsing, logical judgment, and mouse clicks within a very short time (usually 3-5 seconds per defect). This high-intensity repetitive labor easily leads to operator visual fatigue, resulting in missed or incorrect judgments. Moreover, with rising labor costs, maintaining a large inspection team has become a heavy burden for manufacturing companies.

[0005] 2. Subjectivity and Inconsistency in Judgment Standards: Manual re-judgment relies heavily on the operator's personal experience and understanding of the IPC-A-610 standard. Different operators often have different judgment criteria for the same defect (such as insufficient solder, tombstoning, and cold solder joints), and even the same operator may have inconsistent judgment results under different fatigue conditions. This seriously affects the stability of product quality and the traceability of data.

[0006] 3. Limitations of existing automated assistance methods: Some existing "automatic interpretation" technologies are based on simple rule engines or traditional image processing algorithms (such as fixed threshold segmentation), which cannot handle complex defect scenes (such as distinguishing between light and shadow artifacts and real bridging).

[0007] Other deep learning-based defect classification methods, while improving recognition accuracy, typically operate as "black box" models, lacking explainability. In industrial quality inspection scenarios, simply outputting "NG" results without providing justification (such as "why it was judged as bridging" or "whether it violates a clause of IPC-A-610") is unacceptable, making it difficult for engineers to trust and adopt fully automated interpretation results. Summary of the Invention

[0008] To address the aforementioned technical problems, this paper provides an automatic AOI defect re-judgment method and system based on AI intelligent agents. This technical solution solves the problems mentioned in the background section.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: In a first aspect of the present invention, an automatic review method for AOI defects based on an AI agent is provided, comprising: Obtain the interface information and defect image corresponding to the current defect from the graphical user interface of the AOI (Automated Optical Inspection) re-judgment software; Visual features are extracted from defect images, and the defect confidence value and preliminary defect type label output by the AOI device are extracted from the interface information. The extracted visual features, defect confidence scores, and preliminary defect type labels are input into a multimodal large model. Chain-of-Thought multi-step reasoning is performed in conjunction with the SMT defect knowledge base to output defect judgment results and reasoning reasons. The SMT defect knowledge base includes real defect features, false alarm causes, and the IPC-A-610 standard. Based on the defect judgment results, defect judgment operations are performed on the AOI review software interface in a simulated manual manner, including switching defect records, selecting judgment buttons, entering remarks, and saving review records. Record the review results, reasoning process data, and raw detection data from the AOI equipment, and store them in the database; When new defect samples or feedback data accumulate in the database, they are added to the SMT defect knowledge base for fine-tuning and updating of the multimodal large model.

[0010] Preferably, the step of obtaining the interface information and defect image corresponding to the current defect from the graphical user interface of the AOI re-judgment software specifically includes: The status of the foreground window of the AOI review software can be monitored in real time through the operating system-level screen capture interface or hook function. When a focus switch or item selection event is detected in the defect list control, a specified area of ​​the current screen is captured as an image. The text region in the cropped image is scanned using an optical character recognition (OCR) engine to locate and extract a string containing the defect number, coordinates (X, Y), rotation angle, and the original confidence value output by the AOI device. By parsing the UI automation tree or DOM structure of the review software, the metadata tags corresponding to the currently selected defect are obtained, including the defect type code determined in the preliminary judgment and the handle ID of the image display control. The absolute pixel coordinates of the defect image in the screen coordinate system are calculated based on the obtained handle ID, using the following formula: ; in, and These represent the horizontal and vertical coordinates of the top-left corner of the defect image in the screen coordinate system, respectively. and This represents the coordinates of the top-left corner of the AOI review software window in the screen coordinate system. and This indicates the relative offset of the defect image control within the client area of ​​the software window.

[0011] Preferably, the extraction of visual features from the defective image specifically includes: Preprocessing of defect images includes noise reduction filtering, contrast enhancement, and region of interest (ROI) cropping based on defect coordinates; Geometric features of the image are extracted using multi-scale convolutional neural networks or SIFT / SURF algorithms, including the straightness of pad edges, the relative position offset between pins and pads, and the area ratio of component outlines. The texture features of an image are calculated using the Gray-Level Co-occurrence Matrix (GLCM), and the contrast value is extracted. The calculation formula is as follows: ; Where Con represents texture contrast and N is the number of gray levels in the image. Indicates grayscale value and The probability of a pixel pair appearing at a specific direction and distance; In the HSV or Lab color space, calculate the color difference histogram between the defect area and the surrounding substrate, extract the color moment features, and use the histogram kernel to calculate the color similarity. The calculation formula is as follows: ; in, The chi-square distance, This represents the number of intervals in the histogram. For the defect area in the first The pixel distribution probability in each interval For the background substrate region in the first The probability distribution of pixels in each interval.

[0012] Preferably, the Chain-of-Thought multi-step reasoning based on the SMT defect knowledge base specifically includes: The extracted visual feature vectors are matched with standard defect templates in the SMT defect knowledge base using cosine similarity to generate a preliminary list of candidate defect categories. Read the defect confidence value output by the AOI device, and combine it with the acceptance criteria for the corresponding defect type in the IPC-A-610 standard to determine whether the confidence value is in the high false alarm range. The system retrieves the false alarm cause database from the knowledge base and performs a logical comparison between the current image features and false alarm features such as light and shadow artifacts, surface texture interference, and solder paste reflection. If a false alarm feature is matched, the probability weight of the real defect is reduced. Based on a weighted scoring mechanism, visual similarity, confidence score, standard compliance, and false alarm exclusion results are fused and calculated to output the final binary classification result. The fusion score calculation formula is as follows: ; in, For the final overall score, The visual semantic similarity score output by the multimodal large model, ranging from 0 to 1. The original confidence score output by the AOI device is normalized to 0-1. This is the false alarm penalty factor; it is 0 if it matches the cause of the false alarm, and 1 otherwise. Let be the weighting coefficient, satisfying and ; when ≥ If the condition is met, it is considered NG; otherwise, it is considered OK. This is the preset classification threshold.

[0013] Preferably, the SMT defect knowledge base is constructed and indexed in the following way: Defect knowledge is stored using a graph database structure, with nodes containing defect type entities, visual feature vectors, IPC-A-610 standard clause numbers, and false alarm scenario labels. Edge relationships include "belonging to Is-A", "causing Causes", "violating Violates", and "similar to Similar-To"; During the reasoning process, by using a hybrid indexing method of vector retrieval and keyword retrieval, at least three related neighboring defect cases and their historical re-judgment records in the knowledge base can be quickly located based on the visual feature vector of the current defect and the preliminary label. The vector retrieval uses the cosine similarity calculation formula: ; in, Represents the current defect feature vector Feature vectors of historical samples in the knowledge base The similarity is calculated by dividing the numerator by the dot product of the two vectors and the denominator by the L2 norm product of the two vectors; the top K samples with the highest similarity are selected as the reference for inference.

[0014] Preferably, the defect judgment operation performed on the AOI review software interface in a manner simulating manual operation specifically includes: Calculate the absolute pixel coordinates of the defective image on the screen based on the obtained image display control handle ID; Generate mouse movement trajectory data, control the cursor to move from the current position to the center position of the absolute pixel coordinates in a non-linear curve, and perform mouse wheel zoom operation until the image resolution reaches a preset threshold. The mouse movement trajectory is generated using Bézier curve interpolation: ; in, For a moment Mouse position coordinates, The coordinates of the starting point, The coordinates of the target point, and These are the coordinates of the control points, used to simulate the acceleration and deceleration process of a human hand; Based on the judgment result, simulate a left mouse click event, with the click coordinates as follows: The "OK" button or coordinates are The "NG" button; Activate the remarks input box, fill in the output reasoning text word by word through simulated keyboard input, and simulate pressing the "Tab" key to switch to the next input item; Capture and identify the confirmation dialog box that pops up in the software, and simulate clicking the "Save" or "Confirm" button to submit the review record.

[0015] Preferably, the storage in the database specifically includes: Establish a defect review fact table, which includes the following fields: unique defect identifier, original image hash value, original AOI confidence score, visual feature vector, judgment result output by the multimodal large model, inference process text, operation timestamp, and operator account. Establish a feedback association table for storing the true yield data of this defect fed back by subsequent processes; Perform word segmentation on the inference process text and establish an inverted index for subsequent retrieval of specific types of inference logic errors.

[0016] Preferably, the fine-tuning and updating of the multi-modal large model specifically include: Set data cleaning rules, and screen out samples with inconsistent "model judgment results" and "subsequent process feedback data" from the database, as well as samples corrected by manual sampling inspection, to form a negative sample set and a positive sample set; Perform data augmentation on the screened samples, including randomly rotating, cropping, adjusting the brightness of the defect images, and adding Gaussian noise; Adopt the Low-Rank Adaptation or P-Tuning method to freeze the main parameters of the multi-modal large model and only fine-tune the parameters of the adapter layer; Construct a new instruction fine-tuning data set in the format of "<visual feature + confidence + context>, <CoT inference chain>, <final judgment>", and use this data set to iteratively train the model; The loss function during training adopts the cross-entropy loss function, and the calculation formula is as follows: ; Where, is the average loss value, is the number of batch samples, is the true label of the th sample, 0 represents OK, 1 represents NG, is the probability value that the model predicts the th sample as NG;

[0017] Preferably, when extracting visual features from the defect images, if the AOI device provides top-view images and side-view images: Extract the two-dimensional plane features of the top-view image and the three-dimensional height features of the side-view image, such as the height of the pin翘起 (pin翘起 should be translated accurately, but it seems there is a typo, maybe it should be like "pin lifting height"); ; Perform spatial registration on the two-dimensional plane features and the three-dimensional height features to generate a fused feature vector; In the Chain-of-Thought inference, add an independent judgment branch for the side-view height, and calculate the deviation rate of the pin lifting height from the standard allowable value: ; Where, is the height deviation rate, To measure the actual pin lift height, This refers to the maximum permissible tilt height of the corresponding component in the IPC-A-610 standard. like If the result is not found, the NG (Not Given) decision is triggered directly, overriding the reasoning results of other visual features, and serving as a strong constraint for the final decision.

[0018] In a second aspect of the invention, an AOI defect automatic review system based on an AI agent is also provided, comprising: The acquisition module is used to acquire interface information and defect images corresponding to the current defect from the graphical user interface of the AOI re-judgment software. The extraction module is used to extract visual features from the defect image and extract the defect confidence value and preliminary defect type label output by the AOI device from the interface information. The output module is used to input the extracted visual features, defect confidence scores, and preliminary defect type labels into the multimodal large model, combine them with the SMT defect knowledge base to perform chain-of-thought multi-step reasoning, and output the defect judgment result and reasoning reasoning; wherein, the SMT defect knowledge base includes real defect features, false alarm causes and IPC-A-610 standard; The judgment module is used to perform defect judgment operations on the AOI review software interface in a manner that simulates manual operation based on the defect judgment result, including switching defect records, selecting a judgment button, inputting remarks information and saving the review record. The recording module is used to record the re-judgment results, reasoning process data, and the original detection data of the AOI device, and store them in the database; The update module is used to add new defect samples or feedback data to the SMT defect knowledge base when they accumulate in the database, and to fine-tune and update the multimodal large model.

[0019] Compared with existing technologies, this invention provides an automatic AOI defect re-judgment method and system based on AI intelligent agents, which has the following beneficial effects: This invention utilizes an AI agent to automatically acquire defect information and extract visual features. Combining a multimodal large model with an SMT defect knowledge base for multi-step reasoning significantly improves the efficiency and accuracy of defect review, reducing the false positive and false negative rates of manual review. It automatically executes defect judgment operations, simulating manual clicking and input of notes, reducing the workload of review operators, thereby lowering labor costs and improving the automation level of the production line. Furthermore, by incorporating industry standards such as IPC-A-610, the system ensures the objectivity and consistency of judgment standards, avoiding inconsistencies in judgment criteria caused by differences in experience among different operators. The system records review results and reasoning process data, and automatically updates the SMT defect knowledge base and fine-tunes the multimodal large model as new defect samples or feedback data accumulate, ensuring continuous optimization and adaptability. Through automated review and detailed recording, it improves product quality stability and data traceability, providing manufacturing enterprises with a powerful quality control tool. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the method flow of S101-S106 in this invention; Figure 2 This is a schematic diagram of the method flow for S201-S205 in this invention; Figure 3 This is a schematic diagram of the method flow for S301-S304 in this invention; Figure 4 This is a schematic diagram of the method flow for S401-S405 in this invention. Detailed Implementation

[0021] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0022] Example 1 Please refer to Figure 1 As shown, in a first aspect of the present invention, an automatic review method for AOI defects based on an AI agent is provided, comprising: S101. Obtain the interface information and defect image corresponding to the current defect from the graphical user interface of the AOI (Automated Optical Inspection) re-judgment software. S102. Extract visual features from the defect image, and extract the defect confidence value and preliminary defect type label output by the AOI device from the interface information. S103. Input the extracted visual features, defect confidence scores, and preliminary defect type labels into the multimodal large model, and perform chain-of-thought multi-step reasoning in conjunction with the SMT defect knowledge base, outputting the defect judgment result and reasoning reasoning; wherein, the SMT defect knowledge base includes real defect features, false alarm causes and IPC-A-610 standard. S104. Based on the defect judgment results, perform defect judgment operations on the AOI review software interface by simulating manual operation, including switching defect records, selecting judgment buttons, entering remarks information and saving review records. S105. Record the re-judgment results, reasoning process data, and original detection data of the AOI equipment, and store them in the database; S106. When new defect samples or feedback data accumulate in the database, add them to the SMT defect knowledge base and fine-tune and update the multimodal large model.

[0023] As will be understood by those skilled in the art, this invention automatically acquires defect information and extracts visual features through an AI agent, and performs multi-step reasoning by combining a multimodal large model with an SMT defect knowledge base. This significantly improves the efficiency and accuracy of defect review, and reduces the misjudgment and omission rates of manual review. It automatically executes defect judgment operations, simulating manual clicking and input of notes, reducing the workload of review operators, thereby lowering labor costs and improving the automation level of the production line. Furthermore, by incorporating industry standards such as IPC-A-610, the system ensures the objectivity and consistency of judgment standards, avoiding inconsistencies in judgment criteria caused by differences in experience among different operators. The system can record review results and reasoning process data, and automatically updates the SMT defect knowledge base and fine-tunes the multimodal large model when accumulating new defect samples or feedback data, ensuring continuous optimization and adaptability of the system. Through automated review and detailed recording, it improves the stability of product quality and the traceability of data, providing manufacturing enterprises with a powerful quality control tool.

[0024] Please refer to Figure 2 As shown, the interface information and defect image corresponding to the current defect are obtained from the graphical user interface of the AOI (Automated Optical Inspection) re-judgment software, specifically including: S201. Monitor the foreground window status of the AOI review software in real time through the operating system-level screen capture interface or hook function. S202. When a focus switch or item selection event is detected in the defect list control, capture an image of a specified area of ​​the current screen. S203. Use an optical character recognition (OCR) engine to scan the text region in the cropped image, locate and extract a string containing the defect number, coordinates (X, Y), rotation angle and the original confidence value output by the AOI device; S204. By parsing the UI automation tree or DOM structure of the review software, obtain the metadata tag corresponding to the currently selected defect, including the defect type code of the preliminary judgment and the handle ID of the image display control; S205. Calculate the absolute pixel coordinates of the defect image in the screen coordinate system based on the obtained handle ID. The calculation formula is as follows: ; in, and These represent the horizontal and vertical coordinates of the top-left corner of the defect image in the screen coordinate system, respectively. and This represents the coordinates of the top-left corner of the AOI review software window in the screen coordinate system. and This indicates the relative offset of the defect image control within the client area of ​​the software window.

[0025] Please refer to Figure 3 As shown, visual features are extracted from defective images, specifically including: S301. Preprocess the defect image, including noise reduction filtering, contrast enhancement, and region of interest (ROI) cropping based on defect coordinates. S302. Use a multi-scale convolutional neural network or SIFT / SURF algorithm to extract the geometric features of the image, including the straightness of the pad edge, the relative position offset of the pin and the pad, and the area ratio of the component outline. S303. Calculate the texture features of the image using the Gray-Level Co-occurrence Matrix (GLCM) and extract the contrast value. The calculation formula is as follows: ; Where Con represents texture contrast and N is the number of gray levels in the image. Indicates grayscale value and The probability of a pixel pair appearing at a specific direction and distance; S304. In the HSV or Lab color space, calculate the color difference histogram between the defect area and the surrounding substrate, extract the color moment features, and use the histogram kernel to calculate the color similarity. The calculation formula is as follows: ; in, The chi-square distance, This represents the number of intervals in the histogram. For the defect area in the first The pixel distribution probability in each interval For the background substrate region in the first The probability distribution of pixels in each interval.

[0026] Please refer to Figure 4 As shown, Chain-of-Thought multi-step reasoning is performed using the SMT defect knowledge base, specifically including: S401. Perform cosine similarity matching between the extracted visual feature vectors and the standard defect templates in the SMT defect knowledge base to generate a preliminary list of candidate defect categories. S402. Read the defect confidence value output by the AOI device, and in conjunction with the acceptance criteria for the corresponding defect type in the IPC-A-610 standard, determine whether the confidence value is in the high false alarm range. S403. Search the false alarm cause database in the knowledge base and logically compare the current image features with false alarm features such as light and shadow artifacts, surface texture interference, and solder paste reflection; if a false alarm feature is matched, reduce the probability weight of the real defect. S404. Based on a weighted scoring mechanism, visual similarity, confidence score, standard compliance, and false alarm exclusion results are fused and calculated to output the final binary classification result. The fusion score calculation formula is as follows: ; in, For the final overall score, The visual semantic similarity score output by the multimodal large model, ranging from 0 to 1. The original confidence score output by the AOI device is normalized to 0-1. This is the false alarm penalty factor; it is 0 if it matches the cause of the false alarm, and 1 otherwise. Let be the weighting coefficient, satisfying and ; S405, when ≥ If the condition is met, it is considered NG; otherwise, it is considered OK. This is the preset classification threshold.

[0027] The SMT defect knowledge base is constructed and indexed in the following way: Defect knowledge is stored using a graph database structure, with nodes containing defect type entities, visual feature vectors, IPC-A-610 standard clause numbers, and false alarm scenario labels. Edge relationships include "belonging to Is-A", "causing Causes", "violating Violates", and "similar to Similar-To"; During the reasoning process, by using a hybrid indexing method of vector retrieval and keyword retrieval, at least three related neighboring defect cases and their historical re-judgment records in the knowledge base can be quickly located based on the visual feature vector of the current defect and the preliminary label. Vector retrieval uses the cosine similarity calculation formula: ; in, Represents the current defect feature vector Feature vectors of historical samples in the knowledge base The similarity is calculated by dividing the numerator by the dot product of the two vectors and the denominator by the L2 norm product of the two vectors; the top K samples with the highest similarity are selected as the reference for inference.

[0028] Defect judgment operations are performed on the AOI review software interface in a manner that simulates manual operation, specifically including: Calculate the absolute pixel coordinates of the defective image on the screen based on the obtained image display control handle ID; Generate mouse movement trajectory data, control the cursor to move from the current position to the center position of absolute pixel coordinates in a non-linear curve, and perform mouse wheel zoom operation until the image resolution reaches the preset threshold; The mouse movement trajectory is generated using Bézier curve interpolation: ; in, For a moment Mouse position coordinates, The coordinates of the starting point, The coordinates of the target point, and These are the coordinates of the control points, used to simulate the acceleration and deceleration process of a human hand; Based on the judgment result, simulate a left mouse click event, with the click coordinates as follows: The "OK" button or coordinates are The "NG" button; Activate the remarks input box, fill in the output reasoning text word by word through simulated keyboard input, and simulate pressing the "Tab" key to switch to the next input item; Capture and identify the confirmation dialog box that pops up in the software, and simulate clicking the "Save" or "Confirm" button to submit the review record.

[0029] Storing to the database, specifically including: Establish a defect review fact table, which includes the following fields: unique defect identifier, original image hash value, original AOI confidence score, visual feature vector, judgment result output by the multimodal large model, inference process text, operation timestamp, and operator account. Establish a feedback association table to store the actual yield data of the defect reported by subsequent processes; The reasoning process text is segmented into words, and an inverted index is built to facilitate subsequent retrieval of specific types of reasoning logic errors.

[0030] Fine-tune and update the multi-modal large model, specifically including: Set data cleaning rules, and screen out samples with inconsistent "model determination results" and "subsequent process feedback data" from the database, as well as samples corrected by manual random inspection, to form negative sample sets and positive sample sets; Perform data augmentation on the screened samples, including randomly rotating, cropping, adjusting the brightness of defective images, and adding Gaussian noise; Adopt the low-rank adaptation or P-Tuning method to freeze the main parameters of the multi-modal large model and only fine-tune the parameters of the adapter layer; Construct a new instruction fine-tuning dataset in the format of "<visual feature + confidence + context>, <CoT reasoning chain>, <final determination>", and use this dataset to iteratively train the model; The loss function during training adopts the cross-entropy loss function, and the calculation formula is as follows: ; Among them, is the average loss value, is the number of batch samples, is the true label of the th sample, 0 represents OK, 1 represents NG, is the probability value that the model predicts the th sample as NG;

[0031] When the loss value on the validation set continuously decreases by less than the preset threshold for M times, stop training and update the model weight file. When extracting visual features from defective images, if the AOI device provides top-view images and side-view images: Extract the two-dimensional plane features of the top-view image and the three-dimensional height features of the side-view image respectively, such as the height of the pin翘起 ; Perform spatial registration on the two-dimensional plane features and the three-dimensional height features to generate a fused feature vector; ; Among them, [[ID=[]47]]is the height deviation rate, is the measured height of the pin翘起, is the maximum allowable翘起 height of the corresponding component in the IPC-A-610 standard; If , directly trigger the NG determination, overwrite the inference results of other visual features, and serve as a strong constraint condition for the final determination.

[0032] In a second aspect of the invention, an AOI defect automatic review system based on an AI agent is also provided, comprising: The acquisition module is used to acquire the interface information and defect image corresponding to the current defect from the graphical user interface of the AOI re-judgment software. The extraction module is used to extract visual features from defect images and extract the defect confidence value and preliminary defect type label output by the AOI device from the interface information. The output module is used to input the extracted visual features, defect confidence scores, and preliminary defect type labels into the multimodal large model, and perform chain-of-thought multi-step reasoning in conjunction with the SMT defect knowledge base, outputting the defect judgment result and reasoning; wherein, the SMT defect knowledge base includes real defect features, false alarm causes and IPC-A-610 standard. The judgment module is used to perform defect judgment operations on the AOI review software interface in a manner that simulates manual operation based on the defect judgment results. This includes switching defect records, selecting judgment buttons, entering remarks, and saving review records. The recording module is used to record the review results, reasoning process data, and the original detection data of the AOI device, and store them in the database. The update module is used to add new defect samples or feedback data to the SMT defect knowledge base when they accumulate in the database, and to fine-tune and update the multimodal large model.

[0033] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for automatic re-judgment of AOI defects based on AI intelligent agents, characterized in that, include: Obtain the interface information and defect image corresponding to the current defect from the graphical user interface of the AOI (Automated Optical Inspection) re-judgment software; Visual features are extracted from defect images, and the defect confidence value and preliminary defect type label output by the AOI device are extracted from the interface information. The extracted visual features, defect confidence scores, and preliminary defect type labels are input into a multimodal large model. Chain-of-Thought multi-step reasoning is performed in conjunction with the SMT defect knowledge base to output defect judgment results and reasoning reasons. The SMT defect knowledge base includes real defect features, false alarm causes, and the IPC-A-610 standard. Based on the defect judgment results, defect judgment operations are performed on the AOI review software interface in a simulated manual manner, including switching defect records, selecting judgment buttons, entering remarks, and saving review records. Record the review results, reasoning process data, and raw detection data from the AOI equipment, and store them in the database; When new defect samples or feedback data accumulate in the database, they are added to the SMT defect knowledge base for fine-tuning and updating of the multimodal large model.

2. The method for automatic re-judgment of AOI defects based on AI intelligent agents according to claim 1, characterized in that, The process of obtaining the interface information and defect image corresponding to the current defect from the graphical user interface of the AOI (Automated Optical Inspection) re-judgment software specifically includes: The status of the foreground window of the AOI review software can be monitored in real time through the operating system-level screen capture interface or hook function. When a focus switch or item selection event is detected in the defect list control, a specified area of ​​the current screen is captured as an image. The text region in the cropped image is scanned using an optical character recognition (OCR) engine to locate and extract a string containing the defect number, coordinates (X, Y), rotation angle, and the original confidence value output by the AOI device. By parsing the UI automation tree or DOM structure of the review software, the metadata tags corresponding to the currently selected defect are obtained, including the defect type code determined in the preliminary judgment and the handle ID of the image display control. The absolute pixel coordinates of the defect image in the screen coordinate system are calculated based on the obtained handle ID, using the following formula: ； in, and These represent the horizontal and vertical coordinates of the top-left corner of the defect image in the screen coordinate system, respectively. and This represents the coordinates of the top-left corner of the AOI review software window in the screen coordinate system. and This indicates the relative offset of the defect image control within the client area of ​​the software window.

3. The method for automatic re-judgment of AOI defects based on AI intelligent agents according to claim 2, characterized in that, The extraction of visual features from defective images specifically includes: Preprocessing of defect images includes noise reduction filtering, contrast enhancement, and region of interest (ROI) cropping based on defect coordinates; Geometric features of the image are extracted using multi-scale convolutional neural networks or SIFT / SURF algorithms, including the straightness of pad edges, the relative position offset between pins and pads, and the area ratio of component outlines. The texture features of an image are calculated using the Gray-Level Co-occurrence Matrix (GLCM), and the contrast value is extracted. The calculation formula is as follows: ； Where Con represents texture contrast and N is the number of gray levels in the image. Indicates grayscale value and The probability of a pixel pair appearing at a specific direction and distance; In the HSV or Lab color space, calculate the color difference histogram between the defect area and the surrounding substrate, extract the color moment features, and use the histogram kernel to calculate the color similarity. The calculation formula is as follows: ； in, The chi-square distance, This represents the number of intervals in the histogram. For the defect area in the first The pixel distribution probability in each interval For the background substrate region in the first The probability distribution of pixels in each interval.

4. The method for automatic re-judgment of AOI defects based on AI intelligent agents according to claim 3, characterized in that, The Chain-of-Thought multi-step reasoning method, which combines the SMT defect knowledge base, specifically includes: The extracted visual feature vectors are matched with standard defect templates in the SMT defect knowledge base using cosine similarity to generate a preliminary list of candidate defect categories. Read the defect confidence value output by the AOI device, and combine it with the acceptance criteria for the corresponding defect type in the IPC-A-610 standard to determine whether the confidence value is in the high false alarm range. The system retrieves the false alarm cause database from the knowledge base and performs a logical comparison between the current image features and false alarm features such as light and shadow artifacts, surface texture interference, and solder paste reflection. If a false alarm feature is matched, the probability weight of the real defect is reduced. Based on a weighted scoring mechanism, visual similarity, confidence score, standard compliance, and false alarm exclusion results are fused and calculated to output the final binary classification result. The fusion score calculation formula is as follows: ； in, For the final overall score, The visual semantic similarity score output by the multimodal large model, ranging from 0 to 1. The original confidence score output by the AOI device is normalized to 0-1. This is the false alarm penalty factor; it is 0 if it matches the cause of the false alarm, and 1 otherwise. For the weighting coefficients, satisfying and ; when ≥ If the condition is met, it is considered NG; otherwise, it is considered OK. This is the preset classification threshold.

5. The method for automatic re-judgment of AOI defects based on AI intelligent agents according to claim 4, characterized in that, The SMT defect knowledge base is constructed and indexed in the following way: Defect knowledge is stored using a graph database structure, with nodes containing defect type entities, visual feature vectors, IPC-A-610 standard clause numbers, and false alarm scenario labels. Edge relationships include "belongs to Is-A", "causes", "violates Violates", and "similar to Similar-To"; During the reasoning process, by using a hybrid indexing method of vector retrieval and keyword retrieval, at least three related neighboring defect cases and their historical re-judgment records in the knowledge base can be quickly located based on the visual feature vector of the current defect and the preliminary label. The vector retrieval uses the cosine similarity calculation formula: ； in, Represents the current defect feature vector Feature vectors of historical samples in the knowledge base The similarity is calculated by dividing the numerator by the dot product of the two vectors and the denominator by the L2 norm product of the two vectors; the top K samples with the highest similarity are selected as the reference for inference.

6. The method for automatic re-judgment of AOI defects based on AI intelligent agents according to claim 5, characterized in that, The defect judgment operation performed on the AOI review software interface in a manner simulating manual operation specifically includes: Calculate the absolute pixel coordinates of the defective image on the screen based on the obtained image display control handle ID; Generate mouse movement trajectory data, control the cursor to move from the current position to the center position of the absolute pixel coordinates in a non-linear curve, and perform mouse wheel zoom operation until the image resolution reaches a preset threshold. The mouse movement trajectory is generated using Bézier curve interpolation: ； in, For a moment Mouse position coordinates, The coordinates of the starting point, The coordinates of the target point, and These are the coordinates of the control points, used to simulate the acceleration and deceleration process of a human hand; Based on the judgment result, simulate a left mouse click event, with the click coordinates as follows: The "OK" button or coordinates are The "NG" button; Activate the remarks input box, fill in the output reasoning text word by word through simulated keyboard input, and simulate pressing the "Tab" key to switch to the next input item; Capture and identify the confirmation dialog box that pops up in the software, and simulate clicking the "Save" or "Confirm" button to submit the review record.

7. The method for automatic re-judgment of AOI defects based on AI intelligent agents according to claim 6, characterized in that, The storage to the database specifically includes: Establish a defect review fact table, which includes the following fields: unique defect identifier, original image hash value, original AOI confidence score, visual feature vector, judgment result output by the multimodal large model, inference process text, operation timestamp, and operator account. Establish a feedback association table to store the actual yield data of the defect reported by subsequent processes; The reasoning process text is segmented into words, and an inverted index is built to facilitate subsequent retrieval of specific types of reasoning logic errors.

8. The method for automatic re-judgment of AOI defects based on AI intelligent agents according to claim 7, characterized in that, The fine-tuning and updating of the multimodal large model specifically includes: Set data cleaning rules to filter out samples from the database where the "model judgment result" and "subsequent process feedback data" are inconsistent, as well as samples that have been manually checked and corrected, to form a negative sample set and a positive sample set; Data augmentation is performed on the selected samples, including random rotation, cropping, brightness adjustment, and addition of Gaussian noise to defective images; Adopt the low-rank adaptation or P-Tuning method to freeze the main parameters of the multimodal large model and only fine-tune the parameters of the adapter layer; Construct a new instruction fine-tuning dataset in the format of "<visual feature + confidence + context>, <CoT reasoning chain>, <final determination>", and use this dataset to iteratively train the model; The loss function during training uses the cross-entropy loss function, and the calculation formula is as follows: ； in, This is the average loss value. This refers to the batch sample size. For the first The true label for each sample is 0 for OK and 1 for NG. The model predicts the first The probability value of a sample being NG; When the loss value on the validation set continuously drops by less than the preset threshold for M times, stop training and update the model weight file.

9. The method for automatic re-judgment of AOI defects based on AI intelligent agents according to claim 8, characterized in that, When extracting visual features from the defect image, if the AOI device provides top-view images and side-view images: Extract the two-dimensional planar features of the top view image and the three-dimensional height features of the side view image, such as the pin protrusion height. ; Perform spatial registration on the two-dimensional plane features and three-dimensional height features to generate a fused feature vector; In the Chain-of-Thought reasoning, add an independent judgment branch for the side-view height, and calculate the deviation rate between the pin翘起 height and the standard allowable value: ； in, For the height deviation rate, To measure the actual pin lift height, This refers to the maximum permissible tilt height of the corresponding component in the IPC-A-610 standard. like If the result is not found, the NG (Not Given) decision is triggered directly, overriding the reasoning results of other visual features, and serving as a strong constraint for the final decision.

10. An AOI defect automatic review system based on AI intelligent agents, used to implement the AOI defect automatic review method based on AI intelligent agents as described in any one of claims 1-9, characterized in that, Include: An acquisition module, which is used to acquire the interface information and defect image corresponding to the current defect from the graphical user interface of the automatic optical inspection (AOI) rejudgment software; An extraction module, which is used to extract visual features from the defect image, and extract the defect confidence value and the preliminary defect type label output by the AOI device from the interface information; An output module, which is used to input the extracted visual features, defect confidence value, and preliminary defect type label into the multimodal large model, perform Chain-of-Thought multi-step reasoning in combination with the SMT defect knowledge base, and output the defect determination result and the reasoning reason; where the SMT defect knowledge base contains real defect features, false alarm causes, and IPC-A-610 standards; A determination module, which is used to perform defect determination operations on the AOI rejudgment software interface by simulating manual operations according to the defect determination result, including switching defect records, selecting determination buttons, inputting remarks information, and saving the rejudgment records; A recording module, which is used to record the rejudgment results, reasoning process data, and the original detection data of the AOI device, and store them in the database; An update module, which is used to add new defect samples or feedback data to the SMT defect knowledge base and fine-tune and update the multimodal large model when they are accumulated in the database.

Images