A Deep Learning-Based Intelligent Review and Control System for AOI Equipment

Abstract

This application relates to the field of industrial automation control technology and discloses an intelligent review and control system for AOI equipment based on deep learning. The system includes an equipment docking and data adaptation module, a review process execution module, a confidence-based closed-loop control module, a manual review and feedback module, and a model training and iteration module. The equipment docking module converts heterogeneous data into standardized objects; the review module outputs confidence scores based on a CNN model and executes a diversion strategy; the control module calculates parameters based on review differences to correct gradients and adjust AOI equipment accordingly; and the training module uses manually judged data to perform incremental updates. This invention establishes a closed-loop feedback mechanism from algorithm inference to physical equipment through a confidence-weighted algorithm, realizing adaptive dynamic adjustment of AOI detection parameters and continuous model iteration, which helps reduce the false alarm rate on the production line and improve the automated operation and maintenance level of the detection system.

Description

Technical Field

[0001] This invention relates to the field of industrial automation control technology, specifically to an intelligent AOI equipment judgment and control system based on deep learning. Background Technology

[0002] Automatic optical inspection equipment, as a core edge sensing node in the industrial Internet of Things architecture of electronic manufacturing, is configured to use a high-resolution optical imaging system and image processing algorithms to capture the surface features of printed circuit board components and compare the acquired image data with a preset gold standard in real time to identify manufacturing quality problems such as welding defects, missing components or polarity deviations.

[0003] Existing surface mount technology production lines typically employ a multi-brand hybrid deployment model and rely on independent industrial gateways or vendor-specific host computer software to obtain inspection logs and image files from automated optical inspection equipment through periodic polling and document scanning. Production line operators need to manually review intercepted products using a local human-machine interface, and process engineers need to periodically export historical data for offline statistical analysis and manually adjust the equipment's inspection thresholds or light source parameters based on their personal experience to maintain production line operation.

[0004] However, due to the lack of a unified data interaction standard between heterogeneous automated optical inspection equipment and the lack of AI-based protocol adaptive parsing and multimodal data normalization capabilities in traditional data acquisition schemes, production data collected across devices suffers from problems such as chaotic format, inconsistent dimensions, and high redundancy, making it difficult to extract decision-valuable operational features in real time. Furthermore, the lack of a robust hardware-level encryption channel and data integrity verification mechanism throughout the data transmission process, coupled with the absence of breakpoint resume and intelligent synchronization functions in the face of network bandwidth fluctuations or strong electromagnetic interference in industrial settings, poses a risk of core production data loss, tampering, or temporal discrepancies during transmission. Finally, the system disconnect between the underlying physical inspection end and the upper-level data management end, coupled with the lack of closed-loop feedback and dynamic parameter correction mechanisms based on deep learning confidence, prevents the equipment from automatically optimizing physical layer inspection parameters based on manual review results. This leads to a decrease in system accuracy after prolonged operation and increases reliance on manual maintenance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an intelligent AOI device review and control system based on deep learning. It aims to solve the problems of lack of data interaction standards for existing heterogeneous detection devices, lack of a confidence-based closed-loop feedback mechanism leading to the inability to adaptively and dynamically adjust physical detection parameters, and high dependence on manual operation and maintenance with low model iteration efficiency.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention provides a deep learning-based intelligent AOI (Automated Optical Inspection) device review and control system, comprising:

[0008] The system includes a device docking and data adaptation module, a review process execution module, a confidence-based closed-loop management module, a manual review and feedback module, and a model training and iteration module.

[0009] The device docking and data adaptation module establishes communication connections with heterogeneous AOI inspection equipment on the production line, captures inspection data packets containing NG images and metadata, and sends the converted standardized data objects to a dedicated buffer on the production line. To address the inconsistency in scoring dimensions between different brands of equipment, this module loads the device mapping configuration file to obtain the theoretical minimum and maximum scores. It calculates the difference between the original inspection score and the theoretical minimum score as the numerator, and the difference between the theoretical maximum score and the theoretical minimum score as the denominator. Finally, it divides the numerator by the denominator to obtain the normalized standard inspection confidence score. Simultaneously, this module maps manufacturer-specific defect codes to a system-wide unified defect enumeration type.

[0010] The re-judgment process execution module retrieves the standardized data object from the production line-specific cache, performs inference using a pre-trained convolutional neural network model, and outputs the re-judgment classification result and defect probability value. Based on the defect probability value, a triage strategy is executed. When calculating the defect probability value, this module selects the larger of the true defect probability and the false defect probability output by the model as the baseline probability. It calculates the absolute value of the difference between the true and false defect probabilities and performs an exponential operation on this absolute value based on a preset penalty coefficient to reduce the weight of samples with ambiguous classification boundaries. Finally, the exponential operation result is multiplied by the baseline probability. If the judgment result is a false defect and the calculated defect probability value is higher than the automatic release threshold, the system generates a release instruction; otherwise, manual routing logic is triggered.

[0011] The confidence-based closed-loop control module monitors the judgment differences between the AOI inspection equipment and the review process execution module. When a drift trend is detected in the parameters of the AOI inspection equipment, it calculates the detection threshold correction amount and adjusts the physical inspection parameters of the AOI inspection equipment in reverse. This module maintains a statistical queue based on a time sliding window, marking data that are judged as NG by the AOI inspection equipment but as false defects by the review process execution module as false alarm samples of the target component reference number.

[0012] The logic for determining parameter drift must simultaneously satisfy three statistical characteristics: the frequency of false alarm samples within the sliding window exceeds a preset upper limit; the weighted average of the defect probability values ​​corresponding to the false alarm samples is higher than a preset confidence threshold; and the standard deviation of the original AOI detection scores corresponding to the false alarm samples is less than a preset dispersion threshold. Meeting these conditions indicates that the judgment bias of the AOI equipment is systematic rather than random.

[0013] After triggering parameter correction, the system calculates the difference between the mean of the original AOI detection scores of the false alarm samples and the detection threshold currently in effect of the AOI detection equipment. It then uses a preset learning rate coefficient and the mean of the defect probability value of the false alarm samples to perform a weighted calculation on this difference, and adds a preset safety boundary margin to the weighted calculation result to obtain the detection threshold correction amount.

[0014] The manual review and feedback module receives suspected defect data sent by the review process execution module, provides a human-computer interaction interface to obtain the final manual decision, and generates labeled sample data. This module compares the final manual decision with the review classification result in real time. When the two are inconsistent, the corresponding image sample is marked as a label difference sample, and the feature map data corresponding to the review classification result is extracted, packaged, and sent to the backend.

[0015] The model training and iteration module updates the convolutional neural network model using the labeled sample data and distributes the updated model parameters to the review process execution module. When constructing the training set, the system extracts confirmed NG samples and label difference samples from the data stream to build an incremental dataset, and mixes anchor samples from the historical benchmark sample library with the incremental dataset according to a preset ratio. Furthermore, the model version update operation is restricted to idle time slots when the detection task queue is empty.

[0016] This invention provides an intelligent AOI (Automated Optical Inspection) device review and control system based on deep learning. It has the following beneficial effects:

[0017] 1. This invention achieves unified access and data standardization for heterogeneous testing equipment through equipment docking and data adaptation modules. Utilizing preset normalization logic, the system maps the original test scores output by AOI equipment from different brands to standard test confidence levels with unified dimensions, and converts private defect codes into general enumeration types. This helps eliminate data differences between multi-brand equipment on the production line, enabling the upper-level management system to conduct centralized quality analysis and review across equipment hardware barriers, and helps reduce the complexity of multi-equipment collaborative management.

[0018] 2. This invention establishes a confidence-based closed-loop control mechanism, realizing automatic reverse correction of parameters from the back-end algorithm to the front-end physical equipment. By monitoring the false alarm rate of specific component reference numbers, the average AI defect probability value, and the dispersion of the original AOI score, the system can accurately identify the parameter drift trend of the equipment and calculate the threshold correction amount including the safety boundary margin. It can dynamically adjust the physical detection parameters of the AOI equipment, suppress systematic false alarms caused by environmental fluctuations or equipment aging, and reduce the maintenance cost of frequent manual debugging of the equipment.

[0019] 3. This invention employs a defect probability calculation method that introduces a penalty coefficient and an incremental learning strategy that mixes anchor samples, ensuring the stability and adaptability of the model during long-term operation. By penalizing fuzzy samples with small differences between true and false defect probabilities, the system reduces the risk of misjudgment during automatic release. At the same time, when updating the model, the newly added difference samples are mixed with historical benchmark samples for training, which helps prevent the convolutional neural network from forgetting old knowledge when learning new process features, ensuring that the re-judgment accuracy is continuously optimized with the production cycle. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall system structure according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic diagram of the main workflow of the system according to an embodiment of the present invention;

[0022] Figure 3 This is a flowchart illustrating the data adaptation and standardization process in an embodiment of the present invention.

[0023] Figure 4 This is a logic diagram of intelligent review and routing decision-making in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of parameter closed-loop management in an embodiment of the present invention;

[0025] Figure 6 This is a flowchart of the incremental training and iteration process of the model in an embodiment of the present invention.

[0026] The module includes: 100, Equipment Interconnection and Data Adaptation Module; 200, Review Process Execution Module; 300, Confidence-Based Closed-Loop Control Module; 400, Manual Review and Feedback Module; 500, Model Training and Iteration Module; and 600, Data Management and Visualization Module. Detailed Implementation

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] See attached document Figure 1-6 This invention provides a deep learning-based intelligent AOI (Automated Optical Inspection) equipment review and control system. Deployed in the network environment of an electronic manufacturing production line, the system is configured to achieve centralized management, image review, and closed-loop control of heterogeneous AOI inspection equipment. The intelligent AOI equipment review and control system includes:

[0029] The system includes a device docking and data adaptation module 100, a review process execution module 200, a confidence-based closed-loop control module 300, a manual review and feedback module 400, a model training and iteration module 500, and a data management and visualization module 600.

[0030] The device interface and data adaptation module 100 is used to establish a communication connection with AOI inspection equipment on the production line via an industrial local area network. The module 100 has a built-in device configuration parsing unit and drivers for various communication protocols. When the AOI equipment intelligent review and control system starts, the module 100 reads a preset device mapping configuration file to obtain the brand identifier, IP address, and communication method of the connected AOI inspection equipment, and automatically loads the corresponding RS communication protocol driver, HTTP interface listening service, or shared folder polling service based on the brand identifier.

[0031] The equipment docking and data adaptation module 100 is used to capture the inspection data packets output by the AOI inspection equipment in real time. The inspection data packets include NG images and XML metadata files. The equipment docking and data adaptation module 100 also includes a decryption and parsing unit, which is used to decode the encrypted XML files and manufacturer-specific image formats, extract PCB identification codes, component reference numbers, original inspection scores, and defect type fields, and convert heterogeneous data into a unified standardized data object within the AOI equipment intelligent review and control system, storing it in the production line's dedicated cache area.

[0032] The review process execution module 200 communicates with the equipment interface and data adaptation module 100 and the data management and visualization module 600. The review process execution module 200 is configured to retrieve NG images from standardized data objects in the production line's dedicated buffer and execute core inference tasks. Internally, the review process execution module 200 deploys a pre-trained convolutional neural network model, trained on a sample set of defect images including scratch grayscale gradient features and weld point morphology features. The review process execution module 200 performs forward inference calculations on the input NG images and outputs the review classification result and the corresponding confidence score.

[0033] If the reassessment classification result is a false defect and the confidence level is higher than the preset automatic release threshold, the reassessment process execution module 200 directly generates an OK release instruction and feeds the release instruction back to the AOI device or production line control node through the equipment docking and data adaptation module 100; if the reassessment classification result is a true defect or the confidence level is lower than the automatic release threshold, the reassessment process execution module 200 routes the data to the manual review and feedback module 400.

[0034] The confidence-based closed-loop control module 300 establishes bidirectional data connections with both the review process execution module 200 and the equipment interface and data adaptation module 100. The confidence-based closed-loop control module 300 is configured to continuously monitor historical review data for specific component reference numbers. When the AOI inspection equipment continuously reports NG results for a specific component reference number, and the confidence value of the review process execution module 200 in determining it as a false defect remains consistently higher than a preset drift threshold, the confidence-based closed-loop control module 300 determines that there is parameter drift in the component reference number.

[0035] The confidence-based closed-loop control module 300 executes a parameter correction algorithm, calculates the deviation between the original AOI detection score and the preset standard value, and uses the confidence value as a weighting coefficient to calculate the parameter correction gradient. Subsequently, the confidence-based closed-loop control module 300 encapsulates the parameter correction gradient into a control command, which is written back to the configuration file or control interface of the AOI detection device through the device docking and data adaptation module 100 to dynamically adjust the detection threshold parameters of the AOI detection device.

[0036] The manual review and feedback module 400 is connected to the review process execution module 200 and the data management and visualization module 600. The manual review and feedback module 400 is configured to provide a human-machine interface for processing suspected defect data routed from the review process execution module 200. The manual review and feedback module 400 displays NG images, AI review suggestions, and component location diagrams on a display terminal, and receives the final decision result input by the operator. The manual review and feedback module 400 is configured to synchronously return the final decision result to the AOI device via the device docking and data adaptation module 100 to deactivate the AOI device alarm or confirm the rejection of defective products.

[0037] The model training and iteration module 500 is connected to the manual review and feedback module 400 and the re-judgment process execution module 200. The model training and iteration module 500 is configured to perform closed-loop sample management, automatically collecting true defect images confirmed by the manual review and feedback module 400, as well as sample images where the re-judgment classification results are inconsistent with the final manual decision. The model training and iteration module 500 uses the collected image samples to incrementally train the convolutional neural network model in the re-judgment process execution module 200, updates the model weight parameters, and distributes the updated model to the re-judgment process execution module 200 to adapt to changes in production processes.

[0038] The data management and visualization module 600 establishes data connections with all the aforementioned modules. It is configured to provide persistent data storage and report display capabilities. The module includes a local database and a web visualization engine for real-time recording of the original images, AI re-judgment results, confidence levels, manual judgment results, and parameter correction logs for each inspection. The module is also configured to generate statistical reports, displaying real-time first-pass yields, equipment false alarm rate rankings, and Pareto charts of defect type distribution for each production line on the display terminal. It also provides historical data tracing and query functions based on time, board model, or defect type.

[0039] In practice, the workflow of the AOI equipment intelligent review and control system is as follows:

[0040] The front-end AOI inspection equipment performs optical scanning on the PCB board. When the inspection algorithm determines that a defect exists, the front-end AOI inspection equipment generates an inspection data package containing NG images and metadata. The equipment docking and data adaptation module 100 captures the inspection data package through a preset communication protocol, uses a decryption algorithm to parse the XML file and image data, extracts key features, and generates a standardized data object. The re-judgment process execution module 200 reads the standardized data object, uses a convolutional neural network to extract deep semantic features of the image, outputs the re-judgment result as a true defect or a false defect, and includes a confidence score.

[0041] If the review process execution module 200 determines a defect as a false defect with high confidence, it directly sends a release signal. If it determines a defect as genuine or the confidence level is questionable, the standardized data object enters the manual review and feedback module 400. Simultaneously, the confidence-based closed-loop control module 300 performs statistical analysis on the historical review data in the background. If the confidence-based closed-loop control module 300 detects a systematic deviation in the false alarm rate of a specific AOI inspection device at a particular solder joint location, and the confidence level output by the review process execution module 200 remains consistently above 0.95, the confidence-based closed-loop control module 300 calculates a parameter adjustment amount weighted by the deviation and confidence level. This parameter adjustment amount is converted into a device-specific control code and sent to the AOI inspection device via the network to modify its parameter settings, thereby reducing the subsequent false alarm frequency at the specific solder joint location.

[0042] Operators view images and make final decisions on the interface of the manual review and feedback module 400, while the data management and visualization module 600 records the final decision-making process in real time. Based on the final decision results, if the manual decision corrects the AI's initial judgment or confirms a new process defect, the model training and iteration module 500 marks the image as a high-value sample for subsequent model iteration training, achieving adaptive performance iteration of the AOI equipment intelligent review and control system.

[0043] The equipment docking and data adaptation module 100, serving as the interface component connecting the physical production line and the digital system, is logically divided into an equipment configuration parsing unit, a multi-protocol communication driver unit, a heterogeneous data capture and decryption unit, and a data standardization mapping unit. Through the collaborative work of these units, the equipment docking and data adaptation module 100 utilizes a software-defined approach to shield the hardware differences between AOI inspection equipment from different brands, achieving unified access and standardized processing of heterogeneous data.

[0044] The device configuration parsing unit is configured to read the device mapping configuration file during the system initialization phase. The device mapping configuration file is stored in a structured text format and records the physical and network attributes of all AOI inspection devices to be connected in the current production environment. The device configuration parsing unit parses the device mapping configuration file line by line, extracting the unique identifier, brand type field, network IP address, communication port number, and data exchange path for each AOI inspection device.

[0045] Based on the extracted brand type field, the device configuration parsing unit utilizes assembly reflection to dynamically locate and load the corresponding dynamic link library according to predefined interface specifications, thereby instantiating a communication driver adapted to the AOI detection device brand. Through this reflection loading mechanism, the device configuration parsing unit allows the system to extend its support for new device brands by adding driver plugins without modifying the main program code.

[0046] The multi-protocol communication driver unit has built-in adaptation interfaces for communication protocols from different manufacturers. For AOI inspection devices supporting the RS communication protocol, the multi-protocol communication driver unit establishes a long-lived Socket connection based on the TCP / IP protocol and starts a heartbeat keep-alive mechanism after the connection is established, periodically sending handshake data packets to maintain link stability. For AOI inspection devices using file exchange mode, the multi-protocol communication driver unit starts a file system listening service and uses the operating system's file handle hooks to monitor the preset shared folder path in real time. For AOI inspection devices supporting WebAPI, the multi-protocol communication driver unit instantiates an HTTP client and configures a polling timer or Webhook callback interface. Through the parallel operation of the above multiple driver modes, the multi-protocol communication driver unit realizes physical layer interconnection of heterogeneous devices from multiple brands on the production line.

[0047] The heterogeneous data capture and decryption unit is responsible for executing the specific data capture and preprocessing logic. When the multi-protocol communication driver unit detects an alarm signal output by the AOI inspection device or the generation of a new inspection file, the heterogeneous data capture and decryption unit immediately triggers the capture process. For file exchange mode, the heterogeneous data capture and decryption unit executes file exclusivity check logic, i.e., it attempts to open the target file in exclusive read-write mode. If opening fails, it enters a millisecond-level delay waiting loop until a valid file handle is obtained, preventing data corruption caused by reading the file before the AOI inspection device has completed writing. After successfully capturing data, the heterogeneous data capture and decryption unit calls a pre-built decryption algorithm library to handle the proprietary encryption processing of XML metadata files or image data by some manufacturers. For binary data streams using XOR encryption, the heterogeneous data capture and decryption unit performs bitwise operations to restore the data based on the key matrix provided by the manufacturer.

[0048] For data encrypted with AES symmetric encryption, the heterogeneous data capture and decryption unit loads a pre-stored key vector for decryption, thereby restoring the encrypted data to a readable plaintext XML structure and the original image stream. Subsequently, the heterogeneous data capture and decryption unit uses an XMLDOM parser to traverse the document tree, extracting PCB barcodes, component reference numbers, defect area coordinates, and original defect codes. Based on the extracted defect area coordinates, the heterogeneous data capture and decryption unit performs a local cropping operation on the original image stream, generating an image to be reviewed that only contains the defect area and a preset edge range.

[0049] The data standardization mapping unit is used to convert the raw data extracted by the heterogeneous data capture and decryption unit into a unified data format within the system, thereby eliminating differences in data dimensions between different devices. The data standardization mapping unit first performs score normalization processing. Because different brands of AOI inspection equipment have different scoring standards for defect probability, the data standardization mapping unit, based on the equipment brand attribute, calls the following normalization formula to normalize the raw inspection score output by the AOI inspection equipment. Mapped to standard confidence levels within the interval [0,1] :

[0050] ;

[0051] In the formula, The normalized standard confidence level; The raw inspection score output by the AOI inspection equipment; The theoretical minimum score defined in the device mapping configuration file; The theoretical maximum score defined in the device mapping configuration file.

[0052] In addition, the data standardization mapping unit maintains a global defect code mapping dictionary. This dictionary defines key-value pairs between vendor-specific error codes and system standard defect types. The data standardization mapping unit queries this dictionary, converts the extracted vendor-specific codes into a system-wide unified enumeration type, and encapsulates the normalized standard confidence level, the converted standard defect type, the location information in the unified coordinate system, and the cropped image to be reviewed into a standard data transmission object. The data standardization mapping unit serializes this standard data transmission object and stores it in a production line-specific cache for subsequent modules to access.

[0053] The review process execution module 200, as the core inference engine of the system, has an internal logical structure divided into an image preprocessing unit, a deep feature extraction unit, a classification inference unit, and an intelligent routing decision unit. Utilizing the serial processing mechanism of these units, the review process execution module 200 transforms standardized image data into detection conclusions with clear semantics, and drives subsequent control processes based on confidence levels.

[0054] The image preprocessing unit is configured to perform tensor transformation and data augmentation operations to adapt to the input layer requirements of the convolutional neural network model. The image preprocessing unit reads a standard data transmission object from the production line's dedicated buffer and extracts the image data to be reviewed. Due to differences in image resolution output by different AOI inspection devices, the image preprocessing unit performs bicubic interpolation to uniformly scale the image to be reviewed to a preset fixed size required by the convolutional neural network model. The image preprocessing unit performs channel normalization on the scaled image, subtracting the channel mean calculated based on the training dataset from the pixel values ​​of the RGB channels and dividing by the channel standard deviation to generate an input tensor conforming to a standard normal distribution. To enhance the model's adaptability to changes in illumination, the image preprocessing unit is configured to perform histogram equalization on the input tensor before inference to eliminate the influence of production line ambient light fluctuations on the image grayscale distribution.

[0055] The deep feature extraction unit is built upon a deep convolutional neural network architecture and is used to extract multi-dimensional semantic features from the input tensor. The deep feature extraction unit comprises multiple cascaded convolutional layers, activation layers, and pooling layers. In the shallow network portion, the deep feature extraction unit utilizes small-sized convolutional kernels to extract edge, texture, and scratch grayscale gradient features of the image. In the deep network portion, the deep feature extraction unit extracts light and shadow texture features characterizing the three-dimensional morphology of the solder joints by increasing the channel dimension and reducing the spatial resolution.

[0056] Since AOI light sources create specific reflected light spot distributions when illuminating solder surfaces with varying heights and curvatures, the depth feature extraction unit learns these spatial distribution patterns to infer the solder joint's creep height and wetting angle. The depth feature extraction unit then fuses these feature maps from different dimensions to generate a high-dimensional feature vector, which comprehensively characterizes the physical quality state of the component under test.

[0057] The classification inference unit is connected to the deep feature extraction unit and is used to calculate classification probabilities based on the high-dimensional feature vectors. The classification inference unit contains a fully connected layer and a Softmax classifier. It maps the high-dimensional feature vectors to the class space of the output layer, which contains probability values ​​corresponding to true and false defects. The classification inference unit calculates the defect probability value based on the probability distribution of the output layer. To suppress ambiguous decisions, the classification inference unit uses a weighted formula based on probability differences to calculate the final defect probability value. :

[0058] ;

[0059] In the formula, This is the corrected defect probability value; This represents the true defect probability value output by the model. The false defect probability value output by the model; This is the absolute value of the difference between the two probabilities; The preset confidence level penalty coefficient, The value ranges from 0.5 to 2.0 and is used to adjust the model's sensitivity to samples with ambiguous classification boundaries.

[0060] The intelligent routing decision unit is configured to execute a tiered control strategy based on classification results and defect probability values. The intelligent routing decision unit has pre-stored automatic release thresholds and manual review thresholds. The intelligent routing decision unit reads the classification labels and defect probability values ​​output by the classification reasoning unit and executes the following logical judgment steps:

[0061] Automatic release judgment:

[0062] If the classification label is a false defect and the defect probability value is greater than the automatic release threshold, the intelligent routing decision unit determines that the image to be re-evaluated is a false alarm from the AOI inspection equipment. The intelligent routing decision unit generates an OK release command and sends the OK release command to the production line control node through the device docking and data adaptation module 100. The production line control node then drives the transmission track to release the PCB board accordingly. The S210 process does not require manual intervention and achieves automatic filtering of high-confidence false alarms.

[0063] Manual route verification:

[0064] If the classification label is a true defect, or if the classification label is a false defect but the defect probability value is less than or equal to the automatic release threshold, the intelligent routing decision unit determines the result to be pending confirmation. The intelligent routing decision unit generates a review task package containing the image to be reviewed, classification suggestions, and defect probability values, and routes the review task package to the manual review and feedback module 400. Through this logic, the intelligent routing decision unit ensures that low-confidence decisions must be manually confirmed to prevent the risk of missed detections.

[0065] Abnormal data marker:

[0066] If the classification reasoning unit outputs and If the difference is less than the preset confusion threshold, it indicates that the convolutional neural network model cannot effectively distinguish defect features, and the intelligent routing decision unit marks the image to be reviewed as a difficult sample. While routing the difficult sample to the manual review and feedback module 400, the intelligent routing decision unit simultaneously sends the index information of the difficult sample to the data management and visualization module 600 for key annotation, so that the incremental model training and iteration module 500 can prioritize the extraction of this type of sample for model optimization in the future.

[0067] The confidence-based closed-loop control module 300 serves as a feedback processing component connecting the front-end equipment and the back-end intelligent algorithm. Its internal logic structure is divided into a trend monitoring and statistics unit, a drift judgment logic unit, a parameter gradient calculation unit, and a control command encapsulation unit. Through the serial processing of these units, the confidence-based closed-loop control module 300 transforms discrete single-step judgment results into continuous parameter optimization actions, thereby achieving dynamic adaptive adjustment of the process parameters of the AOI inspection equipment.

[0068] The trend monitoring and statistics unit is responsible for constructing time-series data windows for specific component reference numbers. It maintains a sliding window queue in memory based on a hash mapping, where the hash mapping key is a unique combination generated by concatenating the PCB identifier, component reference number, and defect type field. When the review process execution module 200 outputs each review result, the trend monitoring and statistics unit writes the result to the corresponding sliding window queue. The sliding window queue follows a first-in, first-out (FIFO) principle, retaining only the most recent preset number of historical records. Each historical record includes the original AOI detection score, the AI ​​review conclusion, and the AI ​​confidence score. The trend monitoring and statistics unit calculates in real-time the proportion of samples in the sliding window queue that are judged as NG by AOI and as false defects by AI, i.e., the false alarm rate.

[0069] The drift detection logic unit is configured to identify systematic deviations in device parameters. It periodically traverses the sliding window queue maintained by the trend monitoring statistics unit and executes the parameter drift detection algorithm. The drift detection logic unit determines that parameter drift is established only if the following three conditions are met simultaneously:

[0070] First, in the current sliding window queue, the false alarm rate of a specific component reference number exceeds the preset false alarm tolerance limit. Second, among all samples judged as false alarms, the weighted average of the AI ​​confidence scores is higher than the preset confidence threshold, indicating that the convolutional neural network model has extremely high certainty in judging false alarms. Third, the original AOI detection scores of the false alarm samples exhibit a normal distribution clustering characteristic, that is, the standard deviation of the original AOI detection scores is less than the preset dispersion threshold. When all three conditions are met simultaneously, the drift determination logic unit generates a parameter correction trigger signal for the specific component reference number.

[0071] The parameter gradient calculation unit is used to quantitatively calculate the adjustment step size of the AOI inspection equipment's detection parameters based on the degree of drift. Since the AOI inspection equipment uses a scoring logic where higher scores represent a greater likelihood of defects, to prevent missed detections due to large parameter fluctuations, the parameter gradient calculation unit employs a confidence-weighted damping correction algorithm. The parameter gradient calculation unit calculates the correction amount for the detection threshold according to the following formula. :

[0072] ;

[0073] In the formula, The suggested threshold adjustment increment is used to accumulate to the current threshold in order to raise the judgment threshold; The learning rate coefficient is a preset value, ranging from 0.01 to 0.1. The mean AI confidence score of false positive samples in the sliding window queue; The mean of the original AOI detection scores for the false alarm sample group; The current effective detection threshold for the AOI inspection equipment; This is a preset safety boundary margin, which is used to retain redundancy in detection sensitivity.

[0074] The control instruction encapsulation unit is responsible for converting the calculated parameter corrections into executable instructions. The control instruction encapsulation unit first processes the corrections output by the parameter gradient calculation unit. Perform a safety clamp check. The safety clamp check logic limits the modified target threshold to the absolute safety range allowed by the device. If the modified target threshold exceeds the absolute safety range, the control command encapsulation unit forcibly truncates the target threshold to the safety boundary value to prevent the parameter from being excessively relaxed, which could lead to the missed detection of true defects.

[0075] After passing the safety clamp check, the control command encapsulation unit constructs a control command package containing parameter writing instruction code, target component reference number, and new threshold parameters according to the protocol specifications provided by the device interface and data adaptation module 100. The control command encapsulation unit sends the control command package to the device interface and data adaptation module 100, which then completes the final parameter write-back operation.

[0076] The manual review and feedback module 400 serves as the interactive terminal for processing low-confidence data and executing the final decision. Its internal logical structure is divided into a review interface rendering unit, an interactive logic processing unit, a decision signal synchronization unit, and a sample differentiation labeling unit. Through the collaborative work of these units, the manual review and feedback module 400 ensures that when the convolutional neural network model is uncertain, manual judgment logic is introduced to correct the decision, while simultaneously controlling the physical actions of the production line and accumulating model training data.

[0077] The review interface rendering unit is configured to transform the data routed from the review process execution module 200 into a visual human-computer interaction view. The review interface rendering unit receives a review task package containing the image to be reviewed, AI classification suggestions, and confidence scores, and requests a standard good product image corresponding to the target component reference number from the data management and visualization module 600. The review interface rendering unit uses split-screen comparison rendering technology, drawing the current image to be reviewed on the left side of the display terminal and the standard good product image on the right side.

[0078] To assist operators in quickly locating defects, the review interface rendering unit overlays a highlighted vector border onto the image layer based on the coordinates of the defect area extracted from the image to be reviewed. The unit also dynamically adjusts the color of the highlighted vector border by querying a preset color map based on the type suggested by the AI ​​classification, and renders text labels next to the highlighted vector border to display the AI-predicted defect type and confidence level.

[0079] The interaction logic processing unit listens to input device actions and maps them to standard decision commands. It is bound to keyboard event listeners and mouse click event listeners. When an operator triggers a release operation, confirms a defective operation, or modifies a defect type via an input device, the interaction logic processing unit captures the corresponding input event.

[0080] To improve review efficiency, the interactive logic processing unit incorporates batch processing logic. When the review interface rendering unit simultaneously displays multiple images with the same component reference number and the same AI classification result, the interactive logic processing unit allows operators to perform a one-click select-all release operation. Upon receiving the adjudication instruction, the interactive logic processing unit immediately locks the status of the review task package to prevent duplicate submissions and transmits the adjudication result, including the final classification label and operation timestamp, to subsequent units.

[0081] The decision signal synchronization unit is responsible for converting the final decision made manually into control signals for the physical production line. The decision signal synchronization unit maintains a communication connection with the equipment interface and data adaptation module 100. When the interactive logic processing unit outputs a release decision command, the decision signal synchronization unit constructs a release command package containing a remote release operation code and a PCB identification code, and sends the release command package to the production line control node through the equipment interface and data adaptation module 100 to deactivate the alarm status of the AOI inspection equipment and drive the transmission track to continue running. When the interactive logic processing unit outputs a confirmation defect decision command, the decision signal synchronization unit constructs maintenance work order data containing the defect coordinates and the final defect type, sends the maintenance work order data to the production line maintenance station system, and simultaneously sends a confirmation defect signal to the AOI inspection equipment to trigger the automatic rejection mechanism or audible and visual alarm.

[0082] The sample differentiation labeling unit is configured to execute sample cleaning and value assessment logic. Upon receiving the final human decision, the sample differentiation labeling unit immediately retrieves the original AI inference result generated by the review process execution module 200. The sample differentiation labeling unit performs the following logical comparison operation:

[0083] If the final human decision aligns with the AI ​​classification suggestion, the sample differentiation labeling unit will label the image sample to be reviewed as a normal verification sample; if the final human decision differs from the AI ​​classification suggestion, the sample differentiation labeling unit will label the image sample to be reviewed as a high-value conflict sample. The sample differentiation labeling unit further analyzes the reasons for the conflict:

[0084] If the AI ​​confidence score is higher than the preset high confidence threshold but is still manually corrected, the sample differentiation labeling unit determines that the model parameters of the re-judgment process execution module 200 have overfitting or feature recognition bias, and adds the highest priority label to the image sample to be re-judged. The sample differentiation labeling unit encapsulates the image data and metadata with specific labels and sends them to the data management and visualization module 600 for storage, which is then used by the model training and iteration module 500 as the core dataset.

[0085] The model training and iteration module 500, as the core processing component of the system, establishes data connections with both the manual review and feedback module 400 and the re-judgment process execution module 200. The model training and iteration module 500 is configured to perform closed-loop sample management, automatically collecting true defect images confirmed by the manual review and feedback module 400, as well as sample images where the re-judgment classification results differ from the final manual decision. The model training and iteration module 500 uses the collected image samples to incrementally train the convolutional neural network model in the re-judgment process execution module 200, updating the model weight parameters, and distributing the updated model to the re-judgment process execution module 200 to adapt to changes in production processes. To achieve adaptive model updates, the internal logical structure of the model training and iteration module 500 is specifically divided into a sample screening and storage unit, a training task scheduling unit, a weight update calculation unit, and a model version distribution unit.

[0086] The sample selection and storage unit is responsible for extracting image samples with high training value from the data stream output by the manual review and feedback module 400. This unit monitors the operation logs of the manual review and feedback module 400, capturing two types of key data in real time: the first type is confirmed NG samples, which are images ultimately judged as true defects by humans; these NG samples contain the current actual defect morphology on the production line. The second type is conflict correction samples, which are images whose judgment results from the review process execution module 200 have been manually modified; these conflict correction samples reflect the feature recognition bias of the current model. The sample selection and storage unit performs format verification on the confirmed NG samples and conflict correction samples, extracts the image data and corresponding human confirmation labels, and encapsulates the image data and human confirmation labels into a standard training sample package. The sample selection and storage unit stores the training sample package in a dedicated incremental dataset database and assigns a unique version index number to each training sample.

[0087] The training task scheduling unit triggers the model optimization process based on sample accumulation. It monitors the number of samples in the incremental dataset database in real time. When the number of new samples reaches a preset training threshold of, for example, 1000, or when a preset periodic update time is reached, the training task scheduling unit automatically generates a training task sheet. It then retrieves the currently running convolutional neural network model from the review process execution module 200 as the baseline model and loads the latest training sample package from the incremental dataset database. To prevent the model from forgetting old knowledge while learning new samples, the training task scheduling unit also extracts historical core samples according to a preset ratio, mixing these historical core samples with the new samples to construct a balanced training set.

[0088] The weight update computation unit is configured to execute a specific backpropagation algorithm to optimize model parameters. The weight update computation unit loads the baseline model and a balanced training set, utilizing a graphics processing unit (GPU) to accelerate the computation process. It inputs the balanced training set into the baseline model and calculates the loss function value between the model output and the manually confirmed labels. Based on the loss function value, the weight update computation unit calculates the gradient and fine-tunes the weight parameters of the convolutional neural network model using either stochastic gradient descent or adaptive moment estimation (IME) optimization algorithms. During training, the weight update computation unit monitors the accuracy metric of the validation set in real time. When the accuracy of the validation set no longer improves for a preset number of epochs, the weight update computation unit stops training and saves the updated weight parameters as a candidate model file.

[0089] The model version distribution unit is responsible for deploying validated candidate models to the production environment. The unit first performs integrity checks and inference speed tests on the candidate model files to ensure they meet production line cycle time requirements. After successful verification, the unit sends the candidate model files to the review process execution module 200 via the internal network. The unit then sends a hot update command to the review process execution module 200, instructing it to load new weight parameters during idle time slots in the detection task.

[0090] The model version distribution unit also records the version number and change log of this update, so that it can quickly roll back to the previous stable version in case of anomalies. Through the deployment operation of the model version distribution unit, the model training and iteration module 500 enables the detection system to automatically adapt to changes in material batches and fluctuations in process parameters without interrupting production.

[0091] The data management and visualization module 600 serves as the system's data storage and human-computer interaction display layer, establishing bidirectional data pathways with all other functional modules within the system. The data management and visualization module 600 is configured to perform persistent storage of detection data, multi-dimensional statistical analysis, and visualization report generation. The internal logical structure of the data management and visualization module 600 is specifically divided into a multi-level storage management unit, an index building and retrieval unit, a statistical analysis engine unit, and a visualization dashboard rendering unit.

[0092] The multi-level storage management unit (MLM) executes a tiered storage strategy based on data value and access frequency to balance storage costs and access efficiency. The MLM is equipped with a high-speed solid-state drive (SSD) cache and a large-capacity hard disk drive (HDD) archive. For real-time inspection images and review results generated within the most recent preset time period, the MLM writes these images and results to the high-speed SSD cache to support quick querying and review by production line operators. For historical data exceeding the preset time period, the MLM performs hot and cold data separation logic, migrating the original image data to the large-capacity HDD archive or cloud object storage service, while retaining only metadata including PCB identification codes, component reference numbers, judgment results, and confidence levels in the local database. The MLM also has a built-in automatic cleanup strategy. When storage space occupancy exceeds a preset warning threshold, the MLM prioritizes deleting normal, good product images that are not marked as high-value and have been stored for the longest time, ensuring the long-term retention of critical defect data.

[0093] The index building and retrieval unit is responsible for establishing an efficient query entry point for massive amounts of unstructured image data. While writing data to the storage medium, the unit extracts key fields to construct a multi-dimensional inverted index. Key fields include the PCB globally unique identifier, production batch number, component reference number, defect type, and judgment timestamp. The index building and retrieval unit allows external modules to initiate query requests using any combination of key fields. For example, when the production line traceability system needs to query the inspection records of all components on a specific PCB board, the index building and retrieval unit quickly locates all associated image file paths and judgment details using the PCB globally unique identifier and returns the query results in the form of a structured list.

[0094] The statistical analysis engine unit is configured to perform statistical calculations on accumulated inspection data to reveal fluctuation trends in production line process quality. The unit periodically extracts inspection records from the database and calculates key quality indicators (KPIs). These KPIs include the top ranking of defect rates for specific component reference numbers, first-pass yield trends over different time periods, and the rejection rate of AI-based re-judgments relative to the original AOI judgments. The unit also pays particular attention to the distribution patterns of false defects. By aggregating and analyzing frequent false alarms of a certain type of component at specific locations, it identifies problems with improper AOI equipment light source or programming parameter settings and generates process improvement suggestion reports.

[0095] The visualization dashboard rendering unit transforms statistical data into intuitive charts for management. It utilizes web technology to build an interactive dashboard interface. The unit prominently displays a real-time inspection overview, showing the current proportions of good products, true defects, and false defects through pie charts. It also employs heatmap technology to display the spatial distribution of defects on PCB boards, marking high-frequency defect areas with dark blocks to help process engineers intuitively identify regional quality issues caused by uneven reflow oven temperatures or abnormal pick-and-place machine nozzles. Furthermore, the unit supports drill-down analysis; users can click on specific areas on the heatmap to view a list of corresponding original defect images and a histogram showing the confidence distribution of AI-generated defects.

Claims

1. A deep learning-based intelligent AOI (Automated Optical Inspection) device review and control system, characterized in that, include: The equipment docking and data adaptation module is configured to establish a communication connection with heterogeneous AOI inspection equipment on the production line, capture inspection data packets containing NG images and metadata, convert the inspection data packets into a unified standardized data object within the system, and then send them to the production line's dedicated buffer area. The re-judgment process execution module is configured to obtain the standardized data object from the production line-specific cache area, perform inference using a pre-trained convolutional neural network model, output the re-judgment classification result and defect probability value, and execute a diversion strategy of automatic release or routing to manual review based on the defect probability value. The confidence-based closed-loop control module is configured to monitor the judgment difference between the AOI detection device and the re-judgment process execution module. When a drift trend is detected in the parameters of the AOI detection device, the detection threshold correction amount is calculated and the physical detection parameters of the AOI detection device are adjusted in reverse. The manual review and feedback module is configured to receive suspected defect data sent by the review process execution module, provide a human-computer interaction interface to obtain the final manual decision result, and generate tagged sample data. The model training and iteration module is configured to update the convolutional neural network model using the labeled sample data and distribute the updated model parameters to the re-judgment process execution module. The re-judgment process execution module includes a deep feature extraction unit, a classification reasoning unit, and an intelligent routing decision unit; The deep feature extraction unit is configured to extract edge texture features and high-dimensional semantic features of the image using multi-layer cascaded convolutional layers; The classification reasoning unit is configured to map the high-dimensional semantic features into true defect probability and false defect probability, and calculate the corrected defect probability value based on the probability difference between the two. The intelligent routing decision unit is configured to compare the defect probability value with the automatic release threshold. When it is determined to be a false defect and the defect probability value is higher than the automatic release threshold, a release instruction is generated; otherwise, manual routing logic is triggered. The classification reasoning unit is configured to execute the following logic to calculate the defect probability value: The larger of the true defect probability and the false defect probability output by the model is selected as the baseline probability; the absolute value of the difference between the true defect probability and the false defect probability is calculated; an exponential operation based on a preset penalty coefficient is performed on the absolute value to reduce the weight of samples with fuzzy classification boundaries; the result of the exponential operation is multiplied by the baseline probability to obtain the final defect probability value.

2. The intelligent AOI device review and control system based on deep learning according to claim 1, characterized in that, The device interface and data adaptation module includes a device configuration parsing unit, a heterogeneous data capture unit, and a data standardization mapping unit. The device configuration parsing unit is configured to load the device mapping configuration file and dynamically load the corresponding communication driver according to the brand identifier of the AOI detection device. The heterogeneous data capture unit is configured to parse the detection data packet, decode non-standard format image data, and extract the original detection score and defect code; The data standardization mapping unit is configured to map the original detection score to a standard detection confidence level with a uniform value range, and to convert the defect code into a system-unified defect enumeration type to generate the standardized data object.

3. The intelligent AOI device review and control system based on deep learning according to claim 2, characterized in that, The data normalization mapping unit is configured to perform a normalization operation with the following logic to generate the standard detection confidence: Obtain the theoretical minimum score and theoretical maximum score defined in the device mapping configuration file; calculate the difference between the original detection score and the theoretical minimum score, and use it as the numerator; Calculate the difference between the theoretical maximum score and the theoretical minimum score, and use it as the denominator; Divide the numerator by the denominator to obtain the normalized value, which is then used as the standard detection confidence level.

4. The intelligent AOI device review and control system based on deep learning according to claim 1, characterized in that, The confidence-based closed-loop control module includes a trend monitoring and statistics unit, a drift determination logic unit, and a parameter gradient calculation unit. The trend monitoring and statistics unit is configured to maintain a statistical queue based on a time sliding window. For the target component reference number, data that is judged as NG by the AOI inspection device but as a false defect by the re-judgment process execution module is marked as false alarm samples, and the characteristics of the false alarm samples are statistically analyzed. The drift determination logic unit is configured to determine whether the parameter drift condition is met based on the false alarm rate in the statistical queue, the mean defect probability value corresponding to the false alarm sample, and the dispersion of the original AOI detection score. The parameter gradient calculation unit is configured to calculate the detection threshold correction amount of the AOI detection device when it is determined that the parameter drift condition is met.

5. The intelligent AOI device review and control system based on deep learning according to claim 4, characterized in that, The drift determination logic unit is configured to determine the existence of parameter drift when the following three conditions are met simultaneously: The frequency of false alarm samples appearing within the sliding window exceeds a preset upper limit; The weighted average of the defect probability values ​​corresponding to the false alarm samples is higher than a preset confidence threshold; The standard deviation of the original AOI detection score corresponding to the false alarm sample is less than the preset dispersion threshold.

6. The intelligent AOI device review and control system based on deep learning according to claim 5, characterized in that, The parameter gradient calculation unit is configured to perform the following logical calculation to determine the detection threshold correction amount: Calculate the difference between the mean original AOI detection score of the false alarm samples and the detection threshold currently in effect of the AOI detection device; use a preset learning rate coefficient and the mean defect probability value of the false alarm samples to perform a weighted calculation on the difference; The weighted calculation result is superimposed with a preset safety boundary margin to obtain the detection threshold correction amount.

7. The intelligent AOI device review and control system based on deep learning according to claim 1, characterized in that, The manual review and feedback module includes an interactive logic processing unit and a sample differentiation labeling unit; The interactive logic processing unit is configured to respond to the operator's confirmation command and drive the physical production line action through the decision signal synchronization interface. The sample differentiation labeling unit is configured to compare the final human decision result with the re-judgment classification result in real time. When the final human decision result is inconsistent with the re-judgment classification result, the corresponding image sample is labeled as a label difference sample, and the feature map data corresponding to the re-judgment classification result is extracted, packaged together and sent to the model training and iteration module.

8. The intelligent AOI device review and control system based on deep learning according to claim 1, characterized in that, The model training and iteration module includes a sample balancing and screening unit and a model version hot update unit; The sample balancing and screening unit is configured to extract confirmed NG samples and label difference samples marked by the manual review and feedback module from the data stream of the manual review and feedback module to construct an incremental dataset, and to extract anchor samples from the historical benchmark sample library according to a preset ratio and mix them with the incremental dataset; the model version hot-update unit is configured to verify the trained model file and monitor the task load of the review process execution module, and perform hot-loading of model weights only during idle time gaps when the detection task queue is empty.

Images