SMT patch multi-defect real-time detection system and method

Abstract

The invention discloses an SMT patch multi-defect real-time detection system and method. The SMT patch multi-defect real-time detection system comprises an image acquisition unit, an algorithm reasoning unit, a software and hardware cooperation module and a hardware execution unit. The algorithm reasoning unit adopts a double-layer distillation model, a main network receives SMT process semantic features and control association semantic features, and sub-networks synchronously output defect detection results and pre-calculation control parameters through knowledge distillation. The software and hardware cooperation module comprises a cooperation triggering unit and a reverse calibration unit, parallel processing of algorithm reasoning and hardware control is achieved, and control parameters are corrected through real-time working condition data feedback. Through a model training and reasoning-precomputation parallel mechanism guided by process semantics, the technical problems that a traditional lightweight model is insufficient in precision and high in system delay are solved, and the real-time detection requirement of a high-speed SMT production line is met.

Description

Technical Field

[0001] This invention relates to the field of quality inspection and control technology in surface mount technology (SMT) production lines, and particularly to a real-time detection system and method for multiple defects in SMT components. Background Technology

[0002] Surface mount technology (SMT) is widely used in the electronics manufacturing industry, and its production process involves multiple stages such as component placement, soldering, and defect detection. With the increasing level of production automation, real-time detection and defect handling during the placement process have become key aspects to ensure product quality and production efficiency.

[0003] In actual production, SMT placement equipment typically uses image acquisition systems to inspect solder joints and components to identify soldering defects or placement misalignments. However, due to the diversity of different process parameters, component package types, and pad topologies, traditional inspection systems lack generalization performance in complex scenarios, often requiring frequent manual calibration or reliance on empirical parameters, resulting in unstable inspection accuracy and affecting the efficiency of automated control.

[0004] Furthermore, in the patch detection and control process, there are often problems of low coupling and high feedback latency between the algorithm model and the hardware execution unit. The detection results output by the algorithm cannot be converted into hardware control instructions in a timely manner, or there is a lack of real-time correction mechanism after execution, making it difficult to cope with the accumulation of errors caused by changes in operating conditions in a dynamic production environment.

[0005] Therefore, how to achieve high-precision image detection and control parameter co-generation in SMT placement scenarios, and improve the linkage and adaptability between algorithm inference results and hardware execution actions, has become an urgent technical problem to be solved. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide a real-time detection system and method for multiple defects in SMT (Surface Mount Technology) components, which significantly improves the detection accuracy, control response speed, and system adaptability of SMT components, and realizes an efficient, intelligent, and transferable integrated solution for component defect detection and control.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a real-time detection system for multiple defects in SMT (Surface Mount Technology) components, comprising: Image acquisition unit, used to acquire RGB images of SMT components; An algorithm inference unit, connected to the image acquisition unit, is used to receive and process the RGB image. The algorithm inference unit includes: The main network is used to receive SMT process semantic features and control-related semantic features, and to train them through a composite loss function to output trained model parameters. The SMT process semantic features include pad topology, component package grayscale distribution and standard solder joint shape, and the control-related semantic features include the center coordinate offset range of the defect region and the defect area threshold. A sub-network, connected to the main network, migrates data from the main network through double-layer distillation, is used to receive the RGB image and synchronously output defect detection results and pre-calculated control parameters. The defect detection results include defect category, coordinates, and confidence level, and the pre-calculated control parameters include the feeding coordinates, robotic arm correction angle, or laser marking range required for hardware execution. The software-hardware co-operation module, which connects the algorithm inference unit and the hardware execution unit respectively, includes: The collaborative triggering unit is used to synchronously send the pre-calculated control parameters to the hardware execution unit in a lightweight data frame format when the sub-network outputs the defect detection result; The reverse calibration unit is used to receive real-time operating condition data fed back by the hardware execution unit, and to correct the pre-calculated control parameters in real time based on the SMT process semantic features migrated in the sub-network, so as to generate corrected pre-calculated control parameters. The hardware execution unit, connected to the hardware-software co-operation module, is used to perform sorting or marking operations based on the pre-calculated control parameters or the modified pre-calculated control parameters.

[0008] Furthermore, the composite loss function includes a feature alignment loss term, a context consistency loss term, and a control semantic consistency loss term; The feature alignment loss term is used to constrain the consistency of the distribution of the features output by the main network with the standard features in the multidimensional space; The context consistency loss term is used to maintain the spatial correlation between adjacent feature points during feature extraction; The control semantic consistency loss term is used to ensure that the defect geometric features output by the main network match the hardware control parameter thresholds in the control association semantic features; The composite loss function integrates the above three losses through a weighted summation method, wherein the weighting coefficients are dynamically adjusted according to the process requirements of the SMT production line.

[0009] Furthermore, the composite loss function is expressed as: ; in, This is the total loss value. These are the weight coefficients for the feature alignment loss. These are the weighting coefficients for the context consistency loss. To control the weighting coefficients of semantic consistency loss, As the normalization factor, For integration variables, The shape parameter of the Gamma function. For the Gamma function, It is a natural constant. Let be the error function. For the model at position Output eigenvalues To mark the location eigenvalues, The scaling parameter for feature differences. For the summation index, This represents the total number of dimensions of the context features. For a Bessel function of the first kind, Scaling factor for context features For the first Dimensional context feature values, For the parameters of the Gamma function, To control the attenuation coefficient of parameter differences, To control the difference in parameters.

[0010] Furthermore, the SMT process semantic features are obtained in the following ways: Based on the image segmentation model, feature information such as pad topology, component package grayscale distribution and standard solder joint morphology is extracted from SMT patch images. The control-related semantic features are obtained by importing SMT production line process files, including the center coordinate offset range of the defect area and the defect area threshold. The main network simultaneously receives the SMT process semantic features and the control association semantic features as input.

[0011] Furthermore, the sub-network is constructed based on a lightweight convolutional neural network, with fewer than 3M parameters and less than 50M FLOPs of computation. The sub-network is configured to synchronously output defect detection results, including defect category, coordinates and confidence level, as well as pre-calculated control parameters, including material replenishment coordinates, robotic arm correction angle or laser marking range, within a preset time period after receiving the RGB image.

[0012] Furthermore, the cooperative triggering unit is configured to perform the following parallel operations: When the sub-network begins to output the defect detection result, the defect detection result is immediately pushed to the system display terminal; Meanwhile, the pre-calculated control parameters are encapsulated into a lightweight data frame in JSON format, and the data size of the lightweight data frame is less than 1KB. The lightweight data frames are sent to the hardware execution unit in real time via industrial Ethernet.

[0013] Furthermore, the reverse calibration unit is configured as follows: Receive real-time operating data fed back by the hardware execution unit, including the current position error of the robotic arm and the actual power of the laser marker; The SMT process semantic features migrated in the sub-network are invoked, including the mechanical compatibility threshold of the component package and the laser tolerance power of the solder joint; When the real-time operating data exceeds the fault tolerance range defined by the SMT process semantic features, the pre-calculated control parameters are dynamically corrected.

[0014] Furthermore, the hardware execution unit includes: A six-axis sorting robot arm is configured to perform sorting or correction operations based on the replenishment coordinates or robot arm correction angles in the pre-calculated control parameters, with a repeatability of ±0.02mm. The ultraviolet laser marking device is configured to perform marking operations according to the laser marking range in the pre-calculated control parameters, with a marking accuracy of 0.01 mm.

[0015] Furthermore, the image acquisition unit includes a 20-megapixel industrial camera and a telecentric lens, configured to acquire the RGB image at a frame rate of 120fps; The algorithm inference unit is implemented based on an edge computing device; The image acquisition unit, algorithm inference unit, and hardware execution unit are connected via EtherCAT industrial Ethernet with a transmission rate of 1Gbps and an end-to-end latency of less than 1ms.

[0016] A real-time detection method for multiple defects in SMT (Surface Mount Technology) components, applied to the aforementioned real-time detection system for multiple defects in SMT components, includes: Step S1: The image acquisition unit acquires the RGB image of the SMT patch; Step S2: The main network receives SMT process semantic features and control-related semantic features, and trains them using a composite loss function to output trained model parameters. The SMT process semantic features include pad topology, component package grayscale distribution, and standard solder joint shape. The control-related semantic features include the center coordinate offset range of the defect region and the defect area threshold. Step S3: The sub-network migrates data from the main network through double-layer distillation, receives the RGB image, and synchronously outputs defect detection results and pre-calculated control parameters. The defect detection results include defect category, coordinates, and confidence level. The pre-calculated control parameters include the feeding coordinates, robotic arm correction angle, or laser marking range required for hardware execution. Step S4: When the sub-network outputs the defect detection result, the collaborative triggering unit synchronously sends the pre-calculated control parameters to the hardware execution unit in a lightweight data frame format. Step S5: The reverse calibration unit receives real-time operating condition data fed back by the hardware execution unit, and performs real-time correction on the pre-calculated control parameters based on the SMT process semantic features migrated in the sub-network, so as to generate the corrected pre-calculated control parameters. Step S6: The hardware execution unit performs sorting or marking operations according to the pre-calculated control parameters or the modified pre-calculated control parameters.

[0017] The beneficial effects of this invention are: This invention achieves intelligent and high-precision linkage between SMT chip detection and control through the collaborative design of an image acquisition unit, an algorithm inference unit, and a hardware-software co-design module, resulting in the following beneficial effects: 1. Improve the accuracy and robustness of defect detection: This invention introduces SMT process semantic features and control association semantic features into the main network of the algorithm inference unit, and uses a composite loss function for training, so that the model can maintain stable recognition performance under different mounting process conditions, thereby significantly improving the accuracy and generalization ability of defect detection.

[0018] 2. Synchronous generation of detection results and control parameters: The sub-network of this invention transfers knowledge from the main network through a two-layer distillation mechanism. While performing defect detection on RGB images, it can simultaneously output the pre-calculated control parameters required for hardware execution, such as material replenishment coordinates, robotic arm correction angle, or laser marking range. This reduces the instruction delay between the algorithm and the execution layer and improves the system response speed.

[0019] 3. Real-time coordinated control of hardware and software: The present invention sets up a hardware and software coordination module. The coordination triggering unit can synchronously send lightweight control data frames to the hardware execution unit during the detection phase, realizing parallel triggering of detection and execution. The reverse calibration unit can dynamically correct the control parameters according to the real-time operating data fed back by the hardware, so that the control process has the ability to adapt and adjust, significantly improving the execution accuracy and stability of the system.

[0020] 4. Reduce manual intervention and improve automation level: Through the data closed loop between the algorithm inference unit and the hardware execution unit, the system can automatically complete defect identification, control parameter generation and correction without frequent manual calibration or manual adjustment of process parameters, thereby reducing the cost of manual intervention and improving the intelligence level of the chip assembly line.

[0021] 5. Achieving multi-source information fusion and model portability: The two-layer distillation mechanism between the main network and sub-networks enables the model to have knowledge transfer capabilities, allowing it to quickly adapt to different equipment or process conditions, improving the algorithm's versatility and scalability, and providing a technical foundation for the collaborative detection and control of multiple types of patch devices.

[0022] In summary, this invention significantly improves the accuracy of SMT chip detection, control response speed, and system adaptability through a hardware-software co-architecture and semantic feature-driven algorithm reasoning mechanism, achieving an efficient, intelligent, and transferable integrated solution for chip defect detection and control. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the SMT patch multi-defect real-time detection system of the present invention; Figure 2 This is a flowchart of the steps in the real-time detection method for multiple defects in SMT patch panels in this invention.

[0024] Reference numerals: 1. Image acquisition unit; 11. Industrial camera; 12. Telecentric lens; 2. Algorithm inference unit; 21. Main network; 22. Sub-network; 3. Hardware-software co-operation module; 31. Co-operation triggering unit; 32. Reverse calibration unit; 4. Hardware execution unit; 41. Six-axis sorting robot arm; 42. Ultraviolet laser marking device. Detailed Implementation

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Identical components are denoted by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "upper," and "lower" used in the following description refer to directions in the accompanying drawings, and the terms "bottom surface," "top surface," "inner," and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0026] Example 1, referring to Figure 1 This is the first embodiment of the present invention. This embodiment provides a real-time detection system for multiple defects in SMT (Surface Mount Technology) components, including: The system comprises an image acquisition unit 1, an algorithm inference unit 2, a hardware-software co-processing module 3, and a hardware execution unit 4. These units interact with each other via industrial Ethernet or a high-speed bus to transmit control commands. The entire system can be deployed on a standard SMT production line, working in conjunction with pick-and-place machines, reflow soldering machines, and vision inspection equipment to achieve defect detection, location, and automatic correction during the placement process.

[0027] System composition and specific implementation method: (1) Image acquisition unit 1 is used to acquire RGB images during the SMT placement process.

[0028] This unit can be composed of an industrial camera 11, a light source control module, and a vision acquisition control card. The industrial camera 11 uses a high-resolution CMOS sensor with a frame rate of no less than 60 frames per second, and can capture images of the chip position, pad shape, and component orientation in real time. The light source module uses a ring-shaped adjustable light source to suppress reflections and shadows, ensuring uniform image grayscale.

[0029] The acquired RGB images are transmitted to the algorithm inference unit 2 via a gigabit network port through the acquisition control card, realizing high-speed image stream input.

[0030] Technical effect: High-resolution, low-latency image acquisition enables subsequent algorithms to perform high-precision inference based on accurate reconstruction of solder joint topology and grayscale features, thereby improving the effectiveness and stability of detection data.

[0031] (2) Algorithm inference unit 2, connected to the image acquisition unit 1, is used to receive and process RGB images to realize defect detection and control parameter generation. This unit includes two parts: a main network 21 and a sub-network 22, and can be deployed in a GPU-accelerated server or an edge AI inference box.

[0032] Main Network 21: Main Network 21 is constructed based on a deep convolutional neural network structure, and its inputs include SMT process semantic features and control-related semantic features.

[0033] Among them, the SMT process semantic features include pad topology, component package grayscale distribution and standard solder joint shape; the control-related semantic features include the center coordinate offset range of the defect region and the defect area threshold.

[0034] The main network 21 is trained end-to-end using a composite loss function, which comprehensively considers localization error, classification error and process semantic matching error, thereby maintaining the stability of the model output under different mounting process conditions.

[0035] Subnetwork 22: Subnetwork 22 transfers knowledge from main network 21 through a two-layer distillation mechanism, which is used in... Technical results: Through the hierarchical training and double-layer distillation mechanism of the main network 21 and the sub-network 22, the model has strong generalization ability under different working conditions. It can directly generate control parameters while detecting defects, avoiding secondary calculations and manual parameter configuration, and significantly improving inference speed and system adaptability.

[0036] (3) The hardware-software co-operation module 3 is connected to the algorithm inference unit 2 and the hardware execution unit 4 respectively, and is used to realize synchronous triggering and adaptive correction between algorithm output and physical control. This module includes: Collaborative triggering unit 31: When the sub-network 22 outputs the defect detection result, it sends the corresponding pre-calculated control parameters to the hardware execution unit 4 in a lightweight data frame format (e.g., UDP compressed frame) to ensure seamless connection between the detection and execution processes.

[0037] Reverse calibration unit 32: Receives real-time operating condition data (including placement position deviation, material replenishment execution status, temperature and illumination parameters, etc.) fed back by hardware execution unit 4, and corrects the pre-calculated control parameters based on the SMT process semantic features migrated in sub-network 22 to generate corrected control parameters.

[0038] Technical effects: The collaborative triggering unit 31 realizes real-time parallel detection and execution, reducing signal delay; the reverse calibration unit 32 forms a closed-loop feedback mechanism from algorithm to execution, ensuring that control commands are dynamically adjusted according to operating conditions, thereby significantly improving the system's response accuracy and production consistency.

[0039] (4) Hardware execution unit 4, connected to the hardware-software co-operation module 3, is used to perform actions such as feeding, robotic arm adjustment or laser marking. This unit may consist of a robotic arm control module, a vision positioning module, an execution end (suction nozzle or laser head) and a motion control driver.

[0040] The robotic arm control module performs precise pose adjustments based on the received pre-calculated control parameters or correction parameters to compensate for patch offset; the laser module can mark or sort in a designated area based on the coordinates of the defect location.

[0041] Technical effect: The hardware execution unit 4 realizes real-time parameter access and action response through the hardware-software co-operation module 3, which can quickly perform compensation or marking after defect identification, reduce production downtime, and improve the continuity and automation level of the chip assembly line.

[0042] Working principle of Example 1: The system in this embodiment achieves end-to-end automation of SMT surface mount defect detection, control parameter generation, and hardware execution through a deep coupling design of "algorithm layer (process semantic-guided two-layer distillation model) + system layer (inference-pre-computation hardware-software collaboration)". The system forms a closed-loop feedback at the algorithm end and the control end, enabling the detection results and control actions to be triggered in parallel in time, thereby achieving a dual improvement in detection accuracy and execution efficiency.

[0043] (I) Algorithm layer: Two-layer distillation model guided by process semantics and control requirements The algorithm layer is the core intelligent part of this system, consisting of a main network 21 and a sub-network 22. This structure achieves the functional integration of "accurate defect identification + synchronous generation of control parameters" through the method of "knowledge transfer from the main network 21 - functional expansion of the sub-network 22".

[0044] Main Network 21: Dual Semantic Feature Enhancement Training Mechanism The main network 21 adopts a deep convolutional neural network framework based on ResNet-50, using two types of core semantic features from the SMT domain as input: Basic process semantics: This includes inherent parameters of the SMT process, such as pad topology (e.g., 1.0mm × 0.5mm pad size for 0402 packages), component package grayscale distribution (e.g., grayscale difference threshold between ceramic capacitors and PCB substrate), and standard solder joint morphology (typical "half-moon" structure). These semantics are extracted by SAM (Segment Anything Model) to form a feature map.

[0045] Control-related semantics: Geometric and physical parameters directly associated with downstream execution control, including the center coordinate offset range of the defect region (e.g., ±0.1mm) and defect area thresholds (e.g., the minimum area of ​​a bridging defect is 0.02mm). 2 ).

[0046] The main network 21 is trained using a composite loss function. To control the semantic consistency loss, the deviation between the defect geometric features output by the main network 21 and the control parameter threshold is calculated to ensure that the main network 21 has the ability to generate control parameters while recognizing defects.

[0047] Technical effect: The main network 21 can learn "defect semantics + control logic" simultaneously during the training phase, providing high-precision semantic support for the real-time inference of the subsequent sub-network 22, making the model more domain-adaptive and control-consistent.

[0048] Subnetwork 22: Lightweight Inference and Pre-computed Control Parameter Generation Subnetwork 22 adopts an improved MobileNetV4 structure (parameter count <3M, computational cost <50MFLOPs) and transfers bi-semantic knowledge from the main network 21 through a two-layer distillation mechanism.

[0049] During the inference phase, subnetwork 22 can simultaneously output the following after receiving the SMT patch RGB image: Defect detection results include defect type (such as missing parts, misalignment, poor soldering, bridging, etc.), coordinates (accuracy to 0.01mm), and confidence level (≥0.95). Pre-calculated control parameters include feeding coordinates (x±0.05mm, y±0.05mm), robotic arm correction angle (θ≈3°), and laser marking range (0.3mm×0.1mm).

[0050] The output parameter data size is less than 1KB, and it can be directly transmitted to the downstream control system.

[0051] Technical effect: This structure realizes an integrated mechanism of "inference output is control parameter", which significantly reduces the latency in the traditional detection → calculation → execution link (average reduction of 20-30ms) and ensures that the algorithm output and hardware execution are triggered synchronously.

[0052] (II) System Layer: Inference-Pre-computation Parallel Software-Hardware Co-working Module 3 The system layer consists of a collaborative triggering unit 31 and a process semantic reverse calibration unit 32, which are responsible for realizing real-time collaboration and adaptive correction between algorithm output and hardware actions.

[0053] Collaborative triggering unit 31: When subnetwork 22 outputs preliminary defect detection results (without waiting for complete batch inference), this unit executes immediately: The test results are pushed to the system display for manual review; The pre-calculated control parameters are sent in real time to the hardware execution unit 4 (such as a robotic arm controller or laser marker driver) in a lightweight JSON data frame, triggering the downstream pre-calculation process (trajectory planning, laser power setting, etc.).

[0054] Technical effect: Through the parallel mechanism of "inference and control at the same time", the overall system response time is reduced by more than 25%, and the single frame latency is less than 50ms, which meets the real-time requirements of high-speed SMT production lines with a speed of 85,000 CPH or more.

[0055] Reverse calibration unit 32: To avoid mismatch between pre-calculated parameters and actual working conditions (such as execution errors caused by changes in the robot arm load), this unit introduces a two-way feedback mechanism: The hardware execution unit 4 feeds back the execution status (position error, laser power, etc.) to the calibration module in real time; The reverse calibration unit 32 calls the process semantics (such as pad tolerance ±0.05mm, laser power threshold) migrated by the sub-network 22 for dynamic correction.

[0056] For example, when the robotic arm detects a position error >0.03mm, the system automatically adjusts the correction angle from 3° to 3.2° based on the tolerance range of the 0402 package pads to ensure the final mounting accuracy.

[0057] Technical benefits: Establishing a closed-loop feedback channel between the algorithm and hardware enables real-time correction of control parameters, improves the system's adaptability and robustness to complex working conditions, and ensures long-term stable operation of the production line.

[0058] (III) System Operation Process Image acquisition: Industrial camera 11 acquires RGB images of the patch area in real time and transmits them to the edge computing device.

[0059] Algorithm inference and pre-computation are executed in parallel: Sub-network 22 completes image inference in 0.0067 seconds and outputs defect results and control parameters; Cooperative triggering unit 31 simultaneously sends data frames to hardware execution unit 4.

[0060] Hardware execution and feedback: The robotic arm completes trajectory planning and execution within 0.04 seconds and provides feedback on execution error data; the reverse calibration unit 32 determines whether the deviation exceeds the limit, and if it does, it corrects the control parameters in real time.

[0061] Result verification and closed-loop update: The system periodically compares the detection results with the execution feedback, updates the semantic feature mapping, and achieves continuous optimization.

[0062] Technical effects of Example 1: This embodiment achieves intelligent and high-precision linkage of SMT chip detection and control through the collaborative design of image acquisition unit 1, algorithm inference unit 2, and hardware-software co-operation module 3, and has the following beneficial effects: 1. Improve the accuracy and robustness of defect detection: By introducing SMT process semantic features and control association semantic features through the main network 21 in the algorithm inference unit 2, and using a composite loss function for training, the model can maintain stable recognition performance under different mounting process conditions, thereby significantly improving the accuracy and generalization ability of defect detection.

[0063] 2. Synchronous generation of detection results and control parameters: Sub-network 22 transfers knowledge from main network 21 through a double-layer distillation mechanism. While performing defect detection on RGB images, it can simultaneously output the pre-calculated control parameters required for hardware execution, such as material replenishment coordinates, robotic arm correction angle, or laser marking range. This reduces the instruction delay between the algorithm and the execution layer and improves the system response speed.

[0064] 3. Real-time coordinated control of hardware and software: The hardware and software coordination module 3 is set up. The coordination triggering unit 31 can synchronously send lightweight control data frames to the hardware execution unit 4 during the detection stage, realizing parallel triggering of detection and execution. The reverse calibration unit 32 can dynamically correct the control parameters according to the real-time operating data fed back by the hardware, so that the control process has the ability to adapt and adjust, significantly improving the execution accuracy and stability of the system.

[0065] 4. Reduce manual intervention and improve automation level: Through the data closed loop between the algorithm inference unit 2 and the hardware execution unit 4, the system can automatically complete defect identification, control parameter generation and correction without frequent manual calibration or manual adjustment of process parameters, thereby reducing the cost of manual intervention and improving the intelligence level of the chip assembly line.

[0066] 5. Achieving multi-source information fusion and model transferability: The two-layer distillation mechanism between the main network 21 and the sub-network 22 enables the model to have knowledge transfer capabilities, which can be quickly adapted to different equipment or process conditions, improve the versatility and scalability of the algorithm, and provide a technical foundation for the collaborative detection and control of multiple types of patch devices.

[0067] In summary, this embodiment significantly improves the accuracy of SMT chip detection, control response speed, and system adaptability through a hardware-software co-architecture and semantic feature-driven algorithm reasoning mechanism, achieving an efficient, intelligent, and transferable integrated solution for chip defect detection and control.

[0068] Example 2 is the second embodiment of the present invention. Based on Example 1, this embodiment further defines the training mechanism of the main network 21 in the algorithm inference unit 2. In particular, the design and dynamic adjustment mechanism of the composite loss function are optimized to improve the detection accuracy, feature matching ability and control parameter consistency of the system in complex SMT patch scenarios.

[0069] The composite loss function consists of a feature alignment loss term, a context consistency loss term, and a control semantic consistency loss term. The loss terms are fused by weighted summation to achieve bidirectional optimization of feature learning and control semantic constraints.

[0070] The overall structure in Example 2 is the same as that in Example 1, including an image acquisition unit 1, an algorithm inference unit 2, a software-hardware co-processing module 3, and a hardware execution unit 4. In this example, the main network 21 in the algorithm inference unit 2 is trained using an improved composite loss function.

[0071] (1) Image acquisition unit 1 Consistent with Example 1, it is used to acquire high-resolution RGB images of the SMT patch area to provide semantic information input to the main network 21, including pad topology, component package grayscale distribution, and solder joint morphology.

[0072] (2) Algorithm reasoning unit 2 The main network 21 in algorithm inference unit 2 maintains the same structure as in Example 1, but a composite loss function is introduced during the model training phase. The composite loss function is expressed as: ; in, This is the total loss value. These are the weight coefficients for the feature alignment loss. These are the weighting coefficients for the context consistency loss. To control the weighting coefficients of semantic consistency loss, As the normalization factor, For integration variables, The shape parameter of the Gamma function. For the Gamma function, It is a natural constant. Let be the error function. For the model at position Output eigenvalues To mark the location eigenvalues, The scaling parameter for feature differences. For the summation index, This represents the total number of dimensions of the context features. For a Bessel function of the first kind, Scaling factor for context features For the first Dimensional context feature values, For the parameters of the Gamma function, To control the attenuation coefficient of parameter differences, To control the difference in parameters.

[0073] The range of the composite loss function is [0, +∞). =0 indicates a perfect match with no loss; a larger value indicates greater loss and worse model performance. Specifically, the range is affected by the parameters, but can be determined through a normalization factor. The sum function structure (such as the squared error function, exponential decay) ensures that the loss value is non-negative and comparable, and is suitable for gradient descent in optimization algorithms.

[0074] How the composite loss function works: Feature alignment loss term (first term): This term is used to constrain the output features of the main network. With annotation features Consistency of distribution in multidimensional feature space.

[0075] Through the error function Combining it with the Gamma integral form enhances sensitivity to high-dimensional feature boundary regions, thereby reducing feature drift.

[0076] Technical effect: Ensures that the model maintains consistent feature representation for the same type of solder joints and packaging structure under multiple production conditions, improving the model's cross-batch stability and repeatability.

[0077] Context consistency loss term (second term): This term is obtained through the Bessel function. Establish a spatial relationship mapping between feature points to constrain the continuity of changes in adjacent features in the feature space.

[0078] This loss term can significantly improve the model's contextual smoothness when the relative distance or grayscale distribution between features is abnormal. Technical effect: Enhances the model's contextual awareness of pads, solder paste, and component edges, making defect detection results more consistent, boundary recognition more accurate, and effectively reducing false positives and false negatives.

[0079] Control semantic consistency loss term (third term): This term ensures that the defect geometric features output by the main network 21 match the hardware control parameter thresholds (such as feeding coordinates, robot arm angle, etc.) in the control-related semantic features. This is achieved through the Gamma function and the exponential decay term. The combination of dynamic penalties for control discrepancies exceeding a threshold range ensures that the model output remains consistent with the physical execution layer.

[0080] Technical benefits: Significantly reduces the deviation between algorithm inference results and actual execution actions, achieving a high degree of consistency between detection output and execution actions, thereby improving the automated calibration capability of the production line.

[0081] Dynamic weight adjustment mechanism: weight coefficients in the composite loss function , , It can be dynamically adjusted according to the process requirements and product complexity of different SMT production lines.

[0082] For example, in high-density packaging (HDI) processes, feature alignment weights can be increased; in multi-defect detection tasks, context consistency weights can be appropriately increased.

[0083] Technical effects: It enables the system to have adaptive optimization capabilities under different process scenarios, achieves a balance between feature learning and control semantics, and improves the overall collaborative performance of detection and control.

[0084] Working principle of Example 2: During system operation, image acquisition unit 1 acquires real-time RGB images and inputs them to algorithm inference unit 2; main network 21 performs forward inference based on a weight model optimized by a composite loss function to generate defect detection results and corresponding control parameters; sub-network 22 performs knowledge distillation inference to generate lightweight output; software and hardware co-operation module 3 triggers the action of execution unit according to the inference results and feeds back operating condition data to form closed-loop control.

[0085] During inference, the composite loss function continuously applies to the model parameter optimization stage, ensuring high stability of the main network 21 during both online learning and inference. The system can instantly adjust the robotic arm's correction angle or replenishment coordinates upon detecting defect offsets, achieving seamless linkage from detection to execution.

[0086] Technical effects of Example 2: By introducing a composite loss function mechanism, this embodiment achieves multi-dimensional optimization in terms of feature consistency, context smoothness, and control matching, enabling the system to maintain high consistency between detection and control under complex operating conditions. Compared with Embodiment 1, the system achieves significant improvements in defect boundary recognition accuracy, control parameter matching degree, and model convergence speed, providing high-precision, low-latency, and adaptive technical support for multi-defect detection of SMT patches.

[0087] Example 3 is the third embodiment of the present invention. Based on Examples 1 and 2, this embodiment further specifies the acquisition method of SMT process semantic features in the system, the lightweight structure design of sub-network 22, the real-time parallel mechanism of software and hardware co-operation module 3, and the implementation configuration of hardware execution unit 4 and communication architecture, forming a complete and implementable real-time detection and control system for multiple defects in SMT chip mounting.

[0088] System composition and implementation method (1) Image acquisition unit 1 is used to acquire high-resolution RGB images of the SMT patch area.

[0089] It includes: The 20-megapixel industrial camera 11 has a resolution of 5472×3648 and a frame rate of up to 120fps. Telecentric lens 12 (magnification 2×) is used to avoid image distortion; A dimmable ring LED light source is used to ensure consistent grayscale at the solder joints.

[0090] This unit acquires RGB images of SMT components in real time and transmits them to the algorithm inference unit 2 via EtherCAT industrial Ethernet (transmission rate 1Gbps, end-to-end latency <1ms).

[0091] Technical benefits: High frame rate and high resolution imaging ensure accurate capture of solder joint topology, grayscale distribution and component morphology, providing high-fidelity input for subsequent process semantic feature extraction.

[0092] (2) Algorithm inference unit 2, deployed on edge computing devices (such as NVIDIA Jetson AGX Orin, computing power 200TOPS), includes two parts: main network 21 and sub-network 22, which are responsible for semantic learning and lightweight inference, respectively.

[0093] The main network 21 is built on the ResNet-50 architecture and is configured to simultaneously receive SMT process semantic features and control-related semantic features.

[0094] SMT process semantic features are extracted from the patch image using an image segmentation model (such as SAM or DeepLabV3+), including: pad topology, component package grayscale distribution, and standard solder joint shape.

[0095] The control-related semantic features are obtained by importing SMT production line process files, including: the center coordinate offset range of the defect area and the defect area threshold.

[0096] The main network 21 simultaneously receives the two types of semantic features mentioned above as input and is jointly trained through a composite loss function (including feature alignment, context consistency and control semantic consistency loss) to achieve dual semantic modeling of "defect discrimination + control parameter generation".

[0097] Technical effect: The main network 21 integrates process semantics and control logic in the feature space, making the detection output highly correlated with the hardware control parameters, thereby improving the model's transferability and execution consistency.

[0098] Subnetwork 22 is built on a lightweight convolutional neural network (improved MobileNetV4), with fewer than 3M parameters and less than 50M FLOPs of computation.

[0099] Subnetwork 22 obtains dual semantic knowledge through distillation of main network 21, which is used to output results in real time during the online reasoning stage.

[0100] Within a preset time period (approximately 6–7 ms) after receiving the RGB image, sub-network 22 synchronously outputs: Defect detection results: include defect category, coordinates, and confidence level (confidence level ≥ 0.95, coordinate accuracy 0.01 mm). Pre-calculated control parameters: including feeding coordinates, robotic arm correction angle or laser marking range, with a data size of less than 1KB.

[0101] Technical effect: Sub-network 22 enables synchronous output of detection and control parameters, significantly reducing computational latency and ensuring real-time performance and compatibility with lightweight deployment.

[0102] (3) The software and hardware co-operation module 3 includes a co-trigger unit 31 and a reverse calibration unit 32, which are used to realize the parallel triggering and dynamic correction of the algorithm layer output and the hardware control layer action.

[0103] Collaborative triggering unit 31: When subnetwork 22 begins to output defect detection results (without waiting for all inference to complete), this unit immediately performs parallel operations: The test results are pushed to the system display terminal; The pre-calculated control parameters are encapsulated into lightweight JSON data frames and sent to the hardware execution unit 4 in real time via EtherCAT industrial Ethernet.

[0104] Technical effect: Through the parallel triggering mechanism, the detection and execution are started synchronously, the average system latency is reduced by 20-30ms compared with the serial mode, and the total latency of a single frame can be controlled below 50ms.

[0105] Reverse calibration unit 32: This unit receives real-time operating condition data (including the current position error of the robotic arm, the actual power of the laser mark, etc.) fed back by the hardware execution unit 4, and calls the SMT process semantic features (including the component package mechanical compatibility threshold and the laser tolerance power of the solder joint) migrated in the sub-network 22 for dynamic correction.

[0106] When the feedback data exceeds the fault tolerance range defined by the process semantic features, the system will automatically adjust the pre-calculated control parameters.

[0107] For example, if the position error of the robotic arm exceeds 0.03mm, the system will adjust the correction angle of the robotic arm from 3° to 3.2° according to the tolerance range of the 0402 package pad (±0.05mm).

[0108] Technical effect: Through a semantically driven adaptive correction mechanism, the system's execution accuracy and reliability under complex working conditions are significantly improved, achieving a closed-loop unification of algorithm, hardware, and process information.

[0109] (4) Hardware execution unit 4, including: Six-axis sorting robot arm 41: Performs sorting or correction operations according to the replenishment coordinates or correction angles in the pre-calculated control parameters, with a repeatability positioning accuracy of ±0.02mm; Ultraviolet laser marking device 42: Performs marking operations according to the laser marking range in the control parameters, with a marking accuracy of 0.01mm.

[0110] Technical effect: Combined with a real-time calibration mechanism, the execution unit can respond to the algorithm output within milliseconds, realizing seamless integrated detection and control operations.

[0111] Working principle of Example 3: During system operation, image acquisition unit 1 continuously acquires SMT patch images at a speed of 120fps and transmits the image data to algorithm inference unit 2.

[0112] Main Network 21 Process Semantic Parsing Stage: Main Network 21 automatically extracts pad topology, package grayscale and solder joint morphology features through segmentation model, and integrates control parameters (such as offset threshold and area threshold) from SMT production line process files to generate a high-dimensional semantic feature map.

[0113] Lightweight inference stage of sub-network 22: Sub-network 22 performs fast inference in the edge computing device, and outputs defect detection results and pre-calculated control parameters simultaneously within 6-7ms without the need for additional computing resources.

[0114] In the hardware-software collaborative parallel execution stage: the collaborative triggering unit 31 pushes the results to the display end when the sub-network 22 starts outputting, and sends the pre-calculated control parameters to the robotic arm and laser device in real time in JSON format, so as to realize the parallel triggering of algorithm output and control action.

[0115] Reverse calibration and closed-loop correction stage: After the hardware execution unit 4 performs the action, it feeds back the operating condition data to the reverse calibration unit 32. If the feedback parameters exceed the semantic tolerance range, the system corrects the control parameters in real time to ensure that the execution action maintains high accuracy and stability.

[0116] Technical effects of Example 3: Fully semantic intelligent detection and control integration: The main network 21 integrates image segmentation semantics and process file parameters to achieve a natural binding between detection results and control logic.

[0117] Lightweight and high real-time parallelism: The sub-network has 22 parameters with less than 3M and a computational cost of less than 50M FLOPs, achieving an inference speed of over 150fps, making it suitable for high-speed production lines.

[0118] Zero-latency communication through hardware and software collaboration: Transmitted via EtherCAT 1Gbps industrial network, data frame <1KB, end-to-end latency <1ms.

[0119] Adaptive reverse calibration closed loop: Real-time correction of control parameters to ensure execution accuracy at the ±0.02mm level.

[0120] Excellent system stability and scalability: It can be directly applied to various types of SMT equipment and different packaging standards, and has good industrial adaptability.

[0121] A real-time detection method for multiple defects in SMT (Surface Mount Technology) components is applied to the aforementioned real-time detection system for multiple defects in SMT components, with reference to... Figure 2 ,include: Step S1: Image acquisition unit 1 acquires the RGB image of the SMT patch; Step S2: The main network 21 receives SMT process semantic features and control-related semantic features, and trains them using a composite loss function to output trained model parameters. The SMT process semantic features include pad topology, component package grayscale distribution and standard solder joint shape. The control-related semantic features include the center coordinate offset range of the defect region and the defect area threshold. In step S3, the sub-network 22 migrates data from the main network 21 through double-layer distillation, receives the RGB image, and synchronously outputs defect detection results and pre-calculated control parameters. The defect detection results include defect category, coordinates, and confidence level. The pre-calculated control parameters include the feeding coordinates, robotic arm correction angle, or laser marking range required for hardware execution. In step S4, when the sub-network 22 outputs the defect detection result, the collaborative triggering unit 31 synchronously sends the pre-calculated control parameters to the hardware execution unit 4 in a lightweight data frame format. In step S5, the reverse calibration unit 32 receives the real-time operating condition data fed back by the hardware execution unit 4, and performs real-time correction on the pre-calculated control parameters based on the SMT process semantic features migrated in the sub-network 22 to generate the corrected pre-calculated control parameters. In step S6, the hardware execution unit 4 performs sorting or marking operations according to the pre-calculated control parameters or the modified pre-calculated control parameters.

[0122] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A real-time inspection system for multiple defects in SMT (Surface Mount Technology) components, characterized in that, include: Image acquisition unit, used to acquire RGB images of SMT components; An algorithm inference unit, connected to the image acquisition unit, is used to receive and process the RGB image. The algorithm inference unit includes: The main network is used to receive SMT process semantic features and control-related semantic features, and to train them through a composite loss function to output trained model parameters. The SMT process semantic features include pad topology, component package grayscale distribution and standard solder joint shape, and the control-related semantic features include the center coordinate offset range of the defect region and the defect area threshold. A sub-network, connected to the main network, migrates data from the main network through double-layer distillation, is used to receive the RGB image and synchronously output defect detection results and pre-calculated control parameters. The defect detection results include defect category, coordinates, and confidence level, and the pre-calculated control parameters include the feeding coordinates, robotic arm correction angle, or laser marking range required for hardware execution. The software-hardware co-operation module, which connects the algorithm inference unit and the hardware execution unit respectively, includes: The collaborative triggering unit is used to synchronously send the pre-calculated control parameters to the hardware execution unit in a lightweight data frame format when the sub-network outputs the defect detection result; The reverse calibration unit is used to receive real-time operating condition data fed back by the hardware execution unit, and to correct the pre-calculated control parameters in real time based on the SMT process semantic features migrated in the sub-network, so as to generate corrected pre-calculated control parameters. The hardware execution unit, connected to the hardware-software co-operation module, is used to perform sorting or marking operations based on the pre-calculated control parameters or the modified pre-calculated control parameters.

2. The SMT chip multi-defect real-time detection system according to claim 1, characterized in that: The composite loss function includes a feature alignment loss term, a context consistency loss term, and a control semantic consistency loss term; The feature alignment loss term is used to constrain the consistency of the distribution of the features output by the main network with the standard features in the multidimensional space; The context consistency loss term is used to maintain the spatial correlation between adjacent feature points during feature extraction; The control semantic consistency loss term is used to ensure that the defect geometric features output by the main network match the hardware control parameter thresholds in the control association semantic features; The composite loss function integrates the above three losses through a weighted summation method, wherein the weighting coefficients are dynamically adjusted according to the process requirements of the SMT production line.

3. The SMT chip multi-defect real-time detection system according to claim 2, characterized in that: The composite loss function is expressed as follows: ; in, This is the total loss value. These are the weight coefficients for the feature alignment loss. These are the weighting coefficients for the context consistency loss. To control the weighting coefficients of semantic consistency loss, As the normalization factor, For integration variables, The shape parameter of the Gamma function. For the Gamma function, It is a natural constant. Let be the error function. For the model at position Output eigenvalues To mark the location eigenvalues, The scaling parameter for feature differences. For the summation index, This represents the total number of dimensions of the context features. For a Bessel function of the first kind, Scaling factor for context features For the first Dimensional context feature values, For the parameters of the Gamma function, To control the attenuation coefficient of parameter differences, To control the difference in parameters.

4. The SMT chip multi-defect real-time detection system according to claim 1, characterized in that: The SMT process semantic features are obtained through the following methods: Based on the image segmentation model, feature information such as pad topology, component package grayscale distribution and standard solder joint morphology is extracted from SMT patch images. The control-related semantic features are obtained by importing SMT production line process files, including the center coordinate offset range of the defect area and the defect area threshold. The main network simultaneously receives the SMT process semantic features and the control association semantic features as input.

5. The SMT chip multi-defect real-time detection system according to claim 1, characterized in that: The sub-network is constructed based on a lightweight convolutional neural network; The sub-network is configured to synchronously output defect detection results, including defect category, coordinates and confidence level, as well as pre-calculated control parameters, including material replenishment coordinates, robotic arm correction angle or laser marking range, within a preset time period after receiving the RGB image.

6. The SMT chip multi-defect real-time detection system according to claim 1, characterized in that: The collaborative triggering unit is configured to perform the following parallel operations: When the sub-network begins to output the defect detection result, the defect detection result is immediately pushed to the system display terminal; Simultaneously, the pre-calculated control parameters are encapsulated into lightweight data frames in JSON format; The lightweight data frames are sent to the hardware execution unit in real time via industrial Ethernet.

7. The SMT chip multi-defect real-time detection system according to claim 1, characterized in that: The reverse calibration unit is configured as follows: Receive real-time operating data fed back by the hardware execution unit, including the current position error of the robotic arm and the actual power of the laser marker; The SMT process semantic features migrated in the sub-network are invoked, including the mechanical compatibility threshold of the component package and the laser tolerance power of the solder joint; When the real-time operating data exceeds the fault tolerance range defined by the SMT process semantic features, the pre-calculated control parameters are dynamically corrected.

8. The SMT chip multi-defect real-time detection system according to claim 1, characterized in that: The hardware execution unit includes: A six-axis sorting robot arm is configured to perform sorting or correction operations based on the feeding coordinates or robot arm correction angle in the pre-calculated control parameters. An ultraviolet laser marking device is configured to perform a marking operation based on the laser marking range in the pre-calculated control parameters.

9. The SMT chip multi-defect real-time detection system according to claim 1, characterized in that: The image acquisition unit includes an industrial camera and a telecentric lens, and is configured to acquire the RGB image at a frame rate of 120fps. The algorithm inference unit is implemented based on an edge computing device; The image acquisition unit, the algorithm inference unit, and the hardware execution unit are connected via EtherCAT industrial Ethernet.

10. A method for real-time detection of multiple defects in SMT components, applied to the real-time detection system for multiple defects in SMT components according to any one of claims 1-9, characterized in that, include: Step S1: The image acquisition unit acquires the RGB image of the SMT patch; Step S2: The main network receives SMT process semantic features and control-related semantic features, and trains them using a composite loss function to output trained model parameters. The SMT process semantic features include pad topology, component package grayscale distribution, and standard solder joint shape. The control-related semantic features include the center coordinate offset range of the defect region and the defect area threshold. Step S3: The sub-network migrates data from the main network through double-layer distillation, receives the RGB image, and synchronously outputs defect detection results and pre-calculated control parameters. The defect detection results include defect category, coordinates, and confidence level. The pre-calculated control parameters include the feeding coordinates, robotic arm correction angle, or laser marking range required for hardware execution. Step S4: When the sub-network outputs the defect detection result, the collaborative triggering unit synchronously sends the pre-calculated control parameters to the hardware execution unit in a lightweight data frame format. Step S5: The reverse calibration unit receives real-time operating condition data fed back by the hardware execution unit, and performs real-time correction on the pre-calculated control parameters based on the SMT process semantic features migrated in the sub-network, so as to generate the corrected pre-calculated control parameters. Step S6: The hardware execution unit performs sorting or marking operations according to the pre-calculated control parameters or the modified pre-calculated control parameters.

Images