Smt production line-oriented collaborative visual inspection method and system

Abstract

The application relates to the technical field of industrial automation detection, in particular to a collaborative visual detection method and system for an SMT production line, which comprises the following steps: acquiring a beat reference signal synchronized with the periodic motion beat of a mounting head in the SMT production line; determining a stable acquisition window with a vibration intensity lower than a preset vibration threshold; uniformly sending a synchronous trigger instruction to a plurality of visual detection cameras arranged on the SMT production line in the stable acquisition window; performing space-time alignment processing on the multiple-channel image data; performing motion blur correction processing on the image data subjected to the space-time alignment processing; performing effectiveness verification on the space-time alignment processing result, and executing a preset abnormal processing strategy when the verification result indicates that the verification fails. The application directly acquires the beat reference signal synchronized with the motion beat of the mounting head, and deeply binds the timing decision of image acquisition with the real-time vibration monitoring result, so that the dual strategies of beat synchronization and vibration avoidance are realized.

Description

Technical Field

[0001] This application relates to the field of industrial automation inspection technology, specifically to a collaborative visual inspection method and system for SMT production lines. Background Technology

[0002] Surface Mount Technology (SMT) is a core process in modern electronics manufacturing, characterized by high speed, high density, and high precision. With the continuous evolution of Moore's Law, the size of electronic components is constantly miniaturizing. The widespread application of ultra-miniature components such as 0201 (0.6mm x 0.3mm) and even 01005 (0.4mm x 0.2mm) has placed unprecedented demands on online inspection of product assembly quality. Traditional single-point, offline sampling inspections or methods relying on manual visual inspection are inherently inefficient, inconsistent, prone to fatigue, and unable to provide full coverage, making them far from meeting the cycle time and micron-level precision requirements of modern high-speed production lines. Therefore, integrating multiple cameras on the production line for online, full-coverage collaborative visual inspection has become a key technology for ensuring SMT production line quality control and improving product yield and competitiveness.

[0003] In high-speed SMT production lines, the core placement equipment (i.e., the placement head) performs cyclical reciprocating movements of picking up, moving, and placing electronic components at extremely high speeds and accelerations, with a cycle time often on the order of hundreds of milliseconds. This not only poses an extremely high challenge to the processing speed of the vision inspection system, but also imposes stringent requirements on the synchronization of data acquisition between multiple cameras deployed at different locations on the production line, at the microsecond or even sub-microsecond level. Any tiny timing deviation may cause image data of the same target acquired from different perspectives to fail to be accurately correlated in time and space, leading to 3D reconstruction failures or feature matching errors, ultimately resulting in serious consequences such as missed detections or misjudgments.

[0004] Furthermore, the placement head generates severe and high-frequency mechanical vibrations during high-speed movement and start-up / stop. These vibrations are directly transmitted to the camera and its mounting bracket through the production line's mechanical structure, causing minute displacements in the image, resulting in blurred images, loss of detail, and a significant reduction in the signal-to-noise ratio and accuracy of the inspection system. This image quality degradation caused by vibration is particularly detrimental when capturing minute defects, such as solder paste spikes, minor component misalignments, or early signs of tombstoning.

[0005] However, in-depth analysis revealed a critical common flaw in the aforementioned existing technologies: their synchronization triggering mechanisms are indirect, passive, and lagging. These solutions generally rely on external physical sensors (such as photoelectric sensors, proximity switches, etc.) to detect the physical arrival signal of the target object or its carrier, using this as the basis for triggering camera acquisition. This material-driven triggering method inherently suffers from technical bottlenecks in synchronization accuracy and robustness when dealing with high-speed, high-density, and tightly scheduled SMT production lines.

[0006] In summary, existing technologies have failed to proactively address vibration interference caused by high-speed motion, ultimately resulting in their detection accuracy, stability, and reliability under extreme conditions failing to meet the needs of the next generation of electronics manufacturing. Summary of the Invention

[0007] To overcome the aforementioned defects and shortcomings in the existing technology, the first aspect of this application provides a collaborative visual inspection method for SMT production lines, comprising the following steps: acquiring a beat reference signal synchronized with the periodic movement beat of a placement head in the SMT production line, the beat reference signal including a timestamp for identifying the acquisition time and position information reflecting the current cycle of the placement head; real-time monitoring of the mechanical vibration state generated by the SMT production line during the movement of the placement head, and based on the mechanical vibration state and the beat reference signal, determining a stable acquisition window within a single movement beat cycle of the placement head where the vibration intensity is lower than a preset vibration threshold; within the stable acquisition window, transmitting data to multiple visual inspection cameras deployed on the SMT production line. A synchronization trigger command is sent to control the multiple visual inspection cameras to simultaneously acquire multi-channel image data containing the target to be inspected; the multi-channel image data acquired by the multiple visual inspection cameras is received, and spatiotemporal alignment processing is performed on the multi-channel image data. The spatiotemporal alignment processing includes compensating the timestamp based on a statistical model of historical communication delay data, and mapping the pixel coordinates of the multi-channel image data to a unified world coordinate system based on a multi-level coordinate system transformation model; motion blur correction processing is performed on the image data after spatiotemporal alignment processing according to the production line motion speed data associated with the acquisition time; the validity of the spatiotemporal alignment processing result is verified, and a preset exception handling strategy is executed when the verification result indicates verification failure.

[0008] By adopting the above technical solution, and directly acquiring the timing reference signal synchronized with the placement head's movement, and deeply binding the timing decision of image acquisition with real-time vibration monitoring results, a dual strategy of timing synchronization and vibration avoidance is achieved. This fundamentally solves the inherent problems of timing lag, uncertainty, and inability to avoid interference in traditional position sensor-based triggering methods. It ensures that multiple cameras can synchronously acquire data at the golden moment when the physical environment of the entire production line is most stable and interference is minimal, greatly improving the quality and synchronization accuracy of raw data acquisition under complex conditions such as high speed and strong vibration, laying a solid and reliable data foundation for the successful operation of all subsequent high-precision detection algorithms.

[0009] Furthermore, the step of acquiring a timing reference signal synchronized with the periodic movement rhythm of the placement head in the SMT production line specifically includes: combining the encoder signal of the servo motor controlling the movement of the placement head and the height sensor signal used to confirm the vertical position of the placement head to generate the timing reference signal containing precise timing and spatial position information.

[0010] By employing the above technical solution, and combining the encoder signal from the servo motor for monitoring and the height sensor signal for confirming the vertical position, highly informative reference data containing millisecond-level precise timing and micrometer-level precise spatial position can be acquired. This multi-signal fusion approach is more reliable and provides more complete information than relying on a single signal source. This allows the system to have a more accurate and unambiguous understanding of the three-dimensional spatial motion state of the mounting head, thus providing higher-quality input for subsequent determination of the stable acquisition window and spatiotemporal alignment.

[0011] Furthermore, determining the stable acquisition window where the vibration intensity is lower than the preset vibration threshold specifically includes: identifying that the placement head is in the idle travel phase between picking up the component and placing the component during the working cycle, and defining the stable acquisition window in the idle travel phase in combination with the mechanical vibration state.

[0012] By adopting the above technical solution, a stable acquisition window is cleverly utilized by prioritizing the idle travel phase between the pick-up and placement actions of the placement head, thus taking advantage of a naturally occurring, relatively low-vibration interval in the SMT process. During this phase, the placement head's main movement is smooth, with minimal acceleration and deceleration, resulting in naturally lower mechanical shock and vibration. Furthermore, by combining this with real-time vibration monitoring data for filtering, stable acquisition windows that meet the requirements can be found more efficiently and reliably. This achieves intelligent utilization of the inherent operating patterns of the production line, improving the frequency and stability of the acquisition window.

[0013] Furthermore, the real-time monitoring of the mechanical vibration state generated by the SMT production line during the movement of the placement head specifically includes: collecting vibration signals using a vibration sensor rigidly connected to the main structure of the SMT production line, and identifying the characteristic frequency vibration caused by the periodic movement of the placement head by performing frequency domain analysis on the vibration signals; the stable acquisition window is determined within the period when the amplitude of the characteristic frequency vibration is lower than the preset vibration threshold.

[0014] By employing the above technical solution and using vibration sensors rigidly connected to the main structure of the production line, and by performing frequency domain analysis on the acquired signals, vibrations within a specific frequency range caused by the periodic movement of the placement head can be accurately identified and quantified. This method effectively separates the target vibration source from complex background industrial noise, making the judgment of vibration status more accurate. It ensures that the determination of the stable acquisition window is based on a direct quantitative assessment of the core interference source, rather than a general, wide-band vibration magnitude judgment, further enhancing the system's anti-interference capability.

[0015] Furthermore, the statistical model based on historical communication delay data compensates for the timestamps by periodically calculating and updating the statistical average of network communication jitter between sending the synchronization trigger command and receiving the multi-channel image data, and using the statistical average as a compensation amount to independently correct the timestamps of each of the multi-channel image data.

[0016] By adopting the above technical solution, an adaptive time alignment correction mechanism is established by creating an independent network communication jitter statistical model for each camera channel and periodically updating and compensating for it. This mechanism can effectively combat nondeterministic communication jitter caused by factors such as network congestion and switch processing delays, ensuring that even in complex industrial Ethernet environments, the final system timestamps of all image data can be accurately unified to the same time base, significantly improving the accuracy and robustness of time alignment.

[0017] Furthermore, the mapping of pixel coordinates of the multi-channel image data to a unified world coordinate system based on the multi-level coordinate system transformation model specifically includes automatically triggering dynamic calibration and parameter updates of the multi-level coordinate system transformation model under at least one of the following preset conditions: the first power-on of the SMT production line equipment, a restart, or after running for a preset period of time; and when the key moving parts of the vision inspection camera or the placement head in the SMT production line are replaced or their positions are adjusted.

[0018] By adopting the above technical solution, dynamic calibration and parameter updates of the multi-level coordinate system transformation model are automatically triggered after the equipment is first powered on, restarted, operated for a long time, or after physical changes occur to key components, ensuring the long-term effectiveness and accuracy of the spatial alignment model. This proactive self-calibration mechanism, driven by both events and time, can promptly compensate for coordinate system deviations caused by factors such as physical thermal expansion and contraction, temperature drift, or mechanical wear over time. It eliminates the need for tedious and easily forgotten manual periodic calibration, ensuring that the system maintains micron-level spatial mapping accuracy throughout its entire lifecycle.

[0019] Furthermore, the motion blur correction process specifically includes: acquiring instantaneous motion velocity vector data of the SMT production line corresponding to the acquisition time of the multi-channel image data in real time, and calculating the displacement and direction of pixel blur based on the instantaneous motion velocity vector data and the exposure time of the visual inspection camera; and performing reverse pixel translation or deconvolution operation on the image data based on the displacement and direction to achieve motion blur correction.

[0020] By employing the above technical solution, and acquiring instantaneous motion velocity vector data of the production line precisely synchronized with the image acquisition time, and performing inverse mathematical operations (such as deconvolution) on the image based on this data, precise correction of dynamic motion blur is achieved. This method can effectively recover image blur caused by conveyor belt movement or minute camera movements during exposure, making previously blurred component edges, solder joint details, and silkscreen characters clearly discernible. This significantly improves the imaging quality of dynamic targets and the recognition rate and reliability of subsequent defect detection algorithms.

[0021] Furthermore, the validity verification of the spatiotemporal alignment processing result specifically includes: automatically extracting at least one pair of corresponding feature points from the multi-channel image data after spatial coordinate mapping, and calculating their three-dimensional spatial coordinate deviation in the world coordinate system; when the deviation exceeds a first preset deviation threshold, the spatiotemporal alignment processing verification is determined to have failed.

[0022] By employing the aforementioned technical solution, and automatically extracting corresponding feature points from multi-channel image data, and calculating their three-dimensional spatial coordinate deviations in a unified world coordinate system for validity verification, a closed-loop, quantitative, and completely objective method for monitoring spatiotemporal alignment quality is provided. This method can evaluate the final effect of each alignment operation in real time and objectively, ensuring that only high-quality, high-precision aligned data can enter subsequent inspection stages, thus providing a crucial internal quality control barrier for the final reliability of the entire inspection process.

[0023] Furthermore, the anomaly handling strategy includes: when the spatiotemporal alignment processing verification fails, a graded resampling mechanism is initiated according to the type and severity of the verification failure; the graded resampling mechanism includes: firstly, performing a first resampling within the same stable acquisition window in the next adjacent motion beat cycle; if the first resampling still fails, then performing multiple supplementary resamplings in subsequent multiple motion beat cycles, provided that the vibration conditions are met, until the resampling is successful or the preset upper limit of the number of resamplings is reached.

[0024] By adopting the above technical solution, a smart and efficient fault-tolerance and data recovery strategy is established by initiating a hierarchical resampling mechanism when alignment verification fails. This mechanism first attempts low-cost, immediate resampling; if unsuccessful, it performs supplementary resampling in subsequent cycles. This hierarchical processing approach balances real-time performance and success rate, maximizing the acquisition of valid detection data without interrupting normal production line operation. This significantly enhances the system's data integrity when facing accidental interference (such as instantaneous vibration, sudden changes in ambient light, and data packet loss).

[0025] To achieve the aforementioned objectives, a second aspect of this application provides a collaborative visual inspection system for SMT production lines, comprising: a central control server for running the aforementioned collaborative visual inspection method for SMT production lines; multiple visual inspection cameras for capturing images of a target area from different angles; a mechanical vibration sensor rigidly fixed to a load-bearing beam of the main structure of the SMT production line near the mounting position of the visual inspection cameras; a signal acquisition interface installed in the central control server, having multiple digital I / O channels for accessing and parsing underlying signals from the SMT production line control system; and an industrial network device for connecting the central control server and the multiple visual inspection cameras.

[0026] This application enables high-speed, high-vibration SMT production line environments to achieve deep synchronization with the movement rhythm of the core drive unit (placement head). Combined with intelligent sensing and avoidance of the physical vibration state of the production line, it allows multiple vision inspection cameras deployed on the production line to achieve precise sub-microsecond-level synchronous triggering and high-quality data acquisition at the golden acquisition moment when the physical environment is most stable and data value is highest. Simultaneously, this application aims to solve the problems of data spatiotemporal inconsistency and image blurring caused by factors such as network communication latency, mechanical structure changes, and dynamic object movement, thereby comprehensively and systematically improving the accuracy, stability, and robustness of online inspection of micro-electronic components and solder joints. Attached Figure Description

[0027] Figure 1 This is a block diagram of the collaborative visual inspection system of this application.

[0028] Figure 2 This is a schematic diagram of the collaborative visual inspection method of this application. Detailed Implementation

[0029] Understandably, the core timing benchmark for SMT production line operation is the movement rhythm of the placement head, with all process steps revolving around this rhythm. However, existing triggering methods based on material position do not directly and precisely correlate the trigger signal with the internal movement rhythm of the placement head. During high-speed operation, minute differences in material position, sensor response delays, and electronic jitter during signal transmission can all lead to unpredictable delays or misalignments between the trigger moment and the critical actions of the placement head, resulting in timing benchmark misalignment. Moreover, since the trigger timing is determined solely by the material position, this mechanism is completely unable to detect or avoid the periodic, strong vibration interference caused by the placement head's own movement. It is highly likely that the moment of trigger acquisition coincides with the placement head undergoing violent acceleration and deceleration, at which point the entire production line is at its vibration peak, inevitably resulting in a significant reduction in the quality of the acquired image.

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be further described in detail below with reference to a preferred embodiment. It should be understood that the specific embodiment described herein is only for explaining this application and is not intended to limit the scope of protection of this application in any way. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0031] This application embodiment can be deployed on a high-speed SMT production line for online inspection of 0201 (0.6mm x 0.3mm) chip resistors or capacitors mounted on a PCB (Printed Circuit Board) by a high-speed mounting head; the inspection tasks include, but are not limited to, whether the components have defects such as offset in the X / Y direction, rotation in the θ direction, wrong parts, missing parts, or tombstoning caused by solder paste problems.

[0032] like Figure 1 As shown in the embodiments of this application, the collaborative visual inspection system for SMT production lines includes a central control server, multiple visual inspection cameras, mechanical vibration sensors, signal acquisition interfaces, and industrial network equipment.

[0033] Understandably, the central control server is a high-performance IPC (Industrial PC) equipped with a multi-core processor, large-capacity memory, and high-speed solid-state drive, used to run the collaborative visual inspection method for SMT production lines proposed in this application, including all logical units such as signal acquisition, synchronization triggering, data alignment, image correction, and anomaly handling.

[0034] As a specific example, this embodiment deploys three industrial cameras equipped with GigE (Gigabit Ethernet) interfaces as visual inspection cameras. These industrial cameras all support the GigE Vision communication standard and have built-in hardware-level support for PTP (Precision Time Protocol). These industrial cameras capture images of the target area on the PCB from different angles. For example, one industrial camera takes a top-down view perpendicular to the PCB surface, while the other two take oblique side views at 45-degree angles from both sides to obtain the three-dimensional contour information of the target.

[0035] The mechanical vibration sensor can employ two high-sensitivity triaxial MEMS (Micro-Electro-Mechanical System) accelerometers, which are rigidly fixed to the main structural load-bearing beam of the SMT production line near the industrial camera mounting location by bolts, to ensure accurate acquisition of vibrations transmitted to the industrial camera.

[0036] The signal acquisition interface uses a dedicated high-speed data acquisition card based on FPGA (Field-Programmable Gate Array), which is installed in the central control server. This card has multiple digital I / O channels for high-speed, low-latency access and parsing of low-level signals from the SMT production line control system.

[0037] The industrial network equipment uses an industrial-grade network switch that supports PTP functionality to connect the central control server with all GigE cameras. This switch is capable of accurately forwarding and compensating PTP packets, which is crucial for achieving high-precision time synchronization across the entire network.

[0038] The software portion of the collaborative visual inspection system of this application consists of multiple functional units deployed on a central control server. These units work together to implement all the steps of the collaborative visual inspection method of this application.

[0039] like Figure 2 As shown, the first step of the collaborative visual inspection method of this application is S01: acquiring a beat reference signal that is precisely synchronized with the periodic movement beat of the placement head in the SMT production line. This signal is the absolute time and space reference for all subsequent synchronization, triggering, alignment and other operations.

[0040] Two low-level signals from the PLC (Programmable Logic Controller) or dedicated motion controller of the SMT production line equipment are physically connected in parallel via an FPGA acquisition card installed in the IPC. The first low-level signal is the encoder signal of the servo motor. The high-speed X, Y, and Z axes of the placement head are all driven by high-precision AC servo motors. The A / B / Z phase pulse signals output by the quadrature encoders driving these motors are monitored in real time through a differential signal receiver. The FPGA performs frequency multiplication and direction determination on the A / B phase pulses at a clock frequency of tens of MHz, thereby accurately resolving the instantaneous position of the placement head in the X, Y, and Z directions with micron-level real-time resolution. The Z phase pulse is used for periodic origin calibration. The second low-level signal is the height sensor signal. In order to obtain absolute position confirmation in the Z-axis direction, a fiber optic or laser displacement sensor deployed near the nozzle of the placement head can be monitored synchronously. When the sensor reading indicates that the nozzle has accurately reached the preset component pickup height or placement height, a precise digital logic level transition signal is generated.

[0041] Then, these two low-level signals are fused at the hardware level within the FPGA. For example, when the Z-axis encoder indicates that the placement head is moving downwards, and a jump signal indicating that the placement height has been reached is received from the height sensor within a very short time, a placement action is confirmed to be completed with nanosecond-level accuracy. Next, a structured clock reference signal data packet is generated. This data packet is generated at each meaningful event point and broadcast to other units within the system via an internal message queue; for example, this data packet is generated at the beginning of the placement cycle, the completion of the pick-up action, the completion of the placement action, and the end of the cycle.

[0042] In this embodiment of the application, the data packet may include: a timestamp with nanosecond precision generated by the PTP master clock on the central control server and stamped by the FPGA at the moment the event is captured; a cycle ID used to uniquely identify which cycle the current placement motion is in; placement head position information containing a floating-point vector of (X, Y, Z) coordinates obtained in real time by the encoder signal parsing; and a cycle state used to clearly identify the current motion stage of the placement head.

[0043] S02: Real-time monitoring of the mechanical vibration state generated by the SMT production line during the movement of the placement head, and based on the mechanical vibration state and the cycle reference signal, determining a stable acquisition window where the vibration intensity is lower than a preset vibration threshold within a single movement cycle of the placement head.

[0044] Specifically, determining a stable acquisition window where the vibration intensity is below a preset vibration threshold involves continuously subscribing to and analyzing cycle reference signal data packets. Based on the periodic state in the data packets, the idle travel phase after the placement head completes component pickup and before it moves at high speed to the target placement position can be identified. Theoretically, during this phase, the placement head's Z-axis remains stationary while its X and Y axes move smoothly at high speed, without any severe pickup or placement impacts, making it an ideal candidate period with relatively low vibration. Therefore, this phase can be locked in, and the final stable acquisition window can be found by combining this with the mechanical vibration state.

[0045] Specifically, real-time monitoring of the mechanical vibration state generated by the SMT production line during the movement of the placement head includes: a MEMS accelerometer rigidly connected to the main structure of the SMT production line continuously acquiring vibration acceleration signals of the X, Y, and Z axes at an extremely high sampling rate (e.g., configured as 25.6 kHz), and then processing these massive amounts of raw data in real time.

[0046] S21: The acquired acceleration signal is preprocessed in real time through a digital bandpass filter, for example, a fourth-order Butterworth filter is used to filter out extremely high-frequency electronic noise and extremely low-frequency slow drift of the entire equipment that are unrelated to the target vibration.

[0047] S22: Preliminary analysis of a large amount of offline acquired data revealed that the periodic motion of the placement head primarily generates significant vibrational energy in the 50Hz to 500Hz frequency band. Therefore, a short-time Fourier transform was performed on the preprocessed signal to continuously observe energy changes in this characteristic frequency band with high time resolution. Analysis of the spectrum clearly revealed the characteristic frequency peaks with significant amplitudes caused by the periodic motion of the placement head, along with their harmonics.

[0048] S23: Through preliminary experiments and calibration, a preset vibration threshold can be determined. This vibration threshold is not a simple acceleration scalar value, but a frequency domain energy threshold. For example, if the integrated energy value in the 50Hz-500Hz characteristic frequency band is lower than a certain preset value, or the amplitude of all characteristic frequency peaks is lower than 0.5g (g is the acceleration due to gravity), the current environment is considered sufficiently stable, and high-quality imaging can be performed.

[0049] S24: The results of beat analysis and vibration analysis are fused in real time. When the beat reference signal indicates that the mounting head has entered the idle travel phase, intensive monitoring of the energy of the real-time vibration signal in the characteristic frequency band immediately begins. Once it is detected that the energy is consistently below a preset energy threshold for a period of time (e.g., 5 milliseconds), this period is precisely defined as the stable acquisition window. This window is defined by a start timestamp and an end timestamp, and its duration may only be 10 to 30 milliseconds, but this is more than enough for high-speed cameras with exposure times of only a few hundred microseconds.

[0050] S03: Within the stable acquisition window, a synchronization trigger command is sent to multiple vision inspection cameras deployed on the SMT production line to control the multiple vision inspection cameras to synchronously acquire multi-channel image data containing the target to be inspected.

[0051] Specifically, after determining the stable acquisition window, instructions need to be sent immediately to all visual inspection cameras to ensure they accurately complete image acquisition at the same time within that window. To achieve sub-microsecond synchronization accuracy, this application preferably adopts a synchronization scheme based on the IEEE 1588 (PTP) protocol.

[0052] S31: Upon system startup, the central control server, acting as the network's PTP master clock, negotiates clock synchronization with all industrial cameras acting as PTP slave clocks through PTP-enabled industrial network devices. Through multiple delay request / response message exchanges and a complex optimal master clock algorithm using the PTP protocol, the deviation between the internal hardware clocks of all cameras and the central control server's master clock can be corrected to within 1 microsecond, typically reaching the level of several hundred nanoseconds. This allows the entire distributed system to share a unified, high-precision time base.

[0053] S32: Once the stable acquisition window is determined, an optimal trigger time Ttrigger is calculated. For example, if the stable acquisition window spans from timestamp T1 to T2, this trigger time is typically chosen at the center of the window to maximize timing margin, i.e., Ttrigger = T1 + (T2 - T1) / 2. Then, the planned action command is immediately broadcast to all industrial cameras via the GigE Vision protocol. The core of this command is not immediate triggering, but rather: to execute a specified action at a precise absolute timestamp in the future. For example, at timestamp Ttrigger, an image acquisition with an exposure time of 500 microseconds is triggered.

[0054] S33: Since the internal clocks of all industrial cameras are highly synchronized with the master clock via PTP, when their respective internal clock counters reach the precise moment of Trigger, they will trigger their respective image sensors to start exposure at almost the exact same physical instant, thus achieving ultimate synchronous acquisition of multiple industrial cameras.

[0055] S04: Receives multi-channel image data acquired by multiple vision inspection cameras and performs spatiotemporal alignment processing on the multi-channel image data. After multiple industrial cameras simultaneously acquire multi-channel image data, these image data still have slight deviations in time and space that need to be corrected. Precise alignment processing must be performed to fuse them into a unified and effective dataset for subsequent analysis.

[0056] Although PTP guarantees synchronization of the trigger moment, each camera experiences different and dynamically changing network transmission delays (i.e., network jitter) during the process of image acquisition, data readout, packaging, and transmission to the central control server via Ethernet. This means that even images acquired simultaneously may arrive at the server at different times, sometimes by several milliseconds, making it completely inaccurate to rely solely on the timestamps of the data packets received by the server.

[0057] To address this issue, an independent historical communication latency statistical model can be maintained for each industrial camera channel. This model is dynamically updated through periodic, proactive online measurements. Specifically, a lightweight heartbeat or ping message with a timestamp Tsend is periodically (e.g., 5 times per second) sent to each industrial camera. Upon receiving the message, the industrial camera immediately returns a response message. The server records the time of receipt, Treceive. By calculating the round-trip time (RTT) = Treceive - Tsend and assuming symmetric network latency, the one-way latency can be estimated. This latency data is continuously collected, and a moving average filter with a forgetting factor or a more advanced Kalman filter is used to calculate the statistically average latency and jitter variance for each industrial camera channel under the current network load. For example, the average latency for industrial camera A is 2.1ms, and for industrial camera B it is 2.5ms, etc.

[0058] When an image frame from an industrial camera is received, the image data itself carries a precise acquisition timestamp, Tcapture, which is generated by the industrial camera at the start of exposure based on its PTP synchronization clock. The delay model corresponding to the industrial camera is then queried based on this acquisition timestamp Tcapture to compensate for the lost timestamp. A simplified compensation logic can be used, directly using the image's built-in Tcapture as the final physical timestamp, since this timestamp is generated locally by the camera and has not been contaminated by network transmission. A more refined approach is to combine this with a confidence assessment based on the delay model. By processing the images from all channels in this way, their time bases are precisely unified.

[0059] Furthermore, it is understandable that each industrial camera has its own independent two-dimensional image coordinate system based on pixels. To perform three-dimensional reconstruction or multi-view feature fusion, the pixel coordinates of all industrial cameras must be transformed into a unified three-dimensional world coordinate system that is fixed in the physical space of the production line.

[0060] Specifically, this application can adopt a transformation model including three levels of core coordinate systems: a. Camera Pixel Coordinate System (PCS): A two-dimensional coordinate system with the top left corner of the image as the origin, the u-axis to the right, and the v-axis downward. b. Camera Physical Coordinate System (CCS): A three-dimensional coordinate system with the camera's optical center as the origin, and the Z-axis along the optical axis. c. World Coordinate System (WCS): A three-dimensional Cartesian coordinate system fixed in the physical space of the production line, serving as the reference for all measurements. For example, the center of a high-precision ceramic calibration board can be defined as the origin (0,0,0) of the world coordinate system. d. SMT Equipment Coordinate System (MCS): A three-dimensional coordinate system defined by the equipment manufacturer based on the movement of the SMT placement head itself.

[0061] The specific calibration process includes: S41: Camera Intrinsic Calibration: For each industrial camera, a high-precision planar calibration target, such as a glass calibration plate with sub-pixel precision positioning dots, is used to capture 15-20 images from different angles and distances. Using the classic Zhang Zhengyou calibration method, the intrinsic parameter matrix K (including focal lengths fx, fy and principal points cx, cy) and a set of distortion coefficients (including radial distortion k1, k2, k3 and tangential distortion p1, p2) of each industrial camera can be accurately calculated. This step establishes the transformation relationship from distorted PCS to distortion-free, normalized image plane coordinates.

[0062] S42: Multi-camera extrinsic calibration: The calibration target is placed within a common field of view that is clearly visible to all industrial cameras. Simultaneously, all industrial cameras are triggered to capture images. By identifying corresponding feature points on the calibration target in each industrial camera image (e.g., the center of the circle in the i-th row and j-th column), the pose of each industrial camera can be calculated, i.e., the extrinsic parameter matrix [R|t] relative to the world coordinate system. This extrinsic parameter matrix consists of a 3x3 rotation matrix R and a 3x1 translation vector t. This step establishes the transformation relationship from the WCS to each independent CCS.

[0063] S43: Hand-eye calibration: In order to correlate the detection results with the behavior of the SMT equipment (such as informing the placement head how much a component has been offset), it is necessary to establish the relationship between WCS and MCS, which can be solved by the classic hand-eye calibration problem AX=ZB.

[0064] Specifically, a calibration object with multiple feature points is fixedly placed in the field of view of an industrial camera (eye); then, the nozzle at the end of the SMT placement head (hand) is controlled to precisely touch or align with multiple known points on the calibration object; each time a point is touched, the coordinates of the placement head in the MCS read from the PLC at that moment, as well as the coordinates of the nozzle tip in the WCS calculated by the camera's visual positioning, are recorded; after collecting multiple sets of such corresponding point pairs, the homogeneous transformation matrix from the MCS to the WCS can be accurately solved using mathematical methods such as singular value decomposition (SVD).

[0065] It is understandable that the relationships between coordinate systems will undergo slight changes due to thermal expansion and contraction, mechanical wear, or maintenance during long-term equipment operation. To ensure long-term accuracy, an automatic dynamic calibration mechanism can be designed. When any of the following preset conditions occur, the system will prompt the operator to perform a simplified recalibration process, typically requiring only the redo of external parameters and manual / eye calibration: after the production line equipment is first started up each day or after an unplanned restart; after the system has run continuously without interruption for a preset duration, such as 24 hours; or after maintenance personnel have recorded maintenance logs in the system indicating camera or lens replacement or mechanical position adjustments to critical moving parts of the placement head, etc.

[0066] Understandably, even when acquiring images within a stable window with minimal vibration, if the PCB conveyor belt is still moving at a constant speed, high-speed moving objects will still produce motion blur within a limited exposure time, reducing image sharpness.

[0067] S05: Based on the production line motion speed data associated with the acquisition time, perform motion blur correction processing on the image data that has undergone spatiotemporal alignment processing, specifically including the following steps.

[0068] S51: Accurately acquire the timestamp Tcapture based on the time-aligned image, and query the PLC via the OPC UA protocol for the instantaneous velocity vector V of the PCB conveyor belt at that acquisition moment. This vector contains the velocity magnitude (e.g., 1.0 m / s) and the direction of motion under WCS.

[0069] S52: Motion blur in image processing can be precisely modeled as a point spread function (PSF). For linear uniform motion, its PSF is a line segment whose length L is determined by the speed and exposure time: L = |V| * Texposure. The direction of the line segment is consistent with the projection direction of the velocity vector V onto the image plane.

[0070] S53: Using the calculated accurate PSF as known input, perform a non-blind deconvolution algorithm on the acquired blurred image. In this embodiment, the Lucy-Richardson algorithm can be used because it has good robustness to noise. This algorithm mathematically solves for the estimate that is most likely to be close to the original clear image through iteration, thereby largely offsetting motion blur and restoring the details of the component edges and solder joints to clarity.

[0071] S06: Perform validity verification on the result of the spatiotemporal alignment process, and execute a preset exception handling strategy when the verification result indicates that the verification failed.

[0072] Specifically, in order to ensure the reliability of the entire complex process, a closed-loop verification of the processing results is required, and possible failures should be handled properly and intelligently, including the following steps.

[0073] S61: Automatically locates predefined, high-contrast fiducial marks on the PCB from spatially aligned multi-channel images. If no fiducial marks are available, it can also automatically extract stable corner points on the component under test that are clearly identifiable from multiple viewpoints, for example, using Harris or FAST corner detection algorithms.

[0074] S62: Using at least two industrial cameras, such as a top-view camera and a left-side oblique-view camera, the two-dimensional pixel coordinates of the same physical feature point extracted from the two images are used to calculate its three-dimensional coordinates P1 and P2 in the three-dimensional world coordinate system by triangulation.

[0075] S63: Ideally, P1 and P2 should be completely coincident. Calculate the Euclidean distance between them, d = ||P1-P2||. This distance is the reprojection error. If the distance d exceeds the preset first deviation threshold (for example, set to 10 micrometers according to the accuracy requirements), the spatiotemporal alignment process is deemed to have failed.

[0076] S64: When the verification fails, it will not immediately give up or alarm and stop the machine, but will execute the preset exception handling strategy.

[0077] Level 1 Anomaly Handling Strategy: Immediate Re-sampling. Within the next SMT cycle, when the same area of ​​the same PCB re-enters the industrial camera's field of view, an attempt is made to re-execute the complete process of determining the stable window, synchronous triggering, acquisition, processing, and verification at the same relative position. This strategy is primarily used to handle momentary, accidental interference, such as airborne dust momentarily obscuring the camera, accidental glare or reflection from solder joints, or single data packet loss.

[0078] The second-level anomaly handling strategy is supplementary resampling. If the first resampling still fails, it indicates that the problem may not be accidental. However, it is not advisable to retry indefinitely in every cycle to avoid affecting the overall production line cycle time. The detection task can be marked as pending retry, and supplementary resampling can be performed as soon as a stable acquisition window that meets the conditions is detected within the subsequent N cycle time (e.g., N=5). An upper limit on the number of resampling attempts can also be set, such as a maximum of 3 supplementary resampling attempts. If this is exceeded, the third-level anomaly handling strategy is adopted.

[0079] The third-level anomaly handling strategy: Alarms and intelligent handling. If the maximum number of re-samples is reached and the problem persists, it can be determined that the valid data for the target cannot be obtained through the current automated method. At this point, a series of escalation operations can be performed: 1) Record the unique ID of the PCB and the precise coordinates of the component under test in the MCS, along with the failure reason code, into the anomaly check list in the database; 2) Issue a clear, non-blocking alarm to the production line operator through the human-machine interface, such as highlighting a virtual image of the PCB on the screen and prompting them to proceed to the subsequent manual re-inspection station or offline analysis station; 3) Package all contextual information collected from all failed attempts, including raw images, timestamps, vibration data, alignment parameters, etc., into a diagnostic file and archive it for R&D personnel to perform root cause analysis and algorithm optimization. Most importantly, throughout the entire process, the SMT production line can continue to operate uninterruptedly without stopping due to accidental or continuous failures at a single test point.

[0080] In summary, this application constructs a complete closed-loop intelligent control system, from signal source perception to final decision-making, through a series of interconnected and precisely coordinated steps. It effectively solves a series of core technical challenges faced by collaborative visual inspection in high-speed SMT production lines, such as synchronization, vibration, alignment, fuzziness, and fault tolerance. This enables high-precision, high-stability, and high-robust online quality control of microelectronic components in harsh industrial environments.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0086] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A collaborative visual inspection method for SMT production lines, characterized in that, Includes the following steps: Acquire a timing reference signal that is synchronized with the periodic movement rhythm of the placement head in the SMT production line. The timing reference signal includes a timestamp to identify the acquisition time and position information reflecting the current cycle of the placement head. The mechanical vibration state generated by the SMT production line during the movement of the placement head is monitored in real time, and based on the mechanical vibration state and the beat reference signal, a stable acquisition window with vibration intensity lower than a preset vibration threshold is determined within a single movement beat cycle of the placement head. Within the stable acquisition window, a synchronous trigger command is sent to multiple vision inspection cameras deployed on the SMT production line to control the multiple vision inspection cameras to synchronously acquire multi-channel image data containing the target to be inspected. The system receives the multi-channel image data acquired by the multiple visual inspection cameras and performs spatiotemporal alignment processing on the multi-channel image data. The spatiotemporal alignment processing includes compensating the timestamps based on a statistical model of historical communication delay data, and mapping the pixel coordinates of the multi-channel image data to a unified world coordinate system based on a multi-level coordinate system transformation model. Based on the production line motion speed data associated with the acquisition time, motion blur correction processing is performed on the image data that has undergone the spatiotemporal alignment process. The validity of the spatiotemporal alignment process is verified, and a preset exception handling strategy is executed when the verification result indicates that the verification has failed.

2. The method according to claim 1, characterized in that, The acquisition of the timing reference signal synchronized with the periodic movement rhythm of the placement head in the SMT production line specifically includes: generating the timing reference signal containing precise timing and spatial position information by listening to the encoder signal of the servo motor that controls the movement of the placement head and the height sensor signal used to confirm the vertical position of the placement head.

3. The method according to claim 1, characterized in that, The determination of a stable acquisition window where the vibration intensity is below a preset vibration threshold specifically includes: identifying that the placement head is in the idle travel phase between picking up and placing components during the working cycle, and defining the stable acquisition window within the idle travel phase in conjunction with the mechanical vibration state.

4. The method according to claim 1, characterized in that, The real-time monitoring of the mechanical vibration state generated by the SMT production line during the movement of the placement head specifically includes: collecting vibration signals using a vibration sensor rigidly connected to the main structure of the SMT production line, and identifying the characteristic frequency vibration caused by the periodic movement of the placement head by performing frequency domain analysis on the vibration signals; the stable acquisition window is determined within the time period when the amplitude of the characteristic frequency vibration is lower than the preset vibration threshold.

5. The method according to claim 1, characterized in that, The statistical model based on historical communication delay data compensates for the timestamps by periodically calculating and updating the statistical average of network communication jitter between sending the synchronization trigger command and receiving the multi-channel image data, and using the statistical average as a compensation amount to independently correct the timestamps of each of the multi-channel image data.

6. The method according to claim 1, characterized in that, The method of mapping the pixel coordinates of the multi-channel image data to a unified world coordinate system based on the multi-level coordinate system transformation model specifically includes automatically triggering dynamic calibration and parameter updates of the multi-level coordinate system transformation model under at least one of the following preset conditions: the first power-on of the SMT production line equipment, a restart, or after running for a preset period of time; And, when the key moving parts of the vision inspection camera or the placement head in the SMT production line are replaced or their positions are adjusted.

7. The method according to claim 1, characterized in that, The motion blur correction process specifically includes: acquiring instantaneous motion velocity vector data of the SMT production line corresponding to the acquisition time of the multi-channel image data in real time, and calculating the displacement and direction of pixel blur based on the instantaneous motion velocity vector data and the exposure time of the visual inspection camera; and performing reverse pixel translation or deconvolution operation on the image data based on the displacement and direction to achieve motion blur correction.

8. The method according to claim 1, characterized in that, The validity verification of the spatiotemporal alignment processing result specifically includes: automatically extracting at least one pair of corresponding feature points from the multi-channel image data after spatial coordinate mapping, and calculating their three-dimensional spatial coordinate deviation in the world coordinate system; when the deviation exceeds a first preset deviation threshold, the spatiotemporal alignment processing verification is determined to have failed.

9. The method according to claim 8, characterized in that, The anomaly handling strategy includes: when the spatiotemporal alignment processing verification fails, a graded resampling mechanism is initiated according to the type and severity of the verification failure; the graded resampling mechanism includes: firstly, a first resampling is performed in the same stable acquisition window within the next adjacent motion beat cycle; if the first resampling still fails, multiple supplementary resamplings are performed in subsequent multiple motion beat cycles, provided that the vibration conditions are met, until the resampling is successful or the preset upper limit of the number of resamplings is reached.

10. A collaborative visual inspection system for SMT production lines, characterized in that, include: A central control server is used to run the collaborative visual inspection method for SMT production lines as described in any one of claims 1-9; Multiple visual inspection cameras are used to capture images of the target area from different angles; The mechanical vibration sensor is rigidly fixed to the main structural load-bearing beam of the SMT production line near the installation position of the vision inspection camera; The signal acquisition interface, installed in the central control server, has multiple digital I / O channels for accessing and parsing low-level signals from the SMT production line control system; and Industrial network equipment for connecting the central control server and the multiple visual inspection cameras.

Images