A visual inspection system and method for SMT chip defects

Abstract

This invention relates to the field of automated visual inspection technology, specifically to a visual inspection system and method for SMT (Surface Mount Technology) defects. It includes modules for multimodal data acquisition, feature extraction, state prediction, risk assessment, and feedback control. The system acquires two-dimensional color and three-dimensional depth image data of the circuit board and extracts three-dimensional morphological topological features. Its core is to input these features into a cross-modal mapping proxy model, outputting physical and mechanical state prediction data, and combining this with environmental stress to calculate a dynamic failure risk quantification index. When the risk index falls within different preset ranges, the system generates corresponding release, process warning fine-tuning, or fatal defect interception and rework instructions. This invention overcomes the limitations of traditional static two-dimensional comparison, truly reflecting the spatial morphology of solder joints, and realizing a transformation from planar inspection to three-dimensional structural analysis and dynamic risk prediction.

Description

Technical Field

[0001] This invention relates to the field of automated visual inspection technology, specifically to a visual inspection system and method for SMT (Surface Mount Technology) defects. Background Technology

[0002] In the current SMT (Surface Mount Technology) inspection environment, the inspection equipment will collect visual images of circuit board components and solder joints to evaluate the soldering quality;

[0003] To analyze this image data, existing image processing solutions generally adopt a static appearance template comparison architecture, which extracts surface visual features such as pixel grayscale from two-dimensional images and matches and classifies them with standard samples. Although this solution has certain processing capabilities in conventional defect screening, it cannot truly reflect the volume distribution, meniscus curvature, and spatial contact morphology of weld joints because it relies heavily on two-dimensional planar images and lacks in-depth three-dimensional spatial structure analysis. This image processing method, which only stays at the level of visual appearance comparison, severs the mapping relationship between image geometry and actual physical stress state, which can easily lead to the misjudgment of weld joints with slightly off appearance but reliable structure, and the missed detection of high-risk weld joints with normal appearance but local stress concentration inside.

[0004] Therefore, how to deepen the extraction of three-dimensional morphological topological features from multimodal visual images and improve the accuracy and reliability of solder joint failure risk assessment has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a visual inspection system and method for SMT (Surface Mount Technology) defects, addressing the following technical problems: Existing SMT image processing solutions have technical deficiencies in the three-dimensional spatial structure depth analysis for solder joint quality assessment and in establishing a mapping relationship between image geometry and actual physical stress state. There is an urgent need for a visual inspection method and system for SMT defects that can deepen the extraction of three-dimensional morphological topological features from multimodal visual images, accurately quantify dynamic failure risks, and generate closed-loop feedback control commands for the production line. The objective of this invention can be achieved through the following technical solutions:

[0006] A visual inspection method for SMT (Surface Mount Technology) defects includes:

[0007] The system acquires traceability identification information of the target object on the high-definition circuit board to be inspected, as well as multimodal visual image data of the target object. The target object on the high-definition circuit board includes SMT surface mount components and solder joints. The multimodal visual image data is acquired based on a preset device global coordinate system and includes two-dimensional color image data and three-dimensional depth image data.

[0008] Feature extraction is performed on multimodal visual image data to generate three-dimensional morphological topological feature data;

[0009] The three-dimensional morphological topological feature data is input into a preset cross-modal mapping proxy model, and the physical and mechanical state prediction data is output; the physical and mechanical state prediction data includes the stress concentration factor feature matrix and the weak point distribution features.

[0010] By combining preset environmental stress parameters and physical and mechanical state prediction data, a dynamic failure risk quantification index is calculated.

[0011] Determine the preset range in which the dynamic failure risk quantification index falls, and generate corresponding feedback control instructions, specifically including:

[0012] If the dynamic failure risk quantification index is lower than the preset first risk threshold, a safety release instruction is generated as a feedback control instruction.

[0013] If the dynamic failure risk quantification index is greater than or equal to the first risk threshold and lower than the preset second risk threshold, a process warning instruction and a front-end fine-tuning instruction are generated as feedback control instructions.

[0014] If the dynamic failure risk quantification index is greater than or equal to the second risk threshold, a fatal defect interception instruction is generated as a feedback control instruction in combination with the traceability identification information, and the corresponding repair location indication data of the target object of the high-definition circuit board is generated; wherein, the first risk threshold is less than the second risk threshold.

[0015] Optionally, feature extraction is performed on the multimodal visual image data to generate three-dimensional morphological topological feature data, including:

[0016] Edge detection is performed on two-dimensional color image data to extract the two-dimensional contour features of the target object on the high-definition circuit board;

[0017] Spatial coordinate registration is performed between two-dimensional color image data and three-dimensional depth image data;

[0018] Based on the registered two-dimensional contour features, the three-dimensional depth image data is mapped and segmented to generate three-dimensional point cloud data of the target region.

[0019] Geometric topology calculations are performed on the 3D point cloud data of the target area to generate 3D morphological topology feature data; among which, the 3D morphological topology feature data includes wetting angle features, meniscus curvature features, volume distribution features, and contact cross-sectional area features.

[0020] Optionally, the cross-modal mapping proxy model is trained through the following steps:

[0021] Obtain historical three-dimensional topological feature data samples;

[0022] Obtain preset SMT solder material property parameters;

[0023] By combining the material property parameters of SMT solder, a finite element analysis of the consolidation and thermo-mechanical coupling of historical three-dimensional morphological topological feature data samples is performed to generate corresponding historical physical and mechanical state prediction data labels.

[0024] Using historical 3D morphological topological feature data samples and historical physical and mechanical state prediction data labels, a pre-set initial deep learning network is trained under supervision until the loss function of the initial deep learning network converges, thus obtaining a cross-modal mapping surrogate model.

[0025] Optionally, by combining preset environmental stress parameters and physical and mechanical state prediction data, a dynamic failure risk quantification index is calculated, including:

[0026] The environmental stress parameters are input into a preset stress transfer function to calculate the environmental stress load matrix; the environmental stress parameters include at least temperature cycle load parameters and mechanical vibration load parameters.

[0027] The environmental stress load matrix and the stress concentration factor feature matrix in the physical and mechanical state prediction data are multiplied by matrix to obtain the local stress tensor distribution matrix, and the local maximum stress value in the local stress tensor distribution matrix is ​​extracted based on the preset equivalent stress criterion.

[0028] Based on the preset material fatigue life curve, the probability of foundation failure corresponding to the local maximum stress value is determined.

[0029] By utilizing the distribution characteristics of weak points in the physical and mechanical state prediction data, the probability of basic failure is weighted and adjusted to generate a dynamic failure risk quantification index.

[0030] Optionally, if the dynamic failure risk quantification index is greater than or equal to a first risk threshold and lower than a preset second risk threshold, a process warning instruction and a front-end fine-tuning instruction are generated as feedback control instructions, including:

[0031] Obtain the preset standard shape range corresponding to the SMT standard process;

[0032] Extract the deviation feature vector from the preset standard morphological range in the three-dimensional morphological topological feature data;

[0033] Input the deviation feature vector into the preset process parameter mapping matrix to calculate the process compensation amount;

[0034] The front-end fine-tuning instructions are generated based on the process compensation amount; these instructions are used to adjust the operating parameters of the front-end production equipment upstream of the SMT production line.

[0035] Optionally, if the dynamic failure risk quantification index is greater than or equal to the second risk threshold, a fatal defect interception instruction is generated as a feedback control instruction by combining the traceability identification information, and corresponding repair location indication data for the high-definition circuit board target object is generated, including:

[0036] Extract the coordinates of the most probable weak point from the weak point distribution features;

[0037] Map the coordinates of the most probable weak point to the device's global coordinate system to generate global rework coordinates;

[0038] Based on global rework coordinates, critical defect interception instructions, and traceability identification information, rework location indication data for the corresponding high-definition circuit board target object is encapsulated and generated.

[0039] A visual inspection system for surface mount technology (SMT) defects, comprising:

[0040] The data acquisition module is used to acquire traceability identification information of the target object of the high-definition circuit board to be inspected, as well as multimodal visual image data of the target object of the high-definition circuit board. The target object of the high-definition circuit board includes SMT surface mount components and solder joints. Among them, the multimodal visual image data is acquired based on the preset device global coordinate system and includes two-dimensional color image data and three-dimensional depth image data.

[0041] The feature extraction module is used to extract features from multimodal visual image data and generate three-dimensional morphological topology feature data.

[0042] The model mapping module is used to input three-dimensional morphological topological feature data into a preset cross-modal mapping proxy model and output physical and mechanical state prediction data; wherein, the physical and mechanical state prediction data includes stress concentration coefficient feature matrix and weak point distribution features;

[0043] The risk quantification module is used to calculate the dynamic failure risk quantification index by combining preset environmental stress parameters and physical and mechanical state prediction data.

[0044] The decision feedback module is used to determine the preset range in which the dynamic failure risk quantification index falls and generate corresponding feedback control instructions. Specifically, it includes: if the dynamic failure risk quantification index is lower than a preset first risk threshold, generating a safety release instruction; if the dynamic failure risk quantification index is greater than or equal to the first risk threshold but lower than a preset second risk threshold, generating a process warning instruction and a front-end fine-tuning instruction; if the dynamic failure risk quantification index is greater than or equal to the second risk threshold, combining traceability identification information to generate a fatal defect interception instruction as a feedback control instruction, and generating rework location indication data for the corresponding high-definition circuit board target object; wherein, the first risk threshold is less than the second risk threshold.

[0045] Optionally, the feature extraction module is specifically used for:

[0046] Edge detection is performed on two-dimensional color image data to extract the two-dimensional contour features of the target object on the high-definition circuit board;

[0047] Spatial coordinate registration is performed between two-dimensional color image data and three-dimensional depth image data;

[0048] Based on the registered two-dimensional contour features, the three-dimensional depth image data is mapped and segmented to generate three-dimensional point cloud data of the target region.

[0049] Geometric topology calculations are performed on the 3D point cloud data of the target area to generate 3D morphological topology feature data.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. To address the problem in the background technology that heavily relies on two-dimensional planar images and lacks three-dimensional spatial structure analysis, this invention acquires multimodal visual image data containing two-dimensional color image data and three-dimensional depth image data, performs spatial coordinate registration and region mapping segmentation, and generates three-dimensional point cloud data of the target region; based on this, geometric topology calculations are performed to extract three-dimensional morphological topological feature data such as wetting angle features, meniscus curvature features, volume distribution features, and contact cross-sectional area features; this mechanism effectively reflects the true volume distribution and spatial contact morphology of the weld joint, overcoming the limitations of traditional static appearance template comparison;

[0052] 2. To address the problem in the background technology of severing the mapping relationship between image geometry and actual physical stress state, which leads to the missed detection of high-risk solder joints, this invention innovatively inputs the extracted three-dimensional morphological topological feature data into a cross-modal mapping proxy model, and outputs physical and mechanical state prediction data including stress concentration coefficient feature matrix and weak point distribution features. This model is trained by combining SMT solder material property parameters with finite element analysis of consolidation and thermo-mechanical coupling, which can penetrate the visual appearance and directly predict the structural fragility inside the solder joint, effectively reducing the missed detection rate of high-risk solder joints that appear normal but have internal stress concentration.

[0053] 3. To address the problem of misjudgment caused by a single appearance comparison in the background technology, this invention does not adopt a static, one-size-fits-all judgment standard. Instead, it inputs preset environmental stress parameters, including temperature cycle load parameters and mechanical vibration load parameters, into a preset stress transfer function, calculates the local maximum stress value by combining physical and mechanical state prediction data, and calculates a dynamic failure risk quantification index based on the material fatigue life curve and weak point distribution characteristics. This mechanism enables the test results to truly reflect the reliability of the weld joint in a specific service environment, effectively avoiding misjudging qualified weld joints that have slight appearance deviations but actually meet preset reliability indicators as defective.

[0054] 4. Based on the preset range of the dynamic failure risk quantification index, this invention generates tiered feedback control commands. For medium risk, it extracts the deviation feature vector and calculates the process compensation amount by combining it with the process parameter mapping matrix, generating front-end fine-tuning commands to guide the front-end production equipment to optimize operating parameters and achieve early process correction of defects. For high risk, it extracts the coordinates of the most probable weak point and generates global rework coordinates, combining them with traceability identification information to generate fatal defect interception commands and rework location indication data. This hierarchical mechanism not only realizes the closed-loop autonomous control of the production line, but also eliminates the blindness of secondary defect point searching after whole-board interception. Attached Figure Description

[0055] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0056] Figure 1 This is a flowchart of a visual inspection method for SMT chip defects provided by the present invention;

[0057] Figure 2 This is a structural diagram of a visual inspection system for SMT (Surface Mount Technology) defects provided by the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0059] Example 1:

[0060] Please see Figure 1 A visual inspection method for SMT (Surface Mount Technology) defects, comprising:

[0061] The system acquires traceability identification information of the target object on the high-definition circuit board to be inspected, as well as multimodal visual image data of the target object. The target object on the high-definition circuit board includes SMT surface mount components and solder joints. The multimodal visual image data is acquired based on a preset device global coordinate system and includes two-dimensional color image data and three-dimensional depth image data.

[0062] Feature extraction is performed on multimodal visual image data to generate three-dimensional morphological topological feature data;

[0063] The three-dimensional morphological topological feature data is input into a preset cross-modal mapping proxy model, and the physical and mechanical state prediction data is output; the physical and mechanical state prediction data includes the stress concentration factor feature matrix and the weak point distribution features.

[0064] By combining preset environmental stress parameters and physical and mechanical state prediction data, a dynamic failure risk quantification index is calculated.

[0065] Determine the preset range in which the dynamic failure risk quantification index falls, and generate corresponding feedback control instructions, specifically including:

[0066] If the dynamic failure risk quantification index is lower than the preset first risk threshold, a safety release instruction is generated as a feedback control instruction.

[0067] If the dynamic failure risk quantification index is greater than or equal to the first risk threshold and lower than the preset second risk threshold, a process warning instruction and a front-end fine-tuning instruction are generated as feedback control instructions.

[0068] If the dynamic failure risk quantification index is greater than or equal to the second risk threshold, a fatal defect interception instruction is generated as a feedback control instruction in combination with the traceability identification information, and the corresponding repair location indication data of the target object of the high-definition circuit board is generated; wherein, the first risk threshold is less than the second risk threshold.

[0069] This embodiment provides a visual inspection method for SMT (Surface Mount Technology) defects. Specifically, this embodiment deploys the method at the end of an SMT production line for automotive controller circuit boards. These circuit boards need to withstand temperature cycling near the engine compartment, road vibration, and long-term power-on / off loads during subsequent service. Therefore, judging solely based on whether the appearance of the solder joints matches the standard reference sample is prone to two types of deviations: first, solder joints with appearance deviations within a preset tolerance range but actually meeting preset reliability indicators are misjudged as defective; second, solder joints with no abnormal features exceeding a preset threshold but internal stress concentrations are missed. Therefore, this embodiment does not set the inspection endpoint as appearance classification, but further transforms multimodal visual information into a failure risk assessment of the solder joints in the actual use environment, and forms three different feedbacks: release, warning, or interception.

[0070] Specifically, when each high-definition circuit board on the production line enters the inspection station, its traceability identification information is first obtained by a barcode reader or radio frequency reading unit. This traceability identification information can be one or more combinations of board number, batch number, work order number, pick-and-place machine program version number, solder paste batch number, and upstream reflow oven curve version, in order to ensure that subsequent inspection results can be traced back to the specific board and process source.

[0071] The two-dimensional imaging component in the inspection station acquires color image data, while the three-dimensional imaging component acquires depth image data. Specifically, the two-dimensional imaging component includes a high-resolution industrial camera and a multi-angle programmable light source, which acquires two-dimensional color image data under different lighting conditions by controlling the light source to alternate illumination. The three-dimensional imaging component includes a phase-shift structured light projector or a laser contour sensor, which generates three-dimensional depth image data by projecting coded grating stripes onto the high-definition circuit board target object and combining it with a phase-shift solution algorithm. To achieve spatial registration of multimodal data, the two-dimensional imaging component and the three-dimensional imaging component are pre-calibrated using an offline calibration board to jointly calibrate the camera's intrinsic and extrinsic parameters, ensuring that the image data acquired by both can be converted in real time and aligned with high precision to the equipment's global coordinate system.

[0072] Two-dimensional images are better suited for identifying component boundaries, pad edges, solder surface gloss and contour shape, while three-dimensional depth images are better suited for reflecting solder joint height, meniscus uplift state, local collapse and volume distribution. Therefore, the two together constitute multimodal visual image data. All of the above data are collected in the same global coordinate system of the same equipment, so that subsequent positioning, comparison, defect output and rework instructions can be unified to the spatial position that the production equipment can execute.

[0073] After acquiring multimodal visual image data, the system extracts features from each solder joint area to generate three-dimensional morphological topology feature data. These three-dimensional morphological topology features differ from simple pixel grayscale statistics; instead, they reflect the geometric features that best reflect the actual connection state of the solder joints. For example, the wetting angle of the solder joint reflects whether the solder has been sufficiently spread onto the pad and pin surfaces; the meniscus curvature reflects whether the stress transition on the surface is smooth after the solder solidifies; the volume distribution reflects whether the solder is concentrated on one side, leading to uneven stress; and the contact cross-sectional area reflects whether the effective load channel for electrical connections and mechanical supports is sufficient. Through these features, the system essentially transforms visual information representing the surface morphology into structural information representing the connection state of the internal structure.

[0074] Furthermore, the system inputs the aforementioned three-dimensional morphological topological feature data into a preset cross-modal mapping proxy model and outputs physical and mechanical state prediction data. The role of this proxy model is to establish the correspondence between visual morphology and stress state. In other words, the system can assess the failure risk of the solder joint based on its current three-dimensional morphology without relying on physical products for long-term reliability testing, and predict the location and degree of stress accumulation that is more likely to occur under temperature expansion and contraction and mechanical vibration conditions. The stress concentration coefficient feature matrix in the output result can be understood as a structured description of the stress sensitivity of different local areas of the solder joint. The weak point distribution feature reflects which areas are more likely to become the starting points of crack initiation, interface peeling, or fatigue fracture. Its data structure is represented as a set of points or a spatial probability thermal distribution map containing three-dimensional spatial relative coordinates and their corresponding failure probability weights.

[0075] After obtaining the physical and mechanical state prediction data, the system calculates the dynamic failure risk quantification index by combining the preset environmental stress parameters. The environmental stress parameters can be preset according to the target product type. For example, the temperature cycle range of the vehicle controller exceeds the preset benchmark value, and the mechanical vibration spectrum is also higher than the preset load benchmark of ordinary consumer electronics. The system considers the environmental stress parameters together with the stress concentration state of the solder joint itself. The purpose is to avoid using the same static judgment standard for all solder joints. Because the same solder shortage phenomenon below the first area threshold may still work stably for a long time on the indoor home appliance control board, but may quickly evolve into a crack propagation source under high temperature and vibration alternating environment. Therefore, the risk quantification index in this embodiment does not simply describe whether the defect has typical characteristics, but describes the probability of the solder joint actually failing under the given service conditions.

[0076] In the risk decision-making stage, the system generates feedback control instructions based on the range of the dynamic failure risk quantification index. The first risk threshold and the second risk threshold are empirical critical values ​​pre-calibrated based on the actual failure statistics of the target product in historical temperature cycling tests and vibration fatigue tests. Specifically, the maximum risk index of historical qualified samples is calibrated as the first risk threshold, and the minimum risk index of samples that have experienced early fracture failure is calibrated as the second risk threshold. When the index is lower than the first risk threshold, it indicates that even if there is a slight appearance deviation, the structural tolerance of the weld joint can still meet the reliability requirements of the preset service environment, so a safe release instruction is generated. Such parts do not need to enter the manual review or rework stage, so as to reduce the need for secondary heating of weld joints that can be used normally.

[0077] If the index is between the first and second risk thresholds, it indicates that the solder joint has not yet reached the level of immediate failure, but its shape has deviated from the ideal process window. If similar shapes continue to appear in subsequent batches, it may cause a decrease in stability. Therefore, process warning instructions and front-end fine-tuning instructions are generated for upstream equipment adjustment. If the index reaches or exceeds the second risk threshold, it indicates that the solder joint has a significant risk of failure under the preset environmental stress. At this time, the system combines the traceability identification information to generate a fatal defect interception instruction and form rework location indication data so that the board can be accurately diverted to the rework station.

[0078] As an anomaly handling mechanism, situations such as image loss, localized glare, board warping, failure to read traceability tags, or equipment coordinate drift may occur in actual production lines. When there is a severe mismatch between 2D and 3D images, making it impossible to obtain reliable 3D topological features, the system can mark the solder joint or the entire board as requiring manual review instead of directly outputting a safe release conclusion, in order to avoid missed inspections due to incomplete data. When the traceability tag information fails to be obtained but the visual inspection result shows a high risk, the system can still perform physical interception first to prevent high-risk boards from flowing into the next process, and at the same time register the board as a traceability anomaly object, waiting for manual supplementation of identity information. When environmental stress parameters are not correctly loaded, the system can revert to the default conservative environment template, such as using a higher level of temperature cycling and vibration load as the highest preset safety level condition, thereby prioritizing the reliability safety boundary.

[0079] For example, on a high-definition circuit board for an onboard power domain control module, the system reads a high-reliability work order corresponding to a certain batch and collects two-dimensional color images and three-dimensional depth images of the solder joints at the four corners of a square flat leadless package chip. The system detects that one of the solder joints only shows that the curvature of one side of the meniscus is lower than the preset curvature lower limit and the surface reflection is uneven. If a traditional appearance template is used for comparison, it may be directly judged as defective due to the grayscale distribution deviation. However, this embodiment further extracts that the contact cross-sectional area of ​​the solder joint is still sufficient and the wetting angle is within the acceptable range. The surrogate model predicts that its stress concentration exceeds the benchmark value but does not exceed the first threshold. Under the corresponding onboard temperature cycle and vibration conditions, the overall risk is still lower than the first risk threshold. Therefore, the system directly generates a safe release command.

[0080] Conversely, for another inductor solder joint on the same board, although the outline is basically complete, the 3D depth map shows that the solder forms a steep cross section near the edge of the solder pad. The surrogate model predicts that there is a significant weak point at this location, and the probability of crack initiation under vibration conditions is high. Therefore, the system generates a fatal defect interception command and attaches rework coordinates for downstream workstations to process.

[0081] The purpose of this step is to upgrade SMT inspection from appearance comparison to risk assessment based on actual service reliability, thereby achieving the comprehensive technical effect of reducing false alarms and rework, reducing the number of risky boards that are missed, and enhancing the closed-loop control capability of the production line.

[0082] Feature extraction is performed on multimodal visual image data to generate three-dimensional morphological topological feature data, including:

[0083] Edge detection is performed on two-dimensional color image data to extract the two-dimensional contour features of the target object on the high-definition circuit board;

[0084] Spatial coordinate registration is performed between two-dimensional color image data and three-dimensional depth image data;

[0085] Based on the registered two-dimensional contour features, the three-dimensional depth image data is mapped and segmented to generate three-dimensional point cloud data of the target region.

[0086] Geometric topology calculations are performed on the 3D point cloud data of the target area to generate 3D morphological topology feature data; among which, the 3D morphological topology feature data includes wetting angle features, meniscus curvature features, volume distribution features, and contact cross-sectional area features.

[0087] This embodiment provides a mechanism for generating three-dimensional morphological topology features. Specifically, in the continuous detection scenario of the above-mentioned vehicle controller circuit board, if the solder joint shape is judged directly using only two-dimensional images, it is easily affected by the high reflectivity of the solder surface, the occlusion of the component body, the color change of the solder pad ink, and local contaminants. At this time, even if the contour abnormality is detected, it may not be able to truly reflect the volume and spatial spread of the solder. To make up for this deficiency, this embodiment further introduces a method of joint registration of two-dimensional contour and three-dimensional depth, which integrates the two-dimensional planar boundary positioning with the three-dimensional spatial height data to generate three-dimensional morphological topology feature data with engineering significance.

[0088] Specifically, the system first performs edge detection on the two-dimensional color image data. For example, it can use classic gradient-based edge detection algorithms such as the Canny operator, Sobel operator, and Laplacian operator, or a deep learning edge extraction model based on convolutional neural networks to extract the component outlines, pad boundaries, and solder joint outer edges of the target object on the circuit board. The purpose of edge detection is not simply to obtain a ring of pixel edges, but to identify the actual transition areas between solder and pads, solder and pins, and solder joints and background. In SMT soldering, when the solder is fully wetted, the solder joint outline usually shows a continuous transition. If there is a tendency for tombstoning, offset, floating, or cold solder joints, the outline will show asymmetry, abrupt gaps, or boundary interruptions. Therefore, two-dimensional contour features can be used to coarsely locate the target area first.

[0089] The system performs spatial coordinate registration between 2D color image data and 3D depth image data. This registration can be completed based on the camera intrinsic and extrinsic parameter relationships established during device calibration, the structured light projection model, and the reference point position of the circuit board. The registration result allows a corresponding spatial region to be found in the 3D depth map for a solder joint outline region in the 2D image. The engineering significance of this is that the 2D image is used to provide the planar location of the target region, and the 3D depth image is used to provide the spatial 3D morphological features of the corresponding target region. Without registration, the height peaks and valleys in the 3D depth data may not accurately correspond to specific solder joint regions, especially in the case of close-pin components or adjacent solder joints with a spacing less than a preset spacing threshold. This would result in a higher probability of mixing the height information of adjacent solder joints into the same analysis object than the preset probability threshold.

[0090] Based on the registered 2D contour features, the system performs region mapping and segmentation on the 3D depth image data to generate 3D point cloud data of the target region. The target region can be a single solder joint or a set of multiple solder joints of the same component. The segmented point cloud data retains the spatial undulations, edge transitions, and true height distribution of the contact area with the solder pad on the solder joint surface. Then, the system performs geometric topology calculations on the 3D point cloud data of the target region to generate 3D morphological topology feature data.

[0091] Specifically, the wetting angle characteristic reflects the spreading slope at the junction of the solder and the pad or pin; the meniscus curvature characteristic reflects the degree of arc continuity of the solder joint surface. If the curvature changes abruptly, it often means uneven stress transition; the volume distribution characteristic reflects whether there is off-center load accumulation of solder in different directions of the solder joint; the contact cross-sectional area characteristic reflects the effective connection range in which the solder actually participates in conduction and support; the above characteristics together constitute a geometric description of the structural health of the solder joint.

[0092] To meet the requirement of full disclosure, the following simplified logical deduction method is used to explain how the data flows. Suppose that a closed contour region R1 is extracted from a 2D image of a solder joint, and after registration, it is mapped to a point cloud set P1 in a 3D depth map. P1 can be simplified into several height sampling points. Based on this, the system obtains: the left wetting angle is less than the lower limit of the standard wetting angle, and the right wetting angle is greater than the upper limit of the standard wetting angle; the surface center curvature is continuous, but the curvature at the lower right corner changes abruptly; the overall solder volume shifts to the left; and the effective contact area with the solder pad shrinks on the right. Then, the feature vector finally formed by the solder joint is composed of the difference between the left and right wetting angles, the local curvature anomaly of the meniscus, the volume distribution offset, and the degree of unevenness of the contact cross-sectional area. Compared with using only the single visual indicator of whether the solder joint area meets the standard, the aforementioned feature vector can more accurately represent the actual soldering formation state.

[0093] As an anomaly handling mechanism, when the edges of a 2D image are broken due to strong reflection, the system can combine adjacent frame compensation, color gradient smoothing, or board-level prior pad templates to perform contour repair. If the contour still cannot be closed, the area is marked as a low-confidence segmentation result, and the direct generation of final topological features is paused. When the 3D depth map has shadows or high noise in some areas, point cloud denoising and smoothing reconstruction can be performed first. If the number of points in the target area after denoising is still lower than the preset lower limit, the area will not be used for high-confidence risk prediction and will be transferred to the review process. If the pin spacing of a certain device is too small, causing the point clouds of adjacent solder joints to stick together, the system can combine the pad spacing information in the 2D contour to perform secondary segmentation to avoid two solder joints being mistaken for the same structure.

[0094] For example, in the inspection of square flat packaged devices on the same vehicle controller circuit board, a certain pin solder joint only appears as a slight dark spot on the two-dimensional image. However, the point cloud set extracted after registration shows that the height is normal on the side near the pin root, while the meniscus on the side near the outer side of the solder pad is obviously collapsed. The system further generates the corresponding wetting angle features and contact cross-sectional area features, and finds that the solder joint still has sufficient contact area, but the surface morphology is slightly asymmetrical. Therefore, the subsequent risk assessment may not directly classify it as high-risk. Another adjacent solder joint has a complete outline on the two-dimensional image, but the point cloud shows that the middle is arched and the contact area at both ends is narrowed. The generated volume distribution and curvature features indicate that its force path is discontinuous, so it deserves more attention in the subsequent risk prediction.

[0095] The purpose of this step is to provide the positioning boundary with two-dimensional contours, provide the spatial structure with three-dimensional point clouds, and transform the weld point from an image object into a structural object that can be used for mechanical deduction through geometric topology calculations, thereby achieving a fine-grained expression of the actual forming quality of the weld point.

[0096] The cross-modal mapping proxy model is trained through the following steps:

[0097] Obtain historical three-dimensional topological feature data samples;

[0098] Obtain preset SMT solder material property parameters;

[0099] By combining the material property parameters of SMT solder, a finite element analysis of the consolidation and thermo-mechanical coupling of historical three-dimensional morphological topological feature data samples is performed to generate corresponding historical physical and mechanical state prediction data labels.

[0100] Using historical 3D morphological topological feature data samples and historical physical and mechanical state prediction data labels, a pre-set initial deep learning network is trained under supervision until the loss function of the initial deep learning network converges, thus obtaining a cross-modal mapping surrogate model.

[0101] This embodiment provides a training mechanism for a cross-modal mapping proxy model. Specifically, in the aforementioned production line scenario, if only manual experience is relied upon to establish an association rule base for specific appearance morphology corresponding to specific failure risks, the rules are prone to failure when there are diverse device packages, changes in solder types, and complex service environments. In particular, for some solder joints with subtle appearance differences but significant differences in stress states, it is difficult to establish a highly generalizable association model through conventional manual preset rules. Therefore, this embodiment introduces a proxy model training mechanism based on historical three-dimensional morphological topological features and finite element analysis results, enabling the system to learn the intrinsic correspondence between morphological changes and physical stress.

[0102] Specifically, first, historical three-dimensional topological feature data samples are acquired; these samples can come from past production line image data, offline experimental data, and specially constructed process test templates; each sample corresponds to the real three-dimensional geometric features of a certain type of solder joint, such as insufficient solder, offset, insufficient wetting, one-sided solder buildup, lead floating, and other morphological states; then, preset SMT solder material property parameters are acquired; these material property parameters may include solder elastic modulus, thermal expansion characteristics, yield behavior, creep characteristics, and related material constants at the solder pad metal interface;

[0103] The reason for introducing material property parameters is that the thermal fatigue response of the same solder joint geometry is not the same under different solder systems or different metal interface conditions; if material property constraints are lacking, the model will only learn surface similarity relationships, rather than physical associations with transferability.

[0104] The system combines the aforementioned material property parameters to perform finite element analysis of consolidation and thermo-mechanical coupling on historical three-dimensional morphological topological feature data samples, generating corresponding historical physical and mechanical state prediction data labels. Here, consolidation can be understood as the stabilization process of the overall structure after weld point formation in the contact interface, support path, and local shrinkage state. Thermo-mechanical coupling reflects the additional stress generated by the weld point due to the mismatch of material thermal expansion when the temperature changes. Through finite element analysis, the system can obtain the stress concentration area, strain distribution characteristics, and potential weak point locations of each sample under specific temperature cycles and mechanical loads, thereby forming label data that can be used for supervised training.

[0105] Specifically, the implementation steps of finite element analysis include: constructing a basic three-dimensional geometric model containing standard pads and pins; using the extracted three-dimensional morphological topology feature data as driving variables, deforming and reconstructing the basic model to generate the corresponding solder joint three-dimensional solid model; importing the three-dimensional solid model into the finite element analysis software and generating a finite element mesh model; applying geometric constraints to the finite element mesh model and loading the corresponding thermal cycling and mechanical vibration load boundary conditions, performing thermo-mechanical coupling calculations through the solver, and outputting the corresponding physical and mechanical state prediction results; specifically, the SMT solder material property parameters are input as material constitutive model parameters into the finite element solver for calculation. After the calculation is completed, the stress distribution cloud map on the model mesh node is extracted as the stress concentration coefficient feature matrix label, and the coordinate set of the top 5% of nodes with the fastest plastic strain accumulation is extracted as the weak point distribution feature label. The two are combined to form a historical physical and mechanical state prediction data label that uniquely corresponds to each historical three-dimensional morphological topology feature data sample.

[0106] To illustrate the correspondence between training samples, a simplified technical deduction example can be given. Assume there are three solder joint morphology samples T1, T2, and T3 in the historical samples. T1 is characterized by a uniform wetting angle and symmetrical volume distribution; T2 is characterized by solder buildup on one side and a slightly reduced contact cross-sectional area; T3 is characterized by appearance meeting the preset surface morphology tolerance but with a sudden change in the curvature of the meniscus at the outer edge of the solder pad. After finite element analysis, the label corresponding to T1 shows a smooth overall stress distribution; the label corresponding to T2 shows an increase in stress on one side; and the label corresponding to T3 shows local high-stress hotspots and obvious weak points at the edge of the solder pad. The initial deep learning network receives morphology samples of T1 to T3 and uses the corresponding physical labels as learning targets, gradually learning to directly infer the distribution of stress-sensitive areas and weak points from geometric topological features.

[0107] Specifically, the initial deep learning network can adopt a multilayer perceptron architecture. At the network input, three-dimensional topological feature data such as wetting angle, meniscus curvature, volume distribution, and contact cross-sectional area are concatenated into a fixed-dimensional input feature vector. At the network output, the full-field stress distribution grid data output by finite element analysis is downsampled into a structured stress concentration coefficient feature matrix and a weak point probability distribution vector. During training, the mean square error is used as the loss function, and the network weights are continuously updated through the backpropagation algorithm, so that the network output continuously approaches the historical physical and mechanical state prediction data labels generated by finite element analysis until the loss function converges to a preset threshold range, thereby establishing a cross-modal mapping proxy model with physical prior constraints. When a new weld point feature to be tested is input, the network does not need to perform a complete finite element simulation again to output an approximate physical and mechanical state prediction result.

[0108] As an anomaly handling mechanism, if the coverage of historical samples is insufficient, such as containing only a few package types or a single solder material, the model may exhibit insufficient generalization on new devices. In this case, the training set can be expanded by adding samples with different packages, different pad structures, and different process deviations. If the material property parameters on which the finite element analysis is based are inconsistent with the current production line solder batches, the material library should be updated first before incremental training to avoid the labels becoming disconnected from the actual process. If, during the training process, it is found that the three-dimensional features of some samples clearly conflict with the finite element labels, such as geometric distortion caused by deviations in the reconstruction of the original point cloud, these anomalous samples should be removed from the training set or marked separately as low-confidence samples to prevent the surrogate model from being misled by noisy samples.

[0109] For example, in the early stages of an onboard controller project, the system can acquire three-dimensional morphological data of multiple batches of square flat leadless packages, ball grid array packages, and power inductor solder joints from history as a training basis, and perform finite element analysis on them according to automotive-grade temperature cycling and vibration conditions. A certain type of power inductor solder joint may only show slightly higher solder accumulation on one side of the solder foot, but the finite element label reflects that it is prone to stress concentration at the root on the opposite side during repeated vibration. After the proxy model is trained, it can directly output mechanical risk warnings exceeding the safety threshold for volume distribution deviations that do not show obvious abnormal appearance characteristics in subsequent online detection, without having to wait for long-term vibration test results.

[0110] The purpose of this step is to use the physical priors provided by finite element analysis to establish a supervisory benchmark for the model, thereby achieving a fast approximate mapping from image geometric features to physical stress state, enabling online detection to have both speed and reliability assessment capabilities.

[0111] By combining preset environmental stress parameters and physical and mechanical state prediction data, a dynamic failure risk quantification index is calculated, including:

[0112] The environmental stress parameters are input into a preset stress transfer function to calculate the environmental stress load matrix. The environmental stress parameters include at least temperature cycling load parameters and mechanical vibration load parameters. The temperature cycling load parameters and mechanical vibration load parameters are set according to the environmental reliability test standard documents of the target product's corresponding industry, or the measured extreme environmental stress data collected by sensors in the actual service scenario of the target product. The preset stress transfer function is a linear transformation matrix constructed with macroscopic environmental stress parameters as independent variables and local characteristic stresses as dependent variables.

[0113] The environmental stress load matrix and the stress concentration factor feature matrix in the physical and mechanical state prediction data are multiplied by matrix to obtain the local stress tensor distribution matrix, and the local maximum stress value in the local stress tensor distribution matrix is ​​extracted based on the preset equivalent stress criterion.

[0114] Based on the preset material fatigue life curve, the probability of foundation failure corresponding to the local maximum stress value is determined.

[0115] By utilizing the distribution characteristics of weak points in the physical and mechanical state prediction data, the probability of basic failure is weighted and adjusted to generate a dynamic failure risk quantification index.

[0116] This embodiment provides a dynamic failure risk quantification mechanism. Specifically, in the aforementioned method, if the quality of solder joints is determined solely based on the stress concentration coefficient feature matrix output by the surrogate model, it will be impossible to fully characterize the true reliability of the product under different actual usage environments. The risk of the same solder joint structure is not the same on different end products. Therefore, this embodiment further introduces environmental stress parameters into the risk quantification process, so that the test results are geared towards specific application scenarios rather than remaining at the level of abstract structural evaluation.

[0117] Specifically, the system first inputs the environmental stress parameters into a preset stress transfer function to obtain the environmental stress load matrix; the environmental stress parameters here include at least temperature cycle load parameters and mechanical vibration load parameters; the stress transfer function is specifically the equivalent stiffness matrix and transfer coefficient polynomial extracted in advance through whole-plate level finite element modal analysis, which is used to linearly or nonlinearly map the macroscopic whole-plate deformation and the environmental temperature gradient into local mechanical boundary conditions applied to the pins and solder joints of individual components;

[0118] The specific construction steps are as follows: establish a whole plate finite element benchmark model and apply a unit environmental load to extract the discrete dataset of macroscopic deformation and local response; then use a multivariate regression algorithm to fit the discrete dataset to generate a transfer coefficient polynomial, and derive the equivalent stiffness matrix from the polynomial.

[0119] Temperature cyclic load mainly reflects the thermal expansion mismatch tension experienced by the solder joint during alternating hot and cold processes; mechanical vibration load mainly reflects the periodic bending and inertial effects caused by vehicle driving, engine operation, or impact conditions; the engineering significance of the stress transfer function lies in transforming external environmental conditions into load expressions that can act on the local structure of the solder joint; since different packages, different pad layouts, and different component mass distributions will change the way external loads are transferred to the solder joint, this function can be pre-set with different templates based on device type, board structure, or product grade;

[0120] The system performs matrix multiplication on the environmental stress load matrix and the stress concentration factor characteristic matrix to generate local maximum stress values. Let the environmental stress load matrix be E, which represents the global nominal stress components under a specific working condition, including normal stress and shear stress in all directions. Let the stress concentration factor characteristic matrix be K, which records the geometric amplification factor of each local area of ​​the weld joint to external loads. Multiplying the environmental stress load matrix E by the stress concentration factor characteristic matrix K yields the true stress tensor distribution matrix S for each local area, calculated using the following formula:

[0121]

[0122] Where × is the matrix multiplication operator; to satisfy the matrix multiplication operation rules, the environmental stress load matrix E is represented as a column vector containing the nominal stress components in each direction, the stress concentration coefficient feature matrix K is a transformation matrix with the same dimension, and the local stress tensor distribution matrix S calculated in this way is the local stress column vector corresponding to the external load.

[0123] According to the preset von Mises equivalent stress criterion, the stress tensor of each region in the stress tensor distribution matrix S is converted into a single scalar equivalent stress value; the highest value among all local equivalent stress values ​​is extracted and output as the local maximum stress value. This process can be understood as follows: the environmental load tells the system how much repeated tension and vibration are applied from the outside, and the stress concentration factor tells the system which location on the weld joint is most likely to amplify the external forces; the sum of the two reflects the local stress intensity that the most dangerous location on the weld joint may bear under actual working conditions; then, based on the preset material fatigue life curve, the system determines the maximum local stress value. The corresponding base failure probability;

[0124] Specifically, the system uses the Coffin-Manson thermal fatigue model or the Baskin high-cycle fatigue equation to calculate the local maximum stress value. The corresponding expected number of failure cycles is substituted into the preset Weibull reliability distribution function to calculate the cumulative damage probability of the target product within the design life cycle, which is then used as the base failure probability.

[0125] The specific calculation logic is as follows: Taking the Baskin high-cycle fatigue equation under mechanical vibration load as an example, the expected number of failure cycles... The calculation formula is:

[0126]

[0127] in, The local maximum stress value; fatigue strength coefficient. With fatigue strength index All materials are retrieved from the preset material database by the system; under temperature cyclic loading, the Coffin-Manson thermal fatigue model is called accordingly; the expected number of failure cycles is set. Substituting into the Weibull distribution function, we obtain the basic failure probability. :

[0128]

[0129] in, The preset total target load cycle number for the target product during its design life; The shape parameters are derived from historical test samples evaluated by the system; the material fatigue life curve can be obtained from the solder material test database, used to represent the durability of a certain solder under different stress levels; the preset material database is constructed by summarizing the discrete data points of real cycle life obtained from standard tensile and shear fatigue bench tests of various known SMT solders; shape parameters It is a constant obtained by fitting a Weibull distribution to the failure time data of pre-collected historical test samples;

[0130] The system extracts the spatial coordinates of the most probable weak point from the weak point distribution characteristics, calculates the spatial Euclidean distance between these spatial coordinates and the region corresponding to the local maximum stress value, and constructs an exponential decay weight function with this spatial Euclidean distance as the independent variable. Specifically, the exponential decay weight function is set as follows: ,in The weight function value, For spatial Euclidean distance. This is a pre-defined attenuation constant based on the characteristic dimensions of the component, used to characterize the rate attenuation of the influence of stress source on weak points as distance increases; attenuation constant The value is the reciprocal of the physical characteristic length of the pad corresponding to the component; the dynamic failure risk quantification index is obtained by multiplying the base failure probability by the value of the index decay weight function.

[0131] The following simplified example illustrates the logic: Assume the stress concentration coefficient feature matrix of a solder joint can be abstracted into three regions, corresponding to local sensitive regions Z1, Z2, and Z3, with stress concentration coefficients k1, k2, and k3 respectively, where region Z2 has the highest coefficient k2. The above region division is obtained by forcibly dividing the solder joint space into three feature partitions—the root, the middle, and the end near the outer edge—along the component pin extension direction based on a preset three-dimensional bounding box geometric boundary. The stress concentration coefficient values ​​of each region are specifically determined based on the ratio of the local maximum principal stress to the overall nominal stress predicted by the cross-modal mapping proxy model. If the board is used in ordinary indoor equipment, the environmental stress load matrix is ​​low, and even if region Z2 is the most sensitive point, the final local maximum stress may still correspond to a basic failure probability lower than the preset safety benchmark. If the same solder joint is used in a high-vibration automotive scenario, the environmental stress load is enhanced, and the amplification effect of coefficient k2 in region Z2 exceeds the safety threshold, thus increasing the basic failure probability.

[0132] Furthermore, if the distribution characteristics of weak points show that the main weak point is located in region Z2, the dynamic failure risk quantification index is further increased; if the weak points are mainly distributed in secondary regions such as region Z1 and region Z3, the final risk can be lower than the former case. It can be seen that this embodiment does not simply treat all appearance defects, but evaluates the structural vulnerability in conjunction with environmental conditions.

[0133] As a fault-tolerance mechanism, if environmental stress parameters cannot be read from the product configuration file, such as when a new work order has not yet been entered into the terminal application level, the system can call a conservative default environment template to avoid underestimating the risk. If a certain type of product has multiple possible service scenarios, such as being compatible with standard automotive-grade and stringent automotive-grade versions, the system can output risk quantification indices for multiple scenarios, which process engineers can select according to the order level. If the confidence level of the stress concentration coefficient feature matrix output by the proxy model is insufficient, or the distribution of weak points is too discrete, making it difficult to determine the main weak area, the system can attach an uncertainty mark to the risk index and trigger manual review instead of directly entering the automatic release process.

[0134] For example, on the same vehicle controller circuit board, a microcontroller pin solder joint and a high-current inductor solder joint only show slight asymmetry in appearance. For the microcontroller pin solder joint, since the device mass is below the preset threshold and the vibration inertial load does not exceed the preset safety range, even if there is some stress concentration, the probability of basic failure corresponding to the local maximum stress is still low, and the final risk quantification index falls into the safe range. For the high-current inductor solder joint, since the device mass exceeds the preset threshold and is located in the vibration amplification area of ​​the board edge, the environmental load is transmitted and forms stronger local stress at the root of the solder joint. The weak point is concentrated in this area, so the final risk quantification index enters the high-risk range, and the system performs interception accordingly.

[0135] The purpose of this step is to link the state of the solder joint structure with the service conditions of the target product, thereby achieving consequence-oriented risk quantification and avoiding a disconnect between static appearance judgment and real reliability requirements.

[0136] If the dynamic failure risk quantification index is greater than or equal to the first risk threshold and lower than the preset second risk threshold, a process warning instruction and a front-end fine-tuning instruction are generated as feedback control instructions, including:

[0137] Obtain the preset standard shape range corresponding to the SMT standard process;

[0138] Extract the deviation feature vector from the preset standard morphological range in the three-dimensional morphological topological feature data;

[0139] Input the deviation feature vector into the preset process parameter mapping matrix to calculate the process compensation amount;

[0140] The front-end fine-tuning instructions are generated based on the process compensation amount; these instructions are used to adjust the operating parameters of the front-end production equipment upstream of the SMT production line.

[0141] This embodiment provides a closed-loop adjustment mechanism for processes under medium-risk conditions. Specifically, in the aforementioned scheme, when the dynamic failure risk quantification index of certain solder joints has not yet reached the level that must be intercepted, if the system only outputs a warning without providing actionable adjustment suggestions to the front-end production equipment, similar deviations may continue to accumulate on subsequent boards, eventually evolving into batch risks. Therefore, this embodiment introduces a linkage mechanism of standard morphology range, deviation feature vector, and process compensation amount in the medium-risk range, so that the system not only points out the existence of problems, but also promotes the production line to make fine-tuning corrections before the problems expand.

[0142] Specifically, the system first obtains the preset standard shape range corresponding to the SMT surface mount standard process. This standard shape range can be set separately for different device packages, pad designs, and solder processes. For example, the wetting angle range of a certain type of square flat leadless package solder joint, the volume distribution balance range of a certain type of chip resistor solder joint, and the lower limit range of the contact cross-sectional area of ​​a certain type of high current device solder pin. The emphasis here is on the range rather than a single ideal value because the shape of solder joints has reasonable fluctuations in actual production. If all solder joints are compressed to the target value with a tolerance less than the preset fluctuation lower limit, it will amplify false alarms and overcorrection.

[0143] The system extracts deviation feature vectors from the standard morphological range from the three-dimensional morphological topology feature data. These deviation feature vectors can comprehensively represent deviations in multiple directions. For example, a solder joint may exhibit a slightly smaller wetting angle, a volume distribution shifted to one side, and a contact cross-sectional area close to the lower limit, but no obvious fatal weak point has yet appeared. The system combines these deviations in a structured manner, which facilitates subsequent mapping to possible process causes. Its engineering significance lies in the fact that different morphological deviations usually correspond to different front-end process causes: insufficient solder may be related to insufficient solder paste printing, offset may be related to pick-and-place machine nozzle calibration or component placement posture, and local collapse may be related to reflow thermal profiles or pad contamination.

[0144] Specifically, the calculation process for extracting the deviation feature vector is as follows: Assume there are a total of three-dimensional morphological feature data. The dimension, the first The actual measured value of each feature is The corresponding standard morphological range is First calculate the absolute deviation. :like , ;like , ;like In Inside, ; Calculate the dimensionless relative deviation ; All of The column vectors are generated by sequential combination, which are the deviation feature vectors. This process eliminates the dimensional differences between parameters, ensuring the accuracy of subsequent matrix mapping;

[0145] The system inputs the deviation feature vector into a preset process parameter mapping matrix to calculate the process compensation amount; this mapping matrix is ​​essentially a sensitivity coefficient matrix established based on historical orthogonal experiments, denoted as... Its rows correspond to various three-dimensional shape deviation features, and its columns correspond to adjustable parameters of the front-end equipment, such as patch X / Y axis offset, nozzle angle compensation, and printer demolding speed; for the extracted deviation feature vectors that deviate from the preset standard shape range The system uses this deviation feature vector With sensitivity coefficient matrix Perform matrix multiplication to obtain the process compensation vector. The calculation formula is as follows:

[0146]

[0147] Among them, the process compensation vector It contains the sum of weighted correction values ​​required for each front-end process parameter, thus forming a comprehensive process compensation quantity containing multiple parameter adjustment instructions; to ensure that the physical dimensions on both sides of the equation are consistent, the deviation feature vector D is a standardized feature vector that has been dimensionless, or each matrix element in the sensitivity coefficient matrix M contains a unit conversion coefficient used to convert the dimensions of different morphological deviation features into the dimensions of the corresponding front-end production equipment process parameters.

[0148] To ensure the stability of the production line operation, before packaging the instruction, the system compares the process compensation amount with the preset safety boundary of the physical limit parameters of the front-end equipment. If a certain compensation amount is calculated to exceed the limit, it is truncated. The system packages and generates a front-end fine-tuning instruction based on the truncated process compensation amount and sends it to the corresponding upstream equipment or manufacturing execution system.

[0149] To illustrate the correspondence between the deviation vector and the compensation amount, a simplified example can be used. Suppose that the standard shape range of a certain type of chip resistance solder joint requires the wetting angle to be in range A, the volume distribution balance to be in range B, and the contact cross-sectional area to be in range C. In a certain detection, the actual characteristics obtained show that the wetting angle is within range A, but the volume distribution is biased to the left, and the contact cross-sectional area falls near the lower edge of range C. Then the system forms a deviation feature vector D1, where the weight of the volume distribution biased to the left is higher than that of other terms. After mapping, the process compensation amount mainly points to the fine adjustment of the chip coordinates and the correction of the printing offset, rather than performing a reflow temperature adjustment with an adjustment amount greater than the preset range. This can avoid misjudging local placement deviations as overall thermal process abnormalities.

[0150] As a fault-tolerance mechanism, if a certain deviation feature vector points to multiple process sources simultaneously, and the mapping result is not unique, the system can divide the front-end fine-tuning instructions into two levels of output: suggested execution items and only prompts for checks. This avoids production line oscillation caused by automatically modifying multiple parameters on a large scale. If several consecutive boards trigger process warnings in the same direction, the system can increase the priority of the fine-tuning suggestion. If it is only an occasional deviation on a single board, the trend can be recorded first without immediately issuing a forced adjustment. If the upstream equipment required for mapping is currently in a locked state or in manual confirmation mode, the system can first write the compensation amount into the process log and Kanban prompts, and then execute it after the engineer confirms it, in order to ensure production safety.

[0151] For example, on the same vehicle controller production line, the system continuously detected that the volume distribution of one side of the solder joints of a certain batch of chip resistors was consistently biased towards the inside of the circuit board, and the risk quantification index was in the medium risk range. Since these solder joints did not reach the point where rework was necessary, the system did not directly intercept the board. Instead, it extracted the deviation feature vectors of the unilateral offset of the volume distribution and the marginalization of the contact cross-sectional area. After mapping, the front-end fine-tuning command pointed to the correction within the preset tolerance range of the chip mounter nozzle center compensation and the coordinates of the component station. After execution, the solder joints of the same type in subsequent batches returned to a more symmetrical shape distribution, and the number of medium risk alarms decreased accordingly.

[0152] The purpose of this mechanism is to transform online detection results into actionable process feedback, thereby enabling early correction when the risk is still within a controllable range and reducing the probability that similar morphological deviations will continue to amplify in subsequent production.

[0153] If the dynamic failure risk quantification index is greater than or equal to the second risk threshold, a fatal defect interception instruction is generated as a feedback control instruction based on the traceability identification information, and corresponding repair location indication data for the high-definition circuit board target object is generated, including:

[0154] Extract the coordinates of the most probable weak point from the weak point distribution features;

[0155] Map the coordinates of the most probable weak point to the device's global coordinate system to generate global rework coordinates;

[0156] Based on global rework coordinates, critical defect interception instructions, and traceability identification information, rework location indication data for the corresponding high-definition circuit board target object is encapsulated and generated.

[0157] This embodiment provides a mechanism for intercepting and locating critical defects under high-risk conditions. Specifically, in the aforementioned process, when a solder joint is determined to be high-risk, if the system only outputs that the entire board is unqualified without specifying the exact dangerous location, the repair personnel still need to search for the defect point on the entire circuit board a second time. Especially in high-density packaging and multi-layer device mixed assembly scenarios, it is easy to cause repeated heating range to exceed the preset repair safety area or even cause incorrect repair. Therefore, this embodiment further outputs the coordinates of the most probable weak point and the global repair coordinates based on the high-risk judgment, so that the defective board can be quickly and accurately diverted and repaired.

[0158] Specifically, the system extracts the coordinates of the most probable weak point from the distribution characteristics of weak points. These coordinates correspond to the location in the local area of ​​the solder joint where crack initiation, interface peeling, or fatigue failure is most likely to occur. Compared to simply outputting the solder joint defect, directly providing the coordinates of the weak point has greater operational significance because rework personnel or rework equipment need to know whether the problem occurs at the root of the solder foot, the outer edge of the solder pad, the bottom center of the component, or a certain side corner. The system maps the coordinates of the most probable weak point to the equipment's global coordinate system to generate global rework coordinates. Since the entire inspection station, sorting device, and rework workstation share or can be converted to the same global coordinate reference, the rework coordinates can be directly called by subsequent equipment to locate the working path of laser indicators, AOI repeaters, or automatic rework heads.

[0159] The system encapsulates and generates rework location indication data based on global rework coordinates, critical defect interception instructions, and traceability identification information. This data can include board identity, component tag number, solder joint number, coordinate location, risk level, and weak point type. Traceability identification information is particularly important at this stage because high-risk boards often need to be traced back to specific batches, process procedures, and material sources for subsequent quality analysis and process rectification. By binding defect types with identity information, the system can quickly determine whether a certain type of high risk is concentrated in a specific furnace temperature profile, a specific material station, or a specific solder batch.

[0160] To meet the requirement of full disclosure, the following simplified example illustrates the positioning process. Assume there are three candidate regions W1, W2, and W3 in the weak point distribution map of a solder joint, with W2 having the highest probability. The system records the position of W2 in the local coordinate system of the solder joint as local coordinate L2. Since the registration of the local region of the solder joint with the global coordinate system of the entire board has been completed during detection, L2 can be converted to global coordinate G2. Subsequently, the rework location indication data can include board number 123, component number 15, pin 7, rework coordinate G2, and the processing level red interception. After reading this data, the rework equipment can directly send the board to the designated workstation and perform local magnification display, resoldering, or replacement operations near G2.

[0161] As a fault-tolerance mechanism, if multiple similar probability peaks appear in the distribution characteristics of weak points, the system can simultaneously output the primary rework coordinates and secondary rework coordinates, prompting rework personnel to prioritize checking the primary coordinates while also considering adjacent areas; if the global coordinate mapping finds that the board reference point identification fails or the board's posture shifts during transport, the system should re-execute the reference correction; before the correction is completed, only board interception can be performed, and automatic rework coordinates should not be directly issued to avoid mispositioning of the rework head; if traceability identification information is missing but the risk has clearly reached a high level, the system can first encapsulate the rework task with a temporary internal serial number and prevent the board from flowing to the next process, and archive it after the traceability information is supplemented later.

[0162] For example, in the power inductor detection of an on-board controller circuit board, the system determines that the dynamic failure risk quantification index at its left solder pad exceeds the second risk threshold; further analysis reveals that the most probable weak point is located at the root of the solder pad near the outer edge of the pad; the system maps this location to a global rework coordinate in the overall board coordinate system, and writes the board number, component reference number, and the reason for the high risk of crack initiation under high vibration conditions into the rework location indication data; the board is sorted to the rework station, and the operator can directly see the highlighted indication of the corresponding coordinate in the microscopic rework interface without having to repeatedly search the entire board;

[0163] The purpose of this mechanism is to further translate the high-risk assessment results into actionable repair and location measures, thereby enabling precise interception, rapid repair, and source tracing of high-density components.

[0164] Example 2:

[0165] Please see Figure 2 A visual inspection system for SMT (Surface Mount Technology) defects, comprising:

[0166] The data acquisition module is used to acquire traceability identification information of the target object of the high-definition circuit board to be inspected, as well as multimodal visual image data of the target object of the high-definition circuit board. The target object of the high-definition circuit board includes SMT surface mount components and solder joints. Among them, the multimodal visual image data is acquired based on the preset device global coordinate system and includes two-dimensional color image data and three-dimensional depth image data.

[0167] The feature extraction module is used to extract features from multimodal visual image data and generate three-dimensional morphological topology feature data.

[0168] The model mapping module is used to input three-dimensional morphological topological feature data into a preset cross-modal mapping proxy model and output physical and mechanical state prediction data; wherein, the physical and mechanical state prediction data includes stress concentration coefficient feature matrix and weak point distribution features;

[0169] The risk quantification module is used to calculate the dynamic failure risk quantification index by combining preset environmental stress parameters and physical and mechanical state prediction data.

[0170] The decision feedback module is used to determine the preset range in which the dynamic failure risk quantification index falls and generate corresponding feedback control instructions. Specifically, it includes: if the dynamic failure risk quantification index is lower than a preset first risk threshold, generating a safety release instruction; if the dynamic failure risk quantification index is greater than or equal to the first risk threshold but lower than a preset second risk threshold, generating a process warning instruction and a front-end fine-tuning instruction; if the dynamic failure risk quantification index is greater than or equal to the second risk threshold, combining traceability identification information to generate a fatal defect interception instruction as a feedback control instruction, and generating rework location indication data for the corresponding high-definition circuit board target object; wherein, the first risk threshold is less than the second risk threshold.

[0171] This embodiment provides a visual inspection system for SMT placement defects. Specifically, the system can be deployed in the end inspection unit of the aforementioned vehicle controller circuit board production line and interconnected with the conveying mechanism, manufacturing execution system, front-end placement equipment, and rework station. Compared to a single automatic optical inspection device that only provides a result of whether the appearance is qualified or not, the system in this embodiment integrates image acquisition, shape extraction, physical mapping, risk quantification, and decision feedback in a modular form.

[0172] Specifically, the data acquisition module is responsible for acquiring traceability identification information and multimodal visual image data of the circuit board to be inspected; this module may include a barcode reader, a 2D industrial camera, a 3D structured light camera, lighting components, and board position sensors; the feature extraction module is responsible for registering, segmenting, and geometrically analyzing the acquired 2D color images and 3D depth images, and outputting 3D morphological topological feature data suitable for expressing the solder joint forming state; the model mapping module calls a pre-trained cross-modal mapping proxy model to convert the above morphological features into physical and mechanical state prediction data; the risk quantification module combines the product application environment template to obtain a dynamic failure risk quantification index; and the decision feedback module generates different control results such as safe release, process warning, front-end fine-tuning, critical defect interception, and rework location indication based on the risk range;

[0173] From a system collaboration perspective, the modules are not isolated from each other; the global coordinate information output by the data acquisition module will run through subsequent feature extraction, rework location and production line execution; the model mapping module outputs not only abstract scores, but also provides the risk quantification module with an intermediate state that can be coupled with environmental stress; the decision feedback module sends the medium-risk and high-risk results back to the front-end equipment or rework station respectively, forming a closed loop; in this way, the whole system can make quality release judgments at the single board level, and also promote process trend optimization at the batch level.

[0174] As a fault-tolerance mechanism, if one module fails during system operation, other modules can operate in a downgraded manner according to a conservative strategy. For example, when the 3D camera is temporarily offline, the system can retain the 2D appearance inspection and traceability recording functions, but should stop outputting safety release conclusions based on physical risks and instead prompt for re-inspection. When the environment template service is unavailable, the risk quantification module can enable local cached templates. When network communication anomalies prevent front-end fine-tuning instructions from being delivered to upstream equipment in a timely manner, the decision feedback module can write the corresponding suggestions to the local log and prompt manual confirmation on the operator interface. Through this basic fault-tolerance mechanism, a single point of failure can prevent the entire production line from going out of control.

[0175] For example, during the mass production of an onboard controller, a circuit board is read by the data acquisition module and multimodal imaging is completed. The feature extraction module extracts asymmetrical three-dimensional morphological features from its inductor solder joints. The model mapping module predicts that there is significant stress concentration at this location. The risk quantification module, combined with an automotive-grade vibration template, determines it to be high-risk. The decision feedback module then outputs a red interception and sends a rework location instruction to the rework station. At the same time, for another chip resistor solder joint with only slight displacement, the system outputs a yellow process warning and sends a fine-tuning suggestion to the pick-and-place machine. This demonstrates the system's ability to differentiate between objects with different risk levels on the same production line.

[0176] The purpose of this system is to improve the integration and control stability of the SMT inspection system by realizing the entire chain operation from visual perception to reliability decision-making and process closed loop through a modular collaborative architecture.

[0177] The feature extraction module is specifically used for:

[0178] Edge detection is performed on two-dimensional color image data to extract the two-dimensional contour features of the target object on the high-definition circuit board;

[0179] Spatial coordinate registration is performed between two-dimensional color image data and three-dimensional depth image data;

[0180] Based on the registered two-dimensional contour features, the three-dimensional depth image data is mapped and segmented to generate three-dimensional point cloud data of the target region.

[0181] Geometric topology calculations are performed on the 3D point cloud data of the target area to generate 3D morphological topology feature data.

[0182] This embodiment provides a feature extraction module implementation mechanism for performing three-dimensional morphological topology extraction in a system. Specifically, in the whole system, if the feature extraction module only outputs simple defect category labels, such as insufficient tin, offset, and bridging, the subsequent model mapping module will find it difficult to perform physical deduction based on sufficiently rich structural information. Therefore, this embodiment designs the module as an intermediate layer for geometric topology reconstruction, so that it outputs structural features with continuous spatial meaning.

[0183] Specifically, the module first performs edge detection on the two-dimensional color image data to extract two-dimensional contour features; based on the device calibration results, it performs spatial coordinate registration between the two-dimensional image and the three-dimensional depth image; after registration, the module performs region mapping and segmentation on the three-dimensional depth image based on the two-dimensional contour to obtain three-dimensional point cloud data of the target region; then, it performs geometric topology calculations on the three-dimensional point cloud to obtain features such as wetting angle, meniscus curvature, volume distribution, and contact cross-sectional area; this module can be composed of an image processing unit, a point cloud processing unit, and a feature encoding unit, or it can be implemented sequentially in software on the same computing platform;

[0184] For ease of understanding, a simplified example can be provided. Suppose that a closed contour region R1 is extracted from a solder joint of a certain device in a two-dimensional image, and after registration, it corresponds to a point cloud region P1 in the depth map. The image processing unit first determines the boundary range of the solder joint from R1, and the point cloud processing unit removes abnormal high points and isolated noise points from P1, and then reconstructs the surface of the solder joint based on the boundary. The feature encoding unit extracts geometric information from the reconstructed surface, such as a smooth transition on the left, a steep drop on the right, a center height greater than the standard height threshold, and a bottom contact band less than the preset contact width lower limit, and encodes it into a set of structured features, which are then output to subsequent modules.

[0185] As a fault-tolerance mechanism, if the edge detection results of the 2D image are greatly affected by changes in illumination, the module can automatically switch illumination parameters or call the fusion results of adjacent frames according to the circuit board type; if the 3D point cloud is severely missing in some areas, the module can perform interpolation reconstruction on the missing areas, but when the missing area exceeds a preset ratio, the feature result should be marked as low confidence to prevent subsequent modules from over-relying on incomplete data; if there are solder joints and component pin surfaces in the same area that are very close in height and difficult to separate, the module can combine the component package template to perform constraint segmentation to improve the accuracy of point cloud segmentation.

[0186] For example, on the vehicle controller production line, the feature extraction module first identifies the pad boundary and solder outer edge from a two-dimensional image for a corner solder joint of a square flat leadless package device, and then maps it to the corresponding area in a three-dimensional depth map. Finally, it is found that although the area of ​​the solder joint is normal, the contact cross-sectional area on the side closer to the device body is smaller and the curvature of the outer meniscus changes faster. The above results are encoded and sent to the model mapping module for subsequent mechanical risk assessment.

[0187] The purpose of this module is to provide stable, detailed, and spatially structurally significant input features for subsequent physical state prediction, thereby enabling high-quality data reception in the system-level detection chain.

[0188] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A visual inspection method for SMT (Surface Mount Technology) defects, characterized in that, include: The system acquires traceability identification information of the target object on the high-definition circuit board to be inspected, as well as multimodal visual image data of the target object. The target object on the high-definition circuit board includes SMT surface mount components and solder joints. The multimodal visual image data is acquired based on a preset device global coordinate system and includes two-dimensional color image data and three-dimensional depth image data. Feature extraction is performed on multimodal visual image data to generate three-dimensional morphological topological feature data; The three-dimensional morphological topological feature data is input into a preset cross-modal mapping proxy model, and the physical and mechanical state prediction data is output; the physical and mechanical state prediction data includes the stress concentration factor feature matrix and the weak point distribution features. By combining preset environmental stress parameters and physical and mechanical state prediction data, a dynamic failure risk quantification index is calculated. Determine the preset range in which the dynamic failure risk quantification index falls, and generate corresponding feedback control instructions, specifically including: If the dynamic failure risk quantification index is lower than the preset first risk threshold, a safety release instruction is generated as a feedback control instruction. If the dynamic failure risk quantification index is greater than or equal to the first risk threshold and lower than the preset second risk threshold, a process warning instruction and a front-end fine-tuning instruction are generated as feedback control instructions. If the dynamic failure risk quantification index is greater than or equal to the second risk threshold, a fatal defect interception instruction is generated as a feedback control instruction in combination with the traceability identification information, and the corresponding repair location indication data of the target object of the high-definition circuit board is generated; wherein, the first risk threshold is less than the second risk threshold.

2. The visual inspection method for SMT (Surface Mount Technology) defects according to claim 1, characterized in that, Feature extraction is performed on multimodal visual image data to generate three-dimensional morphological topological feature data, including: Edge detection is performed on two-dimensional color image data to extract the two-dimensional contour features of the target object on the high-definition circuit board; Spatial coordinate registration is performed between two-dimensional color image data and three-dimensional depth image data; Based on the registered two-dimensional contour features, the three-dimensional depth image data is mapped and segmented to generate three-dimensional point cloud data of the target region. Geometric topology calculations are performed on the 3D point cloud data of the target area to generate 3D morphological topology feature data; among which, the 3D morphological topology feature data includes wetting angle features, meniscus curvature features, volume distribution features, and contact cross-sectional area features.

3. The visual inspection method for SMT (Surface Mount Technology) defects according to claim 1, characterized in that, The cross-modal mapping proxy model is trained through the following steps: Obtain historical three-dimensional topological feature data samples; Obtain preset SMT solder material property parameters; By combining the material property parameters of SMT solder, a finite element analysis of the consolidation and thermo-mechanical coupling of historical three-dimensional morphological topological feature data samples is performed to generate corresponding historical physical and mechanical state prediction data labels. Using historical 3D morphological topological feature data samples and historical physical and mechanical state prediction data labels, a pre-set initial deep learning network is trained under supervision until the loss function of the initial deep learning network converges, thus obtaining a cross-modal mapping surrogate model.

4. The visual inspection method for SMT (Surface Mount Technology) defects according to claim 1, characterized in that, By combining preset environmental stress parameters and physical and mechanical state prediction data, a dynamic failure risk quantification index is calculated, including: The environmental stress parameters are input into a preset stress transfer function to calculate the environmental stress load matrix; the environmental stress parameters include at least temperature cycle load parameters and mechanical vibration load parameters. The environmental stress load matrix and the stress concentration factor feature matrix in the physical and mechanical state prediction data are multiplied by matrix to obtain the local stress tensor distribution matrix, and the local maximum stress value in the local stress tensor distribution matrix is ​​extracted based on the preset equivalent stress criterion. Based on the preset material fatigue life curve, the probability of foundation failure corresponding to the local maximum stress value is determined. By utilizing the distribution characteristics of weak points in the physical and mechanical state prediction data, the probability of basic failure is weighted and adjusted to generate a dynamic failure risk quantification index.

5. The visual inspection method for SMT chip defects according to claim 1, characterized in that, If the dynamic failure risk quantification index is greater than or equal to the first risk threshold and lower than the preset second risk threshold, a process warning instruction and a front-end fine-tuning instruction are generated as feedback control instructions, including: Obtain the preset standard shape range corresponding to the SMT standard process; Extract the deviation feature vector from the preset standard morphological range in the three-dimensional morphological topological feature data; Input the deviation feature vector into the preset process parameter mapping matrix to calculate the process compensation amount; The front-end fine-tuning instructions are generated based on the process compensation amount; these instructions are used to adjust the operating parameters of the front-end production equipment upstream of the SMT production line.

6. The visual inspection method for SMT (Surface Mount Technology) defects according to claim 1, characterized in that, If the dynamic failure risk quantification index is greater than or equal to the second risk threshold, a fatal defect interception instruction is generated as a feedback control instruction based on the traceability identification information, and corresponding repair location indication data for the high-definition circuit board target object is generated, including: Extract the coordinates of the most probable weak point from the weak point distribution features; Map the coordinates of the most probable weak point to the device's global coordinate system to generate global rework coordinates; Based on global rework coordinates, critical defect interception instructions, and traceability identification information, rework location indication data for the corresponding high-definition circuit board target object is encapsulated and generated.

7. A visual inspection system for SMT component defects, used to implement the visual inspection method for SMT component defects as described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to acquire traceability identification information of the target object of the high-definition circuit board to be inspected, as well as multimodal visual image data of the target object of the high-definition circuit board. The target object of the high-definition circuit board includes SMT surface mount components and solder joints. Among them, the multimodal visual image data is acquired based on the preset device global coordinate system and includes two-dimensional color image data and three-dimensional depth image data. The feature extraction module is used to extract features from multimodal visual image data and generate three-dimensional morphological topology feature data. The model mapping module is used to input three-dimensional morphological topological feature data into a preset cross-modal mapping proxy model and output physical and mechanical state prediction data; wherein, the physical and mechanical state prediction data includes stress concentration coefficient feature matrix and weak point distribution features; The risk quantification module is used to calculate the dynamic failure risk quantification index by combining preset environmental stress parameters and physical and mechanical state prediction data. The decision feedback module is used to determine the preset range in which the dynamic failure risk quantification index falls and generate corresponding feedback control instructions. Specifically, it includes: if the dynamic failure risk quantification index is lower than a preset first risk threshold, generating a safety release instruction; if the dynamic failure risk quantification index is greater than or equal to the first risk threshold but lower than a preset second risk threshold, generating a process warning instruction and a front-end fine-tuning instruction; if the dynamic failure risk quantification index is greater than or equal to the second risk threshold, combining traceability identification information to generate a fatal defect interception instruction as a feedback control instruction, and generating rework location indication data for the corresponding high-definition circuit board target object; wherein, the first risk threshold is less than the second risk threshold.

8. The visual inspection system for SMT (Surface Mount Technology) defects according to claim 7, characterized in that, The feature extraction module is specifically used for: Edge detection is performed on two-dimensional color image data to extract the two-dimensional contour features of the target object on the high-definition circuit board; Spatial coordinate registration is performed between two-dimensional color image data and three-dimensional depth image data; Based on the registered two-dimensional contour features, the three-dimensional depth image data is mapped and segmented to generate three-dimensional point cloud data of the target region. Geometric topology calculations are performed on the 3D point cloud data of the target area to generate 3D morphological topology feature data.

Images