Intelligent detection method and system for controlling mainboard interlayer welding points

Abstract

The application provides an intelligent detection method and system for controlling mainboard interlayer welding points, belonging to the technical field of electronic manufacturing and detection, comprising: loading welding point detection data, calculating the local shielding coefficient of each welding point; inputting the above data into a lightweight deep neural network to obtain welding point defect classification results and their confidence; outputting the welding point defect classification results when the confidence is greater than or equal to a first confidence threshold; uploading to a cloud server for secondary reasoning when the confidence is less than the first confidence threshold and greater than or equal to a second confidence threshold; triggering an alarm when the confidence is less than the second confidence threshold; updating the local mis-detection sample cache area; and performing end-side incremental learning when a preset number threshold is reached and the production line is in an idle period. The application solves the composite problems of unreliable end-side reasoning confidence caused by peripheral element shielding and the difficulty of self-adaptive updating of defect classification models due to performance degradation in production line operation in the detection of mainboard interlayer welding points.

Description

Technical Field

[0001] This invention relates to the field of electronic manufacturing and testing technology, specifically to an intelligent testing method and system for controlling solder joints in the motherboard interlayer. Background Technology

[0002] In motherboard manufacturing, critical solder joints such as USB connectors are located in the sandwich area between the core board and the motherboard. Traditional automated optical inspection cannot directly observe the solder joint morphology due to the obstruction of the double-layer board. Existing solutions, such as CN111862067A, use multi-task learning networks for general solder joint defect detection. However, this method is designed based on complete imaging conditions and does not consider the image feature degradation caused by obstruction. Furthermore, the model is deployed in a high-performance computing environment, which is difficult to meet the real-time inference requirements at the edge of the production line. CN113516706A uses residual networks for solder joint detection, but it is geared towards the scenario of solder joint location on the surface of the third track and does not involve the obstruction problem of the sandwich structure. To address the shortcomings of edge detection of sandwich solder joints, such as the decrease in confidence due to component obstruction, poor adaptability of fixed thresholds, and the inability of the model to adapt to process fluctuations, there is an urgent need for a lightweight edge inference method that can sense the degree of obstruction and dynamically adjust the detection strategy. Summary of the Invention

[0003] This invention provides an intelligent detection method and system for controlling motherboard interlayer solder joints, which solves the complex problems in controlling motherboard interlayer solder joint detection, such as unreliable end-side inference confidence due to obstruction by surrounding components, high latency of full upload to the cloud, and difficulty in adaptively updating the defect classification model due to performance degradation during production line operation.

[0004] In view of the above problems, the present invention provides an intelligent detection method and system for controlling the solder joints in the motherboard interlayer.

[0005] In a first aspect, the present invention provides an intelligent detection method for controlling solder joints in the motherboard interlayer, comprising:

[0006] Load the solder joint detection data of the core board to be tested, and calculate the local occlusion coefficient of each solder joint based on the solder joint position coordinates and the preset surrounding component height model;

[0007] The solder joint detection data and the local occlusion coefficient are input into a lightweight deep neural network on the edge computing unit to obtain the solder joint defect classification results and their confidence levels.

[0008] When the confidence level is greater than or equal to the first confidence threshold, the solder joint defect classification result is output, and the first confidence threshold is negatively correlated with the local occlusion coefficient.

[0009] When the confidence level is less than the first confidence threshold and greater than or equal to the second confidence threshold, the solder joint detection data is uploaded to the cloud server for secondary inference to obtain the review and classification results.

[0010] An alarm is triggered when the confidence level is less than the second confidence threshold.

[0011] The local false detection sample cache is updated based on the discrepancy between the review classification results and the solder joint defect classification results. The update weight of the sample is positively correlated with the local occlusion coefficient.

[0012] When the number of samples in the false detection sample buffer reaches a preset threshold and the production line is in an idle period, edge incremental learning is performed.

[0013] Secondly, the present invention provides an intelligent detection system for controlling solder joints in the motherboard interlayer, comprising:

[0014] The detection data loading module is used to load the solder joint detection data of the core board to be tested, and calculate the local occlusion coefficient of each solder joint based on the solder joint position coordinates and the preset surrounding component height model.

[0015] The local occlusion coefficient calculation module is used to input the solder joint detection data and the local occlusion coefficient into a lightweight deep neural network on the edge computing unit to obtain the solder joint defect classification result and its confidence level.

[0016] A lightweight inference module on the end side is used to output the solder joint defect classification result when the confidence level is greater than or equal to a first confidence threshold, wherein the first confidence threshold is negatively correlated with the local occlusion coefficient;

[0017] The high-confidence direct output module is used to upload the solder joint detection data to the cloud server for secondary inference and obtain the verification and classification results when the confidence level is less than the first confidence threshold and greater than or equal to the second confidence threshold.

[0018] The medium confidence level cloud verification module is used to trigger an alarm when the confidence level is less than the second confidence level threshold;

[0019] The low confidence alarm triggering module is used to update the local false detection sample cache based on samples where the review classification results are inconsistent with the weld joint defect classification results. The update weight of the sample is positively correlated with the local occlusion coefficient.

[0020] The false detection sample caching and incremental learning module is used to perform edge-side incremental learning when the number of samples in the false detection sample cache reaches a preset threshold and the production line is in an idle period.

[0021] One or more technical solutions provided in this invention have at least the following technical effects or advantages:

[0022] The technical solution of this invention first loads multi-angle automatic optical inspection images and structured light three-dimensional topographic data. Combining the solder joint position coordinates with a preset surrounding component height model, it calculates the occlusion intensity at each viewing angle by retrieving the package height, reflection characteristics, and pin offset of the surrounding components, and then weights and normalizes the results to generate a local occlusion coefficient for each solder joint. This provides a priori occlusion perception index reflecting the quality of solder joint data acquisition for subsequent reasoning and decision-making.

[0023] Furthermore, multi-modal feature fusion is performed on multi-angle grayscale images and pseudo-color depth maps in the shallow layers of a lightweight deep neural network. In the intermediate layers, local occlusion coefficients are expanded into additional channels and stitched into the feature map. A compact network constructed through depthwise separable convolution and channel pruning outputs the weld joint defect category and confidence level. Joint perception of texture, shape, and occlusion degree is achieved at the edge, enabling high-precision defect classification with extremely low computational overhead.

[0024] Furthermore, based on the comparison between the confidence level and the dynamic first confidence threshold and the fixed second confidence threshold, the solder joints are divided into three processing paths: high confidence level results in direct output, medium confidence level results in uploading to the cloud for high-precision model verification, and low confidence level results in triggering an alarm. This maximizes the proportion of direct decision-making at the edge while ensuring detection accuracy, reduces cloud communication latency and computing power consumption, and ensures that low-quality data is not incorrectly accepted.

[0025] Finally, false positives that do not match the cloud-based verification results with the edge-side inference results are repeatedly stored in the first-in-first-out (FIFO) cache with update weights positively correlated with the occlusion coefficient. Medium-confidence samples are supplemented with baseline weights. When the number of samples in the cache reaches a preset threshold and the production line is idle, edge-side incremental learning is automatically triggered. The training and update are completed within a preset time window by mixing new and old samples in a proportional manner and using elastic weights to consolidate regularization constraints. This achieves continuous adaptive evolution of the model during production line operation, focusing on strengthening the defect detection capability in occlusion scenarios while preventing catastrophic forgetting of old knowledge.

[0026] In summary, the technical solution of this invention constructs a complete closed-loop detection system, which possesses full-link perception and adaptive response capabilities to the occlusion effects of surrounding components in complex assembly environments; it achieves optimal configuration of computing power and accuracy between the lightweight end-side model and the high-precision cloud-based model; and it can continuously absorb false detection samples and self-optimize during continuous production line operation, ultimately achieving a unified high-precision, low-latency, robust, and self-evolving approach to detecting defects in motherboard interlayer solder joints. This solves the complex problems in motherboard interlayer solder joint detection, such as unreliable end-side inference confidence due to occlusion by surrounding components, high latency of full upload to the cloud, and difficulty in adaptively updating the defect classification model due to performance degradation during production line operation. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating an intelligent detection method for controlling solder joints in the motherboard interlayer provided by the present invention.

[0028] Figure 2 This is a logical diagram illustrating the confidence judgment process in an intelligent detection method for controlling motherboard interlayer solder joints provided by the present invention.

[0029] Figure 3 This is a schematic diagram of the structure of an intelligent detection system for controlling the solder joints in the motherboard interlayer provided by the present invention.

[0030] In the attached diagram, the labels representing each component are as follows:

[0031] The system includes a detection data loading module 11, a local occlusion coefficient calculation module 12, an edge-side lightweight inference module 13, a high-confidence direct output module 14, a medium-confidence cloud-based verification module 15, a low-confidence alarm triggering module 16, and a false detection sample caching and incremental learning module 17. Detailed Implementation

[0032] This invention provides an intelligent detection method and system for controlling motherboard interlayer solder joints, which solves the complex problems in controlling motherboard interlayer solder joint detection, such as unreliable end-side inference confidence due to obstruction by surrounding components, high latency of full upload to the cloud, and difficulty in adaptively updating the defect classification model due to performance degradation during production line operation.

[0033] It should be noted that the terms "comprising" and "having" are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0034] Example 1, as Figure 1 As shown, the present invention provides an intelligent detection method for controlling solder joints in the motherboard interlayer, the method comprising:

[0035] S100: Load the solder joint detection data of the core board to be tested, and calculate the local occlusion coefficient of each solder joint based on the solder joint position coordinates and the preset surrounding component height model.

[0036] This step is used to load the detection data and initialize the calculation of the local occlusion coefficient. By reading the solder joint detection data of the core board to be inspected, and combining the position coordinates of each solder joint with the pre-built height model of surrounding components, the degree of occlusion of surrounding tall components, such as capacitors, inductors, and connectors, on the visual inspection path of the solder joints is quantitatively evaluated.

[0037] In this step, the weld joint detection data includes multi-angle automatic optical detection images and structured light three-dimensional topography data;

[0038] The loading of solder joint inspection data from the core board to be inspected includes:

[0039] Multiple cameras at different angles are controlled to acquire grayscale images of the solder joint area. The grayscale images are preprocessed to form a multi-channel input tensor, which is used as the multi-angle automatic optical detection image.

[0040] The structured light projector projects an coded stripe pattern, and the three-dimensional point cloud of the weld joint area is reconstructed by the phase-shifting method. The height information is then mapped into a pseudo-color depth map as the structured light three-dimensional topography data.

[0041] In this step, multiple cameras positioned around the solder joint area are first controlled to simultaneously acquire grayscale images of the solder joint area from different fixed angles. For the fixed pin solder joints of the USB connector, the cameras need to focus on covering the meniscus in front of the solder joint and the filling area behind it.

[0042] For example, taking the right fixed pin solder joint of the USB Type-C connector on a tablet PC main control board as the target, four CMOS cameras are arranged around it at elevation angles of 30°, 45°, 60°, and 75° respectively, and are simultaneously triggered to acquire four frames of grayscale images with a resolution of 1024×1024, with the exposure time uniformly set to 10ms.

[0043] Secondly, Gaussian filtering for noise reduction and contrast adaptive histogram equalization enhancement are performed sequentially on each grayscale image frame. Then, the preprocessed images of multiple frames are stacked along the channel dimension. The stacking method can be the PyTorch framework, and a multi-channel input tensor is output.

[0044] For example, each of the four pre-processed single-channel images is a 1024×1024 matrix. After being stacked along the channel dimension, they form a 1024×1024×4 tensor. The second channel, corresponding to a 60° viewing angle, can clearly display the texture details of the meniscus at the front of the USB socket solder joint.

[0045] Next, control the structured light projector to project a set of pre-designed coded stripe patterns onto the solder joint area. The stripes need to cover the entire pad and pin area of ​​the USB socket solder joint.

[0046] Furthermore, after acquiring four frames of fringe modulation images, the wrapping phase is calculated pixel-by-pixel using a phase-shifting method. Then, a quality-map-guided path unwrapping algorithm is used to restore the continuous phase distribution and eliminate phase jumps. Using pre-calibrated system parameters between the projector and camera, the continuous phase is converted pixel-by-pixel into spatial three-dimensional coordinates to generate a three-dimensional point cloud of the solder joint area. The aforementioned phase-shifting calculation, phase unwrapping, and triangulation calibration conversion are all mature technologies in the field of optical three-dimensional measurement and will not be elaborated upon here.

[0047] For example, following the previous scenario, the structured light projector uses a DMD digital projector to project four pre-generated sinusoidal stripe patterns with phase offsets of 0°, 90°, 180°, and 270° onto the solder joint area at an external trigger frequency of 10Hz. After the camera simultaneously acquires four frames of reflected images, a three-dimensional point cloud containing approximately 500,000 spatial points is generated through a mature optical three-dimensional reconstruction process.

[0048] Finally, the height information in the 3D point cloud is mapped to a preset color table to generate a pseudo-color depth map as 3D topographic data for structured light. The topographic differences of the USB connector pin solder joints are presented intuitively through color.

[0049] For example, the height range of 0-250μm is linearly mapped to a blue-cyan-green-yellow-red gradient color band. The normal area of ​​the USB connector pin solder joint with a height of 220μm is displayed in red, the recessed area with a height of about 80μm is displayed in cyan, and it can be clearly distinguished that the solder joint vertex with a height of about 218μm is displayed in red, the pad base with a height of about 12μm is displayed in blue, and a small recess with a height of about 85μm is displayed in cyan. Finally, a 1024×1024 RGB pseudo-color depth map is output as the structured light three-dimensional topography data.

[0050] In this step, the preset peripheral component height model is a pre-stored core board component layout database. The core board component layout database includes the package height, pin offset, and metal casing reflection characteristic parameters of adjacent components within a preset radius around each solder point.

[0051] Specifically, the pre-built core board component layout database refers to an associated mapping table extracted and structured in advance from the motherboard's EDA design files, such as PCB layout, BOM, and component datasheets, during the deployment phase of the testing system. Using solder joint coordinates as the primary key, it pre-collects the physical attributes of neighboring components around each solder joint, enabling rapid retrieval and calculation of the degree of local occlusion and optical interference of components during real-time testing, without the need for online measurement.

[0052] For example, using the coordinates (32.5, 18.2) of the fixed pin solder joint on the USB Type-C connector of a tablet PC main control board as the primary key, and with a preset search radius of 4.5mm in the database, the search results show two related components around the solder joint: an inductor with a horizontal distance of 3.2mm, a package height of 4.5mm, a pin offset of 0.3mm, and a metal shell reflectivity of 0.72; and a pin header connector with a horizontal distance of 2.8mm, a package height of 6.8mm, a pin offset of 0.1mm, and a metal shell reflectivity of 0.85. Based on this, the maximum occlusion ratio of the USB connector solder joint is calculated to be 0.45, and it is determined that the high reflectivity of the pin header's metal shell may cause local overexposure interference of structured light stripes at the edge of the solder joint.

[0053] In this step, the local occlusion coefficient is calculated as follows: based on the solder joint position coordinates, the surrounding components are traversed, the occlusion angle and occlusion intensity of each component are calculated, and the weighted sum is normalized to a predetermined value range.

[0054] Specifically, firstly, taking the current position coordinates of the solder joint to be tested as the center, the core board component layout database is searched according to a preset radius to extract the package height, pin offset and metal shell reflection characteristics parameters of all adjacent components within the range.

[0055] For example, using the coordinates (32.5, 18.2) of the fixed pin solder joint of the USB Type-C socket as the center, a search is performed in the database with a radius of 4.5 mm, and two adjacent components are extracted: an inductor with a horizontal distance of 3.2 mm, a package height of 4.5 mm, a pin offset of 0.3 mm, and a reflectivity of 0.72; and a pin header connector with a horizontal distance of 2.8 mm, a package height of 6.8 mm, a pin offset of 0.1 mm, and a reflectivity of 0.85.

[0056] Furthermore, based on the package height of each surrounding component, the horizontal distance between the component and the solder joint, and the current camera optical axis elevation angle, the occlusion angle of the component in each camera viewing direction is calculated. The occlusion angle is determined as follows: when the elevation angle of the line connecting the top edge of the component and the solder joint surface is greater than the camera optical axis elevation angle, the component occludes that viewing angle. The elevation angle of the line connecting the top edge of the component and the solder joint surface = arctan(package height / horizontal distance), and the occlusion angle = elevation angle of the line connecting the top edge of the component and the solder joint surface - camera optical axis elevation angle. If the result is negative, the occlusion angle is counted as zero.

[0057] For example, for a pin header connector, the package height is 6.8mm and the horizontal distance is 2.8mm. In the direction of a 30° elevation camera, the elevation angle of the line connecting the top edge of the component and the solder joint surface is arctan(6.8 / 2.8)≈67.6°>30°, which constitutes obstruction. The obstruction angle is 67.6°-30°=37.6°. In the direction of a 75° elevation camera, the obstruction angle of the line connecting the top edge of the component and the solder joint surface is 67.6°<75°, and the obstruction angle is 0°, meaning the viewing angle is not obstructed.

[0058] Furthermore, the blocking intensity is jointly determined by the blocking angle, component surface reflectivity, and component pin offset. Blocking intensity = blocking angle × component surface reflectivity × (1 + component pin offset / horizontal distance). The higher the component surface reflectivity, the stronger the stray light generated by structured light or imaging light on the component surface, and the more severe the interference with solder joint detection. The larger the component pin offset, the more the actual spatial envelope occupied by the component exceeds its nominal position, thus expanding the blocking range. Both component surface reflectivity and component pin offset are derived from the core board component layout database.

[0059] For example, the pin header connector has a blocking angle of 37.6° at a 30° viewing angle. From the core board component layout database, the component surface reflectivity is 0.85, the component pin offset is 0.1mm, the horizontal distance is 2.8mm, and the blocking strength is 37.6 × 0.85 × (1 + 0.1 / 2.8) ≈ 33.1. The inductor has a blocking angle of arctan(4.5 / 3.2) - 30° ≈ 54.6° - 30° = 24.6° at a 30° viewing angle. The component surface reflectivity is 0.72, the component pin offset is 0.3mm, the horizontal distance is 3.2mm, and the blocking strength is 24.6 × 0.72 × (1 + 0.3 / 3.2) ≈ 19.4.

[0060] Furthermore, by iterating through all surrounding components, the occlusion intensity of each component is accumulated for each camera's viewing angle to obtain the total occlusion intensity for that viewing angle. The weight of each camera's viewing angle is set based on its contribution to the detection of critical areas of the solder joint. Specifically, during the calibration phase, a three-dimensional standard topography model is established for each type of solder joint package, and the spatial range of the front meniscus and the rear filler area is marked. The proportion of the visible projected area of ​​the two areas to the total area of ​​the region is calculated for each viewing angle. Then, according to the importance of the meniscus and the rear filler area for defect detection (e.g., focusing on the meniscus for cold solder joints and the rear filler area for porosity), each is assigned a value of 0.5 or adjusted as needed, and then weighted and summed to obtain the comprehensive detection contribution of each viewing angle. Finally, the contribution of each viewing angle is normalized so that the sum of the weights is 1, which is used as the weight coefficient for multi-view weighted calculation.

[0061] For example, for the fixed pin solder joints of the USB Type-C connector, a three-dimensional standard topographic model is established during the calibration phase, with a total meniscus area of ​​1.2 mm². 2 The total area of ​​the heel padding area is 0.8mm. 2 The importance of the crescent is set to 0.6, and the importance of the heel is set to 0.4. At a 30° viewpoint, the crescent is 29% visible, and the heel is 81% visible, with a contribution of 0.29 × 0.6 + 0.81 × 0.4 = 0.498. Similarly, the contribution of the 45° viewpoint is 0.612; the 60° viewpoint is 0.738; and the 75° viewpoint is 0.824. After normalization, the weights of the four viewpoints are 0.186, 0.229, 0.276, and 0.309, respectively, with the 75° viewpoint receiving the highest weight due to the complete visibility of the crescent.

[0062] For example, at a 30° viewing angle, the blocking strength of the pin header connector is 33.1, and the blocking strength of the inductor is 19.4, so the total blocking strength is 33.1 + 19.4 = 52.5. The total blocking strength at a 45° viewing angle is 28.7; at a 60° viewing angle, it is 12.3; and at a 75° viewing angle, it is 5.8. Multiplying the total blocking strength of each viewing angle by the corresponding normalized weighting coefficient and summing them, we get the weighted total blocking strength = 52.5 × 0.186 + 28.7 × 0.229 + 12.3 × 0.276 + 5.8 × 0.309 = 21.52.

[0063] Finally, the weighted summation result is mapped to the [0,1] interval using a preset normalization function to obtain the final local occlusion coefficient. An upper limit for the occlusion intensity threshold is set; when the weighted value is greater than or equal to the occlusion intensity threshold, the local occlusion coefficient is directly set to 1. The preset normalization function preferably uses a piecewise linear normalization function, i.e., local occlusion coefficient = weighted total occlusion intensity / occlusion intensity threshold.

[0064] The threshold for occlusion intensity is set based on the occlusion simulation experiment determined during the calibration phase: for each solder joint packaging type, occlusion is gradually increased to change the occlusion intensity, and the detection accuracy of the defect classification model under different occlusion intensities is recorded. The occlusion intensity value corresponding to the accuracy dropping to an acceptable lower limit, such as 95%, is taken as the upper limit of the threshold for that solder joint type.

[0065] For example, when calibrating the solder joints of a USB Type-C connector, the model accuracy is stable at over 97% when the occlusion strength is in the range of 0 to 60. When the occlusion strength reaches 65, the accuracy drops to 94.7%, which is lower than the acceptable lower limit of 95%. Therefore, the upper limit of the occlusion strength threshold for this solder joint type is set at 60.

[0066] For example, if the upper limit of the occlusion intensity threshold is set to 60, the above weighted total occlusion intensity is 21.52, and the normalized local occlusion coefficient is approximately 21.52 / 60.

[0067] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0068] In summary, this step assigns a normalized index to each solder joint that reflects the degree of shading caused by surrounding tall components. This enables the subsequent end-side inference model to perceive the reliability differences in the input data, thereby allowing for targeted adjustments to the confidence evaluation criteria and decision-making strategies. This effectively mitigates the problem of misjudgment or missed detection of defects caused by local shading.

[0069] S200: Input the solder joint detection data and the local occlusion coefficient into a lightweight deep neural network on the edge computing unit to obtain the solder joint defect classification result and its confidence level.

[0070] This step incorporates the occlusion coefficient as an auxiliary input feature into the edge-side inference process, enabling the network to perceive the degree of occlusion of the current weld point data when predicting the defect category. This allows for adaptive suppression of blurred or missing features caused by severe occlusion, resulting in a confidence score that better reflects the actual data quality and providing an accurate basis for subsequent hierarchical decision-making based on confidence.

[0071] Step S200 in the method provided by the present invention includes:

[0072] The multi-channel input tensor and the pseudo-color depth map are fused together in the shallow layer of the lightweight deep neural network to obtain a fused feature map.

[0073] The local occlusion coefficient is used as an additional channel and concatenated with the fused feature map in the middle layer of the network before being input into subsequent convolutional blocks.

[0074] The lightweight deep neural network is constructed based on depthwise separable convolution and channel pruning techniques. The output has a dual-task branch: the first branch is used for solder joint semantic segmentation, and the second branch is used for defect classification.

[0075] In this step, the construction of the lightweight deep neural network also includes:

[0076] Collect historical solder joint detection datasets and train an initial version of the lightweight deep neural network on an offline server;

[0077] The initial version is converted into a runtime file executable by the edge computing unit and deployed to the edge computing unit;

[0078] The edge computing unit is deployed inside a multi-angle automated optical inspection device and connects to multiple cameras and structured light projectors at different angles through a standardized interface.

[0079] Specifically, firstly, historical solder joint inspection data from multiple production lines, batches, and core board models are aggregated on an offline server to construct a historical solder joint inspection dataset. Each sample in the dataset includes a multi-angle grayscale image, a pseudo-color depth map, and the corresponding local occlusion coefficient. The annotation information covers solder joint defect category labels and semantic segmentation masks. Annotation is completed collaboratively by manual and semi-automatic auxiliary tools to ensure label quality.

[0080] For example, from the test records of tablet PC main control boards produced in the past 6 months, 50,000 solder joint samples were extracted, including 12 package types such as USB Type-C sockets, BGA solder balls, and QFN pin solder joints. Among them, 28,000 were normal solder joints, 8,000 had excessive porosity, 6,500 had poor solder joints, 5,000 had insufficient solder, and 2,500 had cracks. Each sample was accompanied by a manually verified defect category and segmentation mask label.

[0081] Furthermore, on an offline server, the aforementioned historical solder joint detection dataset was used to conduct supervised training on a lightweight deep neural network. The lightweight deep neural network consists of an input preprocessing layer, a shallow multimodal fusion module, an intermediate feature extraction backbone, and an output dual-task branch.

[0082] Specifically, the multi-channel input tensor and the pseudo-color depth map are first fused in the shallow layers of a lightweight deep neural network. The two are then fused after the first or second convolutional block of the network by element-wise addition or channel concatenation, outputting a fused feature map. This allows the network to simultaneously acquire multi-angle texture information and 3D topographic height information at an early stage.

[0083] For example, the multi-channel input tensor shape of the USB socket solder joint is [4,1024,1024], and the pseudo-color depth map shape is [3,1024,1024]. After each of them is processed by a 3×3 convolutional layer to extract shallow features, they are concatenated in the channel dimension to obtain a fused feature map with a shape of [16,256,256]. The texture feature channel and the height feature channel are arranged alternately, and the network can learn the joint representation of multi-angle texture and three-dimensional shape.

[0084] Secondly, the local occlusion coefficient is used as an additional channel and concatenated with the fused feature map output from the shallow layer in the middle layer of the network. The local occlusion coefficient is a single scalar value, which is first expanded into a single-channel matrix with the same spatial resolution as the fused feature map, and then concatenated along the channel dimension onto the fused feature map. Subsequently, it is input into the subsequent convolutional blocks of the network for deep feature extraction.

[0085] For example, the local occlusion coefficient of the USB socket solder joint is 0.36. It is copied and expanded into a 256×256 single-channel matrix, and then concatenated with the [16,256,256] fused feature map output from the first step along the channel dimension to obtain the [17,256,256] enhanced feature map. This enhanced feature map is then input into four residual blocks composed of depthwise separable convolutions for deep feature extraction. The network can perceive the degree of occlusion at this spatial location during each convolution operation.

[0086] Furthermore, the lightweight deep neural network uses depthwise separable convolution instead of standard convolution, decomposing standard convolution into channel-wise convolution and pointwise convolution, reducing the number of parameters and computational cost. Simultaneously, after network training, convolutional channels with low contribution are pruned. Specifically, after network training, the average L1 norm of the activation values ​​of each output channel on the validation set is calculated layer by layer as the contribution. After sorting from high to low, the lowest-performing channels with a cumulative contribution percentage below a preset ratio are directly removed from the network, effectively compressing the model size and adapting to the storage and computing power constraints of edge computing units.

[0087] The preset ratio refers to the cumulative contribution threshold used to define the range of inefficient channel pruning, which is usually set between 95% and 98%. A contribution greater than the preset ratio can be considered high, such as 97%. The specific value is determined comprehensively based on the number of network layers, the accuracy recovery capability after pruning, and the resource constraints of edge computing units.

[0088] For example, a lightweight deep neural network with 10 layers of depthwise separable convolutional residual blocks and an initial parameter count of 0.8M was trained on 50,000 solder joint samples to achieve a classification accuracy of 97.3%. Using a preset cumulative contribution threshold of 97%, the total contribution of the 64 channels in layer 5 was 25.6. After sorting the channels by their L1 norm mean activation values ​​from highest to lowest and accumulating the contribution channel by channel, the first 48 channels had a cumulative contribution of 24.89, accounting for 97.2%. The remaining 16 channels had a cumulative contribution of only 0.71, accounting for 2.8%, which is below the preset threshold of 97%. These inefficient channels were directly removed, resulting in a pruning rate of 25%. After independent pruning of each layer, an average of 35% of the channels were pruned across the entire network, reducing the parameter count to 0.52M.

[0089] Finally, the lightweight deep neural network output has two parallel task branches. The first branch is the solder joint semantic segmentation branch, which outputs a segmentation mask with the same resolution as the input, and labels the front meniscus, the rear fill area, the pad base, and the background area of ​​the solder joint pixel by pixel; the second branch is the defect classification branch, which outputs the defect category label of the solder joint after global average pooling and fully connected layers, such as normal, excessive porosity, cold solder joint, insufficient solder, crack, etc., as well as the corresponding confidence value.

[0090] For example, after network inference, the semantic segmentation branch outputs a 4-channel segmentation mask for the USB socket solder joints, accurately separating the meniscus region (red area), the follow-up fill region (blue area), and the pad base (green area), with a segmentation crossover ratio of 0.92; the defect classification branch outputs the defect category as excessive porosity, with a confidence level of 0.87.

[0091] In this step, the training process of the lightweight deep neural network uses the cross-entropy loss function, combining the semantic segmentation loss and defect classification loss: the segmentation branch calculates the cross-entropy between the four-class prediction and the labeled mask for each pixel, and the classification branch calculates the cross-entropy between the defect category prediction and the ground truth label. The two losses are directly added together as the total loss. Training is performed on an offline server using the Adam optimizer, with an initial learning rate of 0.001, which decays to 0.1 times the original rate every 30 epochs. The batch size is 64, and training is conducted for approximately 150 to 200 epochs until the total loss no longer decreases. Convergence is defined as the total loss fluctuation on the validation set being less than 0.005 for 10 consecutive epochs; at this point, the model weights are saved as the initial version. After convergence, channel pruning is performed and the recovery accuracy is fine-tuned to obtain a deployable lightweight model.

[0092] For example, taking the USB Type-C connector solder joint detection scenario on a tablet PC main control board as an example, a lightweight deep neural network with 10 layers of separable convolutional residual blocks and 0.8M parameters was used for training with 50,000 labeled solder joint samples. The total loss was the sum of the pixel-wise segmentation cross-entropy and the sample-wise classification cross-entropy. The total loss was approximately 1.85 in the early stages of training, rapidly decreasing to 0.52 in the first 30 rounds, dropping to 0.18 in the 90th round, and then to 0.09 in the 150th round. After that, the loss fluctuated within 0.003 for the next 10 rounds, indicating convergence. After pruning 35% of the low-contribution channels and fine-tuning for 30 rounds, the total loss recovered to 0.10, the classification accuracy was 97.1%, the model size was compressed to 0.52M, and the single solder joint inference time was 6ms after deployment to an edge NPU.

[0093] After obtaining the initial version of the lightweight deep neural network trained offline, this step converts the initial version into a runtime file executable by the edge computing unit using a model conversion toolchain. The conversion process includes steps such as computation graph optimization, operator fusion, and weight quantization, converting the floating-point model file under the training framework into a special format adapted to the target edge computing hardware architecture. At the same time, the model is quantized using INT8 or FP16 to further compress the size and accelerate inference.

[0094] For example, the initial version model file trained under the PyTorch framework, in .pth format and 2.1MB in size, is first exported as an ONNX intermediate representation. Then, graph optimization and INT8 quantization are performed using the conversion tool provided by the edge NPU manufacturer to generate runtime files that can be directly loaded by the edge computing unit, namely the .bin weight file and the .json network description file. After quantization, the size is compressed to 0.6MB, and the quantization accuracy loss is only 0.3% measured on 500 validation samples.

[0095] Furthermore, the generated runtime file is distributed to the edge computing unit deployed inside the multi-angle automated optical inspection equipment via the production line LAN or a secure transmission protocol, and written to its non-volatile storage medium. When the equipment is powered on or reset, the edge computing unit automatically loads the runtime file from the storage medium into memory, completes network initialization, and enters the inference state.

[0096] For example, the runtime file is downloaded from the offline server to the edge computing unit inside the detection device via Gigabit Ethernet and written to a designated directory in the eMMC storage. After the device is powered on, the edge computing unit completes the loading of the runtime file and NPU initialization within 8 seconds, and the status indicator light changes from red to green, indicating that the inference function is ready.

[0097] Finally, the edge computing unit connects to multiple cameras and structured light projectors within the multi-angle automated optical inspection equipment via standardized interfaces. The cameras utilize USB 3.0 interfaces for high-speed image data transmission, while the structured light projectors receive external trigger signals via GPIO interfaces to achieve synchronous switching of the stripe patterns. Furthermore, the edge computing unit communicates with the production line's manufacturing execution system via an industrial Ethernet interface, receiving inspection commands and transmitting inspection results back.

[0098] For example, the edge computing unit is based on an embedded module with an ARM Cortex-A78 and an NPU, and has 4GB of onboard LPDDR memory. Its four USB 3.0 interfaces connect to four CMOS cameras at 30°, 45°, 60°, and 75° respectively, with each channel having a bandwidth of 5Gbps to meet the real-time transmission of 1024×1024 grayscale images; one GPIO interface connects to the external trigger port of the DMD structured light projector, outputting a 10Hz TTL pulse signal to synchronously control stripe switching and camera exposure; and one gigabit Ethernet interface connects to the production line MES system, receiving the serial number of the motherboard to be inspected and transmitting back the defect classification results and confidence levels of each solder joint.

[0099] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0100] In summary, this step inputs multi-angle optical images, structured light depth maps, and local occlusion coefficients into a lightweight deep neural network on the edge computing unit. This enables the model to fuse multimodal features of texture and shape in the shallow layer, perceive the degree of occlusion at each spatial location in the middle layer, and adaptively suppress the response of interference regions. Finally, through dual-task branches, it synchronously outputs accurate segmentation masks and defect categories and their confidence scores for each key region of the solder joint. This achieves high-precision, low-latency real-time inference of solder joint defects at the edge, providing a reliable basis for subsequent confidence-based hierarchical decision-making.

[0101] S300: When the confidence level is greater than or equal to the first confidence threshold, the solder joint defect classification result is output, wherein the first confidence threshold is negatively correlated with the local occlusion coefficient.

[0102] like Figure 2 As shown, this step introduces an adaptive threshold mechanism for occlusion perception in the decision-making process to balance the relationship between confidence decay caused by occlusion and defect discrimination accuracy. This maximizes the proportion of direct decision-making at the edge while ensuring detection accuracy, and reduces reliance on secondary inference in the cloud and communication latency.

[0103] In this step, the first confidence threshold is negatively correlated with the local occlusion coefficient. Specifically, a baseline confidence threshold is set, and for solder joints with a local occlusion coefficient, the first confidence threshold is equal to the baseline confidence threshold multiplied by an adjustment factor negatively correlated with the local occlusion coefficient.

[0104] Specifically, firstly, the confidence score of the solder joint defect classification result output by the lightweight deep neural network is obtained, and simultaneously, the local occlusion coefficient of the solder joint is obtained. The confidence score ranges from [0,1], and the local occlusion coefficient ranges from [0,1].

[0105] For example, after network inference, the defect classification branch outputs a "porosity exceeding the standard" label for the solder joints of a USB Type-C connector, with a confidence level of 0.78; the corresponding local occlusion coefficient of the solder joint, after weighted calculation and normalization, is 0.36.

[0106] Furthermore, the first confidence threshold is negatively correlated with the local occlusion coefficient. A mapping function is used so that the more severe the occlusion, i.e. the larger the occlusion coefficient, the lower the threshold, and the more severe the occlusion, i.e. the smaller the occlusion coefficient, the higher the threshold.

[0107] Optionally, depending on the actual scenario's need to balance occlusion tolerance and false detection risk, linear mapping, piecewise mapping, or nonlinear mapping can be selected. Preferably, a linear mapping function is used in this step, which is simple in form, has low computational overhead, and is suitable for real-time decision-making at the edge.

[0108] For example, using a linear negative correlation mapping function, the first confidence threshold = base threshold - attenuation coefficient × local occlusion coefficient. The base threshold is the first confidence threshold when the occlusion coefficient is 0. It is determined based on the statistical distribution of model confidence in the unoccluded scenario, taking the 95th quantile of the normal class classification confidence on the unoccluded validation set. For example, if the confidence distribution of 10,000 normal solder joints on the unoccluded validation set is statistically analyzed, the 95th quantile is 0.90, which is the base threshold. The attenuation coefficient is calculated by reverse calculation based on the minimum threshold required for the occlusion coefficient to reach 1. Attenuation coefficient = base threshold - minimum threshold. The minimum threshold is set based on the lower limit of the model confidence distribution in the occluded scenario. For example, if the model confidence on the severely occluded validation set is rarely lower than 0.65, then T_minimum threshold = 0.65, and the attenuation coefficient = 0.90 - 0.65 = 0.25.

[0109] For example, to calculate the first confidence threshold, the base threshold is 0.90, the attenuation coefficient is 0.25, and the local occlusion coefficient is 0.36. Then the first confidence threshold = 0.90 - 0.25 × 0.36 = 0.81.

[0110] Finally, the confidence level of the solder joint defect classification is compared with the dynamically calculated first confidence threshold. When the confidence level is greater than or equal to the first confidence threshold, the defect classification result of the end-side inference is directly output without uploading to the cloud, and the result is sent back to the production line MES system.

[0111] For example, if a USB connector solder joint has a confidence level of 0.78 and a first confidence threshold of 0.81, then 0.78 < 0.81, meaning it's below the first confidence threshold. Therefore, the solder joint isn't directly output and proceeds to the next step to determine if cloud verification is needed. If another QFN solder joint has a confidence level of 0.88, a local occlusion coefficient of 0.15, and a first confidence threshold of 0.90 - 0.25 × 0.15 ≈ 0.86, then 0.88 > 0.86, and the classification result for insufficient solder is directly output.

[0112] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0113] S400: When the confidence level is less than the first confidence threshold and greater than or equal to the second confidence threshold, the solder joint detection data is uploaded to the cloud server for secondary inference to obtain the verification and classification results.

[0114] In this step, the second confidence threshold is a fixed value.

[0115] Specifically, when the confidence level of the output solder joint defect classification is less than the first confidence level threshold but greater than or equal to the second confidence level threshold, the solder joint is determined to be in the middle confidence level range. The end-side inference result is not sufficient to be directly accepted, but the data quality is not so low as to be completely indistinguishable. It needs to be uploaded to the cloud for secondary inference verification by a high-precision model.

[0116] The second confidence threshold is a fixed value, set based on the lower limit of the reliability of the edge model's defect classification confidence on the validation set. The confidence distribution of all false positive samples in the validation set, i.e., samples whose edge classification results are inconsistent with the true labels, is statistically analyzed, and the 95th percentile of the confidence of the false positive samples is taken as the second confidence threshold.

[0117] For example, among 10,000 validation samples, there were 270 false positives in the edge model, with confidence levels ranging from 0.12 to 0.62 and a 95th percentile of 0.48. To allow for some margin, the second confidence threshold was rounded up to 0.50. That is, when the solder joint confidence level is below 0.50, the edge inference is considered highly unreliable, triggering an alarm directly instead of uploading to the cloud, thus avoiding invalid data transmission and wasting cloud computing power.

[0118] For example, the confidence level of the USB Type-C connector solder joint is 0.78. The first confidence threshold is dynamically calculated to be 0.81 after applying an occlusion coefficient of 0.36, and the second confidence threshold is a fixed value of 0.50. If the confidence level is less than the first confidence threshold but greater than or equal to the second confidence threshold, it falls into the middle range and triggers the cloud-based review process.

[0119] Furthermore, the complete inspection data of the weld joint, including multi-angle automatic optical inspection images, structured light 3D topographic data, and local occlusion coefficients, is packaged and uploaded to a cloud server via an industrial Ethernet or 5G communication module. The uploaded data retains its original resolution to ensure that the high-precision model in the cloud can fully utilize the complete information for precise judgment.

[0120] For example, four frames of 1024×1024 grayscale images, a 1024×1024 pseudo-color depth map, and a local occlusion coefficient of 0.36 of the USB socket solder joints are losslessly compressed and packaged into a data packet of approximately 4.8MB. This packet is then uploaded to the cloud server via gigabit Ethernet, with a transmission time of approximately 40ms.

[0121] Furthermore, after receiving the weld joint inspection data, the cloud server invokes a high-precision deep neural network model deployed in the cloud for secondary inference. The output flow of the cloud-based high-precision deep neural network model is consistent with that of the lightweight deep neural network model at the edge, both employing a dual-task branch structure. Since the number of parameters in the cloud model is not limited by edge resources, a deeper backbone network and higher input resolution can be used to output the verification defect classification results and verification confidence levels.

[0122] For example, a high-precision model based on a ResNet-50 backbone network is deployed in the cloud. With approximately 25 million parameters and an input resolution of 2048×2048, it performs secondary inference on USB connector solder joints. The output verification classification result is "excessive porosity," with a verification confidence level of 0.91 and an inference time of approximately 120ms. The cloud then transmits the verification result back to the edge computing unit.

[0123] Finally, the edge computing unit receives the review and classification results transmitted from the cloud, uses these as the final defect determination result for the solder joints, and reports the result to the production line MES system. Simultaneously, the review results will be used for updating the false positive sample cache in subsequent tests.

[0124] For example, when the edge computing unit receives the result of excessive porosity from the cloud, which is consistent with the inference result on the edge side, it marks the solder joint as having excessive porosity and reports it to the MES system. At the same time, it records the deviation of 0.13 between the edge confidence level of 0.78 and the cloud confidence level of 0.91.

[0125] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0126] S500: An alarm is triggered when the confidence level is less than the second confidence threshold.

[0127] Specifically, when the confidence level of the output weld defect classification is less than the second confidence level threshold, the judgment end-side inference result is highly unreliable. The weld detection data may have problems such as severe occlusion, image blurring, failure of structured light reconstruction, or abnormal weld morphology that exceeds the known distribution range of the model. It is not suitable to directly accept it or upload it to the cloud for verification.

[0128] For example, if the solder joint of a USB Type-C connector is classified with a confidence level of only 0.32 after inference by the end-side network, which is lower than the second confidence level threshold of 0.50, it is judged as a low-confidence result, triggering an alarm process.

[0129] Furthermore, the edge computing unit generates alarm signals of corresponding levels based on the severity of the low confidence level. This can be further subdivided based on confidence level: a yellow alert is triggered when the confidence level is in the range of [0.30, 0.50), indicating that the quality of the detection data is questionable; a red alert is triggered when the confidence level is below 0.30, indicating a serious anomaly requiring immediate manual intervention.

[0130] For example, if the confidence level of the solder joint is 0.32 and falls within the range of [0.30, 0.50), the edge computing unit outputs a yellow warning signal via GPIO, illuminating the yellow warning light on the detection device. At the same time, it sends an alarm message to the production line MES system. The message includes the solder joint coordinates (32.5, 18.2), the alarm level as yellow warning, the confidence level as 0.32, and the preliminary judgment that the cause of the anomaly is severe occlusion or reconstruction anomaly.

[0131] Finally, the alarm information is synchronously written to the local log and the production line MES system database, recording the location coordinates of the alarm solder joint, the data storage path of the detection data, the end-side inference confidence level, and the timestamp. After receiving the alarm notification, the production line operator can retrieve multi-angle images and depth maps of the solder joint for manual verification, or remove the motherboard for offline precise inspection.

[0132] For example, after receiving a yellow warning, the MES system will pop up a prompt box on the operator interface, showing the location of the USB socket solder joint and a confidence level of 0.32. The operator retrieves the solder joint image and finds that the solder joint is severely blocked by the pin header connector and that the structured light stripes in this area show a large area of ​​phase calculation failure. The detection data is confirmed to be invalid, and the motherboard is marked as waiting for offline re-inspection and sent to the re-inspection buffer.

[0133] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0134] In summary, a complete confidence-based hierarchical decision-making and handling mechanism has been established, with the following overall effect: Based on the confidence level of edge-side inference, combined with the dynamic first confidence threshold and the fixed second confidence threshold for occlusion perception, solder joints are divided into three levels: high confidence direct output, medium confidence cloud verification, and low confidence alarm triggering. This maximizes the proportion of real-time edge-side decision-making while ensuring detection accuracy and reducing cloud communication and computing power overhead. At the same time, it ensures that low-quality data is not incorrectly accepted or invalidally transmitted, achieving an optimized balance between detection efficiency, accuracy, and system resource consumption.

[0135] S600: Update the local false detection sample cache based on samples where the review classification results are inconsistent with the solder joint defect classification results, wherein the update weight of the sample is positively correlated with the local occlusion coefficient.

[0136] This step involves continuously collecting false detection cases from the edge model during actual production line operation and weighting the samples differently based on the degree of occlusion. This allows incremental learning to focus on strengthening the model's defect discrimination ability under complex occlusion conditions, gradually improving the robustness of the edge model to difficult samples, and achieving adaptive evolution of model performance.

[0137] Step S600 provided by the present invention includes:

[0138] The false detection sample buffer is a first-in-first-out queue with a preset maximum capacity;

[0139] When the review classification result is inconsistent with the weld joint defect classification result, the corresponding sample will be repeatedly added to the false detection sample buffer area according to the weight value that is positively correlated with the local occlusion coefficient.

[0140] When the confidence level is less than the first confidence threshold and greater than or equal to the second confidence threshold, the corresponding sample is added to the false detection sample cache with a baseline weight.

[0141] In this step, after receiving the verification and classification results from the cloud, the edge computing unit compares them with the solder joint defect classification results inferred from the edge side. If the two are inconsistent, it is determined that a false detection has occurred on the edge side, and the weights are updated based on the local occlusion coefficient of the solder joints. The weights are positively correlated with the local occlusion coefficient.

[0142] Specifically, the update weight is calculated as follows: Update Weight = Base Weight + (Maximum Weight - Base Weight) × Local Occlusion Coefficient. The base weight is set to 1.0, corresponding to the update weight of false positives when the occlusion coefficient is 0, meaning that false positives in unoccluded scenarios occupy only one baseline entry in the buffer. The maximum weight is set to 5.0, corresponding to the update weight of false positives when the occlusion coefficient is 1, meaning that false positives in fully occluded scenarios occupy five entries in the buffer, with a sampling probability five times that of the baseline sample. The ratio of 5 is experimentally verified to achieve a balance between improved classification accuracy and training stability in occluded scenarios: if the ratio is too small, such as 2-3, the weighting of highly occluded samples is insufficient, and incremental learning has limited improvement in occluded scenarios; if the ratio is too large, such as 8-10, a few highly occluded samples dominate gradient updates, easily leading to performance degradation of the model in normal scenarios.

[0143] For example, the classification result of the USB socket solder joint end is insufficient solder, while the cloud-based review classification result is excessive porosity. The inconsistency between the two results indicates a false detection. The local occlusion coefficient of the solder joint is 0.36, the basic weight is 1.0, and the maximum weight is 5.0. Substituting these values ​​into the linear mapping function, we get the updated weight = 1.0 + (5.0 - 1.0) × 0.36 = 2.44.

[0144] Furthermore, false detection samples are repeatedly added to the false detection sample cache according to the integer values ​​of their updated weights. The higher the weight, the more entries the sample occupies in the cache, and the greater the probability of being sampled in subsequent incremental training, thereby realizing weighted reinforcement learning of false detection samples in severe occlusion scenarios.

[0145] The number of duplicate entries is determined by rounding down or to the nearest integer based on the update weight. One entry is taken when the update weight is less than 1.5, two entries are taken when the weight is between 1.5 and 2.5, and so on, ensuring the weight value is proportional to the number of entries in the cache. The base weight is fixed at 1.0, corresponding to one entry.

[0146] For example, if the updated weight of the USB connector solder joint is 2.44, after rounding, two more samples are added to the false detection sample buffer. If the occlusion coefficient of another row of pins is 0.82 and the updated weight is 4.28, then four more samples are added.

[0147] Furthermore, when the confidence level of a solder joint falls within the middle range of less than the first confidence threshold and greater than or equal to the second confidence threshold, regardless of whether the cloud verification result and the edge inference result are consistent, the sample is added to the false detection sample cache with a baseline weight. The baseline weight is 1.0, and the corresponding sample occupies only one cache entry, which is used to expand the sample diversity of the cache.

[0148] For example, a QFN solder joint has an edge confidence score of 0.68, a first confidence threshold of 0.81, and a second confidence threshold of 0.50, falling into the middle range and triggering cloud-based verification. The cloud-based verification result is consistent with the edge-based result, both being normal, but the sample is still added to the cache with a baseline weight of 1.0. That is, the sample occupies only one entry in the cache, providing a reference sample near the edge-based inference boundary for incremental learning.

[0149] Furthermore, the false positive sample buffer is a first-in, first-out (FIFO) queue, meaning that the data stored first is read first, and the data stored last is read last, with a preset maximum capacity. When the addition of new samples causes the number of entries in the buffer to exceed the maximum capacity, the earliest stored sample entries are removed in the order of their addition time until the number of entries meets the capacity limit.

[0150] For example, the cache has a preset maximum capacity of 150 entries, and currently contains 148 entries. Suppose two USB connector solder joint samples and one QFN solder joint sample are added, for a total of three new entries, bringing the cache count to 151, exceeding the capacity limit by one. In this case, the earliest added BGA solder joint sample will be automatically removed, restoring the cache count to 150. If the removed BGA sample originally had a weight of 3 and was added three times, only the earliest entry will be removed, while the other two will remain in the cache.

[0151] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0152] In summary, this step stores false positive samples with inconsistent cloud verification and edge inference results in a local cache after differentiating them by the degree of occlusion. This allows difficult samples in high-occlusion scenarios to obtain a higher entry proportion and sampling probability in the cache, thereby driving the model to focus on overcoming the weak links in defect discrimination under occlusion conditions in subsequent incremental training. At the same time, the cache size is controlled by the first-in-first-out mechanism and capacity limit to ensure controllable edge storage resources and timeliness of training data, providing an accurate and efficient sample accumulation foundation for the continuous adaptive evolution of the model.

[0153] S700: When the number of samples in the false detection sample buffer reaches a preset threshold and the production line is in an idle period, perform edge incremental learning.

[0154] In this step, the preset quantity threshold is determined based on a predetermined proportion of the maximum capacity of the false detection sample buffer.

[0155] The edge-side incremental learning adopts the elastic weight consolidation method, which adds regularization constraints to the original model parameters;

[0156] The training data consists of samples from the false positive sample buffer and historical samples mixed in a predetermined ratio; the historical samples refer to old samples that were collected and used for training before the current batch of incremental learning.

[0157] The training time is controlled within a preset time window, and the false detection sample cache is cleared after training is completed.

[0158] Specifically, the edge computing unit continuously monitors the number of entries in the false detection sample cache. When the number of entries reaches a predetermined proportion of the cache's maximum capacity, it further queries the production line's operating status through the MES system interface. If the production line is in an idle period and the remaining duration of the idle window is longer than the preset time window, the edge-side incremental learning process is automatically triggered.

[0159] The predetermined ratio refers to the percentage of the maximum capacity of the buffer containing false positive samples used to trigger incremental learning. This is set because a too low predetermined ratio leads to frequent triggering of incremental learning, occupying idle time on the production line and resulting in insufficient training samples per session, making convergence difficult. Conversely, a too high predetermined ratio triggers learning only when the buffer is nearly full, potentially causing high-weight new false positive samples to be prematurely squeezed out due to capacity limitations, thus missing learning opportunities for difficult samples. The predetermined ratio is typically set between 70% and 85%, such as 80%.

[0160] The preset time window is determined based on the minimum available time during typical idle periods on the production line, taking 70% to 80% of the minimum idle time to reserve margin for model validation, file replacement, and anomaly rollback. For example, if the idle period between shift changes on the production line is typically 60 minutes, 75% (45 minutes) is taken as the preset time window, with the remaining 15 minutes used for subsequent operations and buffering. If the production line has a shorter idle window for waiting for materials, such as 20 minutes, the preset time window is adjusted accordingly to 15 minutes, and the number of training rounds is reduced to adapt.

[0161] For example, the maximum capacity of the cache is set to 150 entries, the predetermined ratio is set to 80%, and the preset threshold is 150 × 80% = 120 entries. The cache has currently accumulated 122 entries, reaching the threshold. The edge computing unit queries the MES system and finds that it is currently during the early morning shift change, the production line is idle, and the estimated idle window is 60 minutes remaining. The preset time window is 75% of the idle time, i.e., 45 minutes, with the remaining 15 minutes used for model validation, file replacement, and exception rollback. Since the remaining 60 minutes of the idle window is greater than the preset time window of 45 minutes, the trigger condition is met, and incremental learning is automatically started.

[0162] Furthermore, the training data consists of all samples in the false positive sample cache and old samples randomly selected from the historical sample database, mixed in a predetermined ratio. The label of each false positive sample is verified by cloud-based classification results, while historical samples retain their original labels. After mixing, the data is uniformly shuffled to form the incremental training dataset for this batch.

[0163] The predetermined ratio mixing refers to the ratio of new samples to historical samples in the incremental learning training data, typically set between 6:4 and 8:2. This is based on the principle that a higher proportion of new samples allows the model to learn recent production line false detection patterns more effectively, but also increases the risk of catastrophic forgetting; a higher proportion of historical samples improves the retention of old knowledge, but slows down the model's absorption of new false detection features. Experiments have shown that when the proportion of new samples is 70%, i.e., the predetermined ratio is 7:3, the accuracy improvement on the cached samples after incremental learning is the greatest, while the accuracy decrease on the historical validation set is controlled within 0.5 percentage points, thus balancing the absorption of new knowledge with the retention of old knowledge.

[0164] For example, 122 false positive samples in the cache are used as new samples, with a predetermined mixing ratio of 7:3, meaning 70% new samples and 30% historical samples. The historical sample library currently contains 480 old samples, from which 52 are randomly selected stratified by category and mixed with the 122 new samples to form 174 training data points. The USB connector solder joints, due to a weight update of 2.44, were added twice to the cache, resulting in this sample appearing twice in the training set, thus increasing the sampling probability. After mixing, the data is uniformly shuffled and sent to training in batches.

[0165] Furthermore, incremental learning employs an elastic weight consolidation method, adding regularization constraints to the original model parameters. The total loss function is composed of a weighted sum of the task loss and the regularization constraint term. The regularization term constrains the degree to which the model parameters deviate from the original weight values ​​after offline training convergence. The more important the weight, the greater the penalty for deviation, thus preventing catastrophic forgetting of old knowledge while absorbing new false detection features.

[0166] Wherein, the total loss function = task loss + (regularization strength coefficient / 2) × (importance coefficient × (current weight - original weight)). 2 The task loss is the sum of the cross-entropy losses of the semantic segmentation branch and the defect classification branch, with a regularization strength coefficient typically set between 0.1 and 0.5. A larger regularization strength coefficient means the model parameters tend to remain near their original values ​​after offline training convergence, resulting in less forgetting of old knowledge but slower absorption of new knowledge; conversely, a smaller regularization strength coefficient has the opposite effect. Experiments have shown that when λ=0.4, the accuracy drop on the historical validation set is controlled within 0.3 percentage points, while the accuracy improvement on new samples is the greatest.

[0167] After offline training converges, the pre-built lightweight deep neural network synchronously calculates and stores the diagonal values ​​of the Fisher information matrix for each weight parameter as an importance index, used for elastic weight consolidation constraints in the incremental learning phase. The importance coefficient is obtained by calculating the diagonal values ​​of the Fisher information matrix from historical training data. The larger the importance coefficient, the more important the weight is to the historical task, and the heavier the penalty for the weight deviating from its original value during incremental training, thus protecting the ability to extract key features from being overwritten. The current weight value is the current value of the trainable weight parameter during incremental training, while the original weight value is the original value of the corresponding weight parameter saved after offline training converges.

[0168] For example, the importance matrix of each weight is stored during the offline training phase. During incremental training, the original weight values ​​stored after offline training convergence are 0.75, -0.32, and 1.18, respectively; the importance coefficients of the diagonal of the Fisher information matrix are: 0.82 for the meniscus edge detection channel (high importance), 0.47 for the pad base texture channel (medium importance), and 0.15 for the occluded region blur texture channel (low importance); the regularization strength coefficient is 0.4; the task loss on the current batch of training data is 0.38; the current weight values ​​during incremental training are 0.73, -0.29, and 0.86, respectively. The total loss is 0.38 + (0.4 / 2) × [0.82 × (0.73 - 0.75)]. 2 +0.47×(-0.29+0.32) 2 +0.15×(0.86-1.18) 2 =0.383222. Incremental learning was trained using the Adam optimizer with a learning rate of 0.0001 and a batch size of 16.

[0169] Furthermore, the training process is strictly controlled within a preset time window, and an early stop mechanism is adopted. When the loss decreases by less than 0.005 for three consecutive rounds or the training time reaches the upper limit of the time window, the process is automatically terminated to ensure that the production line can resume normal production without affecting the production line.

[0170] For example, the preset time window is 45 minutes, and the total idle time between production line shifts is 60 minutes. 75% (45 minutes) is used for training, and the remaining 15 minutes are used for model validation, file replacement, and anomaly rollback. After training starts, the time consumption and loss changes are monitored in real time. By the 38th minute, the loss has converged from the initial 0.42 to 0.09, and the loss fluctuation is less than 0.005 for three consecutive epochs, triggering an early stop to terminate training. If training has not converged by the 45th minute, it is forcibly terminated, and the current optimal checkpoint is used to update the model. This batch ended 7 minutes early due to the early stop, leaving a total of 22 minutes for model validation and file replacement. When the next motherboard flows into the inspection station, the new model inference is automatically loaded.

[0171] Furthermore, after incremental training, the performance of the new model is quickly evaluated on a validation subset consisting of a mix of cached samples and historical samples. If the classification accuracy is improved compared to before training and is not lower than 95% of the initial offline version, then the model passes validation. After successful validation, a new runtime file is generated, replacing the currently running model file on the edge computing unit, with the model version number incremented.

[0172] For example, when evaluated on a 10% validation subset of the mixed training set, the accuracy was 62.3% before incremental learning and 91.8% after incremental learning, which is not lower than 95% (92.2%) of the initial offline version accuracy of 97.1%. Since 91.8% is slightly lower than this threshold, after five more rounds of fine-tuning, the accuracy reached 93.5%, passing the validation. A new runtime file is generated to replace the old file, and the model version number is incremented from V2.3.1 to V2.3.2. The new model is automatically loaded for inference when the next motherboard enters the detection station.

[0173] Finally, after training is completed and validation is successful, the false positive sample cache is cleared, allowing it to start accumulating new false positive samples again. At the same time, stratified sampling is performed from the current batch of training data to extract representative samples to supplement the historical sample library. When the historical sample library exceeds its capacity limit, the oldest sample added to the library is removed.

[0174] For example, all 122 samples in the cache were cleared, and the counter was reset to zero. From the 174 training data points, 10 samples each of five categories—normal, excessive porosity, poor soldering, insufficient solder, and cracking—were extracted to supplement the historical sample library, increasing the library size from 480 to 530 samples, exceeding the 500-sample limit. The oldest 30 samples were removed, sorted by entry time, restoring the library size to 500 samples. The incremental learning log was triggered at 02:15, with 174 samples (122 new samples and 52 old samples), and the accuracy increased from 62.3% to 93.5%. The model version was V2.3.2, and the process took 28 minutes.

[0175] It should be noted that the above values ​​are for illustrative purposes only and do not constitute a limitation on the present invention.

[0176] In summary, by automatically triggering edge-side incremental learning when false detection samples accumulate to a preset threshold and the production line is idle, continuous adaptive evolution of the edge-side model during production line operation is achieved, ultimately forming a closed-loop learning mechanism that synergistically optimizes detection accuracy and operational efficiency.

[0177] In summary, this invention constructs a complete closed-loop detection system, which has the ability to perceive and adapt to the occlusion effects of surrounding components in complex assembly environments; it achieves the optimal configuration of computing power and accuracy between the lightweight model on the edge and the high-precision model in the cloud; and it can continuously absorb false detection samples and self-optimize during continuous operation on the production line, ultimately achieving a unified system of high precision, low latency, strong robustness and self-evolution in controlling the detection of defects in the solder joints of the motherboard interlayer.

[0178] Example 2, as Figure 3 As shown, the present invention provides an intelligent detection system for controlling solder joints in the motherboard interlayer, the system comprising:

[0179] The detection data loading module 11 is used to load the solder joint detection data of the core board to be tested, and calculate the local occlusion coefficient of each solder joint based on the solder joint position coordinates and the preset surrounding component height model.

[0180] The weld point detection data includes multi-angle automatic optical detection images and structured light three-dimensional topography data.

[0181] The loading of solder joint inspection data from the core board to be inspected includes:

[0182] Multiple cameras at different angles are controlled to acquire grayscale images of the solder joint area. The grayscale images are preprocessed to form a multi-channel input tensor, which is used as the multi-angle automatic optical detection image.

[0183] The structured light projector projects an coded stripe pattern, and the three-dimensional point cloud of the weld joint area is reconstructed by the phase-shifting method. The height information is then mapped into a pseudo-color depth map as the structured light three-dimensional topography data.

[0184] The preset peripheral component height model is a pre-stored core board component layout database. The core board component layout database includes the package height, pin offset, and metal casing reflection characteristics of adjacent components within a preset radius around each solder point.

[0185] The local occlusion coefficient is calculated as follows: based on the solder joint position coordinates, the surrounding components are traversed, the occlusion angle and occlusion intensity of each component are calculated, and the weighted sum is normalized to a predetermined value range.

[0186] The local occlusion coefficient calculation module 12 is used to input the solder joint detection data and the local occlusion coefficient into a lightweight deep neural network on the edge computing unit to obtain the solder joint defect classification result and its confidence level.

[0187] The process of inputting the solder joint detection data and the local occlusion coefficient into a lightweight deep neural network on the edge computing unit includes:

[0188] The multi-channel input tensor and the pseudo-color depth map are fused together in the shallow layer of the lightweight deep neural network to obtain a fused feature map.

[0189] The local occlusion coefficient is used as an additional channel and concatenated with the fused feature map in the middle layer of the network before being input into subsequent convolutional blocks.

[0190] The lightweight deep neural network is constructed based on depthwise separable convolution and channel pruning techniques. The output has a dual-task branch: the first branch is used for solder joint semantic segmentation, and the second branch is used for defect classification.

[0191] This also includes:

[0192] Collect historical solder joint detection datasets and train an initial version of the lightweight deep neural network on an offline server;

[0193] The initial version is converted into a runtime file executable by the edge computing unit and deployed to the edge computing unit;

[0194] The edge computing unit is deployed inside a multi-angle automated optical inspection device and connects to multiple cameras and structured light projectors at different angles through a standardized interface.

[0195] The end-side lightweight inference module 13 is used to output the solder joint defect classification result when the confidence level is greater than or equal to the first confidence level threshold, wherein the first confidence level threshold is negatively correlated with the local occlusion coefficient.

[0196] The high-confidence direct output module 14 is used to upload the solder joint detection data to the cloud server for secondary inference and obtain the verification and classification results when the confidence level is less than the first confidence level threshold and greater than or equal to the second confidence level threshold.

[0197] The medium confidence level cloud verification module 15 is used to trigger an alarm when the confidence level is less than the second confidence level threshold.

[0198] Specifically, the first confidence threshold is negatively correlated with the local occlusion coefficient. Specifically, a baseline confidence threshold is set, and for solder joints with a local occlusion coefficient, the first confidence threshold is equal to the baseline confidence threshold multiplied by an adjustment factor negatively correlated with the local occlusion coefficient.

[0199] The second confidence threshold is a fixed value.

[0200] The low confidence alarm triggering module 16 is used to update the local false detection sample cache based on samples where the review classification result is inconsistent with the weld joint defect classification result. The update weight of the sample is positively correlated with the local occlusion coefficient.

[0201] Specifically, the local false detection sample cache is updated based on samples where the review classification results and the solder joint defect classification results are inconsistent. The update weight of the sample is positively correlated with the local occlusion coefficient, including:

[0202] The false detection sample buffer is a first-in-first-out queue with a preset maximum capacity;

[0203] When the review classification result is inconsistent with the weld joint defect classification result, the corresponding sample will be repeatedly added to the false detection sample buffer area according to the weight value that is positively correlated with the local occlusion coefficient.

[0204] When the confidence level is less than the first confidence threshold and greater than or equal to the second confidence threshold, the corresponding sample is added to the false detection sample cache with a baseline weight.

[0205] The false detection sample caching and incremental learning module 17 is used to perform edge-side incremental learning when the number of samples in the false detection sample cache reaches a preset number threshold and the production line is in an idle period.

[0206] The preset quantity threshold is determined based on a predetermined proportion of the maximum capacity of the false detection sample buffer.

[0207] The edge-side incremental learning adopts the elastic weight consolidation method, which adds regularization constraints to the original model parameters;

[0208] The training data consists of samples from the false positive sample buffer and historical samples mixed in a predetermined ratio; the historical samples refer to old samples that were collected and used for training before the current batch of incremental learning.

[0209] The training time is controlled within a preset time window, and the false detection sample cache is cleared after training is completed.

[0210] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0211] This specification and accompanying drawings are merely illustrative examples of the invention and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its scope. Therefore, if such modifications and modifications fall within the scope of the invention and its equivalents, the invention is intended to include these modifications and modifications.

Claims

1. A method for intelligently detecting solder joints in the interlayer of a motherboard, characterized in that, include: Load the solder joint detection data of the core board to be tested, and calculate the local occlusion coefficient of each solder joint based on the solder joint position coordinates and the preset surrounding component height model; The solder joint detection data and the local occlusion coefficient are input into a lightweight deep neural network on the edge computing unit to obtain the solder joint defect classification results and their confidence levels. When the confidence level is greater than or equal to the first confidence threshold, the solder joint defect classification result is output, and the first confidence threshold is negatively correlated with the local occlusion coefficient. When the confidence level is less than the first confidence threshold and greater than or equal to the second confidence threshold, the solder joint detection data is uploaded to the cloud server for secondary inference to obtain the review and classification results. An alarm is triggered when the confidence level is less than the second confidence threshold. The local false detection sample cache is updated based on the discrepancy between the review classification results and the solder joint defect classification results. The update weight of the sample is positively correlated with the local occlusion coefficient. When the number of samples in the false detection sample buffer reaches a preset threshold and the production line is in an idle period, edge incremental learning is performed.

2. The intelligent detection method for controlling motherboard interlayer solder joints as described in claim 1, characterized in that, The weld joint detection data includes multi-angle automatic optical detection images and structured light three-dimensional topography data; The loading of solder joint inspection data from the core board to be inspected includes: Multiple cameras at different angles are controlled to acquire grayscale images of the solder joint area. The grayscale images are preprocessed to form a multi-channel input tensor, which is used as the multi-angle automatic optical detection image. The structured light projector projects an coded stripe pattern, and the three-dimensional point cloud of the weld joint area is reconstructed by the phase-shifting method. The height information is then mapped into a pseudo-color depth map as the structured light three-dimensional topography data.

3. The intelligent detection method for controlling motherboard interlayer solder joints as described in claim 1, characterized in that, The preset peripheral component height model is a pre-stored core board component layout database. The core board component layout database includes the package height, pin offset, and metal casing reflection characteristics of adjacent components within a preset radius around each solder point.

4. The intelligent detection method for controlling motherboard interlayer solder joints as described in claim 1, characterized in that, The local occlusion coefficient is calculated as follows: based on the solder joint position coordinates, traverse the surrounding components, calculate the occlusion angle and occlusion intensity of each component, and then normalize the weighted sum to a predetermined value range.

5. The intelligent detection method for controlling motherboard interlayer solder joints as described in claim 2, characterized in that, The solder joint detection data and the local occlusion coefficient are input into a lightweight deep neural network on the edge computing unit, including: The multi-channel input tensor and the pseudo-color depth map are fused together in the shallow layer of the lightweight deep neural network to obtain a fused feature map. The local occlusion coefficient is used as an additional channel and concatenated with the fused feature map in the middle layer of the network before being input into subsequent convolutional blocks. The lightweight deep neural network is constructed based on depthwise separable convolution and channel pruning techniques. The output has a dual-task branch: the first branch is used for solder joint semantic segmentation, and the second branch is used for defect classification.

6. The intelligent detection method for controlling motherboard interlayer solder joints as described in claim 1, characterized in that, Also includes: Collect historical solder joint detection datasets and train an initial version of the lightweight deep neural network on an offline server; The initial version is converted into a runtime file executable by the edge computing unit and deployed to the edge computing unit; The edge computing unit is deployed inside a multi-angle automated optical inspection device and connects to multiple cameras and structured light projectors at different angles through a standardized interface.

7. The intelligent detection method for controlling motherboard interlayer solder joints as described in claim 1, characterized in that, The first confidence threshold is negatively correlated with the local occlusion coefficient. Specifically, a baseline confidence threshold is set, and for solder joints with a local occlusion coefficient, the first confidence threshold is equal to the baseline confidence threshold multiplied by an adjustment factor negatively correlated with the local occlusion coefficient. The second confidence threshold is a fixed value.

8. The intelligent detection method for controlling motherboard interlayer solder joints as described in claim 1, characterized in that, The local false detection sample cache is updated based on samples whose classification results differ from the weld joint defect classification results. The update weight of the sample is positively correlated with the local occlusion coefficient, including: The false detection sample buffer is a first-in-first-out queue with a preset maximum capacity; When the review classification result is inconsistent with the weld joint defect classification result, the corresponding sample will be repeatedly added to the false detection sample buffer area according to the weight value that is positively correlated with the local occlusion coefficient. When the confidence level is less than the first confidence threshold and greater than or equal to the second confidence threshold, the corresponding sample is added to the false detection sample cache with a baseline weight.

9. The intelligent detection method for controlling motherboard interlayer solder joints as described in claim 1, characterized in that: The preset quantity threshold is determined based on a predetermined proportion of the maximum capacity of the false detection sample buffer. The edge-side incremental learning adopts the elastic weight consolidation method, which adds regularization constraints to the original model parameters; The training data consists of samples from the false positive sample buffer and historical samples mixed in a predetermined ratio; the historical samples refer to old samples that were collected and used for training before the current batch of incremental learning. The training time is controlled within a preset time window, and the false detection sample cache is cleared after training is completed.

10. An intelligent detection system for controlling solder joints in the motherboard interlayer, characterized in that, A method for controlling the intelligent detection of solder joints in the motherboard interlayer as described in any one of claims 1-9, comprising: The detection data loading module is used to load the solder joint detection data of the core board to be tested, and calculate the local occlusion coefficient of each solder joint based on the solder joint position coordinates and the preset surrounding component height model. The local occlusion coefficient calculation module is used to input the solder joint detection data and the local occlusion coefficient into a lightweight deep neural network on the edge computing unit to obtain the solder joint defect classification result and its confidence level. A lightweight inference module on the end side is used to output the solder joint defect classification result when the confidence level is greater than or equal to a first confidence threshold, wherein the first confidence threshold is negatively correlated with the local occlusion coefficient; The high-confidence direct output module is used to upload the solder joint detection data to the cloud server for secondary inference and obtain the verification and classification results when the confidence level is less than the first confidence threshold and greater than or equal to the second confidence threshold. The medium confidence level cloud verification module is used to trigger an alarm when the confidence level is less than the second confidence level threshold; The low confidence alarm triggering module is used to update the local false detection sample cache based on samples where the review classification results are inconsistent with the weld joint defect classification results. The update weight of the sample is positively correlated with the local occlusion coefficient. The false detection sample caching and incremental learning module is used to perform edge-side incremental learning when the number of samples in the false detection sample cache reaches a preset threshold and the production line is in an idle period.

Images