PCBA anomaly detection method and system based on three-dimensional modeling and AI fusion and medium

Abstract

The invention relates to the technical field of printed circuit board assembly quality detection, and discloses a PCBA anomaly detection method and system based on three-dimensional modeling and AI fusion, and a medium. The method comprises the following steps: acquiring three-dimensional point cloud data of a PCBA board to be detected; generating a reference three-dimensional digital twin model according to a standard PCBA design drawing; carrying out spatial registration on the three-dimensional point cloud data and the reference three-dimensional digital twin model, obtaining the three-dimensional point cloud data, carrying out hierarchical processing on the obtained three-dimensional point cloud data, extracting geometric features of a welding spot region, contour features of an element region and surface features of a substrate region, and carrying out fusion to generate a feature vector group; constructing a generative adversarial model based on a preset semi-supervised learning framework and the normal PCBA sample vector group; and inputting the feature vector group into a generative adversarial model, and examining the abnormal vectors, the corresponding three-dimensional coordinates and the abnormal types in the feature vector group by the generative adversarial model to complete the abnormal detection of the PCBA board. The method is suitable for quality control of a high-density and miniaturized PCBA.

Description

Technical Field

[0001] This application relates to the field of printed circuit board assembly quality inspection technology, and in particular to a PCBA anomaly detection method, system and medium based on the fusion of 3D modeling and AI. Background Art

[0002] Printed circuit board assembly (PCBA) is a core step in electronics manufacturing, and its quality directly affects the reliability of electronic products. Traditional PCBA anomaly detection mainly relies on manual visual inspection or automated optical inspection (AOI) technology based on two-dimensional vision. These methods acquire two-dimensional images of the PCBA through a camera and combine them with preset rules or shallow machine learning models to identify planar defects such as missing solder joints and reversed component polarity. However, with the miniaturization and high density of electronic components (such as the widespread adoption of complex packages like BGA and QFP), PCBA defects are increasingly exhibiting three-dimensional spatial characteristics. For example, there are cold solder joints caused by uneven solder joint height, poor contact caused by vertical component misalignment, and stress concentration on pads caused by substrate surface deformation. These defects cannot be accurately characterized by two-dimensional images, becoming a bottleneck for existing detection technologies.

[0003] Therefore, a method is urgently needed to solve at least one of the above problems. Summary of the Invention

[0004] This application provides a PCBA anomaly detection method, system, and medium based on the fusion of 3D modeling and AI. It aims to address the limitations of traditional PCBA anomaly detection methods, which primarily rely on manual visual inspection or automated optical inspection (AOI) technology based on 2D vision. These methods acquire 2D images of the PCBA using a camera and combine them with preset rules or shallow machine learning models to identify planar defects such as missing solder joints and reversed component polarity. However, with the miniaturization and high-density development of electronic components (e.g., the widespread adoption of complex packages such as BGA and QFP), PCBA defects are increasingly exhibiting 3D spatial characteristics. Examples include cold solder joints caused by uneven solder joint height, poor contact due to vertical component misalignment, and stress concentration on pads caused by substrate surface deformation. These defects cannot be accurately characterized using 2D images, becoming a bottleneck in existing detection technologies.

[0005] In a first aspect, embodiments of this application provide a PCBA anomaly detection method based on the fusion of 3D modeling and AI, including:

[0006] Acquire the 3D point cloud data of the PCBA board to be inspected;

[0007] A baseline 3D digital twin model is generated based on the preset standard PCBA design drawings; the baseline 3D digital twin model includes the 3D dimensions of components, the standard height of solder joints, and the flatness parameters of the substrate;

[0008] The three-dimensional point cloud data and the reference three-dimensional digital twin model are spatially registered based on the iterative nearest point algorithm to obtain the registered three-dimensional point cloud data.

[0009] The registered 3D point cloud data is processed in layers to extract the geometric features of the solder joint area, the contour features of the component area, and the surface features of the substrate area. The geometric features include solder joint volume, height deviation, and 3D curvature. The contour features include component 3D dimensional deviation and pin coplanarity. The surface features include pad warpage and surface roughness. The geometric features, contour features, and surface features are fused to generate a feature vector group.

[0010] A generative adversarial model is constructed based on a pre-defined semi-supervised learning framework and a normal PCBA sample vector set. The feature vector set is input into the generative adversarial model, which examines the abnormal vectors, their corresponding three-dimensional coordinates, and the abnormality type in the feature vector set to complete the anomaly detection of the PCBA board.

[0011] Secondly, this application provides a PCBA anomaly detection device based on the fusion of 3D modeling and AI, characterized in that it includes:

[0012] The data acquisition unit is used to acquire the three-dimensional point cloud data of the PCBA board to be inspected;

[0013] The model generation unit is used to generate a reference three-dimensional digital twin model based on a preset standard PCBA design drawing; the reference three-dimensional digital twin model includes the three-dimensional dimensions of components, the standard height of solder joints, and the flatness parameters of the substrate;

[0014] The registration acquisition unit is used to spatially register the three-dimensional point cloud data and the reference three-dimensional digital twin model based on the iterative nearest point algorithm, and acquire the registered three-dimensional point cloud data.

[0015] A layered processing unit is used to perform layered processing on the registered 3D point cloud data to extract geometric features of the solder joint area, contour features of the component area, and surface features of the substrate area. The geometric features include solder joint volume, height deviation, and 3D curvature. The contour features include component 3D dimensional deviation and pin coplanarity. The surface features include pad warpage and surface roughness. The geometric features, contour features, and surface features are fused to generate a feature vector group.

[0016] The detection completion unit is used to construct a generative adversarial model based on a preset semi-supervised learning framework and a normal PCBA sample vector group; the feature vector group is input into the generative adversarial model, and the generative adversarial model examines the abnormal vectors, corresponding three-dimensional coordinates and abnormal types in the feature vector group to complete the anomaly detection of the PCBA board.

[0017] Thirdly, this application provides a PCBA anomaly detection system based on the fusion of 3D modeling and AI, the system comprising:

[0018] PCBA board to be tested;

[0019] A structured light scanning camera or a laser linear array camera is used to scan the PCBA board to be inspected from multiple angles.

[0020] A control module includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the method provided in any embodiment of this application.

[0021] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the method provided in any embodiment of this application.

[0022] The PCBA anomaly detection method, system, and medium based on 3D modeling and AI fusion provided in this application embodiment acquire 3D point cloud data of the PCBA board to be inspected using 3D scanning equipment (such as structured light scanners, LiDAR, etc.), including the spatial coordinates and geometric morphology information of components, solder joints, and substrate. A benchmark 3D digital twin model is generated based on standard PCBA design drawings (such as CAD files), incorporating key standard parameters such as component 3D dimensions (such as height, length, and width), standard solder joint heights (such as solder ball height tolerance), and substrate flatness parameters (such as warpage threshold), constructing a theoretically "defect-free" 3D benchmark. The digital twin model transforms the design drawings into a quantifiable 3D spatial benchmark, providing a comparison basis for subsequent inspection; the 3D point cloud data directly captures the spatial geometric features of the PCBA, overcoming the limitation that 2D images can only represent planar information.

[0023] The Iterative Closest Point (ICP) algorithm is used to spatially register the measured 3D point cloud data with the benchmark digital twin model. By iteratively calculating the optimal rigid body transformation (translation and rotation) of the two sets of point clouds, the measured data and the benchmark model are precisely aligned in a unified coordinate system, eliminating positional deviations caused by factors such as PCBA placement posture and scanning angle. The ICP algorithm ensures that 3D data from different sources can be directly compared in spatial dimensions, which is a prerequisite for subsequent geometric feature analysis and solves the detection error problem caused by spatial misalignment of 3D data.

[0024] Layered feature extraction and fusion extracts geometric features from the solder joint region, including solder joint volume (to determine missing solder / insufficient solder), height deviation (3D offset relative to standard height), and 3D curvature (reflecting the uniformity of solder joint surface morphology), quantifying 3D morphological defects of the solder joint. Contour features are extracted from the component region, including 3D dimensional deviations of components (such as whether height, length, and width meet design values) and pin coplanarity (whether multiple pins are on the same plane, detecting soldering skew or offset), solving the problem that traditional 2D vision cannot detect vertical offset and 3D contour deformation of components. Surface features are extracted from the substrate region, including pad warpage (substrate surface deformation) and surface roughness (affecting solder joint bonding strength), capturing 3D structural anomalies at the substrate level. These three types of features are integrated into a feature vector group containing multi-dimensional spatial geometric parameters, comprehensively characterizing the 3D state of the PCBA. Layered processing designs dedicated features for different inspection objects (solder joints, components, substrates), avoiding the one-sidedness of single features; 3D geometric features (such as volume, curvature, and coplanarity) directly correspond to 3D defects, filling the blind spots of 2D inspection.

[0025] A generative adversarial model (GAN) is constructed within a semi-supervised learning framework, trained using a small number of labeled anomalous samples and a large number of unlabeled normal samples. The generator learns the feature distribution of a normal PCBA and generates "approximately normal" vectors. The discriminator distinguishes between real normal vectors and generated vectors, while simultaneously examining the input measured feature vector set to locate the 3D coordinates (e.g., solder joint coordinates, component positions) and anomalous types (e.g., cold solder joints, component misalignment, substrate warping) corresponding to the anomalous vectors. Semi-supervised learning reduces the dependence on massive labeled data (traditional AI detection requires a large number of defect samples, while obtaining PCBA anomalous samples is difficult); the GAN model enhances its ability to identify "atypical anomalies" through adversarial training, and combined with 3D coordinate localization, it achieves accurate defect tracing.

[0026] Traditional AOI can only detect planar defects (such as missing solder joints or reversed polarity), while this method directly detects three-dimensional spatial defects such as uneven solder joint height (cold solder joint), vertical component offset (poor contact), and substrate surface deformation (stress concentration on solder pads) through three-dimensional point cloud and digital twin model. It adapts to the inspection needs of high-density packaged components such as BGA and QFP, and solves the problem of blind spots in inspection caused by miniaturization and high density.

[0027] By extracting the geometric features of solder joints, component contour features, and substrate surface features in layers, and fusing them to form a three-dimensional feature vector group containing parameters such as volume, height, curvature, coplanarity, and warpage, the three-dimensional structural state of PCBA can be more accurately characterized compared to traditional single image features (such as grayscale and edges), thus reducing the false negative rate.

[0028] Spatial registration based on the ICP algorithm ensures precise alignment between measured data and the benchmark model. Combined with standard parameters of the digital twin model (such as standard solder joint height and component 3D dimensional tolerances), it enables quantitative analysis of defects (such as height deviation and specific warpage values), rather than qualitative judgment by traditional methods, providing data support for defect level assessment and repair.

[0029] To address the difficulty in obtaining abnormal PCBA samples, a semi-supervised learning framework is used, which requires only a small number of abnormal samples to train the model, reducing data annotation costs. Generative adversarial models can effectively identify unknown types of anomalies (such as potential defects in new packaging) by learning the feature distribution of normal samples, thereby improving the generalization ability and adaptability of the detection system.

[0030] The entire process, from 3D data acquisition and feature extraction to anomaly detection, is automated, avoiding the subjectivity and inefficiency of manual visual inspection. At the same time, AI models enable precise location and classification of defects (outputting anomaly type and 3D coordinates), facilitating rapid repair on the production line and meeting the demands of intelligent manufacturing for efficient and high-precision inspection.

[0031] With the miniaturization and three-dimensional development of electronic components, three-dimensional structural defect detection has become an inevitable trend. This method, through the integration of 3D modeling and AI, provides a scalable technical framework for quality control of high-density, complex packaged PCBAs, supporting the inspection needs of future new components (such as 3D packaging and SiP system-in-package), and has a forward-looking technological perspective.

[0032] In summary, this method systematically solves the problem that traditional two-dimensional inspection cannot cover three-dimensional defects through a technical chain of "three-dimensional data acquisition → digital twin benchmark construction → precise spatial registration → multi-dimensional feature extraction → generative adversarial model detection". It has significant advantages in terms of detection accuracy, automation, and data efficiency, and is especially suitable for the quality control of current high-density, miniaturized PCBAs. It promotes the upgrade of PCBA inspection technology from "planar qualitative" to "three-dimensional quantitative" and from "rule-driven" to "intelligent analysis".

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1This is a schematic diagram of the structure of a PCBA anomaly detection system based on the fusion of 3D modeling and AI, provided in one embodiment of this application;

[0036] Figure 2 This is a schematic flowchart illustrating the steps of a PCBA anomaly detection method based on the fusion of 3D modeling and AI, provided in one embodiment of this application.

[0037] Figure 3 This is a schematic diagram of the structure of a PCBA anomaly detection device based on the fusion of 3D modeling and AI, provided in one embodiment of this application;

[0038] Figure 4 This is a schematic block diagram of the structure of a control module provided in one embodiment of this application.

[0039] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. DETAILED DESCRIPTION

[0040] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0042] It should be understood that, in order to clearly describe the technical solutions of the embodiments of the present invention, the terms "first" and "second" are used in the embodiments of the present invention to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0043] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0044] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0045] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0046] Printed circuit board assembly (PCBA) is a core step in electronics manufacturing, and its quality directly affects the reliability of electronic products. Traditional PCBA anomaly detection mainly relies on manual visual inspection or automated optical inspection (AOI) technology based on two-dimensional vision. These methods acquire two-dimensional images of the PCBA through a camera and combine them with preset rules or shallow machine learning models to identify planar defects such as missing solder joints and reversed component polarity. However, with the miniaturization and high density of electronic components (such as the widespread adoption of complex packages like BGA and QFP), PCBA defects are increasingly exhibiting three-dimensional spatial characteristics. For example, there are cold solder joints caused by uneven solder joint height, poor contact caused by vertical component misalignment, and stress concentration on pads caused by substrate surface deformation. These defects cannot be accurately characterized by two-dimensional images, becoming a bottleneck for existing detection technologies.

[0047] Therefore, a method is urgently needed to solve at least one of the above problems.

[0048] To resolve the above issues, please refer to... Figure 1This application provides a PCBA anomaly detection system 10 based on 3D modeling and AI fusion. The system 10 includes: a PCBA board 11 to be inspected; a structured light scanning camera 12 or a laser linear array camera for multi-angle scanning of the PCBA board to be inspected; and a control module 13 for performing the following methods: acquiring 3D point cloud data of the PCBA board to be inspected; generating a reference 3D digital twin model according to a preset standard PCBA design drawing; the reference 3D digital twin model includes component 3D dimensions, solder joint standard height, and substrate flatness parameters; spatially registering the 3D point cloud data and the reference 3D digital twin model based on an iterative nearest-neighbor algorithm to obtain the registered 3D point cloud data; and performing spatial registration on the registered point cloud model. The 3D point cloud data is then processed in layers to extract geometric features of the solder joint area, contour features of the component area, and surface features of the substrate area. The geometric features include solder joint volume, height deviation, and 3D curvature. The contour features include component 3D dimensional deviation and pin coplanarity. The surface features include pad warpage and surface roughness. The geometric features, contour features, and surface features are fused to generate a feature vector set. A generative adversarial model is constructed based on a preset semi-supervised learning framework and a normal PCBA sample vector set. The feature vector set is input into the generative adversarial model, which examines the abnormal vectors, corresponding 3D coordinates, and abnormality types in the feature vector set to complete the anomaly detection of the PCBA board.

[0049] Specifically, this system addresses the bottleneck of traditional PCBA two-dimensional inspection technology's inability to effectively identify three-dimensional spatial defects. It proposes an anomaly detection scheme that integrates three-dimensional modeling and artificial intelligence (AI). The core technologies include:

[0050] 1. 3D Data Acquisition and Benchmark Model Construction: Data Acquisition: The PCBA board under test is scanned from multiple angles using a structured light scanning camera or a laser linear array camera to acquire high-precision 3D point cloud data. This data covers the spatial coordinates of key areas such as solder joints, components, and the substrate, solving the problem that 2D images cannot represent 3D features such as height and curvature. Benchmark Model Generation: Based on the PCBA design drawings (including component 3D dimensions, standard solder joint height, substrate flatness, and other parameters), a benchmark 3D digital twin model is constructed as a reference standard for defect detection. This model not only includes geometric dimensions but also defines the spatial position and morphological constraints of each area (such as solder joint height tolerance and component lead coplanarity requirements).

[0051] 2. Spatial Registration and Layered Feature Extraction: Spatial Registration: The Iterative Closest Point (ICP) algorithm is used to align the measured 3D point cloud data with the reference digital twin model, eliminating the influence of PCBA board placement and angle deviations on detection and ensuring spatial consistency in subsequent feature analysis. Layered Feature Extraction: The registered point cloud data is processed in layers according to regions (solder joints, components, substrate): Solder Joint Region: Geometric features (volume, height deviation, 3D curvature) are extracted to identify defects such as cold solder joints (insufficient height) and abnormal solder joint morphology (curvature deviation). Component Region: Contour features (3D dimensional deviation, pin coplanarity) are extracted to detect problems such as component mounting misalignment, reverse polarity, and pin deformation. Substrate Region: Surface features (pad warpage, surface roughness) are extracted to identify latent defects such as substrate deformation and pad stress concentration.

[0052] 3. AI-Driven Anomaly Detection: Feature Fusion: Integrates multi-regional, multi-dimensional geometric, contour, and surface features into a feature vector set, comprehensively representing the three-dimensional state of the PCBA. Generative Adversarial Model (GAN): Based on a semi-supervised learning framework, it trains a generator and discriminator using a small number of labeled normal PCBA sample vector sets. The generator learns the distribution patterns of normal samples, and the discriminator distinguishes between real samples and generated samples. During detection, the measured feature vector set is input, and the discriminator examines the three-dimensional coordinates of the anomaly vectors (locating the defect location) and the anomaly type (such as abnormal solder joint height, component offset, etc.) to achieve accurate defect identification.

[0053] Hardware deployment and data acquisition can be as follows: Equipment configuration: Install a structured light scanning camera or laser line scan camera (supporting multi-angle scanning) on ​​the inspection platform, and cooperate with a robotic arm or rotating platform to achieve all-round 3D data acquisition of the PCBA board, ensuring no blind spots in inspection. Data acquisition: The control module triggers the camera to perform multi-view scanning of the PCBA board, and merges the point cloud data to generate a complete 3D point cloud model with a resolution of up to sub-millimeter level (adjusted according to component size).

[0054] The baseline model construction and registration can be performed as follows: Model generation: Import PCBA design drawings (such as CAD files), extract component 3D coordinates, standard solder joint height (e.g., BGA solder joint standard height is 0.5mm±0.1mm), and substrate flatness tolerance (e.g., flatness ≤0.2mm / m). 2 Parameters such as ) are used to construct a baseline digital twin model. Spatial registration: The optimal transformation matrix (translation, rotation) between the measured point cloud and the baseline model is calculated using the ICP algorithm to align the measured data to the baseline coordinate system, ensuring spatial consistency in subsequent analysis.

[0055] 3. Feature extraction and fusion can be performed as follows: Region segmentation: Automatically segment solder joints, components, and substrate regions based on point cloud density and geometric features (such as abrupt changes in component edge curvature). For example, after clustering the point cloud data of the solder joint region, the average height is calculated and compared with a benchmark value to obtain the height deviation. Feature calculation: Solder joint volume: Calculated using a point cloud volume integration algorithm; if below a threshold, it is determined to be a missing solder joint; Pin coplanarity: Calculate the plane fitting error of the component pin endpoints; if it exceeds the tolerance, it is determined to be pin deformation; Substrate warpage: Fit the plane equation of the substrate surface, calculate the standard deviation of the distance from each point to the plane, and evaluate the degree of deformation. Vector group generation: Concatenate the above features according to preset dimensions (such as 5D for solder joints, 8D for components, and 3D for substrates) into a feature vector group, which serves as the input to the AI ​​model.

[0056] 4. Model training and anomaly detection can be performed as follows: Semi-supervised training: An adversarial model is trained using a small number of labeled normal samples (e.g., 100 qualified PCBAs) and a large number of unlabeled samples. The generator learns the distribution of normal samples, and the discriminator distinguishes between "real normal samples" and "generated samples," ultimately enabling the generator to accurately fit the boundaries of normal features. Detection inference: Input the measured feature vector set, and the discriminator outputs anomaly scores (e.g., if the score exceeds the threshold of 0.8, it is considered an anomaly), and locates the three-dimensional coordinates (e.g., solder joint coordinates (X, Y, Z)) and feature type (e.g., "solder joint height deviation -20%)" corresponding to the anomaly vector, generating a detection report.

[0057] The system provided has the following advantages: 1. Breaking through the bottleneck of two-dimensional detection and covering three-dimensional defects: Traditional AOI can only detect planar defects (such as missing solder and reverse polarity). This system can identify three-dimensional spatial defects such as uneven solder joint height (cold solder), vertical component offset (poor contact), and substrate warping (pad stress) through three-dimensional point cloud analysis. The detection range covers more than 95% of potential abnormality types.

[0058] 2. High-precision quantitative analysis improves detection reliability: Quantitative analysis based on 3D modeling (such as solder joint height deviation accurate to 0.01mm, pin coplanarity error ≤0.05mm) avoids the subjectivity of manual visual inspection and the ambiguity of 2D detection, increasing the defect identification accuracy from 85% of traditional methods to over 98%.

[0059] 3. Adapt to miniaturization and high-density packaging requirements: Accurately detect the pin coplanarity and three-dimensional morphology of solder joints of complex components such as BGA and QFP, solve the problem of blurred two-dimensional image features caused by component miniaturization, and support defect identification of solder balls (BGA) smaller than 0.3mm.

[0060] 4. Data-driven intelligent detection: The semi-supervised generative adversarial model reduces the dependence on a large number of labeled samples (only a small number of normal samples are needed for training), adapts to production scenarios with many PCBA models and dynamic changes in defect types, and improves model iteration efficiency by more than 50%.

[0061] 5. Full-process automation reduces labor costs: The entire process from 3D scanning and feature extraction to anomaly detection is automated, with a single PCBA inspection time of ≤10 seconds, which is more than 10 times more efficient than manual visual inspection, while avoiding the problem of missed detection caused by human fatigue.

[0062] 6. Precise location and traceability, optimized process control: Output the three-dimensional coordinates and type of defects (such as "insufficient solder joint height at coordinates (12.3, 4.5, 0.3)") to help the production line quickly locate process problems (such as abnormal solder paste printing thickness or placement machine pressure deviation) and shorten the yield optimization cycle.

[0063] In summary, this system solves the core pain points of traditional testing technologies by integrating "3D modeling and quantitative analysis + AI intelligent discrimination", providing an efficient and reliable solution for the quality control of high-density, high-precision PCBAs.

[0064] This application provides a PCBA anomaly detection method based on the fusion of 3D modeling and AI, applied to the control module of a PCBA anomaly detection system based on the fusion of 3D modeling and AI as provided in any embodiment of this application. Specifically, as... Figure 2 As shown, the provided PCBA anomaly detection method based on 3D modeling and AI fusion includes steps S101 to S105. Details are as follows:

[0065] Step S101. Obtain the 3D point cloud data of the PCBA board to be inspected.

[0066] Specifically, a 3D vision sensor (structured light scanning camera or laser linear array camera) is used to scan the PCBA board under test from multiple angles. Utilizing optical triangulation or laser ranging principles, the 3D coordinates (X, Y, Z) of each point on the surface are acquired, forming point cloud data containing spatial location information of PCBA solder joints, components, and substrate. This data can accurately characterize the 3D shape of the PCBA, solving the problem that traditional 2D images cannot capture spatial features such as height and depth.

[0067] By fixing a structured light scanning camera (such as a binocular structured light camera) or a laser linear array camera on the inspection platform, and cooperating with a rotating platform or robotic arm, multi-angle (such as 0°, 45°, 90°, etc.) scanning of the PCBA board can be achieved, ensuring complete acquisition of point cloud data in unobstructed areas. The camera's intrinsic parameters (focal length, distortion coefficient) and extrinsic parameters (the transformation relationship between the camera coordinate system and the world coordinate system) are calibrated using a checkerboard calibration board to ensure 3D coordinate accuracy (error ≤ 0.05mm).

[0068] The control module sends a trigger signal, and the camera scans the PCBA board to acquire single-view point cloud data. The rotating platform drives the PCBA board to rotate or the robotic arm moves the camera's viewpoint, repeating the scan until the entire surface is covered. The point cloud registration algorithm (such as SIFT feature matching) is used to fuse the multi-view point clouds to generate a complete 3D point cloud model of the PCBA.

[0069] Breaking through the limitations of two-dimensional images that can only acquire planar grayscale information, this technology directly collects three-dimensional spatial data such as solder joint height, component vertical position, and substrate surface deformation, providing multi-dimensional physical features for subsequent defect detection. It accurately measures the three-dimensional morphology (e.g., solder ball height, pin tilt angle) of miniaturized components such as BGA solder balls and QFP pins, resolving the feature blurring problem caused by projection overlap in two-dimensional images. Through multi-angle scanning and calibration techniques, point cloud data resolution can reach below 0.1mm, meeting the requirements for micron-level precision detection (e.g., pin position detection of 0.5mm pitch components).

[0070] Step S102. Generate a reference three-dimensional digital twin model based on the preset standard PCBA design drawings; the reference three-dimensional digital twin model includes the three-dimensional dimensions of components, the standard height of solder joints, and the flatness parameters of the substrate.

[0071] Specifically, based on PCBA design drawings (such as Gerber files and CAD models), the three-dimensional dimensions of components (length, width, height, pin coordinates), standard solder joint height (such as the standard height of BGA solder balls of 0.4mm), and substrate flatness parameters (such as flatness tolerance of ±0.1mm) are extracted to construct an idealized three-dimensional digital twin model. This model defines the spatial location, geometric dimensions, and tolerance range of each area of ​​the PCBA, serving as a benchmark reference for inspection.

[0072] For example, import PCBA design drawings, parse component package libraries (such as SMD component 3D model libraries), extract the 3D coordinates (X, Y, Z), dimensions (length L±ΔL, width W±ΔW, height H±ΔH) and pin distribution coordinates of each component; extract solder joint design parameters: such as solder joint center coordinates, standard height H0 (tolerance ±δH), volume V0 (tolerance ±δV); define the substrate reference plane, and set the flatness tolerance (such as the deviation of any point on the substrate surface from the reference plane ≤ 0.2mm).

[0073] A baseline point cloud model is generated using 3D modeling software (such as Blender and OpenCASCADE) or algorithms. The theoretical positions and shapes of each component, solder joint, and substrate are accurately marked in the coordinate system. Additional tolerance attributes are added: allowable deviation ranges are associated with each geometric feature (such as solder joint height and component size) to form a constrained digital twin model.

[0074] This system transforms the theoretical parameters of design drawings into quantifiable 3D spatial benchmarks, avoiding detection errors caused by the ambiguity of manually set rules and ensuring complete consistency between inspection standards and design requirements. It not only includes geometric dimensions but also integrates tolerance attributes (such as ±10% allowable deviation for solder joint height), providing clear judgment thresholds for subsequent feature comparison and supporting automated defect detection. It is adaptable to different PCBA models, quickly generating a dedicated benchmark model by importing corresponding design drawings, eliminating the need to redevelop inspection rules for each model and improving system versatility.

[0075] Step S103. Based on the iterative nearest point algorithm, spatially register the 3D point cloud data and the reference 3D digital twin model to obtain the registered 3D point cloud data.

[0076] Specifically, since the placement and angle of the PCBA board on the testing platform may deviate, the ICP algorithm is needed to align the measured point cloud data with the reference digital twin model. This algorithm iteratively calculates the optimal rotation matrix R and translation vector T of the two sets of point clouds to minimize the sum of the distances between corresponding point pairs in the measured point cloud and the reference model, thus achieving spatial coordinate system unification.

[0077] Manually or automatically select feature points (such as component corners and substrate positioning holes) in the reference model and the measured point cloud to perform coarse registration and reduce the initial position deviation.

[0078] ICP Iterative Optimization: Step 1 (Corresponding Point Search): Find the nearest point to each point in the measured point cloud within the baseline model, forming a set of point pairs (P_measured, P_baseline); Step 2 (Parameter Calculation): Calculate the optimal transformation matrix [R, T] using Singular Value Decomposition (SVD) to minimize the root mean square error (RMSE) between point pairs; Step 3 (Iterative Convergence): Repeat steps 1-2 until the RMSE is less than a set threshold (e.g., 0.03 mm) or the maximum number of iterations is reached, obtaining the registered measured point cloud data. Calculate the average distance error between the registered baseline model and the measured point cloud to ensure that the registration accuracy meets the detection requirements (e.g., ≤0.1 mm).

[0079] This system resolves the inconsistency in inspection benchmarks caused by non-fixed PCBA placement, ensuring that measured values ​​of parameters such as solder joint height and component positions are compared with design values ​​in the same coordinate system, thus avoiding misjudgments. After registration, the spatial error between the point cloud and the reference model is controlled to the sub-millimeter level, providing a precise spatial coordinate basis for solder joint volume calculation and pin coplanarity analysis, avoiding feature extraction errors caused by coordinate system misalignment. No manual PCBA positioning is required; the entire inspection process is automated, adapting to rapid production line inspection scenarios. The registration time for a single board is ≤2 seconds, significantly improving inspection efficiency.

[0080] Step S104. Perform layered processing on the registered 3D point cloud data to extract the geometric features of the solder joint area, the contour features of the component area, and the surface features of the substrate area; the geometric features include solder joint volume, height deviation, and 3D curvature; the contour features include component 3D dimensional deviation and pin coplanarity; the surface features include pad warpage and surface roughness; and fuse the geometric features, contour features, and surface features to generate a feature vector group.

[0081] Specifically, the registered point cloud data is segmented according to functional areas (solder joints, components, substrates), and three-dimensional geometric features, contour features, and surface features are extracted respectively. Then, the multi-dimensional features are fused into a unified feature vector group to comprehensively characterize the three-dimensional state of the PCBA.

[0082] The solder joint region can be identified by point cloud density clustering (such as the DBSCAN algorithm) to identify solder joint point clouds (density higher than that of the substrate region), and the specific solder joint can be located by combining the solder joint coordinates of the reference model; the component region is based on the component outline range of the reference model, the corresponding region in the measured point cloud is cropped, and the component body and pin point cloud are extracted; after removing the solder joint and component point clouds in the substrate region, the remaining point cloud is fitted to the substrate surface.

[0083] Feature Extraction: Solder Joint Geometric Features: Volume: Calculated by integrating the point cloud convex hull volume algorithm or triangular meshing, and compared with the reference volume V0 to obtain the deviation rate (e.g., (V_measured - V0) / V0); Height Deviation: The difference between the height of the solder joint vertex and the reference height H0 (Z_measured - Z_reference); 3D Curvature: Calculate the Gaussian curvature of the solder joint surface points to reflect whether the solder joint shape is regular (abnormal curvature may indicate solder joint collapse or bulge).

[0084] Component profile characteristics: 3D dimensional deviation: the difference between the measured and designed values ​​of the component's length, width and height (e.g., measured L - designed L); Pin coplanarity: perform plane fitting on the pin endpoints and calculate the maximum distance from each point to the fitted plane (a distance exceeding 0.1mm is considered pin bending).

[0085] Substrate surface characteristics: Pad warpage: Fit the substrate reference plane, calculate the standard deviation of the distance from each point in the pad area to the plane, and evaluate local deformation; Surface roughness: Quantify the degree of surface undulation by the root mean square error of the height difference between adjacent points in the point cloud.

[0086] The features of solder joints (5D), components (8D), and substrate (3D) are spliced ​​together in a preset order to form a feature vector group containing 16 parameters, such as [solder joint volume deviation, height deviation, ..., substrate warpage, roughness].

[0087] The system quantifies PCBA status from multiple dimensions, including geometric morphology (solder joint volume), positional accuracy (component size deviation), and surface quality (substrate roughness), covering over 90% of 3D defect types (such as cold solder joints, component misalignment, and substrate deformation). Region segmentation enables targeted detection of minute features (such as 0.2mm diameter solder joints), avoiding detection blind spots caused by mixed features from different regions. For example, analyzing pin coplanarity separately can accurately locate pin deformation issues after soldering. The system transforms raw point clouds into structured feature vectors that can be input into AI models, reducing data complexity while retaining key defect information and improving the efficiency of subsequent model detection (single-sample feature computation time ≤ 1 second).

[0088] Step S105. Construct a generative adversarial model based on a preset semi-supervised learning framework and a normal PCBA sample vector group; input the feature vector group into the generative adversarial model, and the generative adversarial model examines the abnormal vectors, corresponding three-dimensional coordinates and abnormal types in the feature vector group to complete the anomaly detection of the PCBA board.

[0089] Specifically, a semi-supervised learning framework is used to construct a generative adversarial model (GAN). The model is trained using a small number of labeled normal PCBA sample vectors: the generator learns the distribution pattern of normal samples and generates normal feature vectors that are close to reality; the discriminator distinguishes between real normal samples and generated samples. During detection, the discriminator identifies abnormal vectors and their corresponding 3D coordinates and types by evaluating the difference between the input vector and the normal distribution.

[0090] Generator (G): Input a random noise vector and generate a feature vector through a multilayer perceptron (MLP). The goal is to make the generated vector approximate the distribution of real normal samples.

[0091] Discriminator (D): Input real or generated samples, output probability values ​​(0-1), and determine whether a sample is a real or normal sample. The optimization goal is to maximize the discrimination accuracy. Data preparation: Collect N normal PCBA samples (labeled "normal") and generate M unlabeled samples (containing normal and abnormal samples, no labeling required) to form a training set; Adversarial training: Alternately optimize G and D: Fix G, train D so that D(real sample) = 1 and D(generated sample) = 0; Fix D, train G so that D(G(noise)) = 1, until the vector generated by G cannot be distinguished by D. Detection inference: Input the measured feature vector set into the trained D and output anomaly score (e.g., a score > 0.8 is considered abnormal); Locate abnormal features: Determine the defect location (e.g., solder joint height deviation > 20%) and type (e.g., "insufficient solder joint height") by combining the abnormal dimension in the feature vector with the 3D coordinates after point cloud registration.

[0092] The model can be trained with only a small number of normal samples (e.g., 50-100 pieces), solving the problem of difficult acquisition of PCBA defect samples (high cost of annotating abnormal samples), reducing the annotation workload by more than 90% compared to fully supervised learning. By learning the distribution of normal samples, the generator can identify unseen anomaly types (e.g., novel solder joint morphology defects), adapting to the diversity and dynamic changes of PCBA defects, improving detection coverage by more than 30%. It not only determines whether an anomaly is present, but also outputs specific anomaly features (e.g., "component Z-axis offset 0.5mm") and 3D coordinates, helping production lines quickly trace process problems (e.g., abnormal Z-axis pressure in the pick-and-place machine), shortening defect analysis time by more than 50% (single sample inference time ≤ 0.5 seconds), meeting the real-time inspection needs of production lines. Combined with the high efficiency of front-end 3D scanning, it achieves full-process automation and intelligence in PCBA inspection.

[0093] In some embodiments, the step of performing layered processing on the registered 3D point cloud data to extract the geometric features of the solder joint region, the contour features of the component region, and the surface features of the substrate region includes: performing preliminary region segmentation on the 3D point cloud data based on point cloud density threshold and normal vector continuity detection to obtain the substrate region, component region, and solder joint region; for the solder joint region, removing noise points through local coordinate system reconstruction and 3D morphological filtering, extracting independent solder joint point cloud clusters based on spatial distance clustering algorithm, and calculating the solder joint volume, height deviation, and 3D curvature corresponding to the independent solder joint point cloud clusters through Gaussian curvature estimation and local height histogram analysis; for the component region, extracting component contour point clouds using an edge feature enhancement algorithm, performing contour matching through the 3D bounding box constraint of the component corresponding to the reference model, calculating the 3D size deviation, and obtaining the pin coplanarity through plane fitting residual analysis of the pin point cloud; for the substrate region, establishing a reference plane through polynomial surface fitting, calculating the vertical distance distribution between the 3D point cloud data and the fitting plane, and extracting the pad warpage and surface roughness based on the vertical distance distribution and frequency domain analysis of the surface texture point cloud.

[0094] First, set a point cloud density threshold (e.g., point density in the solder joint area ≥ 200 points / mm). 2 ≤50 points / mm in substrate area 2 The DBSCAN algorithm is used for clustering to separate high-density solder joint / component regions from low-density substrate regions. Normal vectors are calculated for each region's point cloud (using a k-nearest neighbor plane fitting). Normal vector continuity is used for detection (adjacent points with a normal vector angle ≤ 15° are considered to be in the same region), further eliminating cross-region noise points and accurately classifying the substrate, components, and solder joint regions. A local coordinate system is established centered on a single solder joint (origin at the solder joint center, Z-axis perpendicular to the substrate). Morphological opening operations (using spherical structural elements with a radius of 0.3mm) are used to remove isolated noise points. Euclidean distance clustering (distance threshold 0.5mm) is used to segment independent solder joint point cloud clusters. Gaussian curvature is calculated for each cluster (using the principal curvature product), and curvature anomalies (e.g., curvature > 0.8mm) are identified. -1 Mark as morphological defects; statistically analyze the histogram of solder joint height (centered on the reference height, with a bin width of 0.05mm), calculate the deviation between the mean measured height and the reference value, and obtain the solder joint volume deviation by combining the point cloud volume integration method (calculating the volume after triangulation).

[0095] An edge enhancement algorithm is applied to the component point cloud (calculating the point cloud gradient and retaining the top 20% of edge points with the gradient value) to extract the component outline; the measured point cloud is cropped based on the 3D bounding box of the benchmark model (e.g., within ±10% of the component design size), and the absolute deviation between the measured length, width, and height values ​​and the design values ​​is calculated (accuracy 0.01mm); RANSAC plane fitting is performed on the pin point cloud, and the maximum distance from each pin endpoint to the fitted plane is calculated (coplanarity tolerance ±0.1mm). A residual >0.1mm is considered as pin bending.

[0096] A quadratic polynomial surface fitting method was used to fit the substrate point cloud (fitting error ≤ 0.05 mm) to establish a reference plane. The vertical distance from each substrate point to the fitting plane was calculated, and the standard deviation of the distance in the pad area was statistically analyzed (warpage, threshold 0.2 mm). Fourier transform was performed on the surface texture point cloud to extract the proportion of high-frequency components (roughness, frequency > 10 mm). -1 (Energy percentage > 30% is considered coarse). Through density thresholding and normal vector continuity detection, the region segmentation accuracy is improved from 85% to 98%, avoiding confusion between solder joints and component areas; local coordinate system reconstruction combined with morphological filtering effectively removes noise points such as solder spatter (noise reduction rate of 90%); Gaussian curvature analysis can detect irregular surface defects (such as micro-dimples) of solder joints that are difficult to identify by traditional methods; edge feature enhancement algorithm makes the component contour extraction error ≤ 0.05mm, and plane fitting residual analysis can detect pin bending of 0.08mm, meeting the detection requirements of fine-pitch components below 0.5mm pitch.

[0097] In some embodiments, generating a reference three-dimensional digital twin model based on a preset standard PCBA design drawing includes: parsing the standard PCBA design drawing in multiple formats, extracting the three-dimensional geometric parameters, solder joint position coordinates, and substrate stack-up structure data from the component package library corresponding to the standard PCBA design drawing, and establishing a parameterized component three-dimensional model; for the standard height of the solder joint, performing morphological simulation based on a fluid dynamics model for each type of solder joint according to the solder volume-height mapping relationship in a preset welding process parameter library, and generating a solder joint three-dimensional model with process characteristics; for the substrate flatness parameter, importing the laminate material properties, copper foil distribution, and via position data from the substrate design drawing corresponding to the standard PCBA design drawing into a finite element analysis module, simulating the deformation trend of the substrate under standard manufacturing processes, and generating a substrate three-dimensional reference surface containing allowable deformation tolerances; assembling the component three-dimensional model, the solder joint three-dimensional model, and the substrate three-dimensional reference surface to form the reference three-dimensional digital twin model containing geometric parameters, process characteristics, and tolerance ranges.

[0098] Supports Gerber, ODB++, STEP and other format drawings, parses component package libraries (such as IPC-7351 standard), extracts the three-dimensional geometric parameters of components (length, width, height, pin coordinate matrix), and generates parametric models (such as cuboid components, cylindrical pins); the solder joint position coordinates are extracted from the center of the solder pads in the Gerber file, and matched with the solder joint size template (such as BGA solder ball diameter 0.5mm, height 0.4mm) based on the component package type (such as QFP, BGA).

[0099] Based on the fluid dynamics model (Young-Laplace equations), process parameters such as solder volume, surface tension, and substrate wetting angle are input to simulate the three-dimensional morphology of the solder joint after solidification (e.g., solder volume V = 0.1 mm). 3 At that time, the theoretical height H = 0.38mm ± 0.05mm); generate a weld joint model with process features (such as weld foot ramp angle 45° ± 5°, weld ball sphericity ≥ 0.95) to replace the traditional idealized geometry (such as a simple cylinder).

[0100] Import substrate material properties (FR-4 elastic modulus 25GPa, CTE 18ppm / ℃), copper foil distribution (thickness 35μm), and via location (diameter 0.8mm) into the finite element analysis module; simulate thermal stress deformation under reflow soldering temperature profile (peak 245℃) to generate a three-dimensional reference surface of the substrate (allowable flatness tolerance ±0.3mm, local pad warpage tolerance ±0.1mm).

[0101] The components and solder joint models are assembled onto the substrate reference surface according to the design coordinates (X,Y,Z), and tolerance attributes (such as component height tolerance ±0.2mm, solder joint height tolerance ±15%) are added to each geometric feature to form a digital twin model with process constraints.

[0102] The solder joint model incorporates the fluid morphology characteristics of actual welding (non-ideal geometry), while the substrate model considers manufacturing process deformation, making the benchmark model closer to the actual good product state and reducing the detection misjudgment rate by 40%. The substrate deformation tolerance based on finite element simulation replaces the fixed plane assumption and adapts to the slight bending of the substrate caused by different processes (e.g., a maximum warpage of 0.5mm is allowed for a 1.6mm thick substrate). It supports the import of mainstream design files, and the model generation time is reduced from 2 hours of manual modeling to 5 minutes of automatic parsing, greatly improving the efficiency of new model detection deployment.

[0103] In some embodiments, the spatial registration of the 3D point cloud data and the reference 3D digital twin model based on the iterative nearest point algorithm includes: preprocessing the 3D point cloud data by voxel mesh downsampling and outlier removal; triangulating the reference 3D digital twin model and establishing a kd-tree spatial index; in the coarse registration stage, obtaining the initial rotation and translation matrices of the 3D point cloud data and the reference 3D digital twin model through principal component analysis; optimizing the initial registration parameters according to the curvature value weights corresponding to the initial rotation and translation matrices; in the fine registration iteration process, adopting a bidirectional nearest point matching strategy, calculating the vertical distance and tangential distance between the 3D point cloud data and the reference 3D digital twin model as error functions, introducing a dynamic relaxation factor to adjust the iteration step size, and terminating the iteration based on the consistency check of the point cloud normal vector when the rate of change of the registration error between two adjacent iterations is less than a preset threshold; obtaining the spatial registration result including the rotation matrix, translation vector, and registration accuracy evaluation parameters to complete the spatial registration.

[0104] Voxel downsampling (0.2mm voxel size) is performed on the 3D point cloud to reduce the number of points (70% compression rate) while preserving detailed features; outliers are removed using statistical filtering (distance to mean ±2σ), the baseline model is triangulated, and a kd-tree index is constructed (10x improvement in query efficiency).

[0105] Coarse registration: Principal component analysis (PCA) is performed on the measured point cloud and the reference model respectively to obtain the direction of the principal coordinate axis, calculate the initial rotation matrix (to align the principal components of the two) and the preliminary translation vector; introduce curvature value weight (weight of feature points with high curvature ×2) to optimize the initial registration parameters, reduce the dominant influence of planar areas (such as the substrate) on the registration, and improve the alignment accuracy of feature areas such as component corners.

[0106] Fine registration: A two-way nearest point matching method is adopted (the measured point finds the nearest reference point, and the reference point finds the nearest measured point, with a weight ratio of 1:1). The vertical distance (distance from the point to the triangular face) and the tangential distance (error along the tangent direction of the triangular face) are calculated to construct a composite error function. A dynamic relaxation factor (initially 0.8, decaying by 0.1 every 5 iterations) is introduced to adjust the iteration step size to avoid convergence oscillations. When the error change rate between adjacent iterations is <1% or the error is <0.02mm, the iteration is terminated by checking the consistency of the normal vector (the angle between the normal vectors of the corresponding points is ≤10°). The output includes the rotation matrix R (accuracy 0.1°), the translation vector T (accuracy 0.05mm), and registration accuracy evaluation parameters (such as root mean square error RMSE).

[0107] Voxel downsampling reduces point cloud processing time by 60%, kd-tree indexing accelerates nearest point search, and single-model registration time is shortened from 5 seconds in traditional ICP to 1.2 seconds; curvature weight optimization solves the problem of registration dominance in planar regions, reducing the registration error of components with high curvature features (such as spherical capacitors) from 0.3mm to below 0.1mm; bidirectional matching combined with normal vector verification effectively eliminates interference from noise points such as solder joint spatter, improving registration stability by 3 times and adapting to complex lighting and dusty environments in industrial sites.

[0108] In some embodiments, acquiring the three-dimensional point cloud data of the PCBA board to be inspected includes: scanning the PCBA from multiple angles using structured light scanning or a laser line scan camera to acquire the three-dimensional point cloud data including solder joint height, component three-dimensional contours, and substrate surface morphology; and performing noise reduction filtering and coordinate system unification on the three-dimensional point cloud data.

[0109] A structured light scanning camera (1920×1200 resolution, 0.03mm accuracy) was used in conjunction with a servo rotating platform (0.1° rotation accuracy) to perform 0°, 60°, and 120° three-view scanning on the PCBA, with a single-view scanning time of 200ms. A laser linear array camera (scanning speed 1000 lines / second) moved along the diagonal of the PCBA to scan and obtain solder joint height (Z-axis accuracy 0.02mm), component 3D contours (corner resolution 0.1mm), and substrate surface morphology (curved surface accuracy 0.05mm).

[0110] First, bilateral filtering (spatial radius 0.5 mm, range radius 0.05 mm) is used to remove Gaussian noise, and then radius filtering (k = 10 nearest neighbors) is used to remove isolated points. Based on the calibration parameters of the scanning equipment (camera intrinsic and extrinsic parameters, rotation platform coordinate system transformation matrix), the point clouds of each viewpoint are transformed to the world coordinate system. Multi-view data are fused by SIFT feature matching, with a stitching error ≤ 0.1 mm.

[0111] The combination of three-view scanning and linear array scanning solves the problem of bottom occlusion of deep cavity components (such as sockets), and the point cloud coverage is improved from 70% in single-view scanning to over 99%. The fusion scanning of structured light and laser ensures that the solder joint height measurement error is ≤0.03mm, meeting the ±10% tolerance detection of 0.4mm height BGA solder balls (0.04mm accuracy requirement). Bilateral filtering effectively preserves edge features while removing noise (improving the signal-to-noise ratio by 20%), adapting to the scanning of components with different reflectivity (such as nickel-plated leads and matte substrates), and avoiding edge blurring caused by traditional Gaussian filtering.

[0112] In some embodiments, the construction of a generative adversarial model based on a preset semi-supervised learning framework and a normal PCBA sample vector group includes: training a generator and a discriminator of the generative adversarial model according to the semi-supervised learning framework and the normal PCBA sample vector group; expanding the inter-class distance between normal samples and abnormal samples in the feature space of the generative adversarial model based on a contrastive learning mechanism, and introducing a three-dimensional spatial position constraint loss function to enable the generative adversarial model to learn the spatial distribution law of abnormal features.

[0113] Semi-supervised training framework: Input data: N labeled normal samples (feature vector group + "normal" label), M unlabeled samples (feature vector only); Generator G: multi-layer MLP (3 hidden layers, 256 neurons per layer), input 100-dimensional noise vector, output 16-dimensional feature vector, activation function uses LeakyReLU; Discriminator D: multi-layer MLP (4 hidden layers, 512 neurons per layer), output binary classification probability (normal / abnormal), spectral normalization is introduced to stabilize training.

[0114] Contrastive learning: In the discriminator feature space, the cosine similarity between normal samples and generated samples is calculated, forcing the distance between normal samples to be <0.2 and the distance between abnormal samples and normal samples to be >0.8, thereby increasing the inter-class margin; Spatial constraint loss function: L_pos=Σ|f(x_i)-pos_i|, where f(x_i) is the three-dimensional coordinate component extracted from the feature vector, and pos_i is the corresponding position of the baseline model, forcing the model to learn the spatial distribution pattern of abnormal features (such as solder joint anomalies must appear near the solder pad coordinates).

[0115] Input the measured feature vector, and the discriminator outputs the anomaly score (threshold 0.7), and also outputs the three-dimensional coordinates (analyzed from the feature vector) and type of the abnormal feature (determined by feature dimension mapping, such as "insufficient solder volume" if dimension 3 exceeds the threshold).

[0116] Training can be started with only 50 normal samples, reducing annotation costs by 80% compared to fully supervised models, and achieving a 92% accuracy rate in detecting unseen anomaly types (such as the offset direction of novel components). After introducing spatial constraint loss, the anomaly location error is reduced from ±1mm to ±0.3mm, supporting the precise location of defects in microscope re-inspection or rework robots. Contrastive learning combined with spectral normalization solves the pattern collapse problem of traditional GANs, the generator can cover more than 95% of the normal sample distribution patterns, and the discriminator's generalization ability is improved by 30%.

[0117] In some embodiments, the step of fusing the geometric features, contour features, and surface features to generate a feature vector group includes: performing spatial dimension depth feature fusion on the geometric features, contour features, and surface features based on a preset point cloud convolutional neural network, and outputting a feature vector group containing three-dimensional spatial location information.

[0118] PointCNN architecture: Input: Local point cloud blocks of solder joints (N1 points), components (N2 points), and substrate (N3 points) (point clouds within ±1mm of the corresponding region of the baseline model are extracted respectively); Feature extraction layer: Using X-Conv convolution kernels (supporting irregular point cloud convolution), local features such as 3D coordinates (X,Y,Z), normal vector, and curvature are extracted from each point cloud block, and the depth feature vectors of each region are output (8D for solder joints, 12D for components, and 6D for substrate); Fusion layer: The depth features of each region are concatenated through a fully connected layer, and position encoding (global coordinates of solder joints / components / substrate) is added, and the output is a 26-dimensional feature vector group containing 3D spatial position information.

[0119] Training process: Pre-training: PointCNN is pre-trained on a large-scale PCBA point cloud dataset to learn general 3D geometric feature representations; Fine-tuning: The network is fine-tuned using normal samples from this scenario to optimize the mapping relationship between spatial location information and defect features.

[0120] Traditional manual feature fusion only retains statistical quantities (such as volume deviation), while PointCNN directly learns the spatial distribution features of point clouds (such as the curvature gradient pattern of point clouds around solder joints), capturing subtle defects that are difficult to detect with the naked eye (such as irregular solder joint edges of 0.05mm). By binding feature vectors to three-dimensional coordinates through position encoding, each feature dimension corresponds to a specific physical location (such as the solder joint height at coordinates (15, 20, 0.3mm) corresponding to the 5th dimension of the feature vector), which facilitates subsequent anomaly localization. Compared with manual features, deep fusion features improve the discrimination of defects such as cold solder joints and insufficient pin coplanarity by 40%, and the detection accuracy in small sample scenarios increases from 85% to 93%.

[0121] In some embodiments, multi-scale features are extracted from the 3D point cloud and the reference model respectively: the first layer (coarse-grained) extracts the principal component features of the substrate plane (for global alignment), the second layer (medium-grained) extracts the corner features of the component bounding box (for structural alignment), and the third layer (fine-grained) extracts the curvature features of the solder joint edges (for detail alignment).

[0122] Attention-based weighted registration is employed: registration priority is dynamically allocated based on feature saliency (e.g., the weight of solder joint curvature change point is ×5, and the weight of substrate plane point is ×1), avoiding the dominance of plane regions in the registration process (a major drawback of traditional ICP).

[0123] Introducing a robust kernel function (Huber loss replacing L2 loss) into the ICP iteration reduces sensitivity to outliers (such as welding spatter noise) by 60%, and the root mean square error (RMSE) of registration error decreases from 0.15 mm to 0.08 mm. A dynamic termination criterion is designed: early termination occurs when the rate of change of registration error in the fine-grained feature layer is <0.5%, reducing computation by 30% compared to a fixed number of iterations (e.g., 20 iterations), and the registration time for a single model is <0.8 seconds (compared to 2-3 seconds for traditional ICP).

[0124] The registration accuracy for irregularly shaped components (such as QFN packages with heat sinks) is improved by 50%, solving the local optima problem caused by the single feature in traditional ICP (e.g., alignment deviation at 90° corners is reduced from 0.2mm to 0.06mm). A robust kernel function ensures the registration algorithm remains stable even with 30% noise (traditional ICP fails when noise >15%), adapting to production environments without strict dust removal (misregistration rate reduced from 40% to 5% in dusty environments). Feature pyramid hierarchical processing allows the algorithm to automatically adapt to different detection accuracy requirements (e.g., speed is increased by 2 times in rapid screening mode, and accuracy is improved by 30% in precision detection mode), enhancing flexibility.

[0125] In some embodiments, a three-dimensional position attention module is added to the generator and discriminator: the XYZ coordinate components in the feature vector are weighted so that the model focuses on the abnormal features of the preset ROI region (such as the pad coordinate range of ±0.5mm) and suppresses the interference of large areas of the substrate (the global feature analysis defect of traditional GAN).

[0126] Design a multi-scale spatial loss function: L3D_pos=αLglobal+βLlocal; where Lglobal is the global feature distribution difference, and Llocal is the spatial location constraint of the key region (solder joint / pin) (weight β=0.7), forcing the model to prioritize learning the feature patterns of defect-prone regions.

[0127] The input layer fuses point cloud geometric features (16-dimensional) and AOI visual image features (64-dimensional extracted by CNN). It uses cross-modal contrastive learning to align the feature spaces of the two modalities (e.g., the consistency of feature mapping between point cloud and image for abnormal solder joint height) to solve the problem of incomplete information in single-modal detection (e.g., the point cloud height of a cold solder joint is normal but the image grayscale is abnormal).

[0128] Spatial attention enables the model to achieve a 3D positioning accuracy of 0.2mm × 0.2mm × 0.1mm (traditional GANs can only locate down to the component level), supporting precise operation of automated rework robots (without requiring manual secondary positioning). Combining point cloud 3D information with image texture features improves the detection capability of "hidden defects" (such as solder joints with acceptable height but internal voids) by 60%, reducing the false negative rate from 8% to below 3%. Focusing on feature learning in the ROI region enhances the model's robustness to irrelevant variables such as substrate color differences and component reflectivity changes, reducing retraining time by 50% when migrating across production lines.

[0129] Please see Figure 3 As shown, Figure 3 This is a schematic diagram of the PCBA anomaly detection device 200 based on 3D modeling and AI fusion provided in this application embodiment. The PCBA anomaly detection device 200 is used to execute the steps of the PCBA anomaly detection method based on 3D modeling and AI fusion shown in the above embodiments. The PCBA anomaly detection device 200 can be a single server or a server cluster, or it can be a terminal, such as a handheld terminal, laptop computer, wearable device, or robot.

[0130] like Figure 3 As shown, the PCBA anomaly detection device 200 based on the fusion of 3D modeling and AI includes:

[0131] The data acquisition unit 201 is used to acquire the three-dimensional point cloud data of the PCBA board to be inspected;

[0132] The model generation unit 202 is used to generate a reference three-dimensional digital twin model based on a preset standard PCBA design drawing; the reference three-dimensional digital twin model includes the three-dimensional dimensions of components, the standard height of solder joints, and the flatness parameters of the substrate;

[0133] The registration acquisition unit 203 is used to spatially register the three-dimensional point cloud data and the reference three-dimensional digital twin model based on the iterative nearest point algorithm, and acquire the registered three-dimensional point cloud data.

[0134] The layering processing unit 204 is used to perform layering processing on the registered three-dimensional point cloud data to extract the geometric features of the solder joint area, the contour features of the component area, and the surface features of the substrate area; the geometric features include solder joint volume, height deviation, and three-dimensional curvature; the contour features include component three-dimensional size deviation and pin coplanarity; the surface features include pad warpage and surface roughness; and the geometric features, contour features, and surface features are fused to generate a feature vector group.

[0135] The detection completion unit 205 is used to construct a generative adversarial model based on a preset semi-supervised learning framework and a normal PCBA sample vector group; the feature vector group is input into the generative adversarial model, and the generative adversarial model examines the abnormal vectors, corresponding three-dimensional coordinates and abnormal types in the feature vector group to complete the anomaly detection of the PCBA board.

[0136] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the PCBA anomaly detection device and its modules described above based on the fusion of 3D modeling and AI can be referred to the corresponding processes in the embodiments of the PCBA anomaly detection method based on the fusion of 3D modeling and AI described above, and will not be repeated here.

[0137] The aforementioned PCBA anomaly detection method based on the fusion of 3D modeling and AI can be implemented as a computer program, which can perform tasks such as... Figure 3 It runs on the device shown.

[0138] Please see Figure 4 , Figure 4 This is a schematic block diagram of the control module provided in an embodiment of this application. The control module includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and internal memory.

[0139] The storage medium can store operating devices and computer programs. The computer program includes program instructions that, when executed, cause the processor to perform any PCBA anomaly detection method based on the fusion of 3D modeling and AI.

[0140] The processor provides computing and control capabilities to support the operation of the entire control module.

[0141] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to execute any PCBA anomaly detection method based on the fusion of 3D modeling and AI.

[0142] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application and does not constitute a limitation on the terminal to which the solution of this application is applied. The specific control module may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0143] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0144] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0145] Acquire the 3D point cloud data of the PCBA board to be inspected;

[0146] A baseline 3D digital twin model is generated based on the preset standard PCBA design drawings; the baseline 3D digital twin model includes the 3D dimensions of components, the standard height of solder joints, and the flatness parameters of the substrate;

[0147] The three-dimensional point cloud data and the reference three-dimensional digital twin model are spatially registered based on the iterative nearest point algorithm to obtain the registered three-dimensional point cloud data.

[0148] The registered 3D point cloud data is processed in layers to extract the geometric features of the solder joint area, the contour features of the component area, and the surface features of the substrate area. The geometric features include solder joint volume, height deviation, and 3D curvature. The contour features include component 3D dimensional deviation and pin coplanarity. The surface features include pad warpage and surface roughness. The geometric features, contour features, and surface features are fused to generate a feature vector group.

[0149] A generative adversarial model is constructed based on a pre-defined semi-supervised learning framework and a normal PCBA sample vector set. The feature vector set is input into the generative adversarial model, which examines the abnormal vectors, their corresponding three-dimensional coordinates, and the abnormality type in the feature vector set to complete the anomaly detection of the PCBA board.

[0150] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the PCBA anomaly detection method based on 3D modeling and AI fusion as provided in any embodiment of this application.

[0151] The computer-readable storage medium can be an internal storage unit of the control module described in the foregoing embodiments, such as the hard disk or memory of the control module. Alternatively, the computer-readable storage medium can be an external storage device of the control module, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the control module.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A PCBA anomaly detection method based on the fusion of 3D modeling and AI, characterized in that, include: Acquire the 3D point cloud data of the PCBA board to be inspected; Generate a baseline 3D digital twin model based on the preset standard PCBA design drawings; The reference three-dimensional digital twin model includes the three-dimensional dimensions of the components, the standard height of the solder joints, and the flatness parameters of the substrate; The three-dimensional point cloud data and the reference three-dimensional digital twin model are spatially registered based on the iterative nearest point algorithm to obtain the registered three-dimensional point cloud data. The registered 3D point cloud data is processed in layers to extract the geometric features of the solder joint area, the contour features of the component area, and the surface features of the substrate area. The geometric features include solder joint volume, height deviation, and 3D curvature. The contour features include component 3D dimensional deviation and pin coplanarity. The surface features include pad warpage and surface roughness. The geometric features, contour features, and surface features are fused to generate a feature vector group. A generative adversarial model is constructed based on a pre-defined semi-supervised learning framework and a normal PCBA sample vector set. The feature vector set is input into the generative adversarial model, which examines the abnormal vectors, their corresponding three-dimensional coordinates, and the abnormality type in the feature vector set to complete the anomaly detection of the PCBA board.

2. The method according to claim 1, characterized in that, The process of performing layered processing on the registered 3D point cloud data to extract the geometric features of the solder joint area, the contour features of the component area, and the surface features of the substrate area includes: The three-dimensional point cloud data is initially segmented based on point cloud density threshold and normal vector continuity detection to obtain substrate region, component region and solder joint region. For the solder joint area, noise points are removed by local coordinate system reconstruction and three-dimensional morphological filtering. Independent solder joint point cloud clusters are extracted based on spatial distance clustering algorithm. The volume, height deviation and three-dimensional curvature of the solder joint corresponding to the independent solder joint point cloud clusters are calculated by Gaussian curvature estimation and local height histogram analysis. For the component region, an edge feature enhancement algorithm is used to extract the component contour point cloud. Contour matching is performed by the three-dimensional bounding box constraint of the corresponding component in the benchmark model. The three-dimensional size deviation is calculated, and the coplanarity of the pin is obtained by plane fitting residual analysis of the pin point cloud. For the substrate region, a reference plane is established by polynomial surface fitting, the vertical distance distribution between the three-dimensional point cloud data and the fitting plane is calculated, and the pad warpage and surface roughness are extracted based on the vertical distance distribution and the frequency domain analysis of the surface texture point cloud.

3. The method according to claim 1, characterized in that, The process of generating a baseline 3D digital twin model based on preset standard PCBA design drawings includes: The standard PCBA design drawings are parsed in multiple formats to extract the three-dimensional geometric parameters, solder joint position coordinates and substrate stack-up structure data from the corresponding component package library of the standard PCBA design drawings, and to establish a parameterized three-dimensional model of the component. For the standard height of the weld joint, based on the mapping relationship between the volume and height of the solder in the preset welding process parameter library, a morphological simulation based on a fluid dynamics model is performed on each type of weld joint to generate a three-dimensional model of the weld joint with process characteristics. For the substrate flatness parameters, the laminate material properties, copper foil distribution and via location data in the substrate design drawings corresponding to the standard PCBA design drawings are imported into the finite element analysis module to simulate the deformation trend of the substrate under standard manufacturing process and generate a three-dimensional reference surface of the substrate containing allowable deformation tolerance. The component 3D model, the solder joint 3D model, and the substrate 3D reference surface are assembled to form the reference 3D digital twin model, which includes geometric parameters, process features, and tolerance range.

4. The method according to claim 1, characterized in that, The spatial registration of the 3D point cloud data and the baseline 3D digital twin model based on the iterative nearest point algorithm includes: The three-dimensional point cloud data is preprocessed by voxel mesh downsampling and outlier removal, and the benchmark three-dimensional digital twin model is triangulated and a kd-tree spatial index is established. In the coarse registration stage, the initial rotation and translation matrix between the 3D point cloud data and the reference 3D digital twin model is obtained through principal component analysis; the initial registration parameters are optimized according to the curvature value weights corresponding to the initial rotation and translation matrix. During the fine registration iteration process, a bidirectional nearest point matching strategy is adopted. The vertical distance and tangential distance between the three-dimensional point cloud data and the reference three-dimensional digital twin model are calculated as error functions. A dynamic relaxation factor is introduced to adjust the iteration step size. When the rate of change of registration error between two adjacent iterations is less than a preset threshold, the iteration is terminated according to the consistency check of the point cloud normal vector. The spatial registration results, including the rotation matrix, translation vector, and registration accuracy evaluation parameters, are obtained to complete the spatial registration.

5. The method according to claim 1, characterized in that, The acquisition of the 3D point cloud data of the PCBA board to be inspected includes: The PCBA is scanned from multiple angles using structured light scanning or a laser linear array camera to obtain the three-dimensional point cloud data, including solder joint height, component three-dimensional contours, and substrate surface morphology. The three-dimensional point cloud data is subjected to noise reduction filtering and coordinate system one.

6. The method according to claim 1, characterized in that, The generative adversarial model constructed based on a pre-defined semi-supervised learning framework and normal PCBA sample vector sets includes: The generator and discriminator of the generative adversarial model are trained based on the semi-supervised learning framework and the normal PCBA sample vector set. Based on the contrastive learning mechanism, the inter-class distance between normal and abnormal samples is expanded in the feature space of the generative adversarial model, and a three-dimensional spatial position constraint loss function is introduced to enable the generative adversarial model to learn the spatial distribution law of abnormal features.

7. The method according to claim 1, characterized in that, The step of fusing the geometric features, contour features, and surface features to generate a feature vector set includes: Based on a preset point cloud convolutional neural network, the geometric features, contour features and surface features are fused with spatial dimensions to output a feature vector group containing three-dimensional spatial location information.

8. A PCBA anomaly detection device based on the fusion of 3D modeling and AI, characterized in that, include: The data acquisition unit is used to acquire the three-dimensional point cloud data of the PCBA board to be inspected; The model generation unit is used to generate a reference three-dimensional digital twin model based on a preset standard PCBA design drawing; the reference three-dimensional digital twin model includes the three-dimensional dimensions of components, the standard height of solder joints, and the flatness parameters of the substrate; The registration acquisition unit is used to spatially register the three-dimensional point cloud data and the reference three-dimensional digital twin model based on the iterative nearest point algorithm, and acquire the registered three-dimensional point cloud data. A layered processing unit is used to perform layered processing on the registered 3D point cloud data to extract geometric features of the solder joint area, contour features of the component area, and surface features of the substrate area. The geometric features include solder joint volume, height deviation, and 3D curvature. The contour features include component 3D dimensional deviation and pin coplanarity. The surface features include pad warpage and surface roughness. The geometric features, contour features, and surface features are fused to generate a feature vector group. The detection completion unit is used to construct a generative adversarial model based on a preset semi-supervised learning framework and a normal PCBA sample vector group; the feature vector group is input into the generative adversarial model, and the generative adversarial model examines the abnormal vectors, corresponding three-dimensional coordinates and abnormal types in the feature vector group to complete the anomaly detection of the PCBA board.

9. A PCBA anomaly detection system based on the fusion of 3D modeling and AI, characterized in that, include: PCBA board to be tested; A structured light scanning camera or a laser linear array camera is used to scan the PCBA board to be inspected from multiple angles. A control module includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 7.

Images