An adaptive optical detection system and method for special-shaped PCBA
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHONGXIAN OPTOELECTRONICS TECH (ZHEJIANG) CO LTD
- Filing Date
- 2026-04-10
- Publication Date
- 2026-06-09
Smart Images
Figure CN122171574A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of circuit board inspection technology, and more specifically, to an adaptive optical inspection system and method for irregularly shaped PCBAs. Background Technology
[0002] With the trend towards miniaturization and increasing complexity of electronic products, irregularly shaped circuit boards (PCBs) are widely used in various devices due to their irregular shapes and varied structures. The accuracy of their inspection directly impacts the reliability and safety of these products. However, current technologies often struggle to adapt to the variable shapes of irregularly shaped PCBs. Many methods cannot flexibly handle curved or irregular edges during inspection, leading to deviations in the judgment of component position and orientation. Especially when component mounting postures are diverse, existing solutions often lack dynamic perception of changes in the board shape, failing to accurately identify the specific state of components and thus affecting the reliability of inspection results. The main reason is that changes in the edge contour of irregularly shaped PCBs directly interfere with the reference judgment of the inspection equipment. In other words, when the board is curved or has a multi-layered structure, the inspection system struggles to determine a stable reference plane, leading to errors in component orientation identification and resulting in misjudgments of whether the component is correctly mounted. Summary of the Invention
[0003] In order to overcome the shortcomings of the existing technology, the present invention provides an adaptive optical inspection system and method for irregularly shaped PCBAs, aiming to solve the problems in the existing technology.
[0004] The technical solution adopted by this invention to solve its technical problem is: an adaptive optical inspection system for irregularly shaped PCBAs, comprising: The preliminary processing module is used to acquire the three-dimensional point cloud data of the irregularly shaped circuit board to be inspected through laser scanning, calculate the surface normal vector, and obtain the surface normal vector field of the circuit board. The sub-region extraction module is used to extract the contour of the surface normal vector field to obtain multiple sub-regions; The reference plane determination module is used to determine the local stable reference plane corresponding to each sub-region; The component category and location determination module is used to identify the components on the irregular circuit board and obtain the category of each component and its three-dimensional coordinates in the coordinate system of the three-dimensional point cloud data based on the three-dimensional point cloud data of the irregular circuit board. The component tilt angle determination module is used to obtain the three-dimensional coordinates of the component in each sub-region, fit the plane of the component based on the three-dimensional coordinates and calculate its normal vector; it is used to calculate the angle between the normal vector of the component's plane and the normal vector of the local stable reference plane of the corresponding sub-region, which is used as the tilt angle of the component relative to the local stable reference plane. The output module is used to determine a preset standard tilt threshold based on the type of the component, compare the tilt angle of the component relative to the local stable reference plane with the preset standard tilt threshold, and output the detection result.
[0005] Preferably, in the preliminary processing module, for the three-dimensional point cloud data, the covariance matrix is calculated by principal component analysis in a fixed radius neighborhood around each point, and the eigenvector corresponding to the smallest eigenvalue in the covariance matrix is used as the surface normal vector of that point. The surface normal vectors of all points are combined to form the surface normal vector field of the circuit board.
[0006] Optionally, in the preliminary processing module, before calculating the surface normal vector, a coordinate set of the original point cloud data is generated based on the obtained three-dimensional point cloud data of the irregular circuit board to be detected, and the number of local neighborhood points of each point is calculated based on the coordinate set to generate a point cloud density distribution map. For the point cloud density distribution map, a statistical filtering method is used to remove noise points with significantly low density, resulting in filtered 3D point cloud data.
[0007] Specifically, in the sub-region extraction module, for the surface normal vector field, the angle between the normal vector of each point and the normal vector of the adjacent points in the spatial neighborhood is calculated, the angle difference is obtained, and the points with the angle difference greater than a preset threshold are marked as candidate edge points; Density clustering is performed on candidate edge points to obtain multiple high-density point groups, and each high-density point group is used as the initial point set of a contour edge line. Points with curvature greater than a preset curvature threshold are extracted from the initial point set of each contour edge line as edge feature points; The edge feature points on each contour edge line are arranged in the order of the original 3D point cloud data. The Euclidean distance and the angle between the direction vectors of two adjacent edge feature points are calculated in turn. If the distance between a pair of adjacent edge feature points is less than a preset distance threshold and the angle between the direction vectors is less than a preset angle threshold, the pair of points is retained. Otherwise, the latter point is removed to obtain the refined edge feature point sequence of each contour edge line. Using a refined sequence of edge feature points as a boundary, the 3D point cloud data reflecting the surface of the irregularly shaped circuit board is divided into multiple closed sub-regions.
[0008] It is worth noting that in the reference plane determination module, point cloud data of all points in each sub-region are obtained to form the sampling point set corresponding to each sub-region; The average normal vector of the surface normal vectors of all points in the same sampling point set is calculated as the normal vector of the sub-region. Based on the direction of the normal vector of the sub-region, a local plane with the same direction is fitted as the local stable reference plane corresponding to the sub-region, thus obtaining the local stable reference plane of each sub-region.
[0009] Specifically, in the component category and position determination module, the intrinsic parameter matrix K and extrinsic parameter matrix of the virtual camera are preset, wherein the intrinsic parameter matrix K includes the focal length fx, fy and the principal point coordinates (cx, cy), and the extrinsic parameter matrix is composed of the rotation matrix R and the translation vector t; The coordinates of each point P in the 3D point cloud data acquired from the irregularly shaped circuit board are transformed to obtain the coordinates Pc (Pc.x, Pc.y, Pc.z) in the camera coordinate system, where Pc is calculated as follows: Perform perspective projection on coordinate Pc to obtain normalized coordinates (xn, yn), where , ; The pixel coordinates (u, v) are calculated using the intrinsic parameter matrix K, where , Map the depth value Pc.z of each point to a range of 0-255 as a grayscale value; generate a first grayscale depth image based on the pixel coordinates of all points and the grayscale value of each point; obtain a set of pixels with grayscale values greater than a preset grayscale threshold from the first grayscale depth image; The connectedComponents function is used to perform connected component labeling on the pixel set to obtain multiple connected regions; the boundingRect function is used to calculate the bounding rectangle of each connected region and the corresponding rectangular sub-image is cropped to obtain the pixel coordinates of all points in the bounding box of each rectangular sub-image; the YOLOv8 network is input for each rectangular sub-image to obtain the component category label; Get the depth values corresponding to the pixel coordinates of all points in the bounding box, and take the median as the unified depth value Zd of the bounding box; By combining the intrinsic parameter matrix K, the pixel coordinates (uc, vc) of the points in the bounding box, and the depth value Zd, the 3D coordinates Pc1 (Pc1.x, Pc1.y, Pc1.z) of the points in the bounding box in the camera coordinate system are calculated using the back projection formula. , , ; The 3D coordinates P1 of the points in the bounding box in the coordinate system of the 3D point cloud data are calculated by inverse extrinsic transformation. , R is the inverse of the rotation matrix R, and T represents the matrix transpose; the combination of the three-dimensional coordinates of all points of the bounding box in the coordinate system of the three-dimensional point cloud data is used as the three-dimensional coordinates of the element in the coordinate system of the three-dimensional point cloud data.
[0010] It is worth noting that in the component tilt angle determination module, based on the three-dimensional coordinates of each component in the coordinate system of the three-dimensional point cloud data and the coordinates of the refined edge feature points used to reflect the boundary of the sub-region, the point-in-polygon test method is used to determine whether each component is located in the sub-region, and the three-dimensional coordinates of the components in each sub-region are obtained from this, thus obtaining the component coordinate set corresponding to each sub-region. The coordinate set of the component is fitted by least squares to obtain the component plane, where the plane equation is ax + by + cz + d = 0, a, b, c, and d are fitting coefficients, and the normal vector of the component plane is n1 = (a, b, c). Obtain the normal vector n2 of the local stable reference plane of the sub-region; Calculate the inner product of normal vectors n1 and n2, then calculate the magnitudes of normal vectors n1 and n2 respectively. Divide the inner product by the product of the magnitudes of normal vectors n1 and n2 to obtain the cosine value. Then calculate the inverse cosine value to obtain the angle between normal vectors n1 and n2 as the tilt angle of the component.
[0011] Preferably, in the output module, a preset set of thresholds is queried from the SQLite database according to the category of the component to determine the standard tilt threshold corresponding to the current component; If the tilt angle of a component exceeds the standard tilt threshold, it is judged as abnormal; otherwise, it is normal, and the detection result is output.
[0012] An adaptive optics inspection method using the aforementioned adaptive optics inspection system for irregularly shaped PCBAs.
[0013] The beneficial effects of this invention are as follows: In the adaptive optical inspection system for irregularly shaped PCBAs, three-dimensional point cloud data of the irregularly shaped circuit board is acquired by laser scanning, and the surface normal vector field is calculated. Contour extraction is performed on this normal vector field to divide it into multiple sub-regions with local surface features, and a local stable reference plane is determined for each sub-region. Simultaneously, the categories of each component on the circuit board and their spatial positions in the three-dimensional coordinate system are calibrated. Then, based on the three-dimensional coordinates of the components in each sub-region, the plane where the component is located is fitted, and its normal vector is calculated. The angle between the component plane normal vector and the normal vector of the corresponding sub-region's local stable reference plane is calculated, thereby accurately obtaining the true tilt angle of each component relative to its local mounting reference plane. Finally, based on preset standard tilt thresholds for different component categories, the calculated tilt angle is compared and judged, and the detection result indicating whether the component is abnormal is output. This solution effectively solves the problem of inaccurate component tilt detection caused by the difficulty in unifying the overall reference and local surface undulations in irregularly shaped circuit boards, achieving high-precision and highly adaptable automated tilt defect detection, significantly improving the reliability and efficiency of quality inspection of complex circuit boards. Attached Figure Description
[0014] Figure 1 This is a system block diagram of an adaptive optics inspection system for irregularly shaped PCBAs.
[0015] Figure 2 This is a flowchart of an adaptive optics inspection method for irregularly shaped PCBAs.
[0016] Figure 3 This is a schematic diagram of the least squares method for fitting a plane.
[0017] Figure 4 This is a schematic diagram of candidate edge point clustering using the DBSCAN algorithm. Detailed Implementation
[0018] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that these descriptions are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0019] Combination Figures 1 to 4 An adaptive optical inspection system for irregularly shaped PCBAs, as shown, includes: The preliminary processing module is used to acquire the three-dimensional point cloud data of the irregularly shaped circuit board to be inspected through laser scanning, calculate the surface normal vector, and obtain the surface normal vector field of the circuit board. The sub-region extraction module is used to extract the contour of the surface normal vector field to obtain multiple sub-regions; The reference plane determination module is used to determine the local stable reference plane corresponding to each sub-region; The component category and location determination module is used to identify the components on the irregular circuit board and obtain the category of each component and its three-dimensional coordinates in the coordinate system of the three-dimensional point cloud data based on the three-dimensional point cloud data of the irregular circuit board. The component tilt angle determination module is used to obtain the three-dimensional coordinates of the component in each sub-region, fit the plane of the component based on the three-dimensional coordinates and calculate its normal vector; it is used to calculate the angle between the normal vector of the component's plane and the normal vector of the local stable reference plane of the corresponding sub-region, which is used as the tilt angle of the component relative to the local stable reference plane. The output module is used to determine a preset standard tilt threshold based on the type of the component, compare the tilt angle of the component relative to the local stable reference plane with the preset standard tilt threshold, and output the detection result.
[0020] In the adaptive optical inspection system for irregularly shaped PCBAs, three-dimensional point cloud data of the irregularly shaped circuit board is acquired by laser scanning, and the surface normal vector field is calculated. Contour extraction is performed on this normal vector field to divide it into multiple sub-regions with local surface features, and a local stable reference plane is determined for each sub-region. Simultaneously, the categories of each component on the circuit board and their spatial positions in the three-dimensional coordinate system are calibrated. Then, based on the three-dimensional coordinates of the components in each sub-region, the plane containing the component is fitted, and its normal vector is calculated. The angle between the component plane normal vector and the normal vector of the corresponding sub-region's local stable reference plane is calculated, thereby accurately obtaining the true tilt angle of each component relative to its local mounting reference plane. Finally, based on preset standard tilt thresholds for different component categories, the calculated tilt angle is compared and judged, and the detection result indicating whether the component is abnormal is output. This solution effectively solves the problem of inaccurate component tilt detection caused by the difficulty in unifying the overall reference and local surface undulations in irregularly shaped circuit boards. It achieves high-precision, highly adaptable, automated tilt defect detection, significantly improving the reliability and efficiency of quality inspection of complex circuit boards.
[0021] It is worth noting that in the preliminary processing module, for the three-dimensional point cloud data, the covariance matrix is calculated by principal component analysis in the neighborhood of a fixed radius around each point. The eigenvector corresponding to the smallest eigenvalue in the covariance matrix is used as the surface normal vector of that point, and the surface normal vectors of all points are combined to form the surface normal vector field of the circuit board.
[0022] When calculating the covariance matrix using principal component analysis within a fixed-radius neighborhood around each point, a set of points with a radius of approximately 1 millimeter is first selected. The mean coordinates of these points are calculated, and then the covariance matrix is constructed. This matrix captures the variance and covariance distribution of the point set in three dimensions. By performing eigenvalue decomposition on the matrix, three eigenvalues and corresponding eigenvectors are obtained. The smallest eigenvalue represents the minimum variation of the point set along the surface normal, and its eigenvector naturally serves as the surface normal vector for that point.
[0023] In one possible implementation, the surface normal vectors of all points are combined into a surface normal vector field for the circuit board, which can then be visualized in a software environment such as PointCloudLibrary, with each normal vector pointing outwards as an arrow. This normal vector field can reveal curved or irregular edges on the board surface, while a high degree of consistency in the direction of the normal vectors indicates a smooth surface.
[0024] Preferably, in the preliminary processing module, before calculating the surface normal vector, a coordinate set of the original point cloud data is generated based on the obtained three-dimensional point cloud data of the irregular circuit board to be detected, and the number of local neighborhood points of each point is calculated based on the coordinate set to generate a point cloud density distribution map. For the point cloud density distribution map, a statistical filtering method is used to remove noise points with significantly low density, resulting in filtered 3D point cloud data.
[0025] In actual circuit board inspection scenarios, the first step is to collect data from irregularly shaped circuit boards using a laser scanner. This type of equipment typically uses the principle of triangulation, projecting a laser beam onto the board surface and capturing the reflected light to obtain three-dimensional point cloud data.
[0026] In one possible implementation, when calculating the number of local neighboring points for each point based on these coordinate sets, a fixed radius, such as 0.5 mm, can be chosen as the neighborhood range. The points around each point are counted, generating a point cloud density distribution map in this way. Specifically, when using statistical filtering to remove noise points from the point cloud density distribution map, the standard deviation of the entire point cloud is first calculated. Assuming an average density of 20 points per cubic millimeter, if the neighborhood density of a point is lower than the standard deviation, it is considered noise and removed, resulting in filtered 3D point cloud data. This improves the purity of the data, thus avoiding error accumulation in subsequent steps and achieving more accurate surface modeling.
[0027] Optionally, in the sub-region extraction module, for the surface normal vector field, the angle between the normal vector of each point and the normal vector of the adjacent points in the spatial neighborhood is calculated, the angle difference is obtained, and the points with the angle difference greater than a preset threshold are marked as candidate edge points; Density clustering is performed on candidate edge points to obtain multiple high-density point groups, and each high-density point group is used as the initial point set of a contour edge line. Points with curvature greater than a preset curvature threshold are extracted from the initial point set of each contour edge line as edge feature points; The edge feature points on each contour edge line are arranged in the order of the original 3D point cloud data. The Euclidean distance and the angle between the direction vectors of two adjacent edge feature points are calculated in turn. If the distance between a pair of adjacent edge feature points is less than a preset distance threshold and the angle between the direction vectors is less than a preset angle threshold, the pair of points is retained. Otherwise, the latter point is removed to obtain the refined edge feature point sequence of each contour edge line. Using a refined sequence of edge feature points as a boundary, the 3D point cloud data reflecting the surface of the irregularly shaped circuit board is divided into multiple closed sub-regions.
[0028] In one possible implementation, for the surface normal vector field, it is first necessary to calculate the angle between the normal vector of each point and the normal vector of its neighboring points in the spatial neighborhood. This calculation is based on the vector dot product formula to obtain the angle value. For example, select points within a neighborhood radius of 0.5 mm, and calculate the angle for each center point by traversing the neighborhood points. If the angle difference exceeds a preset threshold, such as 30 degrees, then mark the point as a candidate edge point. This can initially identify areas where the surface normal vector changes drastically. These areas often correspond to the geometric edges or turning points of the circuit board. The candidate point set generated by this method provides basic data support for subsequent edge extraction.
[0029] Specifically, these candidate edge points are then subjected to density clustering. This density clustering can employ the DBSCAN algorithm, which identifies high-density clusters by setting a minimum number of points and a search radius. Figure 4 As shown, for example, if the minimum number of points is set to 5 and the search radius is 0.5 mm, the algorithm will scan all candidate points, find the core points and expand the clusters, and finally obtain multiple high-density point groups. Each point group represents the initial point set of a potential contour edge line. This clustering helps to organize scattered candidate points into a coherent structure and avoid the interference of isolated points.
[0030] In one possible implementation, curvature can be estimated by fitting a local quadratic surface to a point and its neighboring points. For example, the curvature of the neighboring points of each point in the initial point set can be calculated. If the curvature exceeds 0.1, it is considered an edge feature point. These points are usually located at the curved parts of the edge and can highlight the geometric characteristics of the contour, thus laying the foundation for key nodes in edge refinement.
[0031] Specifically, the edge feature points on each contour edge line are arranged in the order of the original 3D point cloud data. Then, the Euclidean distance and the angle between the direction vectors of two adjacent edge feature points are calculated in turn. Here, the Euclidean distance is obtained by taking the square root of the sum of the squares of the coordinate differences, and the angle between the direction vectors is obtained by taking the dot product. For example, if the Euclidean distance between a pair of points is less than 0.2 mm and the angle between the direction vectors is less than 15 degrees, then the pair is retained; otherwise, the pair is discarded. By iterating through this process, a refined edge feature point sequence is obtained. This refinement can remove redundant points and make the sequence more concise and efficient.
[0032] In one possible implementation, these refined edge feature point sequences are used as boundaries to divide the 3D point cloud data reflecting the surface of the irregular circuit board into multiple closed sub-regions. This division is similar to a region growing algorithm, which expands and fills the point cloud from the boundaries. For example, the boundaries divide the point cloud into sub-regions such as the motherboard area and the connector area, ensuring that the point cloud inside each region is connected and closed.
[0033] Specifically, in the reference plane determination module, point cloud data of all points in each sub-region are acquired to form a sampling point set corresponding to each sub-region; The average normal vector of the surface normal vectors of all points in the same sampling point set is calculated as the normal vector of the sub-region. Based on the direction of the normal vector of the sub-region, a local plane with the same direction is fitted as the local stable reference plane corresponding to the sub-region, thus obtaining the local stable reference plane of each sub-region.
[0034] In one possible implementation, after acquiring the point cloud data of all points in each sub-region, these points are first grouped and organized according to their respective sub-regions to form independent sampling point sets. Each sub-region corresponds to a complete sampling point set, and the point cloud in the sampling point set retains the original spatial coordinates and surface normal vector information.
[0035] Specifically, when calculating the average surface normal vector for each set of sampling points, a common approach is to sum the vectors and then normalize the result. It should be noted that the surface normal vector itself is a unit vector, typically using... Let n be the surface normal vector. Therefore, we can directly perform an arithmetic mean on each component of all surface normal vectors to obtain the averaged component. Then, we can perform length normalization on the averaged component to obtain the final component. Finally, we can form an average normal vector based on the three final components to obtain the representative normal vector direction of the sub-region.
[0036] In one embodiment, after obtaining the average normal vector of the sub-region, this average normal vector can be used as the normal reference to directly construct a locally stable reference plane with consistent orientation. Specifically, an arbitrary point within the sub-region is selected as the reference point on the plane, and then combined with the average normal vector as the plane's normal, a unique mathematical plane equation can be determined. In practical applications, to improve the representativeness and stability of the plane, the geometric center point of the sub-region is usually preferred as the reference point. It is understandable that there are often significant attitude differences between the locally stable reference planes of different sub-regions.
[0037] It is worth noting that in the component category and position determination module, the intrinsic parameter matrix K and extrinsic parameter matrix of the virtual camera are preset. The intrinsic parameter matrix K includes the focal length fx, fy and the principal point coordinates (cx, cy), and the extrinsic parameter matrix is composed of the rotation matrix R and the translation vector t. The coordinates of each point P in the 3D point cloud data acquired from the irregularly shaped circuit board are transformed to obtain the coordinates Pc (Pc.x, Pc.y, Pc.z) in the camera coordinate system, where Pc is calculated as follows: Perform perspective projection on coordinate Pc to obtain normalized coordinates (xn, yn), where , ; The pixel coordinates (u, v) are calculated using the intrinsic parameter matrix K, where , Map the depth value Pc.z of each point to a range of 0-255 as a grayscale value; generate a first grayscale depth image based on the pixel coordinates of all points and the grayscale value of each point; obtain a set of pixels with grayscale values greater than a preset grayscale threshold from the first grayscale depth image; The connectedComponents function is used to perform connected component labeling on the pixel set to obtain multiple connected regions; the boundingRect function is used to calculate the bounding rectangle of each connected region and the corresponding rectangular sub-image is cropped to obtain the pixel coordinates of all points in the bounding box of each rectangular sub-image; the YOLOv8 network is input for each rectangular sub-image to obtain the component category label; Get the depth values corresponding to the pixel coordinates of all points in the bounding box, and take the median as the unified depth value Zd of the bounding box; By combining the intrinsic parameter matrix K, the pixel coordinates (uc, vc) of the points in the bounding box, and the depth value Zd, the 3D coordinates Pc1 (Pc1.x, Pc1.y, Pc1.z) of the points in the bounding box in the camera coordinate system are calculated using the back projection formula. , , ; The 3D coordinates P1 of the points in the bounding box in the coordinate system of the 3D point cloud data are calculated by inverse extrinsic transformation. , R is the inverse of the rotation matrix R, and T represents the matrix transpose; the combination of the three-dimensional coordinates of all points of the bounding box in the coordinate system of the three-dimensional point cloud data is used as the three-dimensional coordinates of the element in the coordinate system of the three-dimensional point cloud data.
[0038] In one possible implementation, when pre-setting the intrinsic and extrinsic parameter matrices of the virtual camera, the actual acquisition environment of the irregularly shaped circuit board needs to be considered. For example, when processing an irregularly shaped circuit board used in an electronic control unit, the intrinsic parameter matrix K is configured according to the focal length and principal point coordinates of the simulated camera to ensure that the projection process simulates real imaging. Specifically, the focal lengths fx and fy are usually set to values that match the point cloud resolution, while the principal point coordinates cx and cy are placed at the pixel position at the center of the image. This avoids the impact of projection distortion on subsequent detection. The extrinsic parameter matrix, composed of a rotation matrix R and a translation vector t, is used to align the point cloud data to the virtual camera's viewpoint. For example, adjusting R rotates the circuit board so that its front faces the camera, thus facilitating the generation of a clear depth image.
[0039] In one possible implementation, when the coordinates of each point P in the 3D point cloud data are transformed to obtain Pc in the camera coordinate system, this transformation ensures an accurate mapping of the point cloud from world coordinates to camera coordinates. Specifically, the calculation of Pc involves matrix multiplication and translation addition, which helps to unify point cloud data from different acquisition angles. Performing perspective projection on Pc to obtain normalized coordinates xn and yn simulates the perspective effect of camera imaging. For example, when processing densely packed component areas on a circuit board, this projection can compress 3D points into a 2D plane, highlighting depth differences. Specifically, this normalization method allows the normalized coordinates to capture the proportional relationships under the viewpoint, avoiding size distortion of objects at different distances.
[0040] The step of calculating pixel coordinates u and v using the intrinsic parameter matrix K converts the normalized coordinates into image pixel positions. The calculation of u and v ensures the precise positioning of components such as resistors and capacitors in the image. Specifically, by combining the multiplication and addition of parameters such as fx, fy, cx, and cy, a pixel distribution similar to that captured by an actual camera can be generated. In one possible implementation, mapping the depth value Pc.z of each point to a grayscale value in the range of 0-255 highlights the depth gradient, thus forming a visual depth map. Specifically, in circuit board inspection, this grayscale representation facilitates the differentiation of stacked structures. When obtaining the set of pixels with grayscale values greater than a preset grayscale threshold from the first grayscale depth image, this step filters background noise. For example, setting the threshold to 100 can extract pixel groups that highlight components. This set represents potential component regions, avoiding interference from irrelevant points.
[0041] In one possible implementation, OpenCV's `connectedComponents` function is used to label multiple connected regions on a set of pixels, leveraging the connectivity principles of image processing, such as marking separate resistor and chip areas on a circuit board. Specifically, the `connectedComponents` function checks pixel neighborhoods to group pixels, forming independent component labels, which helps separate densely packed components. When OpenCV's `boundingRect` function is used to calculate the bounding rectangle for each connected region and crop the corresponding rectangular sub-image, this cropping focuses on local areas, generating independent rectangular sub-images for each component, facilitating fine-grained detection. Specifically, `boundingRect` calculates the bounding box pixel coordinates of the rectangular sub-image, ensuring that the sub-image contains the complete component without redundant background. When each rectangular sub-image is input into the YOLOv8 network to obtain component category labels and bounding box pixel coordinates, YOLOv8, as an object detection model, can identify component types in real time, such as classifying rectangular objects in the rectangular sub-image as capacitors or resistors. Specifically, the YOLOv8 network extracts features and predicts bounding boxes through convolutional layers; this detection improves automation efficiency in circuit board assembly verification.
[0042] In one possible implementation, using the median as the uniform depth value Zd for the bounding box can resist the effects of noise, such as dust on the circuit board, and still obtain a stable depth representation. Specifically, this uniform depth simplifies the processing of multi-point depths and ensures the accuracy of backprojection.
[0043] When calculating the 3D coordinates Pc1 of points within the bounding box in the camera coordinate system using the backprojection formula, the backprojection recovers depth information, for example, extending the points within the bounding box back into 3D space. This method achieves a reverse mapping from image to 3D in circuit board dimension measurement.
[0044] When calculating the 3D coordinates P1 of the bounding box corner points in the coordinate system of the 3D point cloud data using the inverse extrinsic transformation, the original coordinates are recovered using the transpose of R and the subtraction of t. This inverse transformation ensures the consistency of the coordinates, facilitating subsequent comparison of the element's position coordinates with the local stable reference plane in the same coordinate system. Combining the 3D coordinates of all points in the bounding box in the coordinate system of the 3D point cloud data forms the spatial description of the element in the coordinate system of the 3D point cloud data.
[0045] Specifically, in the component tilt angle determination module, based on the three-dimensional coordinates of each component in the coordinate system of the three-dimensional point cloud data and the coordinates of the refined edge feature points used to reflect the boundaries of the sub-regions, the Point-in-Polygon Test method is used to determine whether each component is located within a sub-region, thereby obtaining the three-dimensional coordinates of the components within each sub-region and obtaining the component coordinate set corresponding to each sub-region; specifically, the polygon in-polygon test method can be the Raycasting method. like Figure 3 As shown, for the coordinate set of the component, the least squares method is used to fit the coordinate set of NumPy using the linalg.lstsq tool to obtain the component plane, where the plane equation is ax + by + cz + d = 0, a, b, c, and d are fitting coefficients, and the normal vector of the component plane is n1 = (a, b, c). Obtain the normal vector n2 of the local stable reference plane of the sub-region; The dot function of NumPy is used to calculate the dot product of normal vectors n1 and n2. The linalg.norm function of NumPy is used to calculate the magnitudes of normal vectors n1 and n2 respectively. The dot product is divided by the product of the magnitudes of normal vectors n1 and n2 to obtain the cosine value. The math.acos function is used to calculate the arccosine value. The angle between normal vectors n1 and n2 is used as the tilt angle of the component.
[0046] In one possible implementation, a point-within-polygon test method is used to determine whether each element is located within a sub-region. For example, raycasting involves projecting a ray from a test point in one direction and calculating the number of intersections between the ray and the polygon boundary. If the number of intersections is odd, the point is within the sub-region; otherwise, it is outside the sub-region. Specifically, the coordinates of the boundary feature points of the sub-region are first obtained. These points form a closed polygon. Then, raycasting is applied to multiple 3D coordinate points of each element one by one to check whether they fall within the polygon, thereby filtering out the 3D coordinates of elements located within the sub-region and forming a set of element coordinates.
[0047] In one possible implementation, the least squares method is used to fit the component plane to the obtained component coordinate set. For example, a matrix A is constructed using the x_i, y_i, and z_i values of the component's three-dimensional coordinates, where each row of A is [x_i, y_i, 1] and the target vector is -z_i. Then, the coefficients a, b, c, and d of ax + by + cz + d = 0 are solved to minimize the fitting error. This can accurately describe the planar orientation of a component, and the normal vector n1 is (a, b, c), which provides a mathematical basis for tilt detection.
[0048] It should be noted that the normal vector of the sub-region is used as the normal vector n2 of the local stable reference plane of the sub-region. When calculating the angle between n1 and n2, the inner product is first calculated as the result of the vector dot product. Then, the Euclidean magnitude of each vector is calculated separately. The cosine value is equal to the inner product divided by the product of the magnitudes, and the inverse cosine function outputs the angle value. This calculation chain, from vector operations to angle conversion, ensures the accuracy of quantitative evaluation and supports automated correction processes in business operations.
[0049] Preferably, in the output module, a preset set of thresholds is queried from the SQLite database according to the category of the component to determine the standard tilt threshold corresponding to the current component; If the tilt angle of a component exceeds the standard tilt threshold, it is judged as abnormal; otherwise, it is normal, and the detection result is output.
[0050] SQLite is a lightweight embedded database system that supports the SQL query language and can efficiently store and manage structured data. This database can pre-store tilt threshold data for various component categories, such as resistors, capacitors, or sensors. Each category corresponds to a standard tilt threshold, which is set based on historical test data and industry standards to ensure fast query response. If the tilt angle of a component exceeds the standard tilt threshold, it is considered abnormal; otherwise, it is considered normal. This judgment logic is a threshold comparison mechanism. For example, if the calculated tilt angle is 5.2 degrees, while the standard tilt threshold is 3 degrees, it exceeds the threshold and is considered abnormal. For instance, when the tilt angle of a chipset is detected to be 2.8 degrees, which does not exceed the standard tilt threshold of 4 degrees, it is considered normal.
[0051] An adaptive optics inspection method using the aforementioned adaptive optics inspection system for irregularly shaped PCBAs.
[0052] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.
Claims
1. An adaptive optical inspection system for irregularly shaped PCBAs, characterized in that, include: The preliminary processing module is used to acquire the three-dimensional point cloud data of the irregularly shaped circuit board to be inspected through laser scanning, calculate the surface normal vector, and obtain the surface normal vector field of the circuit board. The sub-region extraction module is used to extract the contour of the surface normal vector field to obtain multiple sub-regions; The reference plane determination module is used to determine the local stable reference plane corresponding to each sub-region; The component category and location determination module is used to identify the components on the irregular circuit board and obtain the category of each component and its three-dimensional coordinates in the coordinate system of the three-dimensional point cloud data based on the three-dimensional point cloud data of the irregular circuit board. The component tilt angle determination module is used to obtain the three-dimensional coordinates of the component in each sub-region, fit the plane of the component based on the three-dimensional coordinates and calculate its normal vector; it is used to calculate the angle between the normal vector of the component's plane and the normal vector of the local stable reference plane of the corresponding sub-region, which is used as the tilt angle of the component relative to the local stable reference plane. The output module is used to determine a preset standard tilt threshold based on the type of the component, compare the tilt angle of the component relative to the local stable reference plane with the preset standard tilt threshold, and output the detection result.
2. The adaptive optical inspection system for irregularly shaped PCBAs according to claim 1, characterized in that: In the preliminary processing module, for the three-dimensional point cloud data, the covariance matrix is calculated by principal component analysis in the neighborhood of a fixed radius around each point. The eigenvector corresponding to the smallest eigenvalue in the covariance matrix is used as the surface normal vector of that point. The surface normal vectors of all points are combined to form the surface normal vector field of the circuit board.
3. The adaptive optical inspection system for irregularly shaped PCBAs according to claim 1, characterized in that: In the preliminary processing module, before calculating the surface normal vector, the coordinate set of the original point cloud data is generated based on the three-dimensional point cloud data of the irregular circuit board to be detected. The number of local neighborhood points of each point is calculated based on the coordinate set, and a point cloud density distribution map is generated. For the point cloud density distribution map, a statistical filtering method is used to remove noise points with significantly low density, resulting in filtered 3D point cloud data.
4. The adaptive optical inspection system for irregularly shaped PCBAs according to claim 1, characterized in that: In the sub-region extraction module, for the surface normal vector field, the angle between the normal vector of each point and the normal vector of the adjacent points in the spatial neighborhood is calculated, the angle difference is obtained, and the points with the angle difference greater than the preset threshold are marked as candidate edge points. Density clustering is performed on candidate edge points to obtain multiple high-density point groups, and each high-density point group is used as the initial point set of a contour edge line. Points with curvature greater than a preset curvature threshold are extracted from the initial point set of each contour edge line as edge feature points; The edge feature points on each contour edge line are arranged in the order of the original 3D point cloud data. The Euclidean distance and the angle between the direction vectors of two adjacent edge feature points are calculated in turn. If the distance between a pair of adjacent edge feature points is less than a preset distance threshold and the angle between the direction vectors is less than a preset angle threshold, the pair of points is retained. Otherwise, the latter point is removed to obtain the refined edge feature point sequence of each contour edge line. Using a refined sequence of edge feature points as a boundary, the 3D point cloud data reflecting the surface of the irregularly shaped circuit board is divided into multiple closed sub-regions.
5. The adaptive optical inspection system for irregularly shaped PCBAs according to claim 1, characterized in that: In the reference plane determination module, point cloud data of all points in each sub-region are acquired to form a sampling point set corresponding to each sub-region; The average normal vector of the surface normal vectors of all points in the same sampling point set is calculated as the normal vector of the sub-region. Based on the direction of the normal vector of the sub-region, a local plane with the same direction is fitted as the local stable reference plane corresponding to the sub-region, thus obtaining the local stable reference plane of each sub-region.
6. The adaptive optical inspection system for irregularly shaped PCBAs according to claim 1, characterized in that: In the component category and position determination module, the intrinsic parameter matrix K and extrinsic parameter matrix of the virtual camera are preset. The intrinsic parameter matrix K includes the focal lengths fx and fy and the principal point coordinates (cx, cy), and the extrinsic parameter matrix consists of the rotation matrix R and the translation vector t. The coordinates of each point P in the 3D point cloud data acquired from the irregularly shaped circuit board are transformed to obtain the coordinates Pc (Pc.x, Pc.y, Pc.z) in the camera coordinate system, where Pc is calculated as follows: Perform perspective projection on coordinate Pc to obtain normalized coordinates (xn, yn), where , ; The pixel coordinates (u, v) are calculated using the intrinsic parameter matrix K, where , ; Map the depth value Pc.z of each point to the range of 0-255 as the gray value; generate a first gray-scale depth image based on the pixel coordinates of all points and the gray value of each point; obtain the set of pixels with gray values greater than a preset gray-scale threshold from the first gray-scale depth image; The connectedComponents function is used to perform connected component labeling on the pixel set to obtain multiple connected regions; the boundingRect function is used to calculate the bounding rectangle of each connected region and the corresponding rectangular sub-image is cropped to obtain the pixel coordinates of all points in the bounding box of each rectangular sub-image; the YOLOv8 network is input for each rectangular sub-image to obtain the component category label; Get the depth values corresponding to the pixel coordinates of all points in the bounding box, and take the median as the unified depth value Zd of the bounding box; By combining the intrinsic parameter matrix K, the pixel coordinates (uc, vc) of the points in the bounding box, and the depth value Zd, the 3D coordinates Pc1 (Pc1.x, Pc1.y, Pc1.z) of the points in the bounding box in the camera coordinate system are calculated using the back projection formula. , , ; The 3D coordinates P1 of the points in the bounding box in the coordinate system of the 3D point cloud data are calculated by inverse extrinsic transformation. , R is the inverse of the rotation matrix R, and T represents the matrix transpose; the combination of the three-dimensional coordinates of all points of the bounding box in the coordinate system of the three-dimensional point cloud data is used as the three-dimensional coordinates of the element in the coordinate system of the three-dimensional point cloud data.
7. The adaptive optical inspection system for irregularly shaped PCBAs according to claim 1, characterized in that: In the component tilt angle determination module, based on the three-dimensional coordinates of each component in the coordinate system of the three-dimensional point cloud data and the coordinates of the refined edge feature points used to reflect the boundary of the sub-region, the point-in-polygon test method is used to determine whether each component is located in the sub-region, and the three-dimensional coordinates of the components in each sub-region are obtained from it, thus obtaining the component coordinate set corresponding to each sub-region. The coordinate set of the component is fitted by least squares to obtain the component plane, where the plane equation is ax + by + cz + d = 0, a, b, c, and d are fitting coefficients, and the normal vector of the component plane is n1 = (a, b, c). Obtain the normal vector n2 of the local stable reference plane of the sub-region; Calculate the inner product of normal vectors n1 and n2, then calculate the magnitudes of normal vectors n1 and n2 respectively. Divide the inner product by the product of the magnitudes of normal vectors n1 and n2 to obtain the cosine value. Then calculate the inverse cosine value to obtain the angle between normal vectors n1 and n2 as the tilt angle of the component.
8. The adaptive optical inspection system for irregularly shaped PCBAs according to claim 1, characterized in that: In the output module, a preset set of thresholds is queried from the SQLite database according to the category of the component to determine the standard tilt threshold corresponding to the current component; If the tilt angle of a component exceeds the standard tilt threshold, it is judged as abnormal; otherwise, it is normal, and the detection result is output.
9. An adaptive optics detection method, characterized in that: The adaptive optical inspection system for irregularly shaped PCBAs as described in any one of claims 1-8.