A PCB board segmentation method and system based on X-ray and three-dimensional point cloud

Abstract

The application provides a PCB segmentation method and system based on X-ray and three-dimensional point cloud, the application extracts two-dimensional perspective features of X-ray by using a residual network combined with a feature pyramid network, and extracts three-dimensional space geometric features of three-dimensional point cloud by using RandLA-Net. Two-dimensional projection of three-dimensional point cloud is completed through a projection matrix, and parameters such as ray propagation direction, path space density and material absorption intensity are calculated to construct a physical information field; two-dimensional local features are obtained by adaptive deformation sampling according to three-dimensional geometric feature prediction projection offset; dynamic perspective confidence weight is constructed by combining occlusion rate, penetration length and the like, and multi-perspective fusion two-dimensional features are obtained by weighted aggregation. Cross-modal adaptive fusion of two-dimensional and three-dimensional features is realized through a gating mechanism, feature restoration and resolution reconstruction are completed through a point cloud decoding network, and a PCB point-level structure segmentation result is output. The application can accurately segment structures such as copper wires, copper columns and dielectric layers in the PCB.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, artificial intelligence and PCB board inspection technology, and in particular to a PCB board segmentation method and system based on X-rays and three-dimensional point clouds. Background Technology

[0002] As the electronics manufacturing industry moves towards high-density integration and complex packaging, the internal structure of PCBs is becoming increasingly complex. Traditional manual inspection and single-modal vision methods are no longer sufficient to meet the demands for high-precision, automated structural segmentation. Currently, PCB structural inspection mainly relies on data such as two-dimensional images, X-ray images, and three-dimensional point clouds. Two-dimensional images can obtain surface texture information but are difficult to reflect the internal structure; X-ray images, although able to penetrate packaging materials to obtain internal information, are two-dimensional projection imaging, which suffers from structural overlap, lack of depth information, and significant penetration noise; three-dimensional point clouds can provide spatial geometric structure information, but have limited ability to express internal material properties and occluded areas.

[0003] Therefore, cross-modal fusion of 2D X-ray images and 3D point clouds has become an important research direction in the field of intelligent PCB inspection. However, most existing fusion methods employ simple feature stitching or attention mechanisms, lacking modeling of the physical imaging process of X-rays. This leads to problems such as spatial alignment errors, projection ambiguity, and information redundancy between 2D and 3D features. Especially in complex PCB structures, the X-ray imaging quality varies significantly from different viewpoints. Existing methods struggle to effectively assess the reliability of information from each viewpoint, easily introducing fuzzy features and noise, thus affecting the accuracy of structural segmentation. Furthermore, existing methods typically use fixed projection positions for 2D feature mapping, making it difficult to adapt to local offset problems in complex spatial structures. They also lack multi-scale cross-modal deep interaction mechanisms, resulting in insufficient characterization capabilities for fine-grained structures, occluded areas, and internal defects.

[0004] Therefore, there is an urgent need to provide a PCB board structure segmentation method based on X-ray physical penetration modeling and cross-modal dynamic fusion of 3D point cloud to improve the segmentation accuracy and robustness in complex PCB structure scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a PCB board segmentation method and system based on X-rays and three-dimensional point clouds. This invention can effectively improve the segmentation accuracy and robustness in complex PCB structure scenarios.

[0006] In a first aspect, the present invention provides a PCB board segmentation method based on X-rays and three-dimensional point clouds, comprising: Acquire 3D point cloud data, multi-view X-ray images, and projection matrices corresponding to each view X-ray image of the PCB board; Based on residual networks and feature pyramid networks, multi-scale feature encoding and cross-scale feature fusion are performed on single-channel X-ray images from various perspectives. The shallow edge texture features, mid-level structural contour features and deep semantic internal structure features of the X-ray images are extracted step by step and finally fused into two-dimensional perspective features. Based on the RandLA-Net architecture, hierarchical random sampling and local neighborhood aggregation of 3D point cloud data are performed to obtain the spatial geometric features of 3D point clouds at each level. The projection matrix is ​​used to project the three-dimensional point cloud onto the two-dimensional X-ray image plane to obtain the point cloud projection coordinates. Based on the point cloud projection coordinates, the ray propagation direction, ray path, path spatial density, ray path depth and cumulative material absorption intensity are calculated to construct the physical information field corresponding to each three-dimensional point cloud. Based on the spatial geometric features of each 3D point cloud, predict the 2D projection offset, perform adaptive deformation sampling in the neighborhood of the base projection position, and match the 3D spatial structure to obtain 2D local features. The target occlusion rate of each viewpoint is constructed using the neighborhood point set of the ray path. Based on the target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length, dynamic view confidence weights are constructed. Based on the dynamic view confidence weights, two-dimensional local features are aggregated and weighted to obtain multi-view fused two-dimensional global features. Multi-view fusion of two-dimensional global features is mapped to three-dimensional space and then fused with the spatial geometric features of three-dimensional point cloud through a gating mechanism to generate cross-modal fused features; By using a point cloud decoding network to perform step-by-step channel semantic restoration and point cloud spatial resolution reconstruction on cross-modal fusion features, the point-level structure segmentation results of the PCB board are obtained.

[0007] Preferably, multi-scale feature encoding and cross-scale feature fusion are performed on single-channel X-ray images from various perspectives based on residual networks and feature pyramid networks. This process progressively extracts shallow edge texture features, mid-level structural contour features, and deep semantic internal structure features from the X-ray images, ultimately fusing them into two-dimensional perspective features, including: The X-ray image is fed into the ResNet residual coding network, and downsampled and encoded through convolutional layers, pooling layers and stacked residual blocks to output multi-scale basic features in sequence. A feature pyramid network is used to start from the deep basic features and perform bilinear upsampling operations from top to bottom to restore the deep basic features to the same size as the lower layer features. Then, it is fused with the basic features of the corresponding level through horizontal connection to obtain shallow edge texture features, mid-layer structural contour features and deep semantic internal structure features. Two-dimensional perspective features are obtained by fusing shallow edge texture features, mid-level structural contour features, and deep semantic internal structural features through channel splicing to achieve multi-scale information fusion.

[0008] Preferably, hierarchical random sampling and local neighborhood aggregation of 3D point cloud data are performed based on the RandLA-Net architecture to obtain the spatial geometric features of 3D point clouds at each level, including: RandLA-Net is used as the backbone network for 3D point cloud feature encoding. Each layer of RandLA-Net performs random downsampling on the 3D point cloud data and randomly selects a portion of the point cloud from the current sampling layer as the sampling key point cloud set for the next layer. For any key point cloud in the current downsampling layer, a K-nearest neighbor search is used to select a neighboring point cloud, and the relative spatial offset of the neighboring area is calculated. The absolute coordinates of the key point cloud and the relative spatial offset are then concatenated to form the initial position features of the neighboring area. The initial location features of the neighborhood and the original point features are nonlinearly mapped using a shared MLP to obtain the neighborhood embedding feature vector; then max pooling is used to aggregate the neighborhood embedding feature vector to obtain the aggregated feature vector. The spatial geometric features of the 3D point cloud at each level are output after multi-level encoding.

[0009] Preferably, a projection matrix is ​​used to project a 3D point cloud onto a 2D X-ray image plane to obtain the point cloud projection coordinates. Based on the point cloud projection coordinates, the ray propagation direction, ray path, path spatial density, ray path depth, and cumulative material absorption intensity are calculated to construct the physical information field corresponding to each 3D point cloud; including: Determine the spatial position of the X-ray source and the calibration projection matrix corresponding to each X-ray imaging viewpoint, traverse each three-dimensional point cloud in the three-dimensional point cloud data of the PCB board, and expand the rectangular coordinates of a single three-dimensional point cloud into homogeneous coordinates. Using the pre-calibrated projection matrices of each viewpoint, the homogenized 3D point cloud is transformed by perspective matrix to obtain the homogenized projection coordinates of the image. The homogeneous projection coordinates of the image are normalized by homogeneous division, and the original two-dimensional projection coordinates of the three-dimensional point cloud on the X-ray image at the current viewpoint are obtained. based on Given the spatial location of the ray source, construct a propagation direction vector from the ray source to the corresponding 3D point cloud for any 3D point cloud; and normalize the propagation direction vector using the L2 norm to obtain the propagation direction of the unit ray. The ray penetration length is calculated based on the Euclidean distance between the spatial location of the ray source and the 3D point cloud, and a continuous ray path is constructed based on the ray penetration length, starting from the spatial location of the ray source and along the unit ray propagation direction. Set a neighborhood distance threshold and define the shortest distance from the 3D point cloud to the continuous ray path space. Construct a ray path neighborhood point set and calculate the path space density based on the ratio of the number of neighborhood points to the ray penetration length. For each neighborhood 3D point cloud in the neighborhood point set of the ray path, solve for its corresponding depth parameter on the ray path, and sort them in ascending order of depth parameter from smallest to largest to obtain the ray path depth sorting sequence. Based on the material absorption coefficient and path element length of the neighborhood 3D point cloud, the cumulative material absorption intensity along the ray path is obtained by summing them. The original two-dimensional projection coordinates, unit ray propagation direction, ray penetration length, path spatial density, ray path depth sorting sequence, and material cumulative absorption intensity are integrated into the physical information field of each three-dimensional point cloud from various perspectives.

[0010] Preferably, the two-dimensional projection offset is predicted based on the spatial geometric features of each 3D point cloud, and adaptive deformation sampling is performed in the neighborhood of the base projection position to match the 3D spatial structure and obtain two-dimensional local features; including: The original two-dimensional projected coordinates of each three-dimensional point cloud are used as the reference anchor point position for two-dimensional feature sampling. A learnable offset prediction mapping function is constructed to drive the adaptive prediction of K sets of two-dimensional coordinate offsets with the spatial geometric features of the three-dimensional point cloud. A scale constraint factor will be introduced to superimpose the predicted offset onto the original two-dimensional projected coordinates, generate deformable actual sampling positions, and impose interval constraints on the sampling coordinates. In the Bilinear interpolation operators are used to perform sub-pixel feature sampling at each deformable sampling location on the two-dimensional viewpoint features; An attention mapping network is constructed to predict the weight score of each sampling position based on the spatial geometric features of the 3D point cloud, and the attention weight is obtained by Softmax normalization based on the weight score. The sub-pixel features obtained from multiple deformation samplings are weighted and aggregated based on attention weights to obtain two-dimensional local features.

[0011] Preferably, the target occlusion rate for each viewpoint is constructed using the neighborhood point set of the ray path, and a dynamic viewpoint confidence weight is constructed based on the target occlusion rate, path spatial density, cumulative material absorption intensity, and ray penetration length. Two-dimensional local features are then aggregated and weighted based on these dynamic viewpoint confidence weights to obtain multi-view fused two-dimensional global features; including: Using the neighborhood point set of the ray path, the front layer occlusion point and the rear layer occlusion point are distinguished along the X-ray propagation direction. The proportion of the number of forward occlusion points to the total number of neighborhood points is counted to construct the target occlusion rate of the view. The first method is constructed by combining target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length. The original confidence score function for 3D point clouds from a given perspective; The Softmax function is used to normalize the original confidence scores of each viewpoint to generate dynamic viewpoint confidence weights. Using the dynamic perspective confidence weight as the weighting coefficient, the two-dimensional local features obtained by adaptive deformation sampling under each single perspective are weighted and aggregated to obtain multi-view fused two-dimensional global features.

[0012] Preferably, the multi-view fused two-dimensional global features are mapped to three-dimensional space, and then fused with the spatial geometric features of the three-dimensional point cloud through a gating mechanism to generate cross-modal fused features; including: The multi-view fused two-dimensional global features are mapped to three-dimensional space through a linear projection layer, and the mapped multi-view fused two-dimensional global features are stitched together with the spatial geometric features of the three-dimensional point cloud to obtain the stitched features. A multilayer perceptron is used to construct a gating mapping function, which performs a nonlinear transformation on the spliced ​​features, and then generates channel-by-channel gating weights through Sigmoid activation. Based on channel-gated weights, a channel-by-channel Hadamard product weighted fusion mechanism is used to dynamically modulate the contribution ratio of the multi-view fusion of two-dimensional global features and spatial geometric features of three-dimensional point clouds, thereby obtaining cross-modal fusion features.

[0013] Preferably, a point cloud decoding network is used to perform step-by-step channel semantic restoration and point cloud spatial resolution reconstruction on cross-modal fusion features to obtain point-level structure segmentation results for the PCB board, including: Based on cross-modal fusion features, the point cloud decoding network performs point cloud upsampling and expansion layer by layer from the deepest layer. It adopts an interpolation weighting method based on feature distance for the current sparse key point features, maps the high-level sparse features onto the encrypted point set, and performs spatial resolution reconstruction to obtain the encrypted point feature vector. For each layer of decoding, the encrypted point feature vector obtained by upsampling and interpolation is combined with the cross-modal fusion feature of the corresponding layer of the encoding network through channel concatenation and fusion to obtain the decoded feature vector; After restoring the original point cloud resolution through multi-layer decoding, a multi-layer perceptron (MLP) classification head is connected to perform semantic category mapping on the decoded feature vector of each 3D point cloud, resulting in a category prediction score vector for the 3D point cloud. The probability distribution of each category is obtained by using Softmax normalization; For each 3D point cloud, the category with the highest probability is taken as the final predicted semantic label to obtain the point-level structure segmentation result of the 3D point cloud data of the PCB board.

[0014] Secondly, the present invention provides a PCB board segmentation system based on X-rays and three-dimensional point clouds, comprising: The data acquisition module is used to acquire 3D point cloud data, multi-view X-ray images, and projection matrices corresponding to each view of the PCB board. The two-dimensional view feature extraction module is used to perform multi-scale feature encoding and cross-scale feature fusion on single-channel X-ray images from various viewpoints based on residual networks and feature pyramid networks. It extracts shallow edge texture features, mid-level structural contour features and deep semantic internal structure features of X-ray images step by step, and finally fuses them into two-dimensional view features. The spatial geometric feature extraction module is used to perform hierarchical random sampling and local neighborhood aggregation on 3D point cloud data based on the RandLA-Net architecture to obtain the spatial geometric features of 3D point clouds at each level. The physical information field construction module is used to project three-dimensional point clouds onto a two-dimensional X-ray image plane using a projection matrix to obtain the projection coordinates of the point clouds, and to calculate the ray propagation direction, ray path, path spatial density, ray path depth and cumulative material absorption intensity based on the point cloud projection coordinates in order to construct the physical information field corresponding to each three-dimensional point cloud. The two-dimensional local feature extraction module is used to predict the two-dimensional projection offset based on the spatial geometric features of each three-dimensional point cloud, perform adaptive deformation sampling in the neighborhood of the basic projection position, and match the three-dimensional spatial structure to obtain two-dimensional local features. The multi-view fusion two-dimensional global feature extraction module is used to construct the target occlusion rate of each view using the neighborhood point set of the ray path, and to construct dynamic view confidence weights based on the target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length. Based on the dynamic view confidence weights, the two-dimensional local features are aggregated and weighted to obtain the multi-view fusion two-dimensional global features. The feature fusion module is used to map multi-view fused two-dimensional global features to three-dimensional space, and perform cross-modal fusion with the spatial geometric features of the three-dimensional point cloud through a gating mechanism to generate cross-modal fused features; The decoding module is used to perform channel semantic restoration and point cloud spatial resolution reconstruction step by step on cross-modal fusion features using a point cloud decoding network to obtain the point-level structure segmentation result of the PCB board.

[0015] Thirdly, the present invention provides an electronic device, comprising: At least one processor; and memory that is communicatively connected to at least one processor; The memory stores a computer program that can be executed by at least one processor, characterized in that the computer program, when executed by at least one processor, implements the PCB board segmentation method based on X-rays and three-dimensional point clouds.

[0016] The beneficial effects of this invention are as follows: 1. This paper models the propagation direction, penetration length, path spatial density, depth ordering relationship and material absorption intensity and other physical parameters to construct the physical information field of each three-dimensional point, which can accurately characterize the imaging attenuation and spatial occlusion relationship in the case of multi-layer stacking, metal occlusion and structural overlap inside the PCB. 2. This invention relies on the three-dimensional point cloud geometric features to adaptively predict the two-dimensional projection offset, breaking through the rigid constraints of traditional fixed projection sampling. Through adaptive deformation sampling and attention weighted aggregation mechanism, it compensates for imaging calibration errors, structural misalignment and X-ray imaging distortion, so that the two-dimensional feature sampling position adaptively matches the local structure morphology of three-dimensional space, significantly enhancing the spatial alignment accuracy between two-dimensional X-ray features and three-dimensional point cloud features. 3. This invention dynamically calculates the confidence weight of each viewpoint based on constraints such as X-ray penetration quality, occlusion rate, path density, material absorption, and multi-view consistency. It adaptively weakens the contribution of views with severe occlusion, increased penetration attenuation, and degraded imaging quality, thereby enhancing the clear and reliable representation of viewpoint features. It effectively suppresses the interference of metal scattering artifacts, structural overlap and aliasing, and penetration noise on the segmentation results, significantly improving the robustness of multi-view feature fusion for complex PCBs. 4. This invention dynamically adjusts the contribution ratio of internal features of two-dimensional X-ray texture and geometric features of three-dimensional point cloud by modal space mapping and channel-by-channel gating weights, thereby enhancing three-dimensional geometric information in areas with complex geometric structures and enhancing two-dimensional X-ray information in areas with significant internal texture and transmission features. 5. This invention embeds a cross-modal fusion module into a multi-level three-dimensional point cloud coding network to achieve deep interaction of two-dimensional and three-dimensional features at different scales. It takes into account both shallow fine-grained lines and copper pillar edge details and deep global layout and stacked structure semantics, effectively improving the feature representation ability of slender copper lines, dense microstructures and occluded hidden areas. 6. This invention employs layer-by-layer feature recovery and spatial resolution reconstruction using a decoding network, combined with Softmax probabilistic classification and maximum probability category determination, to output point-level semantic segmentation results that correspond one-to-one with the original point cloud. It can accurately distinguish typical internal structures such as PCB dielectric layers, copper lines, copper pillars, and pads, with high segmentation accuracy and strong generalization, and can meet the requirements of high-density integrated PCB internal structure analysis, defect identification, and automated industrial non-destructive testing. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the method of an embodiment of the present invention; Figure 2 This is a schematic diagram of the system framework according to an embodiment of the present invention; Figure 3 This is a schematic diagram of an X-ray image according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the reconstructed point cloud data in an embodiment of the present invention; Figure 5 This is a schematic diagram showing the comparison between the final segmentation and classification results output on the test set and the actual labels in an embodiment of the present invention; In the diagram, 100 is the data acquisition module; 200 is the two-dimensional perspective feature extraction module; 300 is the spatial geometric feature extraction module; 400 is the physical information field construction module; 500 is the two-dimensional local feature extraction module; 600 is the multi-view fusion two-dimensional global feature extraction module; 700 is the feature fusion module; and 800 is the decoding module. Detailed Implementation

[0018] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings: like Figure 1 As shown, this embodiment provides a PCB board segmentation method based on X-rays and three-dimensional point clouds, including: S1: Acquire the 3D point cloud data, multi-view X-ray images, and projection matrices corresponding to the X-ray images from each view of the PCB board; S2: Based on residual network and feature pyramid network, multi-scale feature encoding and cross-scale feature fusion are performed on single-channel X-ray images from various perspectives. The shallow edge texture features, middle structure contour features and deep semantic internal structure features of X-ray images are extracted step by step and finally fused into two-dimensional perspective features. S3: Based on the RandLA-Net architecture, hierarchical random sampling and local neighborhood aggregation are performed on 3D point cloud data to obtain the spatial geometric features of 3D point clouds at each level; S4: The three-dimensional point cloud is projected onto the two-dimensional X-ray image plane using the projection matrix to obtain the point cloud projection coordinates. Based on the point cloud projection coordinates, the ray propagation direction, ray path, path spatial density, ray path depth and cumulative material absorption intensity are calculated to construct the physical information field corresponding to each three-dimensional point cloud. S5: Predict the 2D projection offset based on the spatial geometric features of each 3D point cloud, perform adaptive deformation sampling in the neighborhood of the base projection position, and match the 3D spatial structure to obtain 2D local features. S6: Construct the target occlusion rate of each viewpoint using the neighborhood point set of the ray path, and construct dynamic viewpoint confidence weights based on the target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length. Aggregate two-dimensional local features based on the dynamic viewpoint confidence weights and perform weighted aggregation to obtain multi-view fused two-dimensional global features. S7: Map the multi-view fused 2D global features to 3D space, and perform cross-modal fusion with the spatial geometric features of 3D point cloud through a gating mechanism to generate cross-modal fused features; S8: The point cloud decoding network is used to perform channel semantic restoration and point cloud spatial resolution reconstruction step by step on the cross-modal fusion features to obtain the point-level structure segmentation result of the PCB board.

[0019] In this embodiment, step S2 involves multi-scale feature encoding and cross-scale feature fusion of single-channel X-ray images from various perspectives based on residual networks and feature pyramid networks. This process progressively extracts shallow edge texture features, mid-level structural contour features, and deep semantic internal structure features from the X-ray images, ultimately fusing them into two-dimensional perspective features. This includes: S21, Let the input be the... Single-channel X-ray image with viewing angle and the X-ray images are The data is fed into a ResNet residual coding network, where it undergoes downsampling encoding through convolutional layers, pooling layers, and stacked residual blocks, sequentially outputting multi-scale basic features: ; in, These are shallow features; It represents mid-level multi-scale features; Indicates deep features; S22. Employing a feature pyramid network to analyze deep, fundamental features Initially, bilinear upsampling is performed step-by-step from top to bottom to extract deep, fundamental features. The features are restored to the same size as the lower-level features, and then fused element by element with the corresponding level's basic features through horizontal connections to obtain the shallow edge texture features. Mid-layer structure outline features With deep semantic internal structural features ; S23, Apply shallow edge texture features Mid-layer structure outline features With deep semantic internal structural features Two-dimensional view features are obtained by multi-scale information fusion using channel stitching. .

[0020] In this embodiment, step S3 involves performing hierarchical random sampling and local neighborhood aggregation on the 3D point cloud data based on the RandLA-Net architecture to obtain the spatial geometric features of the 3D point cloud at each level; including: S31. RandLA-Net is used as the backbone network for 3D point cloud feature encoding. Each layer of RandLA-Net performs random downsampling on the 3D point cloud data, and randomly selects a portion of the point cloud from the current sampling layer's sampled point cloud set as the next layer's sampled key point cloud set, that is: ; In the formula, For the first The sampling key points of the layer are concentrated. For the first A three-dimensional point cloud; For the first Layered point cloud index set; S32, regarding the first Any key point cloud in any layer K-nearest neighbor search is used to select the neighborhood point cloud. Calculate the relative spatial offset of the neighborhood ; key point cloud Absolute coordinates and relative spatial offset By splicing together, the initial positional features of the neighborhood are formed. ,Right now: ; ; In the formula, This indicates a splicing operation.

[0021] S33, Initial position features of the neighborhood The original point features are nonlinearly mapped using a shared MLP to obtain a neighborhood embedding feature vector; then, max pooling is used to aggregate the neighborhood embedding feature vector to obtain an aggregated feature vector, i.e.: ; ; In the formula, Represents aggregated feature vectors; Representing key point clouds With neighboring point clouds The coupled neighborhood embedding feature vector; Represented by key point cloud The set of K-nearest neighbors in local neighborhood.

[0022] S34. Output the spatial geometric features of the 3D point cloud at each level after multi-level encoding. .

[0023] In this embodiment, in step S4, the three-dimensional point cloud is projected onto the two-dimensional X-ray image plane using a projection matrix to obtain the point cloud projection coordinates. Based on the point cloud projection coordinates, the ray propagation direction, ray path, path spatial density, ray path depth, and cumulative material absorption intensity are calculated to construct the physical information field corresponding to each three-dimensional point cloud, including:

[0024] S41. Determine the spatial position of the X-ray source and the calibration projection matrix corresponding to each X-ray imaging viewpoint. Traverse each three-dimensional point cloud in the three-dimensional point cloud data of the PCB board and expand the rectangular coordinates of a single three-dimensional point cloud into homogeneous coordinates. S42. Using the pre-calibrated projection matrices for each viewpoint, perform perspective matrix transformation on the homogeneous 3D point cloud to obtain the homogeneous projection coordinates of the image, i.e.: ; In the formula, The coordinates are the homogeneous projection coordinates of the image; For the first indivual The calibration projection matrix for the ray-view angle; The homogenized 3D point cloud; The homogeneous projection coordinates of the image are normalized by homogeneous division to obtain the original two-dimensional projection coordinates of the 3D point cloud on the X-ray image at the current viewpoint. ; ; S43, Let the first Perspective The spatial location of the radiation source is For any 3D point cloud Construct a three-dimensional point cloud pointing from the ray source. The propagation direction vector is obtained; and the propagation direction vector is normalized using the L2 norm to obtain the propagation direction of the unit ray, i.e.: ; ; In the formula, For the first Perspective The ray source points to a three-dimensional point cloud. The propagation direction vector; The direction of unit ray propagation; It is an L2 norm; S44, Based on the spatial location of the radiation source With 3D point cloud The Euclidean distance is used to calculate the ray penetration length, and based on the ray penetration length, the spatial location of the ray source is determined. Construct a continuous ray path starting from the unit ray and along the direction of ray propagation, that is: ; ; In the formula, For the ray path; Indicates the path step size parameter; The length of the ray penetration; S45, Set neighborhood distance threshold Define a 3D point cloud The shortest distance to the continuous ray path space is Construct the neighborhood point set of the ray path The path spatial density is calculated based on the ratio of the number of neighborhood points of the line path to the ray penetration length. ,Right now: ; ; In the formula, This represents the number of neighborhood points along the line path. S46, Set of points in the neighborhood of the ray path 3D point cloud of each neighborhood Solve for the depth parameter corresponding to it on the ray path. and according to depth parameters Arrange the rays in ascending order to obtain the ray path depth sorting sequence. ,Right now: ; S47. Define the neighborhood 3D point cloud The corresponding material absorption coefficient is The cumulative absorption intensity of the material along the ray path is then obtained by summing the values. ,Right now: ; In the formula, 3D point cloud of the neighborhood The path element length; S48. Combine the original two-dimensional projection coordinates, unit ray propagation direction, ray penetration length, and path spatial density. Ray path depth sorting sequence Cumulative absorption strength of materials This integrates the physical information field of each 3D point cloud from various perspectives.

[0025] In this embodiment, step S5 involves predicting the two-dimensional projection offset based on the spatial geometric features of each 3D point cloud, performing adaptive deformation sampling within the neighborhood of the base projection position, and matching the 3D spatial structure to obtain two-dimensional local features; including: S51. Project the original two-dimensional coordinates of each 3D point cloud. Using the reference anchor point position for 2D feature sampling, a learnable offset prediction mapping function is constructed to utilize the spatial geometric features of the 3D point cloud. Drive adaptive prediction of K sets of 2D coordinate offsets ,Right now: ; In the formula, This represents the number of neighborhood sampling points corresponding to a single 3D point cloud. S52, A scale constraint factor will be introduced. The predicted offset is superimposed on the normalized projected coordinates to generate a deformable actual sampling position. And apply interval constraints to the sampling coordinates, namely: ; S53, in the Two-dimensional viewpoint features The above uses a bilinear interpolation operator to perform sub-pixel feature sampling at each deformable sampling location; that is: ; In the formula, Subpixel features; S54. Construct an attention mapping network based on the spatial geometric features of 3D point clouds. Predict the weight score for each sampling location And based on weighted scores Attention weights are obtained through Softmax normalization. ,Right now: ; ; S55, Based on attention weights Sub-pixel features obtained from sampling multiple deformations Weighted aggregation is performed to obtain two-dimensional local features. ;Right now: ; In the formula, 3D point cloud In the Two-dimensional local features of the viewpoint.

[0026] In this embodiment, in step S6, the target occlusion rate of each viewpoint is constructed using the ray path neighborhood point set, and a dynamic viewpoint confidence weight is constructed based on the target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length. Two-dimensional local features are aggregated based on the dynamic viewpoint confidence weight and weighted aggregation to obtain multi-view fused two-dimensional global features, including: S61. Utilizing the neighborhood point set of the ray path Distinguish between front-layer occlusion points and rear-layer occlusion points along the X-ray propagation direction, count the proportion of the number of forward-occlusion points to the total number of neighborhood points, and construct the target occlusion rate of the viewpoint. S62. Construct the first [structure / method] by combining target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length. 3D point cloud from a certain perspective The original confidence score function ,Right now: ; In the formula, , , , These are the weights of the corresponding items; For the first 3D point cloud from a certain perspective occlusion rate; S63. The Softmax function is used to normalize the original confidence scores of each viewpoint, generating dynamic viewpoint confidence weights; that is: ; In the formula, This represents the total number of X-ray viewing angles. For dynamic perspective confidence weights; S64. Using the dynamic viewpoint confidence weight as the weighting coefficient, the two-dimensional local features obtained from adaptive deformation sampling under each single viewpoint are weighted and aggregated to obtain the multi-view fused two-dimensional global features. ,Right now: ; In the formula, 3D point cloud In the Two-dimensional local features of the viewpoint.

[0027] In this embodiment, step S7 involves mapping the multi-view fused two-dimensional global features to a three-dimensional space, and then fusing them with the spatial geometric features of the three-dimensional point cloud through a gating mechanism to generate cross-modal fused features; including: S71. Fusing multi-viewpoint 2D global features through a linear projection layer. Mapping to three-dimensional space and fusing the multi-view mapping with two-dimensional global features. Spatial geometric features of 3D point clouds By splicing the pieces together, we can obtain the splicing features. ; ; S72. Constructing a gating mapping function using a multilayer perceptron. splicing features A nonlinear transformation is performed, followed by Sigmoid activation to generate channel-wise gated weights, i.e.: ; In the formula, is Sigmoid activation function; Channel gating weights; S73. Based on channel-gated weights, a channel-wise Hadamard product weighted fusion mechanism is used to dynamically modulate the mapping of multi-view fusion of two-dimensional global features. Spatial geometric features of 3D point clouds The contribution ratio is used to obtain the cross-modal fusion feature, namely: ; In the formula, For cross-modal fusion features; For each channel, the Hadamard product is used.

[0028] In this embodiment, in step S8, a point cloud decoding network is used to process cross-modal fusion features. By performing step-by-step channel semantic restoration and point cloud spatial resolution reconstruction, the point-level structure segmentation results of the PCB board are obtained, including: S81, Based on cross-modal fusion features The point cloud decoding network upsamples and expands the point cloud layer by layer from the deepest layer upwards. It uses a feature distance-based interpolation weighting method to map the high-level sparse features onto the encrypted point set, performing spatial resolution reconstruction to obtain the encrypted point feature vector, i.e.: ; In the formula, This represents the feature vector of the encrypted point after being recovered by nearest neighbor interpolation; The set of nearest key point clouds for the point cloud to be encrypted; For key point cloud Interpolation weights; For key point cloud Cross-modal fusion features; S82. For each layer of decoding, the encrypted point feature vector obtained by upsampling and interpolation is concatenated and fused with the cross-modal fusion features of the corresponding layer of the encoding network to obtain the decoded feature vector, i.e.: ; In the formula, For the first Hierarchical decoding of feature vectors; It is a convolutional layer; S83. After restoring the original point cloud resolution through multi-layer decoding, the MLP classification head is connected to perform semantic category mapping on the decoded feature vector of each 3D point cloud to obtain the category prediction score vector of the 3D point cloud. The probability distributions for each category are obtained by using Softmax normalization; that is: ; ; In the formula, Representing a 3D point cloud Category The probability of; S84. For each 3D point cloud, take the category with the highest probability as the final predicted semantic label. The point-level structure segmentation result of the 3D point cloud data of the PCB board is obtained, namely: ; In the formula, 3D point cloud The final predicted semantic label.

[0029] In this embodiment, to verify its effectiveness and robustness, a virtual voxel model containing typical structures such as copper pillars and copper wires was generated based on the PCB structure design, and a total of 15 simulation datasets were constructed. For each dataset, a corresponding X-ray image was generated using a multi-view projection method, and the accurate imaging projection matrix was calculated. Specific X-ray images are shown below. Figure 3 As shown.

[0030] For 3D data acquisition, the FDK (Feldkamp-Davis-Kress) cone-beam reconstruction algorithm is used to reconstruct 3D volume data from multi-view X-ray data. The reconstructed results are then converted into point cloud data to serve as the 3D input representation for subsequent networks. This processing effectively ensures a strict correspondence between the data in the 2D projection domain and the 3D geometric domain. The reconstructed point cloud is as follows: Figure 4 As shown; To ensure the objectivity of the experimental results and the generalization ability of the model, the above 15 sets of multimodal data (including multi-view X-ray images, projection matrices and 3D point clouds) were randomly divided in a ratio of 6:2:2. That is, 9 sets of data were selected as the training set for network parameter optimization, 3 sets of data were selected as the validation set for hyperparameter tuning and model selection, and the remaining 3 sets of data were selected as the test set for final performance evaluation.

[0031] In terms of qualitative analysis, the final segmentation and classification results output by the method in this embodiment on the test set were compared with the real labels using 3D visualization. Experimental results show that, thanks to the 2D and 3D feature multi-scale interactive machine and cross-modal gating fusion module designed in this embodiment, the algorithm can segment the complex microstructure inside the PCB with high accuracy. In particular, for copper lines that are slender, densely distributed, and prone to scattering artifacts, as well as copper pillars with three-dimensional structures that are prone to local occlusion in a single view, this invention demonstrates excellent segmentation performance, such as... Figure 5As shown, the proposed method exhibits good segmentation performance in the microstructure segmentation within PCBs. Comparison of the predicted results (Pred) and the ground truth (GT) results reveals that in small structural regions such as the connections between copper pillars and copper lines, the proposed method can accurately segment the boundaries of different targets, and the connected regions are segmented completely. Furthermore, the predicted results effectively maintain structural continuity and topological consistency in the elongated copper line regions, demonstrating a high degree of consistency between the predicted and ground truth annotations.

[0032] like Figure 2 As shown, embodiments of this application also provide a PCB board segmentation system based on X-rays and three-dimensional point clouds, including: The data acquisition module 100 is used to acquire 3D point cloud data, multi-view X-ray images, and projection matrices corresponding to the X-ray images from each view of the PCB board. The two-dimensional view feature extraction module 200 is used to perform multi-scale feature encoding and cross-scale feature fusion on single-channel X-ray images from various viewpoints based on residual networks and feature pyramid networks. It extracts shallow edge texture features, mid-level structural contour features and deep semantic internal structure features of X-ray images step by step, and finally fuses them into two-dimensional view features. The spatial geometric feature extraction module 300 is used to perform hierarchical random sampling and local neighborhood aggregation on 3D point cloud data based on the RandLA-Net architecture to obtain the spatial geometric features of 3D point clouds at each level. The physical information field construction module 400 is used to project a three-dimensional point cloud onto a two-dimensional X-ray image plane using a projection matrix to obtain the point cloud projection coordinates, and calculate the ray propagation direction, ray path, path spatial density, ray path depth and cumulative material absorption intensity based on the point cloud projection coordinates to construct the physical information field corresponding to each three-dimensional point cloud. The two-dimensional local feature extraction module 500 is used to predict the two-dimensional projection offset based on the spatial geometric features of each three-dimensional point cloud, perform adaptive deformation sampling in the neighborhood of the basic projection position, and match the three-dimensional spatial structure to obtain two-dimensional local features. The multi-view fusion two-dimensional global feature extraction module 600 is used to construct the target occlusion rate of each view using the neighborhood point set of the ray path, and to construct dynamic view confidence weights based on the target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length. Based on the dynamic view confidence weights, the two-dimensional local features are aggregated and weighted to obtain multi-view fusion two-dimensional global features. The feature fusion module 700 is used to map multi-view fused two-dimensional global features to three-dimensional space, and perform cross-modal fusion with the spatial geometric features of the three-dimensional point cloud through a gating mechanism to generate cross-modal fused features; The decoding module 800 is used to perform channel semantic restoration and point cloud spatial resolution reconstruction step by step on cross-modal fusion features using a point cloud decoding network to obtain the point-level structure segmentation result of the PCB board.

[0033] In this application, the two-dimensional viewpoint feature extraction module 200 extracts X-ray images as... The data is fed into the ResNet residual coding network, and downsampled and encoded through convolutional layers, pooling layers and stacked residual blocks to output multi-scale basic features in sequence. The two-dimensional view feature extraction module 200 uses a feature pyramid network to extract features from deep basic features. Initially, bilinear upsampling is performed step-by-step from top to bottom to extract deep, fundamental features. The features are restored to the same size as the lower-level features, and then fused element by element with the corresponding level's basic features through horizontal connections to obtain the shallow edge texture features. Mid-layer structure outline features With deep semantic internal structural features ; shallow edge texture features Mid-layer structure outline features With deep semantic internal structural features Two-dimensional view features are obtained by multi-scale information fusion using channel stitching. .

[0034] In this embodiment, the spatial geometric feature extraction module 300 uses RandLA-Net as the backbone network for three-dimensional point cloud feature encoding. Each layer of RandLA-Net performs random downsampling on the three-dimensional point cloud data and randomly selects a portion of the point cloud from the current sampling layer as the sampling key point cloud set for the next layer. For the first Any key point cloud in any layer K-nearest neighbor search is used to select the neighborhood point cloud. Calculate the relative spatial offset of the neighborhood ; key point cloud Absolute coordinates and relative spatial offset By splicing together, the initial positional features of the neighborhood are formed. .

[0035] Initial location features of the neighborhood The original point features are nonlinearly mapped using a shared MLP to obtain a neighborhood embedding feature vector; then, max pooling is used to aggregate the neighborhood embedding feature vector to obtain an aggregated feature vector, i.e.: ; ; In the formula, Represents aggregated feature vectors; Representing key point clouds With neighboring point clouds The coupled neighborhood embedding feature vector; Represented by key point cloud The set of K-nearest neighbors in local neighborhood.

[0036] After multi-level encoding, the spatial geometric features of each level of the 3D point cloud are output. .

[0037] In this embodiment, the physical information field construction module 40 traverses each three-dimensional point cloud in the three-dimensional point cloud data of the PCB board, and expands the rectangular coordinates of a single three-dimensional point cloud into a homogeneous coordinate form. Using the pre-calibrated projection matrices of each viewpoint, the homogenized 3D point cloud is transformed by perspective matrix to obtain the homogenized projection coordinates of the image. The homogeneous projection coordinates of the image are normalized by homogeneous division, and the original two-dimensional projection coordinates of the three-dimensional point cloud on the X-ray image at the current viewpoint are obtained. Construct a 3D point cloud pointing from a ray source. The propagation direction vector is obtained; and the propagation direction vector is normalized using the L2 norm to obtain the propagation direction of the unit ray. Based on the spatial location of the radiation source With 3D point cloud The Euclidean distance is used to calculate the ray penetration length, and based on the ray penetration length, the spatial location of the ray source is determined. A continuous ray path is constructed starting from the unit ray propagation direction; Set neighborhood distance threshold Define a 3D point cloud The shortest distance to the continuous ray path space is Construct the neighborhood point set of the ray path The path spatial density is calculated based on the ratio of the number of neighborhood points of the line path to the ray penetration length. ; For the neighborhood point set of the ray path 3D point cloud of each neighborhood Solve for the depth parameter corresponding to it on the ray path. and according to depth parameters Arrange the rays in ascending order to obtain the ray path depth sorting sequence. ; Define a neighborhood 3D point cloud The corresponding material absorption coefficient is The cumulative absorption intensity of the material along the ray path is then obtained by summing the values. ; The original two-dimensional projection coordinates, unit ray propagation direction, ray penetration length, and path spatial density are used to... Ray path depth sorting sequence Cumulative absorption strength of materials This integrates the physical information field of each 3D point cloud from various perspectives.

[0038] In this embodiment, the two-dimensional local feature extraction module 500 extracts the original two-dimensional projected coordinates of each three-dimensional point cloud. Using the reference anchor point position for 2D feature sampling, a learnable offset prediction mapping function is constructed to utilize the spatial geometric features of the 3D point cloud. Drive adaptive prediction of K sets of 2D coordinate offsets A scale constraint factor will be introduced. The predicted offset is superimposed on the normalized projected coordinates to generate a deformable actual sampling position. And apply interval constraints to the sampling coordinates; in the first... Two-dimensional viewpoint features The above employs a bilinear interpolation operator to perform sub-pixel feature sampling at each deformable sampling location; an attention mapping network is constructed based on the spatial geometric features of the 3D point cloud. Predict the weight score for each sampling location And based on weighted scores Attention weights are obtained through Softmax normalization. Based on attention weight Sub-pixel features obtained from sampling multiple deformations Weighted aggregation is performed to obtain two-dimensional local features. .

[0039] In this embodiment, the multi-view fusion two-dimensional global feature extraction module 600 utilizes the ray path neighborhood point set. Distinguish between front-layer occlusion points and rear-layer occlusion points along the X-ray propagation direction, count the proportion of the number of forward-occlusion points to the total number of neighborhood points, and construct the target occlusion rate of the viewpoint. Simultaneously, by combining the target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length, the first... 3D point cloud from a certain perspective The original confidence score function ; The Softmax function is used to normalize the original confidence scores of each viewpoint to generate dynamic viewpoint confidence weights. Using dynamic perspective confidence weights as weighting coefficients, the two-dimensional local features obtained from adaptive deformation sampling under each single perspective are weighted and aggregated to obtain multi-view fused two-dimensional global features. .

[0040] In this embodiment, the feature fusion module 700 maps multi-view fused two-dimensional global features to three-dimensional space, and performs cross-modal fusion with the spatial geometric features of the three-dimensional point cloud through a gating mechanism to generate cross-modal fused features; including: Multi-view fusion of two-dimensional global features via a linear projection layer Mapping to three-dimensional space and fusing the multi-view mapping with two-dimensional global features. Spatial geometric features of 3D point clouds By splicing the pieces together, we can obtain the splicing features. ; Gating mapping function constructed using multilayer perceptron splicing features A nonlinear transformation is performed, followed by Sigmoid activation to generate channel-wise gated weights; Based on channel-gated weights, a channel-wise Hadamard product weighted fusion mechanism is used to dynamically modulate the mapping of multi-view fusion of two-dimensional global features. Spatial geometric features of 3D point clouds The contribution ratio is used to obtain cross-modal fusion features; In this embodiment, the decoding module 800 is based on cross-modal fusion features. The point cloud decoding network upsamples and expands the point cloud layer by layer from the deepest layer upwards. It uses a feature distance-based interpolation weighting method to interpolate and map the high-level sparse features onto the encrypted point set, reconstructing the spatial resolution to obtain encrypted point feature vectors. At each decoding layer, the encrypted point feature vectors obtained from the upsampling and interpolation are concatenated and fused with the cross-modal fusion features of the corresponding layer of the encoding network to obtain the decoded feature vectors. After restoring the original point cloud resolution through multiple decoding layers, the network is connected to a multilayer perceptron (MLP) classification head. Semantic category mapping is performed on the decoded feature vectors of each 3D point cloud to obtain the category prediction score vector of the 3D point cloud. The probability distribution of each category is obtained by using Softmax normalization; the category with the highest probability for each 3D point cloud is taken as the final predicted semantic label. The point-level structure segmentation results of the 3D point cloud data of the PCB board are obtained.

[0041] Embodiments of this application also provide an electronic device, including: At least one processor; and memory that is communicatively connected to at least one processor; The memory stores a computer program that can be executed by at least one processor. The computer program, when executed by at least one processor, implements the PCB board segmentation method based on X-rays and three-dimensional point clouds as described in Example 1.

[0042] In this embodiment, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. A processor, coupled to the memory, is used to execute computer programs stored in the memory.

[0043] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A PCB board segmentation method based on X-rays and 3D point clouds, characterized in that, include: Acquire 3D point cloud data, multi-view X-ray images, and projection matrices corresponding to each view X-ray image of the PCB board; Based on residual networks and feature pyramid networks, multi-scale feature encoding and cross-scale feature fusion are performed on single-channel X-ray images from various perspectives. The shallow edge texture features, mid-level structural contour features and deep semantic internal structure features of the X-ray images are extracted step by step and finally fused into two-dimensional perspective features. Based on the RandLA-Net architecture, hierarchical random sampling and local neighborhood aggregation of 3D point cloud data are performed to obtain the spatial geometric features of 3D point clouds at each level. The projection matrix is ​​used to project the three-dimensional point cloud onto the two-dimensional X-ray image plane to obtain the point cloud projection coordinates. Based on the point cloud projection coordinates, the ray propagation direction, ray path, path spatial density, ray path depth and cumulative material absorption intensity are calculated to construct the physical information field corresponding to each three-dimensional point cloud. Based on the spatial geometric features of each 3D point cloud, predict the 2D projection offset, perform adaptive deformation sampling in the neighborhood of the base projection position, and match the 3D spatial structure to obtain 2D local features. The target occlusion rate of each viewpoint is constructed using the neighborhood point set of the ray path. Based on the target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length, dynamic view confidence weights are constructed. Based on the dynamic view confidence weights, two-dimensional local features are aggregated and weighted to obtain multi-view fused two-dimensional global features. Multi-view fusion of two-dimensional global features is mapped to three-dimensional space and then fused with the spatial geometric features of three-dimensional point cloud through a gating mechanism to generate cross-modal fused features; By using a point cloud decoding network to perform step-by-step channel semantic restoration and point cloud spatial resolution reconstruction on cross-modal fusion features, the point-level structure segmentation results of the PCB board are obtained.

2. The PCB board segmentation method based on X-rays and three-dimensional point clouds according to claim 1, characterized in that, Multi-scale feature encoding and cross-scale feature fusion are performed on single-channel X-ray images from various perspectives based on residual networks and feature pyramid networks. Shallow edge texture features, mid-level structural contour features, and deep semantic internal structure features of the X-ray images are extracted level by level and finally fused into two-dimensional perspective features, including: The X-ray image is fed into the ResNet residual coding network, and downsampled and encoded through convolutional layers, pooling layers and stacked residual blocks to output multi-scale basic features in sequence. A feature pyramid network is used to start from the deep basic features and perform bilinear upsampling operations from top to bottom to restore the deep basic features to the same size as the lower layer features. Then, it is fused with the basic features of the corresponding level through horizontal connection to obtain shallow edge texture features, mid-layer structural contour features and deep semantic internal structure features. Two-dimensional perspective features are obtained by fusing shallow edge texture features, mid-level structural contour features, and deep semantic internal structural features through channel splicing to achieve multi-scale information fusion.

3. The PCB board segmentation method based on X-rays and three-dimensional point clouds according to claim 2, characterized in that, Based on the RandLA-Net architecture, hierarchical random sampling and local neighborhood aggregation are performed on 3D point cloud data to obtain the spatial geometric features of 3D point clouds at each level, including: RandLA-Net is used as the backbone network for 3D point cloud feature encoding. Each layer of RandLA-Net performs random downsampling on the 3D point cloud data and randomly selects a portion of the point cloud from the current sampling layer as the sampling key point cloud set for the next layer. For any key point cloud in the current downsampling layer, a K-nearest neighbor search is used to select a neighboring point cloud, and the relative spatial offset of the neighboring area is calculated. The absolute coordinates of the key point cloud and the relative spatial offset are then concatenated to form the initial position features of the neighboring area. The initial location features of the neighborhood and the original point features are nonlinearly mapped using a shared MLP to obtain the neighborhood embedding feature vector; then max pooling is used to aggregate the neighborhood embedding feature vector to obtain the aggregated feature vector. The spatial geometric features of the 3D point cloud at each level are output after multi-level encoding.

4. The PCB board segmentation method based on X-rays and three-dimensional point clouds according to claim 3, characterized in that, A projection matrix is ​​used to project a 3D point cloud onto a 2D X-ray image plane to obtain the point cloud projection coordinates. Based on the point cloud projection coordinates, the ray propagation direction, ray path, path spatial density, ray path depth, and cumulative material absorption intensity are calculated to construct the physical information field corresponding to each 3D point cloud; including: Determine the spatial position of the X-ray source and the calibration projection matrix corresponding to each X-ray imaging viewpoint, traverse each three-dimensional point cloud in the three-dimensional point cloud data of the PCB board, and expand the rectangular coordinates of a single three-dimensional point cloud into homogeneous coordinates. Using the pre-calibrated projection matrices of each viewpoint, the homogenized 3D point cloud is transformed by perspective matrix to obtain the homogenized projection coordinates of the image. The homogeneous projection coordinates of the image are normalized by homogeneous division, and the original two-dimensional projection coordinates of the three-dimensional point cloud on the X-ray image at the current viewpoint are obtained. based on Given the spatial location of the ray source, construct a propagation direction vector from the ray source to the corresponding 3D point cloud for any 3D point cloud; and normalize the propagation direction vector using the L2 norm to obtain the propagation direction of the unit ray. The ray penetration length is calculated based on the Euclidean distance between the spatial location of the ray source and the 3D point cloud, and a continuous ray path is constructed based on the ray penetration length, starting from the spatial location of the ray source and along the unit ray propagation direction. Set a neighborhood distance threshold and define the shortest distance from the 3D point cloud to the continuous ray path space. Construct a ray path neighborhood point set and calculate the path space density based on the ratio of the number of neighborhood points to the ray penetration length. For each neighborhood 3D point cloud in the neighborhood point set of the ray path, solve for its corresponding depth parameter on the ray path, and sort them in ascending order of depth parameter from smallest to largest to obtain the ray path depth sorting sequence. Based on the material absorption coefficient and path element length of the neighborhood 3D point cloud, the cumulative material absorption intensity along the ray path is obtained by summing them. The original two-dimensional projection coordinates, unit ray propagation direction, ray penetration length, path spatial density, ray path depth sorting sequence, and material cumulative absorption intensity are integrated into the physical information field of each three-dimensional point cloud from various perspectives.

5. The PCB board segmentation method based on X-rays and three-dimensional point clouds according to claim 4, characterized in that, Based on the spatial geometric features of each 3D point cloud, predict the 2D projection offset, perform adaptive deformation sampling in the neighborhood of the base projection position, and match the 3D spatial structure to obtain 2D local features; including: The original two-dimensional projected coordinates of each three-dimensional point cloud are used as the reference anchor point position for two-dimensional feature sampling. A learnable offset prediction mapping function is constructed to drive the adaptive prediction of K sets of two-dimensional coordinate offsets with the spatial geometric features of the three-dimensional point cloud. A scale constraint factor will be introduced to superimpose the predicted offset onto the original two-dimensional projected coordinates, generate deformable actual sampling positions, and impose interval constraints on the sampling coordinates. In the Bilinear interpolation operators are used to perform sub-pixel feature sampling at each deformable sampling location on the two-dimensional viewpoint features; An attention mapping network is constructed to predict the weight score of each sampling position based on the spatial geometric features of the 3D point cloud, and the attention weight is obtained by Softmax normalization based on the weight score. The sub-pixel features obtained from multiple deformation samplings are weighted and aggregated based on attention weights to obtain two-dimensional local features.

6. The PCB board segmentation method based on X-rays and three-dimensional point clouds according to claim 5, characterized in that, The target occlusion rate for each viewpoint is constructed using the neighborhood point set of the ray path. Dynamic viewpoint confidence weights are then built based on the target occlusion rate, path spatial density, cumulative material absorption intensity, and ray penetration length. Two-dimensional local features are aggregated and weighted based on these dynamic viewpoint confidence weights to obtain multi-view fused two-dimensional global features; including: Using the neighborhood point set of the ray path, the front layer occlusion point and the rear layer occlusion point are distinguished along the X-ray propagation direction. The proportion of the number of forward occlusion points to the total number of neighborhood points is counted to construct the target occlusion rate of the view. The first method is constructed by combining target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length. The original confidence score function for 3D point clouds from a given perspective; The Softmax function is used to normalize the original confidence scores of each viewpoint to generate dynamic viewpoint confidence weights. Using the dynamic perspective confidence weight as the weighting coefficient, the two-dimensional local features obtained by adaptive deformation sampling under each single perspective are weighted and aggregated to obtain multi-view fused two-dimensional global features.

7. The PCB board segmentation method based on X-rays and three-dimensional point clouds according to claim 6, characterized in that, Multi-view fusion of 2D global features is mapped to 3D space and then fused with the spatial geometric features of 3D point clouds through a gating mechanism to generate cross-modal fused features; including: The multi-view fused two-dimensional global features are mapped to three-dimensional space through a linear projection layer, and the mapped multi-view fused two-dimensional global features are stitched together with the spatial geometric features of the three-dimensional point cloud to obtain the stitched features. A multilayer perceptron is used to construct a gating mapping function, which performs a nonlinear transformation on the spliced ​​features, and then generates channel-by-channel gating weights through Sigmoid activation. Based on channel-gated weights, a channel-by-channel Hadamard product weighted fusion mechanism is used to dynamically modulate the contribution ratio of the multi-view fusion of two-dimensional global features and spatial geometric features of three-dimensional point clouds, thereby obtaining cross-modal fusion features.

8. The PCB board segmentation method based on X-rays and three-dimensional point clouds according to claim 7, characterized in that, A point cloud decoding network is used to perform step-by-step channel semantic restoration and point cloud spatial resolution reconstruction on cross-modal fusion features, resulting in point-level structure segmentation of the PCB board, including: Based on cross-modal fusion features, the point cloud decoding network performs point cloud upsampling and expansion layer by layer from the deepest layer. It adopts an interpolation weighting method based on feature distance for the current sparse key point features, maps the high-level sparse features onto the encrypted point set, and performs spatial resolution reconstruction to obtain the encrypted point feature vector. For each layer of decoding, the encrypted point feature vector obtained by upsampling and interpolation is combined with the cross-modal fusion feature of the corresponding layer of the encoding network through channel concatenation and fusion to obtain the decoded feature vector; After restoring the original point cloud resolution through multi-layer decoding, a multi-layer perceptron (MLP) classification head is connected to perform semantic category mapping on the decoded feature vector of each 3D point cloud, resulting in a category prediction score vector for the 3D point cloud. The probability distribution of each category is obtained by using Softmax normalization; For each 3D point cloud, the category with the highest probability is taken as the final predicted semantic label to obtain the point-level structure segmentation result of the 3D point cloud data of the PCB board.

9. A PCB board segmentation system based on X-rays and three-dimensional point clouds, characterized in that, include: The data acquisition module is used to acquire 3D point cloud data, multi-view X-ray images, and projection matrices corresponding to each view of the PCB board. The two-dimensional view feature extraction module is used to perform multi-scale feature encoding and cross-scale feature fusion on single-channel X-ray images from various viewpoints based on residual networks and feature pyramid networks. It extracts shallow edge texture features, mid-level structural contour features and deep semantic internal structure features of X-ray images step by step, and finally fuses them into two-dimensional view features. The spatial geometric feature extraction module is used to perform hierarchical random sampling and local neighborhood aggregation on 3D point cloud data based on the RandLA-Net architecture to obtain the spatial geometric features of 3D point clouds at each level. The physical information field construction module is used to project three-dimensional point clouds onto a two-dimensional X-ray image plane using a projection matrix to obtain the projection coordinates of the point clouds, and to calculate the ray propagation direction, ray path, path spatial density, ray path depth and cumulative material absorption intensity based on the point cloud projection coordinates in order to construct the physical information field corresponding to each three-dimensional point cloud. The two-dimensional local feature extraction module is used to predict the two-dimensional projection offset based on the spatial geometric features of each three-dimensional point cloud, perform adaptive deformation sampling in the neighborhood of the basic projection position, and match the three-dimensional spatial structure to obtain two-dimensional local features. The multi-view fusion two-dimensional global feature extraction module is used to construct the target occlusion rate of each view using the neighborhood point set of the ray path, and to construct dynamic view confidence weights based on the target occlusion rate, path spatial density, material cumulative absorption intensity, and ray penetration length. Based on the dynamic view confidence weights, the two-dimensional local features are aggregated and weighted to obtain the multi-view fusion two-dimensional global features. The feature fusion module is used to map multi-view fused two-dimensional global features to three-dimensional space, and perform cross-modal fusion with the spatial geometric features of the three-dimensional point cloud through a gating mechanism to generate cross-modal fused features; The decoding module is used to perform channel semantic restoration and point cloud spatial resolution reconstruction step by step on cross-modal fusion features using a point cloud decoding network to obtain the point-level structure segmentation result of the PCB board.

10. An electronic device, comprising: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores a computer program that can be executed by at least one processor, characterized in that, when the computer program is executed by at least one processor, it implements the PCB board segmentation method based on X-rays and three-dimensional point clouds as described in any one of claims 1-8.

Images