Linear highlight overflow small target detection method based on image and point cloud feature fusion and electronic equipment

Abstract

The invention discloses a linear highlight overflow small target detection method based on image and point cloud feature fusion and electronic equipment. The method comprises the following steps: synchronously acquiring microscopic images and three-dimensional point cloud data at the same position; performing multi-scale visual feature extraction on the image, and performing geometric feature extraction based on multi-radius neighborhood analysis on the point cloud; projecting the extracted bimodal features to a unified aerial view space, and performing deep fusion through a bidirectional cross attention and adaptive gating fusion mechanism; generating a background suppression mask by using the geometric features of the point cloud so as to suppress background interference of a bonding pad, a substrate and the like; and inputting the processed features into an anchor-frame-free three-dimensional detection head, and directly regressing three-dimensional bounding box parameters of each bonding wire. According to the invention, through deep fusion and complementation of the image and the point cloud, industrial detection problems of highlight overflow, weak texture, sparse point cloud, complex background interference and the like are effectively solved, and high-precision and high-robustness three-dimensional positioning and measurement of the bonding wire are realized.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit detection technology, and in particular to a linear specular overflow small target detection method and electronic device based on image and point cloud feature fusion. Background Technology

[0002] Hybrid integrated circuits are widely used in high-reliability applications such as aerospace and precision electronics. Their internal interconnections typically rely on wire bonding technology. The diameter of the bonding wire is usually 20-30 micrometers, which places extremely high demands on the accuracy and stability of the detection system.

[0003] Current detection mainly relies on:

[0004] 1. Manual microscopic examination: low efficiency and poor stability;

[0005] 2. Algorithms based on single-modal images: highly sensitive to reflections, highlights, and occlusions, but lack three-dimensional information;

[0006] 3. Traditional point cloud feature algorithms: The bonding filaments are sparse and thin, and geometric information is easily lost after voxelization.

[0007] Existing technologies struggle to simultaneously achieve high precision, high robustness, and high efficiency in complex industrial environments.

[0008] Therefore, this invention provides a bonding wire detection method based on deep fusion of image and point cloud, which utilizes the complementarity between image texture and point cloud geometry to improve the robustness and accuracy of detection, especially in scenarios such as high reflectivity areas, uneven illumination, weak image texture, and local sparse point cloud, it can still achieve stable detection. Summary of the Invention

[0009] The technical solution of this invention to solve the above-mentioned technical problems is to provide a linear specular overflow small target detection method by fusing image and point cloud features, comprising the following steps:

[0010] Acquire microscopic images and 3D point cloud data from the same location;

[0011] Multi-scale feature extraction is performed on the microscopic image to generate multi-scale image features;

[0012] Multi-scale geometric feature extraction is performed on the three-dimensional point cloud data to generate point cloud geometric features;

[0013] The multi-scale features of the image are projected onto the bird's-eye view space to generate bird's-eye view features of the image.

[0014] The point cloud geometric features are projected onto the bird's-eye view space to generate point cloud bird's-eye view features;

[0015] The bird's-eye view features of the image and the bird's-eye view features of the point cloud are deeply fused to generate a fused bird's-eye view feature.

[0016] A background suppression mask is generated based on the point cloud geometric features, and the background suppression mask is used to suppress the background of the fused bird's-eye view features;

[0017] The fused bird's-eye view features after background suppression are input into the Anchor-Free 3D detection head to predict the 3D bounding box parameters of each bonding wire; the 3D bounding box parameters include center point coordinates, size, orientation, and confidence level.

[0018] Furthermore, the multi-scale feature extraction of the microscopic image includes:

[0019] Multi-scale image features are extracted using a convolutional neural network containing a four-level residual structure;

[0020] The multi-scale image features are enhanced and fused using a feature pyramid network;

[0021] The fused multi-scale features are weighted by a hierarchical attention mechanism to generate the final multi-scale image features.

[0022] Furthermore, the multi-scale geometric feature extraction of the 3D point cloud data includes:

[0023] For each point in the point cloud, search for its neighboring points at multiple preset radius scales, including three radii:

[0024] =50μm, used to capture local morphology of line structures;

[0025] =100μm, used to capture the shape stability scale of the bonding wire;

[0026] =200μm, used to capture the overall bending trend;

[0027] For each scale, the covariance matrix of the neighborhood point set is calculated and eigenvalue decomposition is performed to obtain the eigenvalues. The neighborhood covariance matrix is ​​then obtained.

[0028] ;

[0029] in, It sums the sums of all points in the neighborhood. The number of neighboring points is used to calculate the average. It is the outer product of vectors;

[0030] A geometric descriptor characterizing local geometric properties is calculated based on the eigenvalues, and the geometric descriptor includes linearity and flatness.

[0031] Geometric descriptors at multiple scales are spliced ​​together and fused using an attention mechanism to generate the point cloud geometric features.

[0032] Furthermore, the projection of multi-scale image features onto the bird's-eye view space using a dense UV-to-BEV projection method includes:

[0033] For each pixel in the image, based on the camera's intrinsic and extrinsic parameters, it is back-projected to the intersection of the world coordinate system and the reference plane at a specified height.

[0034] Map the intersection points to the bird's-eye view grid coordinates;

[0035] The image features corresponding to the pixels are filled into the bird's-eye view grid to generate the image bird's-eye view features.

[0036] Furthermore, the deep fusion employs a combination of bidirectional cross-attention and gating fusion, including:

[0037] The image bird's-eye view features and the point cloud bird's-eye view features are used as query vector and key vector, respectively, and bidirectional cross-attention calculation is performed to obtain point cloud enhancement features and image enhancement features, respectively.

[0038] Based on the image enhancement features and the point cloud enhancement features, an image confidence map and a point cloud confidence map are generated through a convolutional network;

[0039] Based on the image confidence map and the point cloud confidence map, the image enhancement features and the point cloud enhancement features are weighted and summed to generate the fused bird's-eye view features.

[0040] Furthermore, the generation of the background suppression mask based on point cloud geometric features includes:

[0041] Based on the linearity in the geometric features of the point cloud, identify linear structure regions and planar background regions;

[0042] Generate a mask corresponding to the bird's-eye view space, wherein the mask value corresponding to the linear structure region is set as an enhancement weight, and the mask value corresponding to the planar background region is set as a suppression weight;

[0043] The mask is used to perform element-wise weighting on the features of the fused bird's-eye view to achieve background suppression.

[0044] Furthermore, the Anchor-Free 3D detection head includes a classification branch and a regression branch;

[0045] The classification branch is used to predict the probability that each bird's-eye view grid contains a bonding wire center;

[0046] The regression branch is used to regress the parameters of the three-dimensional bounding box. The regression of the parameters includes: regressing the two-dimensional offset and height value of the center point and the size value using smooth L1 loss; and predicting the orientation angle using a classification-based and residual regression approach.

[0047] Furthermore, the method is optimized through end-to-end training, and the composite loss function used includes: classification loss for supervising the prediction of bond wire center points, regression loss for supervising the regression of 3D bounding box center points and dimensions, and orientation classification loss and orientation residual loss for supervising the prediction of orientation angles.

[0048] The total loss function is:

[0049] .

[0050] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0051] The beneficial effects of this invention are mainly reflected in the following aspects:

[0052] 1. Break through the bottlenecks of traditional visual inspection and achieve complementary advantages of multimodal approaches:

[0053] To address the common challenges in inspecting gold wire bonding wires for hybrid integrated circuits, such as highlight overflow, weak texture, occlusion, and sparse point cloud, this invention creatively integrates microscopic images with 3D point cloud data. Through a bidirectional cross-attention and adaptive gating fusion mechanism, deep modal complementarity is achieved: in areas where image information is unreliable due to reflections or texture loss, the precise geometric features of the point cloud reinforce the data; in areas where the point cloud is sparse or missing, the rich texture information of the image compensates for the loss. This complementarity fundamentally improves the system's perceptual robustness in complex industrial scenarios.

[0054] 2. Achieve high-precision 3D positioning and parameter analysis:

[0055] The method achieves deep fusion by uniformly projecting multi-scale features onto the bird's-eye view (BEV) space, effectively unifying the coordinate systems of the image and point cloud and ensuring spatial geometric consistency. Combined with point cloud multi-scale geometric descriptors (linearity, flatness) specifically designed for linear structures and the Anchor-Free detector head directly regressing 3D bounding box parameters (center point, size, orientation), it enables millimeter-level high-precision localization and quantization measurement of the spatial attitude and position of each bonding filament, providing a reliable 3D data foundation for subsequent quality assessments (such as curvature and collapse).

[0056] 3. Effectively suppresses background interference and improves the detection signal-to-noise ratio:

[0057] An innovative approach utilizes geometric features (such as linearity) extracted from point clouds to generate a background suppression mask (BSM). This mask effectively distinguishes the foreground (linear bonding wires) from the background (planar pads, substrate) and suppresses the response of background regions at the feature level. This allows the detection network to focus more intently on the target bonding wires themselves, significantly reducing false positives and false negatives caused by complex backgrounds such as strong pad reflections and circuit board textures, thereby improving the purity and confidence of the detection.

[0058] 4. The system is highly robust and adaptable to complex industrial environments:

[0059] The entire design fully considers the challenges of practical industrial applications. Multi-scale feature extraction enhances the algorithm's adaptability to bonding fibers of different sizes and shapes; background suppression mechanisms reduce environmental interference; end-to-end training and a composite loss function ensure model stability. Experiments show that even in production line environments with varying degrees of lighting changes, occlusion, foreign object interference, and equipment vibration noise, the system maintains stable and excellent detection performance.

[0060] 5. High operating efficiency, easy for industrial deployment and integration:

[0061] The network employs a fully convolutional design, with highly parallelized feature extraction and fusion processes. Final detection is performed on structurally regular BEV feature maps, supporting efficient GPU inference. Compared to traditional methods relying on complex post-processing or sliding windows, this solution's Anchor-Free detection head has a simple structure and fast computation speed. Furthermore, its modular design facilitates integration into existing automated optical inspection (AOI) equipment or vision-guided systems, meeting the industry's dual requirements for real-time performance and reliability. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0063] Figure 1 This is a flowchart of the linear specular overflow small target detection method based on image and point cloud feature fusion described in this invention;

[0064] Figure 2 This is a diagram illustrating the overall system architecture of the linear specular overflow small target detection method based on image and point cloud feature fusion as described in this invention.

[0065] Figure 3 This is the image multi-scale feature extraction network structure of the present invention;

[0066] Figure 4 This is a schematic diagram of point cloud multi-scale feature extraction according to the present invention;

[0067] Figure 5 This is a comparison image of point cloud BEV and image BEV, and a fused BEV feature map of the present invention.

[0068] Figure 6 This is a schematic diagram of the predicted output of the 3D detection head of the present invention;

[0069] Figure 7 This is a schematic diagram illustrating the image feature extraction process from shallow to deep layers according to the present invention.

[0070] Figure 8 This is a graph showing the total loss of this invention;

[0071] Figure 9 This is a graph showing the sub-loss of the present invention;

[0072] Figure 10 This is a learning rate scheduling curve diagram for the present invention;

[0073] Figure 11 This is a comparison chart of the ablation experiment results of this invention;

[0074] Figure 12 This is a visualization analysis chart of the results of this invention. Detailed Implementation

[0075] This invention proposes a linear specular overflow detection method and electronic device based on the fusion of image and point cloud features. It aims to effectively solve industrial detection problems such as specular overflow, weak texture, sparse point cloud and complex background interference through deep fusion and complementarity of image and point cloud.

[0076] The linear specular overflow small target detection method based on image and point cloud feature fusion proposed in this invention will be described below in specific embodiments:

[0077] Example 1:

[0078] In the technical solution of this embodiment, such as Figure 1 , Figure 2 As shown, a linear specular clipping small target detection method based on image and point cloud feature fusion includes the following steps:

[0079] S10: Acquire microscopic images and 3D point cloud data collected at the same location;

[0080] S20: Perform multi-scale feature extraction on the microscopic image to generate multi-scale image features;

[0081] S30: Perform multi-scale geometric feature extraction on the three-dimensional point cloud data to generate point cloud geometric features;

[0082] S40: Project the multi-scale features of the image onto the bird's-eye view space to generate image bird's-eye view features;

[0083] S50: Project the point cloud geometric features onto the bird's-eye view space to generate point cloud bird's-eye view features;

[0084] S60: Perform deep fusion of the image bird's-eye view features and the point cloud bird's-eye view features to generate fused bird's-eye view features;

[0085] S70: Generate a background suppression mask based on the point cloud geometric features, and use the background suppression mask to suppress the background of the fused bird's-eye view features;

[0086] S80: Input the fused bird's-eye view features after background suppression into the Anchor-Free 3D detection head to predict the 3D bounding box parameters of each bonding wire; the 3D bounding box parameters include center point coordinates, size, orientation and confidence level.

[0087] Furthermore, the multi-scale feature extraction of the microscopic image, such as... Figure 3 , Figure 4 The above includes:

[0088] Multi-scale image features are extracted using a convolutional neural network containing a four-level residual structure;

[0089] The multi-scale image features are enhanced and fused using a feature pyramid network;

[0090] The fused multi-scale features are weighted by a hierarchical attention mechanism to generate the final multi-scale image features.

[0091] Specifically, the system employs a multi-scale image feature extraction network based on an improved ResNet18, comprising: an initial convolutional layer of 7×7 convolution (Stride=2), 3×3 max pooling, batch normalization, and ReLU. This yields basic features at a resolution of 1 / 4 of the input image, used to capture edge information of the bonding fibers.

[0092] Fourth-order residual structure:

[0093]

[0094] like Figure 7 As shown, multi-scale features from low-level texture to high-level semantics are extracted;

[0095] Channel unification: To facilitate subsequent multi-scale fusion, the number of channels across all layers is unified to 256 (1×1 convolution), reducing computational cost and improving fusion efficiency. This yields standardized feature maps. .

[0096] FPN (Feature Pyramid Network) Enhancement: Through top-down paths and lateral connections, → → → The system is fused step by step, and two 3×3 convolutional refinement modules are added to each layer to eliminate feature conflicts and improve multi-scale consistency.

[0097] Level Attention: This refers to the process of enabling the network to automatically assign importance at different scales. , , , Global average pooling is performed, and the concatenated data is fed into a multilayer perceptron. Softmax outputs four weights, which are then weighted and upsampled to the same resolution across the four scales to obtain the final image enhancement features. .

[0098] Furthermore, the multi-scale geometric feature extraction of the 3D point cloud data includes:

[0099] For each point in the point cloud, search for its neighboring points at multiple preset radius scales, including three radii:

[0100] =50μm, used to capture local morphology of line structures;

[0101] =100μm, used to capture the shape stability scale of the bonding wire;

[0102] =200μm, used to capture the overall bending trend;

[0103] For each scale, the covariance matrix of the neighborhood point set is calculated and eigenvalue decomposition is performed to obtain the eigenvalues. The neighborhood covariance matrix is:

[0104] ;

[0105] in, It sums the sums of all points in the neighborhood. The number of neighboring points is used to calculate the average. It is the outer product of vectors;

[0106] A geometric descriptor characterizing local geometric properties is calculated based on the eigenvalues, and the geometric descriptor includes linearity and flatness.

[0107] Geometric descriptors at multiple scales are spliced ​​together and fused using an attention mechanism to generate the point cloud geometric features.

[0108] Specifically, geometric descriptors:

[0109] (1) Contribution rate of the first principal component: ;

[0110] (2) Linearity: ;

[0111] (3) Flatness: ;

[0112] These values ​​can distinguish between "wire structure (bonding wire)" and "planar structure (pad, circuit board)".

[0113] Multi-scale attention fusion:

[0114] The three-scale geometric features are concatenated and then processed by an MLP to generate three-scale weights, forming a fused geometric feature. .

[0115] Point cloud BEV projection:

[0116] Will Projected onto the BEV mesh using XYZ coordinates. Dense regions are treated with attention fusion; sparse regions with max pooling; and empty regions with neighborhood feature diffusion filling.

[0117] Output: Point cloud BEV feature map (H×W×256).

[0118] Furthermore, the projection of multi-scale image features onto the bird's-eye view space using a dense UV-to-BEV projection method includes:

[0119] For each pixel in the image, based on the camera's intrinsic and extrinsic parameters, it is back-projected to the intersection of the world coordinate system and the reference plane at a specified height.

[0120] Map the intersection points to the bird's-eye view grid coordinates;

[0121] The image features corresponding to the pixels are filled into the bird's-eye view grid to generate the image bird's-eye view features.

[0122] Furthermore, the deep fusion employs a combination of bidirectional cross-attention and gating fusion, including:

[0123] The image bird's-eye view features and the point cloud bird's-eye view features are used as query vector and key vector, respectively, and bidirectional cross-attention calculation is performed to obtain point cloud enhancement features and image enhancement features, respectively.

[0124] Based on the image enhancement features and the point cloud enhancement features, an image confidence map and a point cloud confidence map are generated through a convolutional network;

[0125] Based on the image confidence map and the point cloud confidence map, the image enhancement features and the point cloud enhancement features are weighted and summed to generate the fused bird's-eye view features.

[0126] Specifically, bidirectional cross-attention fusion: and They are mutually Query / Key / Value pairs:

[0127] Branch A: , Output point cloud enhanced features .

[0128] Branch B: , Output image enhancement features .

[0129] This approach allows for the completion of weakly textured areas in an image using point clouds, and sparsely textured areas in the point cloud using the image itself. This deep semantic interaction surpasses simple addition or splicing.

[0130] Gated fusion: Generated using convolutional networks:

[0131] (1) Image credibility;

[0132] (2) Point cloud credibility ( );

[0133] Final Fusion: ;

[0134] like Figure 5 As shown, each BEV pixel has a different fusion weight, achieving local adaptive fusion.

[0135] Furthermore, the generation of the background suppression mask based on point cloud geometric features includes:

[0136] Based on the linearity in the geometric features of the point cloud, identify linear structure regions and planar background regions;

[0137] Generate a mask corresponding to the bird's-eye view space, wherein the mask value corresponding to the linear structure region is set as an enhancement weight, and the mask value corresponding to the planar background region is set as a suppression weight;

[0138] The mask is used to perform element-wise weighting on the features of the fused bird's-eye view to achieve background suppression.

[0139] Specifically, the image will be affected by: strong reflections from the pads, background texture, and uneven lighting.

[0140] Construct a BSM using point cloud geometry.

[0141] ;

[0142] Image features are reweighted:

[0143] ;

[0144] Bonding wire region Mask → 1, image preserved; background pad Mask → 0, image suppressed. Improves the classification head's foreground / background discrimination.

[0145] Furthermore, the Anchor-Free 3D detection head includes a classification branch and a regression branch, such as... Figure 6 As shown;

[0146] The classification branch is used to predict the probability that each bird's-eye view grid contains a bonding wire center;

[0147] The regression branch is used to regress the parameters of the three-dimensional bounding box. The regression of the parameters includes: regressing the two-dimensional offset and height value of the center point and the size value using smooth L1 loss; and predicting the orientation angle using a classification-based and residual regression approach.

[0148] Specifically, the detection head input is: ;

[0149] The detection head contains two branches:

[0150] 1. Classification Header: Outputs the probability of whether each BEV mesh contains bonding wires.

[0151] 2. Regression Head: Outputs the coordinates (x, y, z) of the bond fiber center, dimensions (length, width, height), and orientation (classification + residual regression). Bin-based orientation prediction is used.

[0152] 3. Inference stage: Filter peaks based on classification probability, decode 3D bounding boxes, remove duplicates using NMS, and output the final detection results.

[0153] Furthermore, the method is optimized through end-to-end training, and the composite loss function used includes: classification loss for supervising the prediction of bond wire center points, regression loss for supervising the regression of 3D bounding box center points and dimensions, and orientation classification loss and orientation residual loss for supervising the prediction of orientation angles.

[0154] The total loss curve is as follows Figure 8 As shown, the sub-loss curves are as follows: Figure 9 As shown, the total loss function is:

[0155] .

[0156] Specifically, the supervision includes three parts:

[0157] Classification Loss: Focal Loss ;

[0158] Regression Loss: Smooth L1: ;

[0159] Directional loss: ;

[0160] Directional classification cross-entropy;

[0161] Directional residual Smooth L1;

[0162] Total loss: .

[0163] Its learning rate is as follows Figure 10 As shown.

[0164] The system's advantages are:

[0165] 1. Image + point cloud complementarity solves problems of weak texture, highlights and occlusion.

[0166] 2. Dense UV→BEV projection ensures geometric consistency.

[0167] 3. Point cloud PCA encoding is specifically designed for linear structures.

[0168] 4. Cross-attention allows the two modalities to complement each other in depth.

[0169] 5. Background suppression improves foreground response ratio.

[0170] 6. Anchor-Free detectors directly predict 3D parameters.

[0171] 7. Suitable for industrial deployment (simple structure, fast inference speed)

[0172] Beneficial effects:

[0173] 1. Overcomes the problems of highlight clipping and weak textures.

[0174] The weak texture areas of the image are reinforced by the linearity of the point cloud.

[0175] 2. Solve the problem of sparse point clouds

[0176] Sparse areas of the point cloud are compensated by image texture.

[0177] 3. Achieve high-precision 3D positioning

[0178] By fusing multimodal BEVs, detection accuracy is significantly improved.

[0179] 4. Highly robust and adaptable to industrial environments

[0180] It performs excellently even under conditions of obstruction, reflection, and noise.

[0181] 5. Highly efficient operation, suitable for industrial deployment.

[0182] The BEV architecture supports fast convolutional inference.

[0183] Verification experiment:

[0184] To verify the effectiveness of the proposed method, it is compared with three baseline methods:

[0185] Image-only: Detection is performed using only image BEV features, without using point cloud information;

[0186] Point-cloud-only: Detection is performed using only point cloud BEV features, without using image information;

[0187] Naive Fusion: Simply concatenates the BEV features of the two modalities and feeds them directly into the detection head, without including cross attention, gating fusion, and BSM modules;

[0188] Ours: This patent is a complete multimodal BEV fusion detection framework.

[0189] The table below presents the quantitative comparison results on the test set. Precision, Recall, and F1-score are all calculated under the matching criterion of IoU ≥ 0.3. Mean IoU is the average intersection-union ratio of all matched predicted boxes and ground truth boxes. A one-to-one matching strategy (matching from high to low IoU) is used to match predicted boxes and ground truth boxes to avoid statistical bias caused by repeated matching.

[0190]

[0191] From the results of single-modal analysis, Image-only is slightly better than Point-cloud-only. The main reason is that the bonded filament point cloud is sparse and has a low proportion. Although the geometric information is robust to illumination, it is limited by the sampling density in terms of the expression of target integrity and continuity. In contrast, the image can still provide relatively stable edge cues in the local texture visible area.

[0192] Comparative experiments with existing models:

[0193] To fully verify the advancement of the proposed method, this invention selects representative methods covering single-modal image detection, single-modal point cloud detection, and multi-modal BEV fusion detection as comparison objects. Comparative experiments are conducted under the same dataset partitioning and unified evaluation metrics. All comparison methods use the same training / validation / test partitioning and annotation format, and the relationship between input modality and sensor calibration is kept consistent. The number of training epochs is consistent with the early stopping strategy, and the learning rate and weight decay are searched within the same range to ensure the fairness of the comparison.

[0194] For image-based single-modal detection, YOLOv8 is selected as the baseline. YOLOv8 is a widely used and stable target detection model in the current field of industrial vision inspection, and it can effectively represent the detection capabilities of pure image methods in complex scenes. It should be noted that YOLOv8 is a two-dimensional image planar detection method, and its output is a two-dimensional detection result in the image coordinate system. To maintain consistency with the 3D–2D joint annotation system in this paper, based on the system calibration relationship and point cloud height prior, the two-dimensional detection results of YOLOv8 are restored to three-dimensional space, and matched and evaluated with the 3D annotation results in real three-dimensional physical space. This ensures a fair comparison of different methods under a unified three-dimensional evaluation standard.

[0195] PointPillars was selected as the baseline method for single-modal point cloud detection. PointPillars is a typical point cloud target detection framework based on BEV representation. It achieves efficient 3D target detection by encoding disordered point clouds into regular columnar BEV features and performing convolution processing. This method can reflect the role of pure geometric information in bonded filament detection tasks well. However, due to the slender structure of the bonded filaments and the sparse distribution of the point cloud, relying solely on point cloud information still has limitations in terms of target completeness and recall. BEVFusion was selected as the comparison method for multimodal BEV fusion detection. BEVFusion is a representative BEV space fusion framework in the current multi-sensor perception field. It achieves complementary representation of multimodal information by jointly modeling image and point cloud features in BEV space. This method can significantly improve detection performance, but its fusion strategy adopts an equivalence modeling approach for the two modalities to a certain extent, without explicitly considering the reliability differences between different modalities in complex industrial scenarios.

[0196] In contrast, our method further introduces point cloud geometric priors in the BEV space to guide the attention distribution of image features, and achieves adaptive fusion of multimodal features through a gating mechanism, thereby achieving more stable detection performance in typical industrial challenging scenarios such as strong reflection, weak texture and sparse point clouds.

[0197]

[0198] The proposed method achieves state-of-the-art results in terms of Precision, Recall, F1-score, and MeanIoU. Experimental results fully validate the effectiveness and engineering applicability of the proposed method in complex industrial scenarios.

[0199] The main reason why BEVFusion outperforms the simple stitching fusion NaiveFusion on this task is that its method design and training assumptions are more biased towards large-scale perception scenarios such as automotive applications: the target scale is larger, the point cloud density is higher, and the spatial structure is more stable. Bonded wire detection, on the other hand, is characterized by extremely small targets, sparse point clouds, very low proportions, and significant high reflectivity interference. This makes cross-modal alignment and fusion weight learning more susceptible to interference from sparse points and visual specular noise when directly transferring the automotive multimodal BEV fusion framework, leading to increased missed detections and decreased localization consistency. In contrast, the proposed method, through cross-modal interaction and gated fusion guided by point cloud geometric priors, can more effectively suppress unreliable visual responses and improve the detection stability of sparse geometric regions.

[0200] Ablation Experiments: To verify the effectiveness of each module, this study progressively removed key modules for control experiments. Except for the corresponding modules, the network structure, training strategy, and hyperparameter settings remained consistent throughout the ablation experiments to ensure that performance differences originated from the removed modules themselves. The ablation experiment bar charts compare the results. Figure 11 As shown.

[0201] Ablation term: no cross-attention, only splicing and fusion;

[0202]

[0203] Conclusion: Cross-attention effectively promotes the complementarity of image and point cloud depth information, which greatly improves localization accuracy and recall.

[0204] Ablation term: No point cloud prior, using bidirectional equal cross attention;

[0205]

[0206] Conclusion: Geometric information in point cloud BEV is more stable, and using it as a query to guide attention distribution can effectively reduce misleading of highlight areas in images.

[0207] (3) Ablation term: Background suppression mask (BSM) is not used;

[0208]

[0209] Conclusion: BSM effectively reduces erroneous responses in the background pad region and improves the foreground discrimination ability of the classifier.

[0210] (4) Ablation term: No gated fusion is used, simple weighting is applied;

[0211]

[0212] Conclusion: Gated networks can adaptively select advantageous regions of images or point clouds, making feature fusion more accurate and spatially consistent.

[0213] Visualization results analysis:

[0214] To intuitively evaluate the impact of each key module on bonded wire detection performance, this paper visually compares the detection results of the complete model and different ablation configurations in two-dimensional images from the same perspective. The complete model can stably and continuously detect bonded wire structures against complex backgrounds, and the predicted bounding boxes maintain good consistency in spatial location, scale, and orientation. Figure 12As shown in Figure (a), after removing the cross-attention mechanism, some bond wire detection results showed breakage and missed detections, indicating that simple feature fusion is insufficient to fully model the deep correlation between multimodal features. When using equivalence cross-attention without point cloud priors, the detection results showed a certain offset in position and orientation estimation, indicating that geometric priors play an important role in constraining the attention distribution, as shown in Figure (b). After removing the background suppression mask, the false detections in the pad and background regions increased significantly, verifying the effectiveness of point cloud geometric features in foreground / background differentiation, as shown in Figure (c). When the gated fusion mechanism was removed, the detection boxes exhibited scale and position instability in local regions, further indicating that the adaptive fusion strategy helps improve the spatial consistency of multimodal features, as shown in Figure (d). The above visualization results are consistent with the quantitative ablation experiment conclusions, qualitatively verifying the effectiveness of each module in the proposed method.

[0215] Through numerous comparative experiments, the point cloud prior-guided cross-attention, gating fusion mechanism, point cloud geometry-based background suppression, and BEV2.5D detection and analytical recovery framework proposed in this study demonstrate significantly better detection capabilities than traditional single-modal methods and simple fusion methods in complex industrial scenarios. It is particularly suitable for dealing with challenging industrial scenarios such as high reflectivity, weak texture, and sparse point clouds.

[0216] During the model training phase, a joint 3D-2D annotation method is used to annotate the bonding wire targets. Specifically, firstly, the spatial position, orientation, and geometric dimensions of the bonding wires are annotated in the real 3D physical space; then, based on the system calibration relationship, the 3D annotation results are projected onto the 2D image plane and the bird's-eye view plane, respectively, to generate supervisory information for image features and bird's-eye view features, thereby supporting the detection and 3D parsing of multimodal features in the bird's-eye view space.

[0217] Example 2:

[0218] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in Embodiment 1.

[0219] Example 3:

[0220] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in Example 1.

[0221] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting small targets with linear specular overflow by fusing image and point cloud features, characterized in that, Includes the following steps: Acquire microscopic images and 3D point cloud data from the same location; Multi-scale feature extraction is performed on the microscopic image to generate multi-scale image features; Multi-scale geometric feature extraction is performed on the three-dimensional point cloud data to generate point cloud geometric features; The multi-scale features of the image are projected onto the bird's-eye view space to generate bird's-eye view features of the image. The point cloud geometric features are projected onto the bird's-eye view space to generate point cloud bird's-eye view features; The bird's-eye view features of the image and the bird's-eye view features of the point cloud are deeply fused to generate a fused bird's-eye view feature. A background suppression mask is generated based on the point cloud geometric features, and the background suppression mask is used to suppress the background of the fused bird's-eye view features; The fused bird's-eye view features after background suppression are input into the Anchor-Free 3D detection head to predict the 3D bounding box parameters of each bonding wire; the 3D bounding box parameters include center point coordinates, size, orientation, and confidence level.

2. The method according to claim 1, characterized in that, The multi-scale feature extraction of the microscopic image includes: Multi-scale image features are extracted using a convolutional neural network containing a four-level residual structure; The multi-scale image features are enhanced and fused using a feature pyramid network; The fused multi-scale features are weighted by a hierarchical attention mechanism to generate the final multi-scale image features.

3. The method according to claim 1, characterized in that, The multi-scale geometric feature extraction of the 3D point cloud data includes: For each point in the point cloud, search for its neighboring points at multiple preset radius scales, including three radii: =50μm, used to capture local morphology of line structures; =100μm, used to capture the shape stability scale of the bonding wire; =200μm, used to capture the overall bending trend; For each scale, the covariance matrix of the neighborhood point set is calculated and eigenvalue decomposition is performed to obtain the eigenvalues. The neighborhood covariance matrix is: ； in, It sums the sums of all points in the neighborhood. The number of neighboring points is used to calculate the average. It is the outer product of vectors; A geometric descriptor characterizing local geometric properties is calculated based on the eigenvalues, and the geometric descriptor includes linearity and flatness. Geometric descriptors at multiple scales are spliced ​​together and fused using an attention mechanism to generate the point cloud geometric features.

4. The method according to claim 1, characterized in that, The projection of multi-scale image features onto the bird's-eye view space using a dense UV-to-BEV projection method includes: For each pixel in the image, based on the camera's intrinsic and extrinsic parameters, it is back-projected to the intersection of the world coordinate system and the reference plane at a specified height. Map the intersection points to the bird's-eye view grid coordinates; The image features corresponding to the pixels are filled into the bird's-eye view grid to generate the image bird's-eye view features.

5. The method according to claim 1, characterized in that, The deep fusion employs a combination of bidirectional cross-attention and gated fusion, including: The image bird's-eye view features and the point cloud bird's-eye view features are used as query vector and key vector, respectively, and bidirectional cross-attention calculation is performed to obtain point cloud enhancement features and image enhancement features, respectively. Based on the image enhancement features and the point cloud enhancement features, an image confidence map and a point cloud confidence map are generated through a convolutional network; Based on the image confidence map and the point cloud confidence map, the image enhancement features and the point cloud enhancement features are weighted and summed to generate the fused bird's-eye view features.

6. The method according to claim 1, characterized in that, The background suppression mask generated based on point cloud geometric features includes: Based on the linearity in the geometric features of the point cloud, identify linear structure regions and planar background regions; Generate a mask corresponding to the bird's-eye view space, wherein the mask value corresponding to the linear structure region is set as an enhancement weight, and the mask value corresponding to the planar background region is set as a suppression weight; The mask is used to perform element-wise weighting on the features of the fused bird's-eye view to achieve background suppression.

7. The method according to claim 1, characterized in that, The Anchor-Free 3D detection head includes a classification branch and a regression branch; The classification branch is used to predict the probability that each bird's-eye view grid contains a bonding wire center; The regression branch is used to regress the parameters of the three-dimensional bounding box, and the regression of the parameters includes: using smooth L1 loss to regress the two-dimensional offset and height values ​​of the center point, as well as the size value; The orientation angle is predicted using a classification and residual regression approach.

8. The method according to claim 1, characterized in that, The method is optimized through end-to-end training, and the composite loss function used includes: classification loss for supervising the prediction of bond wire center points, regression loss for supervising the regression of 3D bounding box center points and dimensions, and orientation classification loss and orientation residual loss for supervising the prediction of orientation angles. The total loss function is: 。 9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

Images