Machine vision-based method and system for identifying assembly defects of btb terminals

Abstract

This application relates to the field of artificial intelligence and discloses a machine vision-based method and system for identifying defects in BTB terminal assembly. It aims to address the problems of low detection rate, high false alarm rate, and poor adaptability of existing technologies due to their reliance on a single visual modality and fixed feature rules, resulting in defects that are small, hidden, or complex. The method includes: simultaneously acquiring two-dimensional bright-field images, three-dimensional point cloud data, and multi-angle polarized light images of the BTB terminal; extracting two-dimensional texture, three-dimensional morphology, and polarization stress features through a parallel multi-scale network; deeply fusing heterogeneous features using a cross-modal attention mechanism; generating a pixel-level defect probability map via a defect segmentation decoder, and outputting a mask image identifying the defect location, contour, and category. This application, through the fusion of multimodal perception and deep learning, significantly improves the defect detection rate and stability, reduces missed detections and false alarms, and supports rapid adaptation to new product models.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a method and system for identifying defects in BTB terminal assembly based on machine vision. Background Technology

[0002] As a core component for electrical connections and signal transmission between multilayer circuit boards in electronic devices, the assembly quality of BTB connectors directly affects the reliability and stability of the entire product. With the continuous development of high-density integration and miniaturization in consumer electronics and automotive electronics, the pin pitch of BTB terminals has shrunk to 0.3 mm or even smaller, placing increasingly stringent requirements on assembly precision. In automated production lines, any minute misalignment, tilting, bending, missing, or surface damage to terminals can lead to poor contact, signal attenuation, or even functional failure. Therefore, efficient and accurate defect detection of BTB terminals during the manufacturing process has become a crucial step in ensuring the yield rate of electronic products.

[0003] Currently, machine vision-based defect detection technology is widely used in the quality inspection of BTB terminals due to its advantages such as non-contact operation, high efficiency, and ease of integration. Traditional methods mainly rely on manually set image processing workflows, such as extracting regions of interest through threshold segmentation, and then combining edge detection, template matching, or geometric feature analysis to identify defects. However, practical applications face multiple challenges. First, BTB terminals have a highly dense structure and strong reflectivity, making them susceptible to uneven lighting, lens distortion, and background interference. Second, the differences in defect morphology are subtle, and traditional threshold segmentation or template matching methods struggle to balance generalization and stability. Third, the production line environment is complex and variable; camera shake, product batch differences, and even slight variations in assembly posture can lead to false positives or false negatives.

[0004] In existing technologies, most BTB defect identification solutions still rely on manually set feature rules or shallow image processing procedures, lacking the ability to deeply perceive the three-dimensional spatial posture and microscopic deformation of terminals. While some methods incorporating convolutional neural networks have improved recognition accuracy, model training depends on a large number of labeled samples, and the decision-making logic is difficult to interpret, resulting in insufficient adaptability when facing new defects or production line changes. Furthermore, existing systems typically design image acquisition, preprocessing, feature extraction, and judgment modules in a fragmented manner, failing to build an end-to-end collaborative optimization mechanism. This leads to high response latency, high false alarm rates, and high maintenance costs in high-speed, high-precision scenarios. Especially in fields with stringent reliability requirements such as consumer electronics and automotive electronics, there is an urgent need for a novel detection solution that can integrate multi-source information and possess adaptive capabilities. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a method and system for identifying defects in BTB terminal assembly based on machine vision, which aims to overcome the technical bottlenecks in the prior art, which are low recognition rate, high false alarm rate and poor adaptability to new product models, due to reliance on a single visual modality and the use of fixed feature rules.

[0006] To address the aforementioned technical problems, this invention provides a machine vision-based method for identifying defects in BTB terminal assembly. First, a multimodal imaging system is constructed to simultaneously acquire composite image information of the BTB terminal under test under bright field, three-dimensional structured light, and multi-angle polarized light illumination. Next, a parallel multi-scale feature extraction network is designed to deeply mine unique defect representations from image data of different modalities, including two-dimensional texture details, three-dimensional spatial morphology, and surface and subsurface stress distribution of the material. Finally, a pioneering cross-modal attention fusion mechanism is introduced to deeply couple and enhance the extracted heterogeneous features, generating a fusion feature representation with high discriminative power for defects, and based on this, pixel-level accurate segmentation and classification of defects are achieved.

[0007] On one hand, the present invention provides a machine vision-based method for identifying defects in BTB terminal assembly, which includes the following steps: S1, acquire multimodal image data of the BTB terminal assembly under test. The multimodal image data includes two-dimensional high-resolution bright-field images, three-dimensional point cloud data, and polarized light images at at least three polarization angles. S2 performs parallel multi-scale feature extraction on multimodal image data to obtain two-dimensional visual features characterizing the surface texture and contour of the BTB terminal assembly under test, three-dimensional geometric features characterizing its spatial geometry, and polarization optical features characterizing its material stress and micro-scratches. S3 constructs a cross-modal attention fusion network, inputting two-dimensional visual features, three-dimensional geometric features, and polarization optical features into the network. Through feature interaction between modalities and adaptive weight adjustment, a deeply fused multimodal feature vector is generated. S4, input the deep fused multimodal feature vector into a defect segmentation decoder, and the decoder outputs a pixel-level defect probability map corresponding to the size of the original image; S5. Based on the comparison between the probability value of each pixel in the defect probability map and the preset defect judgment threshold, generate the final defect mask image, and mark the specific location, outline, and preset defect category of the defect in the defect mask image.

[0008] Preferably, acquiring multimodal image data of the BTB terminal assembly under test specifically includes: A telecentric lens industrial camera with a built-in coaxial light source is used to acquire two-dimensional high-resolution bright-field images of the BTB terminal assembly under test. The telecentric lens ensures that the image is free of geometric distortion, and the coaxial light source provides uniform and shadowless illumination. A series of pre-coded stripe patterns are projected onto the surface of the BTB terminal assembly under test using a structured light module. An industrial camera simultaneously acquires distorted stripe images at different projection angles. The distorted stripe images are analyzed using phase deflection or triangulation algorithms to reconstruct high-density three-dimensional point cloud data of the surface of the BTB terminal assembly under test. A rotatable analyzer is set in front of the lens of an industrial camera, and a polarizer that rotates synchronously with it is set in the optical path of the coaxial light source. The polarizer and analyzer are controlled to rotate synchronously to four polarization angles: 0 degrees, 45 degrees, 90 degrees and 135 degrees. One frame of image is acquired at each angle to obtain a polarized light image.

[0009] Furthermore, after acquiring the multimodal image data, the method also includes a data preprocessing step, specifically including: Spatial registration is performed on 2D high-resolution bright-field images, 3D point cloud data, and polarized light images to ensure strict alignment of pixel coordinate systems for all modal data. Spatial registration is achieved through pre-calibrated camera intrinsic and extrinsic parameters, as well as the relative pose transformation matrix between the structured light module and the camera. 3D point cloud data is converted into 2D depth maps through depth rendering or orthographic projection, so that its data structure is consistent with that of 2D images; The polarized light images at four polarization angles are combined into a single multi-channel Stokes vector parameter image. Each channel of the Stokes vector parameter image corresponds to a Stokes vector. With four components, this image can fully characterize the polarization state information of light waves.

[0010] Preferably, parallel multi-scale feature extraction of multimodal image data includes: Three independent encoder branches with feature pyramid network structures are constructed, corresponding to the two-dimensional high-resolution bright-field image, the two-dimensional depth map, and the Stokes vector parameter image, respectively; Each encoder branch consists of a series of convolutional layers, pooling layers, and activation function layers stacked together, outputting feature maps of different resolutions at different depth levels of the network. Deep feature maps contain global, semantic information, while shallow feature maps contain local, detailed information, thus forming a multi-scale representation of two-dimensional visual features, three-dimensional geometric features, and polarization optical features.

[0011] Furthermore, a cross-modal attention fusion network is constructed to deeply couple and enhance the information of 2D visual features, 3D geometric features, and polarization optical features. Specifically, this is implemented as follows: Design a cascaded cross-modal interactive attention module, which contains multiple attention computation units; In the first-level attention calculation, two-dimensional visual features are used as the query tensor, and three-dimensional geometric features are used as the key tensor and value tensor. An attention weight matrix is ​​generated by calculating the similarity between the query tensor and the key tensor, and the value tensor is weighted by this weight matrix, thereby realizing the guidance and focus of three-dimensional geometric structural information on two-dimensional texture features. In the second-level attention calculation, the output of the first-level attention calculation is used as a new query tensor, and the polarization optical features are used as the bond tensor and value tensor. The above attention weighting process is repeated, thereby integrating the material stress and micro-defect information into the feature representation that has been enhanced by geometric information. Through the above cascading process, a deep fusion multimodal feature vector that fully absorbs the complementary information of the three modalities is finally generated.

[0012] Preferably, the defect segmentation decoder adopts the decoding path in a U-shaped network structure, the structure of which is symmetrical with the encoder branches; The decoder gradually restores the spatial resolution of the feature map through a series of upsampling layers or deconvolution layers; After each upsampling step, the shallow feature map from the corresponding level of the encoder is concatenated with the currently upsampled feature map through skip connections. This is intended to fuse high-level semantic information with low-level detail information and prevent the loss of accurate defect boundary information during the decoding process. In the last layer of the decoder, a convolutional layer with a kernel size of one by one and a logistic regression activation function are used to map the multi-channel feature map into a single-channel defect probability map with values ​​between zero and one.

[0013] On the other hand, the present invention also provides a machine vision-based BTB terminal assembly defect identification system, which includes: A multimodal image data acquisition module is used to acquire two-dimensional high-resolution bright-field images, three-dimensional point cloud data, and multi-angle polarized light images of the BTB terminal assembly under test. A data preprocessing module, connected to the multimodal image data acquisition module, is used to perform spatial registration, data format conversion, and polarization information synthesis on the acquired multimodal image data; A multi-scale feature extraction module, connected to the data preprocessing module, contains three parallel encoders based on feature pyramid networks, used to extract two-dimensional visual features, three-dimensional geometric features, and polarization optical features from the preprocessed multimodal data, respectively. A cross-modal feature fusion module is connected to a multi-scale feature extraction module. Its core is a cascaded cross-modal interactive attention network, which is used to deeply fuse three heterogeneous features to generate a unified multimodal feature representation. A defect segmentation and decoding module is connected to a cross-modal feature fusion module. It adopts a decoder structure symmetrical to the encoder and uses skip connections to fuse multi-level information to convert a unified multimodal feature representation into a pixel-level defect probability map. A defect determination and output module, connected to the defect segmentation and decoding module, is used to compare the defect probability map with a preset threshold and generate a final defect mask image that identifies the defect location, outline and category.

[0014] Preferably, the multimodal image data acquisition module specifically includes: A megapixel industrial camera equipped with a telecentric lens; A ring-shaped light-emitting diode bright field light source coaxially positioned with the optical axis of an industrial camera; A structured light projection unit, whose optical axis forms a preset angle with the optical axis of an industrial camera, is used to project a coded pattern onto the area to be measured. A polarization optical assembly consisting of a polarizer and an analyzer. The polarizer is located in front of the bright field light source, and the analyzer is located in front of the telecentric lens. Both are driven by a precision stepper motor and can rotate synchronously. A five-axis motion platform is used to carry and precisely position the BTB terminal assembly to be tested.

[0015] Furthermore, the multi-scale feature extraction module, cross-modal feature fusion module, defect segmentation and decoding module, and defect determination and output module are all deployed in an industrial computing terminal. The industrial computing terminal includes a central processing unit, a graphics processing unit, a high-speed memory, and a non-volatile memory for storing neural network model parameters and processing programs. The graphics processing unit is dedicated to performing parallel computing tasks for the neural network.

[0016] In summary, this application includes at least one of the following beneficial technical effects: 1. By integrating information from three modes—two-dimensional bright field, three-dimensional geometry, and polarization optics—this invention can comprehensively perceive the state of BTB terminal assemblies. It can not only identify traditional macroscopic geometric defects such as bending and missing parts, but also effectively detect pin coplanarity deviations that are difficult to distinguish in two-dimensional images, as well as internal stresses in plastic housings, injection molding defects, and minute scratches on the surface of metal pins that cannot be detected by conventional optics. This greatly improves the coverage and accuracy of defect detection and significantly reduces the missed detection rate.

[0017] 2. The cross-modal attention fusion mechanism adopted in this invention enables the model to automatically learn the intrinsic correlation between information from different modalities. It uses the strong features of one modality to enhance and calibrate the weak or fuzzy features of another modality, thereby effectively suppressing interference caused by factors such as changes in illumination, surface reflection, or minor stains in a single modality. This achieves stability against environmental changes and normal process fluctuations, and significantly reduces the false alarm rate.

[0018] 3. The end-to-end defect segmentation model based on deep learning in this invention eliminates the cumbersome manual feature design and threshold setting process in traditional methods. For new BTB terminal models, only a small number of labeled samples are needed for incremental training or fine-tuning of the model to quickly adapt, demonstrating extremely high flexibility and scalability, and shortening the detection deployment cycle for new product introduction.

[0019] 4. This invention outputs pixel-level defect mask images, which not only determine whether there are defects, but also accurately mark the shape, size, location and type of defects, providing detailed and intuitive data support for subsequent quality traceability, process improvement and automated rework, realizing a leap from simple pass / fail judgment to in-depth defect diagnosis and analysis. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention.

[0021] Figure 2 This is a schematic diagram illustrating the core principle framework of the cross-modal attention fusion mechanism in this invention.

[0022] Figure 3 This is a flowchart illustrating the logical flow of the multimodal image data acquisition and preprocessing stage in this invention. Detailed Implementation

[0023] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of specific embodiments based on the present invention is provided in conjunction with the accompanying drawings and preferred embodiments.

[0024] This invention provides a machine vision-based method and system for identifying defects in BTB terminal assembly. It aims to address the technical bottlenecks in existing technologies, which suffer from low recognition rates, high false alarm rates, and poor adaptability to new product models due to reliance on a single visual modality and fixed feature rules. Simultaneously, by constructing a multimodal imaging system, it simultaneously acquires composite image information of the BTB terminal under test under bright field, three-dimensional structured light, and multi-angle polarized light illumination. Furthermore, it designs a parallel multi-scale feature extraction network and a cross-modal attention fusion mechanism to achieve pixel-level accurate segmentation and classification of defects.

[0025] A machine vision-based method for identifying defects in BTB terminal assembly includes the following steps: S1, acquire multimodal image data of the BTB terminal assembly under test; S2 performs parallel multi-scale feature extraction on multimodal image data; S3, construct a cross-modal attention fusion network to generate deeply fused multimodal feature vectors; S4, input the feature vector into the defect segmentation decoder and output a pixel-level defect probability map; S5. Generate the final defect mask image by comparing the defect probability map with a preset threshold.

[0026] In step S1, multimodal image data of the BTB terminal assembly under test is acquired. The multimodal image data includes two-dimensional high-resolution bright-field images, three-dimensional point cloud data, and polarized light images at at least three polarization angles.

[0027] In practice, a two-dimensional high-resolution bright-field image is acquired by a telecentric lens industrial camera with a built-in coaxial light source. The optical design of the telecentric lens ensures that there is no perspective distortion during the imaging process, and the size of all objects in the image does not change with their distance from the lens, thereby ensuring the accuracy of geometric dimension measurement of small structures such as terminal pins.

[0028] The coaxial light source consists of a high-brightness light-emitting diode array. Its light is incident perpendicularly onto the surface of the object under test through a beam splitter and reflected back to the camera sensor along the original path, providing uniform and shadow-free illumination conditions and effectively suppressing overexposure or information loss caused by specular reflection on the surface of the metal pins.

[0029] Meanwhile, a series of sinusoidal stripe patterns pre-coded by the system are projected onto the surface of the BTB terminal assembly under test through a structured light module. The structured light module is composed of a digital micromirror device or a liquid crystal spatial light modulator and can project a stripe sequence with continuously changing phase at a frame rate of not less than 60 Hz.

[0030] The industrial camera and the structured light module are strictly synchronized. A frame of distorted fringe image is acquired at each fringe phase. After acquisition, the distorted fringe image is analyzed by phase deflection algorithm or triangulation principle.

[0031] The phase deflection algorithm is based on the mapping relationship between the fringe phase and the surface slope, and is suitable for highly reflective surfaces; triangulation relies on the known baseline distance and angle between the camera and the projection unit, and calculates the depth by matching corresponding points.

[0032] Both methods can reconstruct high-density three-dimensional point cloud data of the surface of the BTB terminal assembly under test. The point cloud density is no less than one hundred points per square millimeter, and the spatial resolution can reach the micrometer level, which is sufficient to distinguish three-dimensional geometric defects such as pin coplanarity deviation and plastic shell warping.

[0033] In addition, a rotatable analyzer is set in front of the lens of the industrial camera, and a polarizer that rotates synchronously with it is set in the optical path of the coaxial light source. Both the polarizer and the analyzer are driven by high-precision stepper motors, and the two maintain a strict synchronous rotation relationship. The polarizer and the analyzer are controlled to rotate synchronously to four polarization angles: zero degrees, forty-five degrees, ninety degrees and one hundred and thirty-five degrees. At each angle, one frame of image is acquired, thereby obtaining four polarized light images.

[0034] Polarized light images can reflect the stress distribution, micro-scratches, injection flow marks, and other defect information that is difficult to capture by conventional bright-field images, such as those on the surface and subsurface of materials.

[0035] The physical basis of polarization imaging is that when unpolarized light is incident on the surface of a material, the polarization state of the reflected or transmitted light will change due to the birefringence, surface roughness and internal stress of the material. By measuring the polarization of light at multiple angles, the polarization state of the light wave can be completely reconstructed.

[0036] After acquiring multimodal image data, a data preprocessing step is performed.

[0037] First, spatial registration is performed on the two-dimensional high-resolution bright-field image, the three-dimensional point cloud data, and the polarized light image. The goal of spatial registration is to ensure that all modal data are strictly aligned in the same physical coordinate system, so that the same physical location corresponds to the same pixel coordinates in different modal images.

[0038] Spatial registration is achieved through pre-calibrated camera intrinsic and extrinsic parameter matrices, as well as the relative pose transformation matrix between the structured light module and the camera. The camera intrinsic parameters include focal length, principal point coordinates, and lens distortion coefficients; the extrinsic parameters describe the rotation and translation relationship between the camera coordinate system and the world coordinate system; the relative pose between the structured light module and the camera is obtained through joint calibration using a calibration board. The registration process employs an iterative nearest-point algorithm or rigid transformation estimation based on feature points, and the registration error is controlled at the sub-pixel level.

[0039] Secondly, the 3D point cloud data is converted into a 2D depth map through orthographic projection (or depth rendering). Orthographic projection ignores perspective effects and maps the Z coordinate (depth value) of each point in the point cloud to the gray value of the corresponding pixel in the depth map. The X and Y coordinates directly correspond to the row and column indices of the image. The converted depth map has the same resolution and pixel layout as the 2D bright field image, which facilitates subsequent feature extraction and fusion.

[0040] It should be noted that depth rendering or orthographic projection transformation is a common technique in this field, and specific implementations can be found in computer vision standard libraries (such as Open3D, PCL, etc.).

[0041] Next, the polarized light images at the four polarization angles are combined into a multi-channel Stokes vector parameter image. The Stokes vector consists of four components. The components represent the total light intensity, the difference between the horizontal and vertical linear polarization components, the difference between the ±45 degree linear polarization components, and the difference between the left and right circular polarization components. The synthesis formula is as follows: ; ; ; ; in, Indicates in The intensity values ​​of the polarized image acquired at the specified angle. Since this embodiment uses linearly polarized light and does not incorporate a quarter-wave plate, Since the component is always zero, the Stokes vector parameter image is a three-channel or four-channel image, with each channel corresponding to a Stokes component, which fully characterizes the polarization state information of the light wave and provides a data basis for subsequent polarization optical feature extraction.

[0042] In step S2, parallel multi-scale feature extraction is performed on the multimodal image data to obtain two-dimensional visual features, three-dimensional geometric features, and polarization optical features, respectively.

[0043] In practice, three independent encoder branches with feature pyramid network structures are constructed. The first encoder branch receives the preprocessed two-dimensional high-resolution bright-field image, the second branch receives the two-dimensional depth map, and the third branch receives the Stokes vector parameter image.

[0044] Each encoder branch consists of a series of convolutional layers, pooling layers, and activation function layers stacked together. The convolutional layers use 3x3 or 5x5 convolutional kernels with a stride of one and the same padding method to keep the feature map size unchanged. The activation function uses a modified linear unit, whose output is the maximum value of the input value and zero. The pooling layer uses a 2x2 max pooling operation with a stride of two to downsample and retain the most salient local features.

[0045] The network depth is set to five levels, with each level outputting a feature map. The resolutions are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively. The deep feature maps contain global, semantic information, such as the overall layout of the terminals and the type of defects. The shallow feature maps contain local, detailed information, such as pin edges and minor scratches.

[0046] The three branches output multi-scale feature representations of the corresponding modes, forming two-dimensional visual features, three-dimensional geometric features, and polarization optical features.

[0047] In step S3, a cross-modal attention fusion network is constructed. Two-dimensional visual features, three-dimensional geometric features, and polarization optical features are input into the network. Through feature interaction and weight adaptive adjustment between modalities, a deeply fused multimodal feature vector is generated. Specifically, a cascaded cross-modal interactive attention module is designed.

[0048] In the first-level attention calculation, two-dimensional visual features are used as the query tensor, and three-dimensional geometric features are used as the key tensor and value tensor. The query tensor, key tensor, and value tensor all come from the same level feature map of the encoder branch. The attention weight matrix is ​​generated by calculating the dot product similarity between the query tensor and the key tensor and normalizing it by scaling and the Softmax function.

[0049] The weight matrix reflects the degree of attention each location in the two-dimensional image pays to the three-dimensional geometric information. Subsequently, the value tensor is weighted and summed using this weight matrix to obtain the enhanced two-dimensional features guided by the three-dimensional geometric information. This process realizes the guidance and focus of the three-dimensional geometric structure information on the two-dimensional texture features. For example, when the texture of a certain area in a two-dimensional image is blurred, if its corresponding three-dimensional depth map shows obvious depression, the attention mechanism will enhance the feature response of that area, thereby improving the confidence of defect identification.

[0050] The specific implementation process of using this weight matrix to perform weighted summation of the value tensors is as follows: First, let the attention weight matrix be denoted as A, with dimensions [H,W,H,W], where the element A[i,j,k,l] represents the association weight between position (i,j) on the query feature map and position (k,l) on the key feature map; let the value tensor be denoted as V, with the same dimensions as the key tensor, which is [H,W,C_v], where C_v is the number of feature channels.

[0051] Subsequently, for each spatial location (i,j) on the query feature map: Extract the attention weight vector A[i,j,:,:] corresponding to the query position. This vector represents the importance distribution of all key / value positions to the current query position (i,j).

[0052] The weight vector A[i,j,:,:] is reshaped into a two-dimensional matrix and copied and expanded along the channel dimension so that its dimension is consistent with the dimension [H,W,C_v] of the value tensor V.

[0053] The expanded weight matrix is ​​multiplied element-wise with the value tensor V to obtain the weighted value tensor.

[0054] The weighted value tensor is summed along its spatial dimensions (height and width) to generate a feature vector of dimension [1, 1, C_v]. This vector is the new feature representation of the current query position (i, j) updated based on global context information.

[0055] Finally, after traversing all query positions (i,j) and performing the above operations, all new feature vectors are reorganized in their original spatial order, and the final output is a feature tensor with dimensions [H,W,C_v] that has been enhanced with cross-modal information.

[0056] In the second-level attention calculation, the output of the first-level attention calculation is used as a new query tensor, and the polarization optical features are used as the bond tensor and value tensor, and the above attention weighting process is repeated.

[0057] Polarization optical features contain information such as material stress and microscopic scratches, which may appear as low contrast or invisible in two-dimensional bright-field images. Through attention mechanisms, this implicit information can be incorporated into feature representations enhanced by geometric information. For example, residual stress inside a plastic shell appears as specific birefringence fringes in a polarization image. Attention mechanisms can associate this fringing information with corresponding two-dimensional locations, thereby identifying potential structural vulnerabilities.

[0058] Through the above cascade process, a deep fusion multimodal feature vector that fully absorbs the complementary information of the three modalities is finally generated. It not only contains rich semantic information, but also integrates multi-dimensional physical characteristics, and has a high degree of discrimination against defects.

[0059] In step S4, the deep fused multimodal feature vector is input into a defect segmentation decoder, and the decoder outputs a pixel-level defect probability map corresponding to the original image size.

[0060] The defect segmentation decoder adopts the decoding path in the U-shaped network structure, which is symmetrical to the encoder branch. The decoder gradually restores the spatial resolution of the feature map through a series of deconvolution layers. The deconvolution layers use a 2x2 convolution kernel with a stride of 2 to achieve upsampling.

[0061] After each upsampling step, a skip connection is used to concatenate the shallow feature map from the corresponding level of the encoder with the currently upsampled feature map along the channel dimension. The purpose of the skip connection is to fuse high-level semantic information with low-level detail information, preventing the loss of accurate defect boundary information during decoding. For example, the shallow feature map of the encoder contains clear pin edge information, while the high-level feature map of the decoder contains semantic information of the defect category. The fusion of the two can generate a segmentation result with sharp boundaries and accurate categories.

[0062] In the last layer of the decoder, a convolutional layer with a kernel size of one by one and a logistic regression activation function are used to map the multi-channel feature map into a single-channel defect probability map with values ​​between zero and one. The value of each pixel in the probability map represents the probability that the location belongs to a defect region.

[0063] In step S5, the final defect mask image is generated based on the comparison between the probability value of each pixel in the defect probability map and the preset defect judgment threshold.

[0064] The preset defect judgment threshold is set based on historical data statistics and false alarm rate requirements, with a typical value of 0.5. If the probability value of a pixel is greater than or equal to the threshold, it is judged as a defective pixel and marked as white in the mask image; otherwise, it is marked as black. Subsequently, connected component analysis is performed on the defect mask image to extract the geometric attributes such as the contour, area, and centroid of each defect region. Combined with the category information output by the classification head, the specific location, contour, and preset defect category of the defect are identified in the mask image with different colors or labels, such as pin bending, coplanarity deviation, plastic crack, and surface scratch.

[0065] The BTB terminal assembly defect identification system based on machine vision includes a multimodal image data acquisition module, a data preprocessing module, a multi-scale feature extraction module, a cross-modal feature fusion module, a defect segmentation and decoding module, and a defect judgment and output module.

[0066] The multimodal image data acquisition module includes: A megapixel industrial camera equipped with a telecentric lens has a resolution of no less than five million pixels and a frame rate of no less than thirty frames per second. A ring-shaped light-emitting diode bright field light source coaxially positioned with the optical axis of an industrial camera; A structured light projection unit, the optical axis of which forms a 30-degree angle with the optical axis of an industrial camera; A polarization optical component consisting of a polarizer and an analyzer, both driven by a precision stepper motor, with an angular positioning accuracy of no less than 0.1 degrees. A five-axis motion platform is used to carry and precisely position the BTB terminal assembly to be tested, with a positioning repeatability of no less than five micrometers.

[0067] The data preprocessing module, multi-scale feature extraction module, cross-modal feature fusion module, defect segmentation and decoding module, and defect judgment and output module are all deployed in an industrial computing terminal. The industrial computing terminal includes a central processing unit (CPU), a graphics processing unit (GPU), a high-speed memory, and a non-volatile memory. The GPU has no fewer than 8,000 computing cores and a video memory capacity of no less than 16 gigabytes, dedicated to performing parallel computing tasks for neural networks. The non-volatile memory stores pre-trained neural network model parameters, image processing algorithm libraries, and system control programs.

[0068] During system operation, the five-axis motion platform moves the BTB terminal assembly under test to the center of the imaging area. The multimodal image data acquisition module sequentially acquires bright-field images, structured light stripe image sequences, and four polarization images. The data preprocessing module completes spatial registration, depth map generation, and Stokes image synthesis. The multi-scale feature extraction module extracts multi-scale features of three modalities in parallel. The cross-modal feature fusion module generates fused features through a cascaded attention mechanism. The defect segmentation and decoding module outputs a defect probability map. The defect judgment and output module generates the final defect mask image and transmits the results to the host computer or quality management system.

[0069] This embodiment achieves high-precision, low-false-alarm, and robust identification of BTB terminal assembly defects by fusing two-dimensional, three-dimensional, and polarization modal information and combining deep learning and attention mechanisms, thus meeting the stringent requirements of modern electronic manufacturing for automated visual inspection.

[0070] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0071] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A machine vision-based method for identifying defects in BTB terminal assembly, characterized in that, include: S1, acquire multimodal image data of the BTB terminal assembly under test. The multimodal image data includes two-dimensional high-resolution bright-field images, three-dimensional point cloud data, and polarized light images at at least three polarization angles. S2 performs parallel multi-scale feature extraction on multimodal image data to obtain two-dimensional visual features characterizing the surface texture and contour of the BTB terminal assembly under test, three-dimensional geometric features characterizing its spatial geometry, and polarization optical features characterizing its material stress and micro-scratches. S3 constructs a cross-modal attention fusion network, inputting two-dimensional visual features, three-dimensional geometric features, and polarization optical features into the network. Through feature interaction between modalities and adaptive weight adjustment, a deeply fused multimodal feature vector is generated. Specifically, the implementation involves designing a cascaded cross-modal interactive attention module, which contains multiple attention calculation units. In the first-level attention calculation, two-dimensional visual features are used as the query tensor, and three-dimensional geometric features are used as the key tensor and value tensor. An attention weight matrix is ​​generated by calculating the similarity between the query tensor and the key tensor, and the value tensor is weighted using this weight matrix, thereby enabling the guidance and focus of three-dimensional geometric structural information on two-dimensional texture features. In the second-level attention calculation, the output of the first-level attention calculation is used as a new query tensor, and the polarization optical features are used as the bond tensor and value tensor. The above attention weighting process is repeated, thereby integrating the material stress and micro-defect information into the feature representation that has been enhanced by geometric information. Through the above cascade process, a deep fusion multimodal feature vector that fully absorbs the complementary information of the three modes is finally generated. S4, input the deep fused multimodal feature vector into a defect segmentation decoder, and the decoder outputs a pixel-level defect probability map corresponding to the size of the original image; S5. Based on the comparison between the probability value of each pixel in the defect probability map and the preset defect judgment threshold, generate the final defect mask image, and mark the specific location, outline, and preset defect category of the defect in the defect mask image.

2. The BTB terminal assembly defect identification method based on machine vision according to claim 1, characterized in that, Step S1 includes: A telecentric lens industrial camera with a built-in coaxial light source is used to acquire two-dimensional high-resolution bright-field images of the BTB terminal assembly under test. The telecentric lens ensures that the image is free of geometric distortion, and the coaxial light source provides uniform and shadowless illumination. A series of pre-coded stripe patterns are projected onto the surface of the BTB terminal assembly under test using a structured light module. An industrial camera simultaneously acquires distorted stripe images at different projection angles. The distorted stripe images are analyzed using phase deflection or triangulation algorithms to reconstruct high-density three-dimensional point cloud data of the surface of the BTB terminal assembly under test. A rotatable analyzer is set in front of the lens of an industrial camera, and a polarizer that rotates synchronously with it is set in the optical path of the coaxial light source. The polarizer and analyzer are controlled to rotate synchronously to four polarization angles: 0 degrees, 45 degrees, 90 degrees and 135 degrees. One frame of image is acquired at each angle to obtain a polarized light image.

3. The BTB terminal assembly defect identification method based on machine vision according to claim 2, characterized in that, After acquiring multimodal image data, a data preprocessing step is also included, specifically: Spatial registration is performed on two-dimensional high-resolution bright-field images, three-dimensional point cloud data, and polarized light images to ensure that the pixel coordinate systems of all modal data are strictly aligned. 3D point cloud data is converted into 2D depth maps through depth rendering or orthographic projection, so that its data structure is consistent with that of 2D images; The polarized light images at four polarization angles are combined into a single multi-channel Stokes vector parameter image. Each channel of the Stokes vector parameter image corresponds to a Stokes vector. , , , With four components, this image fully represents the polarization state information of the light wave.

4. The BTB terminal assembly defect identification method based on machine vision according to claim 3, characterized in that, Step S2 includes: Three independent encoder branches with feature pyramid network structures are constructed, corresponding to the two-dimensional high-resolution bright-field image, the two-dimensional depth map, and the Stokes vector parameter image, respectively; Each encoder branch consists of a series of convolutional layers, pooling layers, and activation function layers stacked together, outputting feature maps of different resolutions at different depth levels of the network. Deep feature maps contain global, semantic information, while shallow feature maps contain local, detailed information, thus forming a multi-scale representation of two-dimensional visual features, three-dimensional geometric features, and polarization optical features.

5. The BTB terminal assembly defect identification method based on machine vision according to claim 1, characterized in that, In step S4, the defect segmentation decoder adopts the decoding path in the U-shaped network structure, whose structure is symmetrical with the encoder branches; The decoder gradually restores the spatial resolution of the feature map through a series of upsampling layers or deconvolution layers; After each upsampling step, the shallow feature map from the corresponding level of the encoder is concatenated with the currently upsampled feature map through skip connections. This is intended to fuse high-level semantic information with low-level detail information and prevent the loss of accurate defect boundary information during the decoding process. In the last layer of the decoder, a convolutional layer with a kernel size of one by one and a logistic regression activation function are used to map the multi-channel feature map into a single-channel defect probability map with values ​​between zero and one.

6. The BTB terminal assembly defect identification method based on machine vision according to claim 3, characterized in that, Spatial registration is achieved through pre-calibrated camera intrinsic and extrinsic parameters, as well as the relative pose transformation matrix between the structured light module and the camera, and the registration error is controlled at the sub-pixel level.

7. The BTB terminal assembly defect identification method based on machine vision according to claim 1, characterized in that, In step S5, if the probability value of a certain pixel is greater than or equal to the preset defect judgment threshold, then the pixel is marked as a defective pixel. Connectivity analysis is performed on the defect mask image to extract the contour, area, and centroid of each defect region; Based on the category information output by the classification head, the defect category is identified in the defect mask image with different colors or labels. The defect categories include pin bending, coplanarity deviation, plastic cracks, or surface scratches.

8. The BTB terminal assembly defect identification method based on machine vision according to claim 1, characterized in that, The preset defect judgment threshold is set based on historical data statistics and false alarm rate requirements.

9. A machine vision-based BTB terminal assembly defect identification system, used to execute the machine vision-based BTB terminal assembly defect identification method according to any one of claims 1-8, characterized in that, include: The multimodal image data acquisition module is used to acquire two-dimensional high-resolution bright-field images, three-dimensional point cloud data, and multi-angle polarized light images of the BTB terminal assembly under test. The data preprocessing module, connected to the multimodal image data acquisition module, is used to perform spatial registration, data format conversion, and polarization information synthesis on the acquired multimodal image data. The multi-scale feature extraction module, connected to the data preprocessing module, contains three parallel encoders based on feature pyramid networks, used to extract two-dimensional visual features, three-dimensional geometric features, and polarization optical features from the preprocessed multimodal data, respectively. The cross-modal feature fusion module is connected to the multi-scale feature extraction module. Its core is a cascaded cross-modal interactive attention network, which is used to deeply fuse three heterogeneous features to generate a unified multimodal feature representation. The defect segmentation and decoding module is connected to the cross-modal feature fusion module. It adopts a decoder structure symmetrical to the encoder and uses skip connections to fuse multi-level information to convert a unified multimodal feature representation into a pixel-level defect probability map. The defect determination and output module, connected to the defect segmentation and decoding module, is used to compare the defect probability map with a preset threshold and generate a final defect mask image that identifies the defect location, outline and category.

Images