PCBA element defect detection method based on multi-view feature fusion

Abstract

The invention belongs to the technical field of PCBA (Printed Circuit Board Assembly) defect detection, and particularly relates to a PCBA element defect detection method based on multi-view feature fusion. The method comprises the following steps: firstly, acquiring a multi-view image of a PCBA (Printed Circuit Board Assembly), including a top view and four side views, and preprocessing the multi-view image of the PCBA; then, a plurality of PCBAs are collected, and a multi-view image of each PCBA is collected; marking elements in the image to obtain an element data set, and training an element detection model by using the element data set; then, a plurality of normal and abnormal PCBAs are collected, multi-view images of the PCBAs are collected and preprocessed, the preprocessed multi-view images are input into an element detection model, elements on the PCBAs are detected, the multi-view images of the PCBAs are cut according to the position information of the elements, and an element multi-view data set is obtained; and finally, constructing a defect detection model composed of an image encoder and a linear layer, training the defect detection model by using the element multi-view data set, and using the trained defect detection model for PCBA element defect detection. According to the method, all-around defect identification is carried out on the element from a plurality of visual angles, and the problem of defect omission possibly caused by a visual blind area under a single visual angle is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of PCBA defect detection, and specifically is a PCBA component defect detection method based on multi-view feature fusion. Background Art

[0002] With the rapid development of electronic technology, the application scope of electronic equipment continues to expand, especially in the fields of industry, medical treatment, communication and automobile. As the core component of electronic equipment, the assembled circuit board (PCBA) installs electronic components on the printed circuit board (PCB) through surface mounting or through-hole insertion technology to achieve circuit connection and specific electronic functions. In industrial production, the quality of PCBA is crucial to the performance and reliability of the equipment. As the number of electronic components on the circuit board gradually increases and the layout becomes more complex, conventional defect detection technology faces many challenges in practical applications.

[0003] The traditional defect detection method is manual visual inspection. During the inspection process, quality inspectors can flexibly adjust the observation angle to ensure that every detail can be inspected. However, it has the disadvantages of strong subjectivity, easy ambiguity and low efficiency, and it is difficult to meet the high-speed and high-precision inspection requirements of modern industry. Visual inspection is a method that uses computer vision technology to achieve automated defect detection. It can largely overcome the disadvantages of low efficiency and high labor intensity of manual inspection methods, and has been increasingly widely studied and applied in modern industry. The current visual inspection technology mainly relies on the PCBA top view for analysis. However, due to the visual blind spot of a single perspective, for some more hidden component defects on the circuit board, such as component side damage, component tombstone, pin solder joint and component leg lift, the PCBA top view cannot clearly reflect the abnormal area. Summary of the invention

[0004] In view of the shortcomings of the existing defect detection technology, the technical problem that the present invention intends to solve is to propose a PCBA component defect detection method based on multi-view feature fusion.

[0005] The present invention solves the technical problem by adopting the following technical solution: A PCBA component defect detection method based on multi-view feature fusion includes the following steps: Step 1: Acquire a multi-view image of the PCBA, including a top view and four side views, and pre-process the multi-view image of the PCBA; Step 2: Collect several PCBAs and acquire multi-view images of each PCBA; annotate the components in the images to obtain a component data set; use the component data set to train a component detection model so that the component detection model can detect components on the PCBA; Step 3: Collect several normal and abnormal PCBAs, collect multi-view images of each PCBA and perform preprocessing, input the preprocessed multi-view images into the component detection model, detect the components on the PCBA, and obtain the component type and component position information; crop the multi-view images of the PCBA according to the component position information to obtain the multi-view images of the components, and then obtain the component multi-view dataset; Step 4: Build a defect detection model, including an image encoder and a linear layer; input the multi-view image of the component into the image encoder to extract the features of each view; splice the features of each view to obtain multi-view fusion features; classify the multi-view fusion features through the linear layer to obtain the component defect detection results; The defect detection model is trained using the component multi-view dataset, and the trained defect detection model is used for PCBA component defect detection.

[0006] Compared with the prior art, the present invention has the following beneficial effects: 1. When the surface damage of the component is located above the component and the component tombstone is very serious, the component defect can be seen from the top view, but the two defects of pin cold soldering and component curling are difficult to see from the top view. Therefore, the present invention collects multi-view images of PCBA including top view and side view, and performs comprehensive defect recognition on the component from multiple view angles, which solves the problem of defect omission caused by visual blind spots under a single view angle and improves the reliability and comprehensiveness of detection.

[0007] 2. First, based on the multi-view image of PCBA, the component detection model is used to identify the components on the PCBA, and the multi-view image of PCBA is cropped according to the position information of the component to obtain the multi-view image of the component; the multi-view features of the component are extracted using the defect detection model, and the multi-view features are fused to effectively capture the detailed feature information of the component at different view angles, further enhance the model's perception of component defects, and achieve more accurate PCBA component defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of a multi-view image acquisition platform of the present invention; Figure 2 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0009] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but the protection scope of the present application is not limited thereto.

[0010] The present invention provides a PCBA component defect detection method based on multi-view feature fusion (hereinafter referred to as method, see Figure 1~2 ), including the following steps: Step 1: Use the multi-view image acquisition platform to obtain the multi-view images of the PCBA, and pre-process the multi-view images of the PCBA; the multi-view images include the top view of the PCBA and four side views , , , , the multi-view image set is denoted as ; like Figure 1 As shown in the figure, the multi-view image acquisition platform includes a motion platform, a light source, a top-view camera and a side-view camera; the PCBA is placed on the motion platform, and the top-view camera and the side-view camera are located directly above and on the side of the PCBA, respectively, for collecting the top view and side view of the PCBA; the motion platform adopts a three-dimensional motion mode, which translates in the x-axis and y-axis directions, rises and falls in the z-axis direction and can rotate around the z-axis, which not only ensures the distance adaptation between the camera and the PCBA, but also drives the PCBA to rotate through the motion platform to collect the side view. Both the top-view camera and the side-view camera use 20-megapixel high-definition color industrial cameras, which can clearly capture the subtle structure of the PCBA; the lens uses an electric zoom lens, which can flexibly adjust the focal length according to the size of the PCBA to achieve the best imaging effect. The light source is a white annular diffuse light source, which provides a uniform lighting environment.

[0011] Preprocessing includes: histogram equalization, denoising and correction; first, histogram equalization and denoising are performed to improve the visual effect of the image; then, the image is tilted and corrected based on perspective transformation technology to alleviate the impact of improper PCBA placement; the correction process includes: Top view of PCBA For example, assuming the top view of the PCBA There are pixel coordinates in , after perspective transformation, the pixel coordinates are converted to , then the perspective transformation process is expressed as: (1) in, is the perspective transformation matrix, and is the scale factor of homogeneous coordinates; In order to find the perspective transformation matrix, let , then: (2) (3) Known PCBA top view The coordinates of the four vertices are , , and , and the coordinates of the four vertices after perspective transformation are , , and , then: (4) (5) Combining equations (4) and (5), we can find the perspective transformation matrix ; Use the perspective transformation matrix to find the top view of the PCBA The coordinates of each pixel in the image are transformed to obtain the corrected top view. .

[0012] Step 2: Build a component data set and use it to train the component detection model so that the component detection model can automatically detect components such as capacitors, resistors, and inductors on the PCBA; Collect several PCBAs from the production line, and use the multi-view image acquisition platform to collect multi-view images of each PCBA (including top view and four side views); annotate all components (such as capacitors, resistors, and inductors) in the top view and side view to obtain a component data set; use the component data set to train the component detection model, that is, input the multi-view images of the PCBA into the component detection model to detect the components on the PCBA; before training, it is necessary to set appropriate hyperparameters, such as learning rate, iteration rounds, and batch size, and calculate the training loss according to the following loss function; (6) In the formula, , and They are classification loss, prediction box regression loss and focal distribution loss. , and are all weight factors; The Adam optimizer is used to optimize the model through back propagation to obtain a trained component detection model; in this embodiment, the YOLOv8 network is selected as the component detection model.

[0013] Step 3: Build a component multi-view dataset for training defect detection models; Collect several normal and abnormal PCBAs from the production line. The defect types of abnormal PCBAs include component surface damage, component tombstone, pin cold soldering and component curled pins. Use the multi-view image acquisition platform to collect the top view and four side views of each PCBA, obtain the multi-view images of each PCBA, and pre-process the multi-view images of the PCBA. The preprocessed multi-view images are input into the component detection model, and the components on the PCBA are detected to obtain the component type and the position information of the components in each view; the multi-view images of the PCBA are cropped according to the component position information to obtain the multi-view images of the components (including the top view and four side views), and then a component multi-view dataset consisting of multi-view images, component types, position information and labels is obtained. The label set of this dataset is {normal, component surface damage, component tombstone, pin cold soldering and component curled pins}, and each label represents a certain quality state of the component; similarly, the multi-view datasets of the remaining components are obtained.

[0014] Step 4: Build a defect detection model, train the defect detection model using the component multi-view dataset, and use the trained defect detection model for PCBA component defect detection; The defect detection model includes an image encoder and a linear layer. The multi-view images of the component are input into the image encoder to extract the features of each view. ;in, is the top view feature, There are four side view features; the features of each view are spliced ​​to obtain the multi-view fusion feature ; Input the multi-view fusion features into the linear layer to detect defects and obtain the defect type of the component.

[0015] Considering that the ViT model is obtained by training with a large-scale dataset and has a very good migration effect in downstream tasks, this embodiment selects the ViT model as the image encoder, uses the component multi-view dataset to train the defect detection model through the pre-training-fine-tuning paradigm training, and uses the trained defect detection model for PCBA component defect detection.

[0016] In practical applications, a multi-view image acquisition platform is used to acquire multi-view images of the PCBA under test, and the multi-view images of the PCBA under test are preprocessed; the preprocessed multi-view images are input into the component detection model, and the components on the PCBA are detected to obtain the component type and position information; the multi-view images of the PCBA are cropped according to the component position information to obtain the multi-view image of the component, and if there are multiple components, the multi-view images of multiple components are obtained; the multi-view image of the component is input into the trained defect detection model to obtain the defect type of the component, and complete the PCBA component defect detection.

[0017] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A PCBA component defect detection method based on multi-view feature fusion, characterized in that: The method comprises the following steps: Step 1: Acquire a multi-view image of the PCBA, including a top view and four side views, and pre-process the multi-view image of the PCBA; Step 2: Collect several PCBAs and acquire multi-view images of each PCBA; annotate the components in the images to obtain a component data set; use the component data set to train a component detection model so that the component detection model can detect components on the PCBA; Step 3: Collect several normal and abnormal PCBAs, collect multi-view images of each PCBA and perform preprocessing, input the preprocessed multi-view images into the component detection model, detect the components on the PCBA, and obtain the component type and component position information; crop the multi-view images of the PCBA according to the component position information to obtain the multi-view images of the components, and then obtain the component multi-view dataset; Step 4: Build a defect detection model, including an image encoder and a linear layer; input the multi-view image of the component into the image encoder to extract the features of each view; splice the features of each view to obtain multi-view fusion features; classify the multi-view fusion features through the linear layer to obtain the component defect detection results; The defect detection model is trained using the component multi-view dataset, and the trained defect detection model is used for PCBA component defect detection.

2. The PCBA component defect detection method based on multi-view feature fusion according to claim 1 is characterized in that: The preprocessing includes histogram equalization, denoising and rectification; The correction process includes: Assume that there are pixel coordinates in the top view of the PCBA , after perspective transformation, the pixel coordinates are converted to , then the perspective transformation process is expressed as: （1） in, is the perspective transformation matrix, and is the scale factor of homogeneous coordinates; make , then: （2） （3） The coordinates of the four vertices of the PCBA top view are known to be , , and , and the coordinates of the four vertices after perspective transformation are , , and , then: （4） （5） Combining equations (4) and (5), we can solve the perspective transformation matrix ; The corrected top view can be obtained by solving the transformed coordinates of each pixel in the PCBA top view according to the perspective transformation matrix.

3. The PCBA component defect detection method based on multi-view feature fusion according to claim 1 or 2, characterized in that: The defect types of abnormal PCBA include component surface damage, component tombstone, pin cold soldering and component lift-off.

4. The PCBA component defect detection method based on multi-view feature fusion according to claim 1, characterized in that: The component detection model adopts the YOLOv8 network.

Images