An SMT surface defect vision detection device based on AI image recognition

Abstract

The present invention relates to the technical field of image analysis, and specifically to an SMT surface defect vision detection device based on AI image recognition, which includes a template image storage module, an offset box and mask image generation module, an image acquisition module, a bare board defect detection module, and a finished product defect detection module. First, according to the mounting process document, the mounting positions of each workpiece and the allowable range of offset error are determined, and multiple offset boxes are generated to obtain a global mask image. An image extraction method based on the global mask image is proposed, and the component coverage area image and the component non-coverage area image are extracted through the global mask image, realizing partitioned defect detection; in addition, a PCBA comprehensive defect detection model based on multi-task learning is proposed to realize defect detection in the component coverage area. By separately analyzing the defects in the component coverage area and the component non-coverage area, the PCBA finished product defect detection result is finally generated, which can realize efficient and accurate SMT surface defect detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and particularly to a visual inspection device for SMT surface defects based on AI image recognition. Background Art

[0002] SMT (Surface Mount Technology) is an advanced electronic assembly technology that directly mounts electronic components on the surface of a printed circuit board without the need for traditional through-hole mounting. SMT surface mount technology is widely used in the manufacturing of modern electronic products and has the characteristics of high efficiency, high density, high precision, and miniaturization. In the SMT process flow, first, surface mount equipment is used to accurately mount electronic components on the surface of the printed circuit board. After subsequent process treatments such as soldering and cleaning, a complete functional electronic circuit board - PCBA (Printed Circuit Board Assembly) is finally formed. In contrast, a PCB (Printed Circuit Board) is a blank circuit board without installed electronic components. Through the SMT process flow, a PCB can be processed into a PCBA.

[0003] As the basic carrier of each electronic component, if a PCB board has defects such as missing pads, broken circuits, and short circuits, it will directly affect the subsequent component mounting and soldering quality. If defect detection is not carried out at the PCB stage, defective PCB boards will enter the next mounting link, increasing the difficulty and cost of subsequent detection and repair. In addition, once these problems are discovered only at the PCBA defect detection stage, the rework will become complicated, seriously affecting the production efficiency and the yield of finished products. Therefore, early detection of PCBs can effectively screen out defective products, reduce the defect rate and rework rate of subsequent production, and ensure the stability and reliability of product quality.

[0004] Although the existing technology can perform defect detection at the PCB and PCBA stages respectively and identify common defects through a target detection model, these two detection stages are often disjointed. The detection of the original PCB and the PCBA finished product stage is often carried out independently, lacking the joint analysis and feature sharing of the detection data in the two stages, resulting in the inability to make full use of the image feature information in the PCB stage to make the detection effect in the PCBA stage more accurate. This separated detection process cannot effectively track the cumulative process of defects, making it difficult to trace the initial problems in the PCB stage when defects are found in the PCBA stage, increasing the complexity of fault troubleshooting and rework, and thus affecting the overall production efficiency and product quality. In addition, there is still a large room for improvement in the existing computer vision-based SMT surface defect detection methods in terms of component offset detection and overall detection efficiency.

[0005] To this end, a visual inspection device for SMT surface defects based on AI image recognition is proposed. Summary of the Invention

[0006] The object of the present invention is to provide a visual inspection device for SMT surface defects based on AI image recognition. The device includes a template image storage module for acquiring and storing PCBA template images; an offset box and mask image generation module for determining the mounting positions and allowable ranges of offset errors of each workpiece according to the mounting process document, and generating a global mask image to define the component coverage area and the component non-coverage area. The image acquisition module is used to collect images of the bare board and PCBA finished products on the production line in real time. The bare board defect detection module detects defects in the original PCB image, and the finished product defect detection module detects defects in the PCBA finished product image, and extracts the component coverage area and the component non-coverage area through the global mask image, respectively performs defect analysis, and finally generates the finished product defect detection result. The device proposes a region extraction method based on the global mask image, realizing accurate partition defect detection; at the same time, using a multi-task AI model to comprehensively detect defects in the component coverage area, so as to realize efficient and accurate SMT surface defect detection.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A visual inspection device for SMT surface defects based on AI image recognition, comprising:

[0009] A template image storage module for acquiring and storing PCBA template images;

[0010] An offset box and mask image generation module for determining the mounting positions and allowable ranges of mounting offset errors of each workpiece according to the mounting process document; determining M offset boxes according to the mounting positions and the allowable ranges of mounting offset errors corresponding to each workpiece, where M is the total number of all workpieces on the PCBA template image; merging all the offset boxes to generate a global mask image; the global mask image is used to define and identify the component coverage area and the component non-coverage area;

[0011] An image acquisition module for collecting the original PCB image before the full process of component surface mounting on a specified production line; collecting the PCBA finished product image after the full process of component surface mounting on the production line;

[0012] A bare board defect detection module for detecting defects in the original PCB image to obtain a bare board defect detection result;

[0013] A finished product defect detection module for detecting defects in the PCBA finished product image to obtain a finished product defect detection result, including the following steps:

[0014] Extract the component-covered area image and the component-uncovered area image from the PCBA finished product image according to the global mask image;

[0015] Perform defect detection on the component-covered area image to obtain the defect detection result of the component-covered area;

[0016] Obtain the original PCB image corresponding to the component-uncovered area image to get the matching bare board image;

[0017] Obtain the bare board defect detection result according to the component-uncovered area image and the matching bare board image.

[0018] Further, the PCBA template image is specifically: the PCBA template image is the surface reference image of a defect-free finished board that has gone through the entire process of component surface mounting on the production line.

[0019] Further, after performing defect detection on the original PCB image to obtain the bare board defect detection result, the following steps are also included: divide the original PCB image into normal raw material samples and defective raw material samples according to the bare board defect detection result; when a defective raw material sample is detected, give an early warning to the production line, including: obtaining the first serial number corresponding to the defective raw material sample, tracking the defective raw material sample according to the first serial number; notifying the staff or sorting equipment to remove the defective raw material sample from the production line and put it into the isolation area.

[0020] Further, the global mask image is specifically: in the global mask image, the area covered by the offset box is marked as 1, and the area not covered by the offset box is marked as 0.

[0021] Further, extracting the component-covered area image and the component-uncovered area image from the PCBA finished product image according to the global mask image is specifically: align the global mask image with the PCBA finished product image to obtain the aligned global mask image and the aligned PCBA finished product image; perform a bitwise AND operation on the aligned global mask image and the aligned PCBA finished product image; correspond the pixel points marked as 1 in the aligned global mask image with the aligned PCBA finished product image to obtain the first corresponding area; extract the first corresponding area from the aligned PCBA finished product image to obtain the component-covered area image; at the same time, correspond the pixel points marked as 0 in the aligned global mask image with the aligned PCBA finished product image to obtain the second corresponding area, and extract the second corresponding area from the aligned PCBA finished product image to obtain the component-uncovered area image.

[0022] Further, defect detection is performed on the component coverage area image, and the component coverage area defect detection result obtained includes:

[0023] Obtain historical defect samples containing different defect categories from the historical defect sample database; the defect categories include surface defects, component missing defects, and component layout angle defects;

[0024] Generate augmented training samples according to the historical defect samples through data augmentation and generative adversarial networks. All the augmented training samples and all the historical defect samples constitute an augmented historical defect sample dataset;

[0025] According to the global mask image, extract the component coverage area from each sample image in the augmented historical defect sample dataset to obtain historical component coverage area images;

[0026] Use the historical component coverage area images to train the PCBA comprehensive defect detection model;

[0027] Input the preprocessed component coverage area image into the trained PCBA comprehensive defect detection model to obtain the component coverage area defect detection result, including surface defect detection result, component missing defect detection result, and component layout angle defect detection result.

[0028] Further, the PCBA comprehensive defect detection model includes:

[0029] An input layer for receiving the preprocessed component coverage area image or the preprocessed historical component coverage area image;

[0030] A shared feature extraction layer for extracting general features and outputting a shared feature map; the shared feature extraction layer includes an initial convolutional layer and multiple depthwise separable convolutional bottleneck blocks; each depthwise separable convolutional bottleneck block consists of a depthwise convolution, a pointwise convolution, and a residual connection;

[0031] A multi-task branch layer for performing feature extraction and detection of each task on the basis of the shared feature map through a parallel multi-thread mechanism, and consists of a surface defect detection branch, a component missing defect detection branch, and a component layout angle defect detection branch;

[0032] The surface defect detection branch includes two first feature extraction convolutional layers and a first global average pooling layer for extracting surface defect features. The first global average pooling layer is connected to a defect classification module and a defect localization module; wherein, the defect classification module realizes the classification output of defect categories through a fully connected layer for judging the type of surface defects; the defect localization module outputs the localization information of the surface defects through a fully connected layer for determining the position and size of the surface defects;

[0033] The component missing defect detection branch is used to determine whether a specified component is missing, and includes a second feature extraction convolutional layer, a second global average pooling layer, and a second fully connected layer. The second feature extraction convolutional layer is used to extract missing detection features. After being reduced to a one-dimensional vector through the second global average pooling layer, it is transmitted to the second fully connected layer, and the missing probability is output through the Sigmoid activation function;

[0034] The component layout angle defect detection branch is used to detect the layout offset angle of the specified component when it is not missing. When the output result of the component missing defect detection branch is not missing, the component layout angle defect detection branch is activated and extracts layout angle features. After being processed by a third feature extraction convolutional layer and a third global average pooling layer, it is connected to a regression module, and the angle offset value is output; wherein, the regression module consists of a fully connected layer and an activation function; when the output result of the component missing defect detection branch is missing, the component layout angle defect detection branch outputs an empty value;

[0035] The fusion layer is used to splice the outputs of the surface defect detection branch, the component missing defect detection branch, and the component layout angle defect detection branch, and merge them into an overall output vector;

[0036] The output layer is used to receive the overall output vector and output it.

[0037] Further, obtaining the corresponding PCB original image of the component uncovered area image to obtain the matching bare board image includes: when collecting the PCB original image, simultaneously recording the first serial number corresponding to the PCB original image; when collecting the PCBA finished product image, simultaneously recording the second serial number corresponding to the PCBA finished product image; obtaining the first serial number corresponding to the component uncovered area image; and determining the matching bare board image by matching the first serial number and the second serial number.

[0038] Further, obtaining the bare board defect detection result according to the component uncovered area image and the matching bare board image includes: preprocessing the component uncovered area image and the matching bare board image, including denoising, image alignment, and contrast enhancement, to obtain a standard component uncovered area image and a standard matching bare board image; performing difference detection on the standard component uncovered area image and the standard matching bare board image to obtain a changed area; extracting the image containing the changed area to obtain a changed image; inputting the changed image into a trained bare board defect recognition model to obtain the bare board defect detection result; the bare board defect recognition model is a Faster R-CNN model trained according to historical bare board images.

[0039] Further, the finished product defect detection module further includes: summarizing the bare board defect detection result and the component coverage area defect detection result to obtain the PCBA finished product defect detection result.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] 1. A method for extracting based on a global mask image, the core of which lies in precisely separating the component coverage area and the non-component coverage area through the global mask image. Combining multi-stage detection of the PCB original image and the PCBA finished product image realizes efficient and accurate tracking and analysis of defects during the component mounting process; the component coverage area mainly focuses on issues such as component missing, position offset, and mounting angle, while the non-component coverage area focuses on detecting defects such as surface scratches, contamination, and soldering residues. Through the partition extraction of the global mask image, the device avoids the occlusion interference of the component on the detection of the non-component coverage area, making the detection more focused and accurate. In addition, independently processing each area can reduce unnecessary computational overhead, speed up the detection speed, and at the same time reduce the probability of false detection and missed detection. By separately extracting the component coverage area image and the non-component coverage area image and separately performing defect detection, more efficient and accurate defect identification is achieved.

[0042] 2. The present invention proposes a PCBA comprehensive defect detection model, which can simultaneously detect three different types of defects, namely surface defects, component missing, and component layout angle deviation, within a single model framework, achieving accurate and efficient multi-task detection. First of all, the shared feature extraction layer, based on the lightweight structure of MobileNetV2, can effectively extract the basic features common to multi-tasks, reduce the number of model parameters, improve the operation efficiency, and is suitable for the real-time detection requirements in industrial environments. The subsequent multi-task branch layer can perform in-depth extraction and detection of specific features for different types of defects, ensuring that each defect detection task can be independently processed. Through the parallel multi-thread mechanism and multi-task branches, the efficient detection of surface defects, component missing, and component layout angle deviation is achieved. By combining the shared feature map and multi-task branches, the detection efficiency and accuracy are greatly improved. This model can comprehensively analyze the potential defects in the component coverage area image, not only identify and locate surface defects, but also accurately judge whether components are missing, and detect the angle deviation degree of components when they are not missing, so as to achieve a comprehensive monitoring of the component installation state. Especially when detecting the component layout angle deviation, when the result of the component missing detection branch is "not missing", the component layout angle defect detection branch is activated to extract and analyze the angle features. This conditional activation mechanism reduces unnecessary computational overhead, avoids invalid detection operations when components are missing, and ensures the optimization of resource utilization. Through the above model structure, the PCBA comprehensive defect detection model realizes the efficient parallel processing of multi-tasks, ensures the accurate detection of various defects in the component coverage area, and significantly improves the accuracy and efficiency of defect detection.

[0043] 3. By combining the detection data of the original PCB and the PCBA finished product stage, the present invention realizes the joint analysis and feature sharing of the image features in the two stages, makes full use of the image feature information in the PCB stage to improve the detection accuracy in the PCBA stage, and effectively solves the problem of fragmented detection processes in the prior art. By performing differential detection on the image of the component-uncovered area in the finished board and the corresponding PCB board image before SMT processing, it is possible to effectively identify minor defects that may be introduced during the production process, such as scratches, contamination, or indentation problems. Differential detection can quickly focus on the changed areas, avoiding pixel-by-pixel scanning of the entire board and improving the detection efficiency. By inputting the extracted changed areas into a trained defect classification model, the defect types can be further refined, different defect features can be distinguished, ensuring the precise tracking and identification of quality changes in the component-uncovered area during production, enabling the defects in the finished product to be traced back to the problem sources in the initial PCB stage, and thus achieving fast and accurate fault location and troubleshooting. This defect detection method reduces the rework time and cost while ensuring product quality, and helps to improve the identification accuracy and detection efficiency of SMT surface defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The flowchart shows the process of processing a PCB into a PCBA finished product through SMT technology in an embodiment of the present invention;

[0045] Figure 2 The schematic structural diagram of a visual inspection device for SMT surface defects based on AI image recognition provided in an embodiment of the present invention;

[0046] Figure 3 The schematic diagram of the global mask image provided in an embodiment of the present invention. Detailed implementation manners

[0047] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0048] As Figure 1 shown, Figure 1 The flowchart shows the process of processing a PCB into a PCBA finished product through SMT technology. Here, the PCB is an initial bare board without any components on its surface, and SMT (Surface Mount Technology) is surface mount technology. After various electronic components are mounted on the surface of the PCB and processes such as soldering and inspection are completed, a PCBA (Printed Circuit Board Assembly) finished product is formed.

[0049] Please refer to Figures 1 to 3 , the present invention provides a visual inspection device for SMT surface defects based on AI image recognition, which will be specifically elaborated in combination with two embodiments as follows:

[0050] Embodiment 1:

[0051] Company A is an electronic product manufacturer, mainly producing electronic devices such as smartphones, tablets, and smart home products. To improve production efficiency and ensure product quality, Company A introduced a visual inspection device for SMT surface defects based on AI image recognition on its SMT production line. As Figure 2 shown, it includes:

[0052] A template image storage module for acquiring and storing PCBA template images;

[0053] Further, the PCBA template image is specifically: the PCBA template image is the surface reference image of a defect-free finished board after the full process of component surface mounting on a specified production line according to the mounting process document.

[0054] By using the PCBA template image generated according to the mounting process document as the surface reference image of the defect-free finished board, an accurate reference standard can be provided, which provides a reliable basis for the determination of the subsequent offset frame.

[0055] Specifically, all current products on this production line belong to the same production batch and execute the same process standards and inspection requirements. The template image, component layout, and mounting method are kept consistent to ensure that the component positions, offset tolerance ranges, and inspection parameters of all PCB boards are exactly the same.

[0056] The offset frame and mask image generation module is used to determine the mounting position and the allowable range of mounting offset error for each workpiece according to the mounting process document; specifically, the mounting process document includes a bill of materials, a placement file, and a layout diagram; M offset frames are determined according to the mounting position and the allowable range of mounting offset error corresponding to each workpiece, as shown in Table 1, where M is the total number of all workpieces on the PCBA template image; during the generation process of the mask image, the offset frame is used to determine the boundary of each component, helping to distinguish which areas belong to the component coverage area and which belong to the non-component coverage area. The mask image needs to be precisely cropped during the generation process to ensure the accurate position and size of each component. In this embodiment, M is 15, and all the offset frames are combined to generate a global mask image, as Figure 3 shown; among them, Figure 3 the area covered by the offset frame described above is marked as 1 (white area), and the area not covered by the offset frame is marked as 0 (black area). White indicates the possible layout range of components, and black indicates that this area is not within the allowable layout range of components. The offset frame is used to determine whether the position of the component meets the requirements. If the mounting position of the component exceeds the offset frame, it is considered that the component has a position offset or misalignment.

[0057] The global mask image marks the area covered by the offset box as 1 and the uncovered area as 0, providing a reliable basis for the extraction of the component-covered area image and the component-uncovered area image. Since the templates and standards for all circuit boards on the production line are consistent, this mask image has high feasibility and consistency and can be reused throughout the production process to ensure the standardization and consistency of the detection results. In this way, the finished product defect detection module can quickly and accurately separate the component-covered area and the component-uncovered area, significantly improving the image processing efficiency, reducing the computational complexity, and effectively avoiding unnecessary misjudgments and missed detections, making the subsequent defect detection more efficient and accurate, thus enhancing the automated detection ability and quality control level of the production line.

[0058] Table 1 Component Offset Box Layout Information Table

[0059]

[0060] An image acquisition module for real-time acquisition of the bare board image before the full process of component surface mounting on a specified production line to obtain the original PCB image; and at the same time, acquisition of the finished board image after the full process of component surface mounting on the production line to obtain the PCBA finished product image;

[0061] Specifically, during the production process, before each PCB bare board undergoes SMT mounting, it will be photographed by an image acquisition device to generate the original PCB image. Each acquired original PCB image will be assigned a first serial number (such as "1001" or "ABC123"), and this serial number is bound to the corresponding original PCB image to ensure that all subsequent data can be associated with it. After the mounting process is completed, the PCBA finished product image will also be assigned a second serial number. Based on these two serial numbers, the SMT process can be traced and defect analysis can be carried out. Real-time acquisition of the bare board image before the full process of component surface mounting on the production line to obtain the original PCB image; storing the original PCB image and the corresponding first serial number in the bare board database; at the same time, acquisition of the finished board image after the full process of component surface mounting on the production line to obtain the PCBA finished product image; storing the original PCB image and the corresponding second serial number in the finished board database;

[0062] A bare board defect detection module for sequentially performing defect detection on the original PCB images in the bare board database according to the order of the first serial number to obtain the bare board defect detection results;

[0063] Further, after performing defect detection on the original PCB image and obtaining the bare board defect detection result, the following steps are also included: dividing the original PCB image into normal raw material samples and defective raw material samples according to the bare board defect detection result; when detecting the defective raw material samples, giving an early warning to the production line, including the following processes:

[0064] Obtain the first serial number corresponding to the defective raw material sample, and track the defective raw material sample according to the first serial number; notify the staff or sorting equipment to remove the defective raw material sample from the production line and place it in the isolation area.

[0065] To ensure that the order of the serial numbers is not affected after the defective raw material sample is removed, this embodiment provides two feasible solutions:

[0066] Solution 1: When the defective raw material sample is removed from the production line, paste a placeholder identification sticker at its corresponding production line position. This method is simple and easy to implement, and visually prompts the production line operators intuitively to ensure that the subsequent second serial number counting maintains sequential continuity. When the production line operators or the automatic flow system perform the second serial number counting, they will visually recognize this position as a placeholder position through the placeholder identification sticker. Although there is no actual product at this position, the operator or system will regard it as a valid placeholder and still assign a serial number to this position to keep the serial number counting continuous, and then sequentially count to the next position. This ensures the integrity and continuity of the serial numbers. Even if the defective sample is removed, it still occupies the corresponding serial number at the original position.

[0067] Solution 2: After the defective raw material sample is removed, the system automatically generates a virtual placeholder mark for it. This method does not require physical stickers, and the system will automatically skip this position, making the subsequent serial number order maintain continuity, which is convenient for data management and traceability.

[0068] By automatically identifying and real-time tracking the detected defective raw material samples, the bare board defect detection module can immediately give an early warning when a defect occurs, and accurately locate the problem samples according to the serial numbers. The bare board defect detection module can quickly notify the staff or sorting equipment to remove the defective samples from the production line and place them in the isolation area, preventing defective raw materials from entering the next link and reducing the interference to the subsequent production process. This function not only improves the automation and response speed of the production line, but also significantly reduces the rework rate and material waste.

[0069] The finished product defect detection module is used to sequentially perform defect detection on the PCBA finished product images in the finished product board database according to the order of the second serial number to obtain the finished product defect detection result, including the following steps:

[0070] Extract the component-covered area image and the component-uncovered area image from the PCBA finished product image according to the global mask image;

[0071] Further, extracting the component-covered area image and the component-uncovered area image from the PCBA finished product image according to the global mask image is specifically as follows: Align the global mask image with the PCBA finished product image to obtain an aligned global mask image and an aligned PCBA finished product image; perform a bitwise AND operation on the aligned global mask image and the aligned PCBA finished product image; correspond the pixel points marked as 1 in the aligned global mask image with the aligned PCBA finished product image to obtain a first corresponding area; extract the first corresponding area from the aligned PCBA finished product image to obtain the component-covered area image; the component-covered area image is the first corresponding area extracted from the PCBA finished product image, and the component-covered area image is composed of M rectangular images of different sizes. Each rectangular image corresponds to a part of the white area in the global mask image, and each rectangular block shows the area covered by components in the PCBA finished product image. M is the total number of all workpieces on the PCBA template image and is also the total number of offset frames. In this embodiment, M is 15, that is, there are 15 component-covered area images in total. At the same time, correspond the pixel points marked as 0 in the aligned global mask image with the aligned PCBA finished product image to obtain a second corresponding area, and extract the second corresponding area from the aligned PCBA finished product image to obtain the component-uncovered area image. The component-uncovered area image is the second corresponding area extracted from the PCBA finished product image. In this image, all parts corresponding to the black area in the global mask image will show the content of the PCBA finished product image, while the pixels corresponding to the white area in the global mask image will be masked. The component-uncovered area image only retains the exposed PCB board part without component coverage in the finished product image, and its shape and position are consistent with the black area in the global mask. The component-uncovered area image as a whole presents a "hollowed-out" effect, where the component-uncovered area is completely visible, while the component-covered area is completely covered by black blocks. This processing method effectively highlights the component-uncovered area and shields component interference.

[0072] Preferably, when collecting the PCBA finished product image, it usually includes some background or redundant areas around the PCB board. These non-PCBA board areas may interfere with subsequent alignment and bitwise operations. Therefore, before the alignment operation, it is necessary to first perform edge cropping on the finished product image to ensure that the image only retains the complete area of the PCBA board itself for accurate matching with the global mask image.

[0073] This method of extracting partitioned images realizes the precise separation of the component-covered area image and the component-uncovered area image by aligning the global mask image with the PCBA finished product image and performing bitwise operations, enabling the detection system to independently analyze the defect conditions in their respective areas. The component-covered area image retains the information of all component positions, which is helpful for detecting the component mounting quality; while the component-uncovered area image (the image of the area on the PCBA board without component coverage) shows the exposed PCB part in a "hollowed-out" form, avoiding the interference of components and making the bare board defect detection more efficient and clear. This processing method not only improves the detection accuracy but also reduces the misjudgment and missed detection caused by component occlusion, thus further enhancing the quality control ability and detection efficiency of the production line.

[0074] Perform defect detection on the component-covered area image to obtain the defect detection result of the component-covered area;

[0075] Furthermore, performing defect detection on the component-covered area image to obtain the defect detection result of the component-covered area includes:

[0076] Obtain historical defect samples containing different defect categories from the historical defect sample database; the defect categories include surface defects, component missing defects, and component layout angle defects;

[0077] Generate augmented training samples according to the historical defect samples through data augmentation and generative adversarial network (GAN), and all the augmented training samples and all the historical defect samples constitute an augmented historical defect sample dataset;

[0078] According to the global mask image, extract the component-covered area from each sample image in the augmented historical defect sample dataset to obtain historical component-covered area images; the historical component-covered area images are 15 rectangular images of different sizes. Preferably, preprocess the historical component-covered area images, and use the super-resolution reconstruction method to enlarge the historical component-covered area images to increase details. The super-resolution method can make the image details clearer and facilitate the model to extract detailed features.

[0079] Perform component defect annotation on each pre - processed historical component coverage area image, including component surface defects, component missing defects, and component layout angle defects; specifically: Open the image using LabelIMG software, and perform component surface defect annotation on each historical component coverage area image. It is necessary to mark both the position and category of the defect simultaneously. The categories include but are not limited to the following defect types: scratches, stains, spots, bubbles, and deformations. At the same time, perform component missing defect annotation on this historical component coverage area image, and determine whether the specified component exists within the offset box area corresponding to the historical component coverage area image. The annotation result is either "exists" or "does not exist". At the same time, perform layout angle defect annotation on this historical component coverage area image. If the specified component exists within the corresponding offset box area, mark its installation offset angle. The reference angle (0°) is defined as the long side of the component being parallel to the upper edge of the PCB board, and the direction from left to right is 0°. If the component is offset 5° clockwise relative to this direction, it is recorded as +5°. If the component is offset 5° counterclockwise relative to this direction, it is recorded as -5°.

[0080] Train the PCBA comprehensive defect detection model using the historical component coverage area image;

[0081] Input the pre - processed component coverage area image into the trained PCBA comprehensive defect detection model to obtain the defect detection results of the component coverage area, including surface defect detection results, component missing defect detection results, and component layout angle defect detection results.

[0082] This method extracts and expands multi - category defect samples from the historical defect sample database, and combines super - resolution reconstruction to enhance the clear details of the small component coverage area image, thereby improving the model's ability to identify defects in a small area. Through data augmentation using a generative adversarial network (GAN), the training data of the model is enriched, enabling the PCBA comprehensive defect detection model to more comprehensively identify various defect types such as surface defects, component missing, and layout angle offsets. Based on the precise extraction method of the offset box, the system can achieve high - precision defect detection in a small range, effectively reducing the missed detection and misjudgment rates of component defects, and providing higher reliability and accuracy guarantees for component defect identification in the production process.

[0083] Furthermore, the PCBA comprehensive defect detection model includes:

[0084] An input layer for receiving the pre - processed component coverage area image or the pre - processed historical component coverage area image;

[0085] Preferably, in the preprocessing stage, the super-resolution reconstruction method is used for the component coverage area image to high-definition process the component coverage area image to enhance its resolution and detail information. Then, the high-definition image is resized to the standard input size of 224×224 to meet the input requirements of the model.

[0086] A shared feature extraction layer for extracting common features; the shared feature extraction layer is composed of a MobileNetV2 structure, and the shared feature extraction layer includes an initial convolutional layer and multiple depthwise separable convolutional layers, and outputs a shared feature map;

[0087] A multi-task branch layer for performing feature extraction and detection of each task on the basis of the shared feature map through a parallel multi-thread mechanism, and is composed of a surface defect detection branch, a component missing defect detection branch, and a component layout angle defect detection branch;

[0088] The surface defect detection branch includes two feature extraction convolutional layers and a global average pooling layer for extracting surface defect features, and the global average pooling layer is connected to a defect classification module and a defect localization module; among them, the defect classification module realizes the classification output of defect categories through a fully connected layer for judging the type of surface defects; the defect localization module outputs the localization information of the surface defects through another fully connected layer for determining the position and size of the surface defects;

[0089] The component missing defect detection branch is used to judge whether a specified component is missing, and includes a feature extraction convolutional layer, a global average pooling layer, and a fully connected layer. The feature extraction convolutional layer is used to extract missing detection features, and after being reduced to a one-dimensional vector through the global average pooling layer, it is transmitted to the fully connected layer, and the missing probability is output through a Sigmoid activation function;

[0090] The component layout angle defect detection branch is used to detect the layout offset angle of the specified component when it is not missing. When the output result of the component missing defect detection branch is not missing, the component layout angle defect detection branch is activated and extracts layout angle features. After being processed by a feature extraction convolutional layer and a global average pooling layer, it is connected to a regression module and outputs an angle offset value; among them, the regression module is composed of a fully connected layer and an activation function; when the output result of the component missing defect detection branch is missing, the component layout angle defect detection branch outputs an empty value;

[0091] A fusion layer for splicing the outputs of the surface defect detection branch, the component missing defect detection branch, and the component layout angle defect detection branch and merging them into an overall output vector;

[0092] An output layer, configured to receive the overall output vector and output it.

[0093] Preferably, a conditional judgment is added in the forward propagation stage of the PCBA comprehensive defect detection model, and the output of the component missing detection branch is used as a boolean condition:

[0094] If the component missing detection is "not missing" (1), activate the layout angle detection branch;

[0095] If the component missing detection is "missing" (0), skip the layout angle detection branch.

[0096] Specifically, in this embodiment, the conditional judgment logic is located after the output of the component missing detection branch but before the component layout angle detection branch. The activation of the layout angle detection branch can be determined according to the result of the missing detection.

[0097] Table 2 Key Parameter Table of the PCBA Comprehensive Defect Detection Model Network Structure

[0098] Layer name Convolution kernel size Stride Number of output channels Output size Input layer - - 3 224×224×3 Initial convolutional layer 3x3 2 32 112×112×32 Bottleneck block (13 layers) - - 320 7×7×320

[0099] Referring to Table 2, the shared feature extraction layer is based on the MobileNetV2 bottleneck structure and includes an initial convolutional layer and multiple depthwise separable convolutional bottleneck blocks. Each bottleneck block consists of a depth convolution, a pointwise convolution, and a residual connection. Through the repeated stacking of 13 MobileNetV2 bottleneck blocks, high-level features of the image are gradually extracted. Finally, the output shared feature map has a size of 7×7×320, providing an efficient and accurate feature representation for the subsequent multi-task branch layer.

[0100] Specifically, this embodiment provides a specific structure of the multi-task branch layer. Refer to Table 3:

[0101] Table 3 Structure Table of the Multi-task Branch Layer

[0102]

[0103] The PCBA comprehensive defect detection model realizes efficient joint detection of component surface defects, missing conditions, and layout angle offsets through shared feature extraction and multi-task branch structures. The shared feature extraction layer is based on the MobileNetV2 structure, with the advantages of lightweight and efficient feature extraction, thereby reducing the model's computational load and improving the detection speed. The multi-task branch layer, through a parallel multi-thread mechanism, simultaneously executes multiple detection tasks, making the detection process more efficient. Each task branch optimizes the feature extraction of surface defects, component missing, and layout angles to ensure the recognition accuracy of each defect type. The final fusion layer integrates the multi-task outputs into an overall output vector, achieving efficient and accurate detection result output. This model structure not only significantly improves the detection efficiency but also reduces the risks of missed detection and misjudgment.

[0104] Obtain the original PCB image corresponding to the image of the uncovered area of the component to obtain a matching bare board image;

[0105] Furthermore, obtaining the original PCB image corresponding to the image of the uncovered area of the component to obtain the matching bare board image includes: when collecting the original PCB image, simultaneously record the first serial number corresponding to the original PCB image; when collecting the PCBA finished product image, simultaneously record the second serial number corresponding to the PCBA finished product image; obtain the first serial number corresponding to the image of the uncovered area of the component; by matching the first serial number with the second serial number, search for the original PCB image corresponding to the image of the uncovered area of the component, thereby determining the matching bare board image.

[0106] Table 4 Serial Number Mapping Table

[0107]

[0108] There is a serial number mapping table in the database, the structure of which is shown in Table 4. When it is necessary to obtain the matching bare board image of the PCBA finished product image B002, the system will search for the first serial number A002 according to B002, and then obtain the corresponding original PCB image / path / to / pcb_image_A002.png from the original image path, and use it as the matching bare board image.

[0109] By recording and matching the serial numbers of the original PCB images and the finished PCBA images, accurate traceability of the uncovered areas of components is achieved, thus making full use of the previously detected original PCB images. By matching the serial numbers, the system can directly find the original PCB image corresponding to the current finished PCBA, make full use of the image information in the previous stage, and significantly improve the detection efficiency. This method not only saves system resources and time, but also ensures the consistency and accuracy of detection, provides efficient and reliable support for defect traceability and root cause analysis, and further optimizes the quality management process of the production line.

[0110] Based on the image of the uncovered area of the component and the matched bare board image, a bare board defect detection result is obtained.

[0111] Further, obtaining the bare board defect detection result according to the image of the uncovered area of the component and the matched bare board image includes: preprocessing the image of the uncovered area of the component and the matched bare board image, including denoising, image alignment and contrast enhancement, to obtain a standard image of the uncovered area of the component and a standard matched bare board image; performing difference detection on the standard image of the uncovered area of the component and the standard matched bare board image to obtain a changed area; extracting the image containing the changed area to obtain a changed image; inputting the changed image into a trained bare board defect recognition model to obtain the bare board defect detection result; the bare board defect recognition model is a Faster R-CNN model trained according to historical bare board images.

[0112] When extracting the changed area image, only extract the changed part in the second corresponding area of the image of the uncovered area of the component according to the global mask image, and set the white area (the first corresponding area) in the global mask image as the ignored area and do not participate in image extraction.

[0113] This method makes full use of the PCB raw material images detected before SMT processing. By difference detection, it quickly judges whether new defects, such as scratches or dents, are introduced in the uncovered areas of components during the SMT process. If there is no difference, there is no need to repeat the detection, and it is directly confirmed that the uncovered areas of components are qualified, thus reducing unnecessary computational effort and significantly improving the detection efficiency. For the areas with differences, the system further uses the Faster R-CNN model for fine classification to identify the specific types of defects. This process not only ensures the accuracy of detection, but also minimizes the consumption of detection resources, realizes fast and efficient defect screening, and provides an intelligent and simplified solution for quality control in the production process.

[0114] Further, the finished product defect detection module further includes: summarizing the bare board defect detection result and the component covered area defect detection result to obtain a PCBA finished product defect detection result.

[0115] By summarizing the bare board defect detection results and the component coverage area defect detection results, the complete PCBA finished product defect detection results are generated, realizing the comprehensive quality assessment of the finished product.

[0116] Specifically, the component coverage area defect detection results need to be processed as follows before summarization:

[0117] Classify the component layout angle defect detection results in the component coverage area defect detection results. Specifically: determine the corresponding standard installation angle and allowable error according to the mounting process document; compare the component layout angle defect detection results in each component coverage area defect detection result with the standard installation angle, and calculate the angle deviation; if the angle deviation is not within the allowable error range, mark the component as having a component layout angle defect.

[0118] The training process of the bare board defect recognition model includes:

[0119] Obtain historical bare board defect samples containing different bare board defect categories from the historical defect sample database; the bare board defect categories include scratch defects, stain defects, oxidation spot defects, hole defects, crack defects, bubble defects, corrosion defects, and edge chipping defects;

[0120] Through data augmentation and generative adversarial network (GAN), augmented training samples are generated according to the historical bare board defect samples. All the augmented training samples and all the historical defect samples constitute an augmented historical bare board defect sample dataset;

[0121] According to the global mask image, the component uncovered area (component uncovered area) is extracted from each sample image in the augmented historical bare board defect sample dataset to obtain the historical bare board image.

[0122] Perform bare board defect annotation on each preprocessed historical component coverage area image, and at the same time annotate the defect position and defect category. The defect categories include scratch defects, stain defects, oxidation spot defects, hole defects, crack defects, bubble defects, corrosion defects, and edge chipping defects; specifically: use LabelIMG software to open the image, and perform component surface defect annotation on each historical bare board image;

[0123] Obtain the labeled historical bare board images from the augmented historical bare board defect sample dataset, and divide them into a training set and a validation set. Use the pre-trained Faster R-CNN model as the basic model and fine-tune it through transfer learning. After training, evaluate the model performance through the validation set, adjust the hyperparameters to improve the detection effect, and finally deploy it for actual detection to obtain the bare board defect recognition model;

[0124] To verify whether the solution of dividing the defect detection task into two parts, namely the component-covered area and the component-uncovered area, for separate detection in the present invention can improve the efficiency and accuracy of defect detection, a set of comparative experiments were conducted. The experimental group used the sub-region detection method, that is, defect detection was carried out separately on the component-covered area and the component-uncovered area, and the component defect detection results and the bare board defect detection results were obtained respectively. Control group 1, control group 2 and the traditional integrated detection method were used to directly perform defect detection on the entire PCBA finished product image without distinguishing between the component-covered area and the component-uncovered area. Among them, control group 1 was based on the Faster R-CNN object detection model; control group 2 was based on the YOLO (You Only Look Once) v5 model. Control group 3 constructed a multi-task learning (MTL) model, the structure of which was the same as the PCBA comprehensive defect detection model of the experimental group, including a surface defect detection branch, a component missing detection branch and a component layout angle detection branch, and used a shared feature extraction layer (based on MobileNetV2) to extract general features. Different from the experimental group, the training data of the control group 3 model was the entire PCBA finished product image without distinguishing between the component-covered area and the component-uncovered area. This model directly performed defect detection on the entire image, enabling the model to learn the defect features of all regions without regionalizing the component area and the component-uncovered area.

[0125] Table 5 Comparative experimental data table of sub-region detection and integrated detection

[0126]

[0127] The experimental results are shown in Table 5. Among them, the average detection time is the defect detection time for each complete PCBA image. The sub-region detection solution proposed in the present invention is not only superior to the traditional integrated detection method in terms of accuracy and over-detection rate control, but also achieves a good balance in terms of efficiency, and is particularly suitable for the actual production environment that requires high precision and low false alarm rate. Although the YOLOv5 of control group 2 is the fastest, it is slightly lacking in accuracy.

[0128] To verify that the PCBA comprehensive defect detection model based on MTL proposed in the present invention is superior to the prior art and common models in terms of accuracy and efficiency, another set of comparative experiments were conducted. To ensure the comparability of the experiments, all experiments used the sub-region detection method, and the defect detection was based on the component-covered area image.

[0129] The experimental group used a PCBA comprehensive defect detection model based on multi-task learning (MTL). This model includes a surface defect detection branch, a component missing detection branch, and a component layout angle detection branch. It uses a shared feature extraction layer (such as MobileNetV2) to extract common features and processes multiple detection tasks in parallel in the multi-task branches, outputting the detection results of all defects in one detection.

[0130] Control group 1: Three independent single-task detection models were used to detect surface defects, component missing, and component layout angle deviation respectively.

[0131] Surface defect detection model: Faster R-CNN was used to detect surface defects (such as scratches, stains, etc.) and could output the location and category of surface defects.

[0132] Component missing detection model: VGG16 was used to determine whether a component was missing.

[0133] Component layout angle detection model: InceptionV3 was used to detect the layout angle deviation of components.

[0134] Table 6 Data table of model comparison experiments

[0135]

[0136] Refer to Table 6 for the experimental results. The PCBA comprehensive defect detection model of the experimental group was superior to Control group 1 in terms of accuracy (96.2%) and average detection time (0.50 seconds).

[0137] The design of separately detecting the component-covered area and the non-component-covered area can better meet different detection requirements and improve detection accuracy and efficiency. The component-covered area mainly detects defects such as the presence, location, installation angle, and surface damage of components, while the non-component-covered area focuses on issues such as scratches, contamination, and welding residues. By separating the detection, the occlusion interference of components on the non-component-covered area can be avoided, making the detection results more focused and accurate. At the same time, such separation processing can use different detection algorithms for different areas, which can not only use the MTL model to efficiently detect complex component defects but also achieve efficient and accurate detection of the non-component-covered area through differential detection combined with classification models. Thus, a better balance is achieved in the overall detection accuracy and efficiency. Optimize the detection efficiency. In addition, separate detection helps to more accurately locate the source of defects, facilitating quick problem tracing and quality control, thereby improving the overall quality and reliability of SMT production.

[0138] Example 2:

[0139] On the SMT production line of a large electronic manufacturing factory B, after the PCBA board completes component placement and soldering, it enters the vision inspection device in sequence through the conveyor belt to ensure that each PCBA board meets the quality standards. In traditional inspections, traditional machine vision systems often have difficulty detecting tiny surface defects and component position offsets efficiently and accurately, easily resulting in missed inspections or misjudgments, which affect the quality of the finished products.

[0140] To improve the inspection efficiency and accuracy, the factory introduced a vision inspection device for SMT surface defects based on AI image recognition. This device can detect PCBA surface defects in real time to identify various defects in the SMT process and ensure that each PCBA board meets the factory standards.

[0141] A vision inspection device for SMT surface defects based on AI image recognition includes:

[0142] A template image storage module for acquiring and storing the PCBA template image;

[0143] An offset box and mask image generation module for determining the placement positions of each workpiece and the allowable range of placement offset errors according to the placement process file; determining M offset boxes based on the placement positions and the allowable range of placement offset errors corresponding to each workpiece, where M is the total number of all workpieces on the PCBA template image; merging all the offset boxes to generate a global mask image; the global mask image is used to define and identify the component coverage area and the component non-coverage area;

[0144] An image acquisition module for acquiring in real time the bare board image before the full process of component surface placement on the specified production line to obtain the PCB original image; and simultaneously acquiring the finished board image after the full process of component surface placement on the production line to obtain the PCBA finished product image;

[0145] A bare board defect detection module for detecting defects in the PCB original image to obtain the bare board defect detection result;

[0146] A finished product defect detection module for detecting defects in the PCBA finished product image to obtain the finished product defect detection result, including the following steps:

[0147] Extracting the component coverage area image and the component non-coverage area image from the PCBA finished product image according to the global mask image;

[0148] Detecting defects in the component coverage area image to obtain the component coverage area defect detection result;

[0149] Obtaining the PCB original image corresponding to the component non-coverage area image to obtain the matching bare board image;

[0150] Based on the image of the component-uncovered area and the matching bare board image, a bare board defect detection result is obtained.

[0151] Further, the PCBA template image is specifically: the PCBA template image is the surface reference image of a defect-free finished board that has gone through the full process of component surface mounting on the production line.

[0152] Further, after defect detection is performed on the original PCB image to obtain the bare board defect detection result, the following steps are also included: the original PCB image is divided into a normal raw material sample and a defective raw material sample according to the bare board defect detection result; when a defective raw material sample is detected, a warning is issued to the production line, including the following process:

[0153] Obtain the first serial number corresponding to the defective raw material sample, and track the defective raw material sample according to the first serial number; notify the staff or sorting equipment to remove the defective raw material sample from the production line and place it in an isolation area.

[0154] Further, the global mask image is specifically: in the global mask image, the area covered by the offset box is marked as 1, and the area not covered by the offset box is marked as 0.

[0155] Further, according to the global mask image, extracting the component-covered area image and the component-uncovered area image from the PCBA finished product image specifically includes: aligning the global mask image with the PCBA finished product image to obtain an aligned global mask image and an aligned PCBA finished product image; performing a bitwise AND operation on the aligned global mask image and the aligned PCBA finished product image; corresponding the pixel points marked as 1 in the aligned global mask image with the aligned PCBA finished product image to obtain a first corresponding area; extracting the first corresponding area from the aligned PCBA finished product image to obtain the component-covered area image; at the same time, corresponding the pixel points marked as 0 in the aligned global mask image with the aligned PCBA finished product image to obtain a second corresponding area, and extracting the second corresponding area from the aligned PCBA finished product image to obtain the component-uncovered area image.

[0156] Further, defect detection is performed on the component-covered area image, and obtaining the component-covered area defect detection result includes:

[0157] Obtain historical defect samples containing different defect categories from the historical defect sample database; the defect categories include surface defects, component missing defects, and component layout angle defects;

[0158] Generate augmented training samples according to the historical defect samples through data augmentation and generative adversarial networks. All the augmented training samples and all the historical defect samples constitute an augmented historical defect sample dataset;

[0159] Extract the component coverage area from each sample image in the augmented historical defect sample dataset according to the global mask image to obtain a historical component coverage area image;

[0160] Train a PCBA comprehensive defect detection model using the historical component coverage area image;

[0161] Input the preprocessed component coverage area image into the trained PCBA comprehensive defect detection model to obtain the defect detection results of the component coverage area, including surface defect detection results, component missing defect detection results, and component layout angle defect detection results.

[0162] Further, the PCBA comprehensive defect detection model includes:

[0163] An input layer for receiving the preprocessed component coverage area image or the preprocessed historical component coverage area image;

[0164] A shared feature extraction layer for extracting common features and outputting a shared feature map; the shared feature extraction layer includes an initial convolutional layer and multiple depthwise separable convolutional bottleneck blocks; each depthwise separable convolutional bottleneck block consists of a depthwise convolution, a pointwise convolution, and a residual connection;

[0165] A multi-task branch layer for performing feature extraction and detection of each task on the basis of the shared feature map through a parallel multi-thread mechanism, consisting of a surface defect detection branch, a component missing defect detection branch, and a component layout angle defect detection branch;

[0166] The surface defect detection branch includes two first feature extraction convolutional layers and a first global average pooling layer for extracting surface defect features. The first global average pooling layer is connected to a defect classification module and a defect localization module; wherein, the defect classification module realizes the classification output of defect categories through a fully connected layer for judging the type of surface defects; the defect localization module outputs the localization information of the surface defects through a fully connected layer for determining the position and size of the surface defects;

[0167] The component missing defect detection branch is used to determine whether a specified component is missing, and includes a second feature extraction convolutional layer, a second global average pooling layer, and a second fully connected layer. The second feature extraction convolutional layer is used to extract missing detection features. After being reduced to a one-dimensional vector through the second global average pooling layer, it is transmitted to the second fully connected layer, and the missing probability is output through the Sigmoid activation function;

[0168] The component layout angle defect detection branch is used to detect the layout offset angle of the specified component when it is not missing. When the output result of the component missing defect detection branch is not missing, the component layout angle defect detection branch is activated and extracts layout angle features. After being processed by a third feature extraction convolutional layer and a third global average pooling layer, it is connected to a regression module, and the angle offset value is output; wherein, the regression module consists of a fully connected layer and an activation function; when the output result of the component missing defect detection branch is missing, the component layout angle defect detection branch outputs an empty value;

[0169] The fusion layer is used to splice the outputs of the surface defect detection branch, the component missing defect detection branch, and the component layout angle defect detection branch, and merge them into an overall output vector;

[0170] The output layer is used to receive the overall output vector and output it.

[0171] Further, obtaining the corresponding PCB original image of the component uncovered area image to obtain the matching bare board image includes: when collecting the PCB original image, simultaneously recording the first serial number corresponding to the PCB original image; when collecting the PCBA finished product image, simultaneously recording the second serial number corresponding to the PCBA finished product image; obtaining the first serial number corresponding to the component uncovered area image; by matching the first serial number with the second serial number, searching for the PCB original image corresponding to the component uncovered area image, so as to determine the matching bare board image.

[0172] Further, obtaining the bare board defect detection result according to the component uncovered area image and the matching bare board image includes: preprocessing the component uncovered area image and the matching bare board image, including denoising, image alignment, and contrast enhancement, to obtain a standard component uncovered area image and a standard matching bare board image; performing difference detection on the standard component uncovered area image and the standard matching bare board image to obtain a changed area; extracting the image containing the changed area to obtain a changed image; inputting the changed image into a trained bare board defect recognition model to obtain the bare board defect detection result; the bare board defect recognition model is a Faster R-CNN model trained according to historical bare board images.

[0173] Further, the finished product defect detection module further includes: summarizing the bare board defect detection result and the component coverage area defect detection result to obtain the PCBA finished product defect detection result.

[0174] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An SMT surface defect vision detection device based on AI image recognition, characterized in that, Including: A template image storage module, configured to obtain and store a PCBA template image; An offset box and mask image generation module, configured to determine the mounting positions and the allowable ranges of mounting offset errors of each workpiece according to the mounting process file; determine M offset boxes according to the mounting positions and the allowable ranges of mounting offset errors corresponding to each workpiece, where M is the total number of all workpieces on the PCBA template image; merge all the offset boxes to generate a global mask image; the global mask image is used to define and identify the component covered area and the component uncovered area; An image acquisition module, configured to acquire a PCB original image before the full process of component surface mounting on a specified production line; acquire a PCBA finished product image after the full process of component surface mounting on the production line; A bare board defect detection module, configured to perform defect detection on the PCB original image to obtain a bare board defect detection result; A finished product defect detection module, configured to perform defect detection on the PCBA finished product image to obtain a finished product defect detection result, including the following steps: extract a component covered area image and a component uncovered area image from the PCBA finished product image according to the global mask image; perform defect detection on the component covered area image to obtain a component covered area defect detection result; obtain the PCB original image corresponding to the component uncovered area image to obtain a matching bare board image; obtain a bare board defect detection result according to the component uncovered area image and the matching bare board image; Align the global mask image with the PCBA finished product image to obtain an aligned global mask image and an aligned PCBA finished product image; perform a bitwise AND operation on the aligned global mask image and the aligned PCBA finished product image; correspond the pixel points marked as 1 in the aligned global mask image with the aligned PCBA finished product image to obtain a first corresponding area; extract the first corresponding area from the aligned PCBA finished product image to obtain a component covered area image; meanwhile, correspond the pixel points marked as 0 in the aligned global mask image with the aligned PCBA finished product image to obtain a second corresponding area, and extract the second corresponding area from the aligned PCBA finished product image to obtain a component uncovered area image.

2. The SMT surface defect vision detection device based on AI image recognition according to claim 1, characterized in that The PCBA template image specifically is: The PCBA template image is the surface reference image of a defect-free finished board that has gone through the full process of component surface mounting on the production line.

3. The visual inspection device for SMT surface defects based on AI image recognition according to claim 1, characterized in that, After performing defect detection on the PCB original image to obtain the bare board defect detection result, the following steps are further included: divide the PCB original image into a normal raw material sample and a defective raw material sample according to the bare board defect detection result; when a defective raw material sample is detected, give an alarm to the production line, including: obtain the first serial number corresponding to the defective raw material sample, and track the defective raw material sample according to the first serial number; notify the staff or the sorting equipment to remove the defective raw material sample from the production line and put it into an isolation area.

4. An SMT surface defect vision detection device based on AI image recognition according to claim 1, characterized in that, The global mask image specifically is: In the global mask image, the area covered by the offset box is marked as 1, and the area not covered by the offset box is marked as 0.

5. An SMT surface defect vision detection device based on AI image recognition according to claim 1, characterized in that, Performing defect detection on the image of the component coverage area to obtain the defect detection result of the component coverage area, including: Obtaining historical defect samples containing different defect categories from the historical defect sample database; the defect categories include surface defects, component missing defects, and component layout angle defects; Generating augmented training samples according to the historical defect samples through data augmentation and generative adversarial networks. All the augmented training samples and all the historical defect samples constitute an augmented historical defect sample dataset; According to the global mask image, extracting the component coverage area from each sample image in the augmented historical defect sample dataset to obtain historical component coverage area images; Training a PCBA comprehensive defect detection model using the historical component coverage area images; Inputting the preprocessed image of the component coverage area into the trained PCBA comprehensive defect detection model to obtain the defect detection result of the component coverage area, including surface defect detection results, component missing defect detection results, and component layout angle defect detection results.

6. An SMT surface defect vision detection device based on AI image recognition according to claim 5, characterized in that, The PCBA comprehensive defect detection model includes: An input layer for receiving the preprocessed image of the component coverage area or the preprocessed historical component coverage area image; A shared feature extraction layer for extracting common features and outputting a shared feature map; the shared feature extraction layer includes an initial convolutional layer and multiple depthwise separable convolutional bottleneck blocks; each depthwise separable convolutional bottleneck block consists of a depthwise convolution, a pointwise convolution, and a residual connection; A multi-task branch layer for performing feature extraction and detection of each task on the basis of the shared feature map through a parallel multi-thread mechanism, and consists of a surface defect detection branch, a component missing defect detection branch, and a component layout angle defect detection branch; The surface defect detection branch includes two first feature extraction convolutional layers and a first global average pooling layer for extracting surface defect features. The first global average pooling layer is connected to a defect classification module and a defect localization module; wherein, the defect classification module realizes the classification output of the defect category through a first fully connected layer for judging the type of the surface defect; the defect localization module outputs the localization information of the surface defect through a second fully connected layer for determining the position and size of the surface defect; The component missing defect detection branch is used to judge whether a specified component is missing, and includes a second feature extraction convolutional layer, a second global average pooling layer, and a third fully connected layer. The second feature extraction convolutional layer is used to extract missing detection features, which are reduced to a one-dimensional vector after passing through the second global average pooling layer and then transmitted to the third fully connected layer, and the missing probability is output through a Sigmoid activation function; The component layout angle defect detection branch is used to detect the layout offset angle of the specified component without missing. When the output result of the component missing defect detection branch is "not missing", the component layout angle defect detection branch is activated and extracts layout angle features. After being processed by a third feature extraction convolutional layer and a third global average pooling layer, it is connected to a regression module, and an angle offset value is output. Among them, the regression module consists of a fully connected layer and an activation function. When the output result of the component missing defect detection branch is "missing", the output of the component layout angle defect detection branch is empty. The fusion layer is used to splice the outputs of the surface defect detection branch, the component missing defect detection branch, and the component layout angle defect detection branch, and merge them into an overall output vector. The output layer is used to receive the overall output vector and output it.

7. An SMT surface defect vision detection device based on AI image recognition according to claim 1, characterized in that, Obtaining the corresponding PCB original image of the component uncovered area image to obtain the matching bare board image includes: when collecting the PCB original image, recording the first serial number corresponding to the PCB original image at the same time; when collecting the PCBA finished product image, recording the second serial number corresponding to the PCBA finished product image at the same time; obtaining the first serial number corresponding to the component uncovered area image; and determining the matching bare board image by matching the first serial number with the second serial number.

8. The visual inspection device for SMT surface defects based on AI image recognition according to claim 1, wherein, Obtaining the bare board defect detection result according to the component uncovered area image and the matching bare board image includes: preprocessing the component uncovered area image and the matching bare board image, including denoising, image alignment, and contrast enhancement, to obtain a standard component uncovered area image and a standard matching bare board image; performing difference detection on the standard component uncovered area image and the standard matching bare board image to obtain a changed area; extracting the image containing the changed area to obtain a changed image; inputting the changed image into a trained bare board defect recognition model to obtain the bare board defect detection result; the bare board defect recognition model is a Faster R-CNN model trained according to historical bare board images.

9. The SMT surface defect vision detection device based on AI image recognition according to claim 1, characterized in that, The finished product defect detection module further includes: summarizing the bare board defect detection result and the component covered area defect detection result to obtain the PCBA finished product defect detection result.

Images