SMT surface defect visual detection device based on AI image recognition

Abstract

The invention relates to the technical field of image analysis, in particular to an SMT surface defect visual detection device based on AI image recognition, which comprises a template image storage module, an offset frame and mask image generation module, an image acquisition module, a bare board defect detection module and a finished product defect detection module. The method comprises the following steps: firstly, determining a mounting position and an offset error allowable range of each workpiece according to a mounting process file, and generating a plurality of offset frames to obtain a global mask image; an image extraction method based on a global mask image is provided, and an element coverage area image and an element non-coverage area image are extracted through the global mask image, so that partition defect detection is realized; in addition, a PCBA comprehensive defect detection model based on multi-task learning is provided, so that defect detection of an element coverage area is realized. Through respectively carrying out defect analysis on the element coverage area and the element non-coverage area, a PCBA finished product defect detection result is finally generated, and efficient and accurate SMT surface defect detection can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a SMT surface defect visual detection device based on AI image recognition. Background Art

[0002] SMT (Surface Mount Technology) is an advanced electronic assembly technology that directly mounts electronic components on the surface of a printed circuit board without the need for traditional perforation. SMT surface mounting technology is widely used in the manufacture of modern electronic products and has the characteristics of high efficiency, high density, high precision and miniaturization. In the SMT process, surface mounting equipment is first used to accurately mount electronic components on the surface of a printed circuit board. After subsequent processes such as welding and cleaning, a complete electronic circuit board with functions, PCBA (Printed Circuit Board Assembly), is finally formed. In contrast, PCB (Printed Circuit Board) is a blank circuit board that has not yet been installed with electronic components. After the SMT process, PCB can be processed into PCBA.

[0003] PCB boards are the basic carriers of various electronic components. If there are defects such as missing pads, broken lines and short circuits, it will directly affect the quality of subsequent component placement and welding. If defect detection is not performed at the PCB stage, defective PCB boards will enter the next step of placement, increasing the difficulty and cost of subsequent detection and repair. In addition, once these problems are discovered at the PCBA defect detection stage, rework will become complicated, seriously affecting production efficiency and finished product yield. Therefore, early detection of PCBs can effectively screen out defective products, reduce the defect rate and rework rate of subsequent production, and ensure the stability and reliability of product quality.

[0004] Although the existing technology can perform defect detection at the PCB and PCBA stages respectively, and identify common defects through target detection models, these two detection stages are often separated from each other. The detection of the original PCB and PCBA finished product stages is often carried out independently, lacking joint analysis and feature sharing of the detection data of the two stages, resulting in the inability to fully utilize the image feature information of the PCB stage to make the detection effect of the PCBA stage more accurate. This separate detection process cannot effectively track the accumulation process of defects, making it difficult to trace the defects found in the PCBA stage back to the initial problems in the PCB stage, increasing the complexity of troubleshooting and rework, and thus affecting the overall production efficiency and product quality. In addition, the existing SMT surface defect detection methods based on computer vision still have a lot of room for improvement in component offset detection and overall detection efficiency.

[0005] Therefore, a SMT surface defect visual inspection device based on AI image recognition is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide an SMT surface defect visual inspection device based on AI image recognition, which includes a template image storage module for acquiring and storing PCBA template images; an offset frame and mask image generation module for determining the mounting position and offset error tolerance of each workpiece according to the mounting process file, and generating a global mask image to define the component coverage area and the component uncovered area. The image acquisition module is used to collect images of bare boards and PCBA finished products on the production line in real time. The bare board defect detection module performs defect detection on the PCB original image, and the finished product defect detection module performs defect detection on the PCBA finished product image, and extracts the component coverage area and the component uncovered area through the global mask image, performs defect analysis respectively, and finally generates the finished product defect detection result. The device proposes a region extraction method based on the global mask image, which realizes accurate partition defect detection; at the same time, a multi-task AI model is used to perform comprehensive defect detection on the component coverage area, thereby realizing efficient and accurate SMT surface defect detection.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] An SMT surface defect visual detection device based on AI image recognition, comprising:

[0009] A template image storage module is used to obtain and store PCBA template images;

[0010] An offset frame and mask image generation module is used to determine the mounting position and the allowable range of mounting offset error of each workpiece according to the mounting process file; determine M offset frames according to the mounting position and the allowable range of mounting offset error corresponding to each workpiece, where M is the total number of all workpieces on the PCBA template image; merge all the offset frames to generate a global mask image; the global mask image is used to define and identify component coverage areas and component uncovered areas;

[0011] An image acquisition module is used to acquire the original image of the PCB before the whole process of surface mounting of components on a designated production line; and to acquire the image of the PCBA finished product after the whole process of surface mounting of components on the production line;

[0012] A bare board defect detection module is used to perform defect detection on the PCB original image to obtain a bare board defect detection result;

[0013] The finished product defect detection module is used to perform defect detection on the PCBA finished product image to obtain a finished product defect detection result, including the following steps:

[0014] Extracting a component covered area image and a component uncovered area image from the PCBA finished product image according to the global mask image;

[0015] Performing defect detection on the component coverage area image to obtain a component coverage area defect detection result;

[0016] Acquire the original image of the PCB corresponding to the image of the uncovered area of ​​the component to obtain a matching bare board image;

[0017] The bare board defect detection result is obtained according to the component uncovered area image and the matching bare board image.

[0018] Furthermore, the PCBA template image is specifically: the PCBA template image is a reference image of the surface of a defect-free finished board that has gone through the entire process of component surface mounting on the production line.

[0019] Furthermore, after performing defect detection on the PCB original image and obtaining the bare board defect detection result, the following steps are also included: dividing the PCB original image into normal raw material samples and defective raw material samples according to the bare board defect detection result; when the defective raw material sample is detected, issuing an early warning to the production line, including: obtaining a first serial number corresponding to the defective raw material sample, and tracking the defective raw material sample according to the first serial number; notifying staff or sorting equipment to remove the defective raw material sample from the production line and place it in an isolation area.

[0020] Furthermore, the global mask image is specifically: in the global mask image, the area covered by the offset box is marked as 1, and the area not covered by the offset box is marked as 0.

[0021] Further, according to the global mask image, the component coverage area image and the component uncovered area image are extracted from the PCBA finished product image specifically as follows: the global mask image is aligned with the PCBA finished product image to obtain an aligned global mask image and an aligned PCBA finished product image; a bitwise AND operation is performed on the aligned global mask image and the aligned PCBA finished product image; the pixel points marked as 1 in the aligned global mask image are matched with the aligned PCBA finished image to obtain a first corresponding area; the first corresponding area is extracted from the aligned PCBA finished image to obtain the component coverage area image; at the same time, the pixel points marked as 0 in the aligned global mask image are matched with the aligned PCBA finished image to obtain a second corresponding area, and the second corresponding area is extracted from the aligned PCBA finished image to obtain the component uncovered area image.

[0022] Further, performing defect detection on the component coverage area image to obtain the component coverage area defect detection result includes:

[0023] Acquire historical defect samples containing different defect categories from a historical defect sample database; the defect categories include surface defects, component missing defects, and component layout angle defects;

[0024] Generate an expanded training sample based on the historical defect sample by data enhancement and generative adversarial network, wherein all the expanded training samples and all the historical defect samples constitute an expanded historical defect sample data set;

[0025] According to the global mask image, extracting a component coverage area from each sample image in the expanded historical defect sample data set to obtain a historical component coverage area image;

[0026] Using the historical component coverage area image to train a PCBA comprehensive defect detection model;

[0027] The component coverage area image is preprocessed and then input into the trained PCBA comprehensive defect detection model to obtain the component coverage area defect detection results, including surface defect detection results, component missing defect detection results and component layout angle defect detection results.

[0028] Furthermore, the PCBA comprehensive defect detection model includes:

[0029] An input layer, used for receiving the pre-processed component coverage area image or the pre-processed historical component coverage area image;

[0030] A shared feature extraction layer, used to extract common features and output a shared feature map; the shared feature extraction layer includes an initial convolution layer and multiple depth-separable convolution bottleneck blocks; each of the depth-separable convolution bottleneck blocks consists of depth-wise convolution, point-by-point convolution and residual connection;

[0031] A multi-task branch layer, used for performing feature extraction and detection of each task based on the shared feature map through a parallel multi-threading mechanism, and composed of a surface defect detection branch, a component missing defect detection branch, and a component layout angle defect detection branch;

[0032] The surface defect detection branch includes two first feature extraction convolution layers and a first global average pooling layer, which are used to extract surface defect features, and the first global average pooling layer is connected to a defect classification module and a defect localization module; wherein the defect classification module implements defect category classification output through a fully connected layer, which is used to determine the type of surface defects; the defect localization module outputs the location information of the surface defects through the fully connected layer, which is used to determine the position and size of the surface defects;

[0033] The component missing defect detection branch is used to determine whether a specified component is missing, and includes a second feature extraction convolution layer, a second global average pooling layer, and a second fully connected layer. The second feature extraction convolution layer is used to extract missing detection features, which are reduced to a one-dimensional vector by the second global average pooling layer and then transferred to the second fully connected layer, and the missing probability is output through the Sigmoid activation function;

[0034] The component layout angle defect detection branch is used to detect the layout deviation angle of the specified component when it is not missing. When the output result of the component missing defect detection branch is not missing, the component layout angle defect detection branch is activated and extracts the layout angle feature, which is processed by a third feature extraction convolution layer and a third global average pooling layer, and then connected to a regression module to output the angle deviation value; wherein the regression module is composed of a fully connected layer and an activation function; when the output result of the component missing defect detection branch is missing, the output of the component layout angle defect detection branch is empty;

[0035] A fusion layer, used to splice the outputs of the surface defect detection branch, the component missing defect detection branch and the component layout angle defect detection branch into an overall output vector;

[0036] The output layer is used to receive and output the overall output vector.

[0037] Further, acquiring the PCB original image corresponding to the component uncovered area image to obtain the matching bare board image includes: when acquiring the PCB original image, simultaneously recording a first serial number corresponding to the PCB original image; when acquiring the PCBA finished product image, simultaneously recording a second serial number corresponding to the PCBA finished product image; acquiring the first serial number corresponding to the component uncovered area image; and determining the matching bare board image by matching the first serial number with the second serial number.

[0038] Furthermore, obtaining the bare board defect detection result according to the component uncovered area image and the matched bare board image includes: preprocessing the component uncovered area image and the matched bare board image, including denoising, image alignment and contrast enhancement, to obtain a standard component uncovered area image and a standard matched bare board image; performing difference detection on the standard component uncovered area image and the standard matched bare board image to obtain a changed area; extracting an image containing the changed area to obtain a changed image; inputting the changed image into a trained bare board defect recognition model to obtain the bare board defect detection result; the bare board defect recognition model is a Faster R-CNN model trained based on historical bare board images.

[0039] Furthermore, the finished product defect detection module also includes: summarizing the bare board defect detection result and the component coverage area defect detection result to obtain a PCBA finished product defect detection result.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. An extraction method based on a global mask image, the core of which is to accurately separate component coverage areas and component uncovered areas through global mask images. Combining multi-stage detection of PCB original images and PCBA finished product images, efficient and accurate tracking and analysis of defects in the component mounting process is achieved; the component coverage area mainly focuses on problems such as component missing, position offset and installation angle, while the component uncovered area focuses on detecting defects such as surface scratches, contamination and welding residues. Through the partition extraction of the global mask image, the device avoids the occlusion and interference of components on the detection of component uncovered areas, making the detection more focused and accurate. In addition, independent processing of each area can reduce unnecessary computing overhead, speed up the detection speed, and reduce the probability of false detection and missed detection. By extracting component coverage area images and component uncovered area images separately and performing defect detection separately, more efficient and accurate defect recognition is achieved.

[0042] 2. The present invention proposes a PCBA comprehensive defect detection model, which can simultaneously detect three different types of defects, namely surface defects, component missing and component layout angle deviation, within a model framework, and realize accurate and efficient multi-task detection. First, the shared feature extraction layer is based on the lightweight structure of MobileNetV2, which can effectively extract the basic features common to multiple tasks, reduce the number of model parameters, improve the computing efficiency, and meet the real-time detection needs in industrial environments. The following multi-task branch layer can perform deep extraction and detection of specific features for different types of defects, ensure that each defect detection task can be processed independently, and realize efficient detection of surface defects, component missing and component layout angle deviation through parallel multi-threading mechanism and multi-task branch. Through the combination of shared feature map and multi-task branch, the detection efficiency and accuracy are greatly improved. The model can comprehensively analyze the potential defects in the component coverage area image, not only can it identify and locate surface defects, but also can accurately determine whether the component is missing, and detect the degree of angle deviation of the component when it is not missing, thereby realizing comprehensive monitoring of the component installation status. Especially when detecting component layout angle deviation, when the result of component missing detection branch is not missing, the component layout angle defect detection branch is activated to extract and analyze angle features. This conditional activation mechanism reduces unnecessary computational overhead, avoids invalid detection operations when components are missing, and ensures optimal resource utilization. Through the above model structure, the PCBA comprehensive defect detection model realizes efficient parallel processing of multiple tasks, ensures accurate detection of various defects in the component coverage area, and significantly improves the accuracy and efficiency of defect detection.

[0043] 3. The present invention combines the detection data of the original PCB and the PCBA finished product stage to realize the joint analysis and feature sharing of the image features of the two stages, and makes full use of the image feature information of the PCB stage to improve the detection accuracy of the PCBA stage, thereby effectively solving the problem of detection process fragmentation in the prior art. The image of the uncovered area of ​​the component in the finished board is detected by difference with the corresponding PCB board image before SMT processing, which can effectively identify the minor defects that may be introduced in the production process, such as scratches, contamination or indentation. Difference detection can quickly focus on the area where the change occurs, avoiding pixel-by-pixel scanning of the entire board, and improving the detection efficiency. The extracted change area is input into the trained defect classification model, which can further refine the defect type and distinguish different defect features, ensuring the accurate tracking and identification of the quality changes of the uncovered area of ​​the component in production, so that the defects in the finished product can be traced back to the source of the problem in the initial stage of the PCB, thereby realizing fast and accurate fault location and troubleshooting. This defect detection method reduces rework time and cost while ensuring product quality, and helps to improve the recognition accuracy and detection efficiency of SMT surface defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart of a PCB provided in an embodiment of the present invention being processed into a PCBA finished product through SMT technology;

[0045] Figure 2 A schematic diagram of the structure of a SMT surface defect visual detection device based on AI image recognition provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a global mask image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] like Figure 1 As shown, Figure 1 The flow chart of how PCB is processed into PCBA finished products through SMT technology is shown, where PCB is the initial bare board without any components on the surface, and SMT (Surface Mount Technology) is the surface mounting technology, which forms the PCBA (Printed Circuit Board Assembly) finished product by mounting various electronic components on the PCB surface and completing the process steps such as welding and testing.

[0049] See also Figures 1 to 3 The present invention provides an SMT surface defect visual detection device based on AI image recognition, which will be specifically described in combination with two embodiments below:

[0050] Embodiment 1:

[0051] Company A is an electronic product manufacturer that mainly produces electronic devices such as smartphones, tablets, and smart home products. In order to improve production efficiency and ensure product quality, Company A introduced an SMT surface defect visual inspection device based on AI image recognition on its SMT production line. Figure 2 As shown, including:

[0052] A template image storage module is used to obtain and store PCBA template images;

[0053] Furthermore, the PCBA template image is specifically: the PCBA template image is a reference image of a defect-free finished board surface after the entire component surface mounting process on a designated production line according to a mounting process file.

[0054] By using the PCBA template image generated according to the placement process file as the reference image of the defect-free finished board surface, an accurate reference standard can be provided, providing a reliable basis for the subsequent determination of the offset frame.

[0055] Specifically, all products currently on the production line belong to the same production batch and implement the same process standards and testing requirements. The template image, component layout and mounting method are consistent to ensure that the component position, offset tolerance and testing parameters of all PCB boards are exactly the same.

[0056] The offset frame and mask image generation module is used to determine the mounting position and the allowable range of mounting offset error of each workpiece according to the mounting process file; specifically, the mounting process file includes a bill of materials, a patch file and a layout diagram; M offset frames are determined according to the mounting position corresponding to each workpiece and the allowable range of mounting offset error, as shown in Table 1, where M is the total number of all workpieces on the PCBA template image; in the mask image generation process, the offset frame is used to determine the boundary of each component to help distinguish which areas belong to the component coverage area and which belong to the component uncovered area. The mask image needs to be precisely cropped during the generation process to ensure that the actual position and size of each component are accurate. In this embodiment, M is 15, and all the offset frames are merged to generate a global mask image, such as Figure 3 As shown; among them, Figure 3 The area covered by the offset box is marked as 1 (white area), and the area not covered by the offset box is marked as 0 (black area). White indicates the range of possible component layout, and black indicates that the area is not within the range of allowed component layout. The offset box is used to determine whether the position of the component meets the requirements. If the component mounting position exceeds the offset box, it is considered that the component has been offset or misplaced.

[0057] This global mask image marks the area covered by the offset box as 1 and the uncovered area as 0, providing a reliable basis for the subsequent extraction of component covered area images and component uncovered area images. Since the templates and standards of all circuit boards on the production line are consistent, the mask image is highly feasible and consistent and can be reused throughout the production process to ensure the standardization and consistency of the inspection results. In this way, the finished product defect detection module can quickly and accurately separate the component covered area and the component uncovered area, significantly improving the image processing efficiency, reducing the computational complexity, and effectively avoiding unnecessary misjudgments and missed detections, making subsequent defect detection more efficient and accurate, thereby improving the production line's automated inspection capabilities and quality control level.

[0058] Table 1 Component offset frame layout information table

[0059]

[0060] The image acquisition module is used to collect the bare board image before the whole process of surface mounting of components on the designated production line in real time to obtain the original PCB image; and to collect the finished board image after the whole process of surface mounting of components on the production line to obtain the PCBA finished product image;

[0061] Specifically, during the production process, each PCB bare board will be photographed and a PCB original image will be generated by an image acquisition device before SMT mounting. Each collected PCB original image will be assigned a first serial number (such as "1001" or "ABC123"), which is bound to the corresponding PCB original image to ensure that all subsequent data can be associated with it. When the mounting process is completed, the PCBA finished product image will also be assigned a second serial number. Based on these two serial numbers, the SMT process can be traced and defect analysis can be achieved. Real-time collection of bare board images before the full process of component surface mounting on the production line to obtain PCB original images; the PCB original images and the corresponding first serial number are stored in the bare board database; at the same time, the finished board images after the full process of component surface mounting on the production line are collected to obtain PCBA finished product images; the PCB original images and the corresponding second serial number are stored in the finished board database;

[0062] A bare board defect detection module, used to perform defect detection on the PCB original images in the bare board database in sequence according to the order of the first serial numbers to obtain a bare board defect detection result;

[0063] Furthermore, after performing defect detection on the PCB original image and obtaining the bare board defect detection result, the following steps are also included: dividing the PCB original image into a normal raw material sample and a defective raw material sample according to the bare board defect detection result; when the defective raw material sample is detected, giving an early warning to the production line, including the following processes:

[0064] Obtain a first serial number corresponding to the defective raw material sample, and track the defective raw material sample according to the first serial number; notify staff or sorting equipment to remove the defective raw material sample from the production line and place it in an isolation area.

[0065] To ensure that the order of the serial numbers is not affected after the defective raw material samples are removed, this embodiment provides two feasible solutions:

[0066] Solution 1: When a defective raw material sample is removed from the production line, a placeholder identification sticker is affixed to its corresponding production line position. This method is simple and easy to use, and visually prompts the production line operator to ensure that the sequence is continuous when the second serial number is counted later. When the production line operator or the automatic assembly line system performs the second serial number count, the position will be visually identified as a placeholder position through the placeholder identification sticker. Although there is no actual product at this position, the operator or system will regard it as a valid placeholder and will still assign a serial number to the position to keep the serial number count continuous, and then count to the next position in sequence. This ensures the integrity and continuity of the serial number, and even if the defective sample is removed, it still occupies the corresponding serial number at the original position.

[0067] Solution 2: When the defective raw material sample is removed, the system automatically generates a virtual placeholder mark for it. This method does not require a physical sticker, and the system will automatically skip the position, so that the subsequent serial number sequence remains continuous, which is convenient for data management and traceability.

[0068] By automatically identifying and tracking the detected defective raw material samples in real time, the bare board defect detection module can immediately issue an early warning when a defect occurs and accurately locate the problematic sample based on the serial number. The bare board defect detection module can quickly notify the staff or sorting equipment to remove the defective samples from the production line and place them in an isolated area to prevent bad raw materials from entering the next link and reduce interference with subsequent production processes. This function not only improves the automation and response speed of the production line, but also significantly reduces the rework rate and material waste.

[0069] The finished product defect detection module is used to perform defect detection on the PCBA finished product images in the finished board database in sequence according to the second serial number to obtain the finished product defect detection result, including the following steps:

[0070] Extracting a component covered area image and a component uncovered area image from the PCBA finished product image according to the global mask image;

[0071] Further, according to the global mask image, the component coverage area image and the component uncovered area image are extracted from the PCBA finished product image in the following specific steps: aligning the global mask image with the PCBA finished product image to obtain an aligned global mask image and an aligned PCBA finished product image; performing a bitwise AND operation on the aligned global mask image and the aligned PCBA finished product image; matching the pixel points marked as 1 in the aligned global mask image with the aligned PCBA finished product image to obtain a first corresponding area; extracting the first corresponding area from the aligned PCBA finished product image to obtain the component coverage area image; the component coverage area image is the first corresponding area extracted from the PCBA finished product image, and the component coverage area image is composed of M rectangular images of different sizes, each rectangular image corresponds to a portion of the white area in the global mask image, and each rectangular block displays the area covered by the component in the PCBA finished product image. M is the total number of all workpieces on the PCBA template image, and is also the total number of offset frames. In this embodiment, M is 15, that is, there are 15 component coverage area images. At the same time, the pixel points marked as 0 in the aligned global mask image are matched with the aligned PCBA finished product image to obtain a second corresponding area, and the second corresponding area is extracted from the aligned PCBA finished product image to obtain the component uncovered area image. The component uncovered area image is the second corresponding area extracted from the PCBA finished product image. In this image, all parts corresponding to the black area in the global mask image will display the content in the PCBA finished product image, while the pixels corresponding to the white area in the global mask image will be shielded. The component uncovered area image only retains the exposed PCB board part in the finished image that is not covered by the component, and its shape and position are consistent with the black area in the global mask. The component uncovered area image presents a "hollowed out" effect as a whole, in which the component uncovered area is completely visible, and the component covered area is completely covered by black blocks. This processing method effectively highlights the component uncovered area and shields component interference.

[0072] Preferably, when collecting the finished PCBA image, some background or redundant areas around the PCB board are usually included. These non-PCBA board areas may interfere with subsequent alignment and bitwise operations. Therefore, before the alignment operation, the finished image needs to be edge cropped to ensure that the image only retains the complete area of ​​the PCBA board itself, so as to accurately match the global mask image.

[0073] This partitioned image extraction method achieves accurate separation of component coverage area images and component uncovered area images by aligning the global mask image with the PCBA finished product image and performing bitwise operations, allowing the inspection system to independently analyze the defects in each area. The component coverage area image retains the information of all component positions, which is helpful for the inspection of component mounting quality; while the component uncovered area image (the image of the area without component coverage on the PCBA board) shows the exposed PCB part in a "hollowed out" form, avoiding interference from components and making bare board defect detection more efficient and clear. This processing method not only improves the inspection accuracy, but also reduces misjudgments and missed inspections caused by component occlusion, thereby further improving the quality control capability and inspection efficiency of the production line.

[0074] Performing defect detection on the component coverage area image to obtain a component coverage area defect detection result;

[0075] Further, performing defect detection on the component coverage area image to obtain the component coverage area defect detection result includes:

[0076] Acquire historical defect samples containing different defect categories from a historical defect sample database; the defect categories include surface defects, component missing defects, and component layout angle defects;

[0077] Generate an expanded training sample based on the historical defect sample by data enhancement and a generative adversarial network (GAN), wherein all the expanded training samples and all the historical defect samples constitute an expanded historical defect sample data set;

[0078] According to the global mask image, the component coverage area is extracted from each sample image in the expanded historical defect sample data set to obtain a historical component coverage area image; the historical component coverage area image is 15 rectangular images of different sizes. Preferably, the historical component coverage area image is preprocessed, and the historical component coverage area image is enlarged and detailed using a super-resolution reconstruction method. The super-resolution method can make the image details clearer, which is convenient for the model to extract detail features.

[0079] Component defects are annotated for each pre-processed historical component coverage area image, including component surface defects, component missing defects, and component layout angle defects; specifically: use LabelIMG software to open the image, annotate the component surface defects for each historical component coverage area image, and annotate the location and category of the defects at the same time. The categories include but are not limited to the following defect types: scratches, stains, spots, bubbles, and deformations; at the same time, annotate the component missing defects for the historical component coverage area image, and determine whether the specified component exists in the offset frame area corresponding to the historical component coverage area image. The annotated result is existence or non-existence; at the same time, annotate the layout angle defects for the historical component coverage area image. If the specified component exists in the corresponding offset frame area, mark its installation offset angle. The reference angle (0°) is defined as the long side of the component parallel to the edge of the PCB board, and the direction from left to right is 0°. If the component is offset 5° clockwise relative to this direction, it is recorded as +5°. If the component is offset 5° counterclockwise relative to this direction, it is recorded as -5°.

[0080] Using the historical component coverage area image to train a PCBA comprehensive defect detection model;

[0081] The component coverage area image is preprocessed and then input into the trained PCBA comprehensive defect detection model to obtain the component coverage area defect detection results, including surface defect detection results, component missing defect detection results and component layout angle defect detection results.

[0082] This method extracts and expands multi-category defect samples from the historical defect sample database, and combines it with super-resolution reconstruction to enhance the image of the small component coverage area with clear details, thereby improving the model's ability to identify defects in small areas. Data enhancement is performed through the generative adversarial network (GAN), which enriches the model's training data, allowing the PCBA comprehensive defect detection model to more comprehensively identify various defect types such as surface defects, component missing, and layout angle offset. Based on the precise extraction method of the offset frame, the system can achieve high-precision defect detection in a small range, effectively reducing the missed detection and misjudgment rate of component defects, and providing higher reliability and accuracy for component defect identification in the production process.

[0083] Furthermore, the PCBA comprehensive defect detection model includes:

[0084] An input layer, used for receiving the pre-processed component coverage area image or the pre-processed historical component coverage area image;

[0085] Preferably, the component coverage area image is processed by a super-resolution reconstruction method in the preprocessing stage to enhance its resolution and detail information, and then the high-definition image is resized to a standard input size of 224×224 to meet the input requirements of the model.

[0086] A shared feature extraction layer, used to extract common features; the shared feature extraction layer is composed of a MobileNetV2 structure, and the shared feature extraction layer includes an initial convolution layer and multiple depth-separable convolution layers, and outputs a shared feature map;

[0087] A multi-task branch layer, used for performing feature extraction and detection of each task based on the shared feature map through a parallel multi-threading mechanism, and composed of a surface defect detection branch, a component missing defect detection branch, and a component layout angle defect detection branch;

[0088] The surface defect detection branch includes two feature extraction convolution layers and a global average pooling layer, which are used to extract surface defect features. The global average pooling layer is connected to a defect classification module and a defect localization module. The defect classification module implements defect classification output through a fully connected layer to determine the type of surface defects. The defect localization module outputs the location information of the surface defects through another fully connected layer to determine the position and size of the surface defects.

[0089] The component missing defect detection branch is used to determine whether a specified component is missing, and includes a feature extraction convolution layer, a global average pooling layer, and a fully connected layer. The feature extraction convolution layer is used to extract missing detection features, which are reduced to a one-dimensional vector by the global average pooling layer and then transferred to the fully connected layer, and the missing probability is output through the Sigmoid activation function;

[0090] The component layout angle defect detection branch is used to detect the layout deviation angle of the specified component when it is not missing. When the output result of the component missing defect detection branch is not missing, the component layout angle defect detection branch is activated and extracts the layout angle feature, which is processed by a feature extraction convolution layer and a global average pooling layer, and then connected to a regression module to output the angle deviation value; wherein the regression module is composed of a fully connected layer and an activation function; when the output result of the component missing defect detection branch is missing, the output of the component layout angle defect detection branch is empty;

[0091] A fusion layer, used to splice the outputs of the surface defect detection branch, the component missing defect detection branch and the component layout angle defect detection branch into an overall output vector;

[0092] The output layer is used to receive and output the overall output vector.

[0093] Preferably, conditional judgment is added in the forward propagation stage of the PCBA comprehensive defect detection model, and the output of the component missing detection branch is used as the Boolean condition:

[0094] If the component missing detection is "not missing" (1), activate the layout angle detection branch;

[0095] If the component missing detection is "missing" (0), the layout angle detection branch is skipped.

[0096] Specifically, in this embodiment, the conditional judgment logic is located after the output of the component missing detection branch but before the component layout angle detection branch. Whether to activate the layout angle detection branch can be determined according to the result of the missing detection.

[0097] Table 2 Key parameters of PCBA comprehensive defect detection model network structure

[0098] Layer Name Convolution kernel size Stride Number of output channels Output size Input Layer - - 3 224×224×3 Initial convolutional layer 3x3 2 32 112×112×32 Bottleneck block (13 layers) - - 320 7×7×320

[0099] See Table 2. The shared feature extraction layer is based on the MobileNetV2 bottleneck structure, which includes an initial convolution layer and multiple depth-separable convolution bottleneck blocks. Each bottleneck block consists of depth-wise convolution, point-wise convolution, and residual connection. By repeatedly stacking 13 MobileNetV2 bottleneck blocks, high-level features of the image are gradually extracted. Finally, the output shared feature map size is 7×7×320, which provides efficient and accurate feature representation for the subsequent multi-task branch layer.

[0100] Specifically, this embodiment provides a specific structure of a multi-task branch layer, see Table 3:

[0101] Table 3 Multi-task branch layer structure table

[0102]

[0103] The PCBA comprehensive defect detection model realizes efficient joint detection of component surface defects, missing components and layout angle deviations through shared feature extraction and multi-task branch structure. The shared feature extraction layer is based on the MobileNetV2 structure, which has the advantages of lightweight and efficient feature extraction, thereby reducing the model calculation amount and improving the detection speed; while the multi-task branch layer uses a parallel multi-threading mechanism to perform multiple detection tasks at the same time, making the detection process more efficient. Each task branch optimizes the feature extraction of surface defects, component missing components and layout angles respectively to ensure the recognition accuracy of each defect type. The final fusion layer integrates the multi-task output into an overall output vector, achieving efficient and accurate detection result output. This model structure not only significantly improves the detection efficiency, but also reduces the risk of missed detection and misjudgment.

[0104] Acquire the original image of the PCB corresponding to the image of the uncovered area of ​​the component to obtain a matching bare board image;

[0105] Further, acquiring the PCB original image corresponding to the component uncovered area image to obtain the matching bare board image includes: when acquiring the PCB original image, simultaneously recording the first serial number corresponding to the PCB original image; when acquiring the PCBA finished product image, simultaneously recording the second serial number corresponding to the PCBA finished product image; acquiring the first serial number corresponding to the component uncovered area image; and searching for the PCB original image corresponding to the component uncovered area image by matching the first serial number with the second serial number, thereby determining the matching bare board image.

[0106] Table 4 Serial number mapping table

[0107]

[0108] There is a serial number mapping table in the database, and the structure is shown in Table 4. When it is necessary to obtain the matching bare board image of the PCBA finished product image B002, the system will search for the first serial number A002 according to B002, and then obtain the corresponding PCB original image / path / to / pcb_image_A002.png from the original image path, and use it as the matching bare board image.

[0109] By recording and matching the serial numbers of the original PCB image and the finished PCBA image, accurate tracing of uncovered areas of components is achieved, thereby making full use of the original PCB images that have been inspected before. By matching the serial numbers, the system can directly find the original PCB image corresponding to the current finished PCBA product, making full use of the image information in the previous stage, and significantly improving the inspection efficiency. This method not only saves system resources and time, but also ensures the consistency and accuracy of inspection, providing efficient and reliable support for defect tracing and root cause analysis, thereby further optimizing the quality management process of the production line.

[0110] A bare board defect detection result is obtained according to the component uncovered area image and the matching bare board image.

[0111] Furthermore, obtaining the bare board defect detection result according to the component uncovered area image and the matched bare board image includes: preprocessing the component uncovered area image and the matched bare board image, including denoising, image alignment and contrast enhancement, to obtain a standard component uncovered area image and a standard matched bare board image; performing difference detection on the standard component uncovered area image and the standard matched bare board image to obtain a changed area; extracting an image containing the changed area to obtain a changed image; inputting the changed image into a trained bare board defect recognition model to obtain the bare board defect detection result; the bare board defect recognition model is a Faster R-CNN model trained based on historical bare board images.

[0112] When extracting the changed area image, only the changed part in the second corresponding area in the area image not covered by the component is extracted according to the global mask image, and the white area (first corresponding area) in the global mask image is set as the ignored area and does not participate in the image extraction.

[0113] This method makes full use of the PCB raw material images that have been inspected before SMT processing, and quickly determines whether new defects such as scratches or bruises are introduced in the uncovered area of ​​the component during the SMT process through difference detection. If there is no difference, there is no need to repeat the inspection, and the uncovered area of ​​the component is directly confirmed to be qualified, thereby reducing unnecessary calculations and significantly improving the inspection efficiency. For areas with differences, the system further uses the Faster R-CNN model for fine classification to identify the specific type of defects. This process not only ensures the accuracy of the inspection, but also minimizes the consumption of inspection resources, realizes fast and efficient defect screening, and provides an intelligent and streamlined solution for quality control in the production process.

[0114] Furthermore, the finished product defect detection module also includes: summarizing the bare board defect detection result and the component coverage area defect detection result to obtain a PCBA finished product defect detection result.

[0115] By summarizing the bare board defect detection results and the component coverage area defect detection results, a complete PCBA finished product defect detection result is generated, achieving a comprehensive quality assessment of the finished product.

[0116] Specifically, the component coverage area defect detection results need to be processed as follows before being summarized:

[0117] The component layout angle defect detection results in the component coverage area defect detection results are classified, specifically: the corresponding standard installation angle and allowable error are determined according to the mounting process file; the component layout angle defect detection result in each component coverage area defect detection result is compared with the standard installation angle, and the angle deviation is calculated; if the angle deviation is not within the allowable error range, the component is marked as having a component layout angle defect.

[0118] The training process of the bare board defect recognition model includes:

[0119] Obtain historical bare board defect samples containing different bare board defect categories from a historical defect sample database; the bare board defect categories include scratch defects, stain defects, oxidation spot defects, hole defects, crack defects, bubble defects, corrosion defects and edge collapse defects;

[0120] Generate augmented training samples based on historical bare board defect samples through data enhancement and generative adversarial network (GAN), where all the augmented training samples and all the historical defect samples constitute an augmented historical bare board defect sample data set;

[0121] According to the global mask image, a component uncovered area (component uncovered area) is extracted from each sample image in the expanded historical bare board defect sample data set to obtain a historical bare board image.

[0122] Label the bare board defects for each pre-processed historical component coverage area image, and mark the defect location and defect category. The defect categories include scratch defects, stain defects, oxidation spot defects, hole defects, crack defects, bubble defects, corrosion defects and edge collapse defects. Specifically, use LabelIMG software to open the image and label the component surface defects for each historical bare board image.

[0123] Obtain annotated historical bare board images from the expanded historical bare board defect sample dataset and divide them into training and validation sets. Use the pre-trained Faster R-CNN model as the base model and fine-tune it through transfer learning. After training, evaluate the model performance through the validation set, adjust the hyperparameters to improve the detection effect, and finally deploy it for actual detection to obtain a bare board defect recognition model;

[0124] In order to verify whether the scheme of dividing the defect detection task into two parts, component coverage area and component uncovered area, for separate detection in the present invention can improve the efficiency and accuracy of defect detection, a set of comparative experiments was conducted. The experimental group used a regional detection method, that is, defect detection was performed on the component coverage area and the component uncovered area respectively, and the component defect detection results and the bare board defect detection results were obtained respectively. Control group 1 and control group 2 used the traditional integrated detection method to directly perform defect detection on the entire PCBA finished product image without distinguishing between component coverage area and component uncovered area. Among them, control group 1 was based on the Faster R-CNN target detection model; control group 2 was based on the YOLO (You Only Look Once) v5 model. Control group 3 constructed a multi-task learning (MTL) model, which had the same structure as the PCBA comprehensive defect detection model of the experimental group, including surface defect detection branch, component missing detection branch and component layout angle detection branch, and used a shared feature extraction layer (based on MobileNetV2) to extract common features. Unlike the experimental group, the training data of the control group 3 model was the entire PCBA finished product image, without distinguishing between component coverage area and component uncovered area. The model directly performs defect detection on the entire image, allowing the model to learn defect features in all areas without regionalizing component areas and component uncovered areas.

[0125] Table 5 Comparison experimental data between regional detection and integrated detection

[0126]

[0127] The experimental results are shown in Table 5, where the average detection time is the defect detection time for each complete PCBA image. The regional detection scheme proposed in the present invention is not only superior to the traditional integrated detection method in terms of accuracy and pass rate control, but also achieves a good balance in efficiency, and is particularly suitable for actual production environments that require high accuracy and low false alarm rate. Although the YOLOv5 in the control group 2 is the fastest, it is slightly insufficient in accuracy.

[0128] In order to verify that the MTL-based PCBA comprehensive defect detection model proposed in the present invention is superior to the existing technology and common models in terms of accuracy and efficiency, another set of comparative experiments was carried out. To ensure the comparability of the experiments, all experiments adopted a regional detection method, and defect detection was performed based on the component coverage area image.

[0129] The experimental group used a PCBA comprehensive defect detection model based on multi-task learning (MTL). The model includes a surface defect detection branch, a component missing detection branch, and a component layout angle detection branch. It uses a shared feature extraction layer (such as MobileNetV2) to extract common features, and processes multiple detection tasks in parallel in the multi-task branch, outputting the detection results of all defects at once.

[0130] Control group 1: Three independent single-task inspection models were used to detect surface defects, component missing, and component layout angle deviation.

[0131] Surface defect detection model: Use Faster R-CNN to detect surface defects (such as scratches, stains, etc.), and output the location and category of surface defects.

[0132] Component missing detection model: Use VGG16 to determine whether a component is missing.

[0133] Component layout angle detection model: Use InceptionV3 to detect the layout angle deviation of components.

[0134] Table 6 Model comparison experimental data table

[0135]

[0136] The experimental results are shown in Table 6. The PCBA comprehensive defect detection model of the experimental group is superior to the control group 1 in terms of accuracy (96.2%) and average detection time (0.50 seconds).

[0137] The design of separately detecting component covered areas and component uncovered areas can better meet different detection needs and improve detection accuracy and efficiency. The component covered area mainly detects defects such as the presence, position, installation angle, and surface damage of the component, while the component uncovered area focuses on problems such as scratches, contamination, and solder residues. Through separate detection, the interference of components on the uncovered area of ​​the component can be avoided, making the detection results more focused and accurate. At the same time, such separation processing can use different detection algorithms for different areas, which can not only use the MTL model to efficiently detect complex defects of components, but also realize efficient and accurate detection of uncovered areas of components through differential detection combined with classification models. Thereby achieving a better balance between overall detection accuracy and efficiency. Optimize detection efficiency. In addition, separate detection helps to locate the source of defects more accurately, facilitate rapid tracing of problems and quality control, thereby improving the overall quality and reliability of SMT production.

[0138] Embodiment 2:

[0139] On the SMT production line of a large electronics manufacturing plant B, after the PCBA boards are mounted and soldered, they are sequentially sent to the visual inspection device via a conveyor belt to ensure that each PCBA board meets the quality standards. In traditional inspections, traditional machine vision systems often have difficulty efficiently and accurately detecting tiny surface defects and component position deviations, which can easily lead to missed inspections or false inspections, affecting the quality of finished products.

[0140] In order to improve the detection efficiency and accuracy, the factory introduced an SMT surface defect visual inspection device based on AI image recognition. The device can detect PCBA surface defects in real time to identify various defects in the SMT process and ensure that each PCBA board meets the factory standards.

[0141] An SMT surface defect visual detection device based on AI image recognition, comprising:

[0142] A template image storage module is used to obtain and store PCBA template images;

[0143] An offset frame and mask image generation module is used to determine the mounting position and the allowable range of mounting offset error of each workpiece according to the mounting process file; determine M offset frames according to the mounting position and the allowable range of mounting offset error corresponding to each workpiece, where M is the total number of all workpieces on the PCBA template image; merge all the offset frames to generate a global mask image; the global mask image is used to define and identify component coverage areas and component uncovered areas;

[0144] The image acquisition module is used to collect the bare board image before the whole process of surface mounting of components on the designated production line in real time to obtain the original PCB image; and to collect the finished board image after the whole process of surface mounting of components on the production line to obtain the PCBA finished product image;

[0145] A bare board defect detection module is used to perform defect detection on the PCB original image to obtain a bare board defect detection result;

[0146] The finished product defect detection module is used to perform defect detection on the PCBA finished product image to obtain a finished product defect detection result, including the following steps:

[0147] Extracting a component covered area image and a component uncovered area image from the PCBA finished product image according to the global mask image;

[0148] Performing defect detection on the component coverage area image to obtain a component coverage area defect detection result;

[0149] Acquire the original image of the PCB corresponding to the image of the uncovered area of ​​the component to obtain a matching bare board image;

[0150] A bare board defect detection result is obtained according to the component uncovered area image and the matching bare board image.

[0151] Furthermore, the PCBA template image is specifically: the PCBA template image is a reference image of the surface of a defect-free finished board that has gone through the entire process of component surface mounting on the production line.

[0152] Furthermore, after performing defect detection on the PCB original image and obtaining the bare board defect detection result, the following steps are also included: dividing the PCB original image into a normal raw material sample and a defective raw material sample according to the bare board defect detection result; when the defective raw material sample is detected, giving an early warning to the production line, including the following processes:

[0153] Obtain a first serial number corresponding to the defective raw material sample, and track the defective raw material sample according to the first serial number; notify staff or sorting equipment to remove the defective raw material sample from the production line and place it in an isolation area.

[0154] Furthermore, the global mask image is specifically: in the global mask image, the area covered by the offset box is marked as 1, and the area not covered by the offset box is marked as 0.

[0155] Further, according to the global mask image, the component coverage area image and the component uncovered area image are extracted from the PCBA finished product image specifically as follows: the global mask image is aligned with the PCBA finished product image to obtain an aligned global mask image and an aligned PCBA finished product image; a bitwise AND operation is performed on the aligned global mask image and the aligned PCBA finished product image; the pixel points marked as 1 in the aligned global mask image are matched with the aligned PCBA finished image to obtain a first corresponding area; the first corresponding area is extracted from the aligned PCBA finished image to obtain the component coverage area image; at the same time, the pixel points marked as 0 in the aligned global mask image are matched with the aligned PCBA finished image to obtain a second corresponding area, and the second corresponding area is extracted from the aligned PCBA finished image to obtain the component uncovered area image.

[0156] Further, performing defect detection on the component coverage area image to obtain the component coverage area defect detection result includes:

[0157] Acquire historical defect samples containing different defect categories from a historical defect sample database; the defect categories include surface defects, component missing defects, and component layout angle defects;

[0158] Generate an expanded training sample based on the historical defect sample by data enhancement and generative adversarial network, wherein all the expanded training samples and all the historical defect samples constitute an expanded historical defect sample data set;

[0159] According to the global mask image, extracting a component coverage area from each sample image in the expanded historical defect sample data set to obtain a historical component coverage area image;

[0160] Using the historical component coverage area image to train a PCBA comprehensive defect detection model;

[0161] The component coverage area image is preprocessed and then input into the trained PCBA comprehensive defect detection model to obtain the component coverage area defect detection results, including surface defect detection results, component missing defect detection results and component layout angle defect detection results.

[0162] Furthermore, the PCBA comprehensive defect detection model includes:

[0163] An input layer, used for receiving the pre-processed component coverage area image or the pre-processed historical component coverage area image;

[0164] A shared feature extraction layer, used to extract common features and output a shared feature map; the shared feature extraction layer includes an initial convolution layer and multiple depth-separable convolution bottleneck blocks; each of the depth-separable convolution bottleneck blocks consists of depth-wise convolution, point-by-point convolution and residual connection;

[0165] A multi-task branch layer, used for performing feature extraction and detection of each task based on the shared feature map through a parallel multi-threading mechanism, and composed of a surface defect detection branch, a component missing defect detection branch, and a component layout angle defect detection branch;

[0166] The surface defect detection branch includes two first feature extraction convolution layers and a first global average pooling layer, which are used to extract surface defect features, and the first global average pooling layer is connected to a defect classification module and a defect localization module; wherein the defect classification module implements defect category classification output through a fully connected layer, which is used to determine the type of surface defects; the defect localization module outputs the location information of the surface defects through the fully connected layer, which is used to determine the position and size of the surface defects;

[0167] The component missing defect detection branch is used to determine whether a specified component is missing, and includes a second feature extraction convolution layer, a second global average pooling layer, and a second fully connected layer. The second feature extraction convolution layer is used to extract missing detection features, which are reduced to a one-dimensional vector by the second global average pooling layer and then transferred to the second fully connected layer, and the missing probability is output through the Sigmoid activation function;

[0168] The component layout angle defect detection branch is used to detect the layout deviation angle of the specified component when it is not missing. When the output result of the component missing defect detection branch is not missing, the component layout angle defect detection branch is activated and extracts the layout angle feature, which is processed by a third feature extraction convolution layer and a third global average pooling layer, and then connected to a regression module to output the angle deviation value; wherein the regression module is composed of a fully connected layer and an activation function; when the output result of the component missing defect detection branch is missing, the output of the component layout angle defect detection branch is empty;

[0169] A fusion layer, used to splice the outputs of the surface defect detection branch, the component missing defect detection branch and the component layout angle defect detection branch into an overall output vector;

[0170] The output layer is used to receive and output the overall output vector.

[0171] Further, acquiring the PCB original image corresponding to the component uncovered area image to obtain the matching bare board image includes: when acquiring the PCB original image, simultaneously recording the first serial number corresponding to the PCB original image; when acquiring the PCBA finished product image, simultaneously recording the second serial number corresponding to the PCBA finished product image; acquiring the first serial number corresponding to the component uncovered area image; and searching for the PCB original image corresponding to the component uncovered area image by matching the first serial number with the second serial number, thereby determining the matching bare board image.

[0172] Furthermore, obtaining the bare board defect detection result according to the component uncovered area image and the matched bare board image includes: preprocessing the component uncovered area image and the matched bare board image, including denoising, image alignment and contrast enhancement, to obtain a standard component uncovered area image and a standard matched bare board image; performing difference detection on the standard component uncovered area image and the standard matched bare board image to obtain a changed area; extracting an image containing the changed area to obtain a changed image; inputting the changed image into a trained bare board defect recognition model to obtain the bare board defect detection result; the bare board defect recognition model is a Faster R-CNN model trained based on historical bare board images.

[0173] Furthermore, the finished product defect detection module also includes: summarizing the bare board defect detection result and the component coverage area defect detection result to obtain a PCBA finished product defect detection result.

[0174] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A SMT surface defect visual inspection device based on AI image recognition, characterized in that: include: A template image storage module is used to obtain and store PCBA template images; An offset frame and mask image generation module is used to determine the mounting position and the allowable range of mounting offset error of each workpiece according to the mounting process file; determine M offset frames according to the mounting position and the allowable range of mounting offset error corresponding to each workpiece, where M is the total number of all workpieces on the PCBA template image; and merge all the offset frames to generate a global mask image; The global mask image is used to define and identify component covered areas and component uncovered areas; An image acquisition module is used to acquire the original image of the PCB before the whole process of surface mounting of components on a designated production line; and to acquire the image of the PCBA finished product after the whole process of surface mounting of components on the production line; A bare board defect detection module is used to perform defect detection on the PCB original image to obtain a bare board defect detection result; The finished product defect detection module is used to perform defect detection on the PCBA finished product image to obtain a finished product defect detection result, comprising the following steps: extracting a component coverage area image and a component uncovered area image from the PCBA finished product image according to the global mask image; performing defect detection on the component coverage area image to obtain a component coverage area defect detection result; acquiring the PCB original image corresponding to the component uncovered area image to obtain a matching bare board image; and obtaining the bare board defect detection result according to the component uncovered area image and the matching bare board image.

2. According to claim 1, a SMT surface defect visual detection device based on AI image recognition is characterized in that: The PCBA template image is specifically: the PCBA template image is a reference image of the surface of a defect-free finished board that has gone through the entire process of component surface mounting on the production line.

3. The SMT surface defect visual detection device based on AI image recognition according to claim 1 is characterized in that: After performing defect detection on the PCB original image and obtaining the bare board defect detection result, the following steps are also included: dividing the PCB original image into normal raw material samples and defective raw material samples according to the bare board defect detection result; when the defective raw material sample is detected, an early warning is issued to the production line, including: obtaining a first serial number corresponding to the defective raw material sample, and tracking the defective raw material sample according to the first serial number; notifying staff or sorting equipment to remove the defective raw material sample from the production line and put it into an isolation area.

4. The SMT surface defect visual detection device based on AI image recognition according to claim 1 is characterized in that: The global mask image is specifically: in the global mask image, the area covered by the offset frame is marked as 1, and the area not covered by the offset frame is marked as 0.

5. The SMT surface defect visual detection device based on AI image recognition according to claim 1 is characterized in that: Extracting the component coverage area image and the component uncovered area image from the PCBA finished product image according to the global mask image is specifically as follows: aligning the global mask image with the PCBA finished product image to obtain an aligned global mask image and an aligned PCBA finished product image; performing a bitwise AND operation on the aligned global mask image and the aligned PCBA finished product image; matching the pixel points marked as 1 in the aligned global mask image with the aligned PCBA finished product image to obtain a first corresponding area; The first corresponding area is extracted from the aligned PCBA finished product image to obtain the component coverage area image; at the same time, the pixel points marked as 0 in the aligned global mask image are matched with the aligned PCBA finished product image to obtain the second corresponding area, and the second corresponding area is extracted from the aligned PCBA finished product image to obtain the component uncovered area image.

6. The SMT surface defect visual detection device based on AI image recognition according to claim 1 is characterized in that: Performing defect detection on the component coverage area image to obtain the component coverage area defect detection result includes: Acquire historical defect samples containing different defect categories from a historical defect sample database; the defect categories include surface defects, component missing defects, and component layout angle defects; Generate an expanded training sample based on the historical defect sample by data enhancement and generative adversarial network, wherein all the expanded training samples and all the historical defect samples constitute an expanded historical defect sample data set; According to the global mask image, extracting a component coverage area from each sample image in the expanded historical defect sample data set to obtain a historical component coverage area image; Using the historical component coverage area image to train a PCBA comprehensive defect detection model; The component coverage area image is preprocessed and then input into the trained PCBA comprehensive defect detection model to obtain the component coverage area defect detection results, including surface defect detection results, component missing defect detection results and component layout angle defect detection results.

7. The SMT surface defect visual detection device based on AI image recognition according to claim 6 is characterized in that: The PCBA comprehensive defect detection model includes: An input layer, used for receiving the pre-processed component coverage area image or the pre-processed historical component coverage area image; A shared feature extraction layer, used to extract common features and output a shared feature map; the shared feature extraction layer includes an initial convolution layer and multiple depth-separable convolution bottleneck blocks; each of the depth-separable convolution bottleneck blocks consists of depth-wise convolution, point-by-point convolution and residual connection; A multi-task branch layer, used for performing feature extraction and detection of each task based on the shared feature map through a parallel multi-threading mechanism, and composed of a surface defect detection branch, a component missing defect detection branch, and a component layout angle defect detection branch; The surface defect detection branch includes two first feature extraction convolution layers and a first global average pooling layer, which are used to extract surface defect features, and the first global average pooling layer is connected to a defect classification module and a defect localization module; wherein the defect classification module implements defect category classification output through a first fully connected layer, which is used to determine the type of surface defects; the defect localization module outputs the location information of the surface defects through a second fully connected layer, which is used to determine the position and size of the surface defects; The component missing defect detection branch is used to determine whether a specified component is missing, and includes a second feature extraction convolution layer, a second global average pooling layer, and a third fully connected layer. The second feature extraction convolution layer is used to extract missing detection features, which are reduced to a one-dimensional vector by the second global average pooling layer and then transferred to the third fully connected layer, and the missing probability is output through the Sigmoid activation function; The component layout angle defect detection branch is used to detect the layout deviation angle of the specified component when it is not missing. When the output result of the component missing defect detection branch is not missing, the component layout angle defect detection branch is activated and extracts the layout angle feature, which is processed by a third feature extraction convolution layer and a third global average pooling layer, and then connected to a regression module to output the angle deviation value; wherein the regression module is composed of a fully connected layer and an activation function; when the output result of the component missing defect detection branch is missing, the output of the component layout angle defect detection branch is empty; A fusion layer, used to splice the outputs of the surface defect detection branch, the component missing defect detection branch and the component layout angle defect detection branch into an overall output vector; The output layer is used to receive and output the overall output vector.

8. The SMT surface defect visual detection device based on AI image recognition according to claim 1, characterized in that: Acquiring the PCB original image corresponding to the component uncovered area image to obtain the matching bare board image includes: when acquiring the PCB original image, simultaneously recording a first serial number corresponding to the PCB original image; when acquiring the PCBA finished product image, simultaneously recording a second serial number corresponding to the PCBA finished product image; acquiring the first serial number corresponding to the component uncovered area image; and determining the matching bare board image by matching the first serial number with the second serial number.

9. The SMT surface defect visual detection device based on AI image recognition according to claim 1, characterized in that: Obtaining the bare board defect detection result according to the component uncovered area image and the matched bare board image includes: preprocessing the component uncovered area image and the matched bare board image, including denoising, image alignment and contrast enhancement, to obtain a standard component uncovered area image and a standard matched bare board image; performing difference detection on the standard component uncovered area image and the standard matched bare board image to obtain a changed area; extracting an image containing the changed area to obtain a changed image; inputting the changed image into a trained bare board defect recognition model to obtain the bare board defect detection result; the bare board defect recognition model is a Faster R-CNN model trained according to historical bare board images.

10. The SMT surface defect visual detection device based on AI image recognition according to claim 1, characterized in that: The finished product defect detection module further includes: summarizing the bare board defect detection result and the component coverage area defect detection result to obtain a PCBA finished product defect detection result.

Images