Solder paste plate surface foreign matter detection method and system based on AI drive and medium

Abstract

The invention discloses a solder paste plate surface foreign matter detection method and system based on AI driving and a medium, and the method comprises the following steps: collecting a to-be-detected original image of the surface of a solder paste plate, and carrying out the Mark point positioning of the solder paste plate and the image correction of the solder paste plate; extracting a solder paste detection area in the to-be-detected original image by adopting an AI-based semantic segmentation algorithm; cutting and toning the solder paste detection area to obtain a plurality of local small images; performing foreign matter cooperative detection on each local small image by adopting an AI-based target detection algorithm and a reference image-based detection algorithm; integrating the foreign matter detection results of the plurality of local small graphs, and outputting a final foreign matter recognition result; according to the method, the solder paste detection area can be automatically positioned through AI semantic segmentation, the tedious operation of manually drawing the area is abandoned, and the detection convenience is improved; aI target detection based on small target optimization and a reference image detection algorithm are adopted for cooperative work, and the overall recognition accuracy is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of industrial machine vision inspection technology, and in particular to an AI-driven method, system and medium for detecting foreign objects on the surface of solder paste boards. Background Technology

[0002] Solder paste boards are prone to various foreign objects such as iron filings, dust, and component debris on their surface. Detection of foreign objects on their surface is a key process in reducing the defect rate during the circuit board assembly process.

[0003] Currently, the industry mostly adopts a detection scheme that combines manual quality inspection with traditional machine vision algorithms. Traditional machine vision usually first completes the overall positioning of the solder paste board through the Mark positioning algorithm, and then relies on customized algorithms such as grayscale, vector, and contour to achieve defect detection.

[0004] However, this model has many drawbacks, as follows:

[0005] (1) Manual quality inspection is inefficient;

[0006] (2) Traditional algorithms are sensitive to noise, and the recognition effect drops sharply when the image is blurred. In addition, features need to be manually calibrated, and the development cycle is long.

[0007] (3) At the same time, due to the variety of foreign objects and the complex surface features of solder paste boards, the algorithm has poor universality and a high false detection rate, so a lot of manual verification is still needed. Summary of the Invention

[0008] The purpose of this invention is to provide an AI-driven method, system, and medium for detecting foreign objects on the surface of solder paste boards, thereby solving all or one of the aforementioned problems in the prior art.

[0009] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0010] On one hand, the present invention provides an AI-driven method for detecting foreign objects on the surface of solder paste, comprising the following steps:

[0011] AI-driven region segmentation processing: Acquire the original image of the solder paste board surface to be detected, perform solder paste board Mark point localization and solder paste board image correction; use an AI-based semantic segmentation algorithm to extract the solder paste detection region in the original image to be detected; perform cropping and color adjustment on the solder paste detection region to obtain several local small images;

[0012] Foreign object detection processing using dual algorithms: For each of the local small images, a target detection algorithm based on AI and a detection algorithm based on a reference image are used to perform foreign object collaborative detection; the foreign object detection results of several local small images are integrated to output the final foreign object identification result.

[0013] As an improved solution, the solder paste board mark point positioning and solder paste board image correction further include:

[0014] A template matching, circular or cross-shaped pattern detection method is used to identify several Mark points on the solder paste board and obtain the coordinate information of the Mark points.

[0015] The original image to be detected is corrected by affine transformation.

[0016] As an improved approach, the AI-based semantic segmentation algorithm is a semantic segmentation algorithm that combines a spatial attention module and a convolutional neural network.

[0017] The algorithm encoding layer of the AI-based semantic segmentation algorithm is: a convolutional neural network and a downsampling module for extracting shallow semantic information;

[0018] The decoding layer of the AI-based semantic segmentation algorithm consists of: a convolutional layer for mining deep semantics, a spatial attention module for increasing the weight of solder paste regions through convolution and max pooling, and an upsampling module for spatially expanding features so that the output feature map is the same size as the input image.

[0019] As an improved solution, the process of cropping and color-adjusting the solder paste detection area to obtain several local small images further includes:

[0020] Connected components are extracted from the mask image of the solder paste detection area, the bounding rectangle of each connected component is calculated, and the original image and mask image at the corresponding position are cropped according to the bounding rectangle.

[0021] The image within the solder paste detection area is retained based on the cropped mask image, while the image outside the solder paste detection area is set to black.

[0022] As an improved approach, the method of performing collaborative foreign object detection using an AI-based target detection algorithm and a reference image-based detection algorithm for each of the local small images further includes:

[0023] The AI-based target detection algorithm is used to detect small-sized foreign objects in each of the local small images;

[0024] The reference image-based detection algorithm is used to detect large foreign objects in each of the local small images.

[0025] As an improved solution, the AI-based target detection algorithm includes:

[0026] The shallow semantic information of the local small image is obtained through the shallow feature extraction module;

[0027] The deep features of the local small image are obtained through the deep semantic feature extraction module;

[0028] The deep features are upsampled and then fused with the shallow semantic information;

[0029] During the fusion process, small target feature enhancement is performed based on the channel attention feature extraction module to obtain a large spatial size feature map containing feature information of small-sized foreign objects.

[0030] As an improved approach, the reference image-based detection algorithm includes:

[0031] A normal image without foreign objects is acquired at the corresponding local small image location as a reference image;

[0032] The reference image and the local small image are input into the Siamese neural network to extract basic features, then the feature alignment module completes feature alignment, and finally the deep feature extraction module extracts deep semantics and outputs the detection results.

[0033] As an improved approach, the integration of foreign object detection results from several local small images to output the final foreign object identification result further includes:

[0034] The dual-algorithm detection results of all the aforementioned local small images are summarized, and the recognition results at overlapping locations are merged.

[0035] During the merging process, the overlap ratio is determined based on the location of the foreign object, and the bounding rectangle of the target with a high overlap ratio is output as the final result of the foreign object's location and size.

[0036] On the other hand, the present invention also provides an AI-driven foreign object detection system for solder paste surfaces, comprising:

[0037] The AI-driven region segmentation processing module is used to: acquire the original image of the solder paste board surface to be detected, locate the solder paste board Mark points and correct the solder paste board image; extract the solder paste detection area in the original image to be detected using an AI-based semantic segmentation algorithm; and perform cropping and color adjustment on the solder paste detection area to obtain several local small images.

[0038] The dual-algorithm collaborative foreign object detection processing module is used to: perform collaborative foreign object detection on each of the local small images by using an AI-based target detection algorithm and a reference image-based detection algorithm; integrate the foreign object detection results of several local small images, and output the final foreign object identification result.

[0039] On the other hand, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the AI-driven solder paste surface foreign object detection method.

[0040] The beneficial effects of the technical solution of this invention are:

[0041] 1. The AI-driven foreign object detection method for solder paste board surface described in this invention can automatically locate the solder paste detection area through AI semantic segmentation, eliminating the tedious operation of manually drawing the area and improving the convenience of detection; it adopts a small target optimized AI target detection and reference image detection algorithm to work together, which can adapt to foreign objects of different sizes, effectively distinguish between solder dross and real foreign objects, greatly enhance the robustness of detection, and significantly improve the overall recognition accuracy.

[0042] 2. The AI-driven solder paste surface foreign object detection system of the present invention can realize the AI-driven solder paste surface foreign object detection method of the present invention through the cooperation of system modules.

[0043] 3. The computer-readable storage medium of the present invention can enable the guidance system module to cooperate and thereby realize the AI-driven foreign object detection method on the surface of solder paste board of the present invention. The computer-readable storage medium of the present invention also effectively improves the operability of the AI-driven foreign object detection method on the surface of solder paste board. Attached Figure Description

[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a detailed flowchart illustrating the AI-driven foreign object detection method for solder paste board surface as described in Embodiment 1 of the present invention.

[0046] Figure 2 This is a schematic diagram of the logic flow of the AI-driven foreign object detection method for solder paste board surface described in Embodiment 1 of the present invention;

[0047] Figure 3 This is a schematic diagram of the algorithm logic of the AI-based semantic segmentation algorithm in the AI-driven solder paste surface foreign object detection method described in Embodiment 1 of the present invention;

[0048] Figure 4 This is a schematic diagram of the algorithm logic of the AI-based target detection algorithm in the AI-driven solder paste surface foreign object detection method described in Embodiment 1 of the present invention;

[0049] Figure 5This is a schematic diagram of the algorithm logic of the detection algorithm based on the reference image in the AI-driven solder paste surface foreign object detection method described in Embodiment 1 of the present invention;

[0050] Figure 6 This is a schematic diagram of the architecture of the AI-driven foreign object detection system for solder paste board surface as described in Embodiment 2 of the present invention. Detailed Implementation

[0051] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0052] In the description of this invention, it should be noted that the embodiments described in this invention are only some embodiments of this invention, not all embodiments; based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0053] The terms "first," "second," etc., used in this specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0054] Example 1: This example provides an AI-driven method for detecting foreign objects on the surface of solder paste boards, such as... Figures 1-5 As shown, it includes the following steps:

[0055] The core function of this method is: an artificial intelligence-based method for detecting foreign objects on the surface of solder paste boards. It achieves automatic localization of solder paste areas through AI segmentation and completes foreign object identification by combining dual-algorithm collaborative detection. The specific steps are as follows:

[0056] S100, Image Acquisition Steps, including:

[0057] S101. Use a camera to photograph the surface of the solder paste board to obtain the original image to be inspected, providing basic data for subsequent full-process inspection.

[0058] S200, Mark point localization and image correction steps, including:

[0059] S201. Use a preset Mark point recognition algorithm (such as template matching, circular or cross pattern detection method) to identify multiple Mark points on the solder paste board and obtain their coordinate information;

[0060] S202. The solder paste board image is corrected by affine transformation to achieve initial alignment between the detection box and the image, ensuring the accuracy of subsequent detection areas.

[0061] S300, AI-driven solder paste region segmentation steps, including:

[0062] S301. A semantic segmentation algorithm combining spatial attention module and convolutional neural network is used to extract solder paste detection region;

[0063] The algorithm coding layer, consisting of a convolutional neural network and a downsampling module, is responsible for extracting shallow semantic information.

[0064] The algorithm decoding layer integrates convolutional layers, an upsampling module, and a spatial attention module. The convolutional layer is used to mine deep semantics. The spatial attention module is used to increase the weight of the solder paste region through convolution and max pooling (it multiplies the weights calculated by the spatial attention module with the original features, making the network pay more attention to the solder paste region in space). The upsampling module is used to expand the features in space to ensure that the output feature map is the same size as the input image, thereby automatically dividing the solder paste detection region and the non-detection region, replacing the tedious operation of manually drawing the region.

[0065] S400, Solder paste area image cropping steps, including:

[0066] S401. Extract connected components from the solder paste region mask image, calculate the bounding rectangle of each connected component, and crop the original image and mask image at the corresponding position based on the bounding rectangle.

[0067] S402. Based on the mask image, retain the image within the detection area and set the image outside the area to black to obtain multiple small local images of the solder paste area, which facilitates subsequent fine detection.

[0068] S500, dual-algorithm collaborative foreign object detection steps, including:

[0069] S501. For foreign objects of different sizes and characteristics, an artificial intelligence target detection algorithm and a reference image detection algorithm are used to collaboratively detect each small image, achieving accurate identification of both small and large foreign objects, as follows:

[0070] S5011. Small-sized foreign object detection is performed using an artificial intelligence target detection algorithm: First, shallow semantic information is obtained through a shallow feature extraction module (the core of which is a convolutional neural network), and then deep features are obtained through a deep semantic feature extraction module. After upsampling the deep features, they are fused with the shallow high-resolution features. During the fusion process, the channel attention feature extraction module is used to enhance the features of small targets and generate a large spatial size feature map, ensuring that the feature information of small-sized foreign objects is completely preserved.

[0071] S5012. Large-sized foreign object detection is performed using a reference image-based detection algorithm: First, a normal image without foreign objects at the corresponding location is acquired as a reference image; during the testing phase, the reference image and the image to be tested are input into a Siamese neural network (a convolutional neural network with the same structure and weights) to extract basic features, and then the feature alignment module completes feature alignment. Finally, the deep feature extraction module (including convolutional layers and upsampling modules) extracts deep semantics and outputs the detection results.

[0072] Feature alignment includes:

[0073] Spatial location matching is achieved by calculating the cosine similarity of features Fq in the reference image and Fp in the test image at different spatial locations through 1×1 convolution dimensionality reduction. Specifically, for any feature Fq(i,j) at position i,j in the reference image, the corresponding feature in the test image is argmin(Fq(i,j),Fp(m,n)). The smaller the cosine similarity, the closer the spatial locations are. The feature Fp(m,n) at the position with the smallest cosine similarity in the test image is selected as the corresponding feature at that position, thereby achieving spatial alignment of the two sets of features.

[0074] S600, Combined output of test results:

[0075] S601. Summarize the dual-algorithm detection results of all local small images and merge the recognition results of overlapping positions;

[0076] S602. During the merging process, the location of the foreign object will be used to determine whether there is a high overlap ratio. The bounding rectangle of the target with a high overlap ratio will be calculated as the final location and size of the foreign object, and a unified foreign object detection result on the solder paste surface will be output.

[0077] It should be noted that the above examples are merely for explaining the present invention and should not be construed as limiting the scope of protection of the present invention.

[0078] Example 2: This example is based on the same inventive concept as the AI-driven foreign object detection method for solder paste surfaces described in Example 1, and provides an AI-driven foreign object detection system for solder paste surfaces, such as... Figure 6 As shown, it includes:

[0079] The AI-driven region segmentation processing module is used to: acquire the original image of the solder paste board surface to be detected, locate the solder paste board Mark points and correct the solder paste board image; extract the solder paste detection area in the original image to be detected using an AI-based semantic segmentation algorithm; and perform cropping and color adjustment on the solder paste detection area to obtain several local small images.

[0080] The dual-algorithm collaborative foreign object detection processing module is used to: perform collaborative foreign object detection on each of the local small images by using an AI-based target detection algorithm and a reference image-based detection algorithm; integrate the foreign object detection results of several local small images, and output the final foreign object identification result.

[0081] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0082] Example 3: This example provides a computer-readable storage medium, including:

[0083] The storage medium is used to store computer software instructions for implementing the AI-driven solder paste surface foreign object detection method described in Embodiment 1 above. It includes a program for executing the above-described AI-driven solder paste surface foreign object detection method. Specifically, the executable program can be built into the AI-driven solder paste surface foreign object detection system described in Embodiment 2. In this way, the AI-driven solder paste surface foreign object detection system can implement the AI-driven solder paste surface foreign object detection method described in Embodiment 1 by executing the built-in executable program.

[0084] Furthermore, the computer-readable storage medium in this embodiment can be any combination of one or more readable storage media, wherein the readable storage medium includes an electrical, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0085] Unlike existing technologies, the method, system and medium for detecting foreign objects on the surface of solder paste boards based on AI in this application can automatically locate the solder paste detection area through AI semantic segmentation, eliminating the tedious operation of manually drawing the area and improving the convenience of detection. The AI ​​target detection and reference image detection algorithms optimized for small targets work together to adapt to foreign objects of different sizes, effectively distinguish between solder dross and real foreign objects, greatly enhance the robustness of detection and significantly improve the overall recognition accuracy.

[0086] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0087] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0088] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0089] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0091] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0092] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An AI-driven based paste board surface foreign matter detection method, characterized in that, The method comprises the following steps: AI-driven region segmentation processing: collecting a tin paste plate surface original image to be detected, locating tin paste plate Mark points, and correcting the tin paste plate image; using an AI-based semantic segmentation algorithm to extract the tin paste detection area in the original image to be detected; performing cropping and color adjustment on the tin paste detection area to obtain a plurality of local small images; Dual-algorithm collaborative foreign matter detection processing: using an AI-based target detection algorithm and a reference image-based detection algorithm to collaboratively detect foreign matters in each local small image; integrating foreign matter detection results of a plurality of local small images to output a final foreign matter recognition result.

2. The AI-driven tin paste plate surface foreign matter detection method according to claim 1, wherein: the tin paste plate Mark point positioning and tin paste plate image correction further comprise: using template matching, circular or cross-shaped pattern detection to identify a plurality of Mark points on the tin paste plate to obtain coordinate information of the Mark points; and correcting the original image to be detected through affine transformation.

3. The AI-driven tin paste plate surface foreign matter detection method according to claim 1, wherein: the AI-based semantic segmentation algorithm is a semantic segmentation algorithm combining a spatial attention module and a convolutional neural network; the algorithm encoding layer of the AI-based semantic segmentation algorithm is a convolutional neural network and a down-sampling module for extracting shallow semantic information; the algorithm decoding layer of the AI-based semantic segmentation algorithm is a convolutional layer for mining deep semantics, a spatial attention module for improving the weight of the tin paste area through convolution and maximum pooling, and an up-sampling module for expanding the features in space to make the output feature map consistent with the size of the input image.

4. The AI-driven tin paste plate surface foreign matter detection method according to claim 1, wherein: the cropping and color adjustment of the tin paste detection area to obtain a plurality of local small images further comprises: extracting connected domains from the mask image of the tin paste detection area, calculating the circumscribed rectangle of each connected domain, and cropping the original image and the mask image at the corresponding position according to the circumscribed rectangle; based on the cropped mask image, retaining the image in the tin paste detection area and setting the image outside the tin paste detection area to black.

5. The AI-driven tin paste plate surface foreign matter detection method according to claim 1, wherein: the collaborative detection of foreign matters in each local small image using an AI-based target detection algorithm and a reference image-based detection algorithm further comprises: using the AI-based target detection algorithm to detect small-size foreign matters in each local small image; and using the reference image-based detection algorithm to detect large-size foreign matters in each local small image.

6. The AI-driven tin paste plate surface foreign matter detection method according to claim 5, wherein: the AI-based target detection algorithm comprises: obtaining shallow semantic information of the local small image through a shallow feature extraction module. ​ Deep semantic feature extraction module is used to obtain deep features of the local small image; The deep features are up-sampled and fused with the shallow semantic information; During the fusion process, the channel attention feature extraction module is used to enhance the small target feature, and a large spatial size feature map containing small size foreign object feature information is obtained.

7. The AI-driven Solder Paste Board Surface Foreign Object Detection Method of claim 5, wherein: The reference image-based detection algorithm comprises: Collecting a normal image without foreign objects corresponding to the position of the local small image as a reference image; The reference image and the local small image are input into a twin neural network to extract basic features, then a feature alignment module is used to complete feature alignment, and finally a deep feature extraction module is used to extract deep semantic features and output detection results.

8. The AI-driven Solder Paste Board Surface Foreign Object Detection Method of claim 1, wherein: The integration of foreign object detection results of a plurality of local small images outputs the final foreign object recognition result, further comprising: Merging the double-algorithm detection results of all local small images, and merging the recognition results at the overlapping positions; During the merging process, the overlapping ratio is determined according to the position of the foreign object, and the bounding rectangle of the target with a high overlapping ratio is taken as the final foreign object position and size result for output. 9.The AI-driven SMT paste board surface foreign matter detection system based on any one of claims 1 to 8, characterized in that, The system comprises: An AI-driven region segmentation processing module, configured to collect a raw image of a solder paste board surface to be detected, locate a solder paste board Mark point, and correct the solder paste board image; use an AI-based semantic segmentation algorithm to extract a solder paste detection region in the raw image to be detected; crop and tone the solder paste detection region to obtain a plurality of local small images; A double-algorithm cooperative foreign object detection processing module, configured to use an AI-based target detection algorithm and a reference image-based detection algorithm to cooperatively detect foreign objects for each local small image; and integrate the foreign object detection results of a plurality of local small images to output the final foreign object recognition result.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the AI-driven Solder Paste Board Surface Foreign Object Detection Method of any one of claims 1-8.

Images