Surface Mounting Board Defect Detection Method and Storage Medium Based on Linear Sample Compression

Abstract

A surface mount board defect detection method and storage medium based on linear sample compression according to the present invention include the following steps. Step 1: Randomly take out a set of SMT defect sample images from the storage module. Step 2: Use both the input SMT defect sample images and the SMT defect sample images randomly taken out from the storage module to train a classifier. Step 3: Use linear sample compression technology to compress the input small batch of SMT defect sample images at a frequency K. Step 4: Update the storage module with the compressed SMT defect sample images, and the finally obtained classifier is used to complete the SMT defect detection task. The surface mount board defect detection method based on linear sample compression proposed by the present invention compresses different samples together through linear combination instead of deleting them, thereby effectively reducing the computational cost while improving the accuracy of the algorithm.

Description

Technical Field

[0001] The present invention relates to the technical field of defect detection, and particularly to a surface mount board defect detection method and storage medium based on linear sample compression. Background Art

[0002] Surface mount technology is an electronic manufacturing technology that directly mounts electronic components on the surface of printed circuit boards or other substrates. Although surface mount technology is widely used, there are currently some defects, including: during the production and manufacturing process of SMT, product defects and flaws may occur due to equipment aging, etc. The defect detection of SMT is a time-consuming and laborious process, increasing production costs. With the proposal and development of deep learning and intelligent manufacturing, SMT defect detection has also become intelligent. Due to the great progress made by deep neural networks under fully supervised conditions, the defect detection of SMT has also developed. Deep learning models have achieved state-of-the-art results in fields such as computer vision and natural language processing. However, all these results assume the existence of a static training dataset that represents the entire data distribution. But in the learning environment of SMT defect detection, it is non-stationary because data arrives in the form of a data stream and the underlying data distribution may change over time. In such an environment, incremental learning over time, i.e., online continuous learning (OCL), is necessary. Unfortunately, when the model is trained on new data, most models are unable to retain knowledge about past data, which is a problem called catastrophic forgetting, bringing great difficulties to the detection task. The present invention focuses on studying the ability to continuously learn new types of defects online from a small number of newly added SMT defect samples. Summary of the Invention

[0003] A surface mount board defect detection method, device and storage medium based on linear sample compression proposed by the present invention can solve at least one of the technical problems in the background art.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A surface mount board defect detection method based on linear sample compression includes the following steps

[0006] Step 1: Randomly take out a set of SMT defect sample images from the storage module;

[0007] Step 2: Use both the input SMT defect sample images and the SMT defect sample images randomly taken out from the storage module to train the classifier;

[0008] Step 3: Use linear sample compression technology to compress the input small batch of SMT defect sample images at a frequency of K;

[0009] Step 4: Update the storage module with the compressed SMT defect sample images, and use the finally obtained classifier to complete the SMT defect detection task.

[0010] Further, the specific steps of the said Step 3 include

[0011] Denote as the SMT defect sample data set, where and are respectively the input data and label of the i-th entry in the SMT defect sample data set;

[0012] The goal of compression is to generate a set of synthetic samples and their corresponding labels, , where | | | |, such that the synthetic SMT defect samples can be used to train a neural network to achieve the same performance as training the same network on the original SMT defect sample data set ; this goal is expressed as:

[0013] (1)

[0014] where is the data distribution, is the task-specific loss, is the classification model, and are respectively the network parameters obtained by training the network on the SMT defect sample data set and ;

[0015] Adopt learning the synthetic SMT defect sample images such that the model trained on the set minimizes the training loss on the original data set, i.e.:

[0016] (2)

[0017] Further, the synthetic SMT defect sample images are generated by alternately performing internal and external meta-optimization steps. The goal of the internal step is to find the optimal weights induced by the synthetic SMT defect sample images ; then, during the outer loop, use the weights to quantify the quality of the synthetic SMT defect sample images generated so far;

[0018] In the outer optimization step, optimize the synthetic set to the weights , so that the weights obtained from in the next inner loop iteration improve the performance on the original dataset; learn synthetic SMT defect sample images by minimizing the distance between and in the parameter space;

[0019] Optimize and direct the optimization to , which is achieved by solving the following optimization problem: (3)

[0020] where represents the Euclidean distance between two variables, represents the gradient of the variable, and is the flat vector corresponding to the gradient of the i-th output node.

[0021] Furthermore, the step S3 includes the following steps.

[0022] After randomly initializing the coefficients , generate a set of synthetic SMT defect sample images by linearly combining the input SMT defect sample images; filter the coefficients through the mask to define which input images the synthetic image consists of; then, take a small batch of synthetic SMT defect sample images and input SMT defect sample images, and use the network to calculate the loss for the input SMT defect sample ( ) of , the synthetic SMT defect sample and its gradient;

[0023] Use and to calculate the gradient matching loss, which is used to update the coefficients at the end of each outer loop; after generating a new set of compressed SMT defect sample images, update the parameters by minimizing the loss with the learning rate ;

[0024] Specifically, it includes the following sub-steps S31 to S36:

[0025] S31: Initialize the network parameters and the synthetic SMT defect sample images

[0026] , (4)

[0027] Among them is the coefficient matrix, is the mask, is the SMT defect sample image set to be compressed;

[0028] S32: Execute the loop steps from S33 to S36 for t less than or equal to the number of times of the outer loop T:

[0029] S33: Execute the loop steps from S34 to S35 for c less than or equal to the number of classes C:

[0030] S34: Take out a sample and from and respectively;

[0031] S35: Update , and :

[0032] (5a)

[0033] (5b)

[0034] (5c)

[0035] S36: Update , and the parameter :

[0036] (6)

[0037] On the other hand, the present invention also discloses a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the above method.

[0038] On yet another aspect, the present invention also discloses a computer device including a memory and a processor, the memory storing a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the above method.

[0039] As can be seen from the above technical solutions, the surface mount board defect detection method and system based on linear sample compression of the present invention relate to a learning solution for online continuous learning of linear sample compression for SMT defect detection. The present invention first proposes a new replay-based strategy OLCGM. When updating the memory each time, some examples are merged into a new synthetic sample instead of deleting the samples. This method can alleviate the forgetting caused by deleting old samples. Most of the strategies proposed in the literature are based on replay. Most of these strategies rely on fixed and simple random strategies to add or delete examples. However, although these simple strategies may succeed in the simple data streams used in the literature, they do not effectively use the memory buffer. Instead, the present invention proposes an alternative solution that compresses different samples together through linear combination instead of deleting them, thereby effectively reducing the computational cost while improving the accuracy of the algorithm.

[0040] The advantages of the present invention are that it can not only successfully detect surface mount board defects, but also demonstrate its ability to learn new defects in actual detection tasks, which is an ability not possessed by existing online learning methods. In addition, due to the above technical solutions, while improving the accuracy of the algorithm, the linear sample compression technology is used to effectively reduce the computational cost, greatly reducing the time required to complete the surface mount board defect detection task and saving time costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of the present invention;

[0042] Figure 2 It is an example of expert annotation results;

[0043] Figure 3 It is the detection effect diagram of the model trained by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention.

[0045] The embodiments of the present invention propose a new online continuous learning solution for linear sample compression from SMT new defect classes, and its structure is shown in Figure 1. In this solution, a new operation is used to update the replay buffer: sample compression. At each memory update, instead of deleting samples, some examples are merged into a new synthetic sample. Since the synthetic sample compresses the information of multiple samples into one sample, this solution can, to a certain extent, solve the forgetting problem caused by deleting old samples.

[0046] The following is a specific description:

[0047] An online continuous learning method that uses linear sample compression from newly added SMT defect classes performs the following operation steps on each image in the data stream:

[0048] Step 1: Randomly take out a set of SMT defect sample images from the storage module.

[0049] Step 2: Use both the input SMT defect sample images and the SMT defect sample images randomly taken out from the storage module to train the classifier.

[0050] Step 3: Use the linear sample compression technique to compress the input mini-batch of SMT defect sample images with frequency K (K is taken as 10).

[0051] Step 4: Use the compressed SMT defect sample images to update the storage module, and the finally obtained classifier is used to complete the SMT defect detection task.

[0052] Implementation steps of Step 2:

[0053] A multi-layer perceptron with a hidden layer of 400 neurons and RELU as the activation function is used. In all trainings, Resnet-18 and MLP classifiers are optimized by using SGD with a learning rate equal to 0.1.

[0054] The following details the core idea and specific implementation steps of Step 3:

[0055] Denote as the SMT defect sample data set, where and are the input data and label of the i-th entry in the SMT defect sample data set respectively. The goal of compression is to generate a set of synthetic samples and their corresponding labels, , where | | | |, such that the synthetic SMT defect samples can be used to train the neural network to achieve the same performance as training the same network on the original SMT defect sample data set . This goal can be expressed as:

[0056]

[0057] Among them is the data distribution, is the task-specific loss, is the classification model, and are the network parameters obtained by training the network on the SMT defect sample dataset and respectively. In the above equation, the way compression works is to optimize the synthetic SMT defect sample images The optimization set and thus the obtained will make the right side of the equation close to the left side. Here, we adopt learning the synthetic SMT defect sample images such that the model trained on the set minimizes the training loss on the original dataset, i.e.:

[0058]

[0059] The best synthetic SMT defect sample images are generated by alternately performing internal and external meta-optimization steps. The goal of the internal step is to find the optimal weights induced by the synthetic SMT defect sample images . Then, during the outer loop, the weights are used to quantify the quality of the synthetic SMT defect sample images generated so far. In the outer optimization step, the synthetic set is optimized to the weights so that the weights obtained from in the next inner loop iteration improve the performance on the original dataset. Although this method is very good for generating high-quality synthetic sets, the operation of unfolding the computational graph requires a large amount of computational resources. To alleviate this limitation, we adopt minimizing the and distance in the parameter space to learn the synthetic SMT defect sample images. Thus, in addition to achieving similar performance to , their method will also optimize the optimization of towards . This goal can be achieved by solving the following optimization problem:

[0060]

[0061] Among them represents the Euclidean distance between two variables, represents the gradient of the variable, and is a flat vector of the gradient corresponding to the i-th output node. Its advantage is that during each iteration t, optimizing does not require unfolding the computational graph. This means that the computational cost is reduced and the memory requirement is smaller.

[0062] The following introduces the specific implementation plan: After randomly initializing the coefficients and generating a synthetic SMT defect sample image set through linear combination by inputting the SMT defect sample images The coefficients are filtered through the mask whose purpose is to define which input images the synthetic image consists of. Then, take a mini-batch of synthetic SMT defect sample images and input SMT defect sample images, and use the network to calculate the loss for the input SMT defect sample ( ) of , the synthetic SMT defect sample ( ) and its gradient. Calculate the gradient matching loss using and (where c represents the c-th type of defect), and this loss is used to update the coefficients at the end of each outer loop. After generating a new set of compressed SMT defect sample images, update the parameters by minimizing the loss with a learning rate in total six steps. Specifically, it includes the following sub-steps S31 to S36:

[0063] Specifically, it includes the following sub-steps S31 to S36:

[0064] S31: Initialize the network parameters and the synthetic SMT defect sample images

[0065] ,

[0066] where is the coefficient matrix, is the mask, is the set of SMT defect sample images to be compressed.

[0067] S32: Execute the loop steps of S33 to S36 for t less than or equal to T (the number of outer loops):

[0068] S33: Execute the loop steps of S34 to S35 for c less than or equal to C (the number of classes):

[0069] S34: Take out a sample and from and 。

[0070] S35: Update 、 and :

[0071]

[0072]

[0073]

[0074] S36: Update , and parameter :

[0075]

[0076] Furthermore, for the above-mentioned step S4: Update the storage module : Replace the image removed from with the compressed SMT defect sample image.

[0077] In summary, the advantages of an online continuous learning method for linearly compressing samples from newly added SMT defect classes in the embodiments of the present invention are as follows: First, a new replay-based strategy OLCGM is proposed. When updating the memory each time, some examples are merged into a new synthetic sample instead of deleting the samples. This method can alleviate the forgetting caused by deleting old samples. Most of the strategies proposed in the literature are based on replay. Most of these strategies rely on fixed and simple random strategies to add or delete examples. However, although these simple strategies may succeed in the simple data streams used in the literature, they do not effectively use the memory buffer. Instead, we propose an alternative solution that compresses different samples together through linear combination instead of deleting them, thereby effectively reducing the computational cost while improving the accuracy of the algorithm.

[0078] Based on the original defects (wrong components, displacement, and white inversion), a new type of tombstone defect is added, and the effectiveness of this method is tested. An example of the expert annotation result is shown in Figure 2 , and the detection effect using the trained model is shown in Figure 3 , Figure 2 Please ask experts to specifically annotate the newly added defect class of tombstone defects Figure 3The medium model not only successfully detected a new type of defect, namely tombstone defects, but also retained the ability to detect the original types of defects (wrong parts, displacement, and white inversion). This demonstrates that the present invention has the advantage of being able to learn to detect new SMT defects while retaining the ability to detect the original types of defects, proving the effectiveness of this method in the field of detecting new SMT defects in intelligent manufacturing.

[0079] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to execute the steps of the above method.

[0080] In yet another aspect, the present invention also discloses a computer device including a memory and a processor, where the memory stores a computer program, which, when executed by the processor, causes the processor to execute the steps of the above method.

[0081] In yet another embodiment provided by the present application, there is also provided a computer program product containing instructions, which, when running on a computer, causes the computer to execute any one of the above-mentioned surface mount board defect detection methods based on linear sample compression.

[0082] It can be understood that the system provided by the embodiments of the present invention corresponds to the method provided by the embodiments of the present invention. For the explanations, examples, and beneficial effects of related content, reference can be made to the corresponding parts in the above method.

[0083] The embodiments of the present application also provide an electronic device including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus.

[0084] The memory is used to store a computer program.

[0085] The processor is used to implement the above-mentioned surface mount board defect detection method based on linear sample compression when executing the program stored in the memory.

[0086] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0087] The communication interface is used for communication between the above electronic device and other devices.

[0088] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0089] The aforementioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0090] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)).

[0091] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0092] Each embodiment in this specification is described in a related manner. For the same and similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the method embodiment.

[0093] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A surface mount board defect detection method based on linear sample compression, characterized in that, it includes the following steps, Step 1: Randomly take out a set of SMT defect sample images from the storage module; Step 2: Use both the input SMT defect sample images and the SMT defect sample images randomly taken out from the storage module to train a classifier; Step 3: Use linear sample compression technology to compress the input small batch of SMT defect sample images at frequency K; Step 4: Update the storage module with the compressed SMT defect sample images, and the finally obtained classifier is used to complete the SMT defect detection task; The specific steps of the said Step 3 include, Record as the SMT defect sample data set, where and are the input data and label of the i-th entry in the SMT defect sample data set, respectively; The goal of compression is to generate a set of synthetic samples and their corresponding labels, , where | | | |, such that the synthetic SMT defect samples can be used to train a neural network to achieve the same performance as training the same network on the original SMT defect sample dataset ; this goal is expressed as: Among them is the data distribution, is the task-specific loss, is the classification model, and are the network parameters obtained by training the network on the SMT defect sample datasets and respectively; Adopt learning to synthesize SMT defect sample images , such that the model trained on the set minimizes the training loss on the original data set, that is: ； Synthesize SMT defect sample images Generated by alternating internal and external meta-optimization steps, where the goal of the internal step is to find the optimal weights caused by the synthesized SMT defect sample images ; ; Then, during the outer loop, use the weight to quantify the quality of the synthetic SMT defect sample images generated so far; In the outer optimization step, the synthesis set is optimized to weights so that the weights obtained from in the next inner loop iteration improve the performance on the original dataset; the distance between the minimization and in the parameter space is used to learn the synthesized SMT defect sample images; To optimize the optimization is directed to , which is achieved by solving the following optimization problems: wherein represents the Euclidean distance between two variables, represents the gradient of the variable, and is a flat vector of the gradient corresponding to the i-th output node.

2. The surface mount board defect detection method based on linear sample compression according to claim 1, characterized in that: The said Step 3 includes the following steps, After randomly initializing the coefficients and inputting the SMT defect sample images to generate a synthetic SMT defect sample image set through linear combination ; the coefficients are filtered through a mask whose purpose is to define which input images the synthetic image consists of; then, a mini - batch of synthetic SMT defect sample images and input SMT defect sample images are taken, and the network is used to calculate the loss for the input SMT defect sample , the synthetic SMT defect sample and their gradients; Utilize and to calculate the gradient matching loss, which is used to update the coefficient at the end of each outer loop; after generating a new set of compressed SMT defect sample images, update the parameter by minimizing the loss with a learning rate ; specifically including the following sub-steps S31 to S36: S31: Initialize network parameters and synthesize SMT defect sample images ， Among them is the coefficient matrix, is the mask, is the SMT defect sample image set to be compressed; S32: Execute the loop steps of S33 to S36 for t less than or equal to the number of external loops T: S33: Execute the loop steps of S34 to S35 for c less than or equal to the number of classes C: S34: Take out a sample respectively from and ; and ; S35: Update , and : S36: Update , and parameters : 。 3. A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to execute the steps of the method according to claim 1 or 2.

4. A computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor is caused to execute the steps of the method according to claim 1 or 2.

Images