A defect detection method, device, intelligent terminal and computer readable storage medium

Abstract

The present invention discloses a defect detection method, device, intelligent terminal and computer-readable storage medium, wherein the defect detection method comprises: obtaining a target neural network; obtaining a two-dimensional image to be trained and a three-dimensional image to be trained, and obtaining a model training set after training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network; obtaining a defect detection model to be trained, and training the defect detection model to be trained based on the model training set to obtain a target defect detection model; obtaining an image to be detected, and performing defect detection based on the target defect detection model and the image to be detected. Compared with the prior art, the scheme of the present invention combines a neural network and a defect detection model to automatically perform defect detection, which is conducive to improving the accuracy and efficiency of defect detection.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a defect detection method, device, intelligent terminal and computer-readable storage medium. Background Art

[0002] Efficient and reliable defect detection is an important part of the automation of the electronics manufacturing industry. At present, with the development of science and technology, the size of components used in the production of electronic products is getting smaller and smaller, and the assembly integration density is getting higher and higher, and the requirements for defect detection are also getting higher and higher.

[0003] In the prior art, inspectors usually perform visual inspection on objects that need to be inspected for defects, that is, inspectors perform surface defect inspection by visual inspection. The problem with the prior art is that manual visual inspection is affected by subjective factors and it is difficult to detect small defects, which is not conducive to improving the accuracy and efficiency of defect detection.

[0004] Therefore, the prior art needs to be improved. Summary of the invention

[0005] The main purpose of the present invention is to provide a defect detection method, device, intelligent terminal and computer-readable storage medium, aiming to solve the problem that the manual visual inspection method in the prior art is affected by subjective factors and it is difficult to detect small defects, which is not conducive to improving the accuracy and efficiency of defect detection.

[0006] In order to achieve the above object, the first aspect of the present invention provides a defect detection method, wherein the method comprises:

[0007] Obtain the target neural network;

[0008] Obtaining a two-dimensional image to be trained and a three-dimensional image to be trained, and training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network to obtain a model training set;

[0009] Obtain a defect detection model to be trained, and train the defect detection model to be trained based on the model training set to obtain a target defect detection model;

[0010] Acquire an image to be detected, and perform defect detection based on the target defect detection model and the image to be detected.

[0011] Optionally, the above-mentioned obtaining the target neural network includes:

[0012] Obtaining a neural network training set, wherein the neural network training set includes a plurality of groups of corresponding two-dimensional training images and three-dimensional training images;

[0013] A neural network to be trained is obtained, and the neural network to be trained is trained based on the neural network training set to obtain a target neural network.

[0014] Optionally, the neural network to be trained is a BP neural network.

[0015] Optionally, the acquiring of the two-dimensional image to be trained and the three-dimensional image to be trained, and training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network to obtain a model training set, includes:

[0016] Acquire multiple groups of corresponding two-dimensional images to be trained and three-dimensional images to be trained, wherein the two-dimensional images to be trained are obtained by photographing a charge coupled device camera, and the three-dimensional images to be trained are obtained based on grating projection phase measurement profilometry;

[0017] Based on the target neural network, each group of the two-dimensional images to be trained and the three-dimensional images to be trained are trained respectively to obtain a model training set.

[0018] Optionally, the defect detection model to be trained is a YOLOv3 model.

[0019] Optionally, the acquiring of the image to be detected and performing defect detection based on the target defect detection model and the image to be detected include:

[0020] Acquire an image of the object to be detected as the image to be detected;

[0021] The above-mentioned image to be detected is input into the above-mentioned target defect detection model, and defect detection is performed on the above-mentioned object to be detected.

[0022] Optionally, after acquiring the image to be detected and performing defect detection based on the target defect detection model and the image to be detected, the method further includes:

[0023] Output the defect detection results.

[0024] A second aspect of the present invention provides a defect detection device, wherein the device comprises:

[0025] A target neural network acquisition module, used to acquire a target neural network;

[0026] A model training set acquisition module is used to acquire a two-dimensional image to be trained and a three-dimensional image to be trained, and to obtain a model training set after training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network;

[0027] A target defect detection model acquisition module is used to acquire a defect detection model to be trained, and train the defect detection model to be trained based on the model training set to obtain a target defect detection model;

[0028] The defect detection module is used to obtain the image to be detected and perform defect detection based on the target defect detection model and the image to be detected.

[0029] A third aspect of the present invention provides an intelligent terminal, comprising a memory, a processor, and a defect detection program stored in the memory and executable on the processor, wherein the defect detection program implements any one of the steps of the defect detection method when executed by the processor.

[0030] A fourth aspect of the present invention provides a computer-readable storage medium, on which a defect detection program is stored, and when the defect detection program is executed by a processor, the step of any one of the above-mentioned defect detection methods is implemented.

[0031] As can be seen from the above, in the scheme of the present invention, a target neural network is obtained; a two-dimensional image to be trained and a three-dimensional image to be trained are obtained, and a model training set is obtained after the two-dimensional image to be trained and the three-dimensional image to be trained are trained based on the target neural network; a defect detection model to be trained is obtained, and the defect detection model to be trained is trained based on the model training set to obtain a target defect detection model; an image to be detected is obtained, and defect detection is performed based on the target defect detection model and the image to be detected. Compared with the method in the prior art in which the inspectors perform surface defect detection by visual inspection, the scheme of the present invention combines the neural network and the defect detection model to automatically perform defect detection, which is beneficial to improving the accuracy and efficiency of defect detection. At the same time, the two-dimensional image to be trained and the three-dimensional image to be trained are trained based on the target neural network to obtain a model training set, and the target defect detection model is obtained through training with the model training set, which is beneficial to improving the detection accuracy of the target defect detection model and further improving the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 It is a flowchart of a defect detection method provided by an embodiment of the present invention;

[0034] Figure 2 Embodiment of the present invention Figure 1 Specific flow diagram of step S100;

[0035] Figure 3 Embodiment of the present invention Figure 1Specific flow diagram of step S200;

[0036] Figure 4 Embodiment of the present invention Figure 1 Specific flow diagram of step S400;

[0037] Figure 5 is a schematic diagram of a two-dimensional image provided by an embodiment of the present invention;

[0038] Figure 6 is a schematic diagram of a three-dimensional image provided by an embodiment of the present invention;

[0039] Figure 7 Embodiment of the present invention Figure 6 A magnified image of a solder joint;

[0040] Figure 8 is a structural schematic diagram of a defect detection device provided by an embodiment of the present invention;

[0041] Fig. 9 It is a block diagram of the internal structure principle of an intelligent terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0043] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0044] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0045] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0046] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0048] Efficient and reliable defect detection is an important part of the automation of the electronics manufacturing industry. At present, with the development of science and technology, the size of components used in the production of electronic products is getting smaller and smaller, the assembly integration density is getting higher and higher, and the requirements for defect detection are also getting higher and higher. In the prior art, inspectors usually perform visual inspections on objects that need to be inspected for defects, that is, inspectors perform surface defect detection by visual inspection. The problem with the prior art is that the manual visual inspection method is affected by subjective factors and it is difficult to detect small defects, which is not conducive to improving the accuracy and efficiency of defect detection.

[0049] At present, the length and width of some patch components are only 0.1x0.2mm, and the pin spacing of fine devices is only 0.1mm. The main production links of the surface mount installation production line of printed circuit boards include board installation, solder paste printing, chip shooting, IC chip placement, reflow soldering fixation, perforated component assembly, wave soldering, packaging, etc. Among them, more than 80% of the quality defects of electronic products produced by patch production are caused by solder paste printing defects. However, the quality inspection technology of solder paste printing is difficult, and it is difficult to judge such subtle defects by manual visual inspection.

[0050] Therefore, defect detection can be performed based on microstructure 3D detection equipment and three-dimensional reconstruction technology. The methods for defect detection based on microstructure 3D detection equipment and three-dimensional reconstruction technology generally include three categories: fast laser triangulation, three-dimensional attribute analysis based on two-dimensional images collected under special light source illumination, and three-dimensional reconstruction based on the analysis of the matching position of binocular two-dimensional image feature points collected by multi-view charge-coupled device (CCD, Charge-coupled Device) cameras. However, the accurate model of the two-dimensional image method is not easy to establish; in three-dimensional detection, the matching calculation amount of feature points is large and the speed is slow, and the feature point matching error makes the stability poor, and the obtained three-dimensional accuracy is not high; in the present invention, the grating projection phase measurement profilometry (PMP) further developed based on the laser triangulation method is used for detection, and based on the two-dimensional image, a deep learning model trained by a neural network is used to improve the three-dimensional measurement performance of PMP to obtain a more accurate picture, and then the relevant defects are detected by the YOLO deep learning algorithm.

[0051] Specifically, in order to solve the problems existing in the prior art, in this embodiment, a defect detection method, device, intelligent terminal and computer-readable storage medium are proposed to obtain a target neural network; obtain a two-dimensional image to be trained and a three-dimensional image to be trained, and obtain a model training set after training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network; obtain a defect detection model to be trained, and train the defect detection model to be trained based on the model training set to obtain a target defect detection model; obtain an image to be detected, and perform defect detection based on the target defect detection model and the image to be detected. Compared with the method in the prior art in which the inspector performs surface defect detection by visual inspection, the scheme of the present invention combines the neural network and the defect detection model to automatically perform defect detection, which is conducive to improving the accuracy and efficiency of defect detection. At the same time, based on the target neural network, the two-dimensional image to be trained and the three-dimensional image to be trained are trained to obtain a model training set, and the target defect detection model is obtained through training with the model training set, which is conducive to improving the detection accuracy of the target defect detection model and further improving the accuracy of defect detection.

[0052] Exemplary Methods

[0053] like Figure 1 As shown, an embodiment of the present invention provides a defect detection method. Specifically, the method includes the following steps:

[0054] Step S100, obtaining a target neural network.

[0055] The target neural network is a trained neural network. In this embodiment, the corresponding paired two-dimensional image and three-dimensional image can be trained based on the target neural network to obtain a more accurate three-dimensional image, thereby improving the accuracy of defect detection.

[0056] Step S200, obtaining a two-dimensional image to be trained and a three-dimensional image to be trained, and training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network to obtain a model training set.

[0057] Among them, the above-mentioned two-dimensional images to be trained and three-dimensional images to be trained are images used to input the target neural network. After the above-mentioned two-dimensional images to be trained and three-dimensional images to be trained are input into the target neural network, the target neural network is trained, and the two-dimensional images to be trained and the three-dimensional images to be trained are optimized to obtain accurate three-dimensional images, and a model training set is obtained based on the obtained set of accurate three-dimensional images.

[0058] Step S300, obtaining a defect detection model to be trained, and training the defect detection model to be trained based on the model training set to obtain a target defect detection model.

[0059] Among them, the above-mentioned defect detection model to be trained is a model that needs to be further trained before it can be used for defect detection. A target detection model based on machine learning can be selected as the defect detection model to be trained, such as the YOLO model. Based on the above-mentioned accurate three-dimensional image, a model training set is obtained and the above-mentioned defect detection model to be trained is trained to improve the defect detection performance of the defect detection model to be trained and obtain a target defect detection model. The above-mentioned target defect detection model is the defect detection model to be trained after the training is completed, which can be directly used for defect detection.

[0060] Step S400: acquiring an image to be detected, and performing defect detection based on the target defect detection model and the image to be detected.

[0061] The above-mentioned image to be detected is a three-dimensional image of the object to be detected (such as electronic components), which can be obtained based on the PMP three-dimensional measurement method (the three-dimensional image is the modulation of the projection light source obtained by PMP, and the phase change of the modulation is calculated by collecting the grayscale of the image, and finally the three-dimensional height is calculated). The image to be detected is input into the target defect detection model, and the electronic components in the image to be detected are identified based on the target defect detection model, so as to realize defect detection.

[0062] As can be seen from the above, in the defect detection method provided by the embodiment of the present invention, a target neural network is obtained; a two-dimensional image to be trained and a three-dimensional image to be trained are obtained, and a model training set is obtained after the two-dimensional image to be trained and the three-dimensional image to be trained are trained based on the target neural network; a defect detection model to be trained is obtained, and the defect detection model to be trained is trained based on the model training set to obtain a target defect detection model; an image to be detected is obtained, and defect detection is performed based on the target defect detection model and the image to be detected. Compared with the method in the prior art in which the inspectors perform surface defect detection by visual inspection, the scheme of the present invention combines the neural network and the defect detection model to automatically perform defect detection, which is beneficial to improving the accuracy and efficiency of defect detection. At the same time, the two-dimensional image to be trained and the three-dimensional image to be trained are trained based on the target neural network to obtain a model training set, and the target defect detection model is obtained through training with the model training set, which is beneficial to improving the detection accuracy of the target defect detection model and further improving the accuracy of defect detection.

[0063] Specifically, in this embodiment, Figure 2 As shown, the above step S100 includes:

[0064] Step S101, obtaining a neural network training set, wherein the neural network training set includes a plurality of groups of corresponding two-dimensional training images and three-dimensional training images.

[0065] Step S102, obtaining a neural network to be trained, and training the neural network to be trained based on the neural network training set to obtain a target neural network.

[0066] The plurality of groups of corresponding two-dimensional training images and three-dimensional training images included in the neural network training set are a plurality of groups of two-dimensional images and three-dimensional images corresponding to structures or devices similar to the objects to be detected (such as microstructures or electronic components) obtained in advance. The two-dimensional training images can be obtained by photographing with a CCD camera, and the three-dimensional training images can be obtained by measuring with a PMP method. The neural network to be trained is trained based on each associated group of two-dimensional training images and three-dimensional training images.

[0067] In an application scenario, two-dimensional training images (obtained by a CCD camera) and three-dimensional training images (obtained based on the PMP method) are obtained based on the following method. The optoelectronic aiming device is installed on the working surface of the three-dimensional mobile stage (the laser is turned on). At the initial Z position (X is the horizontal direction, Y is the vertical direction, and Z is the height), the three-dimensional mobile stage is controlled to adjust along the X direction so that the output voltage of the silicon phototransistor in the optoelectronic aiming device is maximized to achieve the precise coincidence of the light plane with the center of the aiming hole on the optoelectronic aiming device. The spatial position of the aiming hole at this time is recorded as the starting position of the object coordinate (the origin of the object coordinate system). The laser is turned off, and the two-dimensional image of the aiming hole on the optoelectronic aiming device is obtained by the CCD camera. The sub-pixel image coordinates of the center of the aiming hole image corresponding to the spatial coordinates can be obtained by edge detection and ellipse fitting. Light up the laser again, control the 3D moving stage to move a certain distance in the vertical (Y) direction, and then control the 3D moving stage to adjust in the horizontal (X) direction again, so that the output voltage of the silicon phototransistor in the photoelectric aiming device is maximized, so as to complete the accurate coincidence of the light plane and the center of the aiming hole on the photoelectric aiming device again, and record the spatial object coordinates of the center of the aiming hole by the movement of the 3D moving stage. Among them, the Z position can be set to a specified position of the photographed object (such as the starting position of the solder joint of the circuit board). In the above process, by first setting an origin, then aligning the center of the hole to the center position of the structure to be detected, and moving the center of the hole to the origin again, the direction and distance of movement are recorded to measure the coordinates of the structure to be detected, so as to obtain the corresponding 3D image.

[0068] In an application scenario, the two-dimensional image obtained by the CCD camera can also be preprocessed to obtain the two-dimensional training image required in this embodiment (as well as other two-dimensional images required in the embodiment, such as the two-dimensional image to be trained). The purpose of preprocessing is to facilitate calculation and save time. The preprocessing process may include graying, filtering, sharpening, pin frame selection, etc. Specifically, the image obtained by CCD shooting is in color. When processing the color image, three channels need to be processed in sequence, and the time cost will be very large. Therefore, in order to improve the processing speed and meet the real-time requirements, it is necessary to reduce the amount of data to be processed. The color image can be grayed to reduce the amount of data to be processed. After the graying process, the image noise is reduced by filtering. Optionally, mean filtering, median filtering, etc. can be used. Further, the processed image is sharpened to highlight the edge contours of the object in the image, so as to facilitate the identification of the edge of the object. Specifically, the sharpening function library provided by OpenCV can be used to sharpen the edge of the image. Furthermore, the pins are selected, and the sharpened image is processed with the minimum circumscribed shape function of OpenCV, and the pin borders of the defects are retained according to the size of the input rectangular areas. Specifically, the minimum circumscribed shape of the entire component and the pins is obtained, and the pin borders of the defects are retained based on the size of the input rectangular area (or the preset rectangular area), where the input rectangular area is the standard area of ​​the rectangle corresponding to each input pin, and the difference between the obtained area size and the standard area size is used to determine whether the marked rectangle is a pin, whether its size is standard, and whether it is defective.

[0069] Among them, OpenCV is an open source function library for image processing, analysis, and machine vision. The library is written in C and C++ and can run on Windows, Linux, and Mac OSX systems. Use OpenCV functions to filter, binarize, and locate chips on the workpiece surface image to achieve pixel optimization of the workpiece surface image and improve the accuracy of subsequent workpiece surface quality inspection. At the same time, the defect positioning and marking functions are completed to facilitate subsequent picking and processing operations.

[0070] In this embodiment, the neural network to be trained is a BP neural network. The BP model is one of the important models of neural networks and is widely used in classification, prediction, fault diagnosis and parameter detection. Its network structure not only has input layer nodes and output layer nodes, but also can have one or more hidden layer nodes. For the input signal, it is first propagated forward to the hidden node, and after the action function, the output information of the hidden node is propagated to the output node, and finally the output node gives the output result through the output function. The learning process of this algorithm consists of forward propagation and reverse propagation. In the forward propagation process, the input signal is processed layer by layer from the input layer through the hidden unit layer and transmitted to the output layer. The state of each layer of neurons only affects the state of the next layer of neurons. If the output layer cannot obtain the expected output, it will turn to reverse propagation, return the error signal along the original connection path, and minimize the error signal by modifying the weights of each layer of neurons. In fact, the BP model turns the I / O problem of a group of samples into a nonlinear optimization problem. The most common gradient descent method in optimization is used, and it is solved by iterative operation. Corresponding to the learning and memory problem, adding hidden nodes increases the adjustable parameters of the optimization problem, so that a more accurate solution can be obtained.

[0071] Specifically, for images that need to be input into the BP neural network for training (or testing), the image is divided into sub-regions of the same size that are not connected to each other, and the local minimum points of each sub-region, that is, the central point of the receptive field, are scanned. The top closest to the central point is searched in the 380 neighborhood directions of the central point, and the pixel points corresponding to each top are the peripheral points. According to the above-mentioned sample data acquisition and processing method, 380 spatial sampling points are obtained (the specific number is related to the number of divided areas, and can be set and adjusted according to actual needs), and the sampling interval is 1mm. 280 of them are used as network training samples. 100 points are used as network test samples. The above-mentioned sample data acquisition method can ensure that the spatial positioning accuracy and image positioning accuracy of the sampling points are about 10μm and 0.1 pixels respectively, with high accuracy and without excessive subdivision. In network training and testing, S-shaped functions and hyperbolic functions can be used as conversion functions of neurons. In this embodiment, hyperbolic functions are preferably used to improve accuracy. Since the hyperbolic function is a saturated function, in the data preprocessing, the input training sample (that is, the image input into the BP neural network for training the BP neural network) is transformed to [-1, 1] and the output sample is transformed to [-0.95, 0.95] to reduce the amount of calculation.

[0072] In the BP neural network of this embodiment, there are 2 nodes in the input layer and 3 nodes in the output layer, and the image coordinates (x, y) of the spatial point are used as the input values ​​of the network input nodes. The above image coordinates are the coordinates determined after the rectangular coordinate system is established on the image acquired by PMP, and the object coordinates (x, y, z) of the corresponding spatial point are used as the output values ​​of the output nodes (the entire image is a rectangular coordinate system, and the detection point is an object coordinate). The network is trained and tested by adjusting the adjustable parameters in the network (the number of hidden layers, the number of hidden layer nodes, the learning rate, the momentum factor, and the characteristic parameters in the hyperbolic function), and the training accuracy and the test accuracy are calculated using the root mean square error.

[0073] Specifically, in this embodiment, the BP neural network is trained and optimized based on the following steps: the first step is to initialize the network and randomly set the weights of neurons in each layer. The second step is to give the network input objects and target outputs; wherein the input objects are two-dimensional training images and three-dimensional training images obtained by PMP, and the output target is a more accurate three-dimensional image. The third step is to calculate the outputs of each unit in the hidden layer and output layer of the network; that is, the network calculates the outputs of each unit in each layer by itself through the input objects and the weights of each layer, and each unit is each neuron. The fourth step is to calculate the deviation between the target value and the actual output; the target value refers to the output value of the network under the ideal state obtained by the user through his own calculation (or other methods) during the training process. The fifth step is to adjust the weights of neurons in each layer, specifically, adjust the weights of each neuron according to the deviation between the target value and the output of each unit; for example, if the calculation is too fast and the expected value cannot be obtained, the learning rate can be reduced to traverse more data. The sixth step is to return to the third step and restart learning until the deviation between the target value and the actual output meets the preset training accuracy requirements (for example, the error between the target value and the expected optimal value does not exceed 0.1). In this way, in each learning process, the optimal weights between neurons in each layer can be obtained by adjusting the number of input layer nodes, the number of hidden layers, the number of hidden layer nodes, the number of output layer nodes, the learning rate, the momentum factor, the output function, the action function and its gain coefficient and other parameters. The output layer can obtain the desired output as much as possible to minimize the deviation signal. Then, the different parts to be tested are determined by the coordinates of the sample points, and the three-dimensional image of the PMP is optimized by the two-dimensional image of the CCD. After the above BP neural network training is completed, it is tested by the test sample to detect the deviation between the test sample and the true value, so as to determine whether its accuracy meets the requirements.

[0074] Specifically, in this embodiment, Figure 3 As shown, the above step S200 includes:

[0075] Step S201, obtaining a plurality of corresponding groups of two-dimensional images to be trained and three-dimensional images to be trained, wherein the two-dimensional images to be trained are obtained by photographing a charge coupled device camera, and the three-dimensional images to be trained are obtained based on grating projection phase measurement profilometry.

[0076] Step S202: training each group of the two-dimensional images to be trained and the three-dimensional images to be trained based on the target neural network to obtain a model training set.

[0077] Among them, the above-mentioned two-dimensional images to be trained and three-dimensional images to be trained are one-to-one corresponding images that need to be trained through the target neural network. During the training process, the accuracy of the three-dimensional image can be improved based on the two-dimensional image to be trained to obtain an accurate three-dimensional image, thereby obtaining a model training set based on the accurate three-dimensional image and training the defect detection model to be trained.

[0078] Grating projection phase measurement profilometry (PMP), which is further developed from laser triangulation, has high accuracy and uses surface structured light projection to improve speed. In addition, multiple images of projected grating phase shifts are collected to calculate the modulated phase, converting the image-based distance measurement into image grayscale measurement. PMP is similar to laser triangulation in that both are active stereoscopic vision measurements, and both measure three-dimensional height based on the modulation of the projected light by the measured object. The difference is that grating projection uses a grating with sinusoidal brightness changes as the surface structure projection light source, so that the entire image can be measured at one time, which improves the measurement speed. In addition, it uses the modulation of the projected light source by the measured object, calculates the phase change of the modulation by collecting the grayscale of the image, and finally calculates the three-dimensional height. However, since there must be a certain angle between the projection angle and the CCD acquisition angle in the structured light projection measurement method, and the solder paste has height changes, there must be shadows in the grating projection. The grating projection image cannot be collected in the shadow, and the height information of the shadow part cannot be calculated. At the same time, it also causes a discontinuous area for phase unwrapping, causing the conventional phase unwrapping method to have errors in this area. And some electronic microstructures also need to detect two-dimensional defects. These two-dimensional information can be used to improve the PMP, so that the PMP can solve the problem of improving phase measurement accuracy, phase expansion and processing of shadow areas, so that the 3D detection of microstructures is accurate and fast. Therefore, in this embodiment, the characteristics of the three-dimensional shape, the grayscale information of the two-dimensional image and the three-dimensional height interpolation of adjacent points are combined to reconstruct the three-dimensional image. The BP neural network of deep learning is used to match and fuse the features extracted from the two-dimensional image with the three-dimensional information obtained by the grating projection phase measurement, so that the two-dimensional image features accurately reflect the three-dimensional changes in the shadow, thereby improving the accuracy of the three-dimensional image. In this way, the BP neural network is trained based on the two-dimensional image features to improve the three-dimensional measurement performance of the PMP, so as to help improve the speed of the PMP phase expansion algorithm and solve the measurement problem of the projected shadow area. The accuracy and speed of microstructure 3D detection can be improved, and the robustness can be enhanced, and it is less affected by noise and other interference.

[0079] It should be noted that the images to be collected and prepared as the two-dimensional images to be trained and the three-dimensional images to be trained can also be preprocessed first. The preprocessing process includes graying, filtering, sharpening, pin selection, etc. The specific preprocessing process can refer to the preprocessing process when obtaining the two-dimensional training image above, which will not be repeated here.

[0080] In this embodiment, the defect detection model to be trained is a YOLOv3 model. Specifically, YOLOv3 is selected as the model in this embodiment, and the model is optimized by modifying the code under its darknet framework. The above step S202 specifically includes: obtaining an accurate three-dimensional image trained by the BP neural network, calibrating the target in the image using the yolo-mark calibration tool, using the prepared data set as the training set of the YOLOv3 model, and training the YOLOv3 model. After the training is completed, the model can also be tested and improved to finally obtain the target defect detection model. Specifically, since the training process of the neural network model is unpredictable, the model test results may not meet the use conditions after the training is completed. It is necessary to adjust the data set to make it as close as possible to the final image to be detected, thereby improving the recognition accuracy of the model.

[0081] Specifically, in this embodiment, Figure 4 As shown, the above step S400 includes:

[0082] Step S401, obtaining an image of the object to be detected as the image to be detected.

[0083] Step S402: input the above-mentioned image to be detected into the above-mentioned target defect detection model, and perform defect detection on the above-mentioned object to be detected.

[0084] Among them, the above-mentioned object to be detected is an object that needs to be defect-detected, such as a circuit board, electronic components or other microstructure workpieces. The above-mentioned image to be detected is a three-dimensional image of the object to be detected obtained by PMP. The image to be detected is input into the trained target defect detection model, and the relevant defects (such as solder joints, etc.) are detected by the YOLO deep learning algorithm. In this way, the two-dimensional and three-dimensional image acquisition of the microstructure on the workpiece can be achieved, and the connection between the two-dimensional and three-dimensional images can be trained by the BP neural network, so as to improve the three-dimensional measurement performance of PMP based on the two-dimensional image and train the deep learning model through the neural network to obtain a more accurate image, and then detect the relevant defects through the YOLO deep learning algorithm to achieve rapid recognition and detection of the three-dimensional structure. Combining the three-dimensional reconstruction technology with the deep learning algorithm only requires relevant training of the data set, and there is no need to solve the relevant function, which improves the accuracy and speed of detection and reduces the amount of calculation.

[0085] Furthermore, in this embodiment, after defect detection, the defect detection result is output so that the inspector can be informed of the defect situation in time. Optionally, the defect detection result can be output by voice broadcast, text output, etc., and a defect report can also be output, which is not specifically limited here.

[0086] In this embodiment, industrial cameras, industrial computers, etc. are used to replace manpower for rapid operation, which can improve the speed and efficiency of detection, and reduce the detection cost in batch processing. At the same time, through machine vision and deep learning technology, high-accuracy recognition of detection targets can be achieved. At the same time, the above method can also be applied to other industrial fields. When applied to other industrial fields, it is only necessary to rebuild the model structure, make appropriate data sets to retrain the model, and thus re-acquire a new defect detection model. The application scalability of this method is high.

[0087] In this embodiment, the performance of the above defect detection method is verified through specific experiments. Figure 5 is a schematic diagram of a two-dimensional image provided by an embodiment of the present invention. Specifically, Figure 5 The two-dimensional image shown in is a two-dimensional image obtained after the chip has been pre-processed by graying, filtering, sharpening, pin selection, etc. Figure 6 is a schematic diagram of a three-dimensional image provided by an embodiment of the present invention. Figure 7 yes Figure 6 Specifically, Figure 6 The three-dimensional image in is the image to be detected. Figure 6 The height of the solder joints in the test were measured to compare the methods. The measurement data of the three printed solder pastes by the two methods are shown in the following table:

[0088] Sample No. 1 2 3 Laser triangulation mean 152.0 157.5 152.5 Standard deviation of laser triangulation 362.4 479.5 514.8 The mean of the method in this embodiment 155.0 162.5 157.5 The standard deviation of the method in this example 122.6 142.7 98.3

[0089] As can be seen from the above table, the standard deviation of the data obtained by the laser triangulation method is greater than the standard deviation of the data obtained by the method of this embodiment. The average value of the height measured by the laser triangulation method is less than the average value measured by the method of this embodiment. The method of this embodiment improves the accuracy of measurement.

[0090] Exemplary Devices

[0091] like Figure 8 As shown in , corresponding to the above defect detection method, an embodiment of the present invention further provides a defect detection device, and the above defect detection device includes:

[0092] The target neural network acquisition module 510 is used to acquire the target neural network.

[0093] The target neural network is a trained neural network. In this embodiment, the corresponding paired two-dimensional image and three-dimensional image can be trained based on the target neural network to obtain a more accurate three-dimensional image, thereby improving the accuracy of defect detection.

[0094] The model training set acquisition module 520 is used to acquire the two-dimensional images to be trained and the three-dimensional images to be trained, and obtain the model training set after training the two-dimensional images to be trained and the three-dimensional images to be trained based on the target neural network.

[0095] Among them, the above-mentioned two-dimensional images to be trained and three-dimensional images to be trained are images used to input the target neural network. After the above-mentioned two-dimensional images to be trained and three-dimensional images to be trained are input into the target neural network, the target neural network is trained, and the two-dimensional images to be trained and the three-dimensional images to be trained are optimized to obtain accurate three-dimensional images, and a model training set is obtained based on the obtained set of accurate three-dimensional images.

[0096] The target defect detection model acquisition module 530 is used to acquire the defect detection model to be trained, and train the defect detection model to be trained based on the model training set to obtain the target defect detection model.

[0097] Among them, the above-mentioned defect detection model to be trained is a model that needs to be further trained before it can be used for defect detection. A target detection model based on machine learning can be selected as the defect detection model to be trained, such as the YOLO model. Based on the above-mentioned accurate three-dimensional image, a model training set is obtained and the above-mentioned defect detection model to be trained is trained to improve the defect detection performance of the defect detection model to be trained and obtain a target defect detection model. The above-mentioned target defect detection model is the defect detection model to be trained after the training is completed, which can be directly used for defect detection.

[0098] The defect detection module 540 is used to obtain the image to be detected and perform defect detection based on the target defect detection model and the image to be detected.

[0099] The above-mentioned image to be detected is a three-dimensional image of the object to be detected (such as electronic components), which can be obtained based on the PMP three-dimensional measurement method (the three-dimensional image is the modulation of the projection light source obtained by PMP, and the phase change of the modulation is calculated by collecting the grayscale of the image, and finally the three-dimensional height is calculated). The image to be detected is input into the target defect detection model, and the electronic components in the image to be detected are identified based on the target defect detection model, so as to realize defect detection.

[0100] As can be seen from the above, in the defect detection device provided by the embodiment of the present invention, the target neural network is obtained by the target neural network acquisition module 510; the two-dimensional image to be trained and the three-dimensional image to be trained are obtained by the model training set acquisition module 520, and the model training set is obtained after the two-dimensional image to be trained and the three-dimensional image to be trained are trained based on the target neural network; the defect detection model to be trained is obtained by the target defect detection model acquisition module 530, and the defect detection model to be trained is trained based on the model training set to obtain the target defect detection model; the image to be detected is obtained by the defect detection module 540, and defect detection is performed based on the target defect detection model and the image to be detected. Compared with the method in the prior art in which the inspector performs surface defect detection by visual inspection, the scheme of the present invention combines the neural network and the defect detection model to automatically perform defect detection, which is conducive to improving the accuracy and efficiency of defect detection. At the same time, the model training set is obtained by training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network, and the target defect detection model is obtained by training the model training set, which is conducive to improving the detection accuracy of the target defect detection model and further improving the accuracy of defect detection.

[0101] Specifically, in this embodiment, the specific functions of the above-mentioned defect detection device and its various modules can also refer to the corresponding description in the above-mentioned defect detection method, which will not be repeated here.

[0102] Based on the above embodiments, the present invention further provides an intelligent terminal, whose principle block diagram can be shown as follows: Fig. 9 As shown. The above-mentioned intelligent terminal includes a processor, a memory, a network interface and a display screen connected through a system bus. Among them, the processor of the intelligent terminal is used to provide computing and control capabilities. The memory of the intelligent terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a defect detection program. The internal memory provides an environment for the operation of the operating system and the defect detection program in the non-volatile storage medium. The network interface of the intelligent terminal is used to communicate with an external terminal through a network connection. When the defect detection program is executed by the processor, the steps of any one of the above-mentioned defect detection methods are implemented. The display screen of the intelligent terminal can be a liquid crystal display screen or an electronic ink display screen.

[0103] Those skilled in the art will understand that Fig. 9 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the smart terminal to which the scheme of the present invention is applied. The specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0104] In one embodiment, a smart terminal is provided. The smart terminal includes a memory, a processor, and a defect detection program stored in the memory and executable on the processor. When the defect detection program is executed by the processor, the following operation instructions are performed:

[0105] Obtain the target neural network;

[0106] Obtaining a two-dimensional image to be trained and a three-dimensional image to be trained, and training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network to obtain a model training set;

[0107] Obtain a defect detection model to be trained, and train the defect detection model to be trained based on the model training set to obtain a target defect detection model;

[0108] Acquire an image to be detected, and perform defect detection based on the target defect detection model and the image to be detected.

[0109] An embodiment of the present invention further provides a computer-readable storage medium, on which a defect detection program is stored. When the defect detection program is executed by a processor, the steps of any one of the defect detection methods provided by the embodiment of the present invention are implemented.

[0110] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0111] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0112] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0114] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0115] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0116] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A defect detection method, characterized in that: The method comprises: Obtain the target neural network; Acquire a two-dimensional image to be trained and a three-dimensional image to be trained, and obtain a model training set after training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network, wherein the two-dimensional image to be trained is obtained by photographing a charge coupled device camera, and the three-dimensional image to be trained is obtained based on grating projection phase measurement profilometry, and the target neural network is used to match and fuse the features extracted from the two-dimensional image to be trained with the three-dimensional information obtained by grating projection phase measurement in the three-dimensional image to be trained, and optimize the three-dimensional image to be trained obtained based on grating projection phase measurement profilometry through the two-dimensional image to be trained, so as to improve the accuracy of the three-dimensional image to be trained; Acquire a defect detection model to be trained, and train the defect detection model to be trained based on the model training set to obtain a target defect detection model; An image to be detected is acquired, and defect detection is performed based on the target defect detection model and the image to be detected, wherein the image to be detected is a three-dimensional image of the object to be detected acquired by grating projection phase measurement profilometry.

2. The defect detection method according to claim 1, characterized in that: The step of obtaining a target neural network comprises: Acquire a neural network training set, wherein the neural network training set includes a plurality of groups of corresponding two-dimensional training images and three-dimensional training images; A neural network to be trained is obtained, and the neural network to be trained is trained based on the neural network training set to obtain a target neural network.

3. The defect detection method according to claim 2, characterized in that: The neural network to be trained is a BP neural network.

4. The defect detection method according to claim 1, characterized in that: The step of acquiring a two-dimensional image to be trained and a three-dimensional image to be trained, and training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network to obtain a model training set includes: Acquire multiple groups of corresponding two-dimensional images to be trained and three-dimensional images to be trained; Based on the target neural network, each group of the two-dimensional images to be trained and the three-dimensional images to be trained are trained respectively to obtain a model training set.

5. The defect detection method according to claim 1, characterized in that: The defect detection model to be trained is a YOLOv3 model.

6. The defect detection method according to claim 1, characterized in that: The acquiring the image to be detected and performing defect detection based on the target defect detection model and the image to be detected includes: Acquire an image of the object to be detected as the image to be detected; The image to be detected is input into the target defect detection model to perform defect detection on the object to be detected.

7. The defect detection method according to claim 1, characterized in that: After acquiring the image to be detected and performing defect detection based on the target defect detection model and the image to be detected, the method further includes: Output the defect detection results.

8. A defect detection device, characterized in that: The device comprises: A target neural network acquisition module, used to acquire a target neural network; A model training set acquisition module, used for acquiring a two-dimensional image to be trained and a three-dimensional image to be trained, and obtaining a model training set after training the two-dimensional image to be trained and the three-dimensional image to be trained based on the target neural network, wherein the two-dimensional image to be trained is obtained by photographing a charge coupled device camera, and the three-dimensional image to be trained is obtained based on grating projection phase measurement profilometry, and the target neural network is used for matching and fusing features extracted from the two-dimensional image to be trained with three-dimensional information obtained by grating projection phase measurement in the three-dimensional image to be trained, and optimizing the three-dimensional image to be trained obtained based on grating projection phase measurement profilometry through the two-dimensional image to be trained, so as to improve the accuracy of the three-dimensional image to be trained; A target defect detection model acquisition module is used to acquire a defect detection model to be trained, and train the defect detection model to be trained based on the model training set to obtain a target defect detection model; The defect detection module is used to acquire an image to be detected and perform defect detection based on the target defect detection model and the image to be detected, wherein the image to be detected is a three-dimensional image of the object to be detected obtained by grating projection phase measurement profilometry.

9. An intelligent terminal, characterized in that: The intelligent terminal includes a memory, a processor, and a defect detection program stored in the memory and executable on the processor. When the defect detection program is executed by the processor, the steps of the defect detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a defect detection program, and when the defect detection program is executed by the processor, the steps of the defect detection method according to any one of claims 1 to 7 are implemented.

Images