Intelligent detection and classification system for PCBA manufacturing defects

Abstract

The present disclosure is about an intelligent detection and classification system for PCBA manufacturing defects. The system includes: a first acquisition module for acquiring a first image of a PCBA to be detected; a recognition module for inputting the first image into a pre-trained defect recognition model to obtain a defect recognition result; the defect recognition result includes whether there is a defect and the defect type; a second acquisition module for acquiring a PCBA defect image data set matching the defect type when the defect recognition result is that there is a defect; a first calculation module for calculating the comprehensive similarity between the first image and each of a plurality of sub-data sets; a first selection module for selecting a target sub-data set and a target defect cause corresponding to the target sub-data set from a plurality of sub-data sets according to the comprehensive similarity, and obtaining the defect cause of the PCBA to be detected. This solution can quickly identify the defect cause of the PCBA to be detected.

Description

Technical Field

[0001] The present disclosure relates to the technical field of defect detection, and in particular to an intelligent detection and classification system for PCBA manufacturing defects. Background Art

[0002] In the related art, PCBA (Printed Circuit Board Assembly) is obtained by mounting components on a blank PCB board through a surface mount device (SMT). In order to ensure the production quality of PCBA, it is usually necessary to perform quality inspection on PCBA to identify whether there are manufacturing defects in PCBA. However, after identifying that there are defects in PCBA, in order to solve the root cause of the manufacturing defects and avoid the same manufacturing defects from occurring again, it is usually necessary to manually analyze and debug the causes of the defects based on experience. This method results in low production efficiency of PCBA. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides an intelligent detection and classification system for PCBA manufacturing defects.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided a system for intelligent detection and classification of PCBA manufacturing defects, comprising:

[0005] A first acquisition module, used to acquire a first image of the PCBA to be inspected;

[0006] A recognition module, used for inputting the first image into a pre-trained defect recognition model to obtain a defect recognition result; the defect recognition result includes whether a defect exists and the defect type;

[0007] A second acquisition module is used to acquire a PCBA defect image data set matching the defect type when the defect recognition result is that a defect exists; the PCBA defect image data set includes multiple sub-data sets, the defect types of the second images in the multiple sub-data sets are the same, and the defect causes of the second images belonging to different sub-data sets are different;

[0008] A first calculation module, used for calculating the comprehensive similarity between the first image and each sub-data set in the plurality of sub-data sets;

[0009] The first selection module is used to select a target sub-dataset and a target defect cause corresponding to the target sub-dataset from the multiple sub-datasets according to the comprehensive similarity, so as to obtain the defect cause of the PCBA to be detected.

[0010] In some embodiments of the present application, the first calculation module includes:

[0011] a determination submodule, configured to determine, for each subdata set in the plurality of subdata sets, a similarity between the first image and each second image in the subdata set;

[0012] The calculation submodule is used to calculate the average value of the similarities of all the second images in the sub-data set to obtain the comprehensive similarity.

[0013] In some embodiments of the present application, the system further includes a generation module, and the generation module includes:

[0014] An acquisition submodule, used to acquire an original image set including a PCBA image; the original image set includes an original defect image and an original non-defect image;

[0015] An extraction submodule, used to extract a first feature of each original defective image, and to extract a second feature of each original non-defective image; the first feature is a feature associated with a PCBA defect, and the second feature is a feature not associated with a PCBA defect;

[0016] A first fusion submodule is used to perform feature fusion processing on a first feature of each original defect image with each second feature to obtain a fused feature map;

[0017] A second fusion submodule is used to fuse each fused feature map with the original defect image to which the first feature belongs, so as to obtain a target defect image;

[0018] The generation submodule is used to obtain the PCBA defect image dataset using the target defect image and the original defect image.

[0019] In some embodiments of the present application, the first fusion submodule is specifically configured to include:

[0020] Using a pre-trained image merging generator to perform K fusion processing on the first feature and the second feature to obtain K candidate fusion feature maps; wherein K is an integer greater than 1;

[0021] For each candidate fused feature map, calculating the degree of difference between the original defect image corresponding to the first feature and the candidate fused feature map;

[0022] The candidate fusion feature map with the smallest difference is selected as the fusion feature map.

[0023] In some embodiments of the present application, each image in the PCBA defect image dataset is bound to a historical configuration parameter associated with the defect and a historical solution to the cause of the defect; the system further includes:

[0024] A second selection module is used to select a target second image with the greatest similarity and a historical solution bound to the target second image from the target sub-dataset, and determine the historical solution as a first candidate solution;

[0025] A second calculation module, configured to obtain a first configuration parameter corresponding to the first image, and calculate a parameter similarity between a historical configuration parameter corresponding to each second image in the target sub-dataset and the first configuration parameter;

[0026] The third selection module is used to select the historical solution corresponding to the maximum parameter similarity to obtain the second candidate solution;

[0027] An output module is used to output the first candidate solution and the second candidate solution.

[0028] In some embodiments of the present application, the first selection module is specifically used to perform the following steps:

[0029] Sorting the multiple sub-datasets according to the corresponding comprehensive similarities to obtain a sorting result;

[0030] Selecting N sub-datasets with the largest comprehensive similarity as the target sub-datasets, and obtaining N target defect causes corresponding to the N target sub-datasets; wherein N is an integer greater than 1;

[0031] The N target defect causes and the comprehensive similarities corresponding to each of the N target defect causes are output according to the sorting result.

[0032] In some embodiments of the present application, the determining submodule is specifically used to perform the following steps:

[0033] Inputting the first image and each second image into a pre-trained feature recognition model to obtain a similarity output by the feature recognition model after calculating the similarity between the first image and each second image;

[0034] The similarity recognition model is obtained by performing transfer learning training using any one or more of a plate defect image set, a printed label defect image set, a cloth printing defect image set, and a glass defect image set.

[0035] According to a second aspect of an embodiment of the present disclosure, there is provided a method for intelligent detection and classification of PCBA manufacturing defects, comprising:

[0036] Acquire a first image of the PCBA to be inspected;

[0037] Inputting the first image into a pre-trained defect recognition model to obtain a defect recognition result; the defect recognition result includes whether a defect exists and the defect type;

[0038] When the defect recognition result is that a defect exists, a PCBA defect image data set matching the defect type is obtained; the PCBA defect image data set includes multiple sub-data sets, the defect types of the second images in the multiple sub-data sets are the same, and the defect causes of the second images belonging to different sub-data sets are different;

[0039] Calculating a comprehensive similarity between the first image and each of the plurality of sub-datasets;

[0040] A target sub-dataset and a target defect cause corresponding to the target sub-dataset are selected from the multiple sub-datasets according to the comprehensive similarity to obtain the defect cause of the PCBA to be detected.

[0041] According to a third aspect of an embodiment of the present disclosure, an electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in any one of the first aspects is implemented.

[0042] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method according to any one of the first aspects is implemented.

[0043] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method as described in any one of the first aspects when executed by a processor.

[0044] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: by acquiring a first image of a PCBA to be inspected, inputting the first image into a pre-trained defect recognition model, obtaining a defect recognition result, the defect recognition result includes whether a defect exists and the defect type, and when the defect recognition result is that a defect exists, obtaining a PCBA defect image data set matching the defect type, calculating the comprehensive similarity between the first image and each of the multiple sub-data sets, selecting a target sub-data set and a target defect cause corresponding to the target sub-data set from the multiple sub-data sets according to the comprehensive similarity, and obtaining the defect cause of the PCBA to be inspected. By matching the similarity between the first image to be inspected and the PCBA defect image data set, the defect cause of the PCBA to be inspected can be quickly identified, so that the production equipment or production materials can be adjusted in time according to the defect cause, and the root cause of the defect can be solved in time, thereby improving the production efficiency of the PCBA.

[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0047] Figure 1 The figure is a schematic diagram of a structure of an intelligent detection and classification system for PCBA manufacturing defects according to an exemplary embodiment.

[0048] Figure 2 The present invention is a flowchart showing a method for intelligent detection and classification of PCBA manufacturing defects according to an exemplary embodiment.

[0049] Figure 3 The invention is a block diagram of a device for applying to a PCBA manufacturing defect intelligent detection and classification system according to an exemplary embodiment. DETAILED DESCRIPTION

[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0051] The terms used in the disclosed embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the disclosed embodiments. The singular forms "a", "an" and "the" used in the disclosed embodiments and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0052] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the disclosed embodiments, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the disclosed embodiments, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at" or "when" or "in response to determination".

[0053] In addition, various forms of processes shown in the embodiments of the present disclosure may be used to reorder, add or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present application can be achieved, and this document does not limit this.

[0054] In the related art, PCBA (Printed Circuit Board Assembly) is obtained by mounting components on a blank PCB board through a surface mount device (SMT). In order to ensure the production quality of PCBA, it is usually necessary to perform quality inspection on PCBA to identify whether there are manufacturing defects in PCBA. However, after identifying that there are defects in PCBA, in order to solve the root cause of the manufacturing defects and avoid the same manufacturing defects from occurring again, it is usually necessary to manually analyze and debug the causes of the defects based on experience. This method results in low production efficiency of PCBA.

[0055] In order to solve the above problems, the present disclosure provides an intelligent detection and classification system for PCBA manufacturing defects, which obtains a first image of the PCBA to be detected, inputs the first image into a pre-trained defect recognition model, and obtains a defect recognition result, which includes whether a defect exists and the defect type. When the defect recognition result is that a defect exists, a PCBA defect image data set matching the defect type is obtained, and the comprehensive similarity between the first image and each of the multiple sub-data sets is calculated. According to the comprehensive similarity, a target sub-data set and a target defect cause corresponding to the target sub-data set are selected from the multiple sub-data sets to obtain the defect cause of the PCBA to be detected. By matching the similarity between the first image to be detected and the PCBA defect image data set, the defect cause of the PCBA to be detected can be quickly identified, so that the production equipment or production materials can be adjusted in time according to the defect cause, and the root cause of the defect can be solved in time, thereby improving the production efficiency of the PCBA.

[0056] Figure 1 FIG. 1 is a schematic diagram of a structure of an intelligent detection and classification system for PCBA manufacturing defects according to an exemplary embodiment. Figure 1 As shown, it should be noted that the system for intelligent detection and classification of PCBA manufacturing defects in the embodiment of the present application is applied to the method for intelligent detection and classification of PCBA manufacturing defects.

[0057] like Figure 1 As shown, the system may include a first acquisition module 101 , a recognition module 102 , a second acquisition module 103 , a first calculation module 104 and a first selection module 105 .

[0058] The first acquisition module 101 is used to acquire a first image of the PCBA to be inspected.

[0059] In one embodiment, the first image may be obtained by photographing the component mounting area with a camera, or may be obtained with a visual sensor.

[0060] The recognition module 102 is used to input the first image into a pre-trained defect recognition model to obtain a defect recognition result.

[0061] The defect identification result includes whether a defect exists and the defect type.

[0062] It should be noted that the above defect recognition model can be constructed based on the YOLOv5 algorithm and can be pre-trained by using transfer learning and other methods.

[0063] In one embodiment, the first image is input into a pre-trained defect recognition model, and the defect recognition model identifies whether a PCBA manufacturing defect exists in the first image and the type of the manufacturing defect.

[0064] In one embodiment, the above manufacturing defect types may include any one of wire drawing, false welding, half welding, tin connection, and insufficient tin.

[0065] The second acquisition module 103 is used to acquire a PCBA defect image dataset matching the defect type when the defect recognition result is that a defect exists.

[0066] The PCBA defect image data set includes multiple sub-data sets, the defect types of the second images in the multiple sub-data sets are the same, and the defect causes of the second images in different sub-data sets are different.

[0067] It should be noted that for the same defect type of PCBA manufacturing defects, the causes of the defects (i.e. the cause of the defects) may be different.

[0068] For example, the reason for the pointed solder joints may be the poor fluidity of tin. The purity of tin, the amount of impurities or the concentration of flux will affect the fluidity of tin. It may also be because the soldering time is too long and the flux evaporates too quickly. It may also be due to insufficient preheating time and low preheating temperature.

[0069] For example, the reason for solder joint bridging may be that the welding temperature is too low or the welding time is too short, which will cause the solder to fail to fully melt and flow; it may also be that the flux is not active enough or the specific gravity is inappropriate, which will affect the welding quality and cause bridging; it may also be that the tin component ratio in the solder is insufficient or there are impurities that affect the fluidity of the solder and cause bridging; it may also be that the welding angle is too large or too small, and the arrangement direction of the components and the PCB design will affect the bridging.

[0070] For another example, the reason for false soldering may be poor reducibility of the flux or insufficient dosage; it may also be because the surface of the welded part has not been cleaned in advance and the tin plating is not strong; it may also be because the welding time is too long or too short, which will affect the welding effect. Too long time may cause the solder joint to burn, and too short time may cause the solder to fail to fully melt; it may also be due to the loosening of the welding component before the solder solidifies during the welding process, resulting in false soldering; it may also be due to the oxidation of the component pins, which will affect the welding effect and cause false soldering.

[0071] It is understandable that for the same defect type, different defect causes (that is, different root causes of the defect) will lead to differences in the shape of the solder joints, but such differences are difficult to identify manually. Therefore, multiple PCBA defect image data sets with different defect types can be pre-constructed, that is, the second images in the same PCBA defect image data set have the same defect type, and each PCBA defect image data set includes multiple sub-data sets, and the second images belonging to the same defect cause are divided into the same sub-data set for classifying the causes of the defects in the first image.

[0072] In one embodiment, the second image may be obtained based on a historical PCBA defect image.

[0073] The first calculation module 104 is used to calculate the comprehensive similarity between the first image and each sub-data set in the plurality of sub-data sets.

[0074] In one embodiment, the comprehensive similarity between the first image and each sub-data set may be calculated. The higher the similarity, the greater the possibility that the defect cause of the first image is consistent with the defect cause corresponding to the sub-data set.

[0075] The first selection module 105 is used to select a target sub-dataset and a target defect cause corresponding to the target sub-dataset from multiple sub-datasets according to the comprehensive similarity, so as to obtain the defect cause of the PCBA to be detected.

[0076] In one embodiment, a sub-dataset with the largest comprehensive similarity may be selected from multiple sub-datasets as the target sub-dataset, and defects corresponding to the target sub-dataset may be used as defects of the PCBA to be detected.

[0077] In some embodiments of the present application, the first selection module 105 may be further configured to perform the following steps:

[0078] Sort the multiple sub-data sets according to the corresponding comprehensive similarities to obtain a sorting result;

[0079] Select N sub-datasets with the largest comprehensive similarity as target sub-datasets, and obtain N target defect causes corresponding to the N target sub-datasets; wherein N is an integer greater than 1;

[0080] According to the sorting results, N target defect causes and the comprehensive similarities corresponding to each of the N target defect causes are output.

[0081] It is understandable that in order to enable users to more accurately find the root causes of manufacturing defects, the defect causes corresponding to N sub-data sets with relatively high comprehensive similarity can be output to the terminal device, and users can check one by one according to the level of comprehensive similarity, thereby improving the efficiency and accuracy of the inspection.

[0082] In some embodiments of the present application, the first calculation module 104 includes:

[0083] a determination submodule, configured to determine, for each subdata set in the plurality of subdata sets, a similarity between the first image and each second image in the subdata set;

[0084] The calculation submodule is used to calculate the average value of the similarities of all the second images in the sub-dataset to obtain the comprehensive similarity.

[0085] In one embodiment, the similarity may be calculated using any one or more of a cosine similarity algorithm, a Euclidean distance algorithm, and a grayscale image algorithm.

[0086] In some embodiments of the present application, the system further includes a generation module, and the generation module includes:

[0087] An acquisition submodule is used to acquire an original image set including a PCBA image; the original image set includes an original defect image and an original non-defect image;

[0088] An extraction submodule, used to extract a first feature of each original defective image, and to extract a second feature of each original non-defective image; the first feature is a feature associated with a PCBA defect, and the second feature is a feature not associated with a PCBA defect;

[0089] A first fusion submodule is used to perform feature fusion processing on the first feature of each original defect image with each second feature to obtain a fused feature map;

[0090] The second fusion submodule is used to fuse each fusion feature map with the original defect image to which the first feature belongs, so as to obtain a target defect image;

[0091] The generation submodule is used to obtain a PCBA defect image dataset using the target defect image and the original defect image.

[0092] It is understandable that the amount of original image data of defective PCBAs is very small, and it is impossible to have enough original defect images corresponding to each defect cause, which is difficult to meet the actual needs of identifying the cause of the defect. Therefore, feature fusion can be used to expand the data volume.

[0093] In one embodiment, a first feature associated with a PCBA defect can be extracted from the original defect image, for example, the first feature can include a shape feature of a solder joint, and can also include a color feature. Also, a second feature not associated with a PCBA defect can be extracted from the original non-defect image, for example, the second feature can be a texture feature. By fusing multiple first features with multiple second features in pairs, multiple fused feature maps different from the original image can be obtained.

[0094] In one embodiment, in order to make the expanded data closer to the real defect image, after obtaining the fused feature map, the fused feature map and its corresponding original defect image (that is, the original defect image to which the first feature used to produce the above fused feature map belongs) can be fused to obtain the target defect image. The template defect image and the original defect image are determined as the PCBA defect image dataset.

[0095] It should be noted that the defect cause of the target defect image is the same as the defect type and defect cause of its corresponding original defect image. When dividing the PCBA defect image dataset and sub-dataset, the PCBA defect image dataset and sub-dataset to which it belongs can be divided in this way.

[0096] In some embodiments of the present application, the first fusion submodule is specifically configured to include:

[0097] The pre-trained image merging generator is used to perform K fusion processing on the first feature and the second feature to obtain K candidate fusion feature maps; wherein K is an integer greater than 1;

[0098] For each candidate fusion feature map, calculating the difference between the original defect image corresponding to the first feature and the candidate fusion feature map;

[0099] The candidate fusion feature map with the smallest difference is selected as the fusion feature map.

[0100] It can be understood that in order to ensure that the deviation between the generated fusion feature map and the actual defect feature map is too large, resulting in the inability to play a classification role, the image merging generator can be used to perform multiple fusion processes on the first feature and the second feature. Due to the characteristics of the image merging generator itself, the candidate fusion feature map generated each time is different. Therefore, the one with the smallest difference from the original defect image can be selected from multiple candidate fusion feature maps as the fusion feature map.

[0101] In some embodiments of the present application, the first fusion submodule is further used to:

[0102] The divergence score of the candidate fusion feature map is calculated by the following formula:

[0103] in, is the divergence score, KL is the relative entropy, x is the original defect image to which the first feature of the candidate fusion feature map belongs, y is the original non-defect image to which the second feature belongs, G is the generator of the generative adversarial network GANs, is a function that computes the hidden representation of the generated image, is a feature extractor;

[0104] The divergence score is determined to obtain the degree of difference.

[0105] In some embodiments of the present application, the determination submodule is specifically used to perform the following steps:

[0106] Inputting the first image and each second image into a pre-trained feature recognition model to obtain a similarity output by the feature recognition model after calculating the similarity between the first image and each second image;

[0107] Among them, the similarity recognition model is obtained by transfer learning training using any one or more of the plate defect image set, printed label defect image set, cloth printing defect image set and glass defect image set.

[0108] It should be noted that due to the small amount of defect image data that can be used for PCBA manufacturing, the generalization ability of the model learned using a data set with only a small number of samples under each label is very limited, which can easily cause overfitting of the model training. When it is put into use, it is bound to cause serious misjudgment of a large amount of unknown data. Therefore, the feature recognition model can be pre-trained by transfer learning, and the accumulated data samples with other similar labels can be fully utilized to learn different tasks. The task characteristics can be gradually weakened during the learning process, so that the model can learn prior knowledge and use the prior knowledge to learn to identify the defect types of PCBA.

[0109] In one embodiment, the plate defect image set, the printed label defect image set, the cloth printing defect image set and the glass defect image set can be pre-sorted according to their similarity to PCBA, and the intermediate model can be trained in sequence according to the sorting. After completing the training of all the above image sets, the network model parameters of the intermediate model are configured in the above defect recognition model, and the defect recognition model is trained using the PCBA defect image sample set, thereby solving the problem that the amount of training sample data is limited and affects the accuracy of the defect recognition model.

[0110] In some embodiments of the present application, each image in the PCBA defect image dataset is bound to historical configuration parameters associated with the defect and historical solutions to the cause of the defect; the system further includes:

[0111] A second selection module is used to select the target second image with the greatest similarity and the historical solution bound to the target second image from the target sub-dataset, and determine the historical solution as the first candidate solution;

[0112] A second calculation module is used to obtain a first configuration parameter corresponding to the first image, and calculate a parameter similarity between a historical configuration parameter corresponding to each second image in the target sub-dataset and the first configuration parameter;

[0113] The third selection module is used to select the historical solution corresponding to the maximum parameter similarity to obtain the second candidate solution;

[0114] The output module is used to output the first candidate solution and the second candidate solution.

[0115] In an embodiment of the present application, each image in the PCBA defect image data set can be bound in advance with configuration parameters associated with the defect and a solution. After determining the target sub-data set (that is, after the cause of the defect), the second image with the greatest similarity and the configuration parameters and solutions bound to the second image are selected from the target sub-data set, thereby obtaining a first candidate solution. The configuration parameters corresponding to the first image can also be obtained, and the second image with the highest similarity to the configuration parameters corresponding to the first image is selected from the PCBA defect image data set. The solution corresponding to the second image is used as the second candidate solution, and the first candidate solution and the second candidate solution are used to debug the equipment or production materials, thereby helping the staff to quickly find an effective solution, thereby solving the root defect problem in a timely manner.

[0116] For example, the configuration parameters may be welding temperature, welding time, flux activity value, flux dosage, etc., and the solution may be an adjustment value of the configuration parameters.

[0117] As an example, the above parameter similarity can be calculated by using any one of a comprehensive feature method and a multi-decision method.

[0118] According to the intelligent detection and classification system for PCBA manufacturing defects proposed in the embodiment of the present application, by obtaining a first image of the PCBA to be detected, the first image is input into a pre-trained defect recognition model to obtain a defect recognition result, the defect recognition result includes whether there is a defect and the defect type, and when the defect recognition result is that there is a defect, a PCBA defect image data set matching the defect type is obtained, and the comprehensive similarity between the first image and each of the multiple sub-data sets is calculated. According to the comprehensive similarity, a target sub-data set and a target defect cause corresponding to the target sub-data set are selected from the multiple sub-data sets to obtain the defect cause of the PCBA to be detected. By matching the similarity between the first image to be detected and the PCBA defect image data set, the defect cause of the PCBA to be detected can be quickly identified, so that the production equipment or production materials can be adjusted in time according to the defect cause, and the root cause of the defect can be solved in time, thereby improving the production efficiency of the PCBA.

[0119] Figure 2 FIG. 1 is a flowchart of a method for intelligent detection and classification of PCBA manufacturing defects according to an exemplary embodiment. Figure 2 , the method comprising:

[0120] Step 101, obtaining a first image of a PCBA to be inspected;

[0121] Step 102, inputting the first image into a pre-trained defect recognition model to obtain a defect recognition result; the defect recognition result includes whether a defect exists and the defect type;

[0122] Step 103, when the defect recognition result is that there is a defect, a PCBA defect image dataset matching the defect type is obtained; the PCBA defect image dataset includes multiple sub-datasets, the defect types of the second images in the multiple sub-datasets are the same, and the defect causes of the second images belonging to different sub-datasets are different;

[0123] Step 104, calculating the comprehensive similarity between the first image and each sub-data set in the plurality of sub-data sets;

[0124] Step 105 , selecting a target sub-dataset and a target defect cause corresponding to the target sub-dataset from the multiple sub-datasets according to the comprehensive similarity, and obtaining the defect cause of the PCBA to be inspected.

[0125] In some embodiments of the present application, calculating the comprehensive similarity between the first image and each sub-data set in the plurality of sub-data sets includes:

[0126] For each sub-dataset in the plurality of sub-datasets, determining a similarity between the first image and each second image in the sub-dataset;

[0127] The average similarity of all the second images in the sub-dataset is calculated to obtain the comprehensive similarity.

[0128] In some embodiments of the present application, the PCBA defect image dataset is generated by following the steps below:

[0129] Acquire an original image set including PCBA images; the original image set includes an original defect image and an original non-defect image;

[0130] Extracting a first feature of each original defective image, and extracting a second feature of each original non-defective image; the first feature is a feature associated with the PCBA defect, and the second feature is a feature not associated with the PCBA defect;

[0131] For each first feature of the original defect image, the first feature is fused with each second feature to obtain a fused feature map;

[0132] Each fused feature map is fused with the original defect image to which the first feature belongs to obtain a target defect image;

[0133] The PCBA defect image dataset is obtained by using the target defect image and the original defect image.

[0134] In some embodiments of the present application, the first feature is fused with each second feature to obtain a fused feature map, including:

[0135] The pre-trained image merging generator is used to perform K fusion processing on the first feature and the second feature to obtain K candidate fusion feature maps; wherein K is an integer greater than 1;

[0136] For each candidate fusion feature map, calculating the difference between the original defect image corresponding to the first feature and the candidate fusion feature map;

[0137] The candidate fusion feature map with the smallest difference is selected as the fusion feature map.

[0138] In some embodiments of the present application, each image in the PCBA defect image data set is bound to a historical configuration parameter associated with the defect and a historical solution to the defect cause; after selecting a target sub-data set and a target defect cause corresponding to the target sub-data set from multiple sub-data sets according to the comprehensive similarity, and obtaining the defect cause of the PCBA to be detected, the method further includes:

[0139] Selecting the target second image with the greatest similarity and the historical solution bound to the target second image from the target sub-dataset, and determining the historical solution as the first candidate solution;

[0140] Acquire a first configuration parameter corresponding to the first image, and calculate a parameter similarity between a historical configuration parameter corresponding to each second image in the target sub-dataset and the first configuration parameter;

[0141] Select the historical solution corresponding to the maximum parameter similarity to obtain the second candidate solution;

[0142] Output the first candidate solution and the second candidate solution.

[0143] In some embodiments of the present application, a target sub-dataset and a target defect cause corresponding to the target sub-dataset are selected from multiple sub-datasets according to the comprehensive similarity to obtain the defect cause of the PCBA to be detected, including:

[0144] Sort the multiple sub-data sets according to the corresponding comprehensive similarities to obtain a sorting result;

[0145] Select N sub-datasets with the largest comprehensive similarity as target sub-datasets, and obtain N target defect causes corresponding to the N target sub-datasets; wherein N is an integer greater than 1;

[0146] According to the sorting results, N target defect causes and the comprehensive similarities corresponding to each of the N target defect causes are output.

[0147] In some embodiments of the present application, determining the similarity between the first image and each second image in the sub-data set includes:

[0148] Inputting the first image and each second image into a pre-trained feature recognition model to obtain a similarity output by the feature recognition model after calculating the similarity between the first image and each second image;

[0149] Among them, the similarity recognition model is obtained by transfer learning training using any one or more of the plate defect image set, printed label defect image set, cloth printing defect image set and glass defect image set.

[0150] Regarding the methods in the above embodiments, the specific manner in which each method performs operations has been described in detail in the embodiments of the system, and will not be elaborated here.

[0151] According to the intelligent detection and classification method for PCBA manufacturing defects proposed in the embodiment of the present application, by obtaining a first image of the PCBA to be detected, the first image is input into a pre-trained defect recognition model to obtain a defect recognition result, the defect recognition result includes whether a defect exists and the defect type, and when the defect recognition result is that a defect exists, a PCBA defect image data set matching the defect type is obtained, and the comprehensive similarity between the first image and each of the multiple sub-data sets is calculated. According to the comprehensive similarity, a target sub-data set and a target defect cause corresponding to the target sub-data set are selected from the multiple sub-data sets to obtain the defect cause of the PCBA to be detected. By matching the similarity between the first image to be detected and the PCBA defect image data set, the defect cause of the PCBA to be detected can be quickly identified, so that the production equipment or production materials can be adjusted in time according to the defect cause, and the root cause of the defect can be solved in time, thereby improving the production efficiency of the PCBA.

[0152] Figure 3 3 is a block diagram of a device for use in a PCBA manufacturing defect intelligent detection and classification system according to an exemplary embodiment. For example, the device 300 may be an electronic device, such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0153] Reference Figure 3 , the device 300 may include one or more of the following components: a processing component 302 , a memory 304 , a power component 306 , a multimedia component 308 , an audio component 310 , an input / output (I / O) interface 312 , a sensor component 314 , and a communication component 316 .

[0154] The processing component 302 generally controls the overall operation of the device 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.

[0155] The memory 304 is configured to store various types of data to support operations on the device 300. Examples of such data include instructions for any application or method operating on the device 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0156] The power component 306 provides power to the various components of the device 300. The power component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 300.

[0157] The multimedia component 308 includes a screen that provides an output interface between the device 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0158] The audio component 310 is configured to output and / or input audio signals. For example, the audio component 310 includes a microphone (MIC), and when the device 300 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.

[0159] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0160] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the device 300. For example, the sensor assembly 314 can detect the open / closed state of the device 300, the relative positioning of components, such as the display and keypad of the device 300, the sensor assembly 314 can also detect the position change of the device 300 or a component of the device 300, the presence or absence of user contact with the device 300, the orientation or acceleration / deceleration of the device 300, and the temperature change of the device 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0161] The communication component 316 is configured to facilitate wired or wireless communication between the device 300 and other devices. The device 300 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0162] In an exemplary embodiment, the apparatus 300 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0163] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, and the instructions can be executed by a processor 320 of the device 300 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0164] In an exemplary embodiment, a computer program product is also provided, comprising a computer program, which implements the above method when executed by the processor 320 of the device 300 .

[0165] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0166] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An intelligent detection and classification system for PCBA manufacturing defects, characterized in that: include: A first acquisition module, used to acquire a first image of the PCBA to be inspected; A recognition module, used for inputting the first image into a pre-trained defect recognition model to obtain a defect recognition result; The defect recognition result includes whether there is a defect and the defect type; the defect type includes any one or more of wire drawing, false welding, half welding, continuous tin, and insufficient tin; A second acquisition module is used to acquire a PCBA defect image data set matching the defect type when the defect recognition result is that a defect exists; the PCBA defect image data set includes multiple sub-data sets, the defect types of the second images in the multiple sub-data sets are the same, and the defect causes of the second images belonging to different sub-data sets are different; A first calculation module, used for calculating the comprehensive similarity between the first image and each sub-data set in the plurality of sub-data sets; A first selection module is used to select a target sub-dataset and a target defect cause corresponding to the target sub-dataset from the multiple sub-datasets according to the comprehensive similarity, so as to obtain the defect cause of the PCBA to be detected; Wherein, the first calculation module includes: a determination submodule, configured to determine, for each subdata set in the plurality of subdata sets, a similarity between the first image and each second image in the subdata set; A calculation submodule, used for calculating the average value of the similarities of all the second images in the sub-dataset to obtain the comprehensive similarity; The system further includes a generation module, which includes: An acquisition submodule, used to acquire an original image set including a PCBA image; the original image set includes an original defect image and an original non-defect image; An extraction submodule, used to extract a first feature of each original defective image, and to extract a second feature of each original non-defective image; the first feature is a feature associated with a PCBA defect, and the second feature is a feature not associated with a PCBA defect; A first fusion submodule is used to perform feature fusion processing on a first feature of each original defect image with each second feature to obtain a fused feature map; A second fusion submodule is used to fuse each fused feature map with the original defect image to which the first feature belongs, so as to obtain a target defect image; A generation submodule, used to obtain the PCBA defect image dataset using the target defect image and the original defect image; The first fusion submodule is specifically configured to include: Using a pre-trained image merging generator to perform K fusion processing on the first feature and the second feature to obtain K candidate fusion feature maps; wherein K is an integer greater than 1; For each candidate fused feature map, calculating the degree of difference between the original defect image corresponding to the first feature and the candidate fused feature map; Selecting the candidate fusion feature map with the smallest difference as the fusion feature map; Each image in the PCBA defect image dataset is bound to a historical configuration parameter associated with the defect and a historical solution to the cause of the defect; the system also includes: A second selection module is used to select a target second image with the greatest similarity and a historical solution bound to the target second image from the target sub-dataset, and determine the historical solution as a first candidate solution; a second calculation module, configured to obtain first configuration parameters corresponding to the first image, and calculate parameter similarity between historical configuration parameters corresponding to each second image in the target sub-dataset and the first configuration parameters; the first configuration parameters include welding temperature, welding time, flux activity value and flux dosage; The third selection module is used to select the historical solution corresponding to the maximum parameter similarity to obtain the second candidate solution; An output module is used to output the first candidate solution and the second candidate solution.

2. The intelligent detection and classification system for PCBA manufacturing defects according to claim 1 is characterized in that: The first selection module is specifically used to perform the following steps: Sorting the multiple sub-datasets according to the corresponding comprehensive similarities to obtain a sorting result; Selecting N sub-datasets with the largest comprehensive similarity as the target sub-datasets, and obtaining N target defect causes corresponding to the N target sub-datasets; wherein N is an integer greater than 1; The N target defect causes and the comprehensive similarities corresponding to each of the N target defect causes are output according to the sorting result.

3. The intelligent detection and classification system for PCBA manufacturing defects according to claim 1 is characterized in that: The determining submodule is specifically used to perform the following steps: Inputting the first image and each second image into a pre-trained similarity recognition model to obtain a similarity output by the feature recognition model after calculating the similarity between the first image and each second image; The similarity recognition model is obtained by performing transfer learning training using any one or more of a plate defect image set, a printed label defect image set, a cloth printing defect image set, and a glass defect image set.

4. A method for intelligent detection and classification of PCBA manufacturing defects, characterized in that: include: Acquire a first image of the PCBA to be inspected; Inputting the first image into a pre-trained defect recognition model to obtain a defect recognition result; The defect recognition result includes whether there is a defect and the defect type; the defect type includes any one or more of wire drawing, false welding, half welding, continuous tin, and insufficient tin; When the defect recognition result is that a defect exists, a PCBA defect image data set matching the defect type is obtained; the PCBA defect image data set includes multiple sub-data sets, the defect types of the second images in the multiple sub-data sets are the same, and the defect causes of the second images belonging to different sub-data sets are different; Calculating a comprehensive similarity between the first image and each of the plurality of sub-datasets; Selecting a target sub-dataset and a target defect cause corresponding to the target sub-dataset from the multiple sub-datasets according to the comprehensive similarity, and obtaining the defect cause of the PCBA to be detected; The step of calculating the comprehensive similarity between the first image and each of the plurality of sub-datasets comprises: For each sub-data set in the plurality of sub-data sets, determining a similarity between the first image and each second image in the sub-data set; Calculating an average value of the similarities of all the second images in the sub-dataset to obtain the comprehensive similarity; The method further includes: Acquire an original image set including PCBA images; the original image set includes original defective images and original non-defective images; Extracting a first feature of each original defective image, and extracting a second feature of each original non-defective image; the first feature is a feature associated with a PCBA defect, and the second feature is a feature not associated with a PCBA defect; For each first feature of the original defect image, perform feature fusion processing on the first feature and each second feature to obtain a fused feature map; Fusing each fused feature map with the original defect image to which the first feature belongs, to obtain a target defect image; Using the target defect image and the original defect image, the PCBA defect image dataset is obtained; The first feature of each original defect image is subjected to feature fusion processing with each second feature to obtain a fused feature map, including: Using a pre-trained image merging generator to perform K fusion processing on the first feature and the second feature to obtain K candidate fusion feature maps; wherein K is an integer greater than 1; For each candidate fused feature map, calculating the degree of difference between the original defect image corresponding to the first feature and the candidate fused feature map; Selecting the candidate fusion feature map with the smallest difference as the fusion feature map; Each image in the PCBA defect image dataset is bound to a historical configuration parameter associated with the defect and a historical solution to the cause of the defect; the method further includes: Selecting a target second image with the greatest similarity and a historical solution bound to the target second image from the target sub-dataset, and determining the historical solution as a first candidate solution; Acquire first configuration parameters corresponding to the first image, and calculate parameter similarity between historical configuration parameters corresponding to each second image in the target sub-dataset and the first configuration parameters; the first configuration parameters include welding temperature, welding time, flux activity value and flux dosage; Select the historical solution corresponding to the maximum parameter similarity to obtain the second candidate solution; The first candidate solution and the second candidate solution are output.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the system according to any one of claims 1 to 3 is implemented.

Images