Electronic device defect detection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an electronic device defect detection method and device, electronic equipment and a storage medium. In an automatic optical detection scene, under the condition that a to-be-detected device is confirmed to be an electronic device with a limited installation direction, image features used for describing the corresponding limited installation directions are extracted from a to-be-detected device image of the to-be-detected device and a corresponding standard device image through a pre-trained direction feature extraction model respectively; and determining whether the installation directions of the electronic devices in the to-be-detected device image and the standard device image are consistent or not according to the similarity of the image features correspondingly identified by the to-be-detected device image and the standard device image so as to complete defect detection about the installation direction. Under the condition that a reference standard device image exists, defect detection about the installation direction is rapidly completed, the universality of detection of the polar electronic device is good, and the robustness of detection of various polar electronic devices is obviously improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of optical detection technology, and in particular to an electronic device defect detection method, apparatus, electronic equipment, and storage medium. Background Art

[0002] In the actual application scenarios of AOI (Automated Optical Inspection), it is usually necessary to obtain the position, shape, size and other parameters of the pins of electronic components (such as integrated circuits, chips, connectors, etc.) to ensure that they are installed normally without installation defects and can be installed and used normally in the future. Traditional AOI is mainly implemented through two-dimensional images, that is, two-dimensional images are detected through cameras, and color comparison is performed based on the images of standard devices to confirm whether there are defects in the installation.

[0003] When the inventor used the existing AOI detection method for detection, he found that due to the variety of device types, the universality of the installation direction detection scheme for polar electronic devices was poor, and the robustness of the direction detection for various polar electronic devices was poor. Summary of the invention

[0004] The present invention provides an electronic device defect detection method, device, electronic device and storage medium to solve the technical problems that the existing installation direction detection scheme for polar electronic devices has poor versatility and the direction detection for various polar electronic devices has poor robustness.

[0005] In a first aspect, an embodiment of the present invention provides an electronic device defect detection method, the electronic device defect detection method comprising:

[0006] When confirming that the device to be detected is an electronic device with a limited installation direction, obtaining a standard device image corresponding to the device to be detected and a device to be detected image corresponding to the device to be detected;

[0007] Input the device under test image and the standard device image into a directional feature extraction model respectively, and extract the image features of the device under test and the image features of the standard device accordingly, wherein the image features include image features corresponding to each limited installation direction, and the directional feature extraction model is obtained by performing direction-based center clustering training on sample device images;

[0008] Calculating the similarity between the image features of the device under test and the image features of the standard device corresponding to each limited installation direction to obtain at least two similarities;

[0009] The defect detection result of the device to be detected is determined according to the relative size of the similarity.

[0010] In the above-mentioned automatic optical inspection process, when it is confirmed that the device to be inspected is an electronic device with a limited installation direction, the device image to be inspected and the corresponding standard device image of the device to be inspected are respectively subjected to a pre-trained direction feature extraction model to extract image features for describing the corresponding limited installation directions. According to the similarity of the image features correspondingly identified between the two, it is confirmed whether the installation directions of the electronic devices in the device image to be inspected and the standard device image are consistent, thereby completing the defect detection regarding the installation direction. Based only on the comparison result of the image features of the limited installation direction, it is determined whether the image of the device to be inspected has defects in the installation direction. For polar electronic devices in the image of the device to be inspected, it is not necessary to adapt to the specific circuit board and electronic device type to obtain a clear direction and then confirm whether the direction is correct. Based on the image features, the defect detection regarding the installation direction can be quickly completed with reference to the standard device image. The detection of polar electronic devices has good versatility and significantly improves the robustness of realizing the detection of various polar electronic devices.

[0011] Wherein, the defect detection result of the device to be detected is determined according to the relative size of the similarity, including:

[0012] The image features of the device under test are matched with the corresponding image features of the standard device. When the matching results are consistent, it is confirmed that the installation direction detection of the device under test is qualified; when the matching results are inconsistent, it is confirmed that the installation direction detection of the device under test is unqualified.

[0013] As described above, through image feature matching, as long as the image features are consistent, it is possible to confirm whether the installation direction is qualified. There is no need to set a standard direction for the specific device type of the device to be detected, which reduces the user's setting operations when the type of the device to be detected changes, and improves the robustness of detection of various types of polarity devices.

[0014] Among them, the standard device image and the device image to be tested are both device images after background removal.

[0015] As described above, using the background-removed standard device image and the device-to-be-tested image to perform direction detection can effectively reduce interference in the process of extracting features related to the installation direction and improve detection accuracy.

[0016] Among them, the directional feature extraction model is pre-trained in the following way:

[0017] Acquire multiple binary images as third training samples, each binary image being a forward device image and a reverse device image of a device;

[0018] The third neural network model is trained by center clustering through the third training sample to obtain a directional feature extraction model. The constraint condition of the center clustering training is that device images with the same direction are clustered around the same direction center, and device images with different directions are scattered in their respective direction centers.

[0019] As described above, through supervised training of the directional dimension, the characteristics of the orientation dimension are identified instead of identifying the specific directional state, so that the directions of various installed devices can be identified without constraints.

[0020] The standard device image includes a forward direction standard device image and a reverse direction standard device image;

[0021] Accordingly, the defect detection result of the device to be detected is determined according to the relative size of the similarity, including:

[0022] When the forward comprehensive similarity between the image features of the device to be tested and the forward direction image features is greater than the reverse comprehensive similarity between the image features of the device to be tested and the reverse direction image features, it is confirmed that the installation direction of the device to be tested is qualified, the forward direction image features are the standard device image features corresponding to the forward direction standard device image, and the reverse direction image features are the standard device image features corresponding to the reverse direction standard device image.

[0023] As described above, by comparing the standard device images in two directions with completely opposite polarity states, flexible comparison can be performed according to the specific states of the devices in the board.

[0024] The standard device image is a standard device image in the positive direction; the device image to be tested includes an original image to be tested and a derived image to be tested obtained by rotating the original image to be tested by 180°;

[0025] Accordingly, the defect detection result of the device to be detected is determined according to the relative size of the similarity, including:

[0026] When the original comprehensive similarity between the original image features and the standard device image features is greater than the derived comprehensive similarity between the derived image features and the standard device image features, it is confirmed that the installation direction of the device to be tested is qualified. The original image features are the image features of the device to be tested corresponding to the original image to be tested, and the derived image features are the image features of the device to be tested corresponding to the derived image to be tested.

[0027] As described above, by comparing the two images to be tested from another angle with completely opposite polarity states, a flexible comparison can be performed according to the specific states of the devices in the board.

[0028] In a second aspect, an embodiment of the present invention provides an electronic device defect detection device, the electronic device defect detection device comprising:

[0029] A type image confirmation unit, used to obtain a standard device image corresponding to the device to be detected and a device to be detected image corresponding to the device to be detected when confirming that the device to be detected is an electronic device with a limited installation direction;

[0030] A feature extraction unit is used to input the image of the device under test and the image of the standard device into a directional feature extraction model, respectively, and correspondingly extract the image features of the device under test and the image features of the standard device, wherein the image features include image features corresponding to each limited installation direction, and the directional feature extraction model is obtained by performing direction-based center clustering training on the sample device images;

[0031] A similarity calculation unit, used to calculate the similarity between the image features of the device under test and the image features of the standard device corresponding to each limited installation direction, to obtain at least two similarities;

[0032] The result determination unit is used to determine the defect detection result of the device to be detected according to the relative size of the similarity.

[0033] The result determination unit includes:

[0034] The feature matching module is used to match the image features of the device to be tested with the corresponding image features of the standard device. When the matching results are consistent, it is confirmed that the installation direction detection of the device to be tested is qualified; when the matching results are inconsistent, it is confirmed that the installation direction detection of the device to be tested is unqualified.

[0035] Among them, the standard device image and the device image to be tested are both device images after background removal.

[0036] Among them, the directional feature extraction model is pre-trained in the following way:

[0037] Acquire multiple binary images as third training samples, each binary image being a forward device image and a reverse device image of a device;

[0038] The third neural network model is trained by center clustering through the third training sample to obtain a directional feature extraction model. The constraint condition of the center clustering training is that device images with the same direction are clustered around the same direction center, and device images with different directions are scattered in their respective direction centers.

[0039] The standard device image includes a forward direction standard device image and a reverse direction standard device image;

[0040] Accordingly, the result determination unit includes:

[0041] The first confirmation module is used to confirm that the installation direction of the device to be tested is qualified when the forward comprehensive similarity between the image feature of the device to be tested and the forward direction image feature is greater than the reverse comprehensive similarity between the image feature of the device to be tested and the reverse direction image feature, the forward direction image feature is the standard device image feature corresponding to the forward direction standard device image, and the reverse direction image feature is the standard device image feature corresponding to the reverse direction standard device image.

[0042] The standard device image is a standard device image in the positive direction; the device image to be tested includes an original image to be tested and a derived image to be tested obtained by rotating the original image to be tested by 180°;

[0043] Accordingly, the result determination unit includes:

[0044] The second confirmation module is used to confirm that the installation direction of the device to be tested is qualified when the original comprehensive similarity between the original image feature and the standard device image feature is greater than the derived comprehensive similarity between the derived image feature and the standard device image feature, the original image feature is the image feature of the device to be tested corresponding to the original image to be tested, and the derived image feature is the image feature of the device to be tested corresponding to the derived image to be tested.

[0045] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0046] one or more processors;

[0047] a memory for storing one or more computer programs;

[0048] When one or more computer programs are executed by one or more processors, the electronic device implements the electronic device defect detection method of the first aspect.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the electronic device defect detection method of the first aspect.

[0050] The electronic device of the third aspect and the computer-readable storage medium of the fourth aspect can be used to execute the electronic device defect detection method provided by any of the above embodiments, and have corresponding functions and beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A method flow chart of an electronic device defect detection method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of the structure of an electronic device defect detection device provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are used to explain the present invention, rather than to limit the present invention. It should also be noted that, for the sake of ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0056] It should be noted that due to space limitations, this application description does not list all optional implementation methods. After reading this application description, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of technical features can constitute an optional implementation method.

[0057] Each embodiment is described in detail below.

[0058] Traditional AOI is mainly achieved through two-dimensional images, that is, the two-dimensional image is detected by a camera, and the image is compared with the image of the standard device to confirm whether there are any defects in the installation.

[0059] In the specific defect detection scheme, the polarity of electronic devices is a very important step in the AOI detection process. For example, diode components have polarity. If they are inserted in the wrong direction during installation, a short circuit will occur.

[0060] The existing detection of installation defects of polar devices is mainly based on training classification models, and then using the classification models to identify the installation direction of electronic devices in the image, and then determine whether there are defects in the installation direction. However, the existing classification models can usually only be trained for one type or even one model of polar devices. The trained classification model can achieve good formation in the corresponding single electronic device detection, but the versatility is not good. The direction detection of multiple electronic devices often requires multiple classification models. With the continuous development of electronic devices, it is usually necessary to train new classification models accordingly. Overall, the existing classification models for installation direction detection of polar electronic devices have poor versatility, and the cost of model training for detecting various polar electronic devices is relatively high.

[0061] In order to solve the above technical problems, the embodiment of the present application proposes a method for detecting defects in electronic devices. During the automatic optical inspection process, when it is confirmed that the device to be inspected is an electronic device with a limited installation direction, the device image to be inspected and the corresponding standard device image of the device to be inspected are respectively extracted through a pre-trained direction feature extraction model to extract image features for describing the corresponding limited installation directions. According to the similarity of the image features correspondingly identified by the two, it is confirmed whether the installation directions of the electronic devices in the device image to be inspected and the standard device image are consistent, thereby completing the defect detection on the installation direction. Based only on the comparison result of the image features of the limited installation direction, it is judged whether the image of the device to be inspected has defects in the installation direction. For polar electronic devices in the image of the device to be inspected, it is not necessary to adapt to the specific circuit board and electronic device type to obtain a clear direction and then confirm whether the direction is correct. Based on the image features, the defect detection on the installation direction can be quickly completed in the case of a reference standard device image. The detection of polar electronic devices has good versatility, and the robustness of realizing the detection of various polar electronic devices is significantly improved.

[0062] Figure 1 is a method flow chart of an electronic device defect detection method provided by an embodiment of the present application, such as Figure 1 As shown, the electronic device defect detection method includes but is not limited to steps S110 to S130.

[0063] Step S110: when it is confirmed that the device to be inspected is an electronic device with a limited installation direction, a standard device image corresponding to the device to be inspected and a device to be inspected image corresponding to the device to be inspected are obtained.

[0064] Confirming the type of device to be detected can be to receive confirmation from the user according to the setting operation required for the current detection, or to run a pre-trained device type recognition model to identify and confirm the device in the PCB board image. Whether it is directly receiving the user's operation confirmation or actively identifying and confirming, the device type can be used for subsequent detection and judgment. It should be understood that in the specific implementation process, it is usually necessary to continuously obtain the PCB board image, from which to confirm the image of the device to be detected corresponding to the device to be detected, and to implement defect detection of the corresponding device to be detected based on the image of the device to be detected. Common device types such as chips, capacitors, resistors, etc., can be divided into polar devices and non-polar devices according to the polarity characteristics of the device. Polar devices limit the installation direction. For example, reverse installation of diodes may cause short circuits. In addition, some capacitors are also polar devices. Non-polar devices do not limit the installation direction. Both forward and reverse installations can be normally turned on to achieve the established design goals. For example, both forward and reverse installations of resistors can normally achieve the corresponding circuit functions. In addition, most capacitors are also non-polar devices.

[0065] The standard device image is obtained by capturing the image of a qualified PCB board that has been produced, and then extracting the image of another device that is the same as the device to be tested from the PCB board image. The image of the device to be tested and the image of the standard device are both acquired through optical camera image acquisition, and the specific acquisition parameters are as similar as possible, such as ambient light status, focal length, camera shooting distance, etc., so as to ensure the accuracy of the test results as much as possible.

[0066] There is at least one standard device image, and the standard device image may be a standard image corresponding to the device image under test, or may be an image completely unrelated to the device image under test. The standard device image and the device image under test may both be images unrelated to the training images of the directional feature extraction model, that is, relative to the directional feature extraction model, the standard device image and the device image under test are both "unseen" images.

[0067] When performing defect detection on a device, the device is in a state where it has been installed on a circuit board. When performing image acquisition during detection, the image of the entire circuit board is usually acquired. In the embodiment of the present application, one device is detected each time, and the image of the device to be detected corresponding to each device to be detected needs to be extracted from the corresponding circuit board image. The specific extraction method can be to use the opencv template matching algorithm, that is, firstly perform image acquisition on a circuit board that is confirmed to have no production defects to obtain a standard circuit board image, and then confirm the standard device image corresponding to each device in the standard circuit board image based on device identification and / or manual calibration, and also confirm the device type of each device in the standard circuit board image. For the image of the circuit board collected in the subsequent detection process, match it according to the calibration in the standard circuit board image, extract the image of the device to be detected corresponding to each device to be detected, which is equivalent to confirming the device type corresponding to each extracted device to be detected. That is, confirming the device type of the device to be detected can be directly confirmed according to the device type calibrated in the standard circuit board image during the extraction process. Confirming that each device corresponds to a standard device image based on device identification and / or manual calibration means that the device type can be identified by image, and the device in the standard circuit board image can be directly confirmed to complete the device type calibration, or the device in the standard circuit board image can be manually calibrated, or a preliminary calibration can be performed by image recognition and then the final calibration can be manually confirmed. The corresponding acquisition of the standard device image can be to obtain the corresponding setting operation of the user and read the corresponding standard device image. By providing information that can identify the type of the device to be detected in various ways, the type of the device to be detected can be quickly identified in different application scenarios, and it can be quickly confirmed whether the detection strategy in the embodiment of the present application needs to be adopted.

[0068] Step S120: Input the image of the device under test and the image of the standard device into the directional feature extraction model respectively, and extract the image features of the device under test and the image features of the standard device accordingly. The image features include image features corresponding to each limited installation direction. The directional feature extraction model is obtained by performing direction-based center clustering training on the sample device images.

[0069] For electronic devices with limited installation directions, for the image of the device under test, the image features of the device under test are extracted; for the image of the standard device, the image features of the standard device are extracted. The image features of the device under test and the image features of the standard device are represented by feature vectors, and the image features corresponding to each limited installation direction are extracted by the pre-trained direction feature extraction model.

[0070] The overall training process of the directional feature extraction model can refer to the implementation process in the relevant technology. In the embodiment of the present application, the detailed design of the specific recognition target in the specific application scenario is mainly described. The directional feature extraction model can use the more mainstream neural network model in the relevant technology, such as Resnet, DenseNet or other networks.

[0071] The directional feature extraction model can identify “unseen” DUT images and standard device images, and realize feature extraction and directional recognition of any DUT image.

[0072] The directional feature extraction model can be pre-trained in the following manner: multiple binary images are obtained as the third training samples, each binary image is a forward device image and a reverse device image of a device; the directional feature extraction model is obtained by performing center clustering training on the preset third neural network model through the third training samples, and the constraint condition of the center clustering training is that device images with the same direction are clustered around the same direction center, and device images with different directions are scattered in their respective direction centers.

[0073] For the direction dimension, based on the neural network model in the previous article, we can refer to the binary input of contrastive loss (contrastive loss function). A binary refers to "one forward and one reverse" data as a unit data, that is, the initial image obtained by cropping and processing is rotated 180° to form "bidirectional" data, and then form "binary data".

[0074] The cropping and processing are based on the original image obtained by originally photographing a circuit board with multiple electronic devices installed. For example, the original image is detected and automatically segmented in units of electronic devices using a target detection or segmentation algorithm to obtain the original device image. Alternatively, an original image can be used as a template image, and the position corresponding to each original device image is marked in the template image. Then, according to the marked position in the template image, the original device image is directly cropped from other original images. In this case, the image obtained is an image without the background of the electronic device, but only the electronic device itself. The method of marking the position of a device image in the template image is, for example, represented by (x, y, w, h), where (x, y) represents the position of the upper left corner vertex of the device image, w represents x pixels to the right from the upper left corner vertex, and h represents y pixels below the upper left corner vertex.

[0075] In the specific implementation process, the original device images obtained by the above method may have different sizes, and the specific training may be based on images of the same size. In this case, the original device images need to be adjusted. For example, if the training uses a 128*128 image, the original device image first needs to be scaled so that the length of the longer side is equal to 128, and then the longer side is kept unchanged and filled with equal amounts on both sides (for example, filling with the number 0), and finally a 128*128 initial image is obtained, and the initial image is rotated 180° to form "bidirectional" data, and then "binary" data is formed.

[0076] Since the style of the directly collected data is relatively simple, and the data in reality is relatively changeable. For example, a picture of "polarity" may be rotated, noisy, etc. in reality. By designing an algorithm to rotate and add noise to the original data, the situation that may be encountered in reality is simulated, so that the model has anti-interference. In addition, in order to ensure that the electronic device will not be deformed, the size of the cropped image can be reset, for example, the image width is controlled to 128 pixels, and the image height is scaled proportionally. After the image height is scaled, it will be smaller than the image width. At this time, the cropped and scaled image needs to be filled with 0 on both sides with it as the center, and the final image will form 128*128.

[0077] During the training process, the bounding box of the data will be randomly changed to simulate the disturbance of the real environment. The network design specifically uses CNN (Convolutional Neural Networks) + FC (Fully Connected Layer). CNN specifically uses PReul active function + Pooling, that is, using activation function with parameters + pooling. FC outputs a 256-dimensional vector for calculating similarity. The output of the FC layer should not be too large. If it is too large, it will take a long time to calculate the similarity between two features. At the same time, the output of the FC layer should not be too small. If it is too small, the feature's ability to represent polarity is insufficient, resulting in performance degradation. The loss of the neural network model can be CenterLoss (center loss function), or other loss functions can be used, such as triplet loss (triplet loss function, the I / O needs to be modified at this time, triplet loss is used in triplets). Each device has a center in each direction, and the center calculation uses the average of the 256-dimensional vector. During the training process, features are considered simultaneously, and the direct associations between "forward and forward", "forward and reverse", and "reverse and reverse" are achieved. The 256-dimensional FC output is more "directional" distinguishable (positive pair and negative pair, that is, positive sample pair and negative sample pair). The directional feature extraction model trained in the above way can handle unseen devices. In the above training process, based on center clustering, it is equivalent to realizing the classification ability after training by constraining the feature center clustering. In the specific implementation, contrastive loss can also be used to achieve it through other constraints under the feature dimension; but the overall implementation goal is to make the feature distance of samples with the same attributes very close, and the feature distance of samples with different attributes very far.

[0078] Device images with the same orientation are gathered around the same orientation center, and device images with different orientations are scattered in their respective orientation centers. This can be understood as mapping the "polarity" data to a high-dimensional space through a neural network model. In the high-dimensional space, each "polarity" will have a corresponding center. During the training process, features with the same "polarity" will be gathered to their corresponding centers. The final result after such training is that the distance between features with the same "polarity" will be very close, while the distance between features with different "polarities" will be very far, so that this feature has the ability to distinguish different "polarities". Center clustering of "polarity" features can enable the trained directional feature extraction model to learn the distribution of "polarity" features in the embedding space. The design of the embedding space makes similar "polarity" features close together and dissimilar "polarity" features far away. When the directional feature extraction model is applied to directional feature extraction, good recognition can be achieved for electronic devices that have not been seen during the training process.

[0079] Through supervised training in the directional dimension, the characteristics of the orientation dimension are identified instead of the specific directional state, so that the image features corresponding to various installed electronic devices in each limited installation direction can be extracted without constraints.

[0080] On the basis of the directional feature extraction model training that has been completed, in the specific detection process, it is only necessary to input the image of the device to be tested and the image of the standard device into the directional feature extraction model for directional feature extraction to obtain the corresponding image features of the device to be tested and the image features of the standard device. Because the directional feature extraction model is obtained by performing direction-based center clustering training on the sample device image, the directional feature extraction model has the ability to extract directional features and can cluster the directional features corresponding to each limited installation direction, and finally obtain the feature vector corresponding to each limited installation direction. Through the pre-trained directional feature extraction model, the image features used to describe the corresponding to each limited installation direction are extracted. There is no need to clarify the specific installation direction corresponding to the type of device to be tested. Based on the image features, the defect detection on the installation direction can be quickly completed with the reference of the standard device image.

[0081] Step S130: calculating the similarity between the image features of the device under test and the image features of the standard device corresponding to each limited installation direction, and obtaining at least two similarities.

[0082] Multiple image features corresponding to each limited installation direction can be extracted from both the image of the device under test and the image of the standard device. The image features of each limited installation direction are obtained by performing feature clustering corresponding to each limited installation direction after the directional feature extraction model extracts the features related to the direction. Because the image of the device under test and the image of the standard device are obtained by taking pictures of the same electronic device in the same or different installation directions, the difference in specific image content may only appear in the direction. Therefore, for the features of the two images in each limited installation direction, the similarity is confirmed according to the respective feature vectors corresponding to the same limited installation direction, thereby obtaining the similarity of at least two limited installation directions. The similarity can be the similarity between the image features of the device under test image and the image features of the standard device in different installation directions, or the similarity between the image features of the device under test image in different installation directions and the image features of the standard device. The specific similarity can be confirmed by Euclidean distance, cosine distance, information entropy, etc.

[0083] Step S140: determining the curve detection result of the device to be detected according to the relative size of the similarity.

[0084] In the embodiment of the present application, considering that the device to be inspected has directionality, but the differences in other characteristic dimensions are small, in the specific detection process, detection is performed according to the feature extraction model corresponding to the directional dimension (i.e., the directional feature extraction model) to confirm whether the device to be inspected has major defects.

[0085] When specifically confirming the test results, the standard device image may include a forward direction standard device image and a reverse direction standard device image; when the forward comprehensive similarity between the image features of the device to be tested and the forward direction image features is greater than the reverse comprehensive similarity between the image features of the device to be tested and the reverse direction image features, it is confirmed that the installation direction of the device to be tested is qualified, the forward direction image features are the standard device image features corresponding to the forward direction standard device image, and the reverse direction image features are the standard device image features corresponding to the reverse direction standard device image.

[0086] The standard device images in two directions are compared with the image of the device to be tested in completely opposite polarity states. Considering that each limited installation direction has a corresponding clustering result, when the image features of the device to be tested are compared with the image features of the standard device, the clustering results of the same limited installation direction have a corresponding similarity. When performing a comprehensive evaluation, the clustering results of each limited installation direction are integrated (for example, weighted summation), which can further amplify the difference between the forward comprehensive similarity and the reverse comprehensive similarity, and can be flexibly compared according to the specific state of the device in the board. The obvious differences that must exist between the forward direction image and the reverse direction image and the image of the device to be tested are compared to obtain the directional defect detection result.

[0087] When specifically confirming the test results, the standard device image can also be a standard device image in the positive direction; the image of the device to be tested includes the original image to be tested and a derived image to be tested obtained by rotating the original image to be tested 180°; when the original comprehensive similarity between the original image features and the standard device image features is greater than the derived comprehensive similarity between the derived image features and the standard device image features, it is confirmed that the installation direction of the device to be tested is qualified, the original image features are the image features of the device to be tested corresponding to the original image to be tested, and the derived image features are the image features of the device to be tested corresponding to the derived image to be tested.

[0088] The two images to be tested are compared with the two standard device images in the previous embodiment, which is equivalent to comparing them in completely opposite polarity states from the perspective of the device to be tested. The directional defect detection results are obtained by comparing the obvious differences that must exist between the original image to be tested and the derived image to be tested and the standard device image.

[0089] The above two detection methods are equivalent to that one of the standard device image and the device under test image requires two images, front and back. For the standard device image, at least one is the standard device image in the positive direction. In both methods, the device under test image can be accurately compared with the standard device image in the positive direction to confirm the positive and negative states.

[0090] At the macro-processing level, when starting an inspection task, the scheme of using positive and negative standard device images or positive and negative images of the device to be tested can be confirmed based on the inspection task information input by the user. The user needs to enter specific information about the image volume. For inspection tasks where the total volume of the device to be tested images is relatively large and the total volume of the standard device images is relatively small, you can choose to use positive and negative standard device images (i.e., positive standard device images and reverse standard device images) for inspection; for inspection tasks where the total volume of the device to be tested images is relatively small and the total volume of the standard device images is relatively large, you can choose to use positive and negative images of the device to be tested (i.e., original images to be tested and derived images to be tested) for inspection, avoiding doubling the processing of larger volumes of data, thereby reducing the consumption of computing resources, alleviating the processing burden of AOI inspection equipment, and improving the overall processing speed.

[0091] When confirming the test results, the identification results of the device under test corresponding to the image of the device under test and the standard identification results of the standard device image can also be directly matched. When the similarity between the two reaches the similarity threshold set according to experience, the matching results can also be regarded as consistent. The comprehensive similarity is a comprehensive evaluation of the similarity between the image features corresponding to each limited installation direction of the two images, which can be specifically the weighted sum of the similarities of each limited installation direction.

[0092] In general, various matching result confirmation methods can be provided to users for flexible selection according to specific detection needs.

[0093] It should be noted that in a circuit board, some diodes may not have any requirements for the installation direction at both ends, that is, installing in any direction is the correct installation method, and no direction judgment is required. For such diodes, when calibrating the standard circuit board image, they are not calibrated as diodes, and accordingly there is no need to detect them through the direction feature extraction model.

[0094] As mentioned above, each training sample is an image after removing the background. Correspondingly, during detection, the standard device image is a device image after removing the background, which can effectively reduce the interference in the process of extracting the installation direction related features, thereby ensuring the accuracy of the detection results and improving the detection accuracy. Specifically, when using the opencv template matching algorithm in positioning, the background can be removed at the same time. Of course, in the specific implementation process, the image of the device to be tested with the background retained can also be directly extracted, and the corresponding detection can also be achieved.

[0095] It should be understood that confirmation of qualified inspection in the embodiment of the present application only refers to qualified inspection within the inspection target defined in the embodiment of the present application, and does not represent confirmation of the inspection results of the overall design and production quality of the PCB board.

[0096] In the above-mentioned electronic device defect detection method, during the automatic optical detection process, when it is confirmed that the device to be detected is an electronic device with a limited installation direction, the image of the device to be detected and the corresponding standard device image of the device to be detected are respectively extracted, including image features corresponding to each limited installation direction, and according to the similarity of the corresponding image features identified between the two, it is confirmed whether the installation directions of the electronic devices in the image of the device to be detected and the image of the standard device are consistent, thereby completing the defect detection on the installation direction. Based only on the comparison result of the image features of the limited installation direction, it is judged whether the image of the device to be detected has defects in the installation direction. For polar electronic devices in the image of the device to be detected, it is not necessary to adapt to the specific circuit board and electronic device type to obtain a clear direction and then confirm whether the direction is correct. The detection of polar electronic devices has good versatility, and significantly improves the robustness of realizing the detection of various polar electronic devices.

[0097] The present application uses a feature extraction model to extract image features and then perform directional defect detection. Compared with the method of obtaining similarity using a matching algorithm, the detection using a matching algorithm requires each time to clearly identify the direction according to the detection object, and then detect the device to be detected based on the direction identifier to obtain a clear direction. When the number of detections is large, the operation of pre-clarifying the direction identifier requires the user to pay a lot of and meticulous labor. In the present application, it is only necessary to first use the model to extract the directional features on the basis of confirming the standard device image corresponding to the device to be detected, and then (using a similarity calculation method, such as cosine similarity) calculate the similarity between the extracted features, and confirm whether the features in the standard device image are consistent based on the similarity to make a complete judgment, reducing the delicate operation of the standard image, and there is no need to adapt to the specific circuit board and electronic device type to obtain a clear direction and then confirm whether the direction is correct. The detection of polar electronic devices has good versatility, and the direction detection can be completed with a standard device image, which significantly improves the robustness of realizing the detection of various polar electronic devices.

[0098] Figure 2 This is a schematic diagram of the structure of an electronic device defect detection device provided in an embodiment of the present application. Figure 2 As shown, the electronic device defect detection device includes a type image confirmation unit 210, a feature extraction unit 220, a similarity calculation unit 230 and a result determination unit 240.

[0099] Among them, the type image confirmation unit 210 is used to confirm that the device to be tested is an electronic device with a limited installation direction, and obtain the standard device image corresponding to the device to be tested and the device to be tested image corresponding to the device to be tested; the feature extraction unit 220 is used to input the device to be tested image and the standard device image into the direction feature extraction model respectively, and correspondingly extract the image features of the device to be tested and the image features of the standard device, the image features include image features corresponding to each limited installation direction, and the direction feature extraction model is obtained by performing direction-based center clustering training on the sample device image; the similarity calculation unit 230 is used to calculate the similarity between the image features of the device to be tested and the image features of the standard device in each limited installation direction, and obtain at least two similarities; the result determination unit 240 is used to determine the defect detection result of the device to be tested according to the relative size of the similarity.

[0100] The result determination unit 240 includes:

[0101] The feature matching module is used to match the image features of the device to be tested with the corresponding image features of the standard device. When the matching results are consistent, it is confirmed that the installation direction detection of the device to be tested is qualified; when the matching results are inconsistent, it is confirmed that the installation direction detection of the device to be tested is unqualified.

[0102] Among them, the standard device image and the device image to be tested are both device images after background removal.

[0103] Among them, the directional feature extraction model is pre-trained in the following way:

[0104] Acquire multiple binary images as third training samples, each binary image being a forward device image and a reverse device image of a device;

[0105] The third neural network model is trained by center clustering through the third training sample to obtain a directional feature extraction model. The constraint condition of the center clustering training is that device images with the same direction are clustered around the same direction center, and device images with different directions are scattered in their respective direction centers.

[0106] The standard device image includes a forward direction standard device image and a reverse direction standard device image;

[0107] Accordingly, the result determination unit 240 includes:

[0108] The first confirmation module is used to confirm that the installation direction of the device to be tested is qualified when the forward comprehensive similarity between the image feature of the device to be tested and the forward direction image feature is greater than the reverse comprehensive similarity between the image feature of the device to be tested and the reverse direction image feature, the forward direction image feature is the standard device image feature corresponding to the forward direction standard device image, and the reverse direction image feature is the standard device image feature corresponding to the reverse direction standard device image.

[0109] The standard device image is a standard device image in the positive direction; the device image to be tested includes an original image to be tested and a derived image to be tested obtained by rotating the original image to be tested by 180°;

[0110] Accordingly, the result determination unit 240 includes:

[0111] The second confirmation module is used to confirm that the installation direction of the device to be tested is qualified when the original comprehensive similarity between the original image feature and the standard device image feature is greater than the derived comprehensive similarity between the derived image feature and the standard device image feature, the original image feature is the image feature of the device to be tested corresponding to the original image to be tested, and the derived image feature is the image feature of the device to be tested corresponding to the derived image to be tested.

[0112] The electronic device defect detection apparatus provided in the embodiment of the present application is included in an electronic device and can be used to execute the corresponding electronic device defect detection method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0113] It is worth noting that in the above-mentioned embodiment of the electronic device defect detection apparatus, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0114] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device includes a processor 310 and a memory 320. The electronic device may also include an input device 330, an output device 340, and a communication device 350. The number of processors 310 in the electronic device may be one or more. Figure 3 A processor 310 is taken as an example; the processor 310, the memory 320, the input device 330, the output device 340 and the communication device 350 in the electronic device can be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.

[0115] The memory 320 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the electronic device defect detection method in the embodiment of the present application. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 320, that is, implements the above-mentioned electronic device defect detection method.

[0116] The memory 320 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0117] The input device 330 may be used to receive network configuration information. The output device 340 may include a display device such as a display screen.

[0118] The above-mentioned electronic device can be used to execute any electronic device defect detection method and has corresponding functions and beneficial effects.

[0119] An embodiment of the present invention also provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to perform relevant operations in the electronic device defect detection method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0120] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0121] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0123] Computer readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0124] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0125] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for detecting defects in electronic devices, characterized in that: include: When confirming that the device to be tested is an electronic device with a limited installation direction, acquiring a standard device image and a device to be tested image corresponding to the device to be tested; Input the device under test image and the standard device image into a directional feature extraction model respectively, and extract the image features of the device under test and the image features of the standard device accordingly, wherein the image features include image features corresponding to each limited installation direction, and the directional feature extraction model is obtained by performing direction-based center clustering training on sample device images; Calculating the similarity between the image features of the device under test and the image features of the standard device corresponding to the image features of the respective limited installation directions, and obtaining at least two similarities; The defect detection result of the device to be detected is determined according to the relative size of the similarity.

2. The method according to claim 1, characterized in that Determining the defect detection result of the device to be detected according to the relative size of the similarity includes: The image features of the device under test are matched with the corresponding image features of the standard device. When the matching results are consistent, it is confirmed that the installation direction detection of the device under test is qualified; when the matching results are inconsistent, it is confirmed that the installation direction detection of the device under test is unqualified. 3 . According to the method of claim 1 , the standard device image and the device under test image are both device images after background removal.

4. The method according to any one of claims 1 to 3, characterized in that: The directional feature extraction model is pre-trained in the following way: Acquire multiple binary images as third training samples, each binary image being a forward device image and a reverse device image of a device; The third training sample is used to perform center clustering training on the preset third neural network model to obtain a directional feature extraction model, and the constraint condition of the center clustering training is that device images with the same direction are clustered around the same direction center, and device images with different directions are scattered in their respective direction centers.

5. The method according to any one of claims 1 to 3, characterized in that: The standard device image includes a forward direction standard device image and a reverse direction standard device image; Correspondingly, determining the defect detection result of the device to be detected according to the relative size of the similarity includes: When the forward comprehensive similarity between the image feature of the device to be tested and the forward direction image feature is greater than the reverse comprehensive similarity between the image feature of the device to be tested and the reverse direction image feature, it is confirmed that the installation direction of the device to be tested is qualified, the forward direction image feature is the standard device image feature corresponding to the forward direction standard device image, and the reverse direction image feature is the standard device image feature corresponding to the reverse direction standard device image.

6. The method according to any one of claims 1 to 3, characterized in that: The standard device image is a standard device image in the positive direction; the device image to be tested includes an original image to be tested and a derived image to be tested obtained by rotating the original image to be tested by 180°; Correspondingly, determining the defect detection result of the device to be detected according to the relative size of the similarity includes: When the original comprehensive similarity between the original image feature and the image feature of the standard device is greater than the derived comprehensive similarity between the derived image feature and the image feature of the standard device, it is confirmed that the installation direction of the device to be tested is qualified. The original image feature is the image feature of the device to be tested corresponding to the original image to be tested, and the derived image feature is the image feature of the device to be tested corresponding to the derived image to be tested.

7. The method according to any one of claims 1 to 3, characterized in that: Whether the device to be detected is an electronic device with a limited installation direction is confirmed by a device name input by a user or a recognition result of a pre-trained device type recognition model.

8. An electronic device defect detection device, characterized in that: include: A type image confirmation unit, used to confirm that the device to be detected is an electronic device with a limited installation direction, and to obtain a device to be detected image corresponding to the device to be detected in the standard device image; A feature extraction unit, used to input the image of the device under test and the image of the standard device into a directional feature extraction model, respectively, to correspondingly extract features of the image of the device under test and the image of the standard device, wherein the image features include image features corresponding to each limited installation direction, and the directional feature extraction model is obtained by performing direction-based center clustering training on sample device images; A result determination unit is used to calculate the similarity between the image features of the device under test and the image features of the standard device corresponding to the image features of the specified installation directions to obtain at least two similarities; The result determination unit is used to determine the defect detection result of the device to be detected according to the relative size of the similarity.

9. An electronic device, characterized in that include: one or more processors; a memory for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the electronic device defect detection method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the electronic device defect detection method as described in any one of claims 1 to 7 is implemented.

Images