Optical inspection target classification method and electronic device

Abstract

The present application discloses an optical inspection classification method and an electronic device. The electronic device comprises: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory is configured for storing a computer instruction, and the processor is configured for executing the computer instruction so that the electronic device performs the following: acquiring a measured image of an inspection target and annotation information; using a first model to extract features of both the measured image and a standard image under different inspection tasks, so as to determine inspection results corresponding to the different inspection tasks, and determining a classification of a first re-inspection according to the inspection results; and if the classification of the first re-inspection indicates that the inspection target is acceptable, using a second model to perform feature extraction on both the measured image and the standard image and then calculate a feature difference, performing enhanced annotation on the feature difference of an abnormal region, and determining a classification of a second re-inspection on the basis of the feature difference of each region. This results in lower false determination rates, improved inspection accuracy, and greater generation efficiency.

Description

Optical detection object classification method and electronic device

[0001] Cross-reference to related applications

[0002] The present application claims priority to Chinese Patent Application No. 202410846982.X, filed on June 27, 2024, and No. 202411003120.7, filed on July 25, 2024, the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of optical vision detection, in particular to an optical detection object classification method and an electronic device. BACKGROUND

[0004] In a highly automated production line, any minor defect of a printed circuit board (PCB) can become a fuse of product failure, triggering a chain reaction and causing more serious quality problems. Therefore, when detecting defects of a PCB, it is crucial to maintain an extremely low false negative rate. Although the automatic optic inspection (AOI) equipment in the related art has the capability of defect detection, it mainly relies on image comparison in the dimensions of color, gradient, and contour, etc., and determines defects by setting an empirical threshold. This strategy is often relatively single, and a high threshold set to reduce the false negative rate may cause normal products to be misjudged. In particular, when a production line is replaced with a new material, this image matching problem is more prominent. In addition, the high threshold setting also increases the workload of manual re-inspection and reduces the efficiency of defect detection. SUMMARY

[0005] The optical detection object classification method and the electronic device provided by some embodiments of the present application are used to solve the problem that the normal products may be misjudged due to the reduction of the false negative rate in the prior art, which increases the workload of manual re-inspection. In some embodiments of the present application, on the basis of the optical detection preliminary inspection result of the AOI equipment, the first model and the second model are used to perform at least two times of re-inspection on the NG (Not Good) object to be inspected, wherein the first model of the first-time re-inspection can efficiently realize the simultaneous detection of different detection tasks of the object to be inspected, so as to quickly and timely screen out normal objects to be inspected, thereby reducing the workload of manual re-inspection, and in order to further reduce the misjudgment rate; the second model of the second-time re-inspection adopts the way of feature difference and strengthens the processing of the non-normal area, which can further more accurately determine the non-normal area; in this way, the real image and the standard image of the object to be inspected can be efficiently and accurately re-inspected and analyzed, thereby improving the accuracy of defect detection, reducing misjudgment, and improving production efficiency.

[0006] In a first aspect, an electronic device in some embodiments of the present application includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory is configured to store computer instructions, and the processor is configured to execute the computer instructions to cause the electronic device to perform: obtaining a measured image of a to-be-inspected object and annotation information, the annotation information including an annotated abnormal region, the abnormal region being determined based on the measured image and a standard image of the to-be-inspected object, a region in the measured image representing that the to-be-inspected object is unqualified; extracting features of the measured image and the standard image under different detection tasks respectively by using a first model, determining detection results corresponding to different detection tasks respectively based on the extracted features, and determining a first-time re-inspection classification according to the detection results; if the first-time re-inspection classification represents that the to-be-inspected object is qualified, performing feature extraction on the measured image and the standard image respectively by using a second model and obtaining feature differences, performing reinforced annotation on feature differences of the abnormal region, and determining a second-time re-inspection classification based on feature differences of each region.

[0007] In a second aspect, an optical detection object classification method in some embodiments of the present application includes: obtaining a measured image of a to-be-inspected object and annotation information, the annotation information including an annotated abnormal region, the abnormal region being determined based on the measured image and a standard image of the to-be-inspected object, a region in the measured image representing that the to-be-inspected object is unqualified; extracting features of the measured image and the standard image under different detection tasks respectively by using a first model, determining detection results corresponding to different detection tasks respectively based on the extracted features, and determining a first-time re-inspection classification according to the detection results; if the first-time re-inspection classification represents that the to-be-inspected object is qualified, performing feature extraction on the measured image and the standard image respectively by using a second model and obtaining feature differences, performing reinforced annotation on feature differences of the abnormal region, and determining a second-time re-inspection classification based on feature differences of each region.

[0008] In a third aspect, a computer-readable nonvolatile storage medium in some embodiments of the present application stores a computer program, and the computer program is configured to cause a computer to execute the method of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0009] FIG. 1 is a schematic diagram of processing logic for product classification using an AOI device in the related art;

[0010] FIG. 2 is a schematic diagram of processing logic for increased re-inspection according to some embodiments of the present application;

[0011] FIG. 3 is a schematic diagram of a flow of an optical detection object classification method according to some embodiments of the present application;

[0012] FIG. 4 is a schematic diagram of adding a third review classification according to some embodiments of the present application;

[0013] FIG. 5 is a schematic diagram of the overall process of the optical detection object classification method according to some embodiments of the present application;

[0014] FIG. 6 is a schematic diagram of the detection content of multi-task detection of elements according to some embodiments of the present application;

[0015] FIG. 7 is a schematic diagram of the corresponding processing flow when multi-task detection recognition is a silk screen according to some embodiments of the present application;

[0016] FIG. 8 is a schematic diagram of the overall process of classification by the first model using a multi-task detection algorithm according to some embodiments of the present application;

[0017] FIG. 9 is a schematic diagram of the training process of the first model according to some embodiments of the present application;

[0018] FIG. 10 is a flowchart of the reinforcement of feature differences by an absolute detection algorithm according to some embodiments of the present application;

[0019] FIG. 11 is a flowchart of the classification by an absolute detection algorithm according to some embodiments of the present application;

[0020] FIG. 12 is a schematic diagram of the overall process of classification by the second model using an absolute detection algorithm according to some embodiments of the present application;

[0021] FIG. 13 is a flowchart of the feature extraction process of the second model according to some embodiments of the present application;

[0022] FIG. 14 is a schematic diagram of the processing flow of CBS according to some embodiments of the present application;

[0023] FIG. 15 is a schematic diagram of the processing flow of CSPRes_X according to some embodiments of the present application;

[0024] FIG. 16 is a schematic diagram of the processing flow of ResSPP according to some embodiments of the present application;

[0025] FIG. 17 is a schematic diagram of the training process of the second model according to some embodiments of the present application;

[0026] FIG. 18 is a schematic diagram of the training process of the third model according to some embodiments of the present application;

[0027] FIG. 19 is a schematic diagram of the process of another optical detection object classification method according to some embodiments of the present application;

[0028] FIG. 20 is a schematic diagram of the process of a third optical detection object classification method according to some embodiments of the present application;

[0029] FIG. 21 is a process diagram of a fourth optical detection object classification method according to some embodiments of the present application;

[0030] FIG. 22 is a process diagram of a fifth optical detection object classification method according to some embodiments of the present application;

[0031] FIG. 23 is a process diagram of a sixth optical detection object classification method according to some embodiments of the present application;

[0032] FIG. 24 is a process diagram of a seventh optical detection object classification method according to some embodiments of the present application;

[0033] FIG. 25 is another processing logic diagram for increasing review according to some embodiments of the present application;

[0034] FIG. 26 is a diagram of a blurred image and a clear image according to some embodiments of the present application;

[0035] FIG. 27A is a diagram of a heat map visualization according to some embodiments of the present application;

[0036] FIG. 27B is a diagram of a heat map visualization according to some embodiments of the present application;

[0037] FIG. 27C is a diagram of a heat map visualization according to some embodiments of the present application;

[0038] FIG. 27D is a diagram of a heat map visualization according to some embodiments of the present application;

[0039] FIG. 28 is a diagram of horizontal and vertical sampling lines in a heat map according to some embodiments of the present application;

[0040] FIG. 29 is a diagram of fitting horizontal group data of a measured blurred image to obtain a measured horizontal fitting curve according to some embodiments of the present application;

[0041] FIG. 30 is a diagram of a position relationship between a measured horizontal fitting curve and a standard horizontal fitting curve according to some embodiments of the present application;

[0042] FIG. 31 is a diagram of an electronic device structure according to some embodiments of the present application; DETAILED DESCRIPTION

[0043] To further illustrate the schemes provided by some embodiments of the present application, the following will provide a detailed description in conjunction with the accompanying drawings and specific embodiments. Although some embodiments of the present application provide the following method operation steps as shown in the embodiments or drawings, more or less operation steps can be included in the method based on conventional or non-creative labor. The execution order of the steps is not limited to the execution order provided by some embodiments of the present application in the logical sense that there is no necessary causality between the steps. The method can be executed in sequence or in parallel when the actual processing process or the control device is executed according to the method order shown in the embodiments or drawings.

[0044] The processing logic of the optical detection object classification method in the related art is shown in FIG. 1. The AOI device uses optical detection technology to perform optical detection on the product to be detected (step 101), such as collecting the measured image of the product to be detected, and comparing it with the standard image, and determining the product to be detected as a normal classification, i.e. determining the product to be detected as a qualified (OK) product (step 102), which does not exist defects, and flows to the next process (step 103), or determining the product to be detected as an abnormal classification, i.e. determining the product to be detected as a unqualified (No Good, NG) product (step 104), which exists defects, and performing manual rejudgment (step 105). Since the AOI device detection mainly relies on image comparison in the dimensions of color, gradient and contour, etc., the defects are judged by setting an empirical threshold. This strategy is often relatively single, and a high threshold set to reduce the missed detection rate may lead to misjudgment of normal products. In addition, the high threshold setting also increases the workload of manual reinspection and reduces the efficiency.

[0045] In view of this, some embodiments of the present application provide an optical detection object classification method configured to perform defect detection and determine the classification of a to-be-detected object, which can be but is not limited to a product on a production line, and a possible form is a component (a component device), as shown in FIG. 2. Based on the optical detection performed by the AOI device and the comparison with the standard image to obtain the preliminary detection result, the optical detection of the to-be-detected object is performed by the AOI device (step 201). If the to-be-detected object is determined to be OK (step 202), it is transferred to the next process (step 203). If the to-be-detected object is determined to be NG (step 204), the AI detection technology is used to perform at least two re-inspections on the to-be-detected object determined to be NG (step 205), and the to-be-detected object determined to be OK by the AI algorithm (step 206) and the to-be-detected object determined to be NG by the AI algorithm (step 207) are obtained. For the to-be-detected product determined to be OK by the AOI device and the to-be-detected object determined to be OK after the re-inspection by the AI algorithm, it is transferred to the next process (step 203). For the to-be-detected object determined to be NG by the AI algorithm, manual re-inspection is performed (step 208). In this way, efficient and accurate re-inspection analysis of the actual measurement image and the standard image of the to-be-detected product can be realized, thereby improving the defect detection accuracy, reducing misjudgment, and improving the efficiency of defect detection.

[0046] FIG. 3 is a flowchart of an optical detection object classification method according to some embodiments of the present application, which includes but is not limited to the following steps:

[0047] In step 301, an actual measurement image of a to-be-detected object and annotation information are obtained, and the annotation information includes an annotated abnormal area. The abnormal area is determined based on the actual measurement image and a standard image of the to-be-detected object, and is a region in the actual measurement image that represents the to-be-detected object as unqualified.

[0048] The optical detection object classification method in the present application is applied to an electronic device, which can be a PC, a computer terminal device, a server, or an AOI device. It can be understood that the specific form of the electronic device is not limited in the present application.

[0049] In some embodiments of the present application, an electronic device such as an AOI device obtains a measured image of a to-be-inspected object by using optical detection technology. The present application pre-stores a standard image of the to-be-inspected object. Taking a component on a printed circuit board (PCBA) as an example, in the changeover preparation stage, the corresponding operator intervenes to set some parameters related to the new product, and a high-precision industrial scanning camera is used to collect an image on a normal PCBA as a standard image. In the actual production process, a measured image of the to-be-inspected object on the PCBA produced in real time is collected. The changeover refers to the operation performed due to the change of processing content and product.

[0050] The AOI device compares the measured image with the standard image. For details, refer to related technologies. Through comparison, a preliminary inspection classification result and labeling information can be obtained. The labeling information can include, but is not limited to, an abnormal area, i.e., an NG frame is used to label the abnormal area. The abnormal area can be an area with defects. In some embodiments, the labeling information can also include other information obtained through image comparison.

[0051] In step 302, a first model is used to extract features of the measured image and the standard image in different detection tasks, respectively. Based on the extracted features, detection results corresponding to different detection tasks are determined, respectively, and a first-time re-inspection classification is determined according to the detection results.

[0052] In some embodiments of the present application, the first model can be constructed by the aforementioned AI detection technology. The model can be based on the input measured image and the corresponding standard image, and can simultaneously implement multi-task detection. The different detection tasks are used to implement different purpose detection, and at least include defect type monitoring. For component products, it can include, but is not limited to, ten major categories of defects such as missing parts, offset, missed detection, and polar reverse (i.e., the polarity of the component is opposite to the standard polarity). In addition, tasks such as silk screen position and silk screen direction can also be detected.

[0053] In some embodiments of the present application, the first model can implement a multi-task detection algorithm. By sending the measured image and the standard image of the to-be-inspected object into the first model, the first model can simultaneously implement multi-task detection through the algorithm processing logic of the first model, obtain the detection results of each detection task, and determine whether the to-be-inspected object is normal according to the detection results. The detection tasks can include, but are not limited to, different types of defect detection tasks.

[0054] In some embodiments of the present application, the first model is used to implement the first re-inspection, which can solve about 95% of defects. The objects to be inspected that are re-inspected as abnormal (i.e., unqualified) by the first model are subjected to manual re-inspection, and the processing of the next measured image is performed. The objects to be inspected that are re-inspected as normal (i.e., qualified) are subjected to the second model for the second re-inspection, and the processing of the next measured image is performed.

[0055] In step 303, if the classification of the first re-inspection indicates that the object to be inspected is qualified, the second model is used to extract features from the measured image and the standard image respectively and calculate feature differences, the feature differences of abnormal regions are enhanced, and the classification of the second re-inspection is determined based on the feature differences of the regions.

[0056] According to the results of the first re-inspection, the measured image and the standard image that are re-inspected as normal in the first re-inspection are input to the second model for further detection. In some embodiments of the present application, the second model is a model constructed by the AI detection technology as described above, which is used to extract features from the measured image and the corresponding standard image and calculate feature differences. In order to more accurately determine whether the abnormal region that is re-inspected as NG in the first inspection indeed has a problem, in some embodiments of the present application, the feature differences of the abnormal regions are enhanced, and the classification of the second re-inspection is determined based on the feature differences of the regions. This embodiment refers to the above algorithm process as an absolute detection algorithm, and the network structure thereof is shown in FIG. 12. If the measured image re-inspected in the second re-inspection is classified as NG, the detection result is returned, the object to be inspected is subjected to manual re-inspection, the detection of the current measured image is ended, and the next measured image is continuously re-inspected.

[0057] In some embodiments of the present application, the second model is used to implement the second re-inspection, which can solve about 4.56% of defects.

[0058] In some embodiments of the present application, at least two re-inspections are performed using AI detection technology. The first model of the first re-inspection adopts a multi-task detection algorithm, which can efficiently implement simultaneous detection of multiple tasks of the object to be inspected, thereby quickly and timely screening out normal objects to be inspected and reducing the workload of manual re-inspection. In order to further reduce the misjudgment rate, the second model of the second re-inspection adopts an absolute detection algorithm based on feature differences and enhanced processing of abnormal regions, which can further more accurately determine abnormal regions. In this way, efficient and accurate re-inspection and analysis of the measured image and the standard image pair of the object to be inspected can be achieved, thereby improving the defect detection effect, reducing misjudgment, and improving production efficiency.

[0059] Some embodiments of the present application are directed to special objects to be inspected, which may have complex and diverse error forms and cannot be classified by a small model. In order to further reduce the misjudgment rate, the annotation information further includes related information for assisting in determining the question statement, as shown in FIG. 4. The method further includes but is not limited to the following:

[0060] Step 401, determining that the classification of the second re-inspection indicates that the object to be inspected is qualified, and the type of the object to be inspected is a specified type.

[0061] After a measured image is re-inspected by a second model and is classified as abnormal, the image enters the step of manual re-inspection. If the second re-inspection is classified as normal, in order to further implement the re-inspection of special objects to be inspected (such as a specified type), a third re-inspection is determined.

[0062] Step 402, determining a question statement matched with the annotation information based on a pre-defined question statement affecting the classification result.

[0063] For the case that the error forms may be complex and diverse, causing the determination criteria of the object to be inspected to be complex, different question statements can be pre-defined for various cases that may affect the determination of the special object to be inspected. For each question statement, a corresponding answer to the question statement can be determined by using a model, so that the classification can be determined according to the answer. Different question statements can be associated with corresponding keywords. By matching the annotation information with the keywords, a question statement associated with a successfully matched keyword is determined as the matched question statement.

[0064] Step 403, inputting the measured image, the standard image, and the matched question statement into a third model to obtain a classification of the third re-inspection.

[0065] In some embodiments of the present application, the measured image, the corresponding standard image, and the matched question statement are input into a third model. The third model can be a visual language model, such as a large visual language model (Large Vision Language Model, LVLM) or an industrial anomaly detection AnomalyGPT large model. In this way, the classification and determination of the special object to be inspected can be realized by using the good understanding ability of the third model.

[0066] In some possible embodiments, the third model is taken as an industrial anomaly detection AnomalyGPT large model. AnomalyGPT is an industrial anomaly detection model based on a visual large model (LVLM), which includes an image decoder and a prompt learner. The image decoder is to align the features extracted by an image encoder with text features representing normal and abnormal semantics through a linear projection layer, to generate an attention map of abnormal region segmentation, to guide the large model to focus on the abnormal region in the image. The prompt learner includes a 6-layer convolutional neural network, which can convert the attention map output by the image decoder into a prompt embedding vector that can be understood by the large model. The object to be detected is an element, and the specified type is a silk screen type. For elements of the silk screen type, because there are many manufacturers and many element models, the judgment standard of the polarity point is complex, and the traditional small model cannot understand the complex language logic. For this situation, the polarity point of part of the silk screen element is judged by means of the good understanding ability of the visual language model, so as to judge whether there is a polarity inversion and a wrong material (i.e. using wrong elements, auxiliary materials, etc.).

[0067] In some embodiments of the present application, the above multi-task detection can include detection of silk screen elements and silk screen direction. After the element is determined to be normal through secondary re-inspection, it can be determined whether it is a silk screen type element according to the multi-task detection result. If it is not a silk screen type element, the reasoning of the element ends, and the element in the actual measurement image is OK. If it is a silk screen type element, the polarity point of the current silk screen type element is further reasoned by the AnomalyGPT large model algorithm, the direction of the polarity point and the character content are judged, and the above results of the standard image are compared to determine whether the actual measurement image is OK or NG. In some embodiments of the present application, the AnomalyGPT large model algorithm used in the third re-inspection can solve about 0.44% defects that cannot be solved by the remaining small models.

[0068] Figure 5 is a flowchart of the overall process of the optical detection object classification method according to some embodiments of the present application, taking components as examples of the objects to be inspected, and the specific process includes but is not limited to: after initial inspection by the AOI equipment, obtaining the initial inspection classification result and the annotation file, which can be stored in xml format, denoted as xml file, and only xml file will be used as an example in the following description, but it should be noted that the aforementioned annotation file is not limited to xml file, but can also be in other forms, which is not limited by the present application. The xml file can include annotation information, and the measured image and the annotated image are input into the first model, and the multi-task detection algorithm of the first model is used to determine whether the measured image is OK or NG (step 501), if the measured image is NG, the classification result of NG is returned for manual re-inspection (step 502); if the measured image is OK, the absolute detection algorithm of the second model is used to determine whether the measured image is OK or NG (step 503), if the measured image is NG, the classification result of NG is returned for manual re-inspection (step 504). If the measured image is OK, further determine whether it is a silk screen type component (step 505), if not (N), return the classification result of OK (step 506) and proceed to the next process; if it is a silk screen type component, use the large model algorithm to identify the polarity point of the measured image and determine whether it is OK or NG, and return the detection result (step 507).

[0069] The following takes components as examples of the objects to be inspected to give the corresponding implementation of the three re-inspections.

[0070] 1) The first re-inspection using a multi-task detection algorithm

[0071] When the object to be inspected is a component, the different detection tasks include some or all of the identification of the existing component, the component type, the component defect type, the silk lead position, and the silk screen direction, as shown in Figure 6, each component involves multiple possible defect types, such as defect a, defect b, …, defect l, etc.; and there can be silk screen, if there is silk screen, it can be further divided into eight directions, such as the directions shown in Figure 6: up, down, left, right, left up, right up, left down, and right down.

[0072] If the multi-task detection algorithm detects the presence of silk screen, the silk screen can be segmented in the following way, and the classification is determined according to the silk screen direction and content, as shown in Figure 7, which includes but is not limited to:

[0073] Step 701, when the detection result includes the position and the silk screen direction, the silk screen position is segmented, and the silk screen position is rotated to the correct position according to the silk screen direction;

[0074] Step 702, identify the content of the silk screen in the rotated silk screen position, determine whether the direction of the silk screen in the standard image is the same, and whether the content of the silk screen is mutually replaced.

[0075] If the direction is different or the content of the silk screen is not mutually replaced, the classification of NG is outputted.

[0076] In some embodiments, the specific overall process is shown in FIG. 8. The measured image and the standard image are inputted into the first model, the image features are extracted, and the multi-task detection algorithm is used to realize the detection of part or all of the element, the defect type, the silk screen direction classification, and the silk screen position (FIG. 8 shows a case of detecting the above-mentioned multiple classifications, denoted as step 801). If the detection result is determined to be NG, the AOI error unqualified rectangle (NgBox) frame, i.e. the abnormal area, is outputted (step 802). If the measured image is detected to have the silk screen direction and the silk screen position, the silk screen position is segmented, and the silk screen position is rotated to be upright according to the silk screen direction (step 803). The text recognition algorithm such as the text detection and recognition (paddleOCR) algorithm is used to recognize the content of the silk screen in the rotated silk screen position and output (step 804). Then, whether the direction and the content of the silk screen are mutually replaced is analyzed to determine the classification result, and the classification result is outputted (step 805). If the direction is different or the content of the silk screen is not mutually replaced, the classification result of NG is outputted, otherwise, the classification result of OK is outputted. It should be noted that the specific type of the text recognition algorithm is not limited in the present application.

[0077] As shown in FIG. 9, the first model of some embodiments of the present application can be trained in the following way:

[0078] Step 901, obtaining a sample measured image, a sample standard image, and a sample label of an object, wherein the sample label includes the detection results of different detection tasks of the object and the sample classification;

[0079] In some embodiments of the present application, a plurality of detection tasks are defined according to the object to be detected, and the detection results and the classification labels are labeled for the object with known multi-task detection results of the measured image.

[0080] Step 902, inputting the sample measured image and the sample standard image into the first model to be trained, extracting the features corresponding to different detection tasks by the first model to be trained, adjusting the model parameters for the purpose of obtaining the labeled detection results of different detection tasks and outputting the labeled sample classification, and the specific model parameter adjustment method can adopt related technologies, which will not be described in detail here.

[0081] 2) Second re-inspection using absolute detection algorithm

[0082] In some embodiments of the present application, the second model is used to extract the features of the measured image and the standard image, and then the differences are obtained, and the feature differences of the abnormal regions are emphasized and labeled, and the classification of the second review is determined based on the feature differences of each region.

[0083] As shown in FIG. 10, in some embodiments of the present application, the feature differences of the abnormal regions are emphasized and labeled, including but not limited to:

[0084] Step 1001, difference is obtained between the features extracted from the measured image and the standard image according to regions, to obtain the feature differences of different regions;

[0085] Step 1002, based on the first weight, the feature differences of the normal regions are determined, and based on the second weight, the feature differences of the abnormal regions are determined, and the second weight is greater than the first weight.

[0086] In some embodiments, according to the labeling information obtained by the AOI equipment initial review, the region labeled with the NG frame is determined as the abnormal region, and the other regions are determined as the normal regions. When step 1002 is executed, the product of the feature difference of the normal region and the first weight is taken as the feature difference of the normal region, and the product of the feature difference of the abnormal region and the second weight is taken as the feature difference of the abnormal region. For example, the first weight can be 1, and the second weight can be any positive number greater than 1.5. The reason why the second weight is set to be greater than the first weight is that the abnormal region is considered to be normal without defects in the first review, and if the second weight is set to be less than or equal to the first weight, the same classification result may be output in the second review, which weakens the emphasis detection of the feature differences in the second review. Therefore, in order to realize the emphasis labeling of the feature differences of the abnormal regions and emphasize the detection focus of the subsequent second review, i.e., to focus on detecting whether the abnormal region has defects, the second weight is set to be greater than the first weight, so as to improve the review accuracy.

[0087] As shown in FIG. 11, in some embodiments of the present application, the classification of the second review is determined based on the feature differences of each region, including but not limited to:

[0088] Step 1101, channel attention processing is performed on the feature differences of each region;

[0089] Step 1102, convolution operation is performed on the features after the channel attention processing, and then adaptive average pooling processing is performed after the activation function operation.

[0090] Step 1103, one-dimensional features obtained after the adaptive average pooling processing are linearly mapped at least twice to obtain the classification of the second review.

[0091] In some embodiments of the present application, the absolute detection algorithm network structure of the second model can be as shown in FIG. 12. The standard image and the actual measurement image are sent into the same parameter feature extraction network model, the high-dimensional features of the standard image and the actual measurement image are extracted through the feature extraction network model, and the difference comparison (Diff) module is used to compare the features of the two images in the high-dimensional space with different weights according to the Ngbox frame (i.e. unqualified rectangular frame, abnormal area) generated when the AOI equipment reports an error. A higher weight λ can be given to the difference calculation in the Ngbox frame area, which can be but not limited to λ>1.5. Then the feature difference result is used for subsequent processing. In some embodiments, the result of the above diff module is first subjected to channel attention (SE) module for feature optimization, which enhances useful input features and suppresses useless input features to obtain optimized features. The optimized features are subjected to convolution operation through a convolution layer, and then the convolution operation result is input into the Silu activation function for operation. The activation function operation result is subjected to adaptive average pooling (Adaptive-avgpool) processing, and then returned to one-dimensional features through the flattening module. Then, the two linear mapping Linear1 and Linear2 are used to regress the result into two categories of OK and NG, i.e. to obtain the second re-inspection classification.

[0092] As shown in FIG. 13, it is a process of feature extraction according to the feature extraction network model in some embodiments of the present application. The feature extraction network model is obtained by the feature extraction layer of the lightweight YOLO5, which belongs to shallow feature extraction, including but not limited to: a first convolution module, a residual module, a pooling module and a second convolution module. The first convolution module can include three levels of convolution layers, batch normalization layers and activation functions (Conv, Batch Normalization, SiLU, CBS); the residual module can include 6 levels of cross-stage feature fusion residual layers CSPRes_6; the pooling module can include ResSPP; and the second convolution module can include two levels of CBS. In some embodiments, after the input image is input into the feature extraction network model, the input image is subjected to three levels of CBS processing through the first convolution module, then subjected to cross-stage feature fusion residual layer CSPRes_6 processing through the residual module, then subjected to ResSPP processing through the pooling module, and then subjected to two levels of CBS processing through the second convolution module.

[0093] As shown in FIG. 14, the processing process of CBS is shown, wherein the batch normalization (batchNorm) layer is configured to perform normalization processing. As shown in FIG. 15, the processing process of CSPRes_x is shown, wherein CBS*x represents the CBS processing of x levels, and the connection (Concat) layer is configured to perform feature fusion. As shown in FIG. 16, the processing process of ResSPP is shown, wherein ResSPP can include three maximum pooling layers in parallel, a connection (Concat) layer, and a convolution layer.

[0094] As shown in FIG. 17, the second model in some embodiments of the present application can be trained in the following way:

[0095] In step 1701, a sample measured image, a sample standard image, and a sample label of an object are obtained, wherein the sample label includes a sample classification of the object.

[0096] In step 1702, the sample measured image and the sample standard image are input into the second model to be trained, and the sample measured image and the sample standard image are respectively subjected to feature extraction and feature difference calculation by the second model to be trained, and the feature difference of the abnormal region is strengthened and labeled. Based on the feature difference of each region, the model parameter adjustment is performed with the output labeled sample classification as the target.

[0097] 3) AnomalyGPT large model algorithm

[0098] In some embodiments of the present application, the third model is taken as an example of the AnomalyGPT large model, the object to be detected is a component, and the specified type is a silk screen type.

[0099] For a component with multiple silk screen points, the shape of the silk screen is not fixed, which can be a white point, a black point, a metal strip, a point of different sizes, an arrow, a notch, etc. The polarity point of the silk screen type can exist in various possible forms. As shown in Table 1, the parameters and conditions that may be involved in identifying the polarity point and the polarity point information corresponding to the polarity point under different packages and components are shown. In some embodiments of the present application, the related question sentences can be defined in advance according to various possible complex situations. During the third re-inspection, the related information of the object to be detected in the label information, such as the package and component information, is matched with the corresponding parameters and conditions that may be involved in identifying the polarity point, and the question sentences associated with the parameters and conditions that may be involved in identifying the polarity point are input into the large model.

[0100] Table 1

[0101] Among the QFP (Quad Flat Package), QFN (Quad Flat No-Lead Package), aluminum capacitor, BGA (Ball Grid Array), QFN and LED, the determination rules and forms of polarity points are different. For example, the judgment method of the polarity point of QFP can be that the special point in shape different from the other three points is the polarity point. By setting a prompt language, calling a large model, and asking the position of the special silk screen point among the four vertices, the polarity point of the element is determined, the polarity direction is determined, and whether the polarity reversal occurs is determined.

[0102] As shown in FIG. 18, the third model in some embodiments of the present application can be trained in the following way:

[0103] Step 1801, obtaining a sample measured image of a silk screen element, a sample standard image, a question sentence for asking parameters and conditions involved in identifying a silk screen polarity point, polarity point information of the silk screen, and labeling information of a classification result;

[0104] Step 1802, inputting the sample measured image of the silk screen element, the sample labeled image, and the question sentence for asking parameters and conditions involved in identifying the silk screen polarity point into the third model to be trained, and performing model training with the goal of outputting the polarity point information of the silk screen and the classification result.

[0105] On the basis of the above-mentioned embodiments, some embodiments of the present application further provide an electronic device, as shown in FIG. 31, which comprises at least one processor 311 and a memory 312 in communication connection with the at least one processor; wherein the memory 312 is configured to store computer instructions; the processor 311 is configured to execute the computer instructions to enable the electronic device to perform: obtaining a measured image of a to-be-inspected object and labeling information, the labeling information comprising a labeled abnormal area, the abnormal area being an area representing that the to-be-inspected object is unqualified in the measured image and being determined based on the measured image and a standard image of the to-be-inspected object; using a first model to extract features of the measured image and the standard image under different detection tasks respectively, determining detection results corresponding to different detection tasks based on the extracted features, and determining a first re-inspection classification according to the detection results; if the first re-inspection classification represents that the to-be-inspected object is qualified, using a second model to perform feature extraction on the measured image and the standard image respectively and to calculate feature differences, performing reinforced labeling on feature differences of the abnormal area, and determining a second re-inspection classification based on feature differences of each area.

[0106] In some possible embodiments, the labeling information further comprises relevant information for assisting in determining the question sentence, and the processor 311 is further configured to execute the computer instructions to cause the electronic device to perform: determining that the classification of the second re-inspection indicates that the to-be-inspected object is qualified, and that the type of the to-be-inspected object is a specified type; determining a question sentence matched with the labeling information based on a pre-defined question sentence affecting the classification result; and inputting the actually-measured image, the standard image, and the matched question sentence into a third model to obtain a classification of a third re-inspection.

[0107] In some possible embodiments, the third model is an AnomalyGPT large model, the to-be-inspected object is a component, and the specified type is a silk screen type; and the third model is trained in the following manner: obtaining sample actually-measured images, sample standard images, question sentences for questioning parameters and conditions involved in identifying silk screen polarity points, polarity point information of the silk screen, and labeling information of classification results of silk screen components; and inputting the sample actually-measured images, the sample standard images, and the question sentences for questioning parameters and conditions involved in identifying silk screen polarity points into the third model to be trained, to perform model training with the output of the polarity point information of the silk screen and the classification results as the target.

[0108] In some possible embodiments, the to-be-inspected object is a component, and the different detection tasks include some or all of identifying the existing component, the component type, the component defect type, the silk screen position, and the silk screen direction; and the processor 311 is further configured to execute the computer instructions to cause the electronic device to perform: when the detection result includes the position and the silk screen direction, segmenting out the silk screen position and rotating the silk screen position to be correct according to the silk screen direction; identifying the content of the silk screen in the rotated silk screen position, determining whether the direction of the silk screen in the rotated silk screen position is the same as that in the standard image, and whether the content of the silk screen in the rotated silk screen position is the same as that in the standard image; and outputting a classification indicating that the to-be-inspected object is unqualified if the direction is different or the content of the silk screen is not the same.

[0109] In some possible embodiments, when the features of the abnormal region are emphasized and labeled, the processor 311 is specifically configured to execute the computer instructions to cause the electronic device to perform: calculating the feature difference of the features extracted from the actually-measured image and the standard image according to regions to obtain the feature difference of different regions; determining the feature difference of the normal region based on a first weight, and determining the feature difference of the abnormal region based on a second weight, the second weight being greater than the first weight.

[0110] In some possible embodiments, when determining the classification of the second review based on the feature difference of each region, the processor 311 is specifically configured to execute the computer instructions to cause the electronic device to perform: channel attention processing on the feature difference of each region, convolution operation on the feature after the channel attention processing, and adaptive average pooling processing after the activation function operation; performing linear mapping on the one-dimensional feature obtained after the adaptive average pooling processing at least twice to obtain the classification of the second review.

[0111] In some possible embodiments, the first model is trained in the following manner: sample measured images, sample standard images, and sample labels of an object are obtained, the sample labels include detection results of different detection tasks of the object and sample classifications; the sample measured images and the sample standard images are input into the first model to be trained, features corresponding to different detection tasks are extracted by the first model to be trained, and model parameter adjustment is performed to obtain the labeled detection results of different detection tasks and the output labeled sample classifications as the target.

[0112] In some possible embodiments, the second model is trained in the following manner: sample measured images, sample standard images, and sample labels of an object are obtained, the sample labels include sample classifications of the object; the sample measured images and the sample standard images are input into the second model to be trained, feature extraction is performed on the sample measured images and the sample standard images by the second model respectively, and the feature difference is obtained, the feature difference of the non-normal region is strengthened, and model parameter adjustment is performed based on the feature difference of each region to obtain the labeled sample classifications as the target.

[0113] It should be noted that the specific connection medium between the memory 312 and the processor 311 is not limited in the present application. In the embodiment of the present application, the memory 312 and the processor 311 are connected through the bus 313, and the connection mode between other components is only illustrative and is not limited. The bus 313 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, only one thick line is used in FIG. 31, but it does not mean that there is only one bus or one type of bus.

[0114] The communication bus mentioned in the above electronic device can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0115] The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0116] The aforementioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0117] FIG. 19 is a process diagram of another optical detection object classification method according to some embodiments of the present application, which can also be used to determine the classification of the object to be detected, to realize the defect detection of the object to be detected (such as the component to be detected), and the process can include but is not limited to the following steps:

[0118] Step 1901: obtaining a measured blurred image of an object to be detected, performing feature extraction on the measured blurred image to obtain a feature value matrix corresponding to the measured blurred image, and generating a thermal image corresponding to the measured blurred image according to the feature value matrix;

[0119] Step 1902: determining each sampling line of the thermal image; for each sampling line, determining the average pixel value of the sampling line according to the pixel value of each pixel point on the sampling line;

[0120] Step 1903: fitting the average pixel value of each sampling line to obtain a first fitting curve corresponding to the measured blurred image;

[0121] Step 1904: obtaining a standard clear image corresponding to the measured blurred image, and determining a second fitting curve corresponding to the standard clear image using the same processing rule;

[0122] Step 1905: determining a defect detection result of the object to be detected according to the positional relationship between the first fitting curve and the second fitting curve.

[0123] In the present application, step 1901 is involved. The electronic device first acquires the measured fuzzy image of the object to be inspected. In some embodiments, for each type of element, the image of the element arranged in the working scene collected by the camera in real time is referred to as a measured image, and a standard clear image of the element is pre-stored. The standard clear image is a clear image of the defect-free element pre-acquired, which can be the aforementioned standard image or another image, and the present application does not make specific limitations. For the measured image of the element collected by the camera in real time, the fuzzy measured image can be manually screened out as the measured fuzzy image, and the measured fuzzy image is sent to the electronic device. In other embodiments, the camera can send the measured image of the element collected in real time to the electronic device, and the electronic device calculates the sharpness of each measured image according to the sharpness of the measured image to determine whether the measured image is a measured fuzzy image. The sharpness of the measured image can be calculated by the Tenengrad gradient method, the Laplacian gradient method, the variance method, etc., or the sharpness of the measured image can be calculated by other sharpness evaluation algorithms.

[0124] In this way, by analyzing the sharpness of the measured image, when it is determined that the measured image is a measured fuzzy image, the above-mentioned optical detection object classification method based on the measured fuzzy image is used to avoid the occurrence of the situation that in the related art, the elements in the fuzzy image are directly determined as defective elements due to high object shielding, camera focusing, etc. in actual production, and the subsequent re-determination link is performed, thereby improving the production efficiency.

[0125] In some embodiments of the present application, after the electronic device acquires the measured fuzzy image of the object to be inspected, the electronic device performs feature extraction on the measured fuzzy image to obtain a feature value matrix corresponding to the measured fuzzy image. In some embodiments, the electronic device can perform feature extraction on the measured fuzzy image by using the Local Binary Patterns (LBP) algorithm, the Histogram of Oriented Gradient (HOG) feature extraction algorithm, the Scale-invariant feature transform (SIFT) algorithm, etc., to obtain a feature value matrix corresponding to the measured fuzzy image. According to the feature value matrix, a thermal image corresponding to the measured fuzzy image is generated. In some embodiments, the feature value matrix can be used as a pixel value matrix of the thermal image, so as to map the measured fuzzy image to a corresponding visualized thermal image.

[0126] In relation to step 1902, each sampling line of the thermal image is determined. In some embodiments, the direction and number of the sampling lines can be preset. The direction of the sampling lines can be, for example, in the horizontal direction, in the vertical direction, in any angle oblique direction, etc. of the thermal image. The number of the sampling lines can be set to any value, which is usually set based on experience, for example, can be 100, 150, 180, 200, etc. According to the preset direction and number of the sampling lines, each sampling line of the thermal image is determined in a uniform or non-uniform manner.

[0127] For each sampling line, the pixel values of each pixel point on the sampling line in the thermal image are obtained, and then the average value of the pixel values of each pixel point on the sampling line is calculated as the average pixel value of the sampling line. In this way, the average pixel value of each sampling line is obtained. In relation to step 1903, for example, the average pixel value of each sampling line is fitted by using a skew distribution function to obtain a first fitting curve corresponding to the measured blurred image.

[0128] In relation to step 1904, the electronic device obtains a standard clear image corresponding to the measured blurred image. In some embodiments, according to the category of the object to be detected, a standard clear image of the category is obtained, which is pre-stored, and the standard clear image is taken as the standard clear image corresponding to the measured blurred image. Then, the same processing rule is used to determine a second fitting curve corresponding to the standard clear image. The same processing rule means that the same rule as that for determining the feature value matrix corresponding to the measured blurred image is used to determine the feature value matrix corresponding to the standard clear image; the same rule as that for generating the thermal image corresponding to the measured blurred image is used to generate the thermal image corresponding to the standard clear image; the same rule as that for determining each sampling line of the thermal image corresponding to the measured blurred image is used to determine each sampling line of the thermal image corresponding to the standard clear image; and then the same rule as that for obtaining the first fitting curve corresponding to the measured blurred image is used to obtain the second fitting curve corresponding to the standard clear image.

[0129] In relation to step 1905, the positional relationship between the first fitting curve and the second fitting curve is determined in the same coordinate system. According to the positional relationship between the first fitting curve and the second fitting curve, the classification result of the object to be detected, also referred to as the defect detection result, is determined. In some embodiments, according to the positional relationship between the first fitting curve and the second fitting curve, the coincidence degree of the first fitting curve and the second fitting curve can be determined, if the coincidence degree is greater than a preset coincidence degree threshold, it is determined that the defect detection result of the object to be detected is no defect, otherwise it is determined that the defect detection result of the object to be detected is defective.

[0130] The application provides an optical detection object classification method based on a measured fuzzy image of a to-be-detected object, relates to a component defect detection scheme, and can reflect the consistency of high-dimensional data distribution of the measured fuzzy image and a standard clear image through the positional relationship between a first fitting curve and a second fitting curve. If the consistency of high-dimensional data distribution of the measured fuzzy image and the standard clear image is high, it can be determined that the component in the measured fuzzy image is defect-free, otherwise, it is determined that there is a defect. Therefore, in some embodiments of the application, the defect detection result of the to-be-detected object can be determined according to the positional relationship between the first fitting curve and the second fitting curve. Compared with the scheme of directly determining that the component in the fuzzy image is a defective component in the related art, the accuracy of component defect detection is improved, the workload of manual rejudgment is reduced, and the efficiency of component defect detection is improved.

[0131] In order to improve the intelligence and accuracy of obtaining the measured fuzzy image of the to-be-detected object, in some embodiments of the application, the obtaining of the measured fuzzy image of the to-be-detected object includes but is not limited to: obtaining a measured image of the to-be-detected object and description parameter information of the to-be-detected object; if the sharpness of the measured image is less than a preset sharpness threshold, and it is determined according to the description parameter information that the to-be-detected object is a component without pins and without silk screen information, the measured image is taken as the measured fuzzy image.

[0132] The electronic device first obtains a measured image of a to-be-detected object, and has previously saved description parameter information of the to-be-detected object for the to-be-detected object, which includes but is not limited to part or all of the name, type, whether containing pins, whether containing silk screen information, etc. of the to-be-detected object. The silk screen information includes but is not limited to part or all of the information such as numbers, letters, characters, etc. The electronic device obtains the description parameter information of the to-be-detected object.

[0133] The electronic device calculates the sharpness of the measured image through the Tenengrad gradient method, the Laplacian gradient method, the variance method, other sharpness evaluation algorithms, etc. It is judged whether the sharpness of the measured image is less than a preset sharpness threshold. If not, it is determined that the measured image is not a measured fuzzy image. If yes, it is further determined according to the description parameter information whether the to-be-detected object is a component without pins and without silk screen information. If not, it is determined that the measured image is not a measured fuzzy image. If yes, it is determined that the measured image is a measured fuzzy image.

[0134] In some embodiments of the present application, the measured image of the component with a clarity less than the preset clarity threshold and without pins and without silk screen information is taken as the measured fuzzy image, which improves the intelligence and accuracy of obtaining the measured fuzzy image of the object to be inspected. In some embodiments of the present application, the scheme of using skew distribution function for component defect detection has high accuracy and good effect for the measured fuzzy image without pins and without silk screen information, thereby improving the accuracy of component defect detection.

[0135] FIG. 20 is a process schematic diagram of a third optical detection object classification method according to some embodiments of the present application, which includes but is not limited to the following steps:

[0136] Step 2001: obtaining a measured image of an object to be inspected and description parameter information of the object to be inspected; if the clarity of the measured image is less than a preset clarity threshold and the object to be inspected is determined to be a component without pins and without silk screen information according to the description parameter information, taking the measured image as the measured fuzzy image, performing feature extraction on the measured fuzzy image to obtain a feature value matrix corresponding to the measured fuzzy image, and generating a heat map image corresponding to the measured fuzzy image according to the feature value matrix;

[0137] Step 2002: determining each sampling line of the heat map image; for each sampling line, determining an average pixel value of the sampling line according to the pixel values of each pixel point on the sampling line;

[0138] Step 2003: fitting the average pixel value of each sampling line to obtain a first fitting curve corresponding to the measured fuzzy image;

[0139] Step 2004: obtaining a standard clear image corresponding to the measured fuzzy image, and determining a second fitting curve corresponding to the standard clear image using the same processing rule;

[0140] Step 2005: determining a defect detection result of the object to be inspected according to the positional relationship between the first fitting curve and the second fitting curve.

[0141] In some embodiments of the present application, in order to make the determination of the feature value matrix corresponding to the measured fuzzy image more accurate, the feature extraction on the measured fuzzy image to obtain the feature value matrix corresponding to the measured fuzzy image includes but is not limited to: inputting the measured fuzzy image into a feature extraction network model, and sequentially performing feature extraction on the measured fuzzy image based on a first convolution module, a residual module, a pooling module and a second convolution module in the feature extraction network model to obtain the feature value matrix corresponding to the measured fuzzy image. The network structure of the feature extraction network model can be as shown in FIG. 13.

[0142] In some embodiments of the present application, the feature extraction network model shown in FIG. 13 can be pre-trained. In training the feature extraction network model, the element image and the corresponding feature value matrix label in the training set are input into the feature extraction network model to be trained, the loss value is determined based on the predicted feature value matrix determined by the feature extraction network model to be trained and the feature value matrix label, the model parameters of the feature extraction network model to be trained are trained according to the loss value, and finally the trained feature extraction network model is obtained. As shown in FIG. 13, the feature extraction network model includes a first convolution module, a residual module, a pooling module and a second convolution module. The parameters of the first convolution module, the residual module, the pooling module and the second convolution module in the feature extraction network model to be trained are trained according to the loss value, and finally the trained feature extraction network model is obtained.

[0143] It should be noted that in some embodiments of the present application, the element category detection model can also be trained based on the element image and the corresponding element category label in the training set. The element category detection model includes an image feature extraction network and a classifier. The image feature extraction network is configured to extract image features, and the classifier is configured to predict categories based on image features. After the element category detection model is trained, the image feature extraction network in the element category detection model can be used as the feature extraction network model. The measured fuzzy image is input into the feature extraction network model, and the feature extraction network model is used to extract features of the measured fuzzy image to obtain the feature value matrix corresponding to the measured fuzzy image.

[0144] It should be noted that in some embodiments of the present application, the element category detection model can also be trained based on the element image and the corresponding element category label in the training set. The element category detection model includes an image feature extraction network and a classifier. The image feature extraction network is configured to extract image features, and the classifier is configured to predict categories based on image features. After the element category detection model is trained, the image feature extraction network in the element category detection model can be used as the feature extraction network model. The measured fuzzy image is input into the feature extraction network model, and the feature extraction network model is used to extract features of the measured fuzzy image to obtain the feature value matrix corresponding to the measured fuzzy image.

[0145] FIG. 21 is a process schematic diagram of a fourth optical detection object classification method according to some embodiments of the present application. The process includes but is not limited to the following steps:

[0146] Step 2101: obtaining a measured image of a to-be-detected object and description parameter information of the to-be-detected object; if the clarity of the measured image is less than a preset clarity threshold, and it is determined according to the description parameter information that the to-be-detected object is an element without pins and without silk screen information, then the measured image is taken as the measured fuzzy image.

[0147] Step 2102: input the measured blurred image into a feature extraction network model, sequentially perform feature extraction on the measured blurred image based on a first convolution module, a residual module, a pooling module and a second convolution module in the feature extraction network model, obtain a feature value matrix corresponding to the measured blurred image, and generate a thermal image corresponding to the measured blurred image according to the feature value matrix;

[0148] Step 2103: determine each sampling line of the thermal image; for each sampling line, determine an average pixel value of the sampling line according to pixel values of each pixel point on the sampling line;

[0149] Step 2104: fit the average pixel value of each sampling line to obtain a first fitting curve corresponding to the measured blurred image;

[0150] Step 2105: obtain a standard clear image corresponding to the measured blurred image, and determine a second fitting curve corresponding to the standard clear image using the same processing rule;

[0151] Step 2106: determine a defect detection result of the object to be detected according to the positional relationship between the first fitting curve and the second fitting curve.

[0152] In some embodiments of the present application, in order to make the determination of each sampling line of the thermal image more accurate, the determination of each sampling line of the thermal image includes but is not limited to: determining each uniformly distributed sampling line of the thermal image according to a horizontal direction of the thermal image and a preset first sampling line number; and / or determining each uniformly distributed sampling line of the thermal image according to a vertical direction of the thermal image and a preset second sampling line number.

[0153] The present application can adopt any of the following three ways to determine the sampling lines of the thermal image. Way one: according to the transverse direction of the thermal image and the preset first sampling line number, the uniformly distributed sampling lines of the thermal image are determined. In this way, the X-axis coordinates of the thermal image are uniformly divided according to the preset first sampling line number, and the preset first sampling line number of the sampling lines perpendicular to the X-axis in the thermal image is obtained. Way two: according to the longitudinal direction of the thermal image and the preset second sampling line number, the uniformly distributed sampling lines of the thermal image are determined. In this way, the Y-axis coordinates of the thermal image are uniformly divided according to the preset second sampling line number, and the preset second sampling line number of the sampling lines perpendicular to the Y-axis in the thermal image is obtained. The preset first sampling line number and the preset second sampling line number can be the same or different. For example, the preset first sampling line number is 180, and the preset second sampling line number is 200, etc. Way three: the above way one and way two are combined, that is, the preset first sampling line number of the sampling lines perpendicular to the X-axis in the thermal image is obtained, and the preset second sampling line number of the sampling lines perpendicular to the Y-axis in the thermal image is also obtained.

[0154] For the above-mentioned way one, the first fitting curve corresponding to the measured blurred image and the second fitting curve corresponding to the quasi-clear image are determined by way one respectively, and the defect detection result of the inspected object is determined according to the positional relationship of the first fitting curve and the second fitting curve. Since the features of the transverse direction of the thermal images of the two images are not presented at the same time, by analyzing the features of the transverse direction of the thermal images of the two images, that is, the positional relationship, the determined defect detection result of the inspected object can be accurate, and the comparison speed can be guaranteed.

[0155] For the above-mentioned way two, the first fitting curve corresponding to the measured blurred image and the second fitting curve corresponding to the quasi-clear image are determined by way two respectively, and the defect detection result of the inspected object is determined according to the positional relationship of the first fitting curve and the second fitting curve. Similarly, since the features of the longitudinal direction of the thermal images of the two images are not presented at the same time, by analyzing the features of the longitudinal direction of the thermal images of the two images, that is, the positional relationship, the determined defect detection result of the inspected object can be accurate, and the comparison speed can be guaranteed.

[0156] For the above-mentioned mode three, that is, the combination of the above-mentioned mode one and mode two. If the position relationship of the first fitting curve and the second fitting curve determined according to the mode one is that the defect detection result of the object under test is no defect, and the position relationship of the first fitting curve and the second fitting curve determined according to the mode two is that the defect detection result of the object under test is no defect, then the defect detection result of the object under test is determined as no defect. When the position relationship of the first fitting curve and the second fitting curve determined according to the mode one or mode two is that the defect detection result of the object under test is defect, then the defect detection result of the object under test is determined as defect. The mode three can further improve the accuracy of the defect detection result of the object under test by analyzing the characteristics of the horizontal direction and the vertical direction of the thermal images of the two images, that is, the position relationship. In actual application, any of the above-mentioned modes can be selected based on actual needs.

[0157] FIG. 22 is a schematic diagram of a fifth process of classifying an object by optical detection according to some embodiments of the present application, which includes but is not limited to the following steps:

[0158] Step 2201: Obtain a measured image of an object under test and description parameter information of the object under test; if the clarity of the measured image is less than a preset clarity threshold, and it is determined according to the description parameter information that the object under test is a component without pins and without silk screen information, then the measured image is taken as the measured blurred image;

[0159] Step 2202: input the measured blurred image into a feature extraction network model, and sequentially perform feature extraction on the measured blurred image based on a first convolution module, a residual module, a pooling module and a second convolution module in the feature extraction network model to obtain a feature value matrix corresponding to the measured blurred image, and generate a thermal image corresponding to the measured blurred image according to the feature value matrix;

[0160] Step 2203: determine each uniformly distributed sampling line of the thermal image according to a horizontal direction of the thermal image and a preset first sampling line number, and / or determine each uniformly distributed sampling line of the thermal image according to a vertical direction of the thermal image and a preset second sampling line number; for each sampling line, determine an average pixel value of the sampling line according to pixel values of each pixel point on the sampling line;

[0161] Step 2204: fit the average pixel value of each sampling line to obtain a first fitting curve corresponding to the measured blurred image;

[0162] Step 2205: obtain a standard clear image corresponding to the measured blurred image, and determine a second fitting curve corresponding to the standard clear image by using the same processing rule;

[0163] Step 2206: determining the defect detection result of the object to be inspected according to the positional relationship between the first fitting curve and the second fitting curve.

[0164] In order to make the determination of the defect detection result of the object to be inspected more accurate, in some embodiments of the present application, determining the defect detection result of the object to be inspected according to the positional relationship between the first fitting curve and the second fitting curve includes but is not limited to: determining the difference between the mean values, the difference between the standard deviations, and the intersection area of the first fitting curve and the second fitting curve according to the positional relationship between the first fitting curve and the second fitting curve; determining the defect detection result of the object to be inspected according to the belonging relationship between the difference between the mean values, the difference between the standard deviations, and the intersection area and the respective threshold range.

[0165] In some embodiments of the present application, according to the positional relationship between the first fitting curve and the second fitting curve, the difference between the mean values of the first fitting curve and the second fitting curve and the intersection area can be determined. The difference between the mean values of the first fitting curve and the second fitting curve is also the difference between the mean values of the horizontal coordinates of the sampling points on the first fitting curve and the second fitting curve, which is reflected in the difference between the horizontal coordinates of the peak points of the first fitting curve and the second fitting curve in the fitting curve. The horizontal coordinate axis of the coordinate system in which the first fitting curve and the second fitting curve are located represents the X-axis coordinate or Y-axis coordinate in the corresponding image coordinate system, and the vertical coordinate axis represents the pixel value. For the above-mentioned mode one, the horizontal coordinate axis of the coordinate system in which the first fitting curve and the second fitting curve are located is the X-axis coordinate in the corresponding image coordinate system; for the above-mentioned mode two, the horizontal coordinate axis of the coordinate system in which the first fitting curve and the second fitting curve are located is the Y-axis coordinate in the corresponding image coordinate system.

[0166] The intersection area of the first fitting curve and the second fitting curve refers to the overlapping area of the first region enclosed by the first fitting curve and the horizontal coordinate axis and the second region enclosed by the second fitting curve and the horizontal coordinate axis. After the first fitting curve and the second fitting curve are determined, the intersection area of the first fitting curve and the second fitting curve can be calculated by using the calculus algorithm.

[0167] The difference between the standard deviations of the first fitting curve and the second fitting curve refers to the difference between the standard deviations of the horizontal coordinates of the sampling points on the first fitting curve and the second fitting curve. In some embodiments, the sampling points on the first fitting curve and the second fitting curve are determined according to the same rule, then the standard deviation of the sampling points on the first fitting curve is calculated, the standard deviation of the sampling points on the second fitting curve is calculated, and the difference between the two standard deviations is calculated to obtain the difference between the standard deviations of the first fitting curve and the second fitting curve.

[0168] The electronic device pre-stores a threshold range corresponding to each of the mean difference, the standard deviation difference, and the intersection area, and determines a defect detection result of the object under test according to the belonging relationship between the mean difference, the standard deviation difference, and the intersection area and the threshold range corresponding to each of them. In some embodiments, if the mean difference, the standard deviation difference, and the intersection area all belong to the corresponding threshold range, it is determined that the defect detection result of the object under test is no defect; if any of the mean difference, the standard deviation difference, and the intersection area does not belong to the corresponding threshold range, it is determined that the defect detection result of the object under test is a defect.

[0169] FIG. 23 is a process diagram of a sixth optical detection object classification method according to some embodiments of the present application, which includes but is not limited to the following steps:

[0170] Step 2301: Obtain a measured image of an object under test and description parameter information of the object under test; if the sharpness of the measured image is less than a preset sharpness threshold, and it is determined according to the description parameter information that the object under test is a component without pins and without silk screen information, the measured image is taken as the measured blurred image;

[0171] Step 2302: input the measured blurred image into a feature extraction network model, sequentially perform feature extraction on the measured blurred image based on a first convolution module, a residual module, a pooling module, and a second convolution module in the feature extraction network model, obtain a feature value matrix corresponding to the measured blurred image, and generate a heat map image corresponding to the measured blurred image according to the feature value matrix;

[0172] Step 2303: determine each uniformly distributed sampling line of the heat map image according to a transverse direction of the heat map image and a preset first sampling line number, and / or determine each uniformly distributed sampling line of the heat map image according to a longitudinal direction of the heat map image and a preset second sampling line number; for each sampling line, determine an average pixel value of the sampling line according to pixel values of each pixel point on the sampling line;

[0173] Step 2304: fit each average pixel value of the sampling lines to obtain a first fitting curve corresponding to the measured blurred image;

[0174] Step 2305: obtain a standard clear image corresponding to the measured blurred image, and determine a second fitting curve corresponding to the standard clear image using the same processing rule;

[0175] Step 2306: determining the mean difference, the standard deviation difference and the intersection area of the first fitting curve and the second fitting curve according to the positional relationship between the first fitting curve and the second fitting curve; and determining the defect detection result of the object under test according to the belonging relationship between the mean difference, the standard deviation difference and the intersection area and the respective threshold range.

[0176] To further make the determination of the defect detection result of the object under test more accurate, in some embodiments of the present application, the determination of the defect detection result of the object under test according to the belonging relationship between the mean difference, the standard deviation difference and the intersection area and the respective threshold range includes but is not limited to: determining the area ratio of the intersection area to the union area of the first fitting curve and the second fitting curve according to the positional relationship between the first fitting curve and the second fitting curve; if the mean difference, the standard deviation difference and the area ratio all belong to the respective threshold range, determining that the defect detection result of the object under test is no defect; otherwise, determining that the defect detection result of the object under test is a defect.

[0177] In the present application, the first region surrounded by the first fitting curve and the horizontal coordinate axis and the second region surrounded by the second fitting curve and the horizontal coordinate axis are determined, the intersection area of the first region and the second region is calculated by the calculus algorithm, and the union area of the first region and the second region can also be calculated by the calculus algorithm. Then the area ratio of the intersection area to the union area is calculated. The electronic device pre-stores the respective threshold range of the mean difference, the standard deviation difference and the area ratio, and determines the defect detection result of the object under test according to the belonging relationship between the mean difference, the standard deviation difference and the area ratio and the respective threshold range. In some embodiments, if the mean difference, the standard deviation difference and the area ratio all belong to the respective threshold range, it is determined that the defect detection result of the object under test is no defect; if any of the mean difference, the standard deviation difference and the area ratio does not belong to the corresponding threshold range, it is determined that the defect detection result of the object under test is a defect.

[0178] FIG. 24 is a process schematic diagram of a seventh optical detection object classification method according to some embodiments of the present application, which includes but is not limited to the following steps:

[0179] Step 2401: obtaining the measured image of the object under test and the description parameter information of the object under test; if the sharpness of the measured image is less than a preset sharpness threshold, and it is determined according to the description parameter information that the object under test is a component without pins and without silk screen information, the measured image is taken as the measured blurred image;

[0180] Step 2402: input the measured blurred image into a feature extraction network model, sequentially perform feature extraction on the measured blurred image based on a first convolution module, a residual module, a pooling module and a second convolution module in the feature extraction network model, obtain a feature value matrix corresponding to the measured blurred image, and generate a thermal image corresponding to the measured blurred image according to the feature value matrix;

[0181] Step 2403: determine each sampling line of the thermal image uniformly distributed according to a transverse direction of the thermal image and a preset first sampling line number, and / or determine each sampling line of the thermal image uniformly distributed according to a longitudinal direction of the thermal image and a preset second sampling line number; for each sampling line, determine an average pixel value of the sampling line according to pixel values of each pixel point on the sampling line;

[0182] Step 2404: fitting the average pixel value of each sampling line to obtain a first fitting curve corresponding to the measured blurred image;

[0183] Step 2405: obtaining a standard clear image corresponding to the measured blurred image, and determining a second fitting curve corresponding to the standard clear image using the same processing rule;

[0184] Step 2406: according to the positional relationship between the first fitting curve and the second fitting curve, determining the area ratio of the intersection area and the union area of the first fitting curve and the second fitting curve; if the mean difference, the standard deviation difference and the area ratio are all within the corresponding threshold range, determining that the defect detection result of the object to be detected is no defect; otherwise, determining that the defect detection result of the object to be detected is defective.

[0185] In some embodiments of the present application, in order to prompt the staff that the element has defects, so as to carry out subsequent manual defect review process, the method further comprises: if it is determined that the defect detection result of the object to be detected is defective, outputting prompt information configured to represent manual defect review of the object to be detected.

[0186] Among them, the voice prompt information configured to represent manual defect review of the object to be detected can be output through a loudspeaker; or the light prompt information configured to represent manual defect review of the object to be detected can be output through an indicator light; or the text prompt information configured to represent manual defect review of the object to be detected can be output through a display screen, etc.

[0187] The method in some embodiments of the present application extracts the high-dimensional features of the measured blurred image by introducing a thermal image, the high-dimensional features are mapped into the form of a thermal image, a set of values of the horizontal and vertical key sampling lines are collected and calculated, and the values are fitted, such as using a skew distribution function to complete the value fitting, the peak value, area and other data between the fitting curves of the two sets of skew distribution functions of the standard clear image and the measured blurred image are calculated and compared with the threshold value, and then it is judged whether the high-dimensional feature distribution of the measured blurred image is consistent with the high-dimensional feature data distribution of the standard clear image. This method makes the defect detection result obtained in some embodiments of the present application have the characteristics of reliability, real-time, reproducibility and controllability. Among them, in some embodiments of the present application, the inference result based on the deep learning model is used for subsequent logical judgment, each model can process the to-be-inferred image transmitted by the electronic equipment (such as AOI equipment) to the algorithm within a given time period, which meets the real-time characteristic in the reliability characteristic; for the same image, the same batch of models are repeatedly inferred, and the same result can still be obtained after multiple inferences in different running environments (different platforms are used, inference time and number of times are different, etc.), which meets the reproducibility characteristic in the reliability characteristic; when in use, it can be opened or closed at any time, the degree of attention to the inference result and the degree of adoption can be intervened by the user, which meets the controllability characteristic in the reliability characteristic.

[0188] The above element defect detection process provided by the present application will be described in combination with some embodiments.

[0189] Fig. 25 is a schematic diagram of another process logic for increasing review according to some embodiments of the present application. As shown in Fig. 25, the AOI device detects that the component has no defect according to the component actual measurement image (step 2501), and the component flows to the next process (step 2502); the AOI device detects that the component has a defect according to the component actual measurement image (step 2503), if the actual measurement fuzzy image is an actual measurement fuzzy image, the actual measurement fuzzy image is detected again, if the secondary detection has a defect (step 2504), the artificial review is performed (step 2505); if the secondary detection has no defect, the component flows to the next process. In some embodiments, the foregoing secondary detection can include but is not limited to the following: obtaining an actual measurement fuzzy image of the object to be detected, performing feature extraction on the actual measurement fuzzy image to obtain a feature value matrix corresponding to the actual measurement fuzzy image; generating a thermal image corresponding to the actual measurement fuzzy image according to the feature value matrix; determining each sampling line of the thermal image; for each sampling line, determining the average pixel value of the sampling line according to the pixel value of each pixel point on the sampling line; fitting the average pixel value of each sampling line to obtain a first fitting curve corresponding to the actual measurement fuzzy image using a skew distribution function; obtaining a standard clear image corresponding to the actual measurement fuzzy image, and determining a second fitting curve corresponding to the standard clear image using the same processing rule; determining the defect detection result of the object to be detected according to the positional relationship between the first fitting curve and the second fitting curve. This method improves the defect detection accuracy, reduces misjudgment, and improves production efficiency on the basis of the original image comparison strategy of the AOI device, and the secondary detection can be completed using AI detection technology.

[0190] When the AOI device completes the preliminary detection of the actual measurement image, the image pair currently being detected (including the standard image and the actual measurement image) will be exported. Subsequently, the AI detection technology in some embodiments of the present application is used to perform secondary review on the actual measurement fuzzy image that is preliminarily detected as having a defect. The use of AI detection technology can intelligently identify and adapt to the change of the replacement material on the production line, effectively avoiding the misjudgment problem caused by the difference between the replacement materials in the traditional image matching mode. In the secondary review process, the AI detection technology will analyze and compare the standard clear image and the actual measurement fuzzy image in detail. For the component determined as having no defect, it will be directly transferred to the next process, thereby realizing fast and continuous production of the production line. For the component determined as having a defect, it will enter the next stage of artificial review.

[0191] Fig. 26 is a schematic diagram of a blurred image and a clear image according to some embodiments of the present application. Due to high object shielding, camera focusing, etc., part of the image may be blurred, as shown in the left image of Fig. 26. The right image of Fig. 26 is a standard clear image. In some embodiments of the present application, the measured blurred image and the standard clear image of the same element correspond to the relevant description information of the element. The relevant description information includes the physical identification of the element, such as the letter C for a capacitor. In the related art, in order to reduce the missed detection rate of the AOI device, the blurred image is directly judged as having a defect. However, although the image is blurred, the category and position information of the element can be obtained from the description parameter information of the object to be detected. For chip elements without silk screen and pins, the above-mentioned method can be used to determine whether there is a defect on the element corresponding to the measured blurred image. Based on the above background, some embodiments of the present application propose a defect detection algorithm for chip elements without silk screen based on blurred images.

[0192] In the present application, the measured blurred image needs to be screened first. The requirement is that the clarity is less than a preset clarity threshold, and the measured image of a small or medium-sized element without pins, silk screen information, and cracks is a measured blurred image. The elements that meet the above conditions include small and medium-sized elements such as chip resistors, capacitors, and inductors.

[0193] For the measured blurred image that meets the conditions, the high-dimensional features of the measured blurred image are extracted by sending it into a feature extraction network model. The structure of the feature extraction network model is shown in Fig. 13, but is not limited to the structure of Fig. 13. After feature extraction, the feature value matrix can present the form of a heat map image as shown in Figs. 27A-27D. Figs. 27A-27D are visualized schematic diagrams of heat map images according to some embodiments of the present application, which respectively show heat map images of different elements. Figs. 27A-27C respectively show heat map images of different types of capacitors, and Fig. 27D shows a heat map image of a PN junction of a semiconductor element.

[0194] Then, the pixel values on the sampling lines are collected from the heat map image in a sampling manner in the horizontal and vertical directions of the heat map image. For each sampling line, the average pixel value of the sampling line is calculated to obtain horizontal group data and vertical group data, and the data is normalized. Fig. 28 is a schematic diagram of horizontal and vertical sampling lines in a heat map image according to some embodiments of the present application.

[0195] For example, in some embodiments of the present application, skew distribution function is used to fit the horizontal group data of the measured blurred image to obtain a measured horizontal fitting curve, and the skew distribution function is used to fit the horizontal group data of the standard clear image to obtain a standard horizontal fitting curve. The skew distribution function is used to fit the longitudinal group data of the measured blurred image to obtain a measured longitudinal fitting curve, and the skew distribution function is used to fit the longitudinal group data of the standard clear image to obtain a standard longitudinal fitting curve. Then, according to the positional relationship between the measured horizontal fitting curve and the standard horizontal fitting curve, and the positional relationship between the measured longitudinal fitting curve and the standard longitudinal fitting curve, the defect detection result of the object under test is determined.

[0196] FIG. 29 is a schematic diagram of fitting the horizontal group data of the measured blurred image to obtain a measured horizontal fitting curve according to some embodiments of the present application. FIG. 30 is a schematic diagram of the positional relationship between the measured horizontal fitting curve and the standard horizontal fitting curve according to some embodiments of the present application. For the horizontal group data, the skew distribution functions of the standard clear image and the measured blurred image are calculated respectively, the difference between the mean values of the two groups of data is calculated, the difference between the standard deviations is calculated, the area below the intersection part of the two curves is calculated, the area below the compared part accounts for the proportion of the integral sum of the two groups of functions, and the proportion is compared with the corresponding preset threshold range. If it exceeds the preset threshold range, it means that the measured blurred image is an element image with defects; if the above indicators are within the threshold range, it means that the measured blurred image is an element image without defects.

[0197] On the basis of the above embodiments, referring to FIG. 31, the processor 311 included in the electronic device is further configured to execute the computer program stored in the memory 312 to make the electronic device perform: acquiring a measured blurred image of an object under test, performing feature extraction on the measured blurred image to obtain a feature value matrix corresponding to the measured blurred image; generating a thermal image corresponding to the measured blurred image according to the feature value matrix; determining each sampling line of the thermal image; for each sampling line, determining the average pixel value of the sampling line according to the pixel value of each pixel point on the sampling line; fitting the average pixel value of each sampling line to obtain a first fitting curve corresponding to the measured blurred image; acquiring a standard clear image corresponding to the measured blurred image, and determining a second fitting curve corresponding to the standard clear image using the same processing rule; and determining the defect detection result of the object under test according to the positional relationship between the first fitting curve and the second fitting curve.

[0198] In some possible embodiments, the processor 311 is further configured to execute the computer instructions to cause the electronic device to perform: acquiring a measured image of the object to be inspected and description parameter information of the object to be inspected; and if the definition of the measured image is less than a preset definition threshold and it is determined according to the description parameter information that the object to be inspected is a component without pins and without silk screen information, taking the measured image as the measured fuzzy image.

[0199] In some possible embodiments, the processor 311 is further configured to execute the computer instructions to cause the electronic device to perform: inputting the measured fuzzy image into a feature extraction network model, and sequentially performing feature extraction on the measured fuzzy image based on a first convolution module, a residual module, a pooling module and a second convolution module in the feature extraction network model to obtain a feature value matrix corresponding to the measured fuzzy image.

[0200] In some possible embodiments, the processor 311 is further configured to execute the computer instructions to cause the electronic device to perform: determining, according to a transverse direction of the thermal image and a preset first number of sampling lines, each sampling line of the thermal image that is uniformly distributed; and / or determining, according to a longitudinal direction of the thermal image and a preset second number of sampling lines, each sampling line of the thermal image that is uniformly distributed.

[0201] In some possible embodiments, the processor 311 is further configured to execute the computer instructions to cause the electronic device to perform: determining, according to the positional relationship between the first fitting curve and the second fitting curve, a difference between mean values, a difference between standard deviations and an intersection area of the first fitting curve and the second fitting curve; and determining, according to the belonging relationship between the difference between mean values, the difference between standard deviations and the intersection area and a respective corresponding threshold range, a defect detection result of the object to be inspected.

[0202] In some possible embodiments, the processor 311 is further configured to execute the computer instructions to cause the electronic device to perform: determining, according to the positional relationship between the first fitting curve and the second fitting curve, an area ratio of the intersection area to the union area of the first fitting curve and the second fitting curve; if the difference between mean values, the difference between standard deviations and the area ratio all belong to respective corresponding threshold ranges, determining that the defect detection result of the object to be inspected is no defect; otherwise, determining that the defect detection result of the object to be inspected is a defect.

[0203] In some possible embodiments, the processor 311 is further configured to execute the computer instructions to cause the electronic device to perform: if it is determined that the defect detection result of the object to be inspected is a defect, outputting prompt information configured to represent manual defect re-trial on the object to be inspected.

[0204] On the basis of the above-mentioned embodiments, the above-mentioned any one method in some embodiments of the present application can also be realized by cooperation of various modules, and the specific manners of the modules have been described in detail in the foregoing embodiments, which will not be described in detail here.

[0205] Some embodiments of the present application also provide a computer readable nonvolatile storage medium, which stores a computer program executable by an electronic device, and when the program runs on the electronic device, causes the electronic device to execute the above any one method step.

[0206] Finally, it should be noted that: the above embodiments are only used to illustrate the schemes of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: they can still modify the schemes recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding scheme deviate from the scope of the schemes of the embodiments of the present application.

Claims

1. An electronic device, comprising at least one processor and a memory communicatively connected to said at least one processor; wherein, The memory is configured to store computer instructions; The processor is configured to execute the computer instructions to cause the electronic device to perform: Obtain the measured image and annotation information of the object to be inspected. The annotation information includes annotated abnormal areas. The abnormal areas are determined based on the measured image and the standard image of the object to be inspected, identifying areas in the measured image that characterize the object to be inspected as unqualified. The first model is used to extract features of the measured image and the standard image under different detection tasks. Based on the extracted features, the detection results corresponding to different detection tasks are determined, and the classification of the first re-inspection is determined according to the detection results. If the classification of the object to be inspected in the first re-inspection indicates that it is qualified, then the second model is used to extract features from the measured image and the standard image respectively and to calculate the feature differences. The feature differences of the abnormal regions are enhanced and labeled, and the classification of the second re-inspection is determined based on the feature differences of each region.

2. The electronic device according to claim 1, wherein the annotation information further includes relevant information for assisting in determining the query statement, and the processor is further configured to execute the computer instructions to cause the electronic device to perform: The classification of the object to be inspected in the second re-inspection is determined to be qualified, and the type of the object to be inspected is a specified type; Based on predefined question statements that affect the classification results, determine the question statements that match the annotation information; The measured image, the standard image, and the question statement matching the annotation information are input into the third model to obtain the classification of the third re-examination.

3. The electronic device according to claim 2, wherein the third model is an AnomalyGPT large model, the object to be inspected is a component, and the specified type is a silkscreen type; The third model is trained in the following manner: Acquire sample measured images of screen-printed components, sample standard images, question statements that ask about the parameters and conditions involved in identifying screen-printed polarity points, screen-printed polarity point information, and annotation information of classification results; The measured images of the screen-printed components, the labeled images of the samples, and the question statements that ask about the parameters and conditions involved in identifying the polarity points of the screen print are input into the third model to be trained. The model is trained with the goal of outputting the polarity point information and classification results of the screen print.

4. The electronic device according to any one of claims 1 to 3, wherein the object to be inspected is a component, and the different inspection tasks include identifying some or all of the existing components, component types, component defect types, silkscreen positions, and silkscreen directions; the processor is further configured to execute the computer instructions to cause the electronic device to perform: When determining that the detection results include position and silkscreen direction, the silkscreen position is segmented, and the silkscreen position is rotated according to the silkscreen direction until the silkscreen position is upright. Identify the content of the silkscreen in the rotated silkscreen position, determine whether it is the same as the direction of the silkscreen in the standard image, and whether the silkscreen content is interchangeable. If the orientation is different or the silkscreen content is not interchangeable, the output will indicate that the object to be inspected is unqualified.

5. The electronic device according to claim 1, wherein when enhancing the feature differences of abnormal regions, the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: The feature differences between the measured image and the standard image are obtained by calculating the difference between the features extracted from different regions according to the region. Based on a first weight, the characteristic differences of normal regions are determined, and based on a second weight, the characteristic differences of abnormal regions are determined, wherein the second weight is greater than the first weight.

6. The electronic device according to any one of claims 1 to 3 and 5, wherein when determining the classification for the second re-inspection based on the feature differences of each region, the processor is specifically configured to execute the computer instructions to cause the electronic device to perform: Channel attention processing is applied to the characteristic differences of each region; The features after channel attention processing are convolved, activated by an activation function, and then subjected to adaptive average pooling. The one-dimensional features obtained after adaptive average pooling are subjected to at least two linear mappings to obtain the classification of the second re-examination.

7. The electronic device according to claim 1, wherein the first model is trained in the following manner: Acquire the measured sample image, the standard sample image, and the sample annotation of the object. The sample annotation includes the detection results of different detection tasks and the sample classification of the object. The measured sample images and standard sample images are input into the first model to be trained. The first model to be trained extracts features corresponding to different detection tasks. The model parameters are adjusted with the goal of obtaining the detection results of different detection tasks and the output of labeled sample classification.

8. The electronic device according to claim 1, wherein the second model is trained in the following manner: Acquire sample measured images, sample standard images, and sample annotations of the object, wherein the sample annotations include the sample classification of the object; The measured sample image and the standard sample image are input into the second model to be trained. The second model extracts features from the measured sample image and the standard sample image respectively and calculates the feature differences. It also enhances the annotation of the feature differences in abnormal regions. Based on the feature differences of each region, the model parameters are adjusted with the output labeled sample classification as the goal.

9. The electronic device according to claim 1, wherein if the clarity of the measured image is less than a preset clarity threshold, and the object under test is determined to be a component without pins or silkscreen information based on the description parameter information of the object under test, the processor is further configured to execute the computer instructions to cause the electronic device to perform: The measured image is used as the measured blurred image of the object to be inspected. Feature extraction is performed on the measured blurred image to obtain the feature value matrix corresponding to the measured blurred image. A thermal image corresponding to the measured blurred image is generated based on the feature value matrix. Determine each sampling line of the thermal image; for each sampling line, determine the average pixel value of the sampling line based on the pixel values ​​of each pixel on the sampling line; The average pixel value of each sampling line is fitted to obtain the first fitting curve corresponding to the measured blurred image; Obtain a standard clear image corresponding to the measured blurred image, and determine the second fitting curve corresponding to the standard clear image using the same processing rules; Based on the positional relationship between the first fitted curve and the second fitted curve, the defect detection result of the object to be inspected is determined.

10. A method for classifying optically detected objects, comprising: Obtain the measured image and annotation information of the object to be inspected. The annotation information includes annotated abnormal areas. The abnormal areas are determined based on the measured image and the standard image of the object to be inspected, identifying areas in the measured image that characterize the object to be inspected as unqualified. The first model is used to extract features of the measured image and the standard image under different detection tasks. Based on the extracted features, the detection results corresponding to different detection tasks are determined, and the classification of the first re-inspection is determined according to the detection results. If the classification of the object to be inspected in the first re-inspection indicates that it is qualified, then the second model is used to extract features from the measured image and the standard image respectively and to calculate the feature differences. The feature differences of the abnormal regions are enhanced and labeled, and the classification of the second re-inspection is determined based on the feature differences of each region.

Images