Multi-model voting surface defect detection methods, apparatus, equipment, and storage media

Abstract

This application discloses a surface defect detection method, apparatus, device, and storage medium using multi-model voting, relating to the field of model detection. The method involves inputting a test image into a multi-model detection system, which outputs the inference results corresponding to each model. Defect classifications are determined based on the image size and target type within the image, and defect mask images are created sequentially according to these classifications. Target defect mask images are selected sequentially based on the defect classifications, and the inference results output by the target model are mapped to the target defect mask image based on the corresponding defect type. Finally, the target defects in the test image are labeled based on the mapped target defect mask image. This scheme can integrate multi-model, multi-dimensional detection results and filter defect identification results based on voting results, resulting in higher accuracy in defect identification.

Description

Technical Field

[0001] This application relates to the field of model testing, and in particular to a method, apparatus, device, and storage medium for detecting surface defects using multi-model voting. Background Technology

[0002] Surface defect detection is a crucial step in ensuring product quality in industrial production. With the rapid development of computer vision and deep learning technologies, machine vision-based surface defect detection methods have been widely applied across various industrial sectors. Compared to traditional manual visual inspection, machine vision inspection offers advantages such as speed, high accuracy, and the ability to operate continuously for extended periods, effectively improving the efficiency and quality of industrial production. The development of surface defect detection technology has gone through several stages, from initial manual visual inspection to optical sensor inspection, and now to machine vision inspection. Machine vision inspection technology uses image sensors to acquire images of objects and leverages image processing and analysis techniques to identify surface defects. This method offers advantages such as non-contact operation, high precision, and high efficiency, making it one of the main methods for surface defect detection in modern industry.

[0003] In terms of technical implementation, key challenges in surface defect detection include the few-shot problem and real-time performance issues. To address these challenges, researchers have proposed various methods, such as data augmentation, network pre-training and transfer learning, rational network structure design, and unsupervised or semi-supervised methods. Current research progress mainly focuses on deep learning-based methods, including CNN-based defect classification methods, object detection-based defect detection methods, semantic segmentation-based defect detection methods, and generative adversarial network (GAN)-based defect detection methods. Future development trends in surface defect detection technology may include few-shot learning, self-supervised learning, multimodal fusion, interpretability research, edge computing, active learning, 3D defect detection, and video sequence defect detection. However, regardless of the mainstream technical solution, none can meet or be compatible with all the requirements of defect image detection technology. There are trade-offs: sacrificing accuracy for improved detection timeliness, or sacrificing timeliness for improved detection accuracy, or failing to balance the issue of small target recognition. Furthermore, in high-precision recognition scenarios, the different false detection rates of different detection models may lead to inconsistent output results. Summary of the Invention

[0004] This application provides a surface defect detection method, apparatus, device, and storage medium using multi-model voting, which solves the problem of balancing detection accuracy and false detection rate among different models.

[0005] On one hand, this application discloses a surface defect detection method using multi-model voting, the method comprising:

[0006] The image to be tested is input into the multi-model detection system, which performs image inference based on different types of target models and outputs the inference results corresponding to each model.

[0007] Defect classification is determined based on the size of the image to be tested and the type of target in the image, and defect mask images are created sequentially according to the defect classification;

[0008] Target defect masks are selected sequentially according to defect classification. Based on the corresponding defect type, the inference results output by the target model are mapped to the target defect mask. Finally, the target defects in the image to be tested are labeled according to the mapped target defect mask.

[0009] Specifically, the step of determining defect classification based on the size of the image to be tested and the target type in the image, and sequentially creating defect mask images according to the defect classification, includes:

[0010] Identify the target object in the image to be tested, and set the M types of defects included in the image detection according to the type of the target object;

[0011] Based on the size of the image to be tested and the N inference results output by the model, N defect mask images are set, and corresponding defect labels are set according to the defect type; the defect mask images have the same size dimension as the image to be tested, and the pixel feature matrix corresponding to the defect mask image is an all-zero matrix.

[0012] Specifically, the inference result includes the defect type, defect coordinates, defect size, and confidence value of the target defect identified from the image under test; different target models output inference results containing different target defect types, quantities, coordinates, sizes, and confidence values.

[0013] Specifically, mapping the inference result output by the target model to the target defect mask image based on the corresponding defect type includes:

[0014] Select the target defect mask image based on the defect label, poll the N inference results output by N target models, and extract the coordinates, size, and confidence value of all target defects under the target defect type; wherein, the confidence values ​​of the size range determined by the same target defect are all the same.

[0015] Based on the coordinates of the target defect, the confidence values ​​within the defect size range are transferred to the target position of the all-zero matrix corresponding to the target defect mask image;

[0016] Poll all inference results until all confidence values ​​of all target defects of the same target defect type in N inference results are transferred to the target position of the all-zero matrix to form a target fusion matrix;

[0017] The target fusion matrix is ​​mapped to the target defect mask image.

[0018] Specifically, the process of forming the fusion matrix includes:

[0019] When the i-th type of target defect is found from the inference results, the confidence values ​​within the defect size range are accumulated into the target region in the feature matrix; when the target regions mapped in multiple inference models overlap, the confidence values ​​are accumulated sequentially to obtain the voting result matrix of the i-th type of target defect.

[0020] The fusion threshold is determined based on the number of target models participating in the vote, and compared with the matrix pixel data of the voting result matrix. The values ​​of all pixels less than the fusion threshold are set to 0, thus forming the target fusion matrix.

[0021] Specifically, the step of identifying target defects in the image to be tested based on the mapped target defect mask includes:

[0022] Determine the target region composed of non-zero pixels in the target fusion matrix;

[0023] The extreme values ​​of the coordinates in the two-dimensional coordinate system are determined based on the position coordinates of all pixels in the target area. A rectangular box is bounded around the extreme values ​​of the coordinates and the bounding box is determined as the coverage area of ​​the target defect.

[0024] The coverage area of ​​the target defect is mapped onto the image to be tested, and the defect is labeled on the image to be tested according to the label information.

[0025] Specifically, all defect categories are polled, and when a target defect category is selected, all inference results are polled, and the image to be tested is mapped and calibrated using the corresponding target defect mask image;

[0026] In the output defect detection map, the identified defective objects are represented by the mapped outer rectangle and a defect label is placed. The defect label stores the defect classification and confidence information of the identified defective objects.

[0027] On the other hand, this application provides a surface defect detection device using multi-model voting.

[0028] The inference module is used to input the image to be tested into the multi-model detection system, perform image inference based on different types of target models, and output the inference results corresponding to each model.

[0029] A module is created to determine the defect classification based on the size of the image to be tested and the type of target in the image, and to create defect mask images sequentially according to the defect classification;

[0030] The calibration module is used to select target defect mask images sequentially according to defect classification, map the inference results output by the target model to the target defect mask image based on the corresponding defect type, and perform defect calibration on the target defects in the image to be tested based on the mapped target defect mask image.

[0031] In another aspect, this application provides a computer device including a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by the processor to implement the multi-model voting surface defect detection method described above.

[0032] In another aspect, this application provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the multi-model voting surface defect detection method described above.

[0033] The beneficial effects of the technical solution provided in this application include at least the following: This application utilizes a composite system integrating multiple detection models to perform inference on the input image to be tested, and outputs the inference results of each selected target model; then, it determines the defect classification based on the type of image content, creating a one-to-one corresponding defect mask map; next, it uses the mapped defect mask map to poll the various inference results, performs fusion voting on the target defects, and finally uses the fused and voted mask map to map back to the image to be tested, thus achieving defect labeling. This solution can fuse detection results from multiple models and multiple dimensions, and filter defect identification results based on voting results, resulting in higher accuracy of defect identification results. Attached Figure Description

[0034] Figure 1 This is a flowchart of the surface defect detection method using multi-model voting provided in this application embodiment;

[0035] Figure 2 The algorithm flowchart of the multi-model voting surface defect detection method is shown;

[0036] Figure 3 This is a schematic diagram showing the mapping of the defect mask image;

[0037] Figure 4 A schematic diagram illustrating the transfer of confidence values ​​to the target defect mask image in one possible form is provided;

[0038] Figure 5 A diagram illustrating how different inference model results map to the same target defect type is provided.

[0039] Figure 6 A schematic diagram of target fusion matrix mapping and defect labeling is shown;

[0040] Figure 7 The diagram shows the wafer image before and after inspection.

[0041] Figure 8 This is a structural block diagram of the surface defect detection device with multi-model voting provided in the embodiments of this application;

[0042] Figure 9 A structural block diagram of a computer device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0044] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0045] Figure 1 This is a flowchart of a surface defect detection method using multi-model voting provided in an embodiment of this application, including the following steps:

[0046] S1. Input the image to be tested into the multi-model detection system, perform image inference based on different types of target models, and output the inference results corresponding to each model.

[0047] In the field of small target detection, particularly for wafer surface defect detection, defects are unavoidable during the manufacturing process of wafers, such as welds, cracks, fractures, and contamination. Special attention is paid to specific defects that tend to occur in clusters, such as mirror loss, damage, and misalignment of wafer pads. These high-resolution small target detection methods typically employ object detection-based defect detection methods, semantic segmentation-based defect detection methods, and generative adversarial networks (GAN)-based defect detection methods. However, each model focuses on a specific area and cannot cover all advantages, particularly exhibiting insufficient detection capability for subtle defects and significant discrepancies in detection results. This application addresses this issue by establishing a multi-model composite defect detection system. This system integrates detection models widely used in the semiconductor and wafer inspection fields, utilizing multiple models to detect the same image under test. Simultaneously, voting fusion based on inference output improves detection accuracy.

[0048] In some embodiments, the multi-model detection system may include at least the Mask-R-CNN model, YOLOv5 model, YOLOX model, DETR model, and Dino model, all of which can be used for surface defect detection. In practice, at least two models can be selected as needed. After inputting the image to be tested into the selected target model, subsequent fusion calculations are performed based on the inference results.

[0049] It should be noted that for multiple target models selected, they need to be trained using the same defect detection dataset. The purpose is to maintain consistency at the data level, which is essential to ensure the defect detection capability of the model.

[0050] S2. Determine the defect classification based on the size of the image to be tested and the type of target in the image, and create defect mask images in sequence according to the defect classification;

[0051] Because different test images may produce different types of defects, such as ordinary wafer particles and gold-plated pads, the types of defects that may arise when the target object type is different will naturally differ, and the defects identified from the images will also vary. Therefore, this application sets a defect classification set in advance based on the type of target object in the input test image before performing subsequent fusion calculations. After each model outputs inference results, the corresponding set is directly selected according to the type of target object, and then a defect mask map is created based on the defect types in the set. The advantage of this is that it can reduce the number of output defect types, especially when multiple models have multiple defect detection functions, which may lead to false detections. This approach can selectively filter out defects that should not exist or do not affect product function and quality, thereby improving the overall detection accuracy. The defect mask map is an intermediate image specifically used for voting fusion based on the inference results. The mask map can be used to extract regions of interest in the image, mask certain areas on the image, and detect and extract structural features in the image similar to the mask.

[0052] In one possible implementation, suppose that the inference output of model A is {a defect, b defect, c defect} and the inference output of model B is {a defect, e defect, g defect}. However, the defect set that is usually of concern to the target object is {a defect, g defect}. In this case, when creating the defect mask map, only the defect mask map for defects a and g will be created. Other types of defects will not be considered and no corresponding mask map will be created.

[0053] S3. Select target defect mask images sequentially according to defect classification, map the inference results output by the target model to the target defect mask image based on the corresponding defect type, and perform defect labeling on the target defects in the image to be tested based on the mapped target defect mask image.

[0054] Assuming M defect categories are determined based on the target type, M corresponding defect mask images are created. Then, a target defect of a certain type (e.g., crack defects) is selected sequentially. Using the corresponding crack defect mask image as a base, all inference results containing crack defects are iterated over. Based on the location of the identified crack defect in the image to be tested, the defect result is mapped onto the crack defect mask image, resulting in a crack defect mask image containing crack detection information from all models. The mapping results of each defect are then voted on and fused. This method can filter out results with insufficient detection accuracy by setting a certain threshold. Conversely, detected target defects can be enhanced by the inference results of multiple models, ensuring detection accuracy. Finally, the fused crack defect mask image is mapped back into the image to be tested, achieving defect labeling.

[0055] Figure 2 The flowchart of a multi-model voting surface defect detection method is presented. Assuming there are N inference results and M defect classifications, this scheme takes defect classification as the objective, iteratively polling the N inference results to complete the voting and fusion of the target defect classification. By polling the M defect classifications, defect mapping is performed sequentially on the M defect mask images until the detection of M types of defects is completed.

[0056] In summary, this application utilizes a composite system integrating multiple detection models to infer the input image to be tested, outputting the inference results of each selected target model. Then, it determines the defect classification based on the image content type, creating a one-to-one corresponding defect mask map. Next, it uses the mapped defect mask map to iterate through the various inference results, performing a fusion vote on the target defects. Finally, it uses the fused, voted mask map to map back to the image to be tested, achieving defect labeling. This scheme can fuse detection results from multiple models and multiple dimensions, filtering defect identification results based on voting results, resulting in higher accuracy in defect identification.

[0057] In one possible implementation, the process of creating a defect mask map can be summarized as follows:

[0058] A. Identify the target object in the image to be tested, and set the M types of defects included in the image detection according to the type of the target object;

[0059] B. Set N defect mask images based on the N inference results output by the model, and set corresponding defect labels according to the defect type.

[0060] like Figure 3 As shown, the defect mask images and the image under test have the same dimensions, and the pixel feature matrix corresponding to the initial N defect mask images is an all-zero matrix. Pixels at different positions in the matrix can be mapped back to the image under test based on their coordinates.

[0061] Although the inference results are outputs from different models, the format and structure of the output content are consistent, including the defect type, defect coordinates, defect size, and confidence score of the target defect identified from the image under test. For different target models, depending on their functionality, accuracy, and characteristics, the output inference results will typically contain different target defect types, quantities, coordinates, sizes, and confidence scores.

[0062] Assuming there are N target models participating in the inference and outputting N inference results, the first inference result can be represented as {a1 defect (x1, y1, w1, h1, s1), b defect (x2, y2, w2, h2, s2), c defect (x3, y3, w3, h3, s3), a2 defect (x4, y4, w4, h4, s4)}. This structure indicates that a total of 4 target defects have been identified, where a1 and a2 represent two defects of the same type (a). (x, y, w, h) represent the parameter information of the target defects; specifically, x and y represent the coordinates of the top-left corner of the target, w represents the defect width, h represents the defect width and height, and s represents the confidence score. To facilitate matrix fusion and result voting later, each created defect mask image is assigned a corresponding defect label, i.e., labelM1, labelM2, ..., labelMm.

[0063] In one possible implementation, the step of mapping the inference result to the target defect mask image may include the following steps:

[0064] A. Select the target defect mask image based on the defect label, poll the N inference results output by N target models, and extract the coordinates, size and confidence value of all target defects under the target defect type;

[0065] B. Based on the coordinates of the target defect, transfer the confidence values ​​within the defect size range to the target position of the all-zero matrix corresponding to the target defect mask image;

[0066] In some embodiments, the detected defects are not necessarily regular rectangles. Therefore, the target model detection output is determined by selecting a rectangle and then determining the coordinates and size of the upper right or upper left corner. For a given target model, after identifying a defect, a confidence value is output. This confidence value represents the confidence value of the defect size range corresponding to the entire target defect area, and all confidence values ​​for the size range determined by the same target defect are the same.

[0067] Figure 4This diagram illustrates one possible form of confidence value transfer to a target defect mask. Assuming M1 classification is selected, the x, y, w, h, and score values ​​for this M1 classification are obtained. Based on x, y, w, and h, an identical region A is obtained on the corresponding pure 0 matrix. The values ​​of all pixels within region A are changed from 0 to the score value. If a certain inference result contains multiple M1 classifications, multiple corresponding regions A are also obtained on the pure 0 matrix. If there is overlap between regions, the score values ​​of the overlapping parts are added together. Figure 4 The confidence values ​​for the two defect size ranges are 0.8 and 0.4, respectively. Therefore, after localization, they can be directly mapped onto the target defect mask image, and the model inference results can be transferred to the mask image for subsequent calculations.

[0068] C. Poll all inference results until all confidence values ​​of all target defects of the same target defect type in N inference results are transferred to the target position of the all-zero matrix to form the target fusion matrix;

[0069] Figure 5 A schematic diagram is shown showing how the results of different inference models map to the same target defect type. This step mainly considers the accuracy and differences of different target models, and summarizes the inference results of multiple models into a single mask diagram to achieve the effect of complementary advantages of each model.

[0070] Similarly, the fusion process of different inference models can be summarized as follows:

[0071] 1. When the i-th type of target defect is found from the inference results, the confidence values ​​within the defect size range are accumulated into the target region in the feature matrix; when the target regions mapped in multiple inference models overlap, the confidence values ​​are accumulated sequentially to obtain the voting result matrix of the i-th type of target defect.

[0072] 2. Determine the fusion threshold based on the number of target models participating in the vote, and compare it with the matrix pixel data of the voting result matrix. Set the value of all pixels less than the fusion threshold to 0 to form the target fusion matrix.

[0073] Given that N dimensions represent the inference results of multiple models on a single classification, directly adding these N matrices yields a combined result matrix of multiple models on that single classification, i.e., a fusion of the inference results across N dimensions. This can be considered a voting result of multiple models on that single classification. Typically, the fusion threshold is positively correlated with the number of models. Setting all pixels in the voting result matrix with values ​​less than the fusion threshold to 0, then all values ​​greater than 0 in the matrix are considered to be true M1 classifications.

[0074] D. Map the target fusion matrix to the target defect mask.

[0075] Figure 6 A schematic diagram of target fusion matrix mapping and defect calibration is shown. The calibration process can be summarized as follows:

[0076] A. Determine the target region composed of non-zero pixels in the target fusion matrix;

[0077] B. Determine the extreme values ​​of the coordinates in the two-dimensional coordinate system based on the position coordinates of all pixels in the target area, circumscribe a rectangle around the extreme values ​​of the coordinates, and determine it as the coverage area of ​​the target defect.

[0078] C. Map the coverage area of ​​the target defect to the image under test, and label the defect in the image under test according to the label information.

[0079] In the specific calibration process, M defect images can be generated sequentially according to the defect classification, and a specific defect classification label can be marked on each defect image. Optionally, all defect classifications can be polled in turn, and when a target defect classification is selected, all inference results can be polled in turn, and the image to be tested can be mapped and calibrated through the corresponding target defect mask image. In this approach, the identified defective object is represented by a mapped bounding rectangle in the output defect detection image, and a defect label is placed in it. The defect label stores the defect classification and confidence information of the identified defective object.

[0080] The following section explains the defect detection calibration using an image of the wafer to be tested as input.

[0081] 1. Use two models, YOLOX and co-detr, for multi-model voting;

[0082] 2. Train both models using the same dataset to obtain yolox-model and co-detr-model;

[0083] 3. Use yolox-model and co-detr-model to infer the same image;

[0084] 4. Normalize the inference results: Convert the inference results of both models into a (label, score, bounding box) format; the bounding box represents the size of the rectangle.

[0085] 5. For the first category, create two matrices of the same size as the original image, with values ​​equal to 0, based on the number of models.

[0086] 6. Based on the results of (label, score, bbox), modify the positions corresponding to these two pure 0 matrices to the values ​​of the score;

[0087] 7. Add these two matrices together to obtain the voting result matrix for the first category;

[0088] 8. Set the threshold threshold thres to 0.6, and set the values ​​at positions less than thres to 0;

[0089] 9. Obtain the x, y, w, and h values ​​for each region of the voting result matrix;

[0090] 10. Continue processing the next category until the last one, and obtain the x, y, w, h values ​​for all categories. Since the label is known and the score has been filtered by the threshold value, we can finally obtain multiple (label, bbox) values. Based on these values, we can perform visualization plots on the original image.

[0091] Figure 7 The diagram shows the image of the wafer before and after inspection. In the output defect calibration map, the selected color can be used as the label type, so that all defect types can be displayed in the same image and are easy to observe.

[0092] In summary, this solution mainly addresses the issue of a single model having a significant bottleneck (e.g., consistently performing poorly in detecting a particular defect). By introducing multiple models to vote on detection, the shortcomings of a single model can be effectively mitigated, resulting in a more balanced final detection result. This approach offers a certain degree of relief for small sample size and imbalanced sample problems.

[0093] Using multi-model voting can make it easier to pinpoint the root cause of a problem. If the detection of a certain type of defect still performs poorly when using multi-model voting, it can be basically ruled out that the problem is caused by the model. It is more likely that the problem is caused by the data, and more effort can be put into optimizing the data.

[0094] Figure 8 This paper shows a structural block diagram of a surface defect detection device with multi-model voting provided in an embodiment of this application. The device includes:

[0095] The inference module 810 is used to input the image to be tested into the multi-model detection system, perform image inference based on different types of target models, and output the inference results corresponding to each model.

[0096] A creation module 820 is used to determine the defect classification based on the size of the image to be tested and the target type in the image, and to create defect mask images sequentially according to the defect classification;

[0097] The calibration module 830 is used to select target defect mask images sequentially according to defect classification, map the inference results output by the target model to the target defect mask image based on the corresponding defect type, and perform defect calibration on the target defects in the image to be tested based on the mapped target defect mask image.

[0098] The wafer loading misalignment correction device provided in this application embodiment can be applied to the wafer loading misalignment correction method provided in the above embodiment. For relevant details, please refer to the above method embodiment. The implementation principle and technical effect are similar, and will not be repeated here.

[0099] It should be noted that the wafer loading misalignment correction device provided in this embodiment is only an example of the above-described division of functional modules / units. In practical applications, the above functions can be assigned to different functional modules / units as needed, that is, the internal structure of the wafer loading misalignment correction device can be divided into different functional modules / units to complete all or part of the functions described above. Furthermore, the implementation method of the wafer loading misalignment correction method provided in the above method embodiments and the implementation method of the wafer loading misalignment correction device provided in this embodiment belong to the same concept. The specific implementation process of the wafer loading misalignment correction device provided in this embodiment is detailed in the above method embodiments and will not be repeated here.

[0100] Figure 9 This illustration shows a structural block diagram of a computer device provided in an exemplary embodiment of this application. The device includes desktop computers, laptops, handheld computers, and cloud servers. This computer device may include, but is not limited to, a processor and memory. The processor and memory can be connected via a bus or other means. The processor may be a Central Processing Unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, graphics processing units (GPUs), embedded neural network processing units (NPUs) or other dedicated deep learning coprocessors, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0101] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor may also include a main processor and coprocessors. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0102] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the above embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include a program storage area and a data storage area, wherein the program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] In some embodiments, the computer device may also optionally include: a peripheral device interface and at least one peripheral device. The processor, memory, and peripheral device interface can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface via a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit, a display screen, and a keyboard.

[0104] Peripheral device interfaces can be used to connect at least one I / O (Input / Output) related peripheral device to the processor and memory. In some embodiments, the processor, memory, and peripheral device interface are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor, memory, and peripheral device interface can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0105] The display screen is used to display a user interface (UI). This UI can include graphics, text, icons, videos, and any combination thereof. When the display screen is a touch screen, it also has the ability to collect touch signals on or above the surface of the display. These touch signals can be input as control signals to a processor for processing. In this case, the display screen can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen, located on the front panel of the computer device; in other embodiments, there may be at least two display screens, respectively located on different surfaces of the computer device or in a foldable design. In other embodiments, the display screen may be a flexible display screen, located on a curved or folded surface of the computer device. Furthermore, the display screen can be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0106] A power supply is used to power the various components in a computer device. The power supply can be alternating current (AC), direct current (DC), a disposable battery, or a rechargeable battery. When the power supply includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is charged via a wired connection, while a wireless rechargeable battery is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0107] Those skilled in the art will understand that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0108] This application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, which, when executed by a processor, implements the methods described in the above-described method embodiments. Those skilled in the art will understand that implementing all or part of the processes in the methods described in the above-described embodiments of this application can be accomplished by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0109] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A surface defect detection method using multi-model voting, characterized in that, The method includes: The image to be tested is input into the multi-model detection system, which performs image inference based on different types of target models and outputs the inference results corresponding to each model. Identify the target object in the image to be tested, and set the M types of defects included in the image detection according to the type of the target object; N defect mask images are set according to the N inference results output by the model, and corresponding defect labels are set according to the defect type; the defect mask images have the same size dimension as the image to be tested, and the pixel feature matrix corresponding to the defect mask image is a matrix of all zeros; Select the target defect mask image based on the defect label, poll N inference results, and extract the coordinates, size, and confidence value of all target defects under the target defect type; the confidence values ​​of the size range determined by the same target defect are all the same. Based on the coordinates of the target defect, the confidence values ​​within the defect size range are transferred to the target position of the all-zero matrix corresponding to the target defect mask image; all inference results are polled until the confidence values ​​of all target defects of the same type are transferred to the target position of the all-zero matrix, thus forming the target fusion matrix; The target fusion matrix is ​​mapped to the target defect mask image, and the target defects in the image to be tested are labeled according to the mapped target defect mask image.

2. The method according to claim 1, characterized in that, The inference result includes the defect type, defect coordinates, defect size, and confidence value of the target defect identified from the image under test; different target models output inference results containing different target defect types, quantities, coordinates, sizes, and confidence values.

3. The method according to claim 1, characterized in that, The process of forming the target fusion matrix includes: When the i-th type of target defect is found from the inference results, the confidence values ​​within the defect size range are accumulated into the target region in the feature matrix; when the target regions mapped in multiple inference models overlap, the confidence values ​​are accumulated sequentially to obtain the voting result matrix of the i-th type of target defect. The fusion threshold is determined based on the number of target models participating in the vote, and compared with the matrix pixel data of the voting result matrix. The values ​​of all pixels less than the fusion threshold are set to 0, thus forming the target fusion matrix.

4. The method according to claim 3, characterized in that, The step of calibrating the target defects in the image to be tested based on the mapped target defect mask includes: Determine the target region composed of non-zero pixels in the target fusion matrix; The extreme values ​​of the coordinates in the two-dimensional coordinate system are determined based on the position coordinates of all pixels in the target area. A rectangular box is bounded around the extreme values ​​of the coordinates and the bounding box is determined as the coverage area of ​​the target defect. The coverage area of ​​the target defect is mapped onto the image to be tested, and the defect is labeled on the image to be tested according to the label information.

5. The method according to any one of claims 1-4, characterized in that, All defect categories are polled, and when a target defect category is selected, all inference results are polled. The image to be tested is then mapped and calibrated using the corresponding target defect mask image. In the output defect detection map, the identified defective objects are represented by the mapped outer rectangle and a defect label is placed. The defect label stores the defect classification and confidence information of the identified defective objects.

6. A surface defect detection device using multi-model voting, characterized in that, The device includes: The inference module is used to input the image to be tested into the multi-model detection system, perform image inference based on different types of target models, and output the inference results corresponding to each model. A module is created to identify target objects in the image to be tested, and the M types of defects included in the image detection are set according to the type of the target object. N defect mask images are set according to the N inference results output by the model, and corresponding defect labels are set according to the defect type; the defect mask images have the same size dimension as the image to be tested, and the pixel feature matrix corresponding to the defect mask image is a matrix of all zeros; The calibration module is used to select a target defect mask image based on the defect label, poll N inference results, and extract the coordinates, size, and confidence value of all target defects under the target defect type; the confidence values ​​of the size range determined by the same target defect are all the same. Based on the coordinates of the target defect, the confidence values ​​within the defect size range are transferred to the target position of the all-zero matrix corresponding to the target defect mask image; all inference results are polled until the confidence values ​​of all target defects of the same type are transferred to the target position of the all-zero matrix, thus forming the target fusion matrix; The target fusion matrix is ​​mapped to the target defect mask image, and the target defects in the image to be tested are labeled according to the mapped target defect mask image.

7. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the multi-model voting surface defect detection method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The readable storage medium stores at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or instruction set is loaded and executed by a processor to implement the multi-model voting surface defect detection method as described in any one of claims 1 to 5.

Images