Multi-model voting surface defect detection method, device and equipment and storage medium

Abstract

The invention discloses a multi-model voting surface defect detection method and device, equipment and a storage medium, and relates to the field of model detection. Inputting the to-be-detected picture into a multi-model detection system, and outputting a reasoning result corresponding to each model; determining defect classification according to the size of the to-be-detected image and the target type in the image, and sequentially creating defect mask patterns according to the defect classification; and sequentially selecting target defect mask patterns according to the defect classification, mapping a reasoning result output by the target model to the target defect mask patterns based on the corresponding defect types, and performing defect calibration on target defects in the to-be-detected picture according to the mapped target defect mask patterns. According to the scheme, multi-model and multi-dimensional detection results can be fused, the defect recognition result is filtered according to the voting result, and the accuracy of the obtained defect recognition result is higher.

Description

Technical Field

[0001] The present application relates to the field of model detection, and in particular to a surface defect detection method, device, equipment and storage medium of multi-model voting. Background Art

[0002] Surface defect detection is an important part of ensuring product quality in industrial production. With the rapid development of computer vision and deep learning technology, surface defect detection methods based on machine vision have been widely used in various industrial fields. Compared with traditional manual visual inspection, machine vision inspection has the advantages of fast speed, high precision, and long-term continuous operation, which can effectively improve the efficiency and quality of industrial production. The development of surface defect detection technology has gone through several stages, from the initial manual visual inspection to optical sensor inspection, and then to the current machine vision inspection. Machine vision inspection technology uses image sensors to obtain images of objects and uses image processing and analysis technology to identify surface defects. This method has the advantages of non-contact, high precision, and high efficiency, and is one of the main methods for surface defect detection in modern industry.

[0003] In terms of technical implementation, the key problems faced by surface defect detection include small sample size and real-time problems. In order to solve these problems, researchers have proposed a variety of methods, such as data enhancement, network pre-training and transfer learning, reasonable network structure design, unsupervised or semi-supervised methods, etc. The current research progress is mainly focused on methods based on deep learning, including defect classification methods based on CNN, defect detection methods based on target detection, defect detection methods based on semantic segmentation, defect detection methods based on generative adversarial networks (GAN), etc. In terms of future development trends, surface defect detection technology may develop in the direction of small sample learning, self-supervised learning, multimodal fusion, interpretability research, edge computing, active learning, 3D defect detection, and video sequence defect detection. However, no matter which mainstream technical solution is used, it cannot meet or be compatible with all the requirements of all defect image detection technologies, improve detection timeliness at the expense of accuracy, or increase detection accuracy at the expense of timeliness, or fail to take into account the balance problem under the premise of focusing on small target recognition. In addition, for high-precision recognition scenarios, different false detection rates of different detection models may also lead to inconsistent output results. Summary of the invention

[0004] The embodiments of the present application provide a surface defect detection method, apparatus, device and storage medium for multi-model voting to solve the problem that it is difficult to balance detection accuracy and false detection rate for different models.

[0005] In one aspect, the present application discloses a surface defect detection method using multi-model voting, the method comprising:

[0006] Input the image to be tested into the multi-model detection system, perform image reasoning according to different types of target models, and output the reasoning results corresponding to each model;

[0007] Determining defect classification according to the size of the image to be tested and the type of the target in the image, and sequentially creating defect mask images according to the defect classification;

[0008] Select the target defect mask map in turn according to the defect classification, map the inference result output by the target model to the target defect mask map based on the corresponding defect type, and perform defect calibration on the target defect in the image to be tested according to the mapped target defect mask map

[0009] Specifically, the defect classification is determined according to the size of the image to be tested and the target type in the image, and defect mask images are created in sequence according to the defect classification; including:

[0010] Identify the target object in the image to be tested, and set M types of defects included in the image detection according to the type of the target object;

[0011] N defect mask images are set in a 1:1 ratio according to the size of the image to be tested and the number of inference results, and corresponding defect labels are set according to the defect type; the defect mask image has the same size dimension as the image to be tested, and the pixel feature matrix corresponding to the defect mask image is an all-0 matrix.

[0012] Specifically, the inference result includes the defect type, defect coordinates, defect size and confidence value of the target defect identified from the image to be tested; different target model output inference results include different target defect types, quantities, coordinates, sizes and confidence values.

[0013] Specifically, mapping the inference result output by the target model to the target defect mask image based on the corresponding defect type includes:

[0014] The target defect mask image is selected according to the defect label, N inference results output by N target models are polled, and coordinates, sizes and confidence values ​​of all target defects under the target defect type are extracted; wherein the confidence values ​​of the size range determined by the same target defect are all the same;

[0015] According to the coordinates of the target defect, the confidence value within the defect size range is transferred to the target position of the all-0 matrix corresponding to the target defect mask image;

[0016] Poll all the inference results until all the confidence values ​​of all target defects under the same target defect type in the N inference results are transferred to the target position of the all-0 matrix to form a target fusion matrix;

[0017] The target fusion matrix is ​​mapped to the target defect mask image.

[0018] Specifically, the process of forming the fusion matrix includes:

[0019] When the i-th target defect is found from the inference results, the confidence values ​​within the defect size range are accumulated to the target area in the feature matrix; when the target areas mapped in multiple inference models overlap, the confidence values ​​are accumulated in sequence to obtain the voting result matrix of the i-th target defect;

[0020] The fusion threshold is determined according to the number of target models participating in the voting, and compared with the matrix pixel data of the voting result matrix. The values ​​of all pixels less than the fusion threshold are set to 0 to form the target fusion matrix.

[0021] Specifically, the defect calibration of the target defect in the image to be tested according to the mapped target defect mask image includes:

[0022] Determine a target area composed of non-zero pixels in the target fusion matrix;

[0023] Determine the coordinate extreme value in the two-dimensional coordinate system according to the position coordinates of all pixel points in the target area, use the coordinate extreme value to circumscribe a rectangular frame, and determine it as the coverage area of ​​the target defect;

[0024] The coverage area of ​​the target defect is mapped to the image to be tested, and the defect is calibrated on the image to be tested according to the label information.

[0025] Specifically, all defect classifications are polled, and when a target defect classification is selected, all inference results are polled, and the image to be tested is mapped and calibrated through the corresponding target defect mask image;

[0026] In the output defect detection image, the identified defect object is represented by a mapped external rectangular frame, and a defect label is placed, in which the defect classification and confidence information of the identified defect object is stored.

[0027] On the other hand, the present application provides a surface defect detection device using multi-model voting.

[0028] The inference module is used to input the image to be tested into the multi-model detection system, perform image inference according to different types of target models, and output the inference results corresponding to each model;

[0029] A creation module, used to determine the defect classification according to the size of the image to be tested and the target type in the image, and to create defect mask images in sequence according to the defect classification;

[0030] A calibration module is used to select a target defect mask image in sequence according to defect classification, map the inference result output by the target model to the target defect mask image based on the corresponding defect type, and perform defect calibration on the target defect in the test image according to the mapped target defect mask image.

[0031] On the other hand, the present application provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the surface defect detection method of multi-model voting described in the above aspects.

[0032] On the other hand, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the surface defect detection method of multi-model voting described in the above aspects.

[0033] The beneficial effects of the technical solution provided by the embodiment of the present application include at least the following: the present application uses a composite system integrating multiple detection models to infer the input image to be tested, and outputs the respective inference results according to the selected target model; then the defect classification is determined by the type of image content, and a one-to-one corresponding defect mask map is created; then the mapped defect mask map is used to poll each inference result, and the target defect is fused and voted, and finally the mask map after fusion voting is used to map back to the image to be tested to achieve defect calibration. This solution can fuse the detection results of multiple models and multiple dimensions, filter the defect recognition results according to the voting results, and obtain a higher accuracy defect recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of a surface defect detection method of multi-model voting provided in an embodiment of the present application;

[0035] Figure 2 The algorithm flow chart of the surface defect detection method of multi-model voting is listed;

[0036] Figure 3 is a schematic diagram showing a mapping of a defect mask image;

[0037] Figure 4 A schematic diagram of transferring the confidence value to the target defect mask image in a possible form is listed;

[0038] Figure 5 A schematic diagram showing the mapping of different inference model results to the same target defect type is listed;

[0039] Figure 6 The schematic diagram of target fusion matrix mapping and defect calibration is listed;

[0040] Figure 7 The schematic diagram of the wafer to be tested before and after the image detection is shown;

[0041] Figure 8 It is a structural block diagram of a surface defect detection device of multi-model voting provided in an embodiment of the present application;

[0042] Fig. 9 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0044] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0045] Figure 1 : is a flow chart of the surface defect detection method of multi-model voting provided in an embodiment of the present application, comprising the following steps:

[0046] S1. Input the image to be tested into the multi-model detection system, perform image reasoning according to different types of target models, and output the reasoning results corresponding to each model;

[0047] In the field of small target detection, especially for wafer surface defect detection, wafer particles and the like will inevitably have defects in the production process, such as welds, cracks, fractures and dirt, especially for different types of special defects that will appear in a concentrated manner, such as mirror loss, damage, offset and other problems on wafer pads. These small target high-pixel detection methods usually use defect detection methods based on target detection, defect detection methods based on semantic segmentation, defect detection methods based on generative adversarial networks (GAN), etc., but each model focuses on a certain direction and cannot take into account all advantages, especially the problems of insufficient detection capabilities for subtle defects and large differences in detection results. Based on this perspective, this application considers the problem and establishes a multi-model composite defect detection system, which integrates detection models widely used in the semiconductor and wafer detection fields, uses multiple models to detect the same image to be tested, and performs voting fusion based on inference output to improve detection accuracy.

[0048] In some embodiments, the multi-model detection system may include at least a mask-rcnn model, a yoloV5 model, a yoloX model, a detr model, and a dino model, etc., all of which can be used for surface defect detection. In actual selection, at least two models can be selected according to demand, and after the image to be tested is input into the selected target model, subsequent fusion operations are performed according to the inference results.

[0049] It should be noted that for the multiple target models selected, the same defect detection data set needs to be used for training in order to maintain consistency at the data level, so as to ensure the defect detection capability of the model.

[0050] S2, determining defect classification according to the size of the image to be tested and the target type in the image, and creating defect mask images in sequence according to the defect classification;

[0051] Because different images to be tested may produce different types of defects, such as ordinary wafer particles and gold surface pads, when the type of target object is different, the types of defects that may be produced are different, and the defects identified from the image are naturally different. Therefore, before performing subsequent fusion calculations, the present application sets a defect classification set according to the type of target object in the input image to be tested in advance. After each model outputs the inference result, the corresponding set is directly selected according to the type of target object, and then a defect mask map is created according to the defect type in the set. The advantage of this is that the number of output defect types can be reduced, especially when multiple models have multiple defect detection functions, there will be false detections. This solution can filter out defect outputs that should not exist or do not affect product functions and quality, thereby improving overall detection accuracy. The defect mask map is an intermediate image specifically used for voting fusion based on the inference results. The mask map can be used to extract areas of interest in the image, shield certain areas on the image, and detect and extract structural features in the image that are similar to the mask.

[0052] In a possible implementation, assume that the inference output of model A is {a defect, b defect, c defect}, and the inference output of model B is {a defect, e defect, g defect}, but the defect set of the target object that is usually concerned is {a defect, g defect}, then when creating a defect mask map, only defect mask maps for a defect and g defect are created, and other types of defects are no longer considered, and no corresponding mask maps are created.

[0053] S3. Select the target defect mask map in turn according to the defect classification, map the inference result output by the target model to the target defect mask map based on the corresponding defect type, and calibrate the target defect in the test image according to the mapped target defect mask map.

[0054] Assuming that M types of defect classifications are determined according to the target type, M defect masks are created accordingly. Then, a certain type of target defect (such as crack defect) is selected in turn, and the corresponding crack defect mask is used as the base, and the crack defects contained in all the reasoning results are polled. According to the position of the crack defect identified in the image to be tested, the defect result is mapped to the crack defect mask, and a crack defect mask with all models for crack detection is obtained. The mapping of each defect is voted and defect fused. This method can be filtered by setting a certain threshold to filter out the results with insufficient detection accuracy. On the contrary, the detected target defects can be strengthened under the reasoning results of multiple models to ensure the accuracy of detection. Finally, the fused crack defect mask is mapped back to the image to be tested to achieve defect calibration.

[0055] Figure 2 The algorithm flow chart of the surface defect detection method of multi-model voting is listed. Assuming that there are N inference results and M defect classifications, the scheme takes defect classification as the goal, polls N inference results in turn, and completes the voting and fusion of the target defect classification. By polling M defect classifications, defect mapping is performed on M defect mask images in turn until the detection of M defects is completed.

[0056] In summary, this application uses a composite system that integrates multiple detection models to infer the input images to be tested, and outputs the respective inference results according to the selected target model; then the defect classification is determined by the type of image content, and a one-to-one corresponding defect mask map is created; then the mapped defect mask map is used to poll each inference result, and the target defect is fused and voted, and finally the mask map after fusion voting is used to map back to the image to be tested to achieve defect calibration. This solution can fuse the detection results of multiple models and multiple dimensions, filter the defect recognition results according to the voting results, and obtain a more accurate defect recognition result.

[0057] In a possible implementation manner, the process of creating a defect mask image can be specifically summarized as follows:

[0058] A. Identify the target object in the image to be tested, and set the M types of defects included in the image detection according to the type of the target object;

[0059] B. Set N defect mask images in a 1:1 ratio according to the size of the image to be tested and the number of inference results, and set the corresponding defect labels according to the defect type.

[0060] like Figure 3 As shown, the defect mask image has the same dimension as the image to be tested, and the pixel feature matrix corresponding to the initial N defect mask images is a matrix of all 0. Pixels at different positions in the matrix can be mapped back to the image to be tested according to the coordinate position.

[0061] Although the inference results are output by different models, the output content form and structure are consistent, including the defect type, defect coordinates, defect size and confidence value of the target defect identified from the image to be tested. For different target models, the output inference results usually contain different target defect types, quantities, coordinates, sizes and confidence values ​​according to different functions, accuracy and characteristics.

[0062] Assuming that there are N target models involved in reasoning and output N reasoning results, the first reasoning result can be expressed as {a1 defect (x1, y1, w1, h1, s1), b defect (x2, y2, w2, h2, s2), c defect (x3, y3, w3, h3, s3), a2 defect (x4, y4, w4, h4, s4)}. The above structure indicates that a total of 4 target defects are identified, where a1 and a2 represent two defects of the same type (type a). (x, y, w, h) represents the parameter information of the target defect, where x and y represent the coordinates of the upper left corner of the target, w represents the defect width, h represents the defect width and height, and s represents the confidence (score) value. In order to facilitate subsequent matrix fusion and result voting, a corresponding defect label is set for each defect mask image created, namely labelM1, labelM2, ..., label Mm.

[0063] In a possible implementation, the step of mapping the inference result to the target defect mask image may include the following steps:

[0064] A. Select the target defect mask image according to the defect label, poll the N inference results output by the N target models, and extract the coordinates, sizes and confidence values ​​of all target defects under the target defect type;

[0065] B. According to the coordinates of the target defect, the confidence value within the defect size range is transferred to the target position of the all-0 matrix corresponding to the target defect mask image;

[0066] In some embodiments, the detected defects are not completely regular rectangles, so the target model detection output is determined by selecting a rectangle and then determining the coordinates and size of the upper right corner or upper left corner. For a determined target model, a confidence value will be output after a defect is identified. The confidence value represents the confidence value of the defect size range corresponding to the entire target defect area, and the confidence values ​​of the size range determined by the same target defect are all the same.

[0067] Figure 4A schematic diagram of transferring the confidence value to the target defect mask in a possible form is listed. Assuming that the M1 classification is selected, the x, y, w, h and score values ​​of this M1 classification are obtained. According to x, y, w, h, a same area A is obtained on the corresponding pure 0 matrix, and the values ​​of all pixels in area A are changed from 0 to the score value. If a certain inference result contains multiple M1 classifications, multiple corresponding areas A must also be obtained on the corresponding pure 0 matrix. If there is overlap between the areas, the score values ​​of the overlapping parts are added together. Figure 4 The confidence values ​​in the two defect size ranges are 0.8 and 0.4 respectively, so after positioning, they can be directly mapped to the target defect mask image, and the model inference results can be transferred to the mask image for subsequent calculations.

[0068] C. Poll all the inference results until all the confidence values ​​of all target defects under the same target defect type in the N inference results are transferred to the target position of the all-0 matrix to form a target fusion matrix;

[0069] Figure 5 A schematic diagram of mapping the results of different inference models to the same target defect type is listed. This step mainly considers the accuracy and differences of different target models, and summarizes the inference results of multiple models into a mask image to achieve the effect of complementary advantages of each model.

[0070] Similarly, the fusion process of different inference models is summarized as follows:

[0071] 1. When the i-th target defect is found from the inference results, the confidence values ​​within the defect size range are accumulated to the target area in the feature matrix; when the target areas mapped in multiple inference models overlap, the confidence values ​​are accumulated in sequence to obtain the voting result matrix of the i-th target defect;

[0072] 2. Determine the fusion threshold according to the number of target models participating in the voting, and compare it with the matrix pixel data of the voting result matrix. Set the values ​​of all pixels less than the fusion threshold to 0 to form the target fusion matrix.

[0073] It is known that N dimensions are the inference results of multiple models on a single classification. These N matrices are directly added together to obtain the comprehensive result matrix of multiple models on this classification, that is, the inference results on the fusion N dimensions are obtained. This can be considered as the voting result of multiple models on this classification. Usually, the size of the fusion threshold is positively correlated with the number of models. If the values ​​of all pixels on the voting result matrix that are less than the fusion threshold are set to 0, then all values ​​greater than 0 on the matrix are considered to be the true M1 classification.

[0074] D. Map the target fusion matrix to the target defect mask image.

[0075] Figure 6 The schematic diagram of target fusion matrix mapping and defect calibration is listed. The calibration process can be summarized as follows:

[0076] A. Determine the target area composed of non-zero pixels in the target fusion matrix;

[0077] B. Determine the coordinate extreme value in the two-dimensional coordinate system according to the position coordinates of all pixel points in the target area, use the coordinate extreme value to circumscribe a rectangular frame, and determine it as the coverage area of ​​the target defect;

[0078] C. Map the coverage area of ​​the target defect to the image to be tested, and calibrate the defect on the image to be tested based on the label information.

[0079] In the specific calibration process, M defect images can be generated in sequence according to the defect classification, and the mark of the specific defect classification can be calibrated on each defect image. Optionally, all defect classifications can be polled, and when the target defect classification is selected, all inference results can be polled, and the image to be tested can be mapped and calibrated through the corresponding target defect mask image. In this approach, in the output defect detection image, the identified defect object is represented by a mapped external rectangular box, and a defect label is placed, and the defect classification and confidence information of the identified defect object are stored in the defect label.

[0080] The following is an explanation of defect detection calibration using the image of the wafer to be tested as input.

[0081] 1. Use yoloX and co-detr models for multi-model voting;

[0082] 2. Use the same dataset to train these two models to obtain yolox-model and co-detr-model;

[0083] 3. Use yolox-model and co-detr-model to infer the same picture;

[0084] 4. Normalize the inference results and convert the inference results of both models into the format of (label, score, bbox); the size of the rectangular box represented by bbox

[0085] 5. For the first classification, create two matrices with the same size as the original image and values ​​equal to 0 according to the number of models;

[0086] 6. According to the results of (label, score, bbox), modify the positions corresponding to the two pure 0 matrices to the values ​​of the score;

[0087] 7. Add these two matrices together to get the voting result matrix for the first category;

[0088] 8. Set the threshold thres to 0.6, and set the value of the position less than thres to 0;

[0089] 9. Get the x, y, w, h values ​​of each area in the voting result matrix;

[0090] 10. Continue processing the next category until the last one, and get the x, y, w, h values ​​of all categories. Because the label is known and the score has been filtered by the thres value, multiple (label, bbox) values ​​can be obtained in the end. Based on these values, visual drawing can be done on the original image.

[0091] Figure 7 The schematic diagram shows the wafer image before and after detection. In the output defect calibration image, the boxed color can be used as the label type, so that all defect types can be displayed in the same image for easy observation.

[0092] In summary, this solution mainly solves the problem when a model has obvious bottlenecks (for example, it has always performed poorly in the detection of a certain defect). By introducing multiple models to jointly vote for detection, the shortcomings of a single model can be effectively alleviated, making the final detection results more balanced. This has a certain solution effect on the small sample problem and sample imbalance problem.

[0093] Using multi-model voting can make it easier to locate the root cause of the problem. If the detection of a certain type of defect still performs poorly when using multi-model voting, then the model can basically be ruled out as the cause. It is most likely caused by the data, and more energy can be spent on data optimization.

[0094] Figure 8 The structural block diagram of the surface defect detection device of multi-model voting provided in an embodiment of the present application is shown, and the device includes:

[0095] The inference module 810 is used to input the image to be tested into the multi-model detection system, perform image inference according to different types of target models, and output the inference results corresponding to each model;

[0096] A creation module 820, configured to determine defect classification according to the size of the image to be tested and the type of target in the image, and to sequentially create defect mask images according to the defect classification;

[0097] The calibration module 830 is used to select the target defect mask image in sequence according to the defect classification, map the inference result output by the target model to the target defect mask image based on the corresponding defect type, and perform defect calibration on the target defect in the test image according to the mapped target defect mask image.

[0098] The wafer loading deviation correction device provided in the embodiment of the present application can be applied to the wafer loading deviation correction method provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0099] It should be noted that the wafer loading misalignment correction device provided in the embodiment of the present application is only illustrated by the division of the above-mentioned functional modules / functional units. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the wafer loading misalignment correction device is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the wafer loading misalignment correction method provided in the above-mentioned method embodiment and the implementation method of the wafer loading misalignment correction device provided in this embodiment belong to the same concept. The specific implementation process of the wafer loading misalignment correction device provided in this embodiment is detailed in the above-mentioned method embodiment, which will not be repeated here.

[0100] Fig. 9 The block diagram of a computer device provided by an exemplary embodiment of the present application is shown. It is a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. Among them, the processor and the memory may be connected by a bus or in other ways. Among them, the processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0101] The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0102] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0103] In some embodiments, the computer device may further optionally include: a peripheral device interface and at least one peripheral device. The processor, the memory and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit, a display screen and a keyboard.

[0104] The peripheral device interface can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor and the memory. In some embodiments, the processor, the memory, and the peripheral device interface are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor, the memory, and the peripheral device interface can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0105] The display screen is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen is a touch screen, the display screen also has the ability to collect touch signals on the surface or above the surface of the display screen. The touch signal can be input to the processor as a control signal for processing. At this time, the display screen can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen can be one, arranged on the front panel of the computer device; in other embodiments, the display screen can be at least two, respectively arranged on different surfaces of the computer device or in a folding design. In other embodiments, the display screen can be a flexible display screen, arranged on a curved surface or a folding surface of the computer device. Even, the display screen can be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0106] The power supply is used to power various components in the computer device. The power supply can be AC, DC, a disposable battery, or a rechargeable battery. When the power supply includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0107] Those skilled in the art will appreciate that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0108] The embodiment of the present application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, and when the computer program is executed by the processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art can understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the process of the implementation of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0109] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A surface defect detection method based on multi-model voting, characterized in that: The method comprises: Input the image to be tested into the multi-model detection system, perform image reasoning according to different types of target models, and output the reasoning results corresponding to each model; Determining defect classification according to the size of the image to be tested and the type of the target in the image, and sequentially creating defect mask images according to the defect classification; A target defect mask image is selected in turn according to the defect classification, the inference result output by the target model is mapped to the target defect mask image based on the corresponding defect type, and defect calibration is performed on the target defect in the image to be tested according to the mapped target defect mask image.

2. The method according to claim 1, characterized in that The method of determining defect classification according to the size of the image to be tested and the target type in the image, and sequentially creating defect mask images according to the defect classification comprises: Identify the target object in the image to be tested, and set M types of defects included in the image detection according to the type of the target object; N defect mask images are set in a 1:1 ratio according to the size of the image to be tested and the number of inference results, and corresponding defect labels are set according to the defect type; the defect mask image has the same size dimension as the image to be tested, and the pixel feature matrix corresponding to the defect mask image is an all-0 matrix.

3. The method according to claim 1, characterized in that The inference result includes the defect type, defect coordinates, defect size and confidence value of the target defect identified from the image to be tested; different target model output inference results include different target defect types, quantities, coordinates, sizes and confidence values.

4. The method according to claim 2, characterized in that: The mapping of the inference result output by the target model to the target defect mask image based on the corresponding defect type includes: The target defect mask image is selected according to the defect label, N inference results output by N target models are polled, and coordinates, sizes and confidence values ​​of all target defects under the target defect type are extracted; wherein the confidence values ​​of the size range determined by the same target defect are all the same; According to the coordinates of the target defect, the confidence value within the defect size range is transferred to the target position of the all-0 matrix corresponding to the target defect mask image; Poll all the inference results until all the confidence values ​​of all target defects under the same target defect type in the N inference results are transferred to the target position of the all-0 matrix to form a target fusion matrix; The target fusion matrix is ​​mapped to the target defect mask image.

5. The method according to claim 4, characterized in that The process of forming a fusion matrix includes: When the i-th target defect is found from the inference results, the confidence values ​​within the defect size range are accumulated to the target area in the feature matrix; when the target areas mapped in multiple inference models overlap, the confidence values ​​are accumulated in sequence to obtain the voting result matrix of the i-th target defect; The fusion threshold is determined according to the number of target models participating in the voting, and compared with the matrix pixel data of the voting result matrix. The values ​​of all pixels less than the fusion threshold are set to 0 to form the target fusion matrix.

6. The method according to claim 5, characterized in that The defect calibration of the target defect in the image to be tested according to the mapped target defect mask image includes: Determine a target area composed of non-zero pixels in the target fusion matrix; Determine the coordinate extreme value in the two-dimensional coordinate system according to the position coordinates of all pixel points in the target area, use the coordinate extreme value to circumscribe a rectangular frame, and determine it as the coverage area of ​​the target defect; The coverage area of ​​the target defect is mapped to the image to be tested, and the defect is calibrated on the image to be tested according to the label information.

7. The method according to any one of claims 1 to 6, characterized in that: Poll all defect categories, and when a target defect category is selected, poll all inference results, and map and calibrate the image to be tested through the corresponding target defect mask image; In the output defect detection image, the identified defect object is represented by a mapped external rectangular frame, and a defect label is placed, in which the defect classification and confidence information of the identified defect object is stored.

8. A surface defect detection device using multi-model voting, characterized in that: The device comprises: The inference module is used to input the image to be tested into the multi-model detection system, perform image inference according to different types of target models, and output the inference results corresponding to each model; A creation module, used to determine the defect classification according to the size of the image to be tested and the target type in the image, and to create defect mask images in sequence according to the defect classification; A calibration module is used to select a target defect mask image in sequence according to defect classification, map the inference result output by the target model to the target defect mask image based on the corresponding defect type, and perform defect calibration on the target defect in the test image according to the mapped target defect mask image.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the surface defect detection method of multi-model voting as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the surface defect detection method of multi-model voting as described in any one of claims 1 to 7.

Images