Method and apparatus for training defect detection model, and electronic device
Patent Information
- Application Number
- MYPI2025004769
- Authority / Receiving Office
- MY · MY
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-02-17
- Filing Date
- 2024-02-05
- Publication Date
- 2024-08-17
AI Technical Summary
In the process of grain image defect identification and classification, direct fusion of the defect-free image and the image to be inspected results in a large amount of calculation and low accuracy. Existing technologies cannot effectively solve this problem.
By obtaining the training sample set, including defect images and corresponding template images, the target loss function is determined, and the stochastic gradient descent method is used to train the model, extract the features of the defect images and template images, output defect category and location information, and use the template image Assist identification and classification.
It improves the accuracy of defect identification and classification in grain images, reduces the amount of calculation, enables accurate extraction of defect features in complex backgrounds, and solves the shortcomings of traditional methods that rely on manual feature extraction.
Smart Images

Figure PI2025004769_DRAWING_001 
Figure PI2025004769_DRAWING_002 
Figure PI2025004769_DRAWING_003
Abstract
Description
Training method, device and electronic equipment for defect detection model
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 17, 2023, with application number 202310184844.5 and application name “Training method, device and electronic device for defect detection model”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of image recognition, and more specifically, to a method, device, and electronic device for training a defect detection model. Background Art
[0003] In related art, when using images of flawless grains to identify and classify defects in images of untested grains, template images are typically not used to assist in identification. Instead, the commonly used method is to directly fuse the images of the untested grains with the images of flawless grains, and then determine the defect category and location in the untested images based on the fused images. Although this method utilizes flawless images to identify and classify defects in the untested images, the fusion of the images and the identification of the fused images are computationally intensive and exhibit low accuracy.
[0004] To address the above-mentioned problems, no effective solutions have been proposed so far.
[0005] Summary of the Invention
[0006] The embodiments of the present application provide a training method, device and electronic device for a defect detection model to at least solve the technical problems of large computational complexity and low accuracy caused by directly fusing defect-free images and images to be detected when performing defect identification and classification on grain images in related technologies.
[0007] According to one aspect of an embodiment of the present application, a training method for a defect detection model is provided, including: obtaining a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of defects at each location in the defect image; determining a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; using a stochastic gradient descent method to train a model to be trained using the training sample set, defect label information and target loss function to obtain a target defect detection model, wherein the target defect detection model is used to determine the defect location information and defect category information in the defect image to be detected, and the model to be trained is used to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, and the second label information includes second category information and second position information of defects at each location in the defect image determined by the model to be trained.
[0008] Optionally, the stochastic gradient descent method is used to train the model to be trained using the training sample set, defect label information and target loss function, and the steps of obtaining the target defect detection model include: a first step, inputting the defect image and the template image into the model to be trained respectively, and obtaining the second label information output by the model to be trained; a second step, inputting the first category information and the second category information into the defect category loss function to obtain the first loss function value, and inputting the first position information and the second position information into the defect position loss function to obtain the second loss function value, and storing the first loss function value and the second loss function value; a third step, using the stochastic gradient descent method, adjusting the model parameters of the model to be trained using the first loss function value and the second loss function value; a fourth step, obtaining all the stored first loss function values and the second loss function values, and when the first loss function value and the second loss function value meet the preset conditions, determining that the adjusted model to be trained is the target defect detection model, otherwise jumping to the first step, wherein the preset conditions include that the sum of a consecutive preset number of first loss function values and the second loss function values are all within the target value range.
[0009] Optionally, the steps of inputting the defect image and the template image into the model to be trained respectively and obtaining the second label information output by the model to be trained include: inputting the defect image and the template image into the backbone network of the model to be trained, wherein the backbone network is configured to extract image features from the defect image and the template image, thereby obtaining a defect feature image and a template feature image; and obtaining the second label information output by the model to be trained based on the defect feature image and the template feature image.
[0010] Optionally, the model to be trained also includes a feature fusion layer and a feature pyramid layer, wherein the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; the feature pyramid layer is configured to output second label information based on the first target feature image.
[0011] Optionally, the feature fusion layer includes a first feature fusion layer, wherein the first feature fusion layer is configured to perform a differential operation on the defect feature image and the template feature image to obtain a differential feature image, wherein the pixel value of any pixel point in the differential feature image is equal to the absolute value of the difference between the defect pixel point corresponding to any pixel point and the template pixel point, the defect pixel point is the pixel point corresponding to any pixel point in the defect feature image, and the template pixel point is the pixel point corresponding to any pixel point in the template feature image; channel stitching processing is performed on the differential feature image and the defect feature image to obtain a second target feature image; and convolution processing is performed on the second target feature image through the target convolution kernel to obtain the first target feature image.
[0012] Optionally, the feature fusion layer includes a second feature fusion layer, wherein the second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image respectively through the target convolution kernel, and generate a first target feature image based on the defect feature image and the template feature image after convolution processing, wherein the pixel value of any pixel point in the first target feature image is equal to the sum of the pixel values of the defect pixel point and the template pixel point corresponding to any pixel point.
[0013] Optionally, the step of determining that the adjusted model to be trained is the target defect detection model includes: obtaining a verification sample set, and third defect label information corresponding to each group of samples in the verification sample set, wherein each group of samples in the verification sample set includes a defect image, and a template image corresponding to the defect image, and the third defect label information includes third category information and third position information of defects in each location in the defect image; in the case where there are multiple adjusted models to be trained, determining the average precision mean of each adjusted model to be trained in the multiple adjusted models to be trained through the verification sample set and the third defect label information; and determining the adjusted model to be trained with the largest average precision mean as the target defect detection model.
[0014] Optionally, the step of obtaining a training sample set includes: obtaining a first defect image and a first template image corresponding to the first defect image; performing geometric transformation processing on the first defect image and the first template image using the same processing method, and using the first defect image processed with geometric transformation as a defect image sample, and using the first template image processed with combined transformation as a template image sample, wherein the processing method includes at least one of the following: random cropping, image flipping, and image splicing.
[0015] Optionally, the category information includes a defect category number of the defect, and the location information includes location information of a defect box corresponding to the defect.
[0016] According to another aspect of an embodiment of the present application, a defect image classification method is also provided, including: determining a defect image to be detected, and a template image corresponding to the defect image to be detected; inputting the defect image to be detected and the template image into a target defect detection model, and obtaining defect location information and defect category information output by the target defect detection model, wherein the target defect detection model is used to extract a third image feature of the defect image to be detected and a fourth image feature of the template image, and output defect location information and defect category information based on the three image features and the fourth image feature; determining a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected based on the defect location information; determining the category confidence of each target defect in the preset number of target defects, and determining the defect category of the target defect with the largest category confidence as the defect category of the defect image to be detected.
[0017] According to another aspect of an embodiment of the present application, a training device for a defect detection model is also provided, including: an input module, configured to obtain a training sample set, and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of defects at each location in the defect image; a first processing module, configured to determine a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; a second processing module, configured to use a stochastic gradient descent method to train a model to be trained using the training sample set, defect label information and target loss function to obtain a target defect detection model, wherein the target defect detection model is configured to determine the defect position information and defect category information in the defect image to be detected.
[0018] According to another aspect of an embodiment of the present application, a non-volatile storage medium is also provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute a defect model training method or a defect image classification method.
[0019] According to another aspect of an embodiment of the present application, an electronic device is also provided, which includes a memory and a processor, and the processor is configured to run a program stored in the memory, wherein when the program is running, a training method for a defect detection model or a defect image classification method is executed.
[0020] In an embodiment of the present application, a training sample set and first defect label information corresponding to each group of samples in the training sample set are obtained, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of defects at each location in the defect image; a target loss function is determined, wherein the target loss function includes a defect category loss function and a defect position loss function; a stochastic gradient descent method is used to train a model to be trained using the training sample set, defect label information and target loss function to obtain a target defect detection model, wherein the target defect detection model is used to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is used to extract the defect image The first image feature and the second image feature of the template image are used, and the second label information of the defect image is output based on the first image feature and the second image feature. The second label information includes the second category information and the second position information of each defect in the defect image determined by the model to be trained. By adopting a training sample set including defect images and template images and defect label information to train the defect detection model, the purpose of obtaining a defect detection model that can use the template image in the defect recognition process is achieved, thereby achieving the technical effect of using the template image to assist classification and recognition in the process of defect recognition and classification of grain images, and thus solving the technical problem of poor recognition and classification results caused by not using the template image when recognizing and classifying defects in grain images in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0022] FIG1 is a schematic structural diagram of a computer terminal according to an embodiment of the present application;
[0023] FIG2 is a flow chart of a method for training a defect detection model according to an embodiment of the present application;
[0024] FIG3 a is a schematic diagram of an image of a gold grain according to an embodiment of the present application;
[0025] FIG3 b is a schematic diagram of a defect image and a template image according to an embodiment of the present application;
[0026] FIG4 is a flow chart of a method for acquiring a training sample set according to an embodiment of the present application;
[0027] 5 is a flowchart of a method for training a defect detection model based on a training sample set and defect label information according to an embodiment of the present application;
[0028] FIG6 is a schematic flow chart of a method for obtaining second tag information according to an embodiment of the present application;
[0029] FIG7 is a schematic diagram of a model structure of a defect detection model according to an embodiment of the present application;
[0030] FIG8 is a schematic diagram of a model structure of another defect detection model according to an embodiment of the present application;
[0031] FIG9 is a flow chart of a method for screening a defect detection model according to an embodiment of the present application;
[0032] FIG10 is a flow chart of a defect image classification method according to an embodiment of the present application;
[0033] FIG11 is a flow chart of a model training and classification process according to an embodiment of the present application;
[0034] FIG12 is a schematic structural diagram of a defect model training device according to an embodiment of the present application. DETAILED DESCRIPTION
[0035] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.
[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0037] With the development of semiconductor device technology, more and more processes are used to manufacture semiconductor devices. Their main purpose is to create circuits and electronic components such as transistors, capacitors, and logic switches within the crystal grain. For example, oxidation and chemical vapor deposition are performed on the crystal grain surface, followed by coating, exposure, development, etching, ion implantation, metal sputtering, and other steps. Finally, several layers of circuits and components are processed and manufactured on the crystal grain. However, due to the complexity of each process in the manufacturing process, each process flow may produce some unexpected structures at different layers in the crystal grain. These structures can cause the circuits on the chip to malfunction. Such structures are usually called crystal grain defects.
[0038] To eliminate die defects, die defect detection is implemented after many key processes in the chip manufacturing process to monitor critical processes and ensure their accuracy. However, due to the extremely complex chip manufacturing process and the wide variety of die defect types, there is currently no unified classification method. Currently, AOI (Automated Optical Inspection) equipment typically uses traditional vision-based detection algorithms, such as image difference (between target and gold grains) or threshold segmentation, to locate defects on die. However, due to the complex background of the die, the diverse processes, and the wide variety of die patterns, while traditional vision algorithms can achieve defect location, manual feature extraction and classification is almost impossible. Often, even after formulating a series of rules (including information such as the defect pattern, its shape and size, the grayscale value of the defect location area, and the defect signal intensity), the classification results are inaccurate, resulting in many over-detections and false detections. Therefore, die defect classification often relies heavily on manual re-inspection. Manual classification is slow, has poor consistency and reliability, and is easily affected by external factors, resulting in certain technical bottlenecks in die defect classification.
[0039] In addition, with the rapid research and development of deep learning algorithms and hardware equipment in recent years, especially the iterative optimization of neural networks in the fields of target classification, target detection, and target segmentation, automated defect detection / classification in the semiconductor field has increasingly used deep learning algorithm technology to achieve increasingly higher detection / classification accuracy.
[0040] However, the related art relies on feature extraction technology when performing defect identification and classification, and defects on grains with complex backgrounds are only identified by a number of manually selected features, and the resulting model classification performance is limited. Convolutional neural networks can automatically learn to extract features suitable for classification, thereby improving detection and classification performance. However, in actual production, the background of grains is complex, the textures are numerous, the defect features are not obvious, and the features of different types of defects are similar and difficult to distinguish. How to ignore useless background and accurately extract the features of the defective area of interest is the key difficulty in the problem of grain defect detection and classification. Therefore, there is a problem of being unable to efficiently and accurately determine the fault type in the grain image and classify the fault. In order to solve this problem, relevant solutions are provided in the embodiments of the present application, which are described in detail below.
[0041] According to an embodiment of the present application, a method embodiment of a method for training a defect detection model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0042] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 shows a hardware structure block diagram of a computer terminal (or mobile device) configured to implement a training method for a defect detection model. As shown in Figure 1, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used in the figure to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that the structure shown in Figure 1 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in Figure 1, or have a configuration different from that shown in Figure 1.
[0043] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0044] The memory 104 can be configured to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the defect detection model training method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the defect detection model training method of the above-mentioned application. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0045] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module configured to communicate with the Internet wirelessly.
[0046] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).
[0047] In the above operating environment, an embodiment of the present application provides a method for training a defect detection model, as shown in FIG2 , the method comprising the following steps:
[0048] Step S202: obtaining a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first location information of each defect in the defect image;
[0049] In the technical solution provided in step S202, the defect image and the template image in the sample image are shown in Figure 3b. As can be seen from Figure 3b, the template image and the defect image correspond to the same position in the grain image, the only difference being that there is no defect or flaw in the template image.
[0050] As an optional implementation, after determining the position of the defect image within the grain image, a corresponding template image can be obtained from the golden grain image shown in Figure 3a based on the position information. The golden grain image is an image free of defects or flaws. The golden grain image is typically stored in the AOI inspection equipment.
[0051] In addition, the first category information described in step S202 includes a category number of the defect. Specifically, the defect categories and corresponding category numbers to be identified in the die image can be determined in advance. For example, assuming that the defect categories to be identified include eight categories, namely, dirty openings, dirty lines, broken openings, etched lines, damaged adhesive surfaces, poor development, line bridging, and edge pressing, the corresponding category numbers can be {0, 1, 2, 3, 4, 5, 6, 7} in sequence.
[0052] The first position information in step S202 includes the coordinate information of the defect frame corresponding to the defect in the image. Specifically, tools such as labelImg can be used to mark all defects in the defect image with defect frames, and during the marking process, images that cannot be analyzed and marked can be filtered out.
[0053] Optionally, the above-mentioned first defect label information can be saved in the form of a label file. Each line in the label file records the relevant information of a defect in the defect image, and the format is: [x0, y0, x1, y1, category label], where x0 represents the horizontal coordinate of the upper left corner of the defect box, y0 represents the vertical coordinate of the upper left corner of the defect box, x1 represents the horizontal coordinate of the lower right corner of the defect box, and y1 represents the vertical coordinate of the lower right corner of the defect box. Since the sides of the defect box are parallel to one side of the defect image, only the two coordinates of the upper left corner and the lower right corner of the defect box are needed to determine the position of the defect box, and then determine the position of the defect. It should be noted that the coordinates of the upper left corner and the lower right corner of the defect box are the coordinates of these two points in the defect plane rectangular coordinate system, and the defect plane rectangular coordinate system is a plane rectangular coordinate system established in the defect image.
[0054] As an optional implementation, in order to efficiently train the defect detection model with limited grain defect data, the training sample set acquisition method shown in FIG4 can also be used to process the training sample set. As shown in FIG4 , the following steps are included:
[0055] Step S402: acquiring a first defect image and a first template image corresponding to the first defect image;
[0056] In step S404, the first defect image and the first template image are subjected to geometric transformation processing using the same processing method, and the first defect image subjected to the geometric transformation processing is used as a defect image sample, and the first template image subjected to the combined transformation processing is used as a template image sample, wherein the processing method includes at least one of the following: random cropping, image flipping, and image splicing.
[0057] Specifically, the random cropping scale can be customized, for example, between 70% and 130% of the original image size. The pixel values of the blank areas of the cropped image can be set to zero. Flipping can be performed horizontally or vertically on multiple rows or columns of pixels within an image based on a preset probability. Image stitching can be processed using methods such as R-Stitch.
[0058] After sorting the sample data in this way, the number of training batches in the training process is indirectly increased, the problem of uneven category distribution is solved, the diversity of samples is increased, the occurrence of overfitting in the training process is effectively prevented, and the robustness of the model to the size of the detected images is improved.
[0059] Step S204, determining a target loss function, wherein the target loss function includes a defect category loss function and a defect location loss function;
[0060] In the technical solution provided in step S204 , the defect category loss function may be a cross entropy loss function, and the defect position loss function may be a smooth L1 loss function.
[0061] Step S206, using the stochastic gradient descent method, the model to be trained is trained using the training sample set, defect label information and target loss function to obtain a target defect detection model, wherein the target defect detection model is used to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is used to extract the first image features of the defect image and the second image features of the template image, and output the second label information of the defect image based on the first image features and the second image features, and the second label information includes the second category information and the second position information of each defect in the defect image determined by the model to be trained.
[0062] In the technical solution provided in step S206, the specific process of training the target defect detection model is shown in FIG5 , and includes the following steps:
[0063] Step S502: Input the defect image and the template image into the model to be trained, respectively, and obtain second label information output by the model to be trained, wherein the second label information includes second category information and second position information of each defect in the defect image determined by the model to be trained;
[0064] Step S504: inputting the first category information and the second category information into a defect category loss function to obtain a first loss function value, and inputting the first position information and the second position information into a defect position loss function to obtain a second loss function value, and storing the first loss function value and the second loss function value;
[0065] Step S506, using the stochastic gradient descent method to adjust the model parameters of the to-be-trained model according to the first loss function value and the second loss function value;
[0066] It's important to note that you can load official pre-trained weights as the initial model parameters for the model you're training. Because the official pre-trained weights are trained on the open-source ImageNet dataset, which contains a large number of sample images, using them can help the model you're training better learn the underlying characteristics of the target, significantly alleviating overfitting issues caused by a small number of training samples and accelerating model convergence.
[0067] Step S508, obtain all the stored first loss function values and second loss function values, and if the first loss function values and the second loss function values meet the preset conditions, determine that the adjusted model to be trained is the target defect detection model, otherwise jump to step S502.
[0068] The preset condition may be that both the first loss function value and the second loss function value no longer fluctuate. For example, the difference between any two first loss function values among a preset number of consecutive first loss function values is less than a first preset difference, and the difference between any two second loss function values among a preset number of consecutive second loss function values is less than a second preset difference.
[0069] Specifically, the common cross-entropy loss is used as the loss function for the category, the smoothed L1 loss is used as the loss function for the ground truth coordinate frame, and the classic stochastic gradient descent (SGD) method is used as the optimizer for network backpropagation. The training set is batch-fed into the model to be trained. After the neural network forward propagation process calculates the output (the second label information), the loss function value is compared with the true value (the first label information). The model parameters are then updated through the backpropagation process (gradient descent). Training is repeated until the loss function value remains stable and no longer changes significantly.
[0070] In the technical solution provided in step S502, the specific method for obtaining the second tag information is shown in FIG6 , and includes the following steps:
[0071] Step S602: inputting the defect image and the template image into the backbone network of the model to be trained, wherein the backbone network is configured to extract image features from the defect image and the template image, thereby obtaining a defect feature image and a template feature image;
[0072] Step S604: obtaining second label information output by the model to be trained based on the defect feature image and the template feature image.
[0073] Optionally, the training model provided in the embodiments of the present application, compared to the models in related art, requires simultaneous input of both the defect image and the template image. Therefore, two backbone networks with identical structures are used to simultaneously extract image features from both the defect image and the template image. The backbone network structure can be configured with appropriate parameters and network types based on available hardware and memory requirements, such as the VGG series, ResNet series, RegNet series, Efficient series, MobileNet series, and the like.
[0074] The model to be trained provided in the embodiment of the present application also includes a feature fusion layer and a feature pyramid layer, wherein the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; the feature pyramid layer is configured to output second label information based on the first target feature image.
[0075] Specifically, as an optional implementation, as shown in FIG7 , the feature fusion layer includes a first feature fusion layer. The first feature fusion layer is configured to perform a differential operation on the defect feature image and the template feature image to obtain a differential feature image, wherein the pixel value of any pixel in the differential feature image is equal to the absolute value of the difference between the defect pixel corresponding to any pixel and the template pixel, the defect pixel is the pixel corresponding to any pixel in the defect feature image, and the template pixel is the pixel corresponding to any pixel in the template feature image; the differential feature image and the defect feature image are channel-joined to obtain a second target feature image; and the second target feature image is convolved with the target convolution kernel to obtain the first target feature image.
[0076] In some embodiments, the structure of the model to be trained may also be shown in Figure 8. As can be seen from Figure 8, the feature fusion layer includes a second feature fusion layer, wherein the second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image respectively through a target convolution kernel, and generate the first target feature image based on the defect feature image and the template feature image after the convolution processing, wherein the pixel value of any pixel point in the first target feature image is equal to the sum of the pixel values of the defect pixel point and the template pixel point corresponding to the any pixel point.
[0077] The models to be trained shown in Figures 7 and 8 can both efficiently utilize information from the template image to locate defects or flaws in the defect image. The difference is that the model shown in Figure 7 first obtains a difference image between the defect image and the template image, then performs channel concatenation on the difference image and the defect image (that is, adding the image channels, equivalent to expanding the feature map to twice the original number of channels), and then performs channel reduction processing using a 1×1 convolution kernel before feeding it into the subsequent feature pyramid layer. The solution embodied in Figure 8, on the other hand, directly superimposes the defect image and the template image before feeding it into the subsequent feature pyramid layer.
[0078] To ensure the training effect, multiple models can be trained simultaneously. After training, the validation set is input into each model to calculate the mean average precision (mAP). The model with the highest mAP is selected as the final target model. The specific process is shown in Figure 9 and includes the following steps:
[0079] Step S902: obtaining a verification sample set and third defect label information corresponding to each group of samples in the verification sample set, wherein each group of samples in the verification sample set includes a defect image and a template image corresponding to the defect image, and the third defect label information includes third category information and third position information of each defect in the defect image;
[0080] It should be noted that the validation sample set can be processed in the same way as the training sample set, and geometric transformation processing can be performed on the images in the validation sample set.
[0081] Step S904: when there are multiple adjusted models to be trained, determine the average precision of each of the multiple adjusted models to be trained by using the verification sample set and the third defect label information;
[0082] Step S906 , determining the adjusted model to be trained with the largest average precision mean as the target defect detection model.
[0083] A training sample set and first defect label information corresponding to each group of samples in the training sample set are obtained, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of each defect in the defect image; a target loss function is determined, wherein the target loss function includes a defect category loss function and a defect position loss function; and a stochastic gradient descent method is used to train a to-be-trained model using the training sample set, the defect label information, and the target loss function to obtain a target defect detection model, wherein the target defect detection model is configured to determine defect location information and defect category information in the defect image to be detected. This method implements die defect detection and classification based on template image feature matching, fully utilizing the die background information obtained in the previous inspection process, using a template image intercepted from a defect-free golden die and a defect image as input to the network model, and matching and fusing the feature images after feature extraction. This method efficiently utilizes the information of the template image, avoids interference from complex background circuit boards, and effectively reduces the impact of slight misalignment between the template image and the defect image due to the translation invariance of the neural network, alleviating the reliance on data preprocessing. Compared with traditional visual inspection algorithms and neural network models with a single input network, this method not only solves the problem that traditional visual inspection algorithms rely too much on manual feature extraction when classifying grain defects in complex backgrounds, but also solves the problem that single-input networks ignore the golden grain information unique to AOI equipment and cannot reasonably use the existing grain background information to extract defect features.
[0084] In addition, although the model to be detected provided in the embodiment of the present application can input defect images and template images at the same time, since the backbone network used by the two images is the same, the number of parameters of the model is almost not increased, thereby ensuring the reasoning speed of the model.
[0085] In addition, the embodiments of the present application adopt pre-training weight fine-tuning and online random geometric transformation, splicing and other data enhancement methods, which solve the problem of uneven category distribution without increasing the amount of data (data storage space), increase the diversity of samples, effectively prevent the occurrence of overfitting during network training, and improve the robustness of the model to the size of the image to be detected.
[0086] Moreover, the feature matching fusion model structure in this application is applicable to the transformation of any neural network, including classification, detection, segmentation, etc. It is not limited to the field of grain defect detection, and is applicable to the field of defect detection in all complex background scenes, and has strong versatility.
[0087] The embodiment of the present application provides a defect image classification method. FIG10 is a flow chart of the defect image classification method. As shown in FIG10 , the method includes the following steps:
[0088] Step S1002, determining a defect image to be detected and a template image corresponding to the defect image to be detected;
[0089] Step S1004: Input the defect image to be detected and the template image into the target defect detection model, and obtain the defect location information and defect category information output by the target defect detection model. The target defect detection model is used to extract the third image feature of the defect image to be detected and the fourth image feature of the template image, and output the defect location information and defect category information based on the third image feature and the fourth image feature.
[0090] Step S1006, determining a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected based on the defect position information;
[0091] Step S1008 : determining the category confidence of each target defect among a preset number of target defects, and determining the defect category of the target defect with the largest category confidence as the defect category of the defect image to be detected.
[0092] Specifically, after obtaining the positions and category labels of all defects in the defect image, the category with the largest category confidence in the topk (k can be any value, such as 3) detection boxes closest to the image center point is selected as the final defect category of the current image.
[0093] Since there may be multiple defects of different types and locations in the defect image obtained in actual production, a simple classification network often cannot obtain the accurate category of the area of interest (the single-classification model may not converge, and the multi-classification model cannot determine the category location). In the embodiment of the present application, a target detection model is designed to first locate and classify the defects in the defect image, and then the final category is determined by the distance from the area of interest and the category confidence, ensuring the normal convergence of the model while obtaining accurate category labels.
[0094] The embodiment of the present application also provides a practical application process of model training and image classification. Figure 11 is a flow chart of the model training and classification process. As shown in Figure 11, first, during the model training process, after obtaining the defect image, the AOI device can intercept the corresponding template image in the golden Die (grain) image according to the defect image, and generate a label file for the defect image. Then the defect image and the template image can be processed by geometric changes, R-stitch splicing, etc. After the processing is completed, the processed images are divided into a training set and a validation set according to a certain ratio. Among them, the training set is used to train the model to be trained. During the training process, the loss function can be constructed to determine whether the training process is completed, and the model parameters are updated by the back propagation method. Specifically, the training completion can be determined when the function value of the loss function remains stable. When the validation set is used to verify the trained model, the validation set can be input into the trained model, and the mAP corresponding to each model is calculated, and the model with the highest mAP is determined to be the best model finally obtained.
[0095] When applying the optimal model to image classification, the AOI device first extracts a corresponding template image from the golden die image based on the image to be inspected. This template image and the image to be inspected are then fed into the optimal model. The optimal model then determines the location and type of all defects in the image to be inspected and the category of the image to be inspected based on the categories of the defect boxes closest to the center of the image to be inspected. Specifically, the defect category corresponding to the defect box with the highest confidence among these defect boxes is used as the category of the image to be inspected.
[0096] The present invention provides a training device for a defect detection model, and FIG12 is a schematic diagram of the structure of the device. As shown in FIG12, the device includes: an input module 120, configured to obtain a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first location information of each defect in the defect image; a first processing module 122, configured to determine a target loss function, wherein the target loss function includes a defect category loss function and a defect location loss function; a second processing module 124, configured to use a stochastic gradient descent method to train a to-be-trained model using the training sample set, the defect label information, and the target loss function to obtain a target defect detection model, wherein the target defect detection model is configured to determine defect location information and defect category information in the to-be-trained defect image, and the to-be-trained model is configured to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, wherein the second label information includes second category information and second location information of each defect in the defect image determined by the to-be-trained model.
[0097] In some embodiments of the present application, the second processing module 124 adopts the stochastic gradient descent method to train the to-be-trained model through the training sample set, the defect label information and the target loss function, and the steps of obtaining the target defect detection model include: a first step, inputting the defect image and the template image into the to-be-trained model respectively, and obtaining the second label information output by the to-be-trained model, wherein the second label information includes the second category information and the second position information of each defect in the defect image determined by the to-be-trained model; a second step, inputting the first category information and the second category information into the defect category loss function to obtain the first loss function value, and inputting the first position information and the second position information into the defect category loss function to obtain the first loss function value. In the defect position loss function, the second loss function value is obtained, and the first loss function value and the second loss function value are stored; in the third step, the random gradient descent method is used to adjust the model parameters of the model to be trained through the first loss function value and the second loss function value; in the fourth step, all the stored first loss function values and second loss function values are obtained. When the difference between any two first loss function values in a continuous preset number of first loss function values is less than the first preset difference, and the difference between any two second loss function values in a continuous preset number of second loss function values is less than the second preset difference, the adjusted model to be trained is determined to be the target defect detection model, otherwise jump to the first step.
[0098] In some embodiments of the present application, the second processing module 124 inputs the defect image and the template image into the model to be trained respectively, and the step of obtaining the second label information output by the model to be trained includes: inputting the defect image and the template image into the backbone network of the model to be trained, wherein the backbone network is configured to extract image features from the defect image and the template image, thereby obtaining a defect feature image and a template feature image; obtaining the second label information output by the model to be trained based on the defect feature image and the template feature image.
[0099] In some embodiments of the present application, the model to be trained also includes a feature fusion layer and a feature pyramid layer, wherein the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; the feature pyramid layer is configured to output second label information based on the first target feature image.
[0100] In some embodiments of the present application, the feature fusion layer includes a first feature fusion layer, wherein the first feature fusion layer is configured to perform a differential operation on the defect feature image and the template feature image to obtain a differential feature image, wherein the pixel value of any pixel point in the differential feature image is equal to the absolute value of the difference between the defect pixel point corresponding to any pixel point and the template pixel point, the defect pixel point is the pixel point corresponding to any pixel point in the defect feature image, and the template pixel point is the pixel point corresponding to any pixel point in the template feature image; the differential feature image and the defect feature image are channel stitched to obtain a second target feature image; the second target feature image is convolved with the target convolution kernel to obtain the first target feature image.
[0101] In some embodiments of the present application, the feature fusion layer includes a second feature fusion layer, wherein the second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image respectively through the target convolution kernel, and generate a first target feature image based on the defect feature image and the template feature image after convolution processing, wherein the pixel value of any pixel point in the first target feature image is equal to the sum of the pixel values of the defect pixel point and the template pixel point corresponding to any pixel point.
[0102] In some embodiments of the present application, the step in which the second processing module 124 determines that the adjusted model to be trained is the target defect detection model includes: obtaining a verification sample set, and third defect label information corresponding to each group of samples in the verification sample set, wherein each group of samples in the verification sample set includes a defect image, and a template image corresponding to the defect image, and the third defect label information includes third category information and third position information of defects at each location in the defect image; in the case where there are multiple adjusted models to be trained, determining the average precision mean of each adjusted model to be trained in the multiple adjusted models to be trained through the verification sample set and the third defect label information; and determining the adjusted model to be trained with the largest average precision mean as the target defect detection model.
[0103] In some embodiments of the present application, the step of the input module 120 obtaining a training sample set includes: obtaining a first defect image and a first template image corresponding to the first defect image; performing geometric transformation processing on the first defect image and the first template image using the same processing method, and using the first defect image processed with geometric transformation as a defect image sample, and using the first template image processed with combined transformation as a template image sample, wherein the processing method includes at least one of the following: random cropping, image flipping, and image splicing.
[0104] In some embodiments of the present application, the category information includes a defect category number of the defect, and the location information includes location information of a defect box corresponding to the defect.
[0105] It should be noted that the various modules in the training device of the above-mentioned defect detection model can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
[0106] The present application also provides a non-volatile storage medium. The non-volatile storage medium stores a program, wherein when the program is executed, the device containing the non-volatile storage medium is controlled to execute the following defect detection model training method: obtaining a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of each defect in the defect image; determining a target loss function, wherein the target loss function includes a defect category loss function and a defect location loss function; using a stochastic gradient descent method to train a model to be trained using the training sample set, the defect label information, and the target loss function to obtain a target defect detection model, wherein the target defect detection model is configured to determine defect location information and defect category information in the defect image to be detected, the model to be trained is configured to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, wherein the second label information includes second category information and second position information of each defect in the defect image determined by the model to be trained.
[0107] In some embodiments of the present application, when the program is running, the device where the non-volatile storage medium is located can also be controlled to execute the following defect image classification method: determine the defect image to be detected, and the template image corresponding to the defect image to be detected; input the defect image to be detected and the template image into the target defect detection model, and obtain the defect position information and defect category information output by the target defect detection model, wherein the target defect detection model is set to extract the third image feature of the defect image to be detected and the fourth image feature of the template image, and output the defect position information and defect category information based on the three image features and the fourth image feature; determine a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected based on the defect position information; determine the category confidence of each target defect in the preset number of target defects, and determine the defect category of the target defect with the largest category confidence as the defect category of the defect image to be detected.
[0108] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, and the processor is configured to run a program stored in the memory, wherein the following defect detection model training method is executed when the program is running: obtaining a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of each defect in the defect image; determining a target loss function, wherein the target loss function includes a defect category loss function and a defect position loss function; using a stochastic gradient descent method, the model to be trained is trained with the training sample set, the defect label information and the target loss function to obtain a target defect detection model, wherein the target defect detection model is configured to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is configured to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, and the second label information includes second category information and second position information of each defect in the defect image determined by the model to be trained.
[0109] In some embodiments of the present application, the following defect image classification method can also be executed when the program is running: determine the defect image to be detected, and the template image corresponding to the defect image to be detected; input the defect image to be detected and the template image into the target defect detection model, and obtain the defect position information and defect category information output by the target defect detection model, wherein the target defect detection model is set to extract the third image feature of the defect image to be detected and the fourth image feature of the template image, and output the defect position information and defect category information based on the three image features and the fourth image feature; determine a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected based on the defect position information; determine the category confidence of each target defect in the preset number of target defects, and determine the defect category of the target defect with the largest category confidence as the defect category of the defect image to be detected.
[0110] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0111] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0112] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0113] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0114] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the relevant technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.
[0115] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application. Industrial Applicability
[0116] The training method, device and electronic device of the defect detection model provided in the embodiment of the present application are applied to the field of image recognition. In the embodiment of the present application, a training sample set and first defect label information corresponding to each group of samples in the training sample set are obtained, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes the first category information and the first position information of each defect in the defect image; a target loss function is determined, wherein the target loss function includes a defect category loss function and a defect position loss function; a random gradient descent method is used to train the model to be trained with the training sample set, defect label information and target loss function to obtain a target defect detection model, wherein the target defect detection model is used to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is used to extract the defect image. The first image feature and the second image feature of the template image are used, and the second label information of the defect image is output based on the first image feature and the second image feature. The second label information includes the second category information and the second position information of each defect in the defect image determined by the model to be trained. By adopting a training sample set including defect images and template images and defect label information to train the defect detection model, the purpose of obtaining a defect detection model that can use the template image in the defect recognition process is achieved, thereby achieving the technical effect of using the template image to assist classification and recognition in the process of defect recognition and classification of grain images, and thus solving the technical problem of poor recognition and classification results caused by not using the template image when recognizing and classifying defects in grain images in the related technology.
Claims
1. A method for training a defect detection model, comprising: Acquire a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of defects at each location in the defect image; Determine a target loss function, wherein the target loss function includes a defect category loss function and a defect location loss function; The stochastic gradient descent method is adopted to train the model to be trained through the training sample set, the defect label information and the target loss function to obtain a target defect detection model, wherein the target defect detection model is used to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is used to extract the first image feature of the defect image and the second image feature of the template image, and output the second label information of the defect image based on the first image feature and the second image feature, and the second label information includes the second category information and the second position information of the defects at each location in the defect image determined by the model to be trained.
2. The defect detection model training method according to claim 1, wherein: The step of using the stochastic gradient descent method to train the to-be-trained model through the training sample set, the defect label information and the target loss function to obtain the target defect detection model comprises: The first step is to input the defect image and the template image into the model to be trained respectively, and obtain the second label information output by the model to be trained; The second step is to input the first category information and the second category information into the defect category loss function to obtain a first loss function value, and input the first position information and the second position information into the defect position loss function to obtain a second loss function value, and store the first loss function value and the second loss function value; The third step is to use the stochastic gradient descent method to adjust the model parameters of the to-be-trained model through the first loss function value and the second loss function value; The fourth step is to obtain all the stored first loss function values and the second loss function values. When the first loss function value and the second loss function value meet the preset conditions, determine that the adjusted model to be trained is the target defect detection model, otherwise jump to the first step, wherein the preset conditions include that the sum of a consecutive preset number of the first loss function values and the second loss function values are all within the target value range.
3. The defect detection model training method according to claim 2, wherein: The step of inputting the defect image and the template image into the model to be trained respectively and obtaining the second label information output by the model to be trained comprises: Inputting the defect image and the template image into the backbone network of the model to be trained, wherein the backbone network is configured to extract image features from the defect image and the template image, thereby obtaining a defect feature image and a template feature image; The second label information output by the model to be trained based on the defect feature image and the template feature image is obtained.
4. The defect detection model training method according to claim 3, wherein: The model to be trained also includes a feature fusion layer and a feature pyramid layer. The feature fusion layer is configured to obtain a first target feature image according to the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; The feature pyramid layer is configured to output the second label information according to the first target feature image.
5. The method for training a defect detection model according to claim 4, wherein: The feature fusion layer comprises a first feature fusion layer, The first feature fusion layer is configured to perform a differential operation on the defect feature image and the template feature image to obtain a differential feature image, wherein the pixel value of any pixel in the differential feature image is equal to the absolute value of the difference between the defect pixel corresponding to the any pixel and the template pixel, the defect pixel is the pixel corresponding to the any pixel in the defect feature image, and the template pixel is the pixel corresponding to the any pixel in the template feature image; channel stitching processing is performed on the differential feature image and the defect feature image to obtain a second target feature image; and convolution processing is performed on the second target feature image through a target convolution kernel to obtain the first target feature image.
6. The method for training a defect detection model according to claim 4, wherein: The feature fusion layer includes a second feature fusion layer, The second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image respectively through a target convolution kernel, and generate the first target feature image based on the defect feature image and the template feature image after the convolution processing, wherein the pixel value of any pixel point in the first target feature image is equal to the sum of the pixel values of the defect pixel point and the template pixel point corresponding to the any pixel point.
7. The defect detection model training method according to claim 2, wherein: The step of determining the adjusted model to be trained as the target defect detection model comprises: Acquire a verification sample set and third defect label information corresponding to each group of samples in the verification sample set, wherein each group of samples in the verification sample set includes a defect image and a template image corresponding to the defect image, and the third defect label information includes third category information and third position information of defects at various locations in the defect image; In the case where there are multiple adjusted models to be trained, determining the average precision mean of each of the multiple adjusted models to be trained through the verification sample set and the third defect label information; The adjusted model to be trained with the largest average precision mean is determined as the target defect detection model.
8. The method for training a defect detection model according to claim 1, wherein: The step of obtaining a training sample set comprises: Acquire a first defect image and a first template image corresponding to the first defect image; The first defect image and the first template image are subjected to geometric transformation processing in the same manner, and the first defect image subjected to the geometric transformation processing is used as the defect image sample, and the first template image subjected to the geometric transformation processing is used as the template image sample, wherein the processing manner includes at least one of the following: random cropping, image flipping, and image stitching.
9. The method for training a defect detection model according to claim 1, wherein: The category information includes a defect category number of the defect, and the location information includes location information of a defect box corresponding to the defect.
10. A defect image classification method, comprising: Determine a defect image to be detected and a template image corresponding to the defect image to be detected; Inputting the defect image to be detected and the template image into a target defect detection model, and obtaining defect location information and defect category information output by the target defect detection model, wherein the target defect detection model is used to extract a third image feature of the defect image to be detected and a fourth image feature of the template image, and outputting the defect location information and the defect category information according to the third image feature and the fourth image feature; Determine a preset number of target defects in the defect image to be detected that are closest to the center point of the defect image to be detected according to the defect position information; The category confidence of each target defect among the preset number of target defects is determined, and the defect category of the target defect with the largest category confidence is determined as the defect category of the defect image to be detected.
11. A training device for a defect detection model, comprising: an input module, configured to obtain a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of defects at each location in the defect image; A first processing module is configured to determine a target loss function, wherein the target loss function includes a defect category loss function and a defect location loss function; The second processing module is configured to adopt a stochastic gradient descent method to train the model to be trained through the training sample set, the defect label information and the target loss function to obtain a target defect detection model, wherein the target defect detection model is configured to determine the defect position information and defect category information in the defect image to be detected, and the model to be trained is configured to extract the first image feature of the defect image and the second image feature of the template image, and output the second label information of the defect image based on the first image feature and the second image feature, and the second label information includes the second category information and the second position information of the defects at each location in the defect image determined by the model to be trained.
12. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the defect detection model training method described in any one of claims 1 to 9, or the defect image classification method described in claim 10.
13. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein when the program is run, the training method of the defect detection model described in any one of claims 1 to 9, or the defect image classification method described in claim 10 is executed.
