Defect inspection model training method, device, and electronic device

Abstract

This application discloses a defect inspection model training method, apparatus, and electronic device, including the steps of: acquiring a training sample set and first defect label information corresponding to each group of samples in the training sample set, where each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first location information of each defect in the defect image; determining a target loss function; and training a target model using a stochastic gradient descent method with the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model, where the target defect inspection model is used to determine defect location information and defect category information in the target defect image.

Description

[Technical Field]

[0001] This application claims priority to a Chinese patent application bearing application number 202310184844.5 and entitled "Method, apparatus and electronic device for training a defect inspection model," filed with the China Patent Office on February 17, 2023, the entire contents of which are incorporated herein by reference.

[0002] This application relates to the field of image identification, and in particular to a method, apparatus and electronic device for training a defect inspection model. [Background technology]

[0003] In related art, when a defect-free grain image is used to identify and classify defects in an image of a target grain, a template image is not used to assist in the identification. Instead, the method typically adopted is to directly fuse the image of the target grain with the defect-free grain image, and then determine the defect category and defect location in the image of the target grain based on the fused image. While this method achieves defect identification and classification in the image of the target grain using the defect-free image, it requires a large amount of calculation and has low accuracy when fusing the images and identifying the fused image.

[0004] As of now, no effective means for solving the above problems has been proposed. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide a method, device, and electronic device for training a defect inspection model to at least solve the technical problems of related art in that when identifying and classifying defects in crystal grain images, direct fusion of a defect-free image with an image to be inspected results in a large amount of calculation and low accuracy. [Means for solving the problem]

[0006] According to one aspect of an embodiment of the present application, there is provided a method for training a defect inspection model, the method including: obtaining a training sample set and first defect label information corresponding to each group of samples in the training sample set, where each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first location information of each defect in the defect image; determining a target loss function including a defect category loss function and a defect location loss function; and training a trained model using a stochastic gradient descent method with the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model, where the target defect inspection model is used to determine the defect location information and the defect category information in the inspection target defect image, and the trained model is used to extract first image features of the defect image and second image features of the template image, and to output second label information of the defect image based on the first image features and the second image features, and the second label information includes second category information and second location information of each defect in the defect image determined by the trained model.

[0007] Optionally, the step of training a training object model with a training sample set, defect label information, and a target loss function using a stochastic gradient descent method to obtain a target defect inspection model includes a first step of inputting a defect image and a template image into the training object model, respectively, to obtain second label information output from the training object model; a second step of inputting first category information and second category information into a defect category loss function to obtain a first loss function value, inputting first position information and second position information into a defect position loss function to obtain a second loss function value, and storing the first loss function value and the second loss function value. a third step of adjusting model parameters of the trained model with the first loss function value and the second loss function value using a stochastic gradient descent method; and a fourth step of acquiring all stored first loss function values ​​and second loss function values, and determining the adjusted trained model as a target defect inspection model if the first loss function value and the second loss function value meet a predetermined condition, and otherwise proceeding to the first step, wherein the predetermined condition includes that the sums of a predetermined number of consecutive first loss function values ​​and second loss function values ​​are all located within a target value interval.

[0008] The step of selectively inputting the defect image and the template image into a trained model, respectively, and obtaining second label information output from the trained model includes the steps of inputting the defect image and the template image into a backbone network of the trained model, where the backbone network is configured to obtain a defect feature image and a template feature image by extracting image features in the defect image and the template image, and obtaining the second label information output by the trained model based on the defect feature image and the template feature image.

[0009] Optionally, the training model further includes a feature fusion layer and a feature pyramid layer, wherein the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image and input the first target feature image into the feature pyramid layer, and the feature pyramid layer is configured to output second label information based on the first target feature image.

[0010] Optionally, the feature fusion layer includes a first feature fusion layer, which is configured to perform a difference operation on the defect feature image and the template feature image to obtain a difference feature image, wherein the pixel value of any one pixel point in the difference feature image is equal to the absolute value of the difference between the defect pixel point corresponding to the any one pixel point and the template pixel point, the defect pixel point being a pixel point corresponding to the any one pixel point in the defect feature image, and the template pixel point being a pixel point corresponding to the any one pixel point in the template feature image, perform a channel connection process on the difference feature image and the defect feature image to obtain a second target feature image, and perform a convolution process on the second target feature image with a target convolution kernel to obtain a first target feature image.

[0011] Optionally, the feature fusion layer includes a second feature fusion layer, which is configured to perform convolution processing on the defect feature image and the template feature image, respectively, via a target convolution kernel, and generate a first target feature image based on the defect feature image and the template feature image after the convolution processing, and the pixel value of any one pixel point in the first target feature image is equal to the sum of the pixel values ​​of the defect pixel point and the template pixel point corresponding to the any one pixel point.

[0012] Optionally, the step of determining the adjusted training target model as the target defect inspection model includes a step of obtaining a validation sample set and third defect label information corresponding to each group of samples in the validation sample set, wherein each group of samples in the validation sample set includes a defect image and a template image corresponding to the defect image, and the third defect label information includes third category information and third position information of each defect in the defect image; and if there are multiple adjusted training target models, a step of determining an average accuracy average value of each adjusted training target model among the multiple adjusted training target models using the validation sample set and the third defect label information; and a step of determining the adjusted training target model with the highest average accuracy average value as the target defect inspection model.

[0013] Optionally, the step of obtaining a training sample set includes the steps of obtaining a first defect image and a first template image corresponding to the first defect image, and performing geometric transformation processing on the first defect image and the first template image using the same processing method, and setting the first defect image that has undergone the geometric transformation processing as a defect image sample, and the first template image that has undergone the geometric transformation processing as a template image sample, wherein the processing method includes at least one of random cropping, image flipping, and image concatenation.

[0014] Optionally, the category information includes a defect category number of the defect, and the location information includes location information of the defect block corresponding to the defect.

[0015] According to another aspect of an embodiment of the present application, a defect image classification method is further provided, comprising the steps of: determining a target defect image to be inspected and a template image corresponding to the target defect image; inputting the target defect image and the template image into a target defect inspection model to obtain defect location information and defect category information output from the target defect inspection model, wherein the target defect inspection model is used to extract a third image feature of the target defect image to be inspected and a fourth image feature of the template image, and output the defect location information and defect category information based on the third image feature and the fourth image feature; determining a predetermined number of target defects in the target defect image to be inspected that are closest to the center point of the target defect image to be inspected based on the defect location information; and determining a category reliability of each target defect among the predetermined number of target defects, and determining the defect category of the target defect with the highest category reliability as the defect category of the target defect image to be inspected.

[0016] According to another aspect of an embodiment of the present application, there is further provided an apparatus for training a defect inspection model, comprising: an input module configured to obtain a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first location information of each defect in the defect image; a first processing module configured to determine a target loss function including a defect category loss function and a defect location loss function; and a second processing module configured to train a trained model using a stochastic gradient descent method with the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model, wherein the target defect inspection model is configured to determine defect location information and defect category information in the inspection target defect image.

[0017] According to another aspect of an embodiment of the present application, a non-volatile storage medium storing a program is further provided, and when the program is executed, the non-volatile storage medium controls an apparatus in which the program is located to execute a defect inspection model training method or a defect image classification method.

[0018] According to another aspect of an embodiment of the present application, there is further provided an electronic device, the electronic device including a memory and a processor, the processor being configured to execute a program stored in the memory, and when the program is executed, a method for training a defect inspection model or a method for classifying defect images is performed.

[0019] In an embodiment of the present application, a training sample set and first defect label information corresponding to each group of samples in the training sample set are obtained, each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, the first defect label information includes first category information and first location information of each defect in the defect image, a target loss function including a defect category loss function and a defect location loss function is determined, a training target model is trained using a stochastic gradient descent method with the training sample set, the defect label information and the target loss function to obtain a target defect inspection model, the target defect inspection model is used to determine the defect location information and defect category information in the inspection target defect image, and the training target model extracts first image features of the defect image and second image features of the template image. The method is used to output second label information of the defect image based on the first image features and the second image features, where the second label information includes second category information and second location information of each defect in the defect image determined by the training target model. By training a defect inspection model using a training sample set including defect images and template images and the defect label information, the purpose of obtaining a defect inspection model that can use the template image during defect identification is achieved, thereby achieving the technical effect of using the template image to assist in classification and identification in the process of identifying and classifying defects in crystal grain images, and ultimately solving the technical problem in the related art that poor identification and classification results are achieved when a template image is not used when identifying and classifying defects in crystal grain images. [Brief explanation of the drawings]

[0020] The drawings described herein are intended to provide further understanding of the present application and constitute a part of the present application, and the schematic examples and descriptions thereof are used to interpret the present application and are not intended to unduly limit the present application. [Figure 1] FIG. 1 is a schematic structural diagram of a computer terminal according to an embodiment of the present application; [Figure 2] 1 is a schematic flow chart of a method for training a defect inspection model according to an embodiment of the present application; [Figure 3a] FIG. 1 is a schematic diagram of a gold grain image according to an embodiment of the present application. [Figure 3b] 1 is a schematic diagram of a defect image and a template image according to an embodiment of the present application; [Figure 4] 1 is a schematic flow chart of a method for obtaining a training sample set according to an embodiment of the present application; [Figure 5] 1 is a schematic flowchart of a training method for training a defect inspection model based on a training sample set and defect label information, according to an embodiment of the present application; [Figure 6] 4 is a schematic flowchart of a method for obtaining second label information according to an embodiment of the present application; [Figure 7] FIG. 2 is a schematic structural diagram of a defect inspection model according to an embodiment of the present application; [Figure 8] FIG. 10 is a schematic structural diagram of another defect inspection model according to an embodiment of the present application; [Figure 9] 1 is a schematic flow chart of a method for screening a defect inspection model according to an embodiment of the present application; [Figure 10] 1 is a schematic flow chart of a defect image classification method according to an embodiment of the present application; [Figure 11] 1 is a schematic flowchart of a model training and classification flow according to an embodiment of the present application; [Figure 12] 1 is a schematic structural diagram of a defect inspection model training device according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0021] In order to help those skilled in the art to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application, but it is clear that the described embodiments are only some of the embodiments of the present application, and do not represent all of the embodiments. Any other embodiments obtained by those skilled in the art based on the embodiments of the present application without any creative efforts shall fall within the protection scope of the present application.

[0022] It should be noted that terms such as "first" and "second" in the specification and claims of this application and in the drawings above are used to distinguish between similar objects and are not necessarily used to describe a particular order or priority. It should be understood that the examples of this application described herein may be performed in orders other than those illustrated or described herein, as the data so used may be interchanged under appropriate circumstances.

[0023] Additionally, the terms "comprise" and "have," and any variations thereof, are intended to be non-exclusive inclusive, for example, including a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units explicitly listed, but may also include other steps or units inherent to the process, method, product, or apparatus that are not explicitly listed.

[0024] With the development of semiconductor device technology, there are more and more processes for manufacturing semiconductor devices, the main purpose of which is to fabricate circuits and electronic components such as transistors, capacitors, and logic switches on the crystal grains, for example, by performing oxidation and chemical vapor deposition on the surface of the crystal grains, followed by steps such as coating, exposure, development, etching, ion implantation, and metal sputtering, and finally completing the processing and fabrication of several layers of circuits and components on the crystal grains.

[0025] However, because each process in the manufacturing process is somewhat complex, the treatment of grains in each process flow may result in unexpected structures in different layers, and these structures may cause the circuits on the chip to not function properly; such structures are usually called grain defects.

[0026] To eliminate grain defects, chip manufacturing processes must include grain defect inspection steps after many critical steps to monitor and ensure the accuracy of key processes. However, due to the complexity of chip manufacturing processes and the large number of grain defect types, there is currently no unified classification method. Currently, automated optical inspection (AOI) typically uses traditional visual inspection, i.e., image subtraction (target grain and gold grain) or threshold segmentation inspection algorithms, to locate and inspect grain defects.

[0027] However, due to the complex background of the crystal grains, the many processes involved, and the diverse shapes of the crystal grains, while defect location can be achieved using traditional vision algorithms, it is nearly impossible to manually extract and classify the defect features. After formulating a set of rules (information such as the defect pattern, shape and size of the defect, the brightness value of the defect location area, and the intensity of the defect signal), the classification results obtained through screening are inaccurate and there are many cases of over-detection and false detection. Therefore, in many cases, the classification of crystal grain defects relies heavily on primitive manual re-inspection methods. Manual classification operations have the disadvantages of being slow, having poor consistency / reliability, and being highly susceptible to external factors. As a result, there are certain technical limitations to the classification of crystal grain defects.

[0028] In addition, in recent years, with the rapid research and development of deep learning algorithms and hardware equipment, especially through the iterative optimization of neural networks in the fields of target classification, target inspection, and target segmentation, automated defect inspection / classification in the semiconductor field has increasingly been made use of deep learning algorithm technology, and the accuracy of inspection / classification has also become increasingly higher.

[0029] However, while related technologies rely on feature extraction techniques to identify and classify defects, the classification performance of the resulting model is limited because defects on crystal grains with complex backgrounds are identified solely through features manually selected by several people. Convolutional neural networks can automatically learn and extract features suitable for classification, improving inspection classification performance.

[0030] However, in actual production, grains have complex backgrounds with many patterns, and defect features are unclear, and the features of different types of defects are similar and difficult to distinguish. Therefore, how to ignore unnecessary background and accurately extract the features of interesting defect areas is a key and difficult point in the inspection and classification of grain defects. Therefore, there is a problem that it is not possible to accurately and efficiently determine the fault type and classify the fault in a grain image. To solve this problem, the embodiments of the present application provide related solutions, which will be described in detail below.

[0031] According to an embodiment of the present application, a method embodiment of a method for training a defect inspection model is provided, and it is noted that the steps shown in the flowcharts of the drawings may be performed in a computer system, such as a set of computer-executable instructions, and that although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown in the specification.

[0032] The method embodiments provided in the present application can be implemented in a mobile terminal, a computer terminal, or a similar computing device. Figure 1 shows a block diagram of the hardware structure of a computer terminal (or mobile device) configured to implement the method for training a defect inspection model.

[0033] 1, the computer terminal 10 (or mobile device 10) may include one or more (indicated in the figure using 102a, 102b, ..., 102n) processors 102 (the processors 102 may include, but are not limited to, processing devices such as microprocessors MCUs or programmable logic devices FPGAs), memory 104 for storing data, and a transmission module 106 for communication functions. In addition to these, the computer terminal 10 (or mobile device 10) may further include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS port), a network interface, a power supply, and / or a camera.

[0034] Those skilled in the art will appreciate that the structure shown in Figure 1 is merely schematic and is not intended to limit the structure of the electronic device described above. For example, the computer terminal 10 may include more or fewer components than those shown in Figure 1, or may have a different configuration than that shown in Figure 1.

[0035] It should be noted that the one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits." The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any combination thereof. The data processing circuits may also be a single, independent processing module, or may be combined in whole or in part with any one of the other components in the computer terminal 10 (or mobile device). As referred to in the embodiments herein, the data processing circuits are processor-controlled (e.g., selection of variable resistance terminal paths connected to an interface).

[0036] The memory 104 can be configured to store software programs and modules of application software, for example, a program instruction / data storage device corresponding to the defect inspection model training method in an embodiment of the present application, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 104, i.e., realizes the defect inspection model training method of the above application program.

[0037] The memory 104 may include random memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may also include memory located remotely from the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, an enterprise network, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the network may include a wireless network provided by the carrier of the computer terminal 10. In one example, the transmission device 106 includes a network interface controller (NIC), which can connect to other network devices via a base station and thereby communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0039] The display may be, for example, a touch-sensitive liquid crystal display (LCD) that allows a user to interact with the user screen of the computer terminal 10 (or mobile device).

[0040] In the above execution environment, an embodiment of the present application provides a method for training a defect inspection model, and as shown in FIG. 2, the method includes steps S202 to S206.

[0041] In step S202, a training sample set and first defect label information corresponding to each group of samples in the training sample set are obtained, and each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of each defect in the defect image.

[0042] In the technical solution provided in step S202, the defect image and the template image in the sample image are as shown in Fig. 3b. As can be seen from Fig. 3b, the template image and the defect image correspond to the same position in the grain image, and the only difference is that the template image has no defects or imperfections.

[0043] In an optional embodiment, after determining the position information of the defect image in the grain image, a corresponding template image can be obtained from the gold grain image as shown in Figure 3a based on the position information, where the gold grain image refers to an image without any defects or imperfections. The gold grain image is typically stored in the AOI inspection equipment.

[0044] The first category information recorded in step S202 also includes defect category numbers. Specifically, defect categories that need to be identified in the grain image and the corresponding category numbers can be determined early. For example, assuming that the defect categories that need to be identified include eight categories, such as opening contamination, wiring contamination, opening damage, wiring etching, adhesive surface damage, development failure, wiring bridging, and edge compression, the corresponding category numbers can be set to {0, 1, 2, 3, 4, 5, 6, 7} in order.

[0045] The first position information described in step S202 includes coordinate information of the defect block corresponding to the defect in the image. Specifically, a tool such as labelImg can be used to label all defects in the defect image as defect blocks, and images that cannot be analyzed and labeled can be screened during labeling.

[0046] Optionally, the first defect label information can be saved in the form of a label file. Each line in the label file records information related to one defect in the defect image, in the format [x0, y0, x1, y1, category number], where x0 represents the abscissa of the upper left corner of the defect block, y0 represents the ordinate of the upper left corner of the defect block, x1 represents the abscissa of the lower right corner of the defect block, and y1 represents the ordinate of the lower right corner of the defect block. Because the sides of the defect block are parallel to one side of the defect image, the location of the defect block and, therefore, the defect location can be determined using only the two coordinates of the upper left and lower right corners of the defect block. Note that the coordinates of the upper left and lower right corners of the defect block are the coordinates of these two points in a defect plane Cartesian coordinate system, which is a plane Cartesian coordinate system constructed within the defect image.

[0047] As an optional embodiment, in order to efficiently train a defect inspection model with finite grain defect data, the training sample set can be processed using the training sample set acquisition method shown in Fig. 4. As shown in Fig. 4, the method includes steps S402 and S404.

[0048] In step S402, a first defect image and a first template image corresponding to the first defect image are obtained.

[0049] In step S404, a geometric transformation process is performed on the first defect image and the first template image using the same processing method, and the first defect image after the geometric transformation process is a defect image sample, and the first template image after the geometric transformation process is a template image sample, and the processing method includes at least one of random cropping, image flipping, and image splicing.

[0050] Specifically, the crop scale range of the random crop can be set manually, for example, between 70% and 130% of the original image size, and the pixel values ​​of blank areas in the image obtained after cropping can be set to zero. When inverting, multiple rows or columns of pixels in an image can be flipped horizontally or vertically according to a preset probability. For image stitching, methods such as R-Stitch stitching can be used to process the images.

[0051] After sorting the sample data in this way, the number of training batches in the training process is indirectly increased, which solves the problem of unbalanced category distribution, increases the diversity of samples, effectively prevents overfitting during training, and improves the robustness of the model to the size of the photo being inspected.

[0052] In step S204, a target loss function is determined, including a defect category loss function and a defect location loss function.

[0053] In the technical solution provided in step S204, the defect category loss function may be a cross-entropy loss function, and the defect location loss function may be a smooth L1 loss function.

[0054] In step S206, a training target model is trained using a stochastic gradient descent method with the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model, which is used to determine defect location information and defect category information in the inspection target defect image, and the training target model is used to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, where the second label information includes second category information and second location information of each defect in the defect image determined by the training target model.

[0055] In the technical solution provided in step S206, the specific flow of training the target defect inspection model is as shown in FIG. 5, which includes steps S502 to S508.

[0056] In step S502, the defect image and the template image are input into a training object model, respectively, to obtain second label information output from the training object model, and the second label information includes second category information and second position information of each defect in the defect image determined by the training object model.

[0057] In step S504, the first category information and the second category information are input into a defect category loss function to obtain a first loss function value, and the first position information and the second position information are input into a defect position loss function to obtain a second loss function value, and the first loss function value and the second loss function value are stored.

[0058] In step S506, the model parameters of the training target model are adjusted using the first loss function value and the second loss function value using the stochastic gradient descent method.

[0059] In addition, the official pre-trained weights can be loaded as the initial model parameters of the trained model. The official pre-trained weights are obtained by training on ImageNet, an open-source dataset containing a large number of sample images. Therefore, using the pre-trained weights can enable the trained model to more effectively learn the basic features of the target, greatly alleviate the overfitting problem caused by the small number of samples in the training sample set, and also accelerate the convergence speed of the model.

[0060] In step S508, all stored first loss function values ​​and second loss function values ​​are obtained, and if the first loss function values ​​and second loss function values ​​meet the preset conditions, the adjusted training target model is determined as the target defect inspection model; otherwise, proceed to step S502.

[0061] The preset condition may be that neither the first loss function value nor the second loss function value fluctuates, for example, that the difference between any two first loss function values ​​among a preset number of consecutive first loss function values ​​is less than a first preset difference, and that the difference between any one second loss function value among a preset number of consecutive second loss function values ​​is less than a second preset difference.

[0062] Specifically, the loss calculation function uses the general cross-entropy loss as a category, the smooth L1 loss as the actual coordinate frame, and the typical stochastic gradient descent (SGD) method is used as the network backpropagation optimizer. The training set is input to the above-mentioned training model in batches, and the neural network's forward propagation process is calculated to obtain the output (second label information). After that, the loss function value is calculated by comparing it with the true value (first label information). Next, the model parameters are updated in the backpropagation process (gradient descent method). Training is repeated until the loss function value stabilizes, and training is stopped when there are no more significant changes.

[0063] In the technical solution provided in step S502, a specific method for obtaining the second label information is as shown in FIG. 6, which includes steps S602 and S604.

[0064] In step S602, the defect image and the template image are input into a backbone network of a training model, and the backbone network is configured to obtain a defect feature image and a template feature image by extracting image features in the defect image and the template image.

[0065] In step S604, second label information output by the training model based on the defect feature image and the template feature image is obtained.

[0066] Optionally, the training object model provided in the embodiments of the present application, compared with the model in the related art, requires simultaneous input of a defect image and a template image, so two backbone networks with the same structure simultaneously extract image features from the defect image and the template image. The structure of the backbone network can be selected according to the memory requirements of the existing hardware, with an appropriate number of parameters and network type, such as the VGG series, ResNet series, RegNet series, Efficient series, MobileNet series, etc.

[0067] The training object model provided in the embodiments of the present application further includes a feature fusion layer and a feature pyramid layer, where the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image and input the first target feature image into the feature pyramid layer, and the feature pyramid layer is configured to output second label information based on the first target feature image.

[0068] Specifically, as an optional embodiment, the feature fusion layer includes a first feature fusion layer, as shown in Fig. 7. The first feature fusion layer is configured to perform a difference operation on the defect feature image and the template feature image to obtain a difference feature image, where a pixel value of any one pixel point in the difference feature image is equal to an absolute value of a difference between the defect pixel point corresponding to the any one pixel point and the template pixel point, where the defect pixel point is a pixel point corresponding to the any one pixel point in the defect feature image, and the template pixel point is a pixel point corresponding to the any one pixel point in the template feature image, perform a channel connection operation on the difference feature image and the defect feature image to obtain a second target feature image, and perform a convolution operation on the second target feature image with a target convolution kernel to obtain a first target feature image.

[0069] In some embodiments, the structure of the training target model may be as shown in Fig. 8. As can be seen from Fig. 8, the feature fusion layer includes a second feature fusion layer, which is configured to perform convolution processing on the defect feature image and the template feature image via a target convolution kernel, respectively, and generate the first target feature image based on the defect feature image and the template feature image after the convolution processing, wherein a pixel value of any one pixel point in the first target feature image is equal to the sum of pixel values ​​of the defect pixel point and the template pixel point corresponding to the any one pixel point.

[0070] The trained models shown in Figures 7 and 8 can both efficiently utilize the information of the template image to locate defects or flaws in the defect image. The difference is that the model shown in Figure 7 first obtains a difference image between the defect image and the template image, then performs channel concatenation on the difference image and the defect image (i.e., addition on the image channels, which corresponds to expanding the feature map to twice the original number of channels), and then reduces the number of channels through a 1x1 convolution kernel before being sent to the subsequent feature pyramid layer. The solution used in Figure 8 is to directly overlap the defect image and the template image and then send it to the subsequent feature pyramid layer.

[0071] To ensure the training effect, multiple training target models can be trained simultaneously, and after training is completed, a validation set can be input to each model to calculate mAP (mean average precision), and the model with the highest mAP can be selected as the final target model. The specific flow is as shown in Figure 9, including steps S902 to S906.

[0072] In step S902, a verification sample set and third defect label information corresponding to each group of samples in the verification sample set are obtained, where each group of samples in the verification sample set includes a defect image and a template image corresponding to the defect image, and the third defect label information includes third category information and third position information of each defect in the defect image.

[0073] It should be noted that the validation sample set can be subjected to geometric transformation processing on the images in the validation sample set using the same processing method as the training sample set.

[0074] In step S904, if there are multiple adjusted trained models, the average accuracy of each adjusted trained model is determined using the validation sample set and the third defect label information.

[0075] In step S906, the adjusted training model with the highest average mean accuracy is determined as the target defect inspection model.

[0076] A training sample set and first defect label information corresponding to each group of samples in the training sample set are obtained, and each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first location information of each defect in the defect image. A target loss function is determined, and the target loss function includes a defect category loss function and a defect location loss function. A training target model is trained using a stochastic gradient descent method with the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model, and the target defect inspection model is obtained based on the defect location information and the target defect location information in the confirmed inspection target defect image. The neural network is configured to determine the defect category information and the background information of the grains obtained in the previous inspection process, thereby enabling the inspection and classification of the grain defects matched based on the template map features to be conveniently utilized. The template map cut from the defect-free golden die and the defect map are both used as the input of the network model, and matching and fusion are performed on the feature map after feature extraction. This method efficiently uses the information of the template map, avoids the interference of the complex background circuit board, and, due to the translation invariance of the neural network, effectively reduces the impact of minor discrepancies between the template map and the defect map, thereby reducing the dependency on data preprocessing.

[0077] Compared with traditional visual inspection algorithms and single-input network neural network models, this not only solves the problem that traditional visual inspection algorithms rely too heavily on manual feature extraction when classifying grain defects in complex backgrounds, but also solves the problem that single-input networks ignore the gold grain information specific to AOI equipment and are unable to rationally utilize existing grain background information to extract defect features.

[0078] In addition, the inspection object model provided in the embodiments of the present application can simultaneously input a defect image and a template image, but since the same backbone network is used for the two images, the number of model parameters hardly increases, and the inference speed of the model is guaranteed.

[0079] In the embodiment of the present application, by using data augmentation methods such as fine-tuning of pre-trained weights and online random geometric transformation and concatenation, the problem of unbalanced category distribution can be solved without adding data volume (data storage space), and the diversity of samples can be improved, which effectively prevents overfitting during network training and improves the robustness of the model to the size of the test image.

[0080] Furthermore, the feature matching fusion model structure in this application is suitable for modifying any neural network, including classification, inspection, division, etc., and is not limited to the field of grain defect inspection, but is suitable for all defect inspection fields in complex background scenes, and is highly versatile.

[0081] An embodiment of the present application provides a defect image classification method, and FIG. 10 is a schematic flowchart of the defect image classification method, which includes steps S1002 to S1008 as shown in FIG.

[0082] In step S1002, an inspection target defect image and a template image corresponding to the inspection target defect image are determined.

[0083] In step S1004, the defect image to be inspected and the template image are input into a target defect inspection model to obtain defect location information and defect category information output from the target defect inspection model, and the target defect inspection model is used to extract a third image feature of the defect image to be inspected and a fourth image feature of the template image, and output defect location information and defect category information based on the third image feature and the fourth image feature.

[0084] In step S1006, a preset number of target defects that are closest to the center point of the inspection target defect image are determined based on the defect position information.

[0085] In step S1008, the category reliability of each target defect among a preset number of target defects is determined, and the defect category of the target defect with the highest category reliability is determined as the defect category of the inspection target defect image.

[0086] Specifically, after obtaining all defect locations and category labels in the defect image, the category with the highest category confidence in the topk (k can be any value, e.g., 3) inspection bounding boxes closest to the center point of the image is selected as the final defect category for the current photo.

[0087] Since defect maps obtained in actual production may contain multiple defects of different types and different locations, a simple classification network cannot obtain an accurate category for the area of ​​interest (a single classification model may not converge, and a multi-classification model cannot determine the category location). In the embodiment of the present application, the target inspection model is designed to first perform defect location and classification inspection on the defect map, and then determine the final category based on the distance from the area of ​​interest and the category confidence, thereby ensuring the normal convergence of the model and the acquisition of accurate category labels.

[0088] The present embodiment further provides a practical application flow of model training and image classification. Figure 11 is a schematic flowchart of the model training and classification flow. As shown in Figure 11, first, during model training, a defect map is acquired, and then the AOI device can cut out the corresponding template map from the golden die (grain) image based on the defect map to generate a label file for the defect image.

[0089] The defect image and template image can then be processed using geometric transformation, R-stitch stitching, and other processes. After processing is complete, the processed photos are divided into a training set and a validation set according to a certain proportion. The training set is used to train the target model. During training, a loss function is constructed to determine whether the training process is complete, and the model parameters can be updated using backpropagation. Specifically, training can be determined to be complete when the function value of the loss function stabilizes. When using the validation set to validate the trained model, the validation set is input to the trained model, the mAP corresponding to each model is calculated, and the model with the highest mAP can be determined as the final optimal model.

[0090] When applying the optimal model to image classification tasks, the AOI equipment first extracts a corresponding template image from the golden die image based on the image to be inspected, and then inputs the template image and the image to be inspected into the optimal model. The optimal model determines the locations and types of all defects in the image to be inspected, and determines the category of the image to be inspected based on the categories of the defect bounding boxes closest to the center of the image to be inspected. Specifically, the defect category corresponding to the defect bounding box with the highest reliability among the defect bounding boxes can be determined as the category of the image to be inspected.

[0091] An embodiment of the present application provides a defect inspection model training device, and FIG. 12 is a schematic structural diagram of the device.

[0092] As shown in FIG. 12 , the apparatus includes: an input module 120 configured to obtain a training sample set and first defect label information corresponding to each group of samples in the training sample set, where each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first position information of each defect in the defect image; a first processing module 122 configured to determine a target loss function including a defect category loss function and a defect location loss function; and a second processing module 124 configured to train a trained model with the training sample set, the defect label information, and the target loss function using a stochastic gradient descent method to obtain a target defect inspection model, where the target defect inspection model is configured to determine defect location information and defect category information in the inspection target defect image, where the trained model is configured to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, where the second label information includes second category information and second position information of each defect in the defect image determined by the trained model.

[0093] In some embodiments of the present application, the step of the second processing module 124 using a stochastic gradient descent method to train a training object model with the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model includes a first step of inputting a defect image and a template image into the training object model, respectively, to obtain second label information output from the training object model, where the second label information includes second category information and second location information of each defect in the defect image determined by the training object model; and inputting the first category information and the second category information into a defect category loss function to obtain a first loss function value, and inputting the first location information and the second location information into a defect location loss function. The method includes a second step of inputting the first loss function value to obtain a second loss function value and storing the first loss function value and the second loss function value; a third step of adjusting model parameters of the trained model with the first loss function value and the second loss function value using a stochastic gradient descent method; and a fourth step of acquiring all the stored first loss function values ​​and second loss function values, and determining the adjusted trained model as a target defect inspection model if a difference between any two first loss function values ​​among a predetermined number of consecutive first loss function values ​​is less than a first predetermined difference and a difference between any two second loss function values ​​among a predetermined number of consecutive second loss function values ​​is less than a second predetermined difference; otherwise, returning to the first step.

[0094] In some embodiments of the present application, the step of the second processing module 124 inputting the defect image and the template image into a trained model, respectively, to obtain second label information output from the trained model includes the steps of inputting the defect image and the template image into a backbone network of the trained model, where the backbone network is configured to obtain a defect feature image and a template feature image by extracting image features in the defect image and the template image, and obtaining the second label information output by the trained model based on the defect feature image and the template feature image.

[0095] In some embodiments of the present application, the training target model further includes a feature fusion layer and a feature pyramid layer, wherein the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image and input the first target feature image into the feature pyramid layer, and the feature pyramid layer is configured to output second label information based on the first target feature image.

[0096] In some embodiments of the present application, the feature fusion layer includes a first feature fusion layer, which is configured to perform a difference operation on the defect feature image and the template feature image to obtain a difference feature image, where the pixel value of any one pixel point in the difference feature image is equal to the absolute value of the difference between the defect pixel point corresponding to the any one pixel point and the template pixel point, where the defect pixel point is a pixel point corresponding to the any one pixel point in the defect feature image, and the template pixel point is a pixel point corresponding to the any one pixel point in the template feature image, perform a channel connection process on the difference feature image and the defect feature image to obtain a second target feature image, and perform a convolution process on the second target feature image with a target convolution kernel to obtain a first target feature image.

[0097] In some embodiments of the present application, the feature fusion layer includes a second feature fusion layer, which is configured to perform convolution processing on the defect feature image and the template feature image, respectively, via a target convolution kernel, and generate a first target feature image based on the defect feature image and the template feature image after the convolution processing, and the pixel value of any one pixel point in the first target feature image is equal to the sum of the pixel values ​​of the defect pixel point and the template pixel point corresponding to the any one pixel point.

[0098] In some embodiments of the present application, the step of the second processing module 124 determining the adjusted trained model as the target defect inspection model includes the steps of obtaining a validation sample set and third defect label information corresponding to each group of samples in the validation sample set, wherein each group of samples in the validation sample set includes a defect image and a template image corresponding to the defect image, and the third defect label information includes third category information and third location information of each defect in the defect image; and, if there are multiple adjusted trained models, determining the average accuracy average value of each adjusted trained model among the multiple adjusted trained models using the validation sample set and the third defect label information; and determining the adjusted trained model with the highest average accuracy average value as the target defect inspection model.

[0099] In some embodiments of the present application, the step of the input module 120 acquiring a training sample set includes the steps of acquiring a first defect image and a first template image corresponding to the first defect image, and performing a geometric transformation process on the first defect image and the first template image using the same processing method, and setting the first defect image after the geometric transformation process as a defect image sample and the first template image after the geometric transformation process as a template image sample, wherein the processing method includes at least one of random cropping, image flipping, and image stitching.

[0100] In some embodiments of the present application, the category information includes a defect category number of the defect, and the location information includes location information of the defective block corresponding to the defect.

[0101] Each module in the defect inspection model training device may be a program module (e.g., a set of program instructions that realize a specific function) or a hardware module. In the latter case, the implementation form of each of the above modules may be a single processor, or the functions of each of the above modules may be realized by a single processor, but is not limited to these.

[0102] An embodiment of the present application further provides a non-volatile storage medium having a program stored therein, the program, when executed, controlling a device in which the non-volatile storage medium is located to perform the following defect inspection model training method:

[0103] a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, wherein the first defect label information includes first category information and first location information of each defect in the defect image; a target loss function including a defect category loss function and a defect location loss function is determined; a training target model is trained using a stochastic gradient descent method with the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model, wherein the target defect inspection model is configured to determine defect location information and defect category information in the inspection target defect image; the training target model is configured to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, wherein the second label information includes second category information and second location information of each defect in the defect image determined by the training target model;

[0104] In some embodiments of the present application, when the program is executed, the program may further control a device in which the non-volatile storage medium is located to execute the following defect image classification method: determine an inspection-target defect image and a template image corresponding to the inspection-target defect image, input the inspection-target defect image and the template image into a target defect inspection model to obtain defect location information and defect category information output from the target defect inspection model, the target defect inspection model is configured to extract a third image feature of the inspection-target defect image and a fourth image feature of the template image, and output the defect location information and the defect category information based on the third image feature and the fourth image feature, determine a predetermined number of target defects in the inspection-target defect image that are closest to a center point of the inspection-target defect image based on the defect location information, determine a category reliability of each target defect among the predetermined number of target defects, and determine the defect category of the target defect with the highest category reliability as the defect category of the inspection-target defect image.

[0105] An embodiment of the present application further provides an electronic device, the electronic device including a memory and a processor, the processor being configured to execute a program stored in the memory, and when the program is executed, performing a defect inspection model training method as follows:

[0106] a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, wherein the first defect label information includes first category information and first location information of each defect in the defect image; a target loss function including a defect category loss function and a defect location loss function is determined; a training target model is trained using a stochastic gradient descent method with the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model, wherein the target defect inspection model is configured to determine defect location information and defect category information in the inspection target defect image; the training target model is configured to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, wherein the second label information includes second category information and second location information of each defect in the defect image determined by the training target model;

[0107] In some embodiments of the present application, the program may further execute the following defect image classification method when executed: determine an inspection target defect image and a template image corresponding to the inspection target defect image; input the inspection target defect image and the template image into a target defect inspection model to obtain defect location information and defect category information output from the target defect inspection model; configure the target defect inspection model to extract a third image feature of the inspection target defect image and a fourth image feature of the template image and output the defect location information and defect category information based on the third image feature and the fourth image feature; determine a predetermined number of target defects in the inspection target defect image that are closest to a center point of the inspection target defect image based on the defect location information; determine a category reliability of each target defect among the predetermined number of target defects; and determine the defect category of the target defect with the highest category reliability as the defect category of the inspection target defect image.

[0108] In the above embodiments of the present application, the description of each embodiment has different emphasis, and for the parts not described in detail in one embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] It should be understood that in the several embodiments provided in this application, the disclosed technical contents can be realized in other ways. The above-described device embodiments are merely schematic. For example, the division of the units may be a logical functional division. In actual implementation, other division methods may be used, for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not implemented. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections via several interfaces, units, or modules, and may be in electrical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed across multiple units. Depending on actual needs, some or all of these units may be selected to achieve the objectives of the solution of this embodiment.

[0111] Furthermore, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically independent, or two or more units may be integrated into one unit. The integrated units may be realized in the form of hardware or in the form of software functional units.

[0112] The integrated unit may be realized in the form of a software functional unit and stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the essential features of the technical solution of the present application, in other words, the part that contributes to the related art, or all or part of the technical solution, may be embodied in the form of a software product, and the computer software product is stored in a storage medium and includes several instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute all or part of the steps described in each embodiment of the present application. The storage medium includes various media capable of storing program code, such as a USB flash memory, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0113] It should be noted that the above-described contents are only preferred embodiments of the present application, and those skilled in the art may make some improvements and modifications without departing from the principles of the present application, and these improvements and modifications should also be regarded as within the protection scope of the present application. [Industrial Applicability]

[0114] The defect inspection model training method, apparatus and electronic device provided in the embodiments of the present application are applied in the field of image identification.

[0115] In an embodiment of the present application, a training sample set and first defect label information corresponding to each group of samples in the training sample set are obtained, each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, the first defect label information includes first category information and first location information of each defect in the defect image, a target loss function is determined, the target loss function includes a defect category loss function and a defect location loss function, a training target model is trained using a stochastic gradient descent method with the training sample set, the defect label information and the target loss function to obtain a target defect inspection model, the target defect inspection model is used to determine the defect location information and defect category information in the inspection target defect image, and the training target model is trained using a first image feature of the defect image and a second image feature of the template image. By using this method to train a defect inspection model using a training sample set including defect images and template images and the defect label information, the objective of obtaining a defect inspection model that can use the template image during defect identification is achieved, thereby achieving the technical effect of using the template image to assist in classification and identification in the process of identifying and classifying defects in crystal grain images, and ultimately solving the technical problem in the related art that poor identification and classification results are achieved when a template image is not used when identifying and classifying defects in crystal grain images.

Claims

1. 1. A method for training a defect inspection model, comprising: obtaining a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first location information of each defect in the defect image; determining a target loss function, including a defect category loss function and a defect location loss function; training a training object model with the training sample set, the defect label information, and the target loss function using a stochastic gradient descent method to obtain a target defect inspection model, wherein the target defect inspection model is used to determine defect position information and defect category information in an inspection object defect image, and the training object model is used to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, wherein the second label information includes second category information and second position information of each defect in the defect image determined by the training object model. How to train a defect inspection model.

2. the step of training a training target model using the training sample set, the defect label information, and the target loss function by using a stochastic gradient descent method to obtain a target defect inspection model, a first step of inputting the defect image and the template image into the training object model, respectively, and obtaining second label information output from the training object model; a second step of inputting the first category information and the second category information into the defect category loss function to obtain a first loss function value, inputting the first position information and the second position information into the defect position loss function to obtain a second loss function value, and storing the first loss function value and the second loss function value; a third step of adjusting model parameters of the training target model using the first loss function value and the second loss function value using stochastic gradient descent; a fourth step of acquiring all the stored first loss function values ​​and second loss function values, and determining the adjusted training target model as the target defect inspection model if the first loss function values ​​and the second loss function values ​​meet a predetermined condition, and otherwise proceeding to the first step, wherein the predetermined condition includes that all of the sums of the first loss function values ​​and the second loss function values ​​for a predetermined number of consecutive times are located within a target value interval. The method for training a defect inspection model according to claim 1 .

3. the step of inputting the defect image and the template image into the training object model, respectively, and obtaining second label information output from the training object model, inputting the defect image and the template image into a backbone network of the trained model, the backbone network being configured to obtain a defect feature image and a template feature image by extracting image features in the defect image and the template image; and acquiring the second label information output by the training model based on the defect feature image and the template feature image. The method for training a defect inspection model according to claim 2 .

4. The training object model further includes a feature fusion layer and a feature pyramid layer; the feature fusion layer is configured to obtain a first target feature image based on the defect feature image and the template feature image, and input the first target feature image into the feature pyramid layer; the feature pyramid layer is configured to output the second label information based on the first target feature image; The method for training a defect inspection model according to claim 3 .

5. the feature fusion layer includes a first feature fusion layer; the first feature fusion layer is configured to: perform a difference operation on the defect feature image and the template feature image to obtain a difference feature image; a pixel value of any one pixel point in the difference feature image is equal to an absolute value of a difference between a defect pixel point corresponding to the any one pixel point and a template pixel point; the defect pixel point is a pixel point corresponding to the any one pixel point in the defect feature image; and the template pixel point is a pixel point corresponding to the any one pixel point in the template feature image; perform a channel connection process on the difference feature image and the defect feature image to obtain a second target feature image; and perform a convolution process on the second target feature image via a target convolution kernel to obtain the first target feature image. The method for training a defect inspection model according to claim 4 .

6. the feature fusion layer includes a second feature fusion layer; the second feature fusion layer is configured to perform convolution processing on the defect feature image and the template feature image via a target convolution kernel, respectively, and generate the first target feature image based on the defect feature image and the template feature image after the convolution processing, wherein a pixel value of any one pixel point in the first target feature image is equal to the sum of pixel values ​​of a defect pixel point and a template pixel point corresponding to the any one pixel point; The method for training a defect inspection model according to claim 4 .

7. The step of determining the adjusted training target model as the target defect inspection model includes: acquiring a validation sample set and third defect label information corresponding to each group of samples in the validation sample set, wherein each group of samples in the validation sample set includes a defect image and a template image corresponding to the defect image, and the third defect label information includes third category information and third location information of each defect in the defect image; If the number of the adjusted trained models is multiple, determining an average accuracy average value of each of the adjusted trained models according to the validation sample set and the third defect label information; determining the adjusted trained model having the highest average accuracy as the target defect inspection model; The method for training a defect inspection model according to claim 2 .

8. The step of obtaining a training sample set comprises: acquiring a first defect image and a first template image corresponding to the first defect image; performing a geometric transformation process on the first defect image and the first template image using the same processing method, and setting the first defect image that has undergone the geometric transformation process as the defect image sample and the first template image that has undergone the geometric transformation process as the template image sample, wherein the processing method includes at least one of random cropping, image flipping, and image splicing; The method for training a defect inspection model according to claim 1 .

9. the category information includes a defect category number of the defect, and the location information includes location information of a defect block corresponding to the defect. The method for training a defect inspection model according to claim 1 .

10. 1. A defect image classification method, comprising: determining an inspection target defect image and a template image corresponding to the inspection target defect image; a step of inputting the inspection target defect image and the template image into a target defect inspection model to obtain defect position information and defect category information output from the target defect inspection model, wherein the target defect inspection model is used to extract a third image feature of the inspection target defect image and a fourth image feature of the template image, and output the defect position information and the defect category information based on the third image feature and the fourth image feature; determining a predetermined number of target defects in the inspection target defect image that are closest to a center point of the inspection target defect image based on the defect position information; determining a category reliability of each target defect among the predetermined number of target defects, and determining a defect category of the target defect having the highest category reliability as a defect category of the inspection target defect image, Defect image classification method.

11. 1. A defect inspection model training device, comprising: an input module configured to acquire a training sample set and first defect label information corresponding to each group of samples in the training sample set, wherein each group of samples in the training sample set includes a defect image and a template image corresponding to the defect image, and the first defect label information includes first category information and first location information of each defect in the defect image; a first processing module configured to determine a target loss function, the target loss function including a defect category loss function and a defect location loss function; a second processing module configured to train a training object model using the training sample set, the defect label information, and the target loss function to obtain a target defect inspection model, the target defect inspection model configured to determine defect position information and defect category information in a confirmed inspection target defect image, the training object model configured to extract first image features of the defect image and second image features of the template image, and output second label information of the defect image based on the first image features and the second image features, the second label information including second category information and second position information of each defect in the defect image determined by the training object model. A defect inspection model training device.

12. a nonvolatile storage medium having a program stored therein, the program controlling a device in which the nonvolatile storage medium is located when the program is executed to execute the defect inspection model training method according to any one of claims 1 to 9 or the defect image classification method according to claim 10; Non-volatile storage media.

13. An electronic device including a memory and a processor, wherein the processor is used to execute a program stored in the memory, and when the program is executed, the defect inspection model training method according to any one of claims 1 to 9 or the defect image classification method according to claim 10 is executed. electronic equipment.

Images