Optimization method, device, equipment and storage medium for defect detection model

Abstract

The present application discloses an optimization method, an optimization device, an electronic device, and a storage medium for a defect detection model. The defect detection model is obtained by training a neural network model based on a basic image set, and the optimization method includes: obtaining terminal device data and product design images; when the product design image changes, the defect detection model is trained based on the product captured image and the basic image set to obtain a target defect detection model, or / and, when the terminal device data changes, the defect detection model is trained based on the product captured image and the basic image set collected by the terminal device to obtain a target defect detection model. In the optimization method of the present application, when the product design image is sent to change, or during the equipment and process transformation, a new image set can be quickly generated, reducing the time for the defect detection model to collect data, thereby shortening the model iteration training cycle and reducing the impact on production line production when products and equipment are replaced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a defect detection model optimization method, a defect detection model optimization device, an electronic device and a storage medium. Background Art

[0002] During the screen production process, a deep learning-based defect detection model is needed to detect bad images. The defect detection model requires a large number of images as training data sets. The number of training data sets has a great impact on the overall effect of the defect detection model. The more training data sets there are, the better the model effect will be. However, equipment replacement and new production products in the factory will cause the images taken by the equipment to change from the original data set, resulting in the inability of the defect detection model to detect and identify defects. To obtain a training data set, it is usually necessary to collect and label, clean data, and other steps, which is time-consuming and inefficient. Summary of the invention

[0003] The present application aims to solve at least one of the technical problems existing in the prior art. To this end, the present application provides a defect detection model optimization method, a defect detection model optimization device, an electronic device and a non-volatile computer-readable storage medium.

[0004] The defect detection model of the embodiment of the present application is obtained by training a neural network model according to a basic image set. The optimization method of the defect detection model includes:

[0005] Obtain terminal equipment data and product design images;

[0006] In the case where the product design image changes, training the defect detection model based on the new product image and the basic image set to obtain a target defect detection model; or / and

[0007] In the case where the terminal device data changes, the defect detection model is trained according to the product shooting images collected by the terminal device and the basic image set to obtain a target defect detection model.

[0008] In some embodiments, the step of training the defect detection model according to the product captured image and the basic image set to obtain a target defect detection model includes:

[0009] When the new product design image matches the basic image set, testing the recall rate and / or accuracy rate of the defect detection model;

[0010] In the case where the recall rate and / or the precision rate do not meet the requirements, target new product images whose test results are lower than a preset threshold are collected from the new product images;

[0011] The labeled and classified target new product images are used as a first training set;

[0012] The defect detection model is trained according to the first training set and the basic image set to obtain the target defect detection model that meets the standards.

[0013] In some embodiments, the step of training the defect detection model according to the product captured image and the basic image set to obtain a target defect detection model further includes:

[0014] When the recall rate and / or accuracy rate meet the requirements, the defect detection model is used as the target defect detection model.

[0015] In some implementations, taking the labeled and classified photographed images of the new product as a first training set includes:

[0016] Constructing a first sample set;

[0017] Import the labeled and classified photographed images of the new product into the first sample set to obtain the first training set.

[0018] In some embodiments, when the product design image changes, the optimization method further includes:

[0019] In a case where the new product design image matches the basic image set, the defect detection model is used as the target defect detection model.

[0020] In some embodiments, when the product design image changes, the optimization method further includes:

[0021] In the case that the new product design images do not match the basic image set, the labeled and classified new product images are used as a second training set;

[0022] The neural network model is trained using the second training set to obtain the target defect detection model.

[0023] In some embodiments, when the terminal device data changes, training the defect detection model according to the product captured image and the basic image set collected by the terminal device to obtain a target defect detection model includes:

[0024] Obtaining a production plan and / or a replacement plan for the terminal device;

[0025] In case the production plan and / or the replacement plan changes, testing the recall rate and / or accuracy rate of the defect detection model according to the product captured images;

[0026] When the recall rate and / or accuracy rate of the defect detection model does not meet the standard, collecting target product images whose test results are lower than a preset threshold among the product images;

[0027] The labeled and classified images of the target product are used as a third training set;

[0028] The defect detection model is trained according to the third training set and the basic image set to obtain a target defect detection model.

[0029] In some embodiments, when the terminal device data changes, training the defect detection model according to the product captured image and the basic image set collected by the terminal device to obtain a target defect detection model includes:

[0030] When the recall rate and / or accuracy rate of the defect detection model meets the requirements, the defect detection model is used as the target defect detection model.

[0031] In some embodiments, taking the labeled and classified captured images of the target product as a third training set includes:

[0032] constructing a second sample set;

[0033] The labeled and classified product images are imported into the second sample set to obtain the third training set.

[0034] In certain embodiments, the optimization method further comprises:

[0035] Build image collections;

[0036] The labeled and classified product images are added to the image set to obtain the basic image set.

[0037] The optimization device of the defect detection model of the embodiment of the present application includes:

[0038] An acquisition module, used to acquire terminal device data and product design images;

[0039] A first optimization module is used for training the defect detection model according to the new product shot image and the basic image set to obtain a target defect detection model when the product design image changes; or / and

[0040] The second optimization module is used to train the defect detection model according to the product shooting images collected by the terminal device and the basic image set to obtain a target defect detection model when the terminal device data changes.

[0041] The electronic device provided in the embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the optimization method of the defect detection model.

[0042] The non-volatile computer-readable storage medium provided in the embodiments of the present application includes a computer program. When the computer program is executed by a processor, the processor executes the optimization method of the defect detection model.

[0043] In the defect detection model optimization method, optimization device, electronic device and computer-readable storage medium of the implementation mode of the present application, when the product design image changes, the new product shot image and the basic image set are formed into a test set, and then the test set is used to train the existing defect detection model to obtain the target defect detection model, or when the terminal device changes, the defect detection model is trained according to the product shot image and the basic image set collected by the terminal device to obtain the target defect detection model. In this way, the time for the defect detection model to collect data can be reduced, thereby shortening the model iteration training cycle, improving the training efficiency of the defect detection model, facilitating the rapid deployment of the model in the factory, reducing the impact of product changes or changes in terminal equipment on production line production, and improving production efficiency.

[0044] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 is a flow chart of a method for optimizing a defect detection model in certain embodiments of the present application;

[0047] Figure 2 It is a schematic diagram of a module of an optimization device of certain embodiments of the present application;

[0048] Figure 3-4 is a flow chart of a method for optimizing a defect detection model in certain embodiments of the present application;

[0049] Figure 5-6 This is a schematic diagram of a scenario for creating a new product training set in certain embodiments of the present application;

[0050] Figure 7-10 is a flow chart of a method for optimizing a defect detection model in certain embodiments of the present application;

[0051] Figure 11-12is a schematic diagram of a scenario for creating a training set in certain embodiments of the present application;

[0052] Figure 13-14 It is a schematic diagram of the process flow of the optimization method of certain embodiments of the present application. DETAILED DESCRIPTION

[0053] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0054] See also Figure 1 The present application provides an optimization method for a defect detection model, wherein the defect detection model is obtained by training a neural network model according to a basic image set, and the optimization method includes:

[0055] 01. Obtain terminal device data and product design images;

[0056] 02. When the product design image changes, the defect detection model is trained based on the new product image and the basic image set to obtain the target defect detection model; or / and

[0057] 03. When the terminal device data changes, the defect detection model is trained based on the product images and basic image sets collected by the terminal device to obtain the target defect detection model.

[0058] See also Figure 2 The embodiment of the present application provides an optimization device 10 for a defect detection model. The optimization device 10 includes an acquisition module 110, a first optimization module 120, and a second optimization module 130. Step 01 can be implemented by the acquisition module 110, step 02 can be implemented by the first optimization module 120, and step 03 can be implemented by the second optimization module 130.

[0059] In other words, the acquisition module 110 can be used to acquire the terminal device data and the product design image, and the first optimization module 120 can be used to train the defect detection model based on the new product shot image and the basic image set to obtain the target defect detection model when the product design image changes. The second optimization module 130 can be used to train the defect detection model based on the product shot image and the basic image set collected by the terminal device to obtain the target defect detection model when the terminal device data changes.

[0060] An embodiment of the present application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program. When the computer program is executed by the processor, the processor executes the above-mentioned defect detection model optimization method, that is, the processor can be used to obtain terminal device data and product design images, and when the product design images change, the defect detection model is trained based on the new product captured images and the basic image set to obtain a target defect detection model; and when the terminal device data changes, the defect detection model is trained based on the product captured images and the basic image set collected by the terminal device to obtain a target defect detection model.

[0061] In the optimization method, optimization device 10, electronic device and storage medium of the implementation mode of the present application, when the product design image changes, the new product shot image and the basic image set are formed into a test set, and then the test set is used to train the existing defect detection model to obtain the target defect detection model, or when the terminal device changes, the defect detection model is trained according to the product shot image and the basic image set collected by the terminal device to obtain the target defect detection model. In this way, the time for the defect detection model to collect data can be reduced, thereby shortening the model iteration training cycle, improving the training efficiency of the defect detection model, facilitating the rapid deployment of the model in the factory, reducing the impact of product changes or changes in terminal equipment on production line production, and improving production efficiency.

[0062] In some implementations, the optimization device 10 may be a part of an electronic device. In other words, the electronic device includes the optimization device 10.

[0063] In some embodiments, as hardware, the optimization device 10 can be independent or installed as an additional peripheral component to a computer or computer system. The optimization device 10 can also be integrated into a computer or computer system, for example, when the optimization device 10 is part of an electronic device, the optimization device 10 can be integrated into a processor.

[0064] In some implementations where the optimization device 10 is part of an electronic device, as software, the code segments corresponding to the optimization device 10 can be stored in a memory and executed by a processor to implement the aforementioned functions. In other words, the optimization device 10 includes the aforementioned computer program, or the aforementioned computer program includes the optimization device 10.

[0065] In this embodiment, the product can be described as a display screen, wherein the basic image set can be product-shot images of the display screen. That is, in the optimization method of the defect detection model in this application, the display screen in the manufacturing process is used as an example for description. It can be understood that the above is only an example of the optimization method, and does not strictly limit the object to which the optimization method in the embodiment of the present invention is applied.

[0066] The electronic device may be a server, and the terminal device may be an automated optical inspection device (AOI) for detecting products. The terminal device may include an automatic classification system (ADC) for detecting images of products to determine whether the display screen is abnormal. The server may be connected to the AOI device for communication, so that the server can obtain product images of the product from the AOI device. The server may include a big data platform, which is a platform integrating data access, data processing, data storage, query retrieval, analysis and mining, application interface, etc. In this way, the electronic device can implement the optimization method of the defect detection model of the implementation mode of the present application.

[0067] It should be noted that AOI equipment refers to equipment that detects common defects encountered in product production based on optical principles. When the AOI equipment automatically detects, the AOI equipment automatically scans through the camera and collects the product image captured by the display screen.

[0068] ADC is also known as an automated detection system. Defect detection models can be integrated into the automated detection system, such as a defect detection model based on a neural network. In this way, artificial intelligence, image recognition and other technologies can be used to automatically detect and classify key defects in the panel production process, which can greatly improve the company's quality inspection efficiency, accuracy, yield and production efficiency.

[0069] Next, the defect detection process and principle of the neural network-based defect detection model are explained.

[0070] The neural network-based defect detection model can be integrated in an automated detection device, which can be one or more devices capable of performing artificial intelligence reasoning (visual recognition) and training tasks using GPU computing resources. The reasoning task based on artificial intelligence can be a defect recognition task based on a defect detection model, which is mainly a neural network model based on deep learning. For example, the defect detection model can be based on a feedforward neural network. The feedforward network can be implemented as an acyclic graph, in which nodes are arranged in layers. Typically, the feedforward network topology includes an input layer and an output layer, which are separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating output in the output layer. The network nodes are fully connected to the nodes in the adjacent layers via edges, but there are no edges between the nodes in each layer. The data received at the nodes of the input layer of the feedforward network is propagated (i.e., "feedforward") to the nodes of the output layer via an activation function, and the activation function calculates the state of the nodes of each continuous layer in the network based on coefficients ("weights"), and the coefficients are respectively associated with each of the edges connecting these layers. The output of the defect detection model can take various forms, and the present disclosure is not limited to this. The defect detection model may also include other neural network models, such as a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a generative adversarial network (GAN) model, and a transformation attention (Transformer) model, but is not limited thereto. Other neural network models known to those skilled in the art may also be used.

[0071] Neural network models usually need to be obtained through training. In the disclosed embodiment, the automated detection device also trains the neural network model, which mainly includes the following steps: selecting a network topology; using a set of training data representing the problem modeled by the network; and adjusting the weights until the network model exhibits the minimum error for all instances of the training image set. For example, during a supervised learning training process for a neural network, the output generated by the network in response to an input representing an instance in the training image set is compared with the "correct" labeled output of the instance; an error signal representing the difference between the output and the labeled output is calculated; and when the error signal is propagated backward through the layers of the network, the weights associated with the connection are adjusted to minimize the error. When the error of each output generated from an instance of the training image set is minimized, the neural network model is considered to be "trained" and can be used for artificial intelligence reasoning tasks.

[0072] The reasoning and training process of the neural network model involves a large number of parameters, activation values, and gradient value buffers, each of which must be completely updated in each training iteration, which places very high demands on computing and throughput capabilities. GPUs have strong parallel capabilities and high video memory bandwidth, and are extremely efficient in processing calculations associated with training deep neural networks. Therefore, using a GPU cluster integrated with multiple GPUs can effectively improve the training and reasoning speed of the neural network model, and the product defect recognition cluster 202 can also schedule and allocate hardware resources.

[0073] In at least one embodiment of the present disclosure, the automated inspection equipment may be configured to identify product defect content in a product shot image through a defect detection model based on a first defect recognition task, wherein the product defect content includes any one or more of the product defect type, defect location, and defect size. The product defect content includes any one or more of the product defect type, defect location, and defect size. After receiving the first defect recognition task, the automated inspection equipment uses a trained defect detection model to identify product defect content in the product shot image stored in the product image database, that is, to identify any one or more of the product defect type, defect location, and defect size.

[0074] In an embodiment of the present disclosure, based on the first defect recognition task, the implementation method of identifying the product defect content in the product shot image through the defect detection model is as follows: first, the product shot image is scaled to a fixed pixel size MxN (or not scaled), and then the MxN image is sent to a deep convolutional neural network (VGG / Resnet / MobileNet, etc.); through multiple layers of convolutional layers, activation layers, and pooling layers, the feature map of the entire image is obtained; the feature map is input into the screening region network (ZF / SSD / RPN, etc.), and after calculation, the proposal region is obtained; then, convolution and pooling operations are performed on the proposal region to obtain the regional feature map (proposal feature) of the proposal region, and then the regional feature map (proposal feature) is sent to the subsequent full connection and softmax network for classification (classification, that is, what kind of defect the proposal is), and the defect category with the highest probability is obtained as the final classification result, and the category and probability are recorded. In addition, the coordinates and size of the proposal region represent the location and size of the defect. The method for identifying product defect content based on the defect model may adopt similar variations of the above method or other methods known to those skilled in the art, and the present disclosure does not limit this.

[0075] For example, in the semiconductor industry, defect types may include particles, remain, line defects, holes, splashes, wrinkles, film color, bubbles, etc., and more subcategories may be made based on each defect type. For example, particle types may be divided into normal particles P1 (Normal Particle), sand particles P2 (Sand Particle), and wall particles P3 (Wall Particle) based on particle defects of different forms. The defect location may indicate the specific coordinate location of the defect in the product image, thereby determining the specific coordinate location of the defect on the circuit board, the relationship between the defect and the background circuit (e.g., whether the defect is completely within the gate, intersecting with it, or near the gate), etc. The defect size may include the length of the defect or the area of ​​the defect.

[0076] In addition, the process of how the image acquisition device AOI acquires images and transmits images is explained.

[0077] The product images generated during the product manufacturing process may include product images acquired by the image acquisition device AOI during the manufacturing process, and may also include product images acquired through other means and stored in the product image database. For example, after the AOI device acquires the product images, it can be directly stored in the product image database, or it can be stored in the factory database, and then the product images in the factory database are transferred to the product image database. The transmission of product images from the factory database to the product image database can ensure that the distributed product defect analysis system will not affect other data in the factory when calling, storing, and other operations on the product images in the product image database. The product image database can be a distributed file system (Distributed File System, DFS) or a relational database (Oracle, Redis, and MySQL).

[0078] When the product image changes, the terminal device may not be able to accurately detect the product defects or abnormalities through the defect detection model. Changes in the product or changes in the terminal device that collects the product image may cause the product image to change.

[0079] It is understandable that in actual production, different products have different corresponding circuit designs, so different products will have different product images after passing through the AOI equipment. Moreover, although the hardware parameters are the same between different terminal devices, due to slight differences in light sources, microscope imaging, etc., it is precisely because of the differences between these hardware devices that the background brightness and clarity of the images taken by different terminal devices during the production of the same product are different. Therefore, when the terminal device or its components used to detect the product are replaced, or the product detected by the terminal device changes, the terminal device may not be able to accurately detect the defects or abnormalities of the product through the defect detection model, resulting in a higher factory defect rate for the product. The defect detection model needs to be retrained to ensure the accuracy of the defect detection model.

[0080] It should be noted that the product design image refers to the pattern obtained by the circuit design of the new product before production. Before the new product is officially mass-produced, the circuit design and trial production will be carried out. After the circuit design is completed, the circuit design pattern of the model has been basically determined and will enter the trial production plan.

[0081] It is also necessary to explain that when the product design image changes, it can be considered that a new product has been replaced. When the terminal device data changes, it can be considered that the terminal device has been changed and has become a new terminal device. The new product image refers to the image of the new product captured by the terminal device.

[0082] Therefore, the present application forms a test set based on the images captured by the new product and the basic image set, and then trains the existing defect detection model with the test set to obtain a target defect detection model. Alternatively, the defect detection model is trained by using the product images captured by the new terminal device and the original basic image set to obtain the target defect detection model. This can ensure the accuracy of the target defect model and shorten the training time.

[0083] See also Figure 3 In some embodiments, step 02 comprises the steps of:

[0084] 021, testing the recall and / or precision of defect detection models when new product design images are matched to a base set of images;

[0085] 022. When the recall rate and / or precision rate do not meet the requirements, target new product images with test results below a preset threshold are collected from the new product images;

[0086] 023, take the labeled and classified target new product images as the first training set;

[0087] 024. Train a defect detection model according to the first training set and the basic image set to obtain a target defect detection model that meets the standards.

[0088] Please further combine Figure 2 In some embodiments, steps 021-024 can be implemented by the first optimization module 120. That is, the first optimization module 120 can be used to test the recall rate and / or accuracy of the defect detection model when the new product design image matches the basic image set, and when the recall rate and / or accuracy do not meet the standards, collect target new product images whose test results are lower than a preset threshold among the new product images, use the labeled and classified target new product images as the first training set, and train the defect detection model according to the first training set and the basic image set to obtain a target defect detection model that meets the standards.

[0089] In certain embodiments, the processor can be used to test the recall rate and / or accuracy rate of the defect detection model when the new product design image matches the basic image set. When the recall rate and / or accuracy rate do not meet the standards, target new product images with test results below a preset threshold are collected from the new product images, and the labeled and classified target new product images are used as the first training set. The defect detection model is trained based on the first training set and the basic image set to obtain a target defect detection model that meets the standards.

[0090] It should be noted that whether the new product design image matches the basic image set is determined by the image similarity between the new product image and the images in the basic image set. For example, when the image similarity between the new product design image and the images in the basic image set reaches more than 90%, it can be determined that the new product design image matches the basic image set.

[0091] It should also be noted that the recall rate refers to the number of correct results divided by the total number of correct results. The higher the recall rate, the higher the detection accuracy of the defect detection model, and the more the defect detection model meets the requirements. The accuracy rate refers to the proportion of all correct predictions (positive and negative).

[0092] It is understandable that if there are differences between the images detected by the defect detection model and the images in the basic image set, such differences can easily lead to fluctuations in the recall rate or accuracy of the defect detection model. If the recall rate and / or accuracy rate do not meet the standards, it means that the defect detection model is difficult to use for detecting new products. Therefore, it is necessary to train the defect detection model to obtain a qualified target detection model.

[0093] In this embodiment, the defect detection model can be tested by the new product images captured by the terminal device to obtain the recall rate and / or accuracy rate of the defect detection model. If the recall rate and / or accuracy rate do not meet the standard, the new product images with test results lower than the preset threshold value are collected from the new product images and used as the target new product images. The preset threshold value is not limited, for example, the recall rate and / or accuracy rate can be 95%, 90%, etc. The preset threshold value can be set according to the actual situation.

[0094] Furthermore, after acquiring the captured images of the new product, the target captured images of the new product can be manually classified and labeled, so that the labeled and classified target captured images of the new product can be used as the first training set, and then the first training set and the existing basic image set of the defect detection model are used together to train the defect detection model, so that the recall rate and / or accuracy of the trained defect detection model meet the requirements, and the defect detection model with the met recall rate and / or accuracy is used as the target defect detection model for detecting new products.

[0095] In this way, by taking the existing defect detection model as the base model and quickly generating a new training set based on the first training set obtained by taking images of new products and the existing basic image set, the time for the defect detection model to collect data can be reduced, thereby shortening the model iteration training cycle, improving the training efficiency of the defect detection model, and reducing the impact of new products on production line production.

[0096] See also Figure 4 In some embodiments, step 023 further includes the sub-steps of:

[0097] 0231, construct the first sample set;

[0098] 0232. Import the labeled and classified target new product images into the first sample set to obtain a first training set.

[0099] Please further combine Figure 2 In some implementations, steps 0231-0232 may be implemented by the first optimization module 120. That is, the first optimization module 120 may be used to construct a first sample set, and import the labeled and classified target new product images into the first sample set to obtain a first training set.

[0100] In some implementations, the processor may be used to construct a first sample set, and import the labeled and classified target new product images into the first sample set to obtain a first training set.

[0101] Please combine Figure 5 and Figure 6For example, in some examples, when building a sample set, first enter the sample set name, select the sample set structure and annotation type, and click Confirm to create a sample set for the new product. After the new sample set is created, click Import to jump to the sample set import interface. Select the local import method, and the annotation status can be selected as no annotation information or with annotation information. In the state with annotation information, select the annotation data format and label category to be imported. After the selection is completed, click "Import" to upload the image set of the new product that has been labeled and classified to the sample set to obtain the training set of the new product.

[0102] In this way, the first training set and the basic image set can be used as training sets to train the defect detection model to obtain a target defect detection model for the new product.

[0103] See also Figure 7 In some embodiments, after sub-step 021, step 02 further includes sub-steps:

[0104] 025. When the recall rate and / or precision rate meet the requirements, the defect detection model is used as the target defect detection model.

[0105] Please further combine Figure 2 In some implementations, step 025 may be implemented by the first optimization module 120. That is, the first optimization module 120 may be used to use the defect detection model as the target defect detection model when the recall rate and / or the accuracy rate meet the requirements.

[0106] In some embodiments, the processor may be configured to use the defect detection model as a target defect detection model when the recall rate and / or the precision rate meet the requirements.

[0107] For example, in some examples, when the recall rate of the defect detection model meets the standard, the defect detection model is used as the target defect detection model for detecting new products. For another example, in some examples, when the accuracy of the defect detection model meets the standard, the defect detection model is used as the target defect detection model for detecting new products. For another example, in some examples, when the accuracy and recall rate of the defect detection model meet the standard, the defect detection model is used as the target defect detection model for detecting new products.

[0108] In this way, by using the existing defect detection model as the target defect detection model for new products when the recall rate and / or accuracy rate meet the requirements, it is possible to obtain the target defect detection model for new products without training the neural network model, and it is also unnecessary to collect, classify and label the images taken of new products, which greatly reduces the impact of new products on production lines. In addition, the accuracy of the target defect detection model for new products is guaranteed.

[0109] See also Figure 8 In some embodiments, after step 01, the optimization method further comprises:

[0110] 04, where the new product design image is matched with the base image set, the defect detection model is used as the target defect detection model.

[0111] Please further combine Figure 2 In some implementations, step 04 may be implemented by the first optimization module 120. That is, the first optimization module 120 may be used to use the defect detection model as the target defect detection model when the new product design image matches the basic image set.

[0112] In some embodiments, the processor may be configured to use the defect detection model as a target defect detection model when the new product design image matches the base image set.

[0113] In this way, by directly converting the existing defect detection model into the target defect detection model for the new product, there is no need to train a neural network model to obtain the target defect detection model for the new product. This also saves the need to collect, classify and label the images of the new product, greatly reducing the impact of the new product on the production line.

[0114] Please combine Fig. 9 In some embodiments, after step 01, the optimization method further comprises:

[0115] 05. In the case that the new product design images do not match the basic image set, the labeled and classified new product images are used as the second training set;

[0116] 06. Train the neural network model using the second training set to obtain a target defect detection model.

[0117] Please further combine Figure 2 In some embodiments, steps 05 and 06 can be implemented by the first optimization module 120. That is, the first optimization module 120 can be used to use the labeled and classified new product images as the second training set when the new product design images do not match the basic image set, and train the neural network model through the second training set to obtain the target defect detection model.

[0118] In some embodiments, the processor can be used to use the labeled and classified new product images as a second training set when the new product design images do not match the basic image set, and train the neural network model with the second training set to obtain a target defect detection model.

[0119] It should be noted that the images of new products can be manually annotated and classified, and then imported into the newly created sample set after the annotated and classified, so as to obtain the second training set.

[0120] It is understandable that if the design image does not match the basic image set, it means that the design image has different features from the images in the existing basic image set, and the defect detection model is trained based on the basic image set. The defect detection model usually only detects images with the same or similar features as the images in the basic image set, and the defect detection model may not be able to detect the design image. Therefore, when the design image of a new product does not match the basic image set, the terminal device can collect images of the new product, and then generate a second training set with the images of the new product that have been manually annotated and classified, so that the neural network model can be trained based on the second training set to obtain a target defect detection model for detecting new products.

[0121] In this way, subsequent terminal devices can detect new products based on the target defect detection model.

[0122] See also Fig.10 In some embodiments, step 03 includes the sub-steps of:

[0123] 031, obtain production plan and / or terminal equipment replacement plan;

[0124] 032, Test the recall and accuracy of the defect detection model based on product images in case of changes in production schedule and / or replacement schedule;

[0125] 033. When the recall rate and / or accuracy rate of the defect detection model does not meet the standard, collect target product images whose test results are lower than a preset threshold among the product images;

[0126] 034, the labeled and classified target product images are used as the third training set;

[0127] 035. Train the defect detection model according to the third training set and the basic image set to obtain a target defect detection model.

[0128] Please further combine Figure 2 In some embodiments, sub-steps 031-035 can be implemented by the second optimization module 130. In other words, the second optimization module 130 can be used to obtain the production plan and the replacement plan of the terminal equipment, and in the case where the production plan and / or the replacement plan change, confirm the recall rate and accuracy of the defect detection model according to the product shooting image, and in the case where the recall rate and / or accuracy of the defect detection model do not meet the standard, collect the target product shooting images whose test results are lower than the preset threshold in the product shooting images, and use the labeled and classified target product shooting images as the third training set, and finally train the defect detection model according to the target product shooting images and the basic image set to obtain the target defect detection model.

[0129] In certain embodiments, the processor can be used to obtain a production plan and a replacement plan for terminal equipment, and in the event that the production plan and / or replacement plan change, confirm the recall and accuracy of the defect detection model based on the product images, and if the recall and / or accuracy of the defect detection model do not meet the standards, collect target product images whose test results are lower than a preset threshold, and use the labeled and classified target product images as a third training set, and finally train the defect detection model based on the target product images and the basic image set to obtain a target defect detection model.

[0130] It should be noted that the production plan or the replacement plan of the terminal equipment can determine whether the terminal equipment is replaced, whether new equipment is used for testing, or whether equipment parts are replaced. When the terminal equipment is replaced, new equipment is used for testing, or parts of the terminal equipment are replaced, the terminal equipment can be regarded as a new terminal equipment.

[0131] Specifically, after the terminal equipment is replaced, the new equipment is used for testing or the equipment parts are replaced according to the production plan and the terminal equipment replacement plan, the recall rate and / or accuracy rate fluctuations of the defect detection model of the product images taken by the terminal equipment are observed. When the recall rate and / or accuracy rate decreases and is lower than the preset threshold, it is confirmed that the recall rate and accuracy rate of the defect detection model do not meet the standards, and the product images with a level lower than the preset threshold value judged by the defect detection model are extracted as the target product images. According to the production quantity and production schedule of the products, products with large production volume and urgent delivery are given priority, and the collected target product images are labeled and classified. After the labeling is completed, the third training set is obtained, and the defect detection model is retrained with the third training set and the basic image set so that the defect detection model meets the standards, and then the defect detection model that meets the standards is used as the target defect detection model of the terminal equipment.

[0132] Please combine Fig.11 and Fig.12, further, when training the defect detection model according to the third training set and the basic image set, it is necessary to import the images in the third training set and the images in the basic image set into the common image set in order to train the defect detection model. Specifically, first create a sample set, and import the product images captured when the terminal device has not changed and labeled and classified to obtain the basic image set. Further, create data management, select "ADC" to import in the "Import method" drop-down menu. Select the start and end time, process name, site name, product name, equipment ID number, code type in turn, and select "fail" for ADC pass, that is, select the product images that have not been identified by the ADC defect detection model and have been reviewed as the target product images, click "ADC query", and then set the "extraction quantity", that is, determine the amount of data to be supplemented, and click "Confirm import" to complete the import of the supplementary image set. After the import is completed, the new image set contains two parts, one is the basic image set A, and the other is the newly collected image set B that has not been identified by the defect detection model. We use the basic model trained with image set A as the defect detection model, and then use A+B as the new image set for training to obtain the target defect detection model.

[0133] In this way, by taking the existing defect detection model as the basic model and quickly generating a new training set based on the first training set obtained from the product images captured by the new terminal device and the existing basic image set, the time for the defect detection model to collect training data can be reduced, thereby shortening the model iteration training cycle, improving the training efficiency of the defect detection model, and reducing the impact of terminal equipment changes on production line production.

[0134] See also Fig.13 In some embodiments, sub-step 034 includes:

[0135] 0341, construct the second sample set;

[0136] 0342, import the labeled and classified target product images into the second sample set to obtain a third training set.

[0137] Please further combine Figure 2 In some implementations, steps 0341-0242 may be implemented by the second optimization module 130. That is, the second optimization module 130 may be used to construct a second sample set, and import the labeled and classified target product images into the second sample set to obtain a third training set.

[0138] In some implementations, the processor may be used to construct a second sample set, and import the labeled and classified photographed images of the target product into the second sample set to obtain a third training set.

[0139] In this way, the third training set and the basic image set can be used as training sets to train the defect detection model to obtain a target defect detection model after the terminal device is changed.

[0140] See also Fig.14 In some embodiments, step 03 further includes the sub-steps of:

[0141] 036. When the recall rate and / or precision rate of the defect detection model meets the requirements, the defect detection model is used as the target defect detection model.

[0142] Please further combine Figure 2 In some embodiments, sub-step 036 may be implemented by the second optimization module 130. In other words, the second optimization module 130 may be used to use the defect detection model as the target defect detection model when the recall rate and / or accuracy rate of the defect detection model meets the requirements.

[0143] In some embodiments, the processor may be configured to use the defect detection model as a target defect detection model when the recall rate and / or accuracy rate of the defect detection model meets the requirements.

[0144] For example, in some examples, when the recall rate of the defect detection model meets the standard, the defect detection model is used as the target defect detection model after the terminal device is changed. For another example, in some examples, when the accuracy rate of the defect detection model meets the standard, the defect detection model is used as the target defect detection model after the terminal device is changed. For another example, in some examples, when the accuracy rate and recall rate of the defect detection model meet the standard, the defect detection model is used as the target defect detection model after the terminal device is changed.

[0145] In this way, by using the existing defect detection model as the target defect detection model after the terminal device changes when the recall rate and / or accuracy rate meet the standards, it is possible to obtain the target defect detection model of the new product without training the neural network model, and it is also unnecessary to re-collect product images with the new terminal device, and it is also avoided to collect, classify and label the product images, which greatly reduces the impact of the terminal device change on the production line. In addition, the accuracy of the terminal device's product detection is guaranteed.

[0146] In certain embodiments, the optimization method further comprises:

[0147] 001, build image set;

[0148] 002, import the labeled and classified product images into the image set to obtain a basic image set.

[0149] Please further combine Fig.12In some embodiments, the optimization device 10 further includes a construction module 140, and steps 001 and 002 can be implemented by the construction module 140, or in other words, the construction module 140 can be used to construct an image set, and import the labeled and classified product images into the image set to obtain a basic image set.

[0150] In some implementations, the processor may be used to construct an image set, and import the labeled and classified product images into the image set to obtain a basic image set.

[0151] In this way, the neural network model can be trained according to the basic image set to obtain a defect detection model.

[0152] The embodiment of the present application also provides a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes the above-mentioned defect detection model optimization method.

[0153] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0154] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0155] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0157] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A defect detection model optimization method, characterized in that: The defect detection model is obtained by training a neural network model according to a basic image set, and the optimization method includes: Acquiring terminal device data and product design images, wherein the terminal device is used to detect the product; In the case where the product design image changes, the defect detection model is trained based on the new product shot image and the basic image set to obtain a target defect detection model; including: in the case where the new product design image matches the basic image set, the recall rate and / or accuracy rate of the defect detection model is tested; in the case where the recall rate and / or accuracy rate does not meet the standard, target new product shot images whose test results are lower than a preset threshold value are collected from the new product shot images; the target new product shot images that have been labeled and classified are used as the first training set; the defect detection model is trained based on the first training set and the basic image set to obtain the target defect detection model that meets the standard; or / and In the case where the terminal device data changes, the defect detection model is trained according to the product captured images collected by the terminal device and the basic image set to obtain a target defect detection model; including: obtaining a production plan and / or a replacement plan for the terminal device; in the case where the production plan and / or the replacement plan changes, the recall rate and / or accuracy rate of the defect detection model is tested according to the product captured images; in the case where the recall rate and / or accuracy rate of the defect detection model does not meet the standards, target product captured images whose test results are lower than a preset threshold in the product captured images are collected; the labeled and classified target product captured images are used as a third training set; and the defect detection model is trained according to the third training set and the basic image set to obtain a target defect detection model.

2. The optimization method according to claim 1, characterized in that: The step of training the defect detection model according to the new product shot image and the basic image set to obtain a target defect detection model further includes: When the recall rate and / or accuracy rate meet the requirements, the defect detection model is used as the target defect detection model.

3. The optimization method according to claim 1, characterized in that: The step of taking the labeled and classified target new product images as a first training set includes: Constructing a first sample set; Import the labeled and classified target new product images into the first sample set to obtain the first training set.

4. The optimization method according to claim 1, characterized in that: In the case where the product design image changes, the optimization method further includes: In the case that the new product design images do not match the basic image set, the labeled and classified new product images are used as a second training set; The neural network model is trained using the second training set to obtain the target defect detection model.

5. The optimization method according to claim 1, characterized in that: When the terminal device data changes, training the defect detection model according to the product shooting image collected by the terminal device and the basic image set to obtain a target defect detection model includes: When the recall rate and / or accuracy rate of the defect detection model meets the requirements, the defect detection model is used as the target defect detection model.

6. The optimization method according to claim 1, characterized in that: The step of taking the labeled and classified target product images as a third training set includes: constructing a second sample set; Import the labeled and classified photographed images of the target product into the second sample set to obtain the third training set.

7. The optimization method according to claim 1, characterized in that: The optimization method further comprises: Build image collections; The labeled and classified product images are imported into the image set to obtain the basic image set.

8. An optimization device for a defect detection model, characterized in that: The defect detection model is obtained by training a neural network model according to a basic image set, and the optimization device comprises: An acquisition module, used to acquire terminal device data and product design images, wherein the terminal device is used to detect the product; A first optimization module is used to train the defect detection model according to the new product shot image and the basic image set to obtain a target defect detection model when the product design image changes; the first optimization module is specifically used to test the recall rate and / or accuracy rate of the defect detection model when the new product design image matches the basic image set; when the recall rate and / or accuracy rate does not meet the standard, collect target new product shot images whose test results are lower than a preset threshold in the new product shot images; use the labeled and classified target new product shot images as the first training set; train the defect detection model according to the first training set and the basic image set to obtain the target defect detection model that meets the standard; or / and A second optimization module is used to train the defect detection model according to the product captured images and the basic image set collected by the terminal device to obtain a target defect detection model when the data of the terminal device changes; the second optimization module is specifically used to obtain a production plan and / or a replacement plan for the terminal device; when the production plan and / or the replacement plan changes, test the recall rate and / or accuracy of the defect detection model according to the product captured images; when the recall rate and / or accuracy of the defect detection model does not meet the standards, collect target product captured images whose test results are lower than a preset threshold in the product captured images; use the labeled and classified target product captured images as a third training set; train the defect detection model according to the third training set and the basic image set to obtain a target defect detection model.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the optimization method of the defect detection model according to any one of claims 1-7.

10. A non-volatile computer-readable storage medium containing a computer program, characterized in that: When the computer program is executed by a processor, the processor is caused to execute the method for optimizing the defect detection model according to any one of claims 1 to 7.

Images