Defect detection method and device and model migration method and device

Abstract

The invention discloses a defect detection method and a model migration method. The model migration method comprises the following steps: obtaining a source domain image and a corresponding label, and obtaining a baseline model from a source domain; obtaining a target domain image, wherein the target domain image is completely labeled or partially labeled or completely unlabeled; inputting the annotation of the source domain image into a de-noising device to obtain a de-noised annotation; inputting the initial baseline model into an unlabeled target domain image to obtain a second target domain prediction result, performing data expansion on the unlabeled target domain image to obtain a second target domain expansion image, and taking the second target domain prediction result as a pseudo label of the second target domain expansion image; and training a baseline model by using the source domain image and the corresponding de-noised annotation, the annotated target domain image and the corresponding annotation and / or the second target domain expansion image and the corresponding pseudo annotation to obtain a migration model. According to the method, training can be completed with a small number of labeled target domain images, and the problem that the model migration performance is poor under the condition that the target domain images are insufficient is solved.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a defect detection method and device, and a model migration method and device. Background Art

[0002] In the field of machine vision, deep learning-based detection technology is widely used. When using deep learning technology to detect defects in industrial images, it is necessary to collect a sufficient number of defect samples for various products produced and manually mark them accordingly, and use these defect samples to train the model to achieve the purpose of detection. However, data collection is usually difficult and the labeling cost is high (requiring a lot of manpower), resulting in high data costs for manufacturers. In real industrial production scenarios, for the same or similar products, due to differences in imaging equipment, shooting environment, lighting conditions, or slight changes in the product itself, the trained model cannot be migrated or quickly migrated. This is a pain point for manufacturing companies in using deep detection technology, which is mainly reflected in the following two aspects:

[0003] (1) The same product: In actual industrial production, the same product can be produced by different production lines. However, due to changes in factors such as lighting and imaging environment, there are slight differences in the data distribution of product images on different production lines (i.e., the "domain shift" problem). The data cannot be universalized, forming a strict correspondence mode of "production line-data-model". For example, the data collected and the trained model on production line A cannot be migrated to production line B or other production lines. In particular, when enterprises expand production, the data and advanced models obtained in the early stage of production cannot be migrated or quickly migrated to the new production line, resulting in waste of resources and increased production costs.

[0004] (2) Similar products: This is mainly reflected in the fact that even if the products produced are slightly different, it is impossible to migrate data and models or migrate them quickly. For example, the surface color of the products is slightly different, but the rest is the same, but there is also a "domain offset" problem between the image data of products of different colors, and the data and models cannot achieve good migration performance. In actual production, there are generally multiple similar products produced and launched to meet consumer demand, but currently in the production process of similar products, data cannot be aligned, resulting in the inability to fully utilize the data and poor model migration performance. Summary of the invention

[0005] The main technical problem solved by the present invention is that the machine vision model has poor migration capability and cannot adapt to different image data.

[0006] According to the first aspect, an embodiment provides a defect detection method, including:

[0007] Obtain the image to be detected in the target domain;

[0008] Inputting the image to be detected into a pre-trained defect detection model for defect detection to obtain a defect detection result of the image to be detected;

[0009] The defect detection model is trained in the following way:

[0010] Obtaining a source domain image and a corresponding defect annotation, and obtaining a baseline model trained by the source domain image and the corresponding defect annotation;

[0011] Acquire a target domain image, wherein the source domain and the target domain are different imaging scenes, and the different imaging scenes refer to different measured objects and / or different imaging conditions; part or all of the target domain image has corresponding defect annotations, or none of the target domain image has corresponding defect annotations, and if part or all of the target domain image has corresponding defect annotations, acquire defect annotations of the target domain image with the corresponding defect annotations;

[0012] Inputting the defect annotations of the source domain image into a learnable denoiser to obtain denoised annotations;

[0013] If there is a target domain image without a corresponding defect annotation, input it into the initial baseline model to obtain a second target domain prediction result, perform a data expansion operation on the target domain image without a corresponding defect annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the corresponding pseudo annotation of the second target domain expanded image;

[0014] The baseline model and the denoiser are trained using the source domain image and the corresponding denoising annotations, as well as the target domain image with corresponding defect annotations and the corresponding defect annotations and / or the second target domain expanded image and the corresponding pseudo annotations; the defect detection model is the baseline model obtained at the end of training; the initial baseline model is the baseline model whose parameters are obtained before training and have not been updated.

[0015] In some embodiments, the using the source domain image and the corresponding denoising annotation, and the target domain image with the corresponding defect annotation and the corresponding defect annotation and / or the second target domain augmented image and the corresponding pseudo annotation to train the baseline model and the denoiser comprises:

[0016] Inputting the source domain image into the baseline model to obtain a source domain prediction result;

[0017] If there is a target domain image with corresponding defect annotations, input it into the baseline model to obtain a first target domain prediction result;

[0018] If there is a target domain image without a corresponding defect annotation, the second target domain expanded image obtained by performing a data expansion operation on the target domain image without a corresponding defect annotation is input into the baseline model to obtain a second target domain expansion prediction result;

[0019] Training the baseline model and the denoiser according to a total loss function Loss;

[0020] Among them, the total loss function Loss is determined by a first loss function L1, a second loss function L2 and / or a third loss function L3. The first loss function L1 is determined according to the difference between the source domain prediction result and the corresponding denoising annotation, the second loss function L2 is determined at least according to the difference between the first target domain prediction result and the corresponding defect annotation, and the third loss function L3 is determined according to the difference between the second target domain expanded prediction result and the corresponding pseudo annotation.

[0021] In some embodiments, the expression of the denoiser is:

[0022] y tc =(1-β)×onehot(y)+β×p,

[0023] Among them, y tc is the denoising annotation, y is the defect annotation of the source domain image, onehot(y) represents performing a one-hot encoding operation on y, p is the source domain prediction result, and β is a learnable parameter.

[0024] The defect detection method in some embodiments further includes:

[0025] If there is a target domain image with a corresponding defect annotation, the same data expansion operation is also performed on the target domain image with the corresponding defect annotation and the corresponding defect annotation to obtain a first target domain expanded image and a corresponding first expanded annotation; the first target domain expanded image is input into the baseline model to obtain a first target domain expanded prediction result;

[0026] The second loss function L2 is also determined according to the difference between the first target domain expansion prediction result and the corresponding first expansion annotation, and the difference between the first target domain expansion prediction result and the corresponding first target domain prediction result.

[0027] In some embodiments, the first target domain prediction result includes a first predicted multi-class score map p1, wherein the elements in the first predicted multi-class score map p1 are used to represent the probability that the corresponding pixel points in the target domain image with the corresponding defect annotation belong to each classification category; at least one classification category in the classification categories is used to represent a defect;

[0028] The defect annotation of the target domain image with the corresponding defect annotation includes a target domain category annotation map y1, and the elements in the target domain category annotation map y1 are used to represent the actual classification category of the corresponding pixel points in the target domain image with the corresponding defect annotation;

[0029] The first target domain expansion prediction result includes a first expansion prediction multi-category score map p 1A , the first augmented prediction multi-class score map p 1A The elements in are used to represent the probability that the corresponding pixel points in the first target domain expanded image belong to each classification category;

[0030] The first expanded annotation includes the target domain category annotation image y1 obtained after the data expansion operation of the target domain category annotation image y 1A ;

[0031] The first predicted multi-category score map p1, the target domain category annotation map y1, and the first expanded predicted multi-category score map p 1A and the target domain expanded category annotation map y 1A The resolution of each of the images is the same as the resolution of the target domain image with the corresponding defect annotation;

[0032] The expression of the second loss function L2 is:

[0033] L2=α1L R1 +α2L R1A +α3L R11A ,

[0034] Among them, α1, α2 and α3 are preset weight coefficients,

[0035]

[0036] Wherein, argmax(p1) represents the first category prediction map obtained by taking the classification category with the largest probability for the elements in the first prediction multi-category score map p1, argmax(p1)∩y1 represents the number of intersection elements between the first category prediction map and the corresponding target domain category annotation map y1, and argmax(p1)∪y1 represents the number of union elements between the first category prediction map and the corresponding target domain category annotation map y1;

[0037]

[0038] Among them, argmax(p 1A ) represents the first expanded prediction multi-category score map p 1A The first expanded category prediction graph is obtained by taking the category with the largest probability, argmax(p 1A)∩y 1A represents the first expanded category prediction map and the corresponding target domain expanded category annotation map y 1A The number of intersection elements between 1A )∪y 1A represents the first expanded category prediction map and the corresponding target domain expanded category annotation map y 1A The number of elements in the union between ;

[0039]

[0040] Where m is the total number of elements, i is the sequence number of the element, (p1) represents the one-dimensional flattened vector of p1, (p1) i is the i-th element of (p1), (p 1A ) indicates p 1A The one-dimensional flattened vector, (p 1A ) i For p 1A The i-th element of , where the one-dimensional flattened vector refers to the vector formed by arranging the values ​​of each channel of each element in the graph in sequence.

[0041] In some embodiments, the source domain prediction result includes the probability that the source domain image belongs to each classification category; at least one of the classification categories is used to represent a defect;

[0042] The defect annotation of the source domain image includes the actual probability that the source domain image belongs to each classification category, and the denoised annotation includes the probability corresponding to each classification category obtained after the defect annotation of the source domain image is processed by the denoiser;

[0043] The expression of the first loss function L1 is:

[0044]

[0045] Among them, n is the total number of classification categories, j is the serial number of the classification category, and p j represents the probability that the source domain image included in the source domain prediction result belongs to the jth classification category, (y tc ) j represents the probability corresponding to the jth classification category in the denoised annotation.

[0046] In some embodiments, the second target domain prediction result includes a second predicted multi-class score map p2, and the elements in the second predicted multi-class score map p2 are used to represent the probability that the corresponding pixel points in the target domain image without the corresponding defect annotation belong to each classification category; at least one classification category in the classification categories is used to represent a defect;

[0047] The second target domain expansion prediction result includes a second expansion prediction multi-category score map p 2A , the second augmented prediction multi-class score map p 2A The elements in are used to represent the probability that the corresponding pixel points in the second target domain expanded image belong to each classification category;

[0048] The second predicted multi-class score map p2 and the second expanded predicted multi-class score map p 2A The resolutions of the images are the same as the resolution of the target domain image without corresponding defect annotations;

[0049] The expression of the third loss function L3 is:

[0050]

[0051] Where w is the total number of elements, k is the sequence number of the element, (p2) represents the one-dimensional flattened vector of p2, (p2) k is the kth element of (p2), (p 2A ) indicates p 2A The one-dimensional flattened vector, (p 2A ) k For p 2A The kth element of , where the one-dimensional flattened vector refers to the vector formed by arranging the values ​​of each channel of each element in the graph in sequence.

[0052] In some embodiments, if all the target domain images have corresponding defect annotations, the expression of the total loss function Loss is:

[0053] Loss=δ1L1+δ2L2,

[0054] If the target domain images do not have corresponding defect annotations, the expression of the total loss function Loss is:

[0055] Loss=δ1L1+δ3L3,

[0056] If the target domain image part has corresponding defect annotations, the expression of the total loss function Loss is:

[0057] Loss=δ1L1+δ2L2+δ3L3,

[0058] Wherein δ1, δ2 and δ3 are preset weight coefficients respectively.

[0059] In some embodiments, the imaging conditions include imaging equipment, shooting parameters, lighting and / or shooting environment.

[0060] According to the second aspect, an embodiment provides a model migration method, characterized in that it is used to migrate a baseline model obtained from a source domain to a target domain to obtain a migration model, and the model migration method includes:

[0061] Obtaining a source domain image and corresponding annotations, and obtaining the baseline model trained by the source domain image and corresponding annotations;

[0062] Acquire a target domain image, wherein part or all of the target domain image has corresponding annotations, or none of the target domain image has corresponding annotations. If part or all of the target domain image has corresponding annotations, acquire annotations of the target domain image having the corresponding annotations;

[0063] Inputting the annotations of the source domain image into a learnable denoiser to obtain denoised annotations;

[0064] If there is a target domain image without a corresponding annotation, input it into the initial baseline model to obtain a second target domain prediction result, perform a data expansion operation on the target domain image without a corresponding annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the corresponding pseudo annotation of the second target domain expanded image;

[0065] The baseline model and the denoiser are trained using the source domain image and the corresponding denoising annotations, as well as the target domain image with corresponding annotations and the corresponding annotations and / or the second target domain expanded image and the corresponding pseudo annotations; the migration model is the baseline model obtained at the end of the training; the initial baseline model is the baseline model whose parameters are not updated before the training.

[0066] In some embodiments, the training of the baseline model and the denoiser using the source domain image and the corresponding denoising annotation, and the target domain image with the corresponding annotation and the corresponding annotation and / or the second target domain augmented image and the corresponding pseudo annotation comprises:

[0067] Inputting the source domain image into the baseline model to obtain a source domain prediction result;

[0068] If there is a target domain image with corresponding annotations, input it into the baseline model to obtain a first target domain prediction result;

[0069] If there is a target domain image without corresponding annotations, the second target domain expanded image obtained by performing a data expansion operation on the target domain image without corresponding annotations is input into the baseline model to obtain a second target domain expansion prediction result;

[0070] Training the baseline model and the denoiser according to a total loss function Loss;

[0071] Among them, the total loss function Loss is determined by a first loss function L1, a second loss function L2 and / or a third loss function L3. The first loss function L1 is determined according to the difference between the source domain prediction result and the corresponding denoising annotation, the second loss function L2 is determined at least according to the difference between the first target domain prediction result and the corresponding annotation, and the third loss function L3 is determined according to the difference between the second target domain expanded prediction result and the corresponding pseudo annotation.

[0072] The model migration method in some embodiments further includes:

[0073] If there is a target domain image with a corresponding annotation, the same data expansion operation is also performed on the target domain image with the corresponding annotation and the corresponding annotation to obtain a first target domain expanded image and a corresponding first expanded annotation; the first target domain expanded image is input into the baseline model to obtain a first target domain expanded prediction result;

[0074] The second loss function L2 is also determined according to the difference between the first target domain expansion prediction result and the corresponding first expansion annotation, and the difference between the first target domain expansion prediction result and the corresponding first target domain prediction result.

[0075] According to a third aspect, an embodiment provides a defect detection device, comprising:

[0076] The image acquisition module to be detected is used to obtain the image to be detected in the target domain;

[0077] A detection module, connected to the image acquisition module to be detected, and used to input the image to be detected into a pre-trained defect detection model for defect detection to obtain a defect detection result of the image to be detected;

[0078] A model training module, connected to the detection module, is used to train and obtain the defect detection model, including:

[0079] A source domain data acquisition unit, used to acquire a source domain image and a corresponding defect annotation, and acquire a baseline model trained by the source domain image and the corresponding defect annotation;

[0080] a target domain data acquisition unit, configured to acquire a target domain image, wherein the source domain and the target domain are different imaging scenes, and the different imaging scenes refer to different measured objects and / or different imaging conditions; part or all of the target domain image has corresponding defect annotations, or none of the target domain image has corresponding defect annotations, and if part or all of the target domain image has corresponding defect annotations, then the defect annotations of the target domain image with the corresponding defect annotations are acquired;

[0081] A denoising unit, connected to the source domain data acquisition unit, and configured to input the defect annotation of the source domain image into a learnable denoiser to obtain a denoised annotation;

[0082] a pseudo-annotation generating unit connected to the target domain data acquiring unit, and configured to, if there is a target domain image without a corresponding defect annotation, input it into the initial baseline model to obtain a second target domain prediction result, perform a data expansion operation on the target domain image without the corresponding defect annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the pseudo-annotation corresponding to the second target domain expanded image;

[0083] a training unit connected to the source domain data acquisition unit, the target domain data acquisition unit, the denoising unit and the pseudo-annotation generation unit, and configured to train the baseline model and the denoiser by using the source domain image and the corresponding denoising annotation, the target domain image with the corresponding defect annotation and the corresponding defect annotation and / or the second target domain augmented image and the corresponding pseudo-annotation;

[0084] The defect detection model is a baseline model obtained at the end of training; the initial baseline model is a baseline model obtained before training whose parameters have not been updated.

[0085] According to a fourth aspect, an embodiment provides a model migration device for migrating a baseline model obtained from a source domain to a target domain to obtain a migration model, the model migration device comprising:

[0086] A source domain data acquisition unit, used to acquire a source domain image and corresponding annotations, and to acquire the baseline model trained by the source domain image and corresponding annotations;

[0087] a target domain data acquisition unit, configured to acquire a target domain image, wherein part or all of the target domain image has corresponding annotations, or none of the target domain image has corresponding annotations; if part or all of the target domain image has corresponding annotations, then acquiring annotations of the target domain image having the corresponding annotations;

[0088] a denoising unit, connected to the source domain data acquisition unit, and configured to input the annotation of the source domain image into a learnable denoiser to obtain a denoised annotation;

[0089] a pseudo-annotation generating unit connected to the target domain data acquiring unit, and configured to, if there is a target domain image without a corresponding annotation, input it into the initial baseline model to obtain a second target domain prediction result, perform a data expansion operation on the target domain image without a corresponding annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the pseudo-annotation corresponding to the second target domain expanded image;

[0090] a training unit connected to the source domain data acquisition unit, the target domain data acquisition unit, the denoising unit and the pseudo-annotation generation unit, and configured to train the baseline model and the denoiser by using the source domain image and the corresponding denoising annotation, the target domain image with the corresponding annotation and the corresponding annotation and / or the second target domain augmented image and the corresponding pseudo-annotation;

[0091] The migration model is a baseline model obtained at the end of training; the initial baseline model is a baseline model whose parameters are not updated and obtained before training.

[0092] According to the fifth aspect, an embodiment provides a computer-readable storage medium, on which a program is stored, and the program can be executed by a processor to implement the defect detection method of any of the above embodiments or the model migration method of any of the embodiments.

[0093] According to the fifth aspect, an embodiment provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implements the defect detection method of any of the above embodiments or the model migration method of any of the embodiments.

[0094] The technical solution of the present invention combines source domain images and target domain images to perform migration training on the baseline model from the source domain to the target domain, wherein the baseline model is trained in the source domain; the source domain image has corresponding annotations, and the annotations of the source domain image are input into a learnable denoiser to obtain denoised annotations; the target domain image may have all corresponding annotations, partially have corresponding annotations, or have no corresponding annotations at all, and for target domain images without annotations, pseudo annotations are generated using the baseline model; the source domain image and the corresponding denoising annotations, as well as the target domain image and the corresponding annotations / pseudo annotations are comprehensively utilized to train the baseline model and the denoiser, so as to achieve migration of the baseline model to the target domain, and the training can be completed with a small amount of annotated target domain images, so as to achieve rapid migration of the model, and solve the problem of poor model migration performance when the target domain images are insufficient. On the other hand, the present invention regards the domain adaptation process from the source domain to the target domain as a "denoising" problem to be solved, models the source domain image as a noise sample of the target domain image, and "denoises" the annotations of the source domain image by designing an annotation denoiser, so that the source domain image matches the target domain image. In addition, the technical solution of the present invention can adapt to situations where the target domain image is fully labeled, partially labeled, or completely unlabeled by generating corresponding pseudo-annotations for the unlabeled target domain image. It can adaptively use the target domain image and the corresponding annotations / pseudo-annotations for model transfer training based on the annotation situation of the input target domain image, thereby improving the flexibility of use and broadening the usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1A schematic diagram of the overall processing flow of a model migration method according to an embodiment;

[0096] Figure 2 A flowchart of a model migration method according to an embodiment;

[0097] Figure 3 A flowchart of training a baseline model and a denoiser using a source domain image and corresponding denoising annotations, a target domain image with corresponding annotations and corresponding annotations, and / or a second target domain augmented image and corresponding pseudo annotations in one embodiment;

[0098] Figure 4 A flowchart of a model migration method according to another embodiment;

[0099] Figure 5 A flowchart of training a baseline model and a denoiser using a source domain image and corresponding denoising annotations, a target domain image with corresponding defect annotations and corresponding defect annotations and / or a second target domain augmented image and corresponding pseudo annotations in one embodiment;

[0100] Figure 6 A flowchart of a defect detection method according to an embodiment;

[0101] Figure 7 A schematic diagram of the result of performing defect detection on an image to be detected by using a defect detection model in an embodiment;

[0102] Figure 8 A schematic diagram of the structure of a model migration device according to an embodiment;

[0103] Fig. 9 A schematic diagram of the structure of a defect detection device according to an embodiment. DETAILED DESCRIPTION

[0104] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0105] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0106] The serial numbers of the components in this document, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "coupling" mentioned in this application, unless otherwise specified, include direct and indirect connections (couplings).

[0107] The two problems mentioned in the background technology are the pain points that manufacturers currently encounter when using deep learning technology in industrial detection. In order to enable the trained model to be used on data with "domain offset", transfer learning technology is usually used to migrate the model to the target domain for use.

[0108] The purpose of transfer learning is to effectively transfer the knowledge or patterns learned from the source domain to the target domain to improve the performance of the task in the target domain. The source domain and the target domain are two key concepts in transfer learning, which refer to the dataset or domain where the model is pre-trained and the new dataset or domain to which the model will be applied. In the source domain, there are usually a large number of labeled samples available for learning, which are used to train and build the starting model, also known as the baseline model. The source domain can be a task, a domain, or a dataset, depending on the specific application scenario. For example, in the image classification task, the ImageNet dataset can be regarded as a source domain, which contains a large number of images and corresponding labels, which can be used to train neural networks. In the target domain, there are usually fewer labeled samples available for learning, so the model needs to use transfer learning to utilize the knowledge and features learned in the source domain in order to obtain good performance on the target task.

[0109] At present, the migration technology commonly used in most detection systems uses self-supervised methods for training. The specific processing method is to add the baseline model obtained from the source domain training to the target domain data for self-supervised learning, that is, to align the source domain data with the target domain data through high-dimensional feature space mapping to achieve migration from the source domain to the target domain. Its advantage is that the target domain data does not need to be labeled, but a sufficient number of target domain sample data is required. In the case of insufficient data in the target domain or difficulty in obtaining data, the false detection rate is high, the detection accuracy is low, and the migration effect is poor. At the same time, it also requires more training time and cannot achieve rapid migration.

[0110] In response to these problems, the present invention proposes an adaptive model migration method to achieve rapid migration from the source domain to the target domain, and solve the problem of poor model migration performance when the target domain data is insufficient. Compared with current similar methods, the adaptive model migration method of the present invention can complete training with a small amount of annotated target domain images, achieve rapid model migration, and solve the problem of poor model migration performance when the target domain images are insufficient. The obtained migration model has excellent performance. Applying the model migration method of the present invention to actual industrial production can greatly reduce the production cost of the enterprise; for the annotation of the target domain image (all annotated, partially annotated, and all unannotated), the present invention proposes an adaptive design to improve the flexibility of use. Users can use it according to actual needs to cope with different detection scenarios, while reducing costs, greatly improving the production efficiency of manufacturers.

[0111] The following is an introduction to the concept of the present invention. The present invention uses deep learning technology to construct the entire training process, please refer to Figure 1 The overall processing flow of the technical solution of the present invention includes three parts: a baseline branch G, a target domain labeled data branch R, and a target domain unlabeled data branch S. The input of branch G is a source domain image, the input of branch R is a target domain labeled image, and the input of branch S is a target domain unlabeled image. Model M is a model to be migrated for performing the target task. It is a baseline model with advanced detection performance that is trained in the source domain and then migrated to the target domain to complete the adaptive migration training of the model. M in different branches refers to the same model, which accepts corresponding inputs and outputs corresponding prediction results when performing processing flows of different branches.

[0112] For the situation where there is a certain difference in the data distribution of source domain samples and target domain samples, that is, the problem of "domain shift", unlike the commonly used methods, the present invention models the source domain image as a noise sample of the target domain image, and regards the domain adaptation process from the source domain to the target domain as a "denoising" problem to be solved.

[0113] The inputs of the three parts are different, and their functions and focuses are also different. The specific design idea is that for the baseline branch G, starting from the annotation perspective of the source domain image, a corresponding annotation denoiser is designed to remove the annotation "noise" of the source domain image so that the source domain image matches the target domain image. The target domain has labeled data branch R and the target domain has unlabeled data branch S, which are adaptive dynamic selection branches, supporting the supervised mode in which all images are labeled, the semi-supervised mode in which some images are labeled, and the unsupervised mode in which all images are unlabeled. Specifically, they can be divided into three cases: first, if all the input target domain images have corresponding annotations, then all the target domain images will undergo the processing flow of the target domain labeled data branch R, and the processing of the target domain unlabeled data branch S will not be performed; second, if some of the input target domain images have annotations, then the labeled target domain images will undergo the processing flow of the target domain labeled data branch R, and the unlabeled target domain images will undergo the processing flow of the target domain unlabeled data branch S; third, if all the input target domain images do not have corresponding annotations, then all the target domain images will undergo the processing flow of the target domain unlabeled data branch S, and the processing of the target domain labeled data branch R will not be performed.

[0114] In use, users can flexibly select training modes according to task requirements. For example, if the number of target domain images collected is sufficient, but the annotation workload is large or the task goal is to quickly migrate a small number of samples, the unsupervised mode of the present invention can be used for pre-training to obtain an initial model that can be used as an auxiliary annotation. When the model training is completed, after manual verification, most of the inference results output by the initial model can be directly used as annotations to reduce the workload of annotation; for target domain images with poor reasoning effects, corresponding annotations can be manually marked, and the corresponding sample data can be provided as important samples or difficult samples for model training to achieve rapid migration. At the same time, the obtained initial model can also be used as a pre-training model for subsequent training, which greatly reduces the training time. If there are fewer target domain images, some or all of the data can be directly manually marked with corresponding annotations and input into the model. The migration training of the model is completed through the supervised mode or semi-supervised mode. The model converges quickly and the migrated model has excellent performance on the target task. In order to ensure the performance of the model on the target task after migration, preferably, a small number of labeled target domain images are used for training, that is, a semi-supervised mode is used.

[0115] The present invention designs corresponding loss functions for training the three parts of the baseline branch G, the target domain labeled data branch R and the target domain unlabeled data branch S respectively. Among them, the "denoising" design of the annotation of the source domain image, the adaptive combination design of the R branch and the S branch and the design of the corresponding loss function are the key.

[0116] Combine the following Figure 1The model migration method of the present invention is specifically described. The model migration method is used to migrate the baseline model obtained from the source domain to the target domain to obtain a migration model. Please refer to Figure 2 , the model migration method in some embodiments includes steps 110 to 150, which are described below respectively.

[0117] Step 110: Obtain a source domain image x and a corresponding annotation y, and obtain a baseline model M trained with the source domain image x and the corresponding annotation y.

[0118] Step 120: Acquire a target domain image.

[0119] Among them, part or all of the target domain image has corresponding annotations, or none of them has corresponding annotations. If part or all of the target domain image has corresponding annotations, the annotations of the target domain image with corresponding annotations are obtained. Whether the target domain image is annotated and the amount of annotations are determined by the user according to the actual task needs. Figure 1 Here, x1 represents the labeled target domain image, y1 represents its label, and x2 represents the unlabeled target domain image.

[0120] Step 130: Input the source domain image annotation y into the learnable denoiser TC to obtain the denoised annotation y tc .

[0121] This step is to "denoise" the annotation y and model the source domain image as a noise sample of the target domain image, thereby converting the domain adaptation process from the source domain to the target domain into a "denoising" problem for solution.

[0122] like Figure 1 As shown, after the source domain image x is input into the baseline model M, the source domain prediction result p is obtained. In one embodiment, the denoiser TC is constructed according to the annotation y and the source domain prediction result p. The specific expression is:

[0123] y tc =(1-β)×onehot(y)+β×p,

[0124] Among them, onehot(y) means performing a one-hot encoding operation on y, and β is a learnable parameter.

[0125] The annotation y is processed by the denoiser TC, outputting the “clean” annotation y tc In essence, it forces the denoiser TC to learn adaptive smoothing (i.e., “denoising”) of the original annotations.

[0126] If there is a target domain image without a corresponding annotation, step 140 is executed, otherwise step 150 is directly executed.

[0127] Step 140: Input the target domain image x2 without corresponding annotations into the initial baseline model M eval The second target domain prediction result p2 is obtained, and the target domain image x2 without corresponding annotation is expanded to obtain the second target domain expanded image x 2A , the second target domain prediction result p2 is used as the second target domain expanded image x 2A The corresponding pseudo-annotations.

[0128] Figure 1 The AUG in refers to the data augmentation operation, which can be flipping, contrast enhancement, and adding noise. The initial baseline model M eval It is a baseline model whose parameters are not updated before training, or it can be understood as a baseline model that performs an eval operation during training (that is, the parameters remain fixed and not updated).

[0129] Step 150: Using the source domain image x and the corresponding denoised annotation y tc , and the target domain image x1 with corresponding annotations and the corresponding annotations y1 and / or the second target domain augmented image x 2A And the corresponding pseudo-annotation p2, the baseline model M and the denoiser TC are trained. The baseline model obtained at the end of the training is the migration model that completes the migration.

[0130] It can be understood that at this time, the source domain image x, the target domain image x1 with corresponding annotations, and the second target domain augmented image x 2A With the corresponding annotation information, they can be used for deep learning training. In the present invention, the denoiser TC is learnable, and the parameters are updated during training.

[0131] For the target domain image x1 and the second target domain augmented image x with corresponding annotations 2A If all target domain images are labeled, only the target domain image x1 is used for training; if all target domain images are not labeled, all target domain images need to be processed in step 140, and finally only the second target domain expanded image x1 is used. 2A Training; if the target domain image is partially annotated and partially unannotated, then the second target domain augmented image x1 is obtained by using the annotated target domain image x1 and the unannotated target domain image x2 2A Train together.

[0132] Please refer to Figure 1 and Figure 3 In some embodiments, step 150 includes the following steps:

[0133] Step 151: input the source domain image x into the baseline model M to obtain the source domain prediction result p;

[0134] If there is a target domain image x1 with a corresponding annotation, then execute step 152; if there is a target domain image x2 without a corresponding annotation, then execute step 153;

[0135] Step 152: input the target domain image x1 into the baseline model M to obtain the first target domain prediction result p1;

[0136] Step 153: Perform data expansion on the target domain image x2 to obtain a second target domain expanded image x 2A Input the baseline model M to obtain the second target domain expansion prediction result p 2A ;

[0137] Step 154: Train the baseline model M and the denoiser TC according to the total loss function Loss.

[0138] It should be noted that there is no fixed execution order for steps 151 to 153, and they can be executed sequentially or simultaneously.

[0139] The total loss function Loss is determined by the first loss function L1, the second loss function L2 and / or the third loss function L3. Please refer to Figure 1 The first loss function L1 is based on the source domain prediction result p and the corresponding denoising annotation y tc The second loss function L2 is determined at least according to the difference between the first target domain prediction result p1 and the corresponding annotation y1, and the third loss function L3 is determined according to the second target domain expansion prediction result p 2A and the corresponding pseudo annotation p2.

[0140] The first loss function L1, the second loss function L2 and the third loss function L3 correspond to the training of the G branch, the R branch and the S branch respectively, and they can be designed according to the training objectives of their respective branches. The training objectives or functions of each branch are described below.

[0141] In the domain adaptation model transfer training, the input of the G branch is all source domain images with corresponding annotations. In terms of design, it has two functions: first, make full use of the source domain images to transfer the recognition ability of the baseline model M to the target domain images; second, there is a "domain shift" problem of data distribution deviation between the source domain images and the target domain images. Therefore, the source domain images are modeled as noise samples of the target domain images, and the annotations of the source domain images are "denoised" by designing the annotation denoiser TC, so that the source domain images match the target domain images, which greatly reduces the false detection rate of the model to the target domain images and improves the detection performance.

[0142] The input of the R branch is the target domain image x1 with corresponding annotations, which is used to control the training fitting of the baseline model M on the target domain image and learn the ability to perform corresponding tasks on the target domain image.

[0143] The input of the S branch is the target domain image x2 without corresponding annotations. Using the data-enhanced target domain image as a training sample, the initial baseline model M eval The prediction of the target domain image x2 without corresponding annotation is used as a pseudo-annotation to guide the baseline model M to fit the enhanced image of the target domain image x2. In essence, it is a self-supervised training, which allows the baseline model M to complete efficient training and improve detection accuracy without labeled data.

[0144] In some embodiments, Figure 1 As shown in FIG. 1 , if there is a target domain image x1 with a corresponding annotation, the same data expansion operation is performed on the target domain image x1 with a corresponding annotation and the corresponding annotation y1 to obtain a first target domain expanded image x 1A and the corresponding first extended annotation y 1A ; Expand the first target domain image x 1A Input the baseline model M to obtain the first target domain expansion prediction result p 1A The second loss function L2 also expands the prediction result p according to the first target domain 1A and the corresponding first extended annotation y 1A The difference between the first target domain expansion prediction result p 1A and the difference between the corresponding first target domain prediction result p1 is determined.

[0145] In the adaptive training process of model migration, source domain images are sufficient and target domain images are scarce. Therefore, this embodiment adds a data expansion branch to the R branch: the same data expansion operation is performed on the target domain image x1 with corresponding annotations and the corresponding annotation y1 to obtain the first target domain expanded image x 1A and the corresponding first extended annotation y 1A ; Expand the first target domain image x 1A Input the baseline model M to obtain the first target domain expansion prediction result p 1A .

[0146] In this embodiment, the first target domain expanded image x is introduced by data expansion 1A and the corresponding first extended annotation y 1A The purpose of learning is to make the baseline model M train and fit the target domain expansion image; at the same time, in order to enhance the attention of the baseline model M to the target domain image, the first target domain expansion prediction result p 1Aand the corresponding first target domain prediction result p1 to construct the second loss function L2, the purpose of which is to train the baseline model M for consistency and guide the baseline model M to predict the target domain image x1 and the first target domain augmented image x 1A Output more consistent prediction results.

[0147] There are two purposes for expanding the target domain images: first, target domain images are difficult to collect and their number is small. By using data augmentation methods, the number of target domain images can be increased. Second, by training the consistency of the original target domain images and the expanded target domain images, the recognition ability of the baseline model M for the target domain images is enhanced. In essence, it is to enhance the pixel consistency between the prediction of the baseline model M for the target domain images and the prediction of the perturbation images (target domain expanded images) corresponding to the target domain images, thereby improving the detection performance and generalization performance of the model.

[0148] For the total loss function Loss, in some embodiments, according to the annotation of the target domain image, it is divided into three cases:

[0149] If all target domain images have corresponding annotations, the expression of the total loss function Loss is:

[0150] Loss=δ1L1+δ2L2,

[0151] If the target domain images do not have corresponding annotations, the expression of the total loss function Loss is:

[0152] Loss=δ1L1+δ3L3,

[0153] If part of the target domain image has corresponding annotations and part does not have corresponding annotations, the expression of the total loss function Loss is:

[0154] Loss=δ1L1+δ2L2+δ3L3,

[0155] δ1, δ2 and δ3 are respectively preset weight coefficients. In one example, δ1 = 1.0, δ2 = 1.0, and δ3 = 0.8.

[0156] The above model transfer method can be applied to defect detection tasks, by transferring the baseline model trained from defect samples in the source domain to the target domain to obtain a defect detection model that can perform defect detection tasks in the target domain. The difference is that in the defect detection task, the source domain image and the target domain image are specifically defect sample images in the source domain and the target domain, and the annotation is specifically the defect annotation.

[0157] Among them, the source domain and the target domain are different imaging scenes. Different imaging scenes refer to different objects to be measured (such as different products) and / or different imaging conditions. The imaging conditions include imaging equipment, shooting parameters (such as exposure time, aperture size, etc.), lighting and / or shooting environment (different backgrounds, different production lines, indoor and outdoor environments, etc.).

[0158] Defect annotation is used to mark defects in images, including the location, size and / or category of the defects, etc., and is usually represented by a defect annotation map with the same resolution as the image. Each pixel in the defect annotation map indicates whether the corresponding pixel in the marked image is a defect. If there are multiple different types of defects, it can also indicate what kind of defect the corresponding pixel in the marked image belongs to.

[0159] Please refer to Figure 4 , the model migration method applied to defect detection includes steps 210 to 250, which are described below respectively.

[0160] Step 210: Obtain a source domain image x and a corresponding defect annotation y, and obtain a baseline model M trained with the source domain image x and the corresponding defect annotation y.

[0161] The baseline model M trained with the source domain image x and the corresponding defect annotation y can perform the defect detection task in the source domain and has good defect detection performance on the source domain images.

[0162] Step 220: Acquire a target domain image.

[0163] Among them, part or all of the target domain image has corresponding defect annotations, or none of them has corresponding defect annotations. If part or all of the target domain image has corresponding defect annotations, the defect annotations of the target domain image with corresponding defect annotations are obtained. Whether to annotate the target domain image with defects and the amount of defect annotations are determined by the user according to actual task needs.

[0164] Step 230: Input the defect annotation y of the source domain image x into the learnable denoiser TC to obtain the denoised annotation y tc .

[0165] In one embodiment, the denoiser TC is constructed according to the defect annotation y and the source domain prediction result p, and the specific expression is:

[0166] y tc =(1-β)×onehot(y)+β×p,

[0167] Among them, onehot(y) means performing a one-hot encoding operation on y, and β is a learnable parameter.

[0168] The defect annotation y is processed by the denoiser TC and the output is the “clean” annotation ytc , in essence, it forces the denoiser TC to learn adaptive smoothing (i.e., “denoising”) of the original defect annotations.

[0169] If there is a target domain image without a corresponding defect annotation, step 240 is executed, otherwise step 250 is directly executed.

[0170] Step 240: Input the target domain image x2 without corresponding defect annotations into the initial baseline model M eval The second target domain prediction result p2 is obtained, and the target domain image x2 without the corresponding defect annotation is subjected to data expansion operation to obtain the second target domain expanded image x 2A , the second target domain prediction result p2 is used as the second target domain expanded image x 2A The corresponding pseudo-annotations.

[0171] Step 250: Using the source domain image x and the corresponding denoised annotation y tc , and the target domain image x1 with the corresponding defect annotation and the corresponding defect annotation y1 and / or the second target domain augmented image x 2A And the corresponding pseudo-annotation p2, the baseline model M and the denoiser TC are trained. The baseline model obtained at the end of the training is the defect detection model of the target domain.

[0172] Please refer to Figure 1 and Figure 5 In some embodiments, step 250 includes the following steps:

[0173] Step 251: input the source domain image x into the baseline model M to obtain the source domain prediction result p;

[0174] If there is a target domain image x1 with a corresponding defect annotation, then execute step 152; if there is a target domain image x2 without a corresponding defect annotation, then execute step 153;

[0175] Step 252: input the target domain image x1 into the baseline model M to obtain the first target domain prediction result p1;

[0176] Step 253: Perform data expansion on the target domain image x2 to obtain a second target domain expanded image x 2A Input the baseline model M to obtain the second target domain expansion prediction result p 2A ;

[0177] Step 254: Train the baseline model M and the denoiser TC according to the total loss function Loss.

[0178] It should be noted that there is no fixed execution order for steps 251 to 253, and they can be executed sequentially or simultaneously.

[0179] The total loss function Loss is determined by the first loss function L1, the second loss function L2 and / or the third loss function L3. Here, the first loss function L1 is based on the source domain prediction result p and the corresponding denoising annotation y tc The second loss function L2 is determined at least according to the difference between the first target domain prediction result p1 and the corresponding defect annotation y1, and the third loss function L3 is determined according to the second target domain expansion prediction result p 2A The difference between p2 and the corresponding pseudo-label p2 is determined. Each loss function can be designed according to the training objectives of the respective branches described above.

[0180] In some embodiments, the source domain prediction result p includes the probability that the source domain image x belongs to each classification category. There are multiple classification categories, and at least one of the multiple classification categories is used to represent defects. For example, there are two classification categories, one for no defect and one for defect; there may also be more classification categories, one for no defect and the others for various defect types.

[0181] Correspondingly, the defect annotation y of the source domain image x includes the actual probability that the source domain image x belongs to each classification category, so after denoising, it also represents the probability of each classification category, that is, the denoised annotation y tc The defect annotation y of the source domain image x is processed by the denoiser TC and the probability corresponding to each classification category is obtained. The expression of the first loss function L1 is:

[0182]

[0183] Among them, n is the total number of classification categories, j is the serial number of the classification category, and p j represents the probability that the source domain image x included in the source domain prediction result p belongs to the jth classification category, (y tc ) j Denotes the denoised annotation y tc The probability corresponding to the jth classification category.

[0184] In some embodiments, if there is a target domain image x1 with a corresponding defect annotation, the same data expansion operation is also performed on the target domain image x1 with the corresponding defect annotation and the corresponding defect annotation y1 to obtain a first target domain expanded image x 1A and the corresponding first extended annotation y 1A ; Expand the first target domain image x 1A Input the baseline model M to obtain the first target domain expansion prediction result p 1A The second loss function L2 also expands the prediction result p according to the first target domain 1A and the corresponding first extended annotation y 1AThe difference between the first target domain expansion prediction result p 1A and the difference between the corresponding first target domain prediction result p1 is determined.

[0185] Please refer to Figure 1 , at this time, the second loss function L2 consists of three parts: L determined according to the difference between the first target domain prediction result p1 and the corresponding defect annotation y1 R1 , according to the first target domain expansion prediction result p 1A and the corresponding first extended annotation y 1A The difference between the L R1A , and expand the prediction result p according to the first target domain 1A The consistency loss function L is determined by the difference between the prediction result p1 of the first target domain and the corresponding first target domain prediction result p2. R11A .

[0186] In some embodiments, the baseline model M is used to perform pixel-level defect detection on the target domain image x1, and the first target domain prediction result includes a first predicted multi-class score map p1, and the elements in the first predicted multi-class score map p1 are used to represent the probability that the corresponding pixel points in the target domain image x1 with the corresponding defect annotation belong to each classification category. Similarly, at least one of the classification categories is used to represent defects. For example, there are two classification categories, one for background and one for defects; there may also be more classification categories, one for background and the others for various defect types. The defect annotation of the target domain image x1 includes a target domain category annotation map y1, and the elements in the target domain category annotation map y1 are used to represent the actual classification category of the corresponding pixel points in the target domain image x1.

[0187] The first target domain expansion prediction result includes the first expansion prediction multi-category score map p 1A , the first expansion predicts the multi-category score map p 1A The elements in are used to represent the probability that the corresponding pixel points in the first target domain expanded image belong to each classification category. The first expanded annotation includes the target domain category annotation map y1 obtained after the data expansion operation. 1A .

[0188] First predicted multi-category score map p1, target domain category annotation map y1, first expanded predicted multi-category score map p 1A And the target domain expanded category annotation map y 1A The resolution of is the same as that of the target domain image x1.

[0189] L R1The intersection-over-union loss function is used. Since the resolution of industrial images is large and the defect area is relatively small, the intersection-over-union loss can better balance the loss contribution between different categories and avoid the loss of some categories being too large, which will cause the model to fail to learn effectively. R1 The expression is:

[0190]

[0191] Among them, argmax(p1) represents the first category prediction map obtained by taking the classification category with the highest probability for the elements in the first prediction multi-category score map p1, that is, for each element in the first prediction multi-category score map p1, the classification category with the highest probability is taken as the final category prediction result. argmax(p1)∩y1 represents the number of intersection elements between the first category prediction map and the corresponding target domain category annotation map y1, and argmax(p1)∪y1 represents the number of union elements between the first category prediction map and the corresponding target domain category annotation map y1.

[0192] L R1A The intersection-over-union loss function is also used, and the expression is:

[0193]

[0194] Among them, argmax(p 1A ) represents the first expanded prediction multi-category score map p 1A The first expanded category prediction graph is obtained by taking the category with the largest probability, argmax(p 1A )∩y 1A Represents the first expanded category prediction map and the corresponding target domain expanded category annotation map y 1A The number of intersection elements between 1A )∪y 1A Represents the first expanded category prediction map and the corresponding target domain expanded category annotation map y 1A The number of elements in the union between .

[0195] L R11A The expression is:

[0196]

[0197] Where m is the total number of elements, i is the sequence number of the element, (p1) represents the one-dimensional flattened vector of p1, (p1) i is the i-th element of (p1), (p 1A ) indicates p 1A The one-dimensional flattened vector, (p 1A ) i For p 1AThe ith element of . The one-dimensional flattened vector refers to the vector formed by arranging the values ​​of each element and channel in the graph in order. For example, a 3×3×3 graph has 9 elements and 3 channels, so its one-dimensional flattened vector is a vector of 27 elements formed by arranging the values ​​of each element and channel in order.

[0198] For the second loss function L2, in one embodiment, its expression is:

[0199] L2=α1L R1 +α2L R1A +α3L R11A ,

[0200] α1, α2 and α3 are respectively preset weight coefficients. In one example, α1 = 1.0, α2 = 1.0, and α3 = 0.6.

[0201] In some embodiments, the second target domain prediction result includes a second predicted multi-class score map p2, and the elements in the second predicted multi-class score map p2 are used to represent the probability that the corresponding pixel points in the target domain image x2 without the corresponding defect annotation belong to each classification category. Similarly, at least one classification category in the classification category is used to represent a defect. Accordingly, the second target domain expanded prediction result includes the second expanded prediction multi-class score map p 2A , the second expansion predicts the multi-category score map p 2A The elements in are used to represent the probability that the corresponding pixel points in the second target domain expanded image belong to each classification category.

[0202] The second predicted multi-category score map p2 and the second expanded predicted multi-category score map p 2A The resolution of is the same as that of the target domain image x2.

[0203] The expression of the third loss function L3 is:

[0204]

[0205] Where w is the total number of elements (the total number of elements here refers to p2 or p 2A The total number of elements), k is the number of elements, (p2) represents the one-dimensional flattened vector of p2, (p2) k is the kth element of (p2), (p 2A ) indicates p 2A The one-dimensional flattened vector, (p 2A ) k For p 2A The kth element of .

[0206] For the expression of the total loss function Loss, please refer to the above and will not be repeated here.

[0207] After the training process is completed, the defect detection model M is obtained. Based on the defect detection model M, the present invention provides a defect detection method. Please refer to Figure 6 In some embodiments, the defect detection method includes:

[0208] Step 310: Acquire an image to be detected in the target domain, that is, the image to be detected and the image in the target domain are images in the same imaging scene, for example, images of products on a production line in the target domain;

[0209] Step 320: Input the image to be detected into the defect detection model M for defect detection to obtain a defect detection result of the image to be detected.

[0210] The defect detection result may include a defect prediction map of the image to be detected. Figure 7 As shown (the left picture is the image to be detected, and the right picture is the defect prediction picture), the defect prediction picture is a binary picture (that is, there are two classification categories), black pixels represent the background, and white pixels represent defects.

[0211] Based on the above model migration method, the present invention also provides a model migration device for migrating the baseline model obtained from the source domain to the target domain to obtain a migration model. Figure 8 In one embodiment, the device includes a source domain data acquisition unit 11, a target domain data acquisition unit 12, a denoising unit 13, a pseudo-annotation generation unit 14 and a training unit 15, which are described below respectively.

[0212] The source domain data acquisition unit 11 is used to acquire source domain images and corresponding annotations, and to acquire a baseline model trained with the source domain images and corresponding annotations.

[0213] The source domain data acquisition unit 12 is used to acquire a target domain image, where part or all of the target domain image has corresponding annotations, or none of the target domain image has corresponding annotations. If part or all of the target domain image has corresponding annotations, the annotations of the target domain image with the corresponding annotations are acquired.

[0214] The denoising unit 13 is connected to the source domain data acquisition unit 11, and is used to input the annotation of the source domain image into a learnable denoiser to obtain denoised annotations. The expression of the denoiser can be referred to above.

[0215] The pseudo-annotation generation unit 14 is connected to the source domain data acquisition unit 12, and is used to input a target domain image without a corresponding annotation into the initial baseline model to obtain a second target domain prediction result, and perform a data expansion operation on the target domain image without a corresponding annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the corresponding pseudo-annotation of the second target domain expanded image.

[0216] The initial baseline model is a baseline model whose parameters are not updated before training.

[0217] The training unit 15 is connected to the source domain data acquisition unit 11, the source domain data acquisition unit 12, the denoising unit 13 and the pseudo-annotation generation unit 14, and is used to train the baseline model and the denoiser using the source domain image and the corresponding denoising annotation, as well as the target domain image with the corresponding annotation and the corresponding annotation and / or the second target domain augmented image and the corresponding pseudo-annotation. The baseline model obtained at the end of the training is the migration model that completes the migration.

[0218] In some embodiments, the training unit 15 trains the baseline model and the denoiser by the following steps: inputting the source domain image into the baseline model to obtain a source domain prediction result; if there is a target domain image with corresponding annotations, inputting it into the baseline model to obtain a first target domain prediction result; if there is a target domain image without corresponding annotations, inputting a second target domain expanded image obtained by performing a data expansion operation on the target domain image without corresponding annotations into the baseline model to obtain a second target domain expanded prediction result; and training the baseline model and the denoiser according to the total loss function Loss.

[0219] Among them, the total loss function Loss is determined by the first loss function L1, the second loss function L2 and / or the third loss function L3. The first loss function L1 is determined according to the difference between the source domain prediction result and the corresponding denoising annotation, the second loss function L2 is determined at least according to the difference between the first target domain prediction result and the corresponding annotation, and the third loss function L3 is determined according to the difference between the second target domain expanded prediction result and the corresponding pseudo annotation.

[0220] In some embodiments, if there is a target domain image with a corresponding annotation, the training unit 15 further performs the same data expansion operation on the target domain image with the corresponding annotation and the corresponding annotation to obtain a first target domain expanded image and a corresponding first expanded annotation; the first target domain expanded image is input into the baseline model to obtain a first target domain expanded prediction result. The second loss function L2 is also determined based on the difference between the first target domain expanded prediction result and the corresponding first expanded annotation, and the difference between the first target domain expanded prediction result and the corresponding first target domain prediction result.

[0221] For how to set each loss function, please refer to the introduction of the model migration method above, which will not be repeated here.

[0222] Based on the above-mentioned defect detection method and model migration device, the present invention also provides a defect detection device, please refer to Fig. 9 In one embodiment, the device includes an image acquisition module 20 to be detected, a detection module 30 and a model training module 40, which are described below respectively.

[0223] The to-be-detected image acquisition module 20 is used to acquire the to-be-detected image in the target domain.

[0224] The detection module 30 is connected to the image acquisition module 20 for inputting the image to be detected into a pre-trained defect detection model for defect detection to obtain a defect detection result of the image to be detected.

[0225] The model training module 40 is connected to the detection module 30 and is used to train the above-mentioned defect detection model. The model training module 40 includes a source domain data acquisition unit 41, a target domain data acquisition unit 42, a denoising unit 43, a pseudo-annotation generation unit 44 and a training unit 45.

[0226] The source domain data acquisition unit 41 is used to acquire source domain images and corresponding defect annotations, and to acquire a baseline model trained with the source domain images and corresponding defect annotations.

[0227] The source domain data acquisition unit 42 is used to acquire a target domain image, where part or all of the target domain image has corresponding defect annotations, or none of the target domain image has corresponding defect annotations. If part or all of the target domain image has corresponding defect annotations, the defect annotations of the target domain image with the corresponding defect annotations are acquired.

[0228] The denoising unit 43 is connected to the source domain data acquisition unit 41 and is used to input the defect annotations of the source domain image into a learnable denoiser to obtain denoised annotations. The expression of the denoiser can be referred to above.

[0229] The pseudo-annotation generation unit 44 is connected to the source domain data acquisition unit 42, and is used to input a target domain image without a corresponding defect annotation into the initial baseline model to obtain a second target domain prediction result, and perform a data expansion operation on the target domain image without a corresponding defect annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the corresponding pseudo-annotation of the second target domain expanded image.

[0230] The training unit 45 is connected to the source domain data acquisition unit 41, the source domain data acquisition unit 42, the denoising unit 43 and the pseudo-annotation generation unit 44, and is used to train the baseline model and the denoiser using the source domain image and the corresponding denoising annotation, as well as the target domain image with the corresponding defect annotation and the corresponding defect annotation and / or the second target domain augmented image and the corresponding pseudo-annotation. The baseline model obtained at the end of the training is the defect detection model of the target domain.

[0231] In some embodiments, the training unit 45 trains the baseline model and the denoiser by the following steps: inputting the source domain image into the baseline model to obtain a source domain prediction result; if there is a target domain image with a corresponding defect annotation, inputting it into the baseline model to obtain a first target domain prediction result; if there is a target domain image without a corresponding defect annotation, performing a data augmentation operation on the target domain image without the corresponding defect annotation to obtain a second target domain augmented image, inputting the second target domain augmented prediction result into the baseline model; and training the baseline model and the denoiser according to the total loss function Loss.

[0232] The total loss function Loss is determined by the first loss function L1, the second loss function L2 and / or the third loss function L3. Here, the first loss function L1 is determined according to the difference between the source domain prediction result and the corresponding denoising annotation, the second loss function L2 is determined at least according to the difference between the first target domain prediction result and the corresponding defect annotation, and the third loss function L3 is determined according to the difference between the second target domain expanded prediction result and the corresponding pseudo annotation.

[0233] In some embodiments, if there is a target domain image with a corresponding defect annotation, the training unit 45 further performs the same data expansion operation on the target domain image with the corresponding defect annotation and the corresponding defect annotation to obtain a first target domain expanded image and a corresponding first expanded annotation; the first target domain expanded image is input into the baseline model to obtain a first target domain expanded prediction result. The second loss function L2 is also determined according to the difference between the first target domain expanded prediction result and the corresponding first expanded annotation, and the difference between the first target domain expanded prediction result and the corresponding first target domain prediction result.

[0234] In some embodiments, the second loss function L2 is composed of three parts: L2 determined according to the difference between the first target domain prediction result p1 and the corresponding defect annotation y1 R1 , according to the first target domain expansion prediction result p 1A and the corresponding first extended annotation y 1A The difference between the L R1A , and expand the prediction result p according to the first target domain 1A The consistency loss function L is determined by the difference between the prediction result p1 of the first target domain and the corresponding first target domain prediction result p2. R11A .

[0235] For details on how to set each loss function, please refer to the introduction of the model migration method above, which will not be repeated here.

[0236] The technical solution of the present invention has the following three advantages compared with the prior art.

[0237] (1) The number of target domain samples required is small, which reduces the data cost of the enterprise and has a fast migration speed. The model migration method proposed in the present invention only requires a small amount of target domain data to participate in the training. The obtained migration model has excellent detection performance, which greatly reduces the production cost of the enterprise.

[0238] (2) The domain adaptation process from the source domain to the target domain is regarded as a "denoising" problem and solved. The source domain image is modeled as a noise sample of the target domain image. The annotation of the source domain image is corrected and designed to achieve the matching of the source domain image and the target domain image. The migration model obtained after training has a low false detection rate and high detection accuracy, and the detection performance is greatly improved.

[0239] (3) An adaptive design is made according to the annotation of the input target domain image, which improves the flexibility of use and broadens the application scenarios of the technology. The model migration method of the present invention has supervised mode, semi-supervised mode and unsupervised mode. The training mode can be automatically determined according to the annotation of the target domain image. The user can organize the input data according to the actual needs of the task without manually selecting the training mode. It is flexible and convenient to use and improves work efficiency. During use, the user can flexibly annotate the target domain image according to actual needs to cope with different task requirements and detection scenarios. The model migration method of the present invention can adaptively perform migration training.

[0240] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0241] The above specific examples are used to illustrate the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art, according to the concept of the present invention, some simple deductions, modifications or substitutions can be made.

Claims

1. A defect detection method, characterized in that: include: Obtain the image to be detected in the target domain; Inputting the image to be detected into a pre-trained defect detection model for defect detection to obtain a defect detection result of the image to be detected; The defect detection model is trained in the following way: Obtaining a source domain image and a corresponding defect annotation, and obtaining a baseline model trained by the source domain image and the corresponding defect annotation; Acquire a target domain image, wherein the source domain and the target domain are different imaging scenes, and the different imaging scenes refer to different measured objects and / or different imaging conditions; part or all of the target domain image has corresponding defect annotations, or none of the target domain image has corresponding defect annotations, and if part or all of the target domain image has corresponding defect annotations, acquire defect annotations of the target domain image with the corresponding defect annotations; Inputting the defect annotations of the source domain image into a learnable denoiser to obtain denoised annotations; If there is a target domain image without a corresponding defect annotation, input it into the initial baseline model to obtain a second target domain prediction result, perform a data expansion operation on the target domain image without a corresponding defect annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the corresponding pseudo annotation of the second target domain expanded image; The baseline model and the denoiser are trained using the source domain image and the corresponding denoising annotations, as well as the target domain image with corresponding defect annotations and the corresponding defect annotations and / or the second target domain expanded image and the corresponding pseudo annotations; the defect detection model is the baseline model obtained at the end of training; the initial baseline model is the baseline model whose parameters are obtained before training and have not been updated.

2. The defect detection method according to claim 1, characterized in that: The method of training the baseline model and the denoiser by using the source domain image and the corresponding denoising annotation, and the target domain image with the corresponding defect annotation and the corresponding defect annotation and / or the second target domain augmented image and the corresponding pseudo annotation comprises: Inputting the source domain image into the baseline model to obtain a source domain prediction result; If there is a target domain image with corresponding defect annotations, input it into the baseline model to obtain a first target domain prediction result; If there is a target domain image without a corresponding defect annotation, the second target domain expanded image obtained by performing a data expansion operation on the target domain image without a corresponding defect annotation is input into the baseline model to obtain a second target domain expansion prediction result; Training the baseline model and the denoiser according to a total loss function Loss; Among them, the total loss function Loss is determined by a first loss function L1, a second loss function L2 and / or a third loss function L3. The first loss function L1 is determined according to the difference between the source domain prediction result and the corresponding denoising annotation, the second loss function L2 is determined at least according to the difference between the first target domain prediction result and the corresponding defect annotation, and the third loss function L3 is determined according to the difference between the second target domain expanded prediction result and the corresponding pseudo annotation.

3. The defect detection method according to claim 2, characterized in that: The expression of the denoiser is: and tc (1-β)×onehot(y)+β×p, Among them, y tc is the denoising annotation, y is the defect annotation of the source domain image, onehot(y) represents performing a one-hot encoding operation on y, p is the source domain prediction result, and β is a learnable parameter.

4. The defect detection method according to claim 2, characterized in that: Also includes: If there is a target domain image with a corresponding defect annotation, the same data expansion operation is also performed on the target domain image with the corresponding defect annotation and the corresponding defect annotation to obtain a first target domain expanded image and a corresponding first expanded annotation; the first target domain expanded image is input into the baseline model to obtain a first target domain expanded prediction result; The second loss function L2 is also determined according to the difference between the first target domain expansion prediction result and the corresponding first expansion annotation, and the difference between the first target domain expansion prediction result and the corresponding first target domain prediction result.

5. The defect detection method according to claim 4, characterized in that: The first target domain prediction result includes a first predicted multi-class score map p1, wherein the elements in the first predicted multi-class score map p1 are used to represent the probability that the corresponding pixel points in the target domain image with the corresponding defect annotation belong to each classification category; at least one classification category in the classification categories is used to represent a defect; The defect annotation of the target domain image with the corresponding defect annotation includes a target domain category annotation map y1, and the elements in the target domain category annotation map y1 are used to represent the actual classification category of the corresponding pixel points in the target domain image with the corresponding defect annotation; The first target domain expansion prediction result includes a first expansion prediction multi-category score map p 1A , the first augmented prediction multi-class score map p 1A The elements in are used to represent the probability that the corresponding pixel points in the first target domain expanded image belong to each classification category; The first expanded annotation includes the target domain category annotation image y1 obtained after the data expansion operation of the target domain category annotation image y 1A ; The first predicted multi-category score map p1, the target domain category annotation map y1, and the first expanded predicted multi-category score map p 1A and the target domain expanded category annotation map y 1A The resolution of each of the images is the same as the resolution of the target domain image with the corresponding defect annotation; The expression of the second loss function L2 is: L2=α1L R1 +α2L R1A +α3L R11A , Among them, α1, α2 and α3 are preset weight coefficients, Wherein, argmax(p1) represents the first category prediction map obtained by taking the classification category with the largest probability for the elements in the first prediction multi-category score map p1, argmax(p1)∩y1 represents the number of intersection elements between the first category prediction map and the corresponding target domain category annotation map y1, and argmax(p1)∪y1 represents the number of union elements between the first category prediction map and the corresponding target domain category annotation map y1; Among them, argmax(p 1A ) represents the first expanded prediction multi-category score map p 1A The first expanded category prediction graph is obtained by taking the category with the largest probability, argmax(p 1A )∩y 1A represents the first expanded category prediction map and the corresponding target domain expanded category annotation map y 1A The number of intersection elements between 1A )∪y 1A represents the first expanded category prediction map and the corresponding target domain expanded category annotation map y 1A The number of elements in the union between ; Where m is the total number of elements, i is the sequence number of the element, (p1) represents the one-dimensional flattened vector of p1, (p1) i is the i-th element of (p1), (p 1A ) indicates p 1A The one-dimensional flattened vector, (p 1A ) i For p 1A The i-th element of , where the one-dimensional flattened vector refers to the vector formed by arranging the values ​​of each channel of each element in the graph in sequence.

6. The defect detection method according to claim 2, characterized in that: The source domain prediction result includes the probability that the source domain image belongs to each classification category; at least one classification category in the classification categories is used to represent a defect; The defect annotation of the source domain image includes the actual probability that the source domain image belongs to each classification category, and the denoised annotation includes the probability corresponding to each classification category obtained after the defect annotation of the source domain image is processed by the denoiser; The expression of the first loss function L1 is: Among them, n is the total number of classification categories, j is the serial number of the classification category, and p j represents the probability that the source domain image included in the source domain prediction result belongs to the jth classification category, (y tc ) j represents the probability corresponding to the jth classification category in the denoised annotation.

7. The defect detection method according to claim 2, characterized in that: The second target domain prediction result includes a second predicted multi-class score map p2, wherein the elements in the second predicted multi-class score map p2 are used to represent the probability that the corresponding pixel points in the target domain image without the corresponding defect annotation belong to each classification category; at least one classification category in the classification categories is used to represent a defect; The second target domain expansion prediction result includes a second expansion prediction multi-category score map p 2A , the second augmented prediction multi-class score map p 2A The elements in are used to represent the probability that the corresponding pixel points in the second target domain expanded image belong to each classification category; The second predicted multi-class score map p2 and the second expanded predicted multi-class score map p 2A The resolutions of the images are the same as the resolution of the target domain image without corresponding defect annotations; The expression of the third loss function L3 is: Where w is the total number of elements, k is the sequence number of the element, (p2) represents the one-dimensional flattened vector of p2, (p2) k is the kth element of (p2), (p 2A ) indicates p 2A The one-dimensional flattened vector, (p 2A ) k For p 2A The kth element of , where the one-dimensional flattened vector refers to the vector formed by arranging the values ​​of each channel of each element in the graph in sequence.

8. The defect detection method according to any one of claims 2 to 7, characterized in that: If all the target domain images have corresponding defect annotations, the expression of the total loss function Loss is: Loss=δ1L1+δ2L2, If the target domain images do not have corresponding defect annotations, the expression of the total loss function Loss is: Loss=δ1L1+δ3L3, If the target domain image part has corresponding defect annotations, the expression of the total loss function Loss is: Loss=δ1L1+δ2L2+δ3L3, Wherein δ1, δ2 and δ3 are preset weight coefficients respectively.

9. The defect detection method according to claim 1, characterized in that: The imaging conditions include imaging equipment, shooting parameters, lighting and / or shooting environment.

10. A model migration method, characterized in that: The method for migrating a baseline model obtained from a source domain to a target domain to obtain a migration model includes: Obtaining a source domain image and corresponding annotations, and obtaining the baseline model trained by the source domain image and corresponding annotations; Acquire a target domain image, wherein part or all of the target domain image has corresponding annotations, or none of the target domain image has corresponding annotations. If part or all of the target domain image has corresponding annotations, acquire annotations of the target domain image having the corresponding annotations; Inputting the annotations of the source domain image into a learnable denoiser to obtain denoised annotations; If there is a target domain image without a corresponding annotation, input it into the initial baseline model to obtain a second target domain prediction result, perform a data expansion operation on the target domain image without a corresponding annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the corresponding pseudo annotation of the second target domain expanded image; The baseline model and the denoiser are trained using the source domain image and the corresponding denoising annotations, as well as the target domain image with corresponding annotations and the corresponding annotations and / or the second target domain expanded image and the corresponding pseudo annotations; the migration model is the baseline model obtained at the end of the training; the initial baseline model is the baseline model whose parameters are not updated before the training.

11. The model migration method according to claim 10, characterized in that: The method of training the baseline model and the denoiser by using the source domain image and the corresponding denoising annotation, and the target domain image with the corresponding annotation and the corresponding annotation and / or the second target domain augmented image and the corresponding pseudo annotation comprises: Inputting the source domain image into the baseline model to obtain a source domain prediction result; If there is a target domain image with corresponding annotations, input it into the baseline model to obtain a first target domain prediction result; If there is a target domain image without corresponding annotations, the second target domain expanded image obtained by performing a data expansion operation on the target domain image without corresponding annotations is input into the baseline model to obtain a second target domain expansion prediction result; Training the baseline model and the denoiser according to a total loss function Loss; Among them, the total loss function Loss is determined by a first loss function L1, a second loss function L2 and / or a third loss function L3. The first loss function L1 is determined according to the difference between the source domain prediction result and the corresponding denoising annotation, the second loss function L2 is determined at least according to the difference between the first target domain prediction result and the corresponding annotation, and the third loss function L3 is determined according to the difference between the second target domain expanded prediction result and the corresponding pseudo annotation.

12. The model migration method according to claim 11, characterized in that: Also includes: If there is a target domain image with a corresponding annotation, the same data expansion operation is also performed on the target domain image with the corresponding annotation and the corresponding annotation to obtain a first target domain expanded image and a corresponding first expanded annotation; the first target domain expanded image is input into the baseline model to obtain a first target domain expanded prediction result; The second loss function L2 is also determined according to the difference between the first target domain expansion prediction result and the corresponding first expansion annotation, and the difference between the first target domain expansion prediction result and the corresponding first target domain prediction result.

13. A defect detection device, characterized in that: include: The image acquisition module to be detected is used to obtain the image to be detected in the target domain; A detection module, connected to the image acquisition module to be detected, and used to input the image to be detected into a pre-trained defect detection model for defect detection to obtain a defect detection result of the image to be detected; A model training module, connected to the detection module, is used to train and obtain the defect detection model, including: A source domain data acquisition unit, used to acquire a source domain image and a corresponding defect annotation, and acquire a baseline model trained by the source domain image and the corresponding defect annotation; a target domain data acquisition unit, configured to acquire a target domain image, wherein the source domain and the target domain are different imaging scenes, and the different imaging scenes refer to different measured objects and / or different imaging conditions; part or all of the target domain image has corresponding defect annotations, or none of the target domain image has corresponding defect annotations, and if part or all of the target domain image has corresponding defect annotations, then the defect annotations of the target domain image with the corresponding defect annotations are acquired; A denoising unit, connected to the source domain data acquisition unit, and configured to input the defect annotation of the source domain image into a learnable denoiser to obtain a denoised annotation; a pseudo-annotation generating unit connected to the target domain data acquiring unit, and configured to, if there is a target domain image without a corresponding defect annotation, input it into the initial baseline model to obtain a second target domain prediction result, perform a data expansion operation on the target domain image without the corresponding defect annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the pseudo-annotation corresponding to the second target domain expanded image; a training unit connected to the source domain data acquisition unit, the target domain data acquisition unit, the denoising unit and the pseudo-annotation generation unit, and configured to train the baseline model and the denoiser by using the source domain image and the corresponding denoising annotation, the target domain image with the corresponding defect annotation and the corresponding defect annotation and / or the second target domain augmented image and the corresponding pseudo-annotation; The defect detection model is a baseline model obtained at the end of training; the initial baseline model is a baseline model obtained before training whose parameters have not been updated.

14. A model migration device, characterized in that: The model migration device is used to migrate the baseline model obtained from the source domain to the target domain to obtain a migration model, and the model migration device includes: A source domain data acquisition unit, used to acquire a source domain image and corresponding annotations, and to acquire the baseline model trained by the source domain image and corresponding annotations; a target domain data acquisition unit, configured to acquire a target domain image, wherein part or all of the target domain image has corresponding annotations, or none of the target domain image has corresponding annotations; if part or all of the target domain image has corresponding annotations, then acquiring annotations of the target domain image having the corresponding annotations; a denoising unit, connected to the source domain data acquisition unit, and configured to input the annotation of the source domain image into a learnable denoiser to obtain a denoised annotation; a pseudo-annotation generating unit connected to the target domain data acquiring unit, and configured to, if there is a target domain image without a corresponding annotation, input it into the initial baseline model to obtain a second target domain prediction result, perform a data expansion operation on the target domain image without a corresponding annotation to obtain a second target domain expanded image, and use the second target domain prediction result as the pseudo-annotation corresponding to the second target domain expanded image; a training unit connected to the source domain data acquisition unit, the target domain data acquisition unit, the denoising unit and the pseudo-annotation generation unit, and configured to train the baseline model and the denoiser by using the source domain image and the corresponding denoising annotation, the target domain image with the corresponding annotation and the corresponding annotation and / or the second target domain augmented image and the corresponding pseudo-annotation; The migration model is a baseline model obtained at the end of training; the initial baseline model is a baseline model whose parameters are not updated and obtained before training.

15. A computer-readable storage medium, characterized in that: The medium stores a program, which can be executed by a processor to implement the defect detection method according to any one of claims 1 to 9 or the model migration method according to any one of claims 10 to 12.

16. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the defect detection method according to any one of claims 1 to 9 or the model migration method according to any one of claims 10 to 12 is implemented.

Images