Method and apparatus for transferring defect detection model, device, and storage medium

Abstract

Disclosed are a method and an apparatus for transferring a defect detection model, a device, and a storage medium. The method comprises: obtaining a pre-trained defect feature extraction model and a defect feature set, wherein the defect feature extraction model is configured to extract, from different product images having similar known defects, different known defect features having a similarity greater than a first preset similarity, and the defect feature set comprises the known defect features extracted by the defect feature extraction model; inputting a new defect product image containing a new defect into the defect feature extraction model, so as to obtain a new defect feature; within the defect feature set, for known defect features whose similarity to the new defect feature satisfies a preset similarity condition, determining corresponding known defects as candidate defects; and for defect detection models corresponding to the determined candidate defects, using the new defect product image to perform transfer training, so as to obtain a new defect detection model used for detecting whether an input product image contains the new defect.

Description

A defect detection model migration method, device, equipment and storage medium Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a defect detection model migration method, device, equipment and storage medium. Background Art

[0002] Image processing technology is currently quite mature, and there are many detection models for detecting whether an image conforms to a certain type. For example, you can capture a product image and use image recognition technology to detect product defects. Specifically, you can set up different defect detection models for different defects to detect whether the product in the image has a certain defect, facilitating subsequent recycling and repair. You can also capture a product image and use image recognition technology to detect the product type. Specifically, you can set up different product type detection models for different product types to detect whether the product in the image belongs to a certain product type, facilitating subsequent sorting.

[0003] As demand increases, the types of tests required are also increasing.

[0004] For example, during the production or transportation of a product, it is often difficult to exhaust all types of product defects, and new product defects may be discovered or generated at any time, which requires training a model to detect new product defects.

[0005] Existing methods often involve collecting new image samples for training. However, sample collection is difficult and time-consuming, and the efficiency of generating models for detecting new types is also low. For example, the efficiency of generating models for detecting new defects is low.

[0006] Summary of the Invention

[0007] The present invention provides a defect detection model migration method, device, equipment and storage medium to address the deficiencies in related technologies.

[0008] According to a first aspect of an embodiment of the present invention, a defect detection model migration method is provided, wherein at least two known defects and different defect detection models corresponding to different known defects are pre-set; the defect detection models are used to detect whether an input product image has the corresponding known defects;

[0009] The method comprises:

[0010] Obtain a pre-trained defect feature extraction model and a defect feature set;

[0011] The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity for different product images with similar known defects; the defect feature set includes the known defect features extracted by the defect feature extraction model for known defect product images with known defects;

[0012] Inputting a new defective product image having a new defect into the defect feature extraction model to obtain a new defect feature;

[0013] In the defect feature set, for known defect features whose similarity to the new defect feature meets a preset similarity condition, determining the corresponding known defects as candidate defects;

[0014] For the defect detection model corresponding to the determined candidate defect, transfer training is performed using the acquired new defect product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

[0015] Optionally, the training method of the defect feature extraction model includes:

[0016] The following training steps are executed repeatedly until the preset training stop condition is met:

[0017] Acquire multiple groups of different product images having the same known defects, and determine each group of product images as a product image having similar known defects;

[0018] The current defect feature extraction model is trained using the acquired multiple groups of product images to improve the similarity of known defect features extracted between product images within a group and / or to improve the difference of known defect features extracted between product images of different groups.

[0019] Optionally, the training step further includes:

[0020] Obtain images of products with known defects;

[0021] Extract known defect features from the acquired product image based on the current defect feature extraction model;

[0022] Clustering the extracted known defect features to obtain at least two similar defect feature groups, and labeling each acquired product image with the similar defect feature group to which the corresponding known defect feature belongs;

[0023] The current defect feature extraction model is trained based on the annotated product images to improve the similarity of known defect features extracted between product images with the same annotation and / or to improve the difference of known defect features extracted between product images with different annotations.

[0024] Optionally, the training of the current defect feature extraction model based on the annotated product image includes:

[0025] The following steps are repeated until the first training stop condition is met:

[0026] Based on the current defect feature extraction model, known defect features are extracted from the annotated product image to obtain a first feature set;

[0027] Based on a preset strategy, a known defect feature is determined based on the known defect features corresponding to each annotation to obtain a second feature set;

[0028] The loss is calculated based on the obtained first feature set and the second feature set, and the current defect feature extraction model is updated.

[0029] Optionally, the training of the current defect feature extraction model based on the annotated product image includes:

[0030] Randomly select one known defect feature from the known defect features corresponding to each annotation and add it to the initial feature set; and add each feature in the initial feature set to the edge strategy feature set and the average strategy feature set respectively;

[0031] The following steps are executed repeatedly until the second training stop condition is met:

[0032] Based on the current defect feature extraction model, known defect features are extracted from the annotated product images to obtain a full feature set.

[0033] Update the edge policy feature set based on the following steps:

[0034] For a target defect feature corresponding to a target annotation in the edge strategy feature set, among known defect features corresponding to the target annotation in the full feature set, determine the known defect feature with the lowest similarity to the target defect feature as an edge feature; and update the target defect feature based on the edge feature;

[0035] The average policy feature set is updated based on the following steps:

[0036] For the target defect feature corresponding to the target annotation in the average strategy feature set, the mean is calculated for each known defect feature corresponding to the target annotation in the full feature set, and the target defect feature is updated according to the calculated mean;

[0037] Based on the updated edge strategy feature set and the updated average strategy feature set, the current initial feature set is updated;

[0038] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0039] Optionally, the defect detection model corresponding to the determined candidate defect is transferred and trained using the acquired new defective product image, including:

[0040] For each of the multiple defect detection models corresponding to the determined candidate defects, determining the detection accuracy for the new defect using a test set; the test set includes images of new defective products having the new defect;

[0041] Select a defect detection model whose detection accuracy meets the preset detection conditions, and use the acquired new defective product images for migration training.

[0042] Optionally, the defect detection model corresponding to the determined candidate defect is transferred and trained using the acquired new defective product image, including:

[0043] For multiple defect detection models corresponding to the determined candidate defects, transfer training is performed using the acquired new defect product image to obtain multiple candidate new defect detection models;

[0044] For each candidate new defect detection model, determining a detection accuracy rate for the new defect using a test set; the test set includes images of new defective products with the new defect;

[0045] The alternative new defect detection model whose detection accuracy meets the preset accuracy condition is determined as the new defect detection model.

[0046] According to a second aspect of an embodiment of the present invention, a method for migrating a defect detection model is provided, comprising:

[0047] For a new defective product image with a new defect, known defects similar to the new defect are determined as candidate defects by feature extraction;

[0048] obtaining a candidate defect detection model for detecting whether an input product image has the candidate defect;

[0049] The obtained candidate defect detection model is transferred and trained using the new defective product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

[0050] According to a third aspect of an embodiment of the present invention, a method for migrating a type detection model is provided, wherein at least two known types and different type detection models corresponding to the different known types are pre-set; the type detection models are used to detect whether an input image conforms to the corresponding known types;

[0051] The method comprises:

[0052] Obtain a pre-trained type feature extraction model and a type feature set;

[0053] The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity for different images of similar known types; the type feature set includes the known type features extracted by the type feature extraction model for images that meet the known type;

[0054] Inputting an image that meets the new type into the type feature extraction model to obtain new type features;

[0055] In the type feature set, for known type features whose similarity to the new type feature meets a preset similarity condition, determining the corresponding known type as an alternative type;

[0056] For the type detection model corresponding to the determined candidate type, migration training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0057] According to a fourth aspect of an embodiment of the present invention, a method for migrating a type detection model is provided, comprising:

[0058] For images that meet the new type, known types similar to the new type are determined as candidate types through feature extraction;

[0059] Obtaining an alternative type detection model for detecting whether an input image conforms to the alternative type;

[0060] For the obtained candidate type detection model, migration training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0061] According to a fifth aspect of an embodiment of the present invention, a device for migrating a defect detection model is provided, which is pre-configured with at least two known defects and different defect detection models corresponding to different known defects; the defect detection models are used to detect whether an input product image has the corresponding known defects; the device includes:

[0062] An acquisition unit, used to acquire a pre-trained defect feature extraction model and a defect feature set;

[0063] The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity for different product images with similar known defects; the defect feature set includes the known defect features extracted by the defect feature extraction model for known defect product images with known defects;

[0064] a similarity unit, configured to input a new defective product image having a new defect into the defect feature extraction model to obtain a new defect feature; and determine, in the defect feature set, for known defect features whose similarity to the new defect feature satisfies a preset similarity condition, the corresponding known defect as a candidate defect;

[0065] The migration unit is used to perform migration training on the defect detection model corresponding to the determined candidate defect using the acquired new defect product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

[0066] Optionally, the training method of the defect feature extraction model includes:

[0067] The following training steps are executed repeatedly until the preset training stop condition is met:

[0068] Acquire multiple groups of different product images having the same known defects, and determine each group of product images as a product image having similar known defects;

[0069] The current defect feature extraction model is trained using the acquired multiple groups of product images to improve the similarity of known defect features extracted between product images within a group and / or to improve the difference of known defect features extracted between product images of different groups.

[0070] Optionally, the training step further includes:

[0071] Obtain images of products with known defects;

[0072] Extract known defect features from the acquired product image based on the current defect feature extraction model;

[0073] Clustering the extracted known defect features to obtain at least two similar defect feature groups, and labeling each acquired product image with the similar defect feature group to which the corresponding known defect feature belongs;

[0074] The current defect feature extraction model is trained based on the annotated product images to improve the similarity of known defect features extracted between product images with the same annotation and / or to improve the difference of known defect features extracted between product images with different annotations.

[0075] Optionally, the training of the current defect feature extraction model based on the annotated product image includes:

[0076] The following steps are repeated until the first training stop condition is met:

[0077] Based on the current defect feature extraction model, known defect features are extracted from the annotated product image to obtain a first feature set;

[0078] Based on a preset strategy, a known defect feature is determined based on the known defect features corresponding to each annotation to obtain a second feature set;

[0079] The loss is calculated based on the obtained first feature set and the second feature set, and the current defect feature extraction model is updated.

[0080] Optionally, the training of the current defect feature extraction model based on the annotated product image includes:

[0081] Randomly select one known defect feature from the known defect features corresponding to each annotation and add it to the initial feature set; and add each feature in the initial feature set to the edge strategy feature set and the average strategy feature set respectively;

[0082] The following steps are executed repeatedly until the second training stop condition is met:

[0083] Based on the current defect feature extraction model, known defect features are extracted from the annotated product images to obtain a full feature set.

[0084] Update the edge policy feature set based on the following steps:

[0085] For a target defect feature corresponding to a target annotation in the edge strategy feature set, among known defect features corresponding to the target annotation in the full feature set, determine the known defect feature with the lowest similarity to the target defect feature as an edge feature; and update the target defect feature based on the edge feature;

[0086] The average policy feature set is updated based on the following steps:

[0087] For the target defect feature corresponding to the target annotation in the average strategy feature set, the mean is calculated for each known defect feature corresponding to the target annotation in the full feature set, and the target defect feature is updated according to the calculated mean;

[0088] Based on the updated edge strategy feature set and the updated average strategy feature set, the current initial feature set is updated;

[0089] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0090] Optionally, the migration unit is used to:

[0091] For each of the multiple defect detection models corresponding to the determined candidate defects, determining the detection accuracy for the new defect using a test set; the test set includes images of new defective products having the new defect;

[0092] Select a defect detection model whose detection accuracy meets the preset detection conditions, and use the acquired new defective product images for migration training.

[0093] Optionally, the migration unit is used to:

[0094] For multiple defect detection models corresponding to the determined candidate defects, transfer training is performed using the acquired new defect product image to obtain multiple candidate new defect detection models;

[0095] For each candidate new defect detection model, determining a detection accuracy rate for the new defect using a test set; the test set includes images of new defective products with the new defect;

[0096] The alternative new defect detection model whose detection accuracy meets the preset accuracy condition is determined as the new defect detection model.

[0097] According to a sixth aspect of an embodiment of the present invention, there is provided a device for migrating a defect detection model, comprising:

[0098] a feature extraction unit configured to extract features from a new defective product image and determine known defects similar to the new defect as candidate defects;

[0099] a model acquisition unit, configured to acquire a candidate defect detection model for detecting whether an input product image has the candidate defect;

[0100] The transfer training unit is used to perform transfer training on the obtained alternative defect detection model using the new defective product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

[0101] According to a seventh aspect of an embodiment of the present invention, there is provided a device for migrating a type detection model, the device being pre-configured with at least two known types and different type detection models corresponding to the different known types; the type detection models being used to detect whether an input image conforms to the corresponding known types;

[0102] The device comprises:

[0103] A type acquisition unit, used to obtain a pre-trained type feature extraction model and a type feature set;

[0104] The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity for different images of similar known types; the type feature set includes the known type features extracted by the type feature extraction model for images that meet the known type;

[0105] A new type unit is configured to input an image that meets the new type into the type feature extraction model to obtain a new type feature; and in the type feature set, for known type features whose similarity to the new type feature satisfies a preset similarity condition, determine the corresponding known type as an alternative type;

[0106] The training unit is used to perform migration training on the type detection model corresponding to the determined alternative type using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0107] According to an eighth aspect of an embodiment of the present invention, there is provided a device for migrating a type detection model, comprising:

[0108] an alternative type unit, configured to, for an image that meets a new type, determine known types similar to the new type as alternative types by extracting features;

[0109] an alternative model unit, configured to obtain an alternative type detection model for detecting whether an input image conforms to the alternative type;

[0110] The result unit is used to perform migration training on the acquired candidate type detection model using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0111] According to the above embodiment, by determining a known type similar to the new type and performing transfer training based on the type detection model for detecting the determined known type to obtain a new type detection model for detecting the new type, the efficiency of generating the new type detection model for detecting the new type can be improved. In particular, when the image type is specifically a defect, the efficiency of generating the new defect detection model for detecting the new defect can be improved.

[0112] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0114] FIG1 is a schematic flow chart of a defect detection model migration method according to an embodiment of the present invention;

[0115] FIG2 is a schematic diagram showing the principle of defect detection model migration according to an embodiment of the present invention;

[0116] FIG3 is a schematic flow chart of another defect detection model migration method according to an embodiment of the present invention;

[0117] FIG4 is a schematic diagram of a flow chart of a method for migrating a type detection model according to an embodiment of the present invention;

[0118] FIG5 is a schematic flow chart of a method for migrating another type of detection model according to an embodiment of the present invention;

[0119] FIG6 is a schematic structural diagram of a defect detection model migration device according to an embodiment of the present invention;

[0120] 7 is a schematic structural diagram of another defect detection model migration device according to an embodiment of the present invention;

[0121] FIG8 is a schematic structural diagram of a migration device for a type detection model according to an embodiment of the present invention;

[0122] FIG9 is a schematic structural diagram of a migration device for another type of detection model according to an embodiment of the present invention;

[0123] FIG10 is a schematic diagram of the hardware structure of a computer device configured with the method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0124] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0126] Currently, image processing technology is relatively mature, and there are many detection models for detecting whether an image conforms to a certain type.

[0127] For example, product defects can be detected by taking product images and using image recognition technology. Different defect detection models can be set up for different defects to detect whether the product in the image has a certain defect, which facilitates subsequent recycling and repair.

[0128] It is also possible to detect product types by taking product images and using image recognition technology. Specifically, different product type detection models can be set for different product types to detect whether the product in the image belongs to a certain product type, which facilitates subsequent sorting.

[0129] As demand increases, the types of tests required are also increasing.

[0130] For example, during the production or transportation of a product, it is often difficult to exhaust all types of product defects, and new product defects may be discovered or generated at any time, which requires training a model to detect new product defects.

[0131] The existing method often involves re-collecting new types of image samples for training, but sample collection is difficult and time-consuming, and the generation efficiency of the model for detecting new types is also low.

[0132] Taking defect detection as an example, the existing method often involves re-collecting samples of new product defects for training. However, sample collection is difficult and time-consuming, and the generation efficiency of the model for detecting new product defects is also low.

[0133] The following first explains it using the defect detection scenario as an example.

[0134] The product defect in the image can be an example of an image type. For example, if the product in the image has a damaged defect, the "damaged" product defect can be an image type, that is, the product in the image has a damaged defect.

[0135] The present invention discloses a defect detection model migration method. Through model transfer learning, an existing defect detection model is used to select existing defects similar to a new defect. A defect detection model used to detect the selected existing defect is then selected and migrated. This allows transfer learning to be performed using a relatively small number of new defect samples, quickly obtaining a defect detection model for detecting the new defect and improving the efficiency of model generation for determining detection of new products.

[0136] Among them, it is often difficult to determine the similarity of product defects.

[0137] Therefore, in this method, a model for measuring the similarity of product defects can be pre-trained. Specifically, the model can be used to map the product defect image into a feature space. The features mapped corresponding to similar defects are close in the feature space, which facilitates the subsequent determination of existing defects that are similar to the new defect.

[0138] The following is a detailed explanation of a defect detection model migration method provided by an embodiment of the present invention.

[0139] As shown in FIG1 , FIG1 is a flow chart of a defect detection model migration method according to an embodiment of the present invention.

[0140] Optionally, the defect here may be a defect of the product in the image, for example, a screen defect may be broken when the screen is assembled, or a shell defect may be broken when the shell is installed, etc.

[0141] The embodiment of the present invention does not limit the execution subject of the method process. Optionally, the execution subject can be any computing device, for example, a server or client for detecting product defects.

[0142] At least two known defects and different defect detection models corresponding to different known defects may be pre-set; the defect detection model may be used to detect whether an input product image has corresponding known defects.

[0143] Optionally, different known defects may correspond one-to-one to different defect detection models, each known defect may correspond to a defect detection model, and each defect detection model may correspond to a known defect.

[0144] The method may include the following steps.

[0145] S101: Obtain a pre-trained defect feature extraction model and a defect feature set.

[0146] The defect feature extraction model may be used to extract different known defect features having a similarity greater than a first preset similarity for different product images having similar known defects.

[0147] The defect feature set may include known defect features extracted by the defect feature extraction model for a known defect product image with a known defect.

[0148] S102: Inputting the new defect product image with the new defect into the defect feature extraction model to obtain new defect features.

[0149] S103: In the defect feature set, for known defect features whose similarity to the new defect feature meets a preset similarity condition, the corresponding known defects are determined as candidate defects.

[0150] S104: performing transfer training on the defect detection model corresponding to the determined candidate defect using the acquired new defect product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

[0151] The above method process limits the image type to defects in product images. By determining known types similar to the new type and performing transfer training based on the type detection model of the known type determined by detection, a new type detection model for detecting the new type is obtained, which can improve the generation efficiency of the new type detection model for detecting the new type.

[0152] Among them, when the image type is specifically a defect, the generation efficiency of a new defect detection model for detecting new defects can be improved.

[0153] Specifically, the efficiency of generating a new defect detection model for detecting new defects can be improved by determining known defects similar to the new defect based on the similarity between defect features and performing transfer training based on the defect detection model of the known defects determined by detection.

[0154] The following is a detailed explanation of each aspect.

[0155] 1. About the structure and function of defect feature extraction model.

[0156] 1. Model structure.

[0157] The process of this method does not limit the structure of the defect feature extraction model.

[0158] Optionally, the defect feature extraction model can be a model for extracting image features such as a graph convolutional network, a graph neural network, etc.

[0159] Optionally, the defect feature extraction model can be a residual neural network ResNet.

[0160] 2. Model function.

[0161] The defect feature extraction model can be used to extract different known defect features with a similarity greater than a first preset similarity for different product images with similar known defects.

[0162] Optionally, the defect feature extraction model can be used to map the input product image into a feature space, that is, to extract the defect features in the product image.

[0163] Among them, different product images with similar known defects can obtain different defect features with close distances after being mapped by the defect feature extraction model, so as to facilitate determining whether the defects are similar based on the distance between the defect features mapped by the defect feature extraction model.

[0164] Optionally, the distance between defect features mapped by the defect feature extraction model can be used to determine the similarity between defects for subsequent business analysis.

[0165] Regarding similar defects, this method process does not limit the specific determination method.

[0166] Optionally, similar defects may be pre-set. Specifically, different defects having similar corresponding defective product images may be pre-set as similar defects.

[0167] For example, product images with "screen stain defects" and product images with "screen damage defects" often have similarities in the images, and thus can be set as similar defects. These two types of defective product images can be treated as images with similar defects, and similar defect features can be extracted by the defect feature extraction model.

[0168] Alternatively, similar defects can be discovered during model training. Specifically, during the iterative training of the defect feature extraction model, different defects corresponding to different defect features that are close in distance are identified as similar defects, thereby enabling similar defects to be discovered based on data analysis.

[0169] 2. About the training method of defect feature extraction model.

[0170] This method does not limit the training method of the defect feature extraction model.

[0171] Optionally, supervised training, unsupervised training, or a combination of supervised training and unsupervised training can be used.

[0172] Optionally, the defect feature extraction model can be trained directly, or an output layer can be added to the defect feature extraction model so that the output layer can predict the defect features extracted by the defect feature extraction model, and then the defect feature extraction model and the output layer can be integrated for training.

[0173] For example, a defect classification output layer can be added to the defect feature extraction model so that the defect classification output layer can predict the corresponding defects based on the defect features extracted by the defect feature extraction model. Furthermore, the defect feature extraction model and the defect classification output layer can be trained in a combined manner. Specifically, supervised training can be used.

[0174] This method process does not limit the source of the training sample set for training the defect feature extraction model.

[0175] Optionally, images of products in a factory production process may be acquired, or images of products with defects may be acquired.

[0176] The generation of defects can be related to the process stages during the product production process. For example, a screen defect may be broken during screen assembly, or a housing defect may be broken during housing installation.

[0177] To facilitate understanding, this method provides a specific example process, where sample images of defective products can be collected based on the AI ​​platform.

[0178] The Automated Optical Inspection (AOI) equipment on product lines can interact with the platform to achieve automated defect data detection and reflow mechanisms. AOI equipment is a crucial inspection tool and process quality control tool for ensuring product quality in the electronics manufacturing industry. During automated inspection, the AOI machine uses a high-definition camera to automatically scan panel products or other industrial products, capture images, and perform subsequent defect detection and location.

[0179] In response to the real-time upload of production line equipment, the AI ​​platform performs initial classification and storage based on parameters such as the product's corresponding ID, process segment, and defect type (what defect).

[0180] In an optional embodiment, in one round of training iteration of the defect feature extraction model, supervised training may be performed, unsupervised training may be performed, or supervised training and unsupervised training may be performed successively.

[0181] The following explains supervised training and unsupervised training respectively.

[0182] 1. Supervised training.

[0183] This method is not limited to a specific supervised training method.

[0184] Optionally, multiple defective product images labeled with similar defects may be obtained and used as supervised training samples for training.

[0185] Optionally, training samples with images of different defective products as features and similar defects as labels can be obtained to perform supervised training on the defect feature extraction model.

[0186] Alternatively, images of different defective products labeled with dissimilar defects may be obtained as supervised training samples for training. Specifically, defect features extracted from defective product images with dissimilar defects may be updated through training to reduce similarity or increase difference.

[0187] Optionally, the above two types of training samples may be combined for training.

[0188] This method does not limit the loss calculation method of supervised training.

[0189] Optionally, the loss of supervised training can be used to characterize the similarity of the extracted defect features between different product images with similar defects, so that the model can be updated to increase the similarity of the defect features of similar defects.

[0190] Optionally, the supervised training loss can be used to characterize the difference in defect features extracted between different product images with dissimilar defects, so that the model can be updated to increase the difference in defect features of similar defects.

[0191] Optionally, the above two types of losses may be combined for updating.

[0192] In an optional embodiment, supervised training can be performed in combination with pre-set similar defect labels.

[0193] Optionally, the training method of the defect feature extraction model may include: looping through the following training steps until a preset training stop condition is met:

[0194] Acquire multiple groups of different product images with similar known defects; and use the acquired multiple groups of product images to train the current defect feature extraction model to improve the similarity of known defect features extracted between product images within a group and / or improve the difference of known defect features extracted between product images of different groups.

[0195] Optionally, different groups of product images may not have similar defects, thereby facilitating improving the differences in known defect features extracted between different groups of product images.

[0196] Alternatively, if only the similarity of known defect features extracted between product images within a group is improved, product images in different groups may also have similar defects.

[0197] This embodiment can be combined with labeled samples to train the defect feature extraction model, thereby improving the model training effect and the accuracy of similar defect recognition.

[0198] This embodiment does not limit the specific forms of the known defect feature similarity and the known defect feature difference.

[0199] Optionally, the similarity and difference of known defect features can be measured using feature distances in the feature space. Specifically, this can be done by reducing the distances in the feature space between known defect features extracted between product images within a group and / or increasing the distances in the feature space between known defect features extracted between product images in different groups.

[0200] Optionally, different product images with the same known defect can also be regarded as product images with similar known defects, thereby facilitating the extraction of more accurate defect features.

[0201] Optionally, the training method of the defect feature extraction model may include: looping through the following training steps until a preset training stop condition is met:

[0202] Acquire multiple groups of different product images with the same known defects, and determine each group of product images as a product image with similar known defects; use the multiple groups of product images acquired to train the current defect feature extraction model to improve the similarity of the known defect features extracted between product images in the same group, and / or improve the difference of the known defect features extracted between product images in different groups.

[0203] This embodiment does not limit the preset training stop condition. Optionally, the preset training stop condition can be at least one of the following: the number of training cycles reaches a preset number, the similarity of known defects within a group is greater than a preset similarity threshold, and the difference of known defects between groups is greater than a preset difference threshold, etc.

[0204] The same known defects can belong to similar defects and, therefore, can be used as labeled samples for supervised training.

[0205] This embodiment can use product images with the same known defects as labeled samples for training, thereby improving training efficiency and reducing the difficulty of sample collection.

[0206] Different groups of product images may have different known defects.

[0207] This embodiment does not limit the specific forms of the known defect feature similarity and the known defect feature difference.

[0208] Optionally, the similarity and difference of known defect features can be measured using feature distances in the feature space. Specifically, this can be done by reducing the distances in the feature space between known defect features extracted between product images within a group and / or increasing the distances in the feature space between known defect features extracted between product images in different groups.

[0209] This method does not limit the loss calculation method of supervised training.

[0210] Optionally, in the case of using multiple groups of different product images with the same known defects as training samples for supervised training, a classification output layer can be added to the defect feature extraction model to predict the defect type.

[0211] Furthermore, the loss can characterize whether the same correct defect is predicted for product images within any group.

[0212] This embodiment can combine product images with the same known defects and use the additional classification output layer to predict the defect type, thereby training the defect feature extraction model.

[0213] To facilitate understanding, the embodiments of the present invention also provide specific examples.

[0214] In one example, intra-class distance optimization can be used to optimize the defect feature extraction model (Resnet50 structure), that is, using a training dataset, intra-class distance optimization and positive sample classification for training.

[0215] For intra-class distance optimization, a custom distance function can be used to calculate the distance between the extracted defect features for each group of different product images with the same known defects.

[0216] Based on the above distance function, the distances between defect features within a group with the same known defect can be made as close as possible.

[0217] For positive sample classification, a new classification output layer can be added to predict the defect type, ensuring that product image samples within a group with the same known defect receive the same defect classification. The greater the probability that the output defect prediction is correct and the same, the smaller the loss.

[0218] 2. Unsupervised training.

[0219] The process of this method is not limited to a specific unsupervised training method.

[0220] Optionally, clustering may be performed on the initially extracted defect features, and defects corresponding to each category of defect features in the clustering results may be determined as similar defects and labeled to obtain product image samples labeled with pseudo labels.

[0221] Optionally, training can be performed based on pseudo labels, specifically referring to supervised training.

[0222] Optionally, training can also be performed in combination with sampling, as explained below.

[0223] In an optional embodiment, unsupervised training may be performed on the defect feature extraction model.

[0224] Optionally, the defect feature extraction model training method includes: looping through the following training steps until a preset training stop condition is met:

[0225] Acquire product images with known defects; extract known defect features from the acquired product images based on a current defect feature extraction model; cluster the extracted known defect features to obtain at least two similar defect feature groups, and label each acquired product image with the similar defect feature group to which the corresponding known defect feature belongs; train the current defect feature extraction model based on the labeled product images to improve the similarity of known defect features extracted between product images with the same labeling, and / or improve the difference of known defect features extracted between product images with different labeling.

[0226] Among them, the marked similar defect feature groups can be regarded as pseudo labels.

[0227] This embodiment does not limit the clustering algorithm. Optionally, a density-based spatial clustering algorithm (DBSCAN), mean shift clustering, or kmeans can be used for clustering.

[0228] This embodiment does not limit the specific method of training the current defect feature extraction model based on the annotated product image. Optionally, the training can be based on a supervised method.

[0229] Optionally, some special known defect features may be sampled for each annotation group, and then the loss may be determined by comparing similar defect feature groups and the similarity between the sampling results.

[0230] For ease of understanding, in a specific example, the distance between known defect features within each annotation group can be continuously shortened to improve similarity. Therefore, by sampling the known defect feature that is most off-center within each annotation group, or the average value of the features within each annotation group, or the weighted average value of the features within each annotation group, etc., the loss can be used to update the distance between features within similar defect feature groups.

[0231] Optionally, training the current defect feature extraction model based on the annotated product image may include: looping through the following steps until a first training stop condition is met:

[0232] Based on the current defect feature extraction model, known defect features are extracted from the annotated product image to obtain a first feature set; based on the preset strategy, a known defect feature is determined based on the known defect features corresponding to each annotation to obtain a second feature set; based on the obtained first feature set and second feature set, the loss is calculated, and the current defect feature extraction model is updated.

[0233] The present embodiment does not limit the first training stop condition. Optionally, the first training stop condition may be at least one of the following: the number of training cycles reaches a preset number, the loss is less than a preset loss threshold, and the like.

[0234] This embodiment does not limit the preset strategy.

[0235] Optionally, the average values ​​of the known defect features corresponding to the respective annotations may be determined and combined into a second feature set. That is, for each annotation, the average values ​​of the known defect features under each annotation may be combined into the second feature set.

[0236] Alternatively, based on the known defect features corresponding to each annotation, the known defect features farthest from the cluster center can be determined and combined into a second feature set. That is, for each annotation, the known defect features farthest from the cluster center can be combined into the second feature set. Alternatively, the known defect features with the lowest similarity to the cluster center can be determined and combined into the second feature set.

[0237] Among them, in different cycles of operation, different second feature sets can be combined to obtain different first feature sets.

[0238] Optionally, one or more known defect features may be randomly selected based on the known defect features corresponding to each annotation and combined into a second feature set.

[0239] Optionally, since the above steps are executed in a loop, the known defect feature farthest from the preset feature can be determined based on the known defect features corresponding to each annotation; the preset feature can be the feature corresponding to the same annotation in the second feature set obtained in the previous loop combination.

[0240] This embodiment does not limit the calculation method of the loss between the first feature set and the second feature set.

[0241] Optionally, the loss between the first feature set and the second feature set can be used to represent the similarity between the first feature set and the second feature set.

[0242] It can be understood that, in one example, when the known defect features farthest from the cluster center are extracted and combined into a second feature set, if the second feature set is still highly similar to the first feature set, it means that the known defect features under the same annotations are highly similar, thereby obtaining a defect feature extraction model that maps different product images with similar known defects into similar defect features.

[0243] Optionally, the loss between the first feature set and the second feature set may be calculated using a cross entropy or contrastive learning loss (info Noise Contrastive Estimation, infoNCE) calculation method.

[0244] For ease of understanding, an optional embodiment is given below.

[0245] Optionally, training the current defect feature extraction model based on the annotated product image may include the following steps:

[0246] Randomly select one known defect feature from the known defect features corresponding to each annotation and add it to the initial feature set; and add each feature in the initial feature set to the edge strategy feature set and the average strategy feature set respectively;

[0247] The following steps are executed repeatedly until the second training stop condition is met:

[0248] Based on the current defect feature extraction model, known defect features are extracted from the annotated product images to obtain a full feature set.

[0249] Update the edge policy signature set based on the following steps:

[0250] For the target defect feature corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect feature with the lowest similarity to the target defect feature is determined as the edge feature; and the target defect feature is updated based on the edge feature;

[0251] Update the average policy feature set based on the following steps:

[0252] For the target defect feature corresponding to the target annotation in the average strategy feature set, the mean is calculated for each known defect feature corresponding to the target annotation in the full feature set, and the target defect feature is updated based on the calculated mean;

[0253] Based on the updated edge strategy feature set and the updated average strategy feature set, the current initial feature set is updated;

[0254] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0255] This embodiment does not limit the number of labeled product images in the process of obtaining the full feature set. Optionally, multiple labeled product images can be obtained. To facilitate subsequent operations, multiple product images labeled with different pseudo labels can be obtained.

[0256] Optionally, for each annotated pseudo-label, the same number of product images can be obtained to extract known defect feature combinations to obtain a full feature set.

[0257] The full feature set obtained in different cycles may be different.

[0258] This embodiment does not limit the second training stop condition. Optionally, the second training stop condition may be at least one of the following: the number of training cycles reaches a preset number, and the loss is less than a preset loss threshold.

[0259] This embodiment can realize unsupervised training of the defect feature extraction model by setting the initial feature set, the edge strategy feature set and the average strategy feature set. For a group of similar defect features under a single label, the known defect features in the group can be further aggregated to improve the similarity of the known defect features in the group.

[0260] In this embodiment, the target label may be any label, and specifically, the edge strategy feature set and the average strategy feature set may be updated for each label.

[0261] This embodiment does not limit the method for updating the target defect feature based on the edge feature. Alternatively, a weighted method can be used to determine the weighted sum of the edge feature and the target defect feature as the new target defect feature. Alternatively, a replacement method can be used to determine the edge feature as the new target defect feature.

[0262] Optionally, the weight values ​​in the weighted sum can be hyperparameters or model parameters that are updated during the training process.

[0263] Similarly, this embodiment does not limit the method of updating the target defect feature according to the calculated mean value. Specific explanations can be found above.

[0264] This embodiment does not limit the method for updating the current initial feature set. Optionally, for each annotation, a weighted sum of the corresponding feature in the updated edge strategy feature set, the corresponding feature in the updated average strategy feature set, and the corresponding feature in the current initial feature set can be calculated. The weighted sum is determined as the new corresponding feature in the initial feature set.

[0265] Optionally, for each annotation, a weighted sum of the corresponding feature in the updated edge strategy feature set and the corresponding feature in the updated average strategy feature set may be calculated, and the weighted sum may be determined as a new corresponding feature in the initial feature set.

[0266] Optionally, the weight values ​​in the weighted sum can be hyperparameters or model parameters that are updated during the training process.

[0267] This embodiment does not limit the calculation method of the loss between the current initial feature set and the full feature set.

[0268] Optionally, the loss between the updated initial feature set and the full feature set can be used to represent the similarity between the updated initial feature set and the full feature set.

[0269] Optionally, the loss between the updated initial feature set and the full feature set can be calculated using a cross entropy or contrastive learning loss method.

[0270] Of course, it is understandable that one or more strategic feature sets may be set to sample known defect features using different strategies.

[0271] 1) Only edge policy updates are performed.

[0272] Optionally, training the current defect feature extraction model based on the annotated product image may include the following steps:

[0273] Randomly select one known defect feature from the known defect features corresponding to each annotation and add it to the initial feature set; and add each feature in the initial feature set to the edge strategy feature set;

[0274] The following steps are executed repeatedly until the third training stop condition is met:

[0275] Based on the current defect feature extraction model, known defect features are extracted from the annotated product images to obtain a full feature set.

[0276] Update the edge policy signature set based on the following steps:

[0277] For the target defect feature corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect feature with the lowest similarity to the target defect feature is determined as the edge feature; and the target defect feature is updated based on the edge feature;

[0278] Based on the updated edge strategy feature set, update the current initial feature set;

[0279] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0280] 2) Only average policy updates are performed.

[0281] Optionally, training the current defect feature extraction model based on the annotated product image may include the following steps:

[0282] Randomly select one known defect feature from the known defect features corresponding to each annotation and add it to the initial feature set; and add each feature in the initial feature set to the average strategy feature set;

[0283] The following steps are executed repeatedly until the fourth training stop condition is met:

[0284] Based on the current defect feature extraction model, known defect features are extracted from the annotated product images to obtain a full feature set.

[0285] Update the average policy feature set based on the following steps:

[0286] For the target defect feature corresponding to the target annotation in the average strategy feature set, the mean is calculated for each known defect feature corresponding to the target annotation in the full feature set, and the target defect feature is updated based on the calculated mean;

[0287] Based on the updated average strategy feature set, update the current initial feature set;

[0288] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0289] 3) Other strategies can also be set. For example, a weighted strategy. Specifically, an additional weighted strategy feature set can be set for update.

[0290] For ease of understanding, the embodiment of the present invention also provides a specific unsupervised training method.

[0291] For full-sample unsupervised learning, this embodiment first uses a clustering algorithm to perform pseudo-label annotation.

[0292] For the clustering algorithm, this method uses DBSCAN to perform unsupervised clustering, so that similar known defect features are clustered together.

[0293] DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm that automatically divides data points into clusters without prior information, while also identifying and filtering out noise points. Its core idea is to determine the cluster boundaries by calculating the density around each data point, centered on it.

[0294] Specifically, the DBSCAN algorithm divides data points into three categories: core points, boundary points, and noise points. Core points are data points whose surrounding density reaches a certain threshold, boundary points are data points that do not meet the core point conditions but have at least one core point around them, and noise points are data points that are neither core points nor boundary points.

[0295] The main parameters of the DBSCAN algorithm are the radius r and the minimum density k, which control the size and density of the clusters respectively. The time complexity of the DBSCAN algorithm is O(n log n), where n is the number of data points.

[0296] The core formula of the DBSCAN clustering algorithm is the definition of density-reachable and density-connected:

[0297] Density direct access: If data point q is in the ε-neighborhood of data point p, and data point p is a core point, then data point q and data point p are density direct access.

[0298] Density reachable: If there exists a chain of data points {p1,p2,…,pn}, for any 1≤i≤n, data point pi and data point pi+1 are density reachable, then data point p1 and data point pn are density reachable.

[0299] Another important parameter in the DBSCAN algorithm is the minimum density k, which represents the minimum number of data points within the radius r. If the number of data points in the ε-neighborhood of a data point is less than k, the point is considered a noise point.

[0300] The cluster expansion process of the DBSCAN algorithm can be described as follows:

[0301] 1) Randomly select a data point p that has not been visited;

[0302] 2) Mark p as visited;

[0303] 3) If p is a core point, find all data points in its ε-neighborhood, mark them as visited, and add them to the current cluster;

[0304] 4) Repeat steps 1-3 until all data points have been visited.

[0305] After obtaining the initial cluster classification using DBSCAN, this embodiment can perform random sampling according to the classification feature information of the initial cluster of pseudo labels, and store the sampled information in the corresponding conceptual Memory feature library to form an initialized Memory feature library for subsequent model training in the current training round.

[0306] After obtaining the initialized Memory feature library, this embodiment designs a corresponding process update and iteration model and the Memory feature library.

[0307] First, in the current training round, each batch of training requires extracting query images to train the defect feature extraction model. Each batch of query datasets contains P*K images of pseudo-labeled defects obtained based on the initial training clustering, where P represents the number of pseudo-labeled categories and K represents the number of pseudo-labeled categories. For each query dataset, NCE Loss can be used to impose relevant constraints.

[0308] The part after feature extraction can use the Batch Hard strategy and Average strategy to update the memory feature library to form a corresponding feature group.

[0309] For the convenience of description, the initialized memory feature library is called the initial feature library, the feature library using the Batch Hard strategy is called the difficult feature library, and the feature library using the Average strategy is called the average feature library.

[0310] Among them, a known defect feature randomly extracted from each pseudo-label category can be added to the initial feature library, the difficult feature library and the average feature library through random sampling.

[0311] Afterwards, the above three feature libraries can be updated separately through the update step.

[0312] First, we can extract defect features from the product images in the query dataset based on the current defect feature extraction model and combine them into a full feature set.

[0313] The features in the difficult feature library can be updated according to the following steps.

[0314] For any pseudo-label, obtain the corresponding first feature in the current difficult feature library; then obtain the corresponding features in the full feature set, and determine the difficult feature with the lowest similarity to the first feature; specifically, KL divergence or Euclidean distance can be used for calculation.

[0315] Afterwards, the weighted sum of the first feature and the difficult feature can be determined as a new first feature, thereby updating the features of the difficult feature library.

[0316] The specific weighted weight may be a momentum update factor, which may be a hyperparameter or an updateable model parameter.

[0317] The features in the average feature library can be updated according to the following steps.

[0318] For any pseudo-label, obtain the corresponding second feature in the current average feature library; then obtain the corresponding feature in the full feature set and determine the average value between the features.

[0319] Afterwards, the weighted sum of the second feature and the feature average value may be determined as a new second feature, thereby updating the features of the average feature library.

[0320] The specific weighted weight may be a momentum update factor, which may be a hyperparameter or an updateable model parameter.

[0321] After updating the difficult feature library and the average feature library, the two feature libraries can be merged. Specifically, the weighted sum of the features corresponding to each pseudo-label can be calculated separately and updated to the current initial feature library.

[0322] The current initial feature library can then be compared with the full feature set using the InfoNCE loss. Updating based on the InfoNCE loss ensures that the features in the current initial feature library are as similar as possible to those in the full feature set with the same pseudo-label, and dissimilar to those with different pseudo-labels.

[0323] 3. The steps of a round of training iteration.

[0324] This method does not limit the steps performed in one training iteration.

[0325] Optionally, one round of training iteration may include supervised training and / or unsupervised training, or other training methods.

[0326] Optionally, the same or different steps may be performed in different training iterations.

[0327] Optionally, in one round of training iteration, supervised training may be performed first, followed by unsupervised training; or unsupervised training may be performed first, followed by supervised training.

[0328] For ease of understanding, this method flow provides an optional embodiment.

[0329] Optionally, the training method of the defect feature extraction model includes:

[0330] The following training steps are executed repeatedly until the preset training stop condition is met:

[0331] Acquire multiple groups of different product images having the same known defects, and determine each group of product images as a product image having similar known defects;

[0332] The current defect feature extraction model is trained using the acquired multiple groups of product images to improve the similarity of known defect features extracted between product images within a group and / or to improve the difference of known defect features extracted between product images of different groups.

[0333] Optionally, the training step may further include:

[0334] Obtain images of products with known defects;

[0335] Extract known defect features from the acquired product image based on the current defect feature extraction model;

[0336] Clustering the extracted known defect features to obtain at least two similar defect feature groups, and labeling each acquired product image with the similar defect feature group to which the corresponding known defect feature belongs;

[0337] The current defect feature extraction model is trained based on the annotated product images to improve the similarity of known defect features extracted between product images with the same annotation and / or to improve the difference of known defect features extracted between product images with different annotations.

[0338] In this embodiment, constraints may be first performed through supervised training, thereby facilitating the improvement of the efficiency of unsupervised training and the training effect of the defect feature extraction model.

[0339] The detailed explanation of this embodiment can be found above.

[0340] 4. Training is over.

[0341] In an optional embodiment, the trained defect feature extraction model can be used to extract different known defect features with a similarity greater than a first preset similarity for different product images with similar known defects.

[0342] This embodiment does not limit the specific form and value of the first preset similarity.

[0343] Optionally, the trained defect feature extraction model can be used to extract similar different known defect features from different product images with similar known defects, thereby facilitating feature extraction based on the defect feature extraction model and determining similar different known defects based on similar different known defect features.

[0344] Among them, different known defect features with a similarity greater than a first preset similarity may be similar known defect features.

[0345] The process of this method does not limit the training end conditions of the defect feature extraction model.

[0346] Optionally, the training termination condition of the defect feature extraction model may include: for any different product images with similar known defects in the training sample set, the similarity between different known defect features extracted is greater than a first preset similarity.

[0347] 3. About defect detection model.

[0348] This method does not limit the structure and training method of the defect detection model.

[0349] Optionally, the defect detection model for known defect detection can be a graph convolution or graph neural structure.

[0350] Optionally, the defect detection model can be trained through supervised training.

[0351] Among them, different defect detection models can be used to detect different defects.

[0352] In the process of this method, at least two known defects and different defect detection models corresponding to different known defects can be pre-set; the defect detection model can be used to detect whether the input product image has the corresponding known defects.

[0353] Afterwards, known defects and corresponding defect detection models can be continuously updated through supervised training or transfer training.

[0354] 4. About transfer training.

[0355] The present method does not limit the specific form and content of the preset similarity condition.

[0356] Optionally, the preset similarity condition may be that the similarity is greater than a first preset similarity; or that the similarity is greater than a second preset similarity, wherein the second preset similarity is greater than the first preset similarity; or that the similarity is the highest; or that the top N known defect features after being sorted from high to low according to similarity.

[0357] Optionally, the defect feature set may include known defect features extracted by the defect feature extraction model for one or more known defect product images with known defects, to facilitate subsequent similarity calculation and determination of candidate defects.

[0358] The present method is also not limited to a specific method for selecting an alternative defect detection model. For ease of description, a defect detection model that corresponds one-to-one to an alternative defect may be referred to as an alternative defect detection model.

[0359] It should be noted that when an alternative defect is determined, screening is not required, and the corresponding defect detection model is directly determined for the alternative, and the obtained new defect product image is used for migration training to obtain a new defect detection model.

[0360] For ease of understanding, the present invention provides two optional embodiments for illustrative purposes.

[0361] 1. Optionally, performing transfer training on the defect detection model corresponding to the determined candidate defect using the acquired new defective product image may include the following steps:

[0362] For each of the multiple defect detection models corresponding to the multiple candidate defects, the detection accuracy of each of the new defects is determined using a test set; the test set may include images of new defective products with the new defects;

[0363] Select a defect detection model whose detection accuracy meets the preset detection conditions, and use the acquired new defective product images for migration training.

[0364] The present embodiment does not limit the specific form and content of the preset detection conditions. Optionally, the preset detection conditions may be the highest detection accuracy, the detection accuracy greater than the preset accuracy, the detection accuracy being in the top N, etc.

[0365] There may be multiple candidate defects determined, and therefore, multiple defect detection models corresponding to the multiple candidate defects determined one by one can be obtained.

[0366] This embodiment can further screen out suitable defect detection models for similar defects based on the test detection of the test set to perform transfer training, thereby improving the efficiency and effect of transfer training.

[0367] 2. Optionally, performing transfer training on the defect detection model corresponding to the determined candidate defect using the acquired new defective product image may include the following steps:

[0368] For the multiple defect detection models corresponding to the multiple candidate defects, transfer training is performed using the acquired new defect product image to obtain multiple candidate new defect detection models;

[0369] For each candidate new defect detection model, the detection accuracy of the new defect is determined using a test set; the test set may include images of new defective products with new defects;

[0370] The alternative new defect detection model whose detection accuracy meets the preset accuracy condition is determined as the new defect detection model.

[0371] The present embodiment does not limit the specific form and content of the preset accuracy condition. Optionally, the preset accuracy condition may be the highest detection accuracy, the detection accuracy greater than the preset accuracy, or the detection accuracy being in the top N.

[0372] This embodiment can be based on the test detection of the test set, migrate the alternative defect detection model, and then calculate the detection accuracy through the test set to screen out a suitable alternative defect detection model as the new defect detection model, which can improve the effect of migration training.

[0373] This method does not limit the specific transfer training method.

[0374] Alternatively, fine-tuning training or supervised training can be performed based on product image samples with new defects. Specifically, some parameters in the model to be migrated can be adjusted and updated.

[0375] Optionally, a smaller learning rate can be used to adjust the parameters of the model to be transferred, which helps to maintain the original knowledge of the model to be transferred.

[0376] For easier understanding, the embodiment of the present invention further provides a specific example.

[0377] As shown in FIG2 , FIG2 is a schematic diagram showing a principle of defect detection model migration according to an embodiment of the present invention.

[0378] Among them, multiple defect detection models corresponding one-to-one to the determined multiple candidate defects can be combined into a backup model library.

[0379] Afterwards, a new defect image dataset can be collected and combined into a test set.

[0380] The test set is then used to determine the defect detection models in the backup model library. The detection accuracy of the new defects can be specifically combined into a model-accuracy table; the test set can include new defective product images with new defects.

[0381] Afterwards, you can select a defect detection model whose detection accuracy meets the preset detection conditions, and use the acquired new defective product images for migration training to facilitate rapid deployment on subsequent platforms.

[0382] For ease of understanding, the embodiment of the present invention further provides an application embodiment.

[0383] In screen production and other industrial scenarios, process defects appear at every stage of the production process as it progresses. With the development of customized deep learning models for defect detection technology, the number of defect detection models has gradually increased. However, when new defects emerge or similar defects occur on other production lines, rapid defect detection model development becomes difficult. Furthermore, due to the lack of comparative analysis data for production line defects, it is difficult to generate a large number of annotation files.

[0384] To this end, this embodiment designs an unsupervised similarity analysis process and specific method for defects in screens or other industrial scenarios, evaluates the similarity of defects occurring in different scenarios, and then quickly develops and deploys model transfer learning of existing models for defects with high similarity.

[0385] Image similarity analysis is a key problem in the field of computer vision, used in applications such as image retrieval, image classification, and image recognition. Deep learning is a key technology that has achieved tremendous success in computer vision in recent years. It can automatically learn image feature representations from large amounts of data and has achieved excellent performance in tasks such as image similarity analysis. Convolutional neural networks (CNNs) are commonly used in deep learning for image processing. CNNs extract meaningful features from images, convert them into high-dimensional feature vectors, and then calculate image similarity by comparing these feature vectors. Common methods for comparing feature vectors include calculating metrics such as Euclidean distance or cosine similarity. These metrics can be used to generate a set of similarity scores, which can then be used to rank images.

[0386] Unsupervised learning is a machine learning method that, unlike supervised learning, does not require pre-labeled data. Instead, it automatically learns patterns and features from unlabeled data. Unsupervised learning can be used in a variety of fields, such as data mining, pattern recognition, image processing, and speech recognition. The goal of this method is to discover inherent structures and patterns in data to better understand the data and subsequent tasks.

[0387] This embodiment designs a defect similarity analysis process and specific method for screens or other industrial scenarios, evaluates the similarity of defects appearing in different scenarios, and then quickly develops and deploys model transfer learning of existing models for defects with high similarity.

[0388] This embodiment designs a defect similarity analysis process and specific method for screens or other industrial scenarios, evaluates the similarity of defects appearing in different scenarios, and then quickly develops and deploys model transfer learning of existing models for defects with high similarity.

[0389] It includes similarity measurement models and processes for screen and other industrial defects; methods and processes for rapid transfer learning based on existing models after model matching; and methods for managing and storing stock models and existing defect sample images based on similarity measurement.

[0390] This embodiment can greatly reduce the repetitive model development work in the industrial production process, improve the efficiency and accuracy of industrial defect detection development, and reduce the data cost of industrial defect detection and the development cycle when new defects appear in different scenarios.

[0391] This embodiment designs an unsupervised similarity analysis process and specific method for defects in screens or other industrial scenarios.

[0392] Specifically, a screen defect feature extraction model can be trained through supervised training and unsupervised training to extract different similar known defect features for different screen product images with similar known defects. Specifically, the different known defect features can have a similarity greater than a first preset similarity.

[0393] Then, for new screen defects, the screen defect feature extraction model can be used to determine the target known defects that are similar to the new defects, so that transfer training can be performed based on the defect detection model of the target known defects to obtain a screen defect detection model for detecting new screen defects.

[0394] For details, please refer to the explanation of the above method embodiment.

[0395] In addition, the similarity between different defect features can be determined based on the defect feature extraction model. The defect feature similarity can then be determined as the similarity between defect detection models and displayed to users for easy viewing and selection.

[0396] In the case of existing defect detection model similarity information (that is, the similarity between defects), this embodiment can use the following steps to manage different models with similarities using a relational database:

[0397] 1. Design the database schema: First, design a relational database schema to store the model metadata (such as model name, type, creation date, performance metrics, etc.) and similarity information. The database schema includes the following tables:

[0398] Model table: stores basic information of the model, such as model ID, name, type, etc.

[0399] Similarity table: stores similarity information between models, including model ID, reference model ID, and similarity value.

[0400] 2. Import similarity information: Import the existing model similarity information into the similarity table. Ensure that each record contains the model ID, reference model ID, and similarity value.

[0401] 3. Implement query function: Implement query function for model similarity according to requirements. For example, the following functions can be implemented:

[0402] Query the model most similar to a given model; query the similarity values ​​between multiple models; query similar models within a given threshold.

[0403] 4. Model organization and management: Based on similarity information, similar models can be organized together for management. This can be achieved through the following methods:

[0404] Add a field (e.g. group_id) to the model table to assign similar models to the same group.

[0405] Create a separate model group table to store information about model groups (such as group ID, group name, description, etc.). In the model table, add a field (for example: group_id) to each model to indicate the model group it belongs to.

[0406] 5. Update and maintenance: According to needs, the update and maintenance functions of model similarity information and model metadata are implemented.

[0407] Update model metadata;

[0408] Update the similarity information between models;

[0409] Add a new model and import similarity information.

[0410] By following the above steps, this embodiment can use a relational database to manage different models with similarities.

[0411] To visualize the model information of the similarity management database on the web page, you can take the following steps:

[0412] 1. Choose a visualization library: Choose a visualization library suitable for the web, such as D3.js, Plotly.js, Chart.js, ECharts, etc.

[0413] 2. Create a web application: Create a basic web application to display the visualization in a browser.

[0414] 3. Connect to the database: On the backend of your web application (e.g., using Node.js, Flask, Django, etc.), connect to the relational database to query model information and similarity information. Implement an API interface to pass data from the database to the frontend.

[0415] 4. Get data: On the front end, use AJAX or the Fetch API to retrieve model information and similarity information from the backend API. Process the data to make it suitable for visualization.

[0416] 5. Create visualizations: Use the selected visualization library to plot the acquired data into charts. For example:

[0417] Scatter plot: Displays the similarity between models. In a scatter plot, each model is represented by a dot, and the x- and y-axes represent the model's key characteristics. Model similarity can be represented using color, shape, or size. Models with high similarity appear closer together in the plot. Hovering over a dot displays detailed information about the model.

[0418] Matrix: Displays the similarity between models. A matrix is ​​a two-dimensional table with rows and columns representing models. The color of the cell indicates the similarity between the models. Darker colors indicate higher similarity. Hovering over a cell displays the specific similarity value.

[0419] Hierarchical Diagram: Displays model clustering based on similarity, using a tree diagram or circular tree diagram to illustrate hierarchical relationships. The root node of the tree represents the entire model set, child nodes represent clusters, and leaf nodes represent models. Models with high similarity are placed on the same branch in the tree diagram. Hovering over a node displays detailed information about the model or cluster.

[0420] Force-Directed Graph: Displays the similarity relationships between models. In a force-directed graph, models are nodes, and similarity is weighted as the edge. The distance between nodes reflects the similarity relationship, with models with high similarity being closer in the graph. Hovering over a node displays detailed information about the model, and hovering over an edge displays the similarity value.

[0421] 6. Interactive functions: Add interactive functions such as zooming, dragging, hover prompts, click events, etc. as needed. This will make the visualization more attractive and help users better understand the model information.

[0422] 7. Responsive Design: To make visualizations work well on different devices (e.g., desktop, tablet, and mobile), apply responsive design. Use CSS and JavaScript to adapt to different screen sizes and device types.

[0423] Through the above steps, this embodiment can visualize the model information of the similarity management database on the web page.

[0424] In addition, an embodiment of the present invention also discloses another defect detection model migration method.

[0425] As shown in FIG3 , FIG3 is a flow chart of another defect detection model migration method according to an embodiment of the present invention.

[0426] The embodiment of the present invention does not limit the execution subject of the method process. Optionally, the execution subject can be any computing device, for example, a server or client for detecting product defects.

[0427] The method may include the following steps.

[0428] S201: For a new defective product image having a new defect, known defects similar to the new defect are determined as candidate defects by feature extraction.

[0429] S202: Acquire a candidate defect detection model for detecting whether an input product image has a candidate defect.

[0430] S203: Perform transfer training on the acquired candidate defect detection model using the new defective product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

[0431] The present method is not limited to a specific feature extraction method. Alternatively, an image recognition model may be used to extract defect features, or the above-mentioned defect feature extraction model may be used to extract defect features.

[0432] Optionally, the method of extracting new defect features from a new defective product image may be the same as the method of extracting known defect features.

[0433] Optionally, for the new defective product image with the new defect, the known defect similar to the new defect is determined as an alternative defect through feature extraction. Specifically, it can be: for the new defective product image with the new defect, the new defect feature is extracted through a preset method; for the known defect feature whose similarity with the new defect feature meets the preset similarity condition, the corresponding known defect is determined as an alternative defect; wherein the known defect feature is extracted through a preset method.

[0434] The method flow is not limited to known defects that are similar to the new defect.

[0435] Optionally, the known defects similar to the new defect may specifically be known defect features whose similarity to the new defect features satisfies a preset similarity condition, and further corresponding known defects. This embodiment does not limit the preset similarity condition; alternatively, the preset similarity condition may be the highest similarity, or the similarity being greater than a preset similarity threshold.

[0436] Optionally, the known defect similar to the new defect may specifically be a known defect feature having a similarity with the new defect feature greater than a first preset similarity, and further a corresponding known defect.

[0437] This method flow does not limit the specific method of determining the known defects similar to the new defect.

[0438] Optionally, the determination may be made based on the similarity between defect features, or based on the similarity between a new defective product image and a known defective product image.

[0439] This method does not limit the method for obtaining the alternative defect detection model.

[0440] Alternatively, a candidate defect detection model for detecting whether an input product image has a candidate defect may be directly selected from different defect detection models that are preset and correspond one-to-one to different known defects. Specific selection methods can be found in the above method embodiments.

[0441] Optionally, since a large number of samples of known defects can generally be accumulated, image samples of candidate defects can also be used for real-time training to obtain a candidate defect detection model for detecting whether an input product image has the candidate defect.

[0442] Optionally, selection and transfer training can be performed in combination with alternative defect detection models based on the recognition accuracy of new defective product images.

[0443] The present method does not limit the specific method of performing transfer training. For details, please refer to the above method embodiments.

[0444] In addition, for a detailed explanation of the process of this method, please refer to the above method embodiment.

[0445] The above article uses the defect detection scenario as an example to explain the specific model migration method.

[0446] The embodiments of the present invention also provide model migration methods in other scenarios.

[0447] For example, a similar migration approach can be used to quickly create a new product type detection model for a new product type using a product type detection model. Specifically, for newly developed products, the new product type detection model can be used to detect them during machine sorting, facilitating subsequent sorting.

[0448] For another example, a similar migration method can be used for the detection model of item types to quickly obtain a new item type detection model for the new item type.

[0449] For another example, a similar migration method can be used for the detection model of an image type to quickly obtain a new image type detection model for a new image type.

[0450] The embodiment of the present invention is not limited to a specific image type.

[0451] Optionally, the image type may include at least one of the following: whether the product in the image has a certain defect, whether the product in the image belongs to a certain product type, whether the item in the image belongs to a certain item type, whether the person in the image belongs to a certain crowd type, etc.

[0452] As shown in FIG4 , FIG4 is a schematic flow chart of a method for migrating a type detection model according to an embodiment of the present invention.

[0453] The embodiment of the present invention does not limit the execution subject of the method process. Optionally, the execution subject can be any computing device, for example, a server or client for detecting types.

[0454] At least two known types and different type detection models corresponding to different known types may be pre-set; the type detection model may be used to detect whether the input image conforms to the corresponding known type.

[0455] The method may include the following steps.

[0456] S301: Obtain a pre-trained type feature extraction model and a type feature set.

[0457] The type feature extraction model may be used to extract different known type features with a similarity greater than a first preset similarity for different images of similar known types.

[0458] The type feature set may include known type features extracted by the type feature extraction model for images that conform to the known type.

[0459] S302: Input the image that meets the new type into the type feature extraction model to obtain new type features.

[0460] S303: In the type feature set, for known type features whose similarity to the new type feature meets a preset similarity condition, the corresponding known type is determined as a candidate type.

[0461] S304: For the type detection model corresponding to the determined candidate type, transfer training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0462] The above method process can improve the generation efficiency of the new type detection model for detecting the new type by determining the known types similar to the new type based on the similarity between type features, and performing transfer training based on the type detection model of the known types determined by detection.

[0463] The embodiment of the present invention does not limit the specific type of image matching.

[0464] Optionally, the type may include at least one of the following: whether the product in the image has a certain defect, whether the product in the image belongs to a certain product type, whether the item in the image belongs to a certain item type, whether the person in the image belongs to a certain crowd type, etc.

[0465] Optionally, the training method of the type feature extraction model includes:

[0466] The following training steps are executed repeatedly until the preset training stop condition is met:

[0467] Acquire multiple groups of different images that conform to the same known type, and determine each group of images as an image that conforms to a similar known type;

[0468] The acquired multiple groups of images are used to train the current type feature extraction model to improve the similarity of known type features extracted between images in the same group and / or to improve the difference of known type features extracted between images in different groups.

[0469] Optionally, the training step may further include:

[0470] Acquire images that match a known type;

[0471] Extracting known type features from the acquired image based on the current type feature extraction model;

[0472] Clustering the extracted known type features to obtain at least two similar type feature groups, and marking each acquired image with the similar type feature group to which the corresponding known type feature belongs;

[0473] The current type feature extraction model is trained based on the annotated images to improve the similarity of known type features extracted between images with the same annotation and / or to improve the difference of known type features extracted between images with different annotations.

[0474] Optionally, the current type feature extraction model is trained based on the annotated images, including:

[0475] The following steps are repeated until the first training stop condition is met:

[0476] Based on the current type feature extraction model, known type features are extracted from the annotated image to obtain a first feature set;

[0477] Based on a preset strategy, a known type feature is determined based on the known type features corresponding to each annotation to obtain a second feature set;

[0478] The loss is calculated based on the obtained first feature set and second feature set, and the current type feature extraction model is updated.

[0479] Optionally, the current type feature extraction model is trained based on the annotated images, including:

[0480] Randomly select a known type feature from the known type features corresponding to each annotation and add it to the initial feature set; and add each feature in the initial feature set to the edge strategy feature set and the average strategy feature set respectively;

[0481] The following steps are executed repeatedly until the second training stop condition is met:

[0482] Based on the current type feature extraction model, known type features are extracted from the annotated image to obtain the full feature set;

[0483] Update the edge policy signature set based on the following steps:

[0484] For the target type feature corresponding to the target annotation in the edge strategy feature set, among the known type features corresponding to the target annotation in the full feature set, the known type feature with the lowest similarity to the target type feature is determined as the edge feature; the target type feature is updated based on the edge feature;

[0485] Update the average policy feature set based on the following steps:

[0486] For the target type feature corresponding to the target label in the average strategy feature set, calculate the mean for each known type feature corresponding to the target label in the full feature set, and update the target type feature based on the calculated mean;

[0487] Based on the updated edge strategy feature set and the updated average strategy feature set, the current initial feature set is updated;

[0488] Calculate the loss based on the updated initial feature set and the full feature set, and update the current type feature extraction model.

[0489] Optionally, for the type detection model corresponding to the determined candidate type, transfer training is performed using the acquired new type image, including:

[0490] For each of the multiple type detection models corresponding to the multiple candidate types determined, the detection accuracy of each new type is determined using a test set; the test set contains new type images that meet the new type;

[0491] Select a type detection model whose detection accuracy meets the preset detection conditions, and use the acquired new type images for migration training.

[0492] Optionally, for the type detection model corresponding to the determined candidate type, transfer training is performed using the acquired new type image, including:

[0493] For the multiple type detection models corresponding to the multiple candidate types determined one by one, transfer training is performed using the acquired new type images to obtain multiple candidate new type detection models;

[0494] For each candidate new type detection model, the detection accuracy of the new type is determined using the test set; the test set contains new type images that meet the new type;

[0495] The alternative new type detection model whose detection accuracy meets the preset accuracy condition is determined as the new type detection model.

[0496] The detailed explanation of the process of this method can be found in the above method embodiment.

[0497] In addition, an embodiment of the present invention also discloses a migration method of another type of detection model.

[0498] As shown in FIG5 , FIG5 is a flow chart of a migration method of another type of detection model according to an embodiment of the present invention.

[0499] The embodiment of the present invention does not limit the execution subject of the method process. Optionally, the execution subject can be any computing device, for example, a server or client for detecting image types.

[0500] The method may include the following steps.

[0501] S401: For images that meet the new type, determine known types similar to the new type as candidate types by feature extraction;

[0502] S402: Acquire a candidate type detection model for detecting whether the input image conforms to the candidate type;

[0503] S403: For the obtained candidate type detection model, transfer training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0504] The present method does not limit the specific feature extraction method. Optionally, an image recognition model can be used to extract type features, or the above-mentioned type feature extraction model can be used to extract type features.

[0505] Optionally, the method for extracting new types of features for new types of images may be the same as the method for extracting features of known types.

[0506] Optionally, for new type images that conform to the new type, known types that are similar to the new type are determined as alternative types through feature extraction. Specifically, it can be: for images that conform to the new type, new type features are extracted through a preset method; for known type features whose similarity to the new type features meets a preset similarity condition, the corresponding known type is determined as an alternative type; wherein the known type features are extracted through a preset method.

[0507] The method flow is not limited to known types that are similar to the new type.

[0508] Optionally, the known type similar to the new type may specifically be a known type feature whose similarity to the new type feature satisfies a preset similarity condition, and further corresponds to the known type. This embodiment does not limit the preset similarity condition. Optionally, the preset similarity condition may be the highest similarity, or the similarity being greater than a preset similarity threshold.

[0509] Optionally, the known type similar to the new type may specifically be a known type feature whose similarity to the new type feature is greater than a first preset similarity, and further corresponds to the known type.

[0510] The present method flow is not limited to a specific method for determining a known type similar to a new type.

[0511] Optionally, the determination may be made based on the similarity between type features, or based on the similarity between a new type image and an image of a known type.

[0512] This method does not limit the method for obtaining the alternative type detection model.

[0513] Alternatively, the candidate type detection model for detecting whether the input image conforms to the candidate type may be directly selected from preset different type detection models corresponding to different known types. Specific selection methods can be found in the above method embodiments.

[0514] Optionally, since more samples of known types can generally be accumulated, real-time training can also be performed using image samples of alternative types to obtain an alternative type detection model for detecting whether an input image conforms to the alternative type.

[0515] Optionally, selection and transfer training can be performed in combination with the recognition accuracy of the alternative type detection model for new types of images.

[0516] The present method does not limit the specific method of performing transfer training. For details, please refer to the above method embodiments.

[0517] In addition, for a detailed explanation of the process of this method, please refer to the above method embodiment.

[0518] Corresponding to the above method embodiment, the embodiment of the present invention also provides an apparatus embodiment.

[0519] Figure 6 is a schematic diagram of a defect detection model migration apparatus according to an embodiment of the present invention. At least two known defects and corresponding defect detection models are pre-set. The defect detection models are used to detect whether an input product image contains the corresponding known defects.

[0520] The apparatus may include the following units.

[0521] An acquisition unit 501 is used to acquire a pre-trained defect feature extraction model and a defect feature set;

[0522] The defect feature extraction model is used to extract different known defect features having a similarity greater than a first preset similarity for different product images having similar known defects; the defect feature set includes the known defect features extracted by the defect feature extraction model for known defect product images having known defects;

[0523] The similarity unit 502 is configured to input the new defect product image having the new defect into the defect feature extraction model to obtain the new defect feature; and in the defect feature set, for known defect features whose similarity to the new defect feature meets a preset similarity condition, determine the corresponding known defect as a candidate defect;

[0524] The migration unit 503 is configured to perform migration training on the defect detection model corresponding to the determined candidate defect using the acquired new defect product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

[0525] Optionally, the training method of the defect feature extraction model includes:

[0526] The following training steps are executed repeatedly until the preset training stop condition is met:

[0527] Acquire multiple groups of different product images having the same known defects, and determine each group of product images as a product image having similar known defects;

[0528] The current defect feature extraction model is trained using the acquired multiple groups of product images to improve the similarity of known defect features extracted between product images within a group and / or to improve the difference of known defect features extracted between product images of different groups.

[0529] Optionally, the training step further includes:

[0530] Obtain images of products with known defects;

[0531] Extract known defect features from the acquired product image based on the current defect feature extraction model;

[0532] Clustering the extracted known defect features to obtain at least two similar defect feature groups, and labeling each acquired product image with the similar defect feature group to which the corresponding known defect feature belongs;

[0533] The current defect feature extraction model is trained based on the annotated product images to improve the similarity of known defect features extracted between product images with the same annotation and / or to improve the difference of known defect features extracted between product images with different annotations.

[0534] Optionally, training the current defect feature extraction model based on the annotated product images includes:

[0535] The following steps are repeated until the first training stop condition is met:

[0536] Based on the current defect feature extraction model, known defect features are extracted from the annotated product image to obtain a first feature set;

[0537] Based on a preset strategy, a known defect feature is determined based on the known defect features corresponding to each annotation to obtain a second feature set;

[0538] The loss is calculated based on the obtained first feature set and the second feature set, and the current defect feature extraction model is updated.

[0539] Optionally, training the current defect feature extraction model based on the annotated product images includes:

[0540] Randomly select one known defect feature from the known defect features corresponding to each annotation and add it to the initial feature set; and add each feature in the initial feature set to the edge strategy feature set and the average strategy feature set respectively;

[0541] The following steps are executed repeatedly until the second training stop condition is met:

[0542] Based on the current defect feature extraction model, known defect features are extracted from the annotated product images to obtain a full feature set.

[0543] Update the edge policy signature set based on the following steps:

[0544] For the target defect feature corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect feature with the lowest similarity to the target defect feature is determined as the edge feature; and the target defect feature is updated based on the edge feature;

[0545] Update the average policy feature set based on the following steps:

[0546] For the target defect feature corresponding to the target annotation in the average strategy feature set, the mean is calculated for each known defect feature corresponding to the target annotation in the full feature set, and the target defect feature is updated based on the calculated mean;

[0547] Based on the updated edge strategy feature set and the updated average strategy feature set, the current initial feature set is updated;

[0548] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0549] Optionally, the migration unit 503 is configured to:

[0550] For each of the multiple defect detection models corresponding to the determined candidate defects, the detection accuracy for the new defect is determined using a test set; the test set includes images of new defective products with the new defect;

[0551] Select a defect detection model whose detection accuracy meets the preset detection conditions, and use the acquired new defective product images for migration training.

[0552] Optionally, the migration unit 503 is configured to:

[0553] For multiple defect detection models corresponding to the determined candidate defects, transfer training is performed using the acquired new defect product image to obtain multiple candidate new defect detection models;

[0554] For each candidate new defect detection model, the detection accuracy for the new defect is determined using a test set; the test set contains images of new defective products with the new defect;

[0555] The alternative new defect detection model whose detection accuracy meets the preset accuracy condition is determined as the new defect detection model.

[0556] For detailed explanation, please refer to the above method embodiment.

[0557] As shown in FIG. 7 , FIG. 7 is a schematic structural diagram of another defect detection model migration device according to an embodiment of the present invention.

[0558] The apparatus may include the following units.

[0559] A feature extraction unit 601 is configured to extract features from a new defective product image and identify known defects similar to the new defect as candidate defects.

[0560] A model acquisition unit 602 is configured to acquire a candidate defect detection model for detecting whether an input product image has the candidate defect;

[0561] The transfer training unit 603 is used to perform transfer training on the obtained candidate defect detection model using the new defective product image to obtain a new defect detection model for detecting whether the input product image has a new defect.

[0562] The detailed explanation of the embodiment of the present device can be found in the above method embodiment.

[0563] As shown in FIG8 , FIG8 is a structural diagram of a migration device for a type of detection model according to an embodiment of the present invention.

[0564] At least two known types and different type detection models corresponding to different known types are pre-set; the type detection model is used to detect whether the input image conforms to the corresponding known type;

[0565] The apparatus may include the following units.

[0566] The type acquisition unit 701 is used to obtain a pre-trained type feature extraction model and a type feature set;

[0567] The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity for different images of similar known types; the type feature set includes the known type features extracted by the type feature extraction model for images that meet the known type;

[0568] The new type unit 702 is configured to input an image that meets the new type into the type feature extraction model to obtain a new type feature; and in the type feature set, for known type features whose similarity to the new type feature meets a preset similarity condition, determine the corresponding known type as an alternative type;

[0569] The training unit 703 is used to perform migration training on the type detection model corresponding to the determined candidate type using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0570] The embodiment of the present invention does not limit the specific type of image matching.

[0571] Optionally, the type may include at least one of the following: whether the product in the image has a certain defect, whether the product in the image belongs to a certain product type, whether the item in the image belongs to a certain item type, whether the person in the image belongs to a certain crowd type, etc.

[0572] Optionally, the training method of the type feature extraction model includes:

[0573] The following training steps are executed repeatedly until the preset training stop condition is met:

[0574] Acquire multiple groups of different images that conform to the same known type, and determine each group of images as an image that conforms to a similar known type;

[0575] The acquired multiple groups of images are used to train the current type feature extraction model to improve the similarity of known type features extracted between images in the same group and / or to improve the difference of known type features extracted between images in different groups.

[0576] Optionally, the training step may further include:

[0577] Acquire images that match a known type;

[0578] Extracting known type features from the acquired image based on the current type feature extraction model;

[0579] Clustering the extracted known type features to obtain at least two similar type feature groups, and marking each acquired image with the similar type feature group to which the corresponding known type feature belongs;

[0580] The current type feature extraction model is trained based on the annotated images to improve the similarity of known type features extracted between images with the same annotation and / or to improve the difference of known type features extracted between images with different annotations.

[0581] Optionally, the current type feature extraction model is trained based on the annotated images, including:

[0582] The following steps are repeated until the first training stop condition is met:

[0583] Based on the current type feature extraction model, known type features are extracted from the annotated image to obtain a first feature set;

[0584] Based on a preset strategy, a known type feature is determined based on the known type features corresponding to each annotation to obtain a second feature set;

[0585] The loss is calculated based on the obtained first feature set and second feature set, and the current type feature extraction model is updated.

[0586] Optionally, the current type feature extraction model is trained based on the annotated images, including:

[0587] Randomly select a known type feature from the known type features corresponding to each annotation and add it to the initial feature set; and add each feature in the initial feature set to the edge strategy feature set and the average strategy feature set respectively;

[0588] The following steps are executed repeatedly until the second training stop condition is met:

[0589] Based on the current type feature extraction model, known type features are extracted from the annotated image to obtain the full feature set;

[0590] Update the edge policy signature set based on the following steps:

[0591] For the target type feature corresponding to the target annotation in the edge strategy feature set, among the known type features corresponding to the target annotation in the full feature set, the known type feature with the lowest similarity to the target type feature is determined as the edge feature; the target type feature is updated based on the edge feature;

[0592] Update the average policy feature set based on the following steps:

[0593] For the target type feature corresponding to the target label in the average strategy feature set, calculate the mean for each known type feature corresponding to the target label in the full feature set, and update the target type feature based on the calculated mean;

[0594] Based on the updated edge strategy feature set and the updated average strategy feature set, the current initial feature set is updated;

[0595] Calculate the loss based on the updated initial feature set and the full feature set, and update the current type feature extraction model.

[0596] Optionally, the training unit 703 is configured to:

[0597] For each of the multiple type detection models corresponding to the multiple candidate types determined, the detection accuracy of each new type is determined using a test set; the test set contains new type images that meet the new type;

[0598] Select a type detection model whose detection accuracy meets the preset detection conditions, and use the acquired new type images for migration training.

[0599] Optionally, the training unit 703 is configured to:

[0600] For the multiple type detection models corresponding to the multiple candidate types determined one by one, transfer training is performed using the acquired new type images to obtain multiple candidate new type detection models;

[0601] For each candidate new type detection model, the detection accuracy of the new type is determined using the test set; the test set contains new type images that meet the new type;

[0602] The alternative new type detection model whose detection accuracy meets the preset accuracy condition is determined as the new type detection model.

[0603] The detailed explanation of the embodiment of the present device can be found in the above method embodiment.

[0604] As shown in Figure 9, Figure 9 is a schematic structural diagram of another type of detection model migration device according to an embodiment of the present invention. The device may include the following units.

[0605] The candidate type unit 801 is configured to determine, for images that meet the new type, known types similar to the new type as candidate types by extracting features;

[0606] The candidate model unit 802 is used to obtain a candidate type detection model for detecting whether the input image conforms to the candidate type;

[0607] The result unit 803 is used to perform migration training on the acquired candidate type detection model using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0608] The detailed explanation of the embodiment of the present device can be found in the above method embodiment.

[0609] An embodiment of the present invention further provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above method embodiments is implemented.

[0610] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned method embodiments.

[0611] FIG10 is a schematic diagram of the hardware structure of a computer device configured to implement the method according to an embodiment of the present invention, according to an embodiment of the present invention. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other within the device via the bus 1050.

[0612] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0613] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present invention are implemented through software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0614] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0615] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0616] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0617] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of the present invention, and does not necessarily include all the components shown in the figure.

[0618] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements any of the above method embodiments when executed by a processor.

[0619] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program implements any of the above method embodiments when executed by a processor.

[0620] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0621] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the embodiments of the present invention can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the embodiments of the present invention can essentially or in other words, the contributing part can be embodied in the form of a software product. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0622] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0623] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and when implementing the embodiment of the present invention, the functions of each module can be implemented in the same one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0624] The above description is only a specific implementation of the embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present invention. These improvements and modifications should also be regarded as protection for the embodiment of the present invention.

[0625] In the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.

[0626] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0627] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A defect detection model migration method, characterized in that: At least two known defects and different defect detection models corresponding to different known defects are preset; The defect detection model is used to detect whether the input product image has a corresponding known defect; The method comprises: Obtain a pre-trained defect feature extraction model and a defect feature set; The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity for different product images with similar known defects; the defect feature set includes the known defect features extracted by the defect feature extraction model for known defect product images with known defects; Inputting a new defective product image having a new defect into the defect feature extraction model to obtain a new defect feature; In the defect feature set, for known defect features whose similarity with the new defect feature meets a preset similarity condition, determining the corresponding known defect as a candidate defect; For the defect detection model corresponding to the determined candidate defect, migration training is performed using the acquired new defect product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

2. The method according to claim 1, characterized in that: The training method of the defect feature extraction model comprises: The following training steps are executed repeatedly until the preset training stop condition is met: Acquire multiple groups of different product images with the same known defects, and determine each group of product images as a product image with similar known defects; The current defect feature extraction model is trained using the acquired multiple groups of product images to improve the similarity of known defect features extracted between product images within a group and / or to improve the difference of known defect features extracted between product images of different groups.

3. The method according to claim 2, characterized in that The training step also includes: Acquire images of products with known defects; Extract known defect features from the acquired product image based on the current defect feature extraction model; Clustering the extracted known defect features to obtain at least two similar defect feature groups, and marking the similar defect feature groups to which the corresponding known defect features belong for each acquired product image; The current defect feature extraction model is trained based on the annotated product images to improve the similarity of known defect features extracted between product images with the same annotations, and / or to improve the difference of known defect features extracted between product images with different annotations.

4. The method according to claim 3, characterized in that The method of training the current defect feature extraction model based on the annotated product image includes: The following steps are repeated until the first training stop condition is met: Based on the current defect feature extraction model, known defect features are extracted from the annotated product image to obtain a first feature set; Based on the preset strategy, a known defect feature is determined based on the known defect features corresponding to each annotation, and a second feature set is obtained; The loss is calculated based on the obtained first feature set and the second feature set, and the current defect feature extraction model is updated.

5. The method according to claim 3, characterized in that: The method of training the current defect feature extraction model based on the annotated product image includes: From the known defect features corresponding to each annotation, a known defect feature is randomly selected and added to the initial feature set; and each feature in the initial feature set is added to the edge strategy feature set and the average strategy feature set respectively; The following steps are executed repeatedly until the second training stop condition is met: Based on the current defect feature extraction model, known defect features are extracted from the annotated product images to obtain a full feature set; The edge policy feature set is updated based on the following steps: For a target defect feature corresponding to a target annotation in the edge strategy feature set, among known defect features corresponding to the target annotation in the full feature set, determine the known defect feature with the lowest similarity to the target defect feature as an edge feature; and update the target defect feature based on the edge feature; The average policy feature set is updated based on the following steps: For the target defect feature corresponding to the target annotation in the average strategy feature set, for each known defect feature corresponding to the target annotation in the full feature set, calculate the mean, and update the target defect feature according to the calculated mean; Based on the updated edge strategy feature set and the updated average strategy feature set, update the current initial feature set; The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

6. The method according to claim 1, characterized in that The defect detection model corresponding to the determined candidate defect is transferred and trained by using the acquired new defect product image, including: For the multiple defect detection models corresponding to the multiple candidate defects determined one by one, the detection accuracy rates for the new defects are determined respectively using a test set; the test set includes a new defect product image with the new defect; Select a defect detection model whose detection accuracy meets the preset detection conditions, and use the acquired new defective product images for migration training.

7. The method according to claim 1, characterized in that The defect detection model corresponding to the determined candidate defect is transferred and trained by using the acquired new defect product image, including: For the multiple defect detection models corresponding to the multiple candidate defects determined one by one, transfer training is performed using the acquired new defect product image to obtain multiple candidate new defect detection models; For each candidate new defect detection model, a detection accuracy rate for the new defect is determined using a test set; the test set contains new defect product images with new defects; The candidate new defect detection model whose detection accuracy meets the preset accuracy condition is determined as the new defect detection model.

8. A defect detection model migration method, characterized in that: include: For a new defective product image having a new defect, known defects similar to the new defect are determined as candidate defects by means of feature extraction; Obtaining a candidate defect detection model for detecting whether an input product image has the candidate defect; For the obtained candidate defect detection model, migration training is performed using the new defective product image to obtain a new defect detection model for detecting whether the input product image has a new defect.

9. The method according to claim 8, characterized in that The method of determining known defects similar to the new defects as candidate defects for the new defect product image with the new defects by feature extraction includes: For new defective product images with new defects, new defect features are extracted by a preset method; For known defect features whose similarity with the new defect feature meets the preset similarity condition, the corresponding known defect is determined For alternative defects; Among them, the known defect features are extracted through the preset method.

10. A migration method for a type detection model, characterized in that: At least two known types and different type detection models corresponding to different known types are preset; the type detection model is used to detect whether the input image conforms to the corresponding known type; The method comprises: Obtain a pre-trained type feature extraction model and a type feature set; The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity for different images of similar known types; the type feature set includes the known type features extracted by the type feature extraction model for images that conform to the known type; Inputting an image that conforms to the new type into the type feature extraction model to obtain new type features; In the type feature set, for known type features whose similarity with the new type feature meets a preset similarity condition, determining the corresponding known type as a candidate type; For the type detection model corresponding to the determined candidate type, migration training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

11. A migration method for a type detection model, characterized in that: include: For images that conform to the new type, known types similar to the new type are determined as candidate types by extracting features; Obtaining an alternative type detection model for detecting whether an input image conforms to the alternative type; For the obtained candidate type detection model, transfer training is performed using images that conform to the new type to obtain a new type detection model for detecting whether an input image conforms to the new type.

12. The method according to claim 11, characterized in that The method of determining known types similar to the new type as candidate types by extracting features for the image that meets the new type includes: For images that meet the new type, extract new type features through a preset method; For known type features whose similarity with the new type features meets a preset similarity condition, determining the corresponding known type as a candidate type; Among them, the known type features are extracted through the preset method.

13. A defect detection model migration device, characterized in that: At least two known defects and different defect detection models corresponding to different known defects are preset; The defect detection model is used to detect whether the input product image has a corresponding known defect; The device comprises: An acquisition unit, used for acquiring a pre-trained defect feature extraction model and a defect feature set; The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity for different product images with similar known defects; the defect feature set includes the known defect features extracted by the defect feature extraction model for known defect product images with known defects; A similarity unit is used to input a new defect product image with a new defect into the defect feature extraction model to obtain a new defect feature; in the defect feature set, for a known defect feature whose similarity with the new defect feature meets a preset similarity condition, determine the corresponding known defect as a candidate defect; The migration unit is used to perform migration training on the defect detection model corresponding to the determined candidate defect using the acquired new defect product image to obtain a new defect detection model for detecting whether the input product image has the new defect.

14. A defect detection model migration device, characterized in that: include: A feature extraction unit is used to determine known defects similar to the new defect as candidate defects by extracting features from the new defective product image having the new defect; A model acquisition unit, used to acquire a candidate defect detection model for detecting whether an input product image has the candidate defect; The transfer training unit is used to perform transfer training on the acquired candidate defect detection model using the new defect product image to obtain a new defect detection model for detecting whether the input product image has a new defect.

15. A migration device for a type detection model, characterized in that: At least two known types and different type detection models corresponding to different known types are preset; the type detection model is used to detect whether the input image conforms to the corresponding known type; The device comprises: A type acquisition unit, used to acquire a pre-trained type feature extraction model and a type feature set; The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity for different images of similar known types; the type feature set includes the known type features extracted by the type feature extraction model for images that conform to the known type; A new type unit is used to input an image that meets the new type into the type feature extraction model to obtain a new type feature; in the type feature set, for a known type feature whose similarity with the new type feature meets a preset similarity condition, the corresponding known type is determined as an alternative type; The training unit is used to perform migration training on the type detection model corresponding to the determined candidate type using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

16. A migration device for a type detection model, characterized in that: include: A candidate type unit, for determining, for images that conform to the new type, known types similar to the new type as candidate types by extracting features; An alternative model unit, used for obtaining an alternative type detection model for detecting whether an input image conforms to the alternative type; The result unit is used to perform migration training on the acquired candidate type detection model using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

17. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 12.

18. A computer-readable storage medium storing a computer program, characterized in that: The computer program implements the method according to any one of claims 1 to 12 when executed by a processor.

Images