A method, apparatus, device, and storage medium for transferring a defect detection model.

Abstract

This invention discloses a method, apparatus, device, and storage medium for transferring a defect detection model. The method includes: acquiring a pre-trained defect feature extraction model and a defect feature set; the defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects; the defect feature set includes the known defect features extracted by the defect feature extraction model; inputting a new defective product image with a new defect into the defect feature extraction model to obtain new defect features; in the defect feature set, identifying known defect features whose similarity to the new defect features meets a preset similarity condition as candidate defects; and performing transfer training on the defect detection model corresponding to the identified candidate defects using the new defective product image to obtain a new defect detection model for detecting whether the input product image has a new defect.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for transferring a defect detection model. Background Technology

[0002] Image processing technology is currently quite mature, with numerous detection models available for determining whether an image conforms to a specific type. For example, product images can be captured and image recognition technology used to detect product defects. Specifically, different defect detection models can be set up for different defects to detect whether the product in the image possesses a particular defect, facilitating subsequent recycling and repair. Similarly, product images can be captured and image recognition technology used to detect product types. Different product type detection models can be set up for different product types to detect whether the product in the image belongs to a specific product type, facilitating subsequent sorting.

[0003] As demand increases, the types of tests required also continue to increase.

[0004] For example, during the production or transportation of a product, the types of product defects are often inexhaustible, and new product defects may be discovered or generated at any time, thus requiring the training of models to detect defects in new products.

[0005] Existing methods often involve collecting new types of image samples for training, but sample collection is difficult and time-consuming, resulting in low generation efficiency for models that detect new types. For example, models for detecting new defects have low generation efficiency. Summary of the Invention

[0006] This invention provides a method, apparatus, device, and storage medium for migrating a defect detection model to address the shortcomings of related technologies.

[0007] According to a first aspect of the present invention, a method for transferring a defect detection model is provided, wherein at least two known defects are pre-set, and different defect detection models corresponding one-to-one with the different known defects are provided; the defect detection model is used to detect whether an input product image has a corresponding known defect;

[0008] The method includes:

[0009] Obtain a pre-trained defect feature extraction model and a set of defect features;

[0010] The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects; the defect feature set includes known defect features extracted by the defect feature extraction model from known defect product images with known defects.

[0011] The image of the product with the new defect is input into the defect feature extraction model to obtain the new defect features;

[0012] In the set of defect features, for known defect features whose similarity to the new defect feature meets the preset similarity condition, the corresponding known defects are determined as candidate defects;

[0013] For the defect detection model corresponding to the identified candidate defects, the new defect product image is used for transfer training to obtain a new defect detection model for detecting whether the input product image has a new defect.

[0014] Optionally, the training method for the defect feature extraction model includes:

[0015] Repeat the following training steps until the preset training stopping condition is met:

[0016] Acquire multiple sets of different product images with the same known defect, and identify each set of product images as product images with similar known defects;

[0017] The current defect feature extraction model is trained using the acquired multiple sets of product images to improve the similarity of known defect features extracted between product images within the same group, and / or improve the difference of known defect features extracted between product images from different groups.

[0018] Optionally, the training step further includes:

[0019] Obtain product images with known defects;

[0020] Based on the current defect feature extraction model, known defect features are extracted from the acquired product images;

[0021] Cluster the extracted known defect features to obtain at least two similar defect feature groups, and label the similar defect feature group to which the corresponding known defect feature belongs for each product image acquired.

[0022] The current defect feature extraction model is trained based on the labeled product images to improve the similarity of known defect features extracted between product images with the same label, and / or improve the difference of known defect features extracted between product images with different labels.

[0023] Optionally, training the current defect feature extraction model based on the labeled product image includes:

[0024] Repeat the following steps until the first training stopping condition is met:

[0025] Based on the current defect feature extraction model, known defect features are extracted from the labeled product image to obtain the first feature set;

[0026] Based on a preset strategy, a known defect feature is determined based on the known defect features of each corresponding annotation, thus obtaining a second feature set;

[0027] The loss is calculated based on the obtained first and second feature sets, and the current defect feature extraction model is updated.

[0028] Optionally, training the current defect feature extraction model based on the labeled product image includes:

[0029] From the known defect features corresponding to each annotation, randomly select one known defect feature and add it to the initial feature set; then add each feature in the initial feature set to the edge policy feature set and the average policy feature set, respectively.

[0030] Repeat the following steps until the second training stopping condition is met:

[0031] Based on the current defect feature extraction model, known defect features are extracted from the labeled product images to obtain the full feature set;

[0032] The edge policy feature set is updated based on the following steps:

[0033] For the target defect features corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect feature with the lowest similarity to the target defect feature is determined as the edge feature; the target defect feature is updated based on the edge feature;

[0034] The average policy feature set is updated based on the following steps:

[0035] For the target defect features corresponding to the target annotation in the average strategy feature set, calculate the mean for each known defect feature corresponding to the target annotation in the full feature set, and update the target defect features based on the calculated mean.

[0036] Update the current initial feature set based on the updated edge policy feature set and the updated average policy feature set;

[0037] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0038] Optionally, the defect detection model corresponding to the determined candidate defects is transferred and trained using the acquired new defective product image, including:

[0039] For each of the identified candidate defects, the detection accuracy for the new defect is determined using a test set; the test set includes images of the new defective product.

[0040] Select a defect detection model whose detection accuracy meets the preset detection conditions, and perform transfer training using the acquired new defective product images.

[0041] Optionally, the defect detection model corresponding to the determined candidate defects is transferred and trained using the acquired new defective product image, including:

[0042] For the multiple defect detection models corresponding to the identified candidate defects, transfer training is performed using the acquired images of the new defective products to obtain multiple new candidate defect detection models.

[0043] For each candidate new defect detection model, the detection accuracy for the new defect is determined using a test set; the test set contains images of the new defective product.

[0044] The candidate new defect detection models whose detection accuracy meets the preset accuracy conditions are determined as new defect detection models.

[0045] According to a second aspect of the present invention, a method for transferring a defect detection model is provided, comprising:

[0046] For a product image with a new defect, a feature extraction method is used to identify known defects similar to the new defect as candidate defects.

[0047] Obtain a candidate defect detection model for detecting whether the input product image has the candidate defects;

[0048] For the acquired candidate defect detection models, transfer training is performed using new defective product images to obtain a new defect detection model for detecting whether the input product image has new defects.

[0049] According to a third aspect of the present invention, a transfer method for a type detection model is provided, wherein at least two known types are pre-set, and different type detection models corresponding one-to-one with the different known types; the type detection model is used to detect whether an input image conforms to the corresponding known type;

[0050] The method includes:

[0051] Obtain a pre-trained type feature extraction model and a set of type features;

[0052] The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity from different images of similar known types; the type feature set includes known type features extracted by the type feature extraction model for images that conform to known types;

[0053] The image that matches the new type is input into the type feature extraction model to obtain the new type features;

[0054] In the set of type features, for known type features whose similarity to the new type feature meets the preset similarity condition, the corresponding known type is determined as a candidate type;

[0055] For the type detection model corresponding to the identified candidate types, transfer training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0056] According to a fourth aspect of the present invention, a transfer method for a type detection model is provided, comprising:

[0057] For images that conform to the new type, known types similar to the new type are identified as candidate types through feature extraction.

[0058] Obtain a candidate type detection model for detecting whether the input image conforms to the candidate type;

[0059] For the obtained candidate type detection models, transfer training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0060] According to a fifth aspect of the present invention, a defect detection model transfer apparatus is provided, wherein at least two known defects are pre-set, and different defect detection models corresponding one-to-one with the different known defects; the defect detection model is used to detect whether an input product image has a corresponding known defect; the apparatus includes:

[0061] The acquisition unit is used to acquire the pre-trained defect feature extraction model and the defect feature set;

[0062] The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects; the defect feature set includes known defect features extracted by the defect feature extraction model from known defect product images with known defects.

[0063] The similarity unit is used to input the image of the product with the new defect into the defect feature extraction model to obtain the new defect feature; in the defect feature set, for the known defect features whose similarity with the new defect feature meets the preset similarity condition, the corresponding known defects are determined as candidate defects;

[0064] The transfer unit is used to train the defect detection model corresponding to the identified candidate defects using the acquired new defect product image, so as to obtain a new defect detection model for detecting whether the input product image has a new defect.

[0065] Optionally, the training method for the defect feature extraction model includes:

[0066] Repeat the following training steps until the preset training stopping condition is met:

[0067] Acquire multiple sets of different product images with the same known defect, and identify each set of product images as product images with similar known defects;

[0068] The current defect feature extraction model is trained using the acquired multiple sets of product images to improve the similarity of known defect features extracted between product images within the same group, and / or improve the difference of known defect features extracted between product images from different groups.

[0069] Optionally, the training step further includes:

[0070] Obtain product images with known defects;

[0071] Based on the current defect feature extraction model, known defect features are extracted from the acquired product images;

[0072] Cluster the extracted known defect features to obtain at least two similar defect feature groups, and label the similar defect feature group to which the corresponding known defect feature belongs for each product image acquired.

[0073] The current defect feature extraction model is trained based on the labeled product images to improve the similarity of known defect features extracted between product images with the same label, and / or improve the difference of known defect features extracted between product images with different labels.

[0074] Optionally, training the current defect feature extraction model based on the labeled product image includes:

[0075] Repeat the following steps until the first training stopping condition is met:

[0076] Based on the current defect feature extraction model, known defect features are extracted from the labeled product image to obtain the first feature set;

[0077] Based on a preset strategy, a known defect feature is determined based on the known defect features of each corresponding annotation, thus obtaining a second feature set;

[0078] The loss is calculated based on the obtained first and second feature sets, and the current defect feature extraction model is updated.

[0079] Optionally, training the current defect feature extraction model based on the labeled product image includes:

[0080] From the known defect features corresponding to each annotation, randomly select one known defect feature and add it to the initial feature set; then add each feature in the initial feature set to the edge policy feature set and the average policy feature set, respectively.

[0081] Repeat the following steps until the second training stopping condition is met:

[0082] Based on the current defect feature extraction model, known defect features are extracted from the labeled product images to obtain the full feature set;

[0083] The edge policy feature set is updated based on the following steps:

[0084] For the target defect features corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect feature with the lowest similarity to the target defect feature is determined as the edge feature; the target defect feature is updated based on the edge feature;

[0085] The average policy feature set is updated based on the following steps:

[0086] For the target defect features corresponding to the target annotation in the average strategy feature set, calculate the mean for each known defect feature corresponding to the target annotation in the full feature set, and update the target defect features based on the calculated mean.

[0087] Update the current initial feature set based on the updated edge policy feature set and the updated average policy feature set;

[0088] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0089] Optionally, the migration unit is used for:

[0090] For each of the identified candidate defects, the detection accuracy for the new defect is determined using a test set; the test set includes images of the new defective product.

[0091] Select a defect detection model whose detection accuracy meets the preset detection conditions, and perform transfer training using the acquired new defective product images.

[0092] Optionally, the migration unit is used for:

[0093] For the multiple defect detection models corresponding to the identified candidate defects, transfer training is performed using the acquired images of the new defective products to obtain multiple new candidate defect detection models.

[0094] For each candidate new defect detection model, the detection accuracy for the new defect is determined using a test set; the test set contains images of the new defective product.

[0095] The candidate new defect detection models whose detection accuracy meets the preset accuracy conditions are determined as new defect detection models.

[0096] According to a sixth aspect of the present invention, a defect detection model transfer apparatus is provided, comprising:

[0097] The feature extraction unit is used to identify known defects similar to the new defect as candidate defects for a new defect product image with a new defect by feature extraction.

[0098] The model acquisition unit is used to acquire a candidate defect detection model for detecting whether the input product image has the candidate defect;

[0099] The transfer training unit is used to transfer train the acquired candidate defect detection models using images of new defective products, so as to obtain a new defect detection model for detecting whether the input product image has a new defect.

[0100] According to a seventh aspect of the present invention, a type detection model transfer apparatus is provided, wherein at least two known types are pre-set, and different type detection models corresponding one-to-one with the different known types; the type detection model is used to detect whether an input image conforms to the corresponding known type;

[0101] The device includes:

[0102] The type acquisition unit is used to acquire a pre-trained type feature extraction model and a set of type features;

[0103] The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity from different images of similar known types; the type feature set includes known type features extracted by the type feature extraction model for images that conform to known types;

[0104] The new type unit is used to input images that conform to the new type into the type feature extraction model to obtain new type features; in the type feature set, for known type features whose similarity to the new type features meets the preset similarity conditions, the corresponding known types are determined as candidate types;

[0105] The training unit is used to transfer train the type detection model corresponding to the determined candidate types using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0106] According to an eighth aspect of the present invention, a type detection model transfer apparatus is provided, comprising:

[0107] The candidate type unit is used to identify known types similar to the new type as candidate types for images that conform to the new type through feature extraction.

[0108] The alternative model unit is used to obtain an alternative type detection model for detecting whether the input image conforms to the alternative type.

[0109] The result unit is used to transfer train the acquired candidate type detection model using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0110] As can be seen from the above embodiments, by identifying known types similar to the new type and performing transfer training based on the type detection model that detects the identified known types to obtain a new type detection model for detecting the new type, the generation efficiency of the new type detection model for detecting the new type can be improved. Specifically, when the image type is a defect, the generation efficiency of the new defect detection model for detecting new defects can be improved.

[0111] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0112] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0113] Figure 1This is a flowchart illustrating a method for migrating a defect detection model according to an embodiment of the present invention;

[0114] Figure 2 This is a schematic diagram illustrating the principle of defect detection model transfer according to an embodiment of the present invention;

[0115] Figure 3 This is a flowchart illustrating another method for migrating a defect detection model according to an embodiment of the present invention;

[0116] Figure 4 This is a flowchart illustrating a type detection model transfer method according to an embodiment of the present invention;

[0117] Figure 5 This is a flowchart illustrating a transfer method for another type of detection model according to an embodiment of the present invention;

[0118] Figure 6 This is a schematic diagram of the structure of a migration device for a defect detection model according to an embodiment of the present invention;

[0119] Figure 7 This is a schematic diagram of the structure of a migration device for another defect detection model according to an embodiment of the present invention;

[0120] Figure 8 This is a schematic diagram of the structure of a type detection model transfer device according to an embodiment of the present invention;

[0121] Figure 9 This is a schematic diagram of the structure of a migration device for another type of detection model according to an embodiment of the present invention;

[0122] Figure 10 This is a schematic diagram of the hardware structure of a computer device configured according to an embodiment of the present invention. Detailed Implementation

[0123] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0125] Currently, image processing technology is relatively mature, and there are many detection models available for detecting whether an image conforms to a certain type.

[0126] For example, product defects can be detected by taking product images and using image recognition technology. Specifically, different defect detection models can be set up for different defects to detect whether the product in the image has a certain defect, which facilitates subsequent recycling and repair.

[0127] Alternatively, product images can be captured and product type detection can be performed using image recognition technology. Specifically, different product type detection models can be set up for different product types to detect whether the product in the image belongs to a certain product type, which facilitates subsequent sorting.

[0128] As demand increases, the types of tests required also continue to increase.

[0129] For example, during the production or transportation of a product, the types of product defects are often inexhaustible, and new product defects may be discovered or generated at any time, thus requiring the training of models to detect defects in new products.

[0130] Existing methods often involve collecting new types of image samples for training, but this is difficult and time-consuming, and the generation efficiency of models for detecting new types is also low.

[0131] Taking defect detection as an example, the existing methods often involve re-collecting samples of defects in new products for training. However, sample collection is difficult and time-consuming, and the generation efficiency of models for detecting defects in new products is also low.

[0132] The following explanation will begin with a defect detection scenario.

[0133] Among them, product defects in an image can be an example of an image type. For example, if the product in the image has a broken defect, the existence of a "broken" product defect can be an image type, that is, the product in the image has a broken defect.

[0134] This invention discloses a method for transferring a defect detection model. By using model transfer learning, an existing defect detection model is used, and an existing defect similar to the new defect is selected. Then, a defect detection model for detecting the selected existing defect is chosen and transferred. This allows for the use of a smaller number of new defect samples for transfer learning, quickly obtaining a defect detection model for detecting new defects and improving the efficiency of model generation for determining the detection of new products.

[0135] However, it is often difficult to determine the similarity of product defects.

[0136] Therefore, in this method, a model for measuring the similarity of product defects can be pre-trained. Specifically, the product defect image can be mapped to a feature space through this model. The features corresponding to similar defects are close in distance in this feature space, which makes it easier to identify existing defects similar to the new defect.

[0137] The following is a detailed explanation of a method for transferring a defect detection model provided by an embodiment of the present invention.

[0138] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a method for migrating a defect detection model according to an embodiment of the present invention.

[0139] Alternatively, the defect here could be a defect in the product shown in the image. For example, during screen assembly, a screen breakage defect might occur; during housing assembly, a housing breakage defect might occur, and so on.

[0140] The embodiments of this invention do not limit the executing entity of this method. Optionally, the executing entity can be any computing device. For example, a server or client used to detect product defects.

[0141] At least two known defects can be pre-set, as well as different defect detection models corresponding to each known defect; the defect detection model can be used to detect whether the input product image has the corresponding known defect.

[0142] Optionally, different known defects can correspond one-to-one with different defect detection models, each known defect can correspond to one defect detection model, and each defect detection model can correspond to one known defect.

[0143] The method may include the following steps.

[0144] S101: Obtain the pre-trained defect feature extraction model and the defect feature set.

[0145] Among them, the defect feature extraction model can be used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects.

[0146] The defect feature set may include known defect features extracted by the defect feature extraction model for known defect product images with known defects.

[0147] S102: Input the image of the product with the new defect into the defect feature extraction model to obtain the new defect features.

[0148] S103: In the defect feature set, for known defect features whose similarity to new defect features meets the preset similarity conditions, the corresponding known defects are identified as candidate defects.

[0149] S104: For the defect detection model corresponding to the identified candidate defects, transfer training is performed using the acquired new defect product image to obtain a new defect detection model for detecting whether the input product image has a new defect.

[0150] The above method limits the image type to defects in product images. By identifying known types similar to the new type and performing transfer training based on the type detection model that detects the identified known types, a new type detection model for detecting the new type can be obtained, which can improve the generation efficiency of the new type detection model for detecting the new type.

[0151] In particular, when the image type is specifically a defect, the generation efficiency of a new defect detection model can be improved to detect new defects.

[0152] Specifically, by identifying known defects similar to new defects based on the similarity between defect features, and then performing transfer training based on the defect detection model that detected the known defects, the generation efficiency of new defect detection models can be improved.

[0153] The following is a detailed explanation of each aspect.

[0154] I. The structure and function of the defect feature extraction model.

[0155] 1. Model structure.

[0156] This method does not limit the structure of the defect feature extraction model.

[0157] Alternatively, the defect feature extraction model can be a graph convolutional network, graph neural network, or other model that extracts image features.

[0158] Alternatively, the defect feature extraction model can be a residual neural network, ResNet.

[0159] 2. The role of the model.

[0160] The defect feature extraction model can be used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects.

[0161] Optionally, the defect feature extraction model can be used to map the input product image into a feature space, that is, to extract the defect features in the product image.

[0162] Among them, different product images with similar known defects can be mapped by the defect feature extraction model to obtain different defect features that are close in distance. This makes it convenient to determine whether the defects are similar based on the distance between the defect features mapped by the defect feature extraction model.

[0163] Optionally, the distance between defect features mapped by the defect feature extraction model can be used to determine the similarity between defects for subsequent business analysis.

[0164] Regarding similar defects, this method does not limit the specific way of determining them.

[0165] Optionally, similar defects can be preset. Specifically, different defects that are similar to the corresponding defective product images can be preset as similar defects.

[0166] For example, product images with "screen smudge defects" and "screen damage defects" often have similarities in the images, so they can be set as similar defects. These two types of defective product images can be used as images with similar defects, and the defect feature extraction model can extract similar defect features.

[0167] Alternatively, similar defects can be discovered during model training. Specifically, during the iterative training of the defect feature extraction model, different defects corresponding to different defect features that are close in distance can be identified as similar defects, thereby enabling the discovery of similar defects based on data analysis.

[0168] II. Training methods for the defect feature extraction model.

[0169] This method does not limit the training method of the defect feature extraction model.

[0170] Optionally, supervised training, unsupervised training, or a combination of supervised and unsupervised training can be used.

[0171] Optionally, the defect feature extraction model can be trained directly, or an output layer can be added to the defect feature extraction model so that the output layer can predict the defect features extracted by the defect feature extraction model. In this way, the defect feature extraction model and the output layer can be trained together.

[0172] For example, a defect classification output layer can be added to the defect feature extraction model. This allows the defect classification output layer to predict the corresponding defects based on the defect features extracted by the defect feature extraction model. Furthermore, the defect feature extraction model and the defect classification output layer can be trained together. Specifically, this can be supervised training.

[0173] This method does not limit the source of the training sample set for training the defect feature extraction model.

[0174] Alternatively, it can be to acquire product images during the factory production process or to acquire images of defective products.

[0175] The occurrence of defects can be related to specific stages in the product manufacturing process. For example, screen breakage may occur during screen assembly; casing breakage may occur during casing installation, and so on.

[0176] To facilitate understanding, a specific example is provided in this methodology. Sample images of defective products can be collected based on an AI platform.

[0177] Among them, the AOI (Automatic Optical Inspection) equipment on the product production line can interact with the platform to achieve an automated defect data detection and feedback mechanism. AOI equipment is an important inspection tool and process quality control tool for ensuring product quality in the electronics manufacturing industry. During automatic inspection, the AOI inspection equipment automatically scans panel products or other industrial products with a high-definition camera and collects images, which are then used for defect detection and location in subsequent stages.

[0178] For real-time uploads from production line equipment, the AI ​​platform performs initial classification and storage based on parameters such as the product's corresponding ID, process segment, and defect type (what kind of defect).

[0179] In one alternative embodiment, during a training iteration of the defect feature extraction model, supervised training, unsupervised training, or a combination of supervised and unsupervised training can be performed.

[0180] The following sections will explain supervised training and unsupervised training respectively.

[0181] 1. Supervised training.

[0182] This method does not limit the specific supervised training method.

[0183] Optionally, multiple defective product images labeled with similar defects can be acquired and used as supervised training samples for training.

[0184] Optionally, training samples can be obtained with images of different defective products as features and similar defects as labels, and supervised training can be performed on the defect feature extraction model.

[0185] Optionally, images of different defective products labeled with dissimilar defects can be obtained and used as supervised training samples for training. Specifically, the defect features extracted from images of products with dissimilar defects can be updated through training to reduce similarity or increase dissimilarity.

[0186] Alternatively, training can be performed by combining the two types of training samples mentioned above.

[0187] This method does not limit the method of calculating loss in supervised training.

[0188] Optionally, the loss from supervised training can be used to characterize the similarity of extracted defect features between different product images with similar defects, thereby allowing the model to be updated to increase the similarity of defect features for similar defects.

[0189] Optionally, the supervised training loss can be used to characterize the difference in extracted defect features between different product images with dissimilar defects, thereby allowing the model to be updated to increase the difference in defect features between similar defects.

[0190] Alternatively, the update can be performed by combining the two types of losses mentioned above.

[0191] In one alternative embodiment, supervised training can be performed by combining pre-defined similar defect labels.

[0192] Optionally, the training method for the defect feature extraction model may include: repeatedly executing the following training steps until a preset training stopping condition is met:

[0193] Acquire multiple sets of different product images with similar known defects; use the acquired multiple sets of product images to train the current defect feature extraction model to improve the similarity of known defect features extracted between product images within a group, and / or improve the difference of known defect features extracted between product images from different groups.

[0194] Optionally, different groups of product images may not have similar defects, which can facilitate improving the difference in known defect features extracted between different groups of product images.

[0195] Alternatively, if only the similarity of known defect features extracted between product images within a group is increased, then product images from different groups can also have similar defects.

[0196] This embodiment can combine labeled samples to train the defect feature extraction model, thereby improving the training effect of the model and the accuracy of similar defect identification.

[0197] This embodiment does not limit the specific form of the known defect feature similarity and the known defect feature difference.

[0198] Optionally, the similarity and difference of known defect features can be measured using feature distance in feature space. Specifically, this can involve reducing the distance in feature space between known defect features extracted from product images within the same group, and / or increasing the distance in feature space between known defect features extracted from product images from different groups.

[0199] Optionally, images of different products with the same known defects can also be regarded as images of products with similar known defects, thereby facilitating the extraction of more accurate defect features.

[0200] Optionally, the training method for the defect feature extraction model may include: repeatedly executing the following training steps until a preset training stopping condition is met:

[0201] Acquire multiple sets of different product images with the same known defects, and identify each set of product images as having similar known defects; use the acquired multiple sets of product images to train the current defect feature extraction model to improve the similarity of known defect features extracted between product images within a group, and / or improve the difference of known defect features extracted between product images from different groups.

[0202] This embodiment does not limit the preset training stopping condition. Optionally, the preset training stopping condition may be at least one of the following: the number of training cycles reaches a preset number, the similarity of known defects within a group is greater than a preset similarity threshold, and the difference of known defects between groups is greater than a preset difference threshold, etc.

[0203] Known defects of the same kind can be classified as similar defects, and therefore can be used as labeled samples for supervised training.

[0204] This embodiment can use product images with the same known defects as labeled samples for training, which improves training efficiency and reduces the difficulty of sample collection.

[0205] Different groups of product images may have different known defects.

[0206] This embodiment does not limit the specific form of the known defect feature similarity and the known defect feature difference.

[0207] Optionally, the similarity and difference of known defect features can be measured using feature distance in feature space. Specifically, this can involve reducing the distance in feature space between known defect features extracted from product images within the same group, and / or increasing the distance in feature space between known defect features extracted from product images from different groups.

[0208] This method does not limit the method of calculating loss in supervised training.

[0209] Optionally, for supervised training using multiple sets of different product images with the same known defects as training samples, a classification output layer can be added to the defect feature extraction model to predict the defect type.

[0210] Furthermore, the loss can characterize whether the same correct defect is predicted for any set of product images.

[0211] This embodiment can combine product images with the same known defects and use an additional classification output layer to predict the defect type, thereby training a defect feature extraction model.

[0212] For ease of understanding, specific examples are also provided in the embodiments of the present invention.

[0213] In one example, intra-class distance optimization can be used to optimize the defect feature extraction model (ResNet50 architecture), which means using the training dataset and training with intra-class distance optimization and positive sample classification.

[0214] For intra-class distance optimization, a custom distance function can be used to calculate the distance between extracted defect features for different product images with the same known defects in each group.

[0215] Based on the above distance function, the distance between defect features within a group that have the same known defects can be made as close as possible.

[0216] For positive sample classification, a new classification output layer can be added to predict defect types, ensuring that product image samples within the same group with the same known defect are classified as having the same defect. The higher the probability that the output defect predictions are correct and identical, the smaller the loss.

[0217] 2. Unsupervised training.

[0218] This method does not limit the specific unsupervised training method.

[0219] Optionally, clustering can be performed on the initially extracted defect features, and the defects corresponding to the defect features of each cluster can be identified as similar defects and labeled to obtain product image samples labeled with pseudo-labels.

[0220] Optionally, training can be performed based on pseudo-labels. Specifically, this can be done by referring to supervised training methods.

[0221] Alternatively, training can be combined with sampling methods, as explained below.

[0222] In one alternative embodiment, the defect feature extraction model can be trained unsupervised.

[0223] Optionally, the training method for the defect feature extraction model includes: repeatedly executing the following training steps until a preset training stopping condition is met:

[0224] Acquire product images with known defects; extract known defect features from the acquired product images based on the current defect feature extraction model; cluster the extracted known defect features to obtain at least two similar defect feature groups, and label the similar defect feature groups to which the corresponding known defect features belong for each acquired product image; train the current defect feature extraction model based on the labeled product images to improve the similarity of known defect features extracted between product images with the same label, and / or improve the difference of known defect features extracted between product images with different labels.

[0225] Among them, the labeled similar defect feature groups can be regarded as pseudo-labels.

[0226] This embodiment does not limit the clustering algorithm. Optionally, density-based clustering (Density-Based Spatial Clustering of Applications with Noise, DBSCAN), mean-shift clustering, or kmems can be used for clustering.

[0227] This embodiment does not limit the specific method of training the current defect feature extraction model based on the labeled product images. Optionally, it can be trained in a supervised manner.

[0228] Optionally, for each labeled group, some specific known defect features can be sampled separately. Then, the loss can be determined by comparing similar defect feature groups and the similarity between the sampling results.

[0229] To facilitate understanding, in a specific example, the similarity can be improved by continuously reducing the distance between known defect features within each labeled group. Therefore, this can be achieved by sampling the known defect feature that is furthest from the center within each labeled group, or the average value of features within each labeled group, or the weighted average value of features within each labeled group, and then updating the data using a loss function to continuously reduce the distance between features within similar defect feature groups.

[0230] Optionally, training the current defect feature extraction model based on the labeled product image may include: repeatedly executing the following steps until a first training stopping condition is met:

[0231] Based on the current defect feature extraction model, known defect features are extracted from the labeled product image to obtain the first feature set; based on the preset strategy, a known defect feature is determined based on the known defect features corresponding to each label to obtain the second feature set; the loss is calculated based on the obtained first and second feature sets, and the current defect feature extraction model is updated.

[0232] In this embodiment, the first training stopping condition is not limited. Optionally, the first training stopping condition may be at least one of the following: the number of training loops reaches a preset number, and the loss is less than a preset loss threshold, etc.

[0233] This embodiment does not limit the preset strategy.

[0234] Optionally, the average value of the known defect features can be determined based on the known defect features of each corresponding label, and then combined into a second feature set. That is, for each label, the average value of the known defect features under each label can be combined into a second feature set.

[0235] Optionally, the second feature set can be formed by determining the known defect features farthest from the cluster center based on the known defect features corresponding to each label, and combining them. That is, for each label, the known defect features farthest from the cluster center under each label are combined into the second feature set. Alternatively, the second feature set can be formed by determining the known defect features with the lowest similarity to the cluster center.

[0236] Different combinations of loop operations can yield different second feature sets, as well as different first feature sets.

[0237] Alternatively, one or more known defect features can be randomly selected based on the known defect features of each corresponding label and combined into a second feature set.

[0238] Optionally, since the above steps are executed cyclically, the known defect feature furthest from the preset feature can be determined based on the known defect features corresponding to each label; the preset feature can be the feature corresponding to the same label in the second feature set obtained in the previous cycle combination.

[0239] This embodiment does not limit the method of calculating the loss between the first feature set and the second feature set.

[0240] Optionally, the loss between the first feature set and the second feature set can be used to characterize the similarity between the first feature set and the second feature set.

[0241] Understandably, in one example, if the known defect features furthest from the cluster center are extracted and combined into a second feature set, and the second feature set is still highly similar to the first feature set, it means that the known defect features under the same label are highly similar. Thus, a defect feature extraction model can be obtained that maps different product images with similar known defects to similar defect features.

[0242] Optionally, the loss between the first feature set and the second feature set can be calculated using cross-entropy or contrastive learning loss (infoNCE).

[0243] For ease of understanding, an optional embodiment is given below.

[0244] Optionally, training the current defect feature extraction model based on the labeled product image may include the following steps:

[0245] From the known defect features corresponding to each annotation, randomly select one known defect feature and add it to the initial feature set; then add each feature in the initial feature set to the edge policy feature set and the average policy feature set, respectively.

[0246] Repeat the following steps until the second training stopping condition is met:

[0247] Based on the current defect feature extraction model, known defect features are extracted from the labeled product images to obtain the full feature set;

[0248] The edge policy feature set is updated based on the following steps:

[0249] For the target defect features corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect features with the lowest similarity to the target defect features are identified as edge features; the target defect features are updated based on the edge features.

[0250] The average policy feature set is updated based on the following steps:

[0251] For the target defect features corresponding to the target annotation in the average strategy feature set, calculate the mean for each known defect feature corresponding to the target annotation in the full feature set, and update the target defect features based on the calculated mean.

[0252] Update the current initial feature set based on the updated edge policy feature set and the updated average policy feature set;

[0253] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0254] This embodiment does not limit the number of labeled product images used in obtaining the full feature set. Optionally, multiple labeled product images can be obtained. To facilitate subsequent operations, multiple product images labeled with different pseudo-labels can be obtained.

[0255] Optionally, for each labeled pseudo-label, the same number of product images can be obtained separately to extract known defect feature combinations to obtain the full feature set.

[0256] The full feature set obtained in different iterations may not be the same.

[0257] This embodiment does not limit the second training stopping condition. Optionally, the second training stopping condition may be at least one of the following: the number of training loops reaches a preset number, and the loss is less than a preset loss threshold.

[0258] This embodiment enables unsupervised training of the defect feature extraction model by setting an initial feature set, an edge strategy feature set, and an average strategy feature set. For similar defect feature groups under a single annotation, the known defect features within the group can be further aggregated to improve the similarity of the known defect features within the group.

[0259] In this embodiment, the target label can be any label, and specifically, the edge policy feature set and the average policy feature set can be updated for each label.

[0260] This embodiment does not limit the method of updating the target defect features based on edge features. Optionally, a weighted method can be used, where the weighted sum of the edge features and the target defect features is used to determine the new target defect features. Optionally, a replacement method can also be used to determine the edge features as the new target defect features.

[0261] Optionally, the weights in the weighted sum can be hyperparameters or model parameters that are updated during the training process.

[0262] Similarly, this embodiment does not limit the method of updating the target defect features based on the calculated mean. See above for a detailed explanation.

[0263] This embodiment does not limit the method of updating the current initial feature set. Optionally, for each label, a weighted sum can be calculated among the features corresponding to the updated edge policy feature set, the features corresponding to the updated average policy feature set, and the features corresponding to the current initial feature set. The weighted sum is then used to determine the new corresponding feature in the initial feature set.

[0264] Alternatively, for each annotation, a weighted sum can be calculated between the features corresponding to the updated edge policy feature set and the features corresponding to the updated average policy feature set. This weighted sum is then used as the new corresponding feature in the initial feature set.

[0265] Optionally, the weights in the weighted sum can be hyperparameters or model parameters that are updated during the training process.

[0266] This embodiment does not limit the calculation method of the loss between the current initial feature set and the full feature set.

[0267] Optionally, the loss between the updated initial feature set and the full feature set can be used to characterize the similarity between the updated initial feature set and the full feature set.

[0268] Optionally, the loss between the updated initial feature set and the full feature set can be calculated using cross-entropy or contrastive learning loss.

[0269] Of course, it is understandable that one or more strategy feature sets can be set, and different strategies can be used to sample known defect features.

[0270] 1) Only perform edge policy updates.

[0271] Optionally, training the current defect feature extraction model based on the labeled product image may include the following steps:

[0272] From the known defect features corresponding to each annotation, randomly select one known defect feature and add it to the initial feature set; then add each feature in the initial feature set to the edge strategy feature set.

[0273] Repeat the following steps until the third training stopping condition is met:

[0274] Based on the current defect feature extraction model, known defect features are extracted from the labeled product images to obtain the full feature set;

[0275] The edge policy feature set is updated based on the following steps:

[0276] For the target defect features corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect features with the lowest similarity to the target defect features are identified as edge features; the target defect features are updated based on the edge features.

[0277] Update the current initial feature set based on the updated edge strategy feature set;

[0278] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0279] 2) Only perform average strategy updates.

[0280] Optionally, training the current defect feature extraction model based on the labeled product image may include the following steps:

[0281] From the known defect features corresponding to each annotation, randomly select one known defect feature and add it to the initial feature set; then add each feature in the initial feature set to the average strategy feature set.

[0282] Repeat the following steps until the fourth training stopping condition is met:

[0283] Based on the current defect feature extraction model, known defect features are extracted from the labeled product images to obtain the full feature set;

[0284] The average policy feature set is updated based on the following steps:

[0285] For the target defect features corresponding to the target annotation in the average strategy feature set, calculate the mean for each known defect feature corresponding to the target annotation in the full feature set, and update the target defect features based on the calculated mean.

[0286] Update the current initial feature set based on the updated average policy feature set;

[0287] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0288] 3) Other strategies can also be set. For example, a weighted strategy. Specifically, an additional weighted strategy feature set can be set for updating.

[0289] For ease of understanding, this embodiment of the invention also provides a specific unsupervised training method.

[0290] For unsupervised learning of all samples, this embodiment first uses a clustering algorithm to label pseudo-labels.

[0291] For clustering algorithms, this method uses DBSCAN for unsupervised clustering, which allows similar known defect features to be grouped together.

[0292] DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm that can automatically divide data points into different clusters without prior information, while also identifying and filtering out noisy points. Its core idea is to determine the cluster boundaries by calculating the density around each data point.

[0293] Specifically, the DBSCAN algorithm divides data points into three categories: core points, boundary points, and noise points. Core points are data points whose surrounding density reaches a certain threshold. Boundary points are data points that do not meet the core point condition but have at least one core point around them. Noise points are data points that are neither core points nor boundary points.

[0294] The main parameters of the DBSCAN algorithm are the radius r and the minimum density k, which control the cluster size and density, respectively. The time complexity of the DBSCAN algorithm is O(n log n), where n is the number of data points.

[0295] The core formula of the DBSCAN clustering algorithm is the definition of density-reachable and density-connected:

[0296] Density reachability: If data point q is within the ε-neighborhood of data point p, and data point p is a core point, then data point q and data point p are density reachable.

[0297] Density reachability: If there exists a chain of data points {p1, p2, ..., pn} such that for any 1 ≤ i ≤ n, data points pi and pi+1 are density-directly reachable, then data points p1 and pn are density-reachable.

[0298] Another important parameter in the DBSCAN algorithm is the minimum density k, which represents the minimum number of data points within a radius r. If the number of data points in the ε-neighborhood of a data point is less than k, then that point is considered a noise point.

[0299] The cluster expansion process of the DBSCAN algorithm can be described by the following process:

[0300] 1) Randomly select an unvisited data point p;

[0301] 2) Mark p as visited;

[0302] 3) If p is a core point, find all data points in its ε-neighborhood, mark them as visited, and add them to the current cluster;

[0303] 4) Repeat steps 1-3 until all data points have been accessed.

[0304] After obtaining the initial cluster classification using DBSCAN, this embodiment can randomly sample the classification feature information of the initial cluster according to the pseudo-label, and store the sampled data in the corresponding conceptual memory feature library to form an initialized memory feature library for subsequent model training in the current training round.

[0305] After obtaining the initialized memory feature library, this embodiment designs a corresponding process update and iteration model and memory feature library.

[0306] Firstly, in the current training epoch, each batch of training requires extracting query images for training the defect feature extraction model. Each batch of query dataset contains P*K images of pseudo-labeled defects obtained from initial training clustering, where P represents the number of pseudo-label categories and K represents the number of categories. For each query dataset, NCE Loss can be used for relevant constraints.

[0307] The extracted features can be updated using the Batch Hard and Average strategies to create a corresponding feature group.

[0308] For ease of description, the initialized Memory feature library is called the initial feature library, the feature library using the Batch Hard strategy is called the hard feature library, and the feature library using the Average strategy is called the average feature library.

[0309] In this process, a known defect feature can be randomly selected from each pseudo-label category and added to the initial feature library, the difficult feature library, and the average feature library through random sampling.

[0310] Afterwards, the three feature libraries mentioned above can be updated through the update steps.

[0311] First, for the product images in the query dataset, defect features can be extracted based on the current defect feature extraction model and combined into a full feature set.

[0312] Features in the difficult feature library can be updated according to the following steps.

[0313] For any pseudo-label, obtain the first feature corresponding to the current difficult feature library; then obtain the corresponding features in the full feature set, and determine the difficult feature with the lowest similarity to the first feature; specifically, KL divergence or Euclidean distance can be used for calculation.

[0314] Then, the weighted sum of the first feature and the difficult features can be used to determine the new first feature, thereby updating the features of the difficult feature library.

[0315] The specific weighting can be a momentum update factor, which can be a hyperparameter or an updatable model parameter.

[0316] The features in the average feature library can be updated according to the following steps.

[0317] For any pseudo-label, obtain the corresponding second feature in the current average feature library; then obtain the corresponding feature in the full feature set and determine the average value among the features.

[0318] Then, the weighted sum between the second feature and the feature average can be used to determine the new second feature, thereby updating the features of the average feature library.

[0319] The specific weighting can be a momentum update factor, which can be a hyperparameter or an updatable model parameter.

[0320] After updating the hard feature library and the average feature library, these two feature libraries can be merged. Specifically, the weighted sum of the features corresponding to each pseudo-label can be calculated separately and then updated into the current initial feature library.

[0321] Then, the current initial feature library can be compared with the full feature set using the InfoNCE loss. Updating based on the InfoNCE loss ensures that the features in the current initial feature library are as similar as possible to features with the same pseudo-label in the full feature set, and dissimilar to features with different pseudo-labels.

[0322] 3. Steps for one round of training iteration.

[0323] This method does not limit the steps to be executed in a single training iteration.

[0324] Optionally, a training iteration may include supervised training and / or unsupervised training, or other training methods.

[0325] Optionally, the same or different steps can be performed in different training iterations.

[0326] Optionally, in a training iteration, supervised training can be performed first, followed by unsupervised training; or unsupervised training can be performed first, followed by supervised training.

[0327] For ease of understanding, an optional embodiment of this method is provided.

[0328] Optionally, the training method for the defect feature extraction model includes:

[0329] Repeat the following training steps until the preset training stopping condition is met:

[0330] Acquire multiple sets of different product images with the same known defect, and identify each set of product images as product images with similar known defects;

[0331] The current defect feature extraction model is trained using the acquired multiple sets of product images to improve the similarity of known defect features extracted between product images within the same group, and / or improve the difference of known defect features extracted between product images from different groups.

[0332] Optionally, the training steps may also include:

[0333] Obtain product images with known defects;

[0334] Based on the current defect feature extraction model, known defect features are extracted from the acquired product images;

[0335] Cluster the extracted known defect features to obtain at least two similar defect feature groups, and label the similar defect feature group to which the corresponding known defect feature belongs for each product image acquired.

[0336] The current defect feature extraction model is trained based on the labeled product images to improve the similarity of known defect features extracted between product images with the same label, and / or improve the difference of known defect features extracted between product images with different labels.

[0337] In this embodiment, supervised training can be used to constrain the process, thereby improving the efficiency of unsupervised training and enhancing the training effect of the defect feature extraction model.

[0338] For a detailed explanation of this embodiment, please refer to the above text.

[0339] 4. Training ends.

[0340] In one optional embodiment, the trained defect feature extraction model can be used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects.

[0341] This embodiment does not limit the specific form and value of the first preset similarity.

[0342] Optionally, the trained defect feature extraction model can be used to extract similar known defect features from different product images with similar known defects, thereby facilitating the identification of similar known defects based on the feature extraction of the defect feature extraction model and the similar known defect features.

[0343] Among them, different known defect features with a similarity greater than the first preset similarity can be similar known defect features.

[0344] This method does not limit the training termination conditions of the defect feature extraction model.

[0345] Optionally, the training termination condition for the defect feature extraction model may include: for any different product images with similar known defects in the training sample set, the similarity between the extracted different known defect features is greater than a first preset similarity.

[0346] III. On Defect Detection Models.

[0347] This method does not limit the structure and training method of the defect detection model.

[0348] Optionally, the defect detection model for detecting known defects can be a graph convolutional or graph neural structure.

[0349] Alternatively, the defect detection model can be trained using supervised training.

[0350] Different defect detection models can be used to detect different defects.

[0351] In this method, at least two known defects can be pre-set, as well as different defect detection models corresponding to different known defects; the defect detection model can be used to detect whether the input product image has the corresponding known defect.

[0352] Then, known defects and corresponding defect detection models can be continuously updated through supervised training or transfer training.

[0353] IV. On transfer training.

[0354] This method does not limit the specific form and content of the preset similarity conditions.

[0355] Optionally, the preset similarity condition can be that the similarity is greater than the first preset similarity; or that the similarity is greater than the second preset similarity, where the second preset similarity is greater than the first preset similarity; or that the similarity is the highest; or that the top N known defect features after sorting by similarity from high to low.

[0356] Optionally, the defect feature set may include known defect features extracted by the defect feature extraction model for one or more known defect product images with known defects, which facilitates subsequent similarity calculations to determine candidate defects.

[0357] This method does not limit the specific way of selecting alternative defect detection models. For ease of description, the defect detection model that corresponds one-to-one with the alternative defects can be referred to as the alternative defect detection model.

[0358] It should be noted that when a candidate defect is identified, no further screening is required. Instead, a corresponding defect detection model can be determined directly for that candidate, and the new defective product image can be used for transfer training to obtain a new defect detection model.

[0359] For ease of understanding, the present invention provides two optional embodiments for illustrative purposes.

[0360] 1. Optionally, for the defect detection model corresponding to the identified candidate defects, transfer training can be performed using the acquired new defective product image, which may include the following steps:

[0361] For each of the identified candidate defects, a test set is used to determine the detection accuracy for the new defect. The test set may include images of the new defective product.

[0362] Select a defect detection model whose detection accuracy meets the preset detection conditions, and perform transfer training using the acquired new defective product images.

[0363] This embodiment does not limit the specific form and content of the preset detection conditions. Optionally, the preset detection conditions can be the highest detection accuracy, a detection accuracy greater than the preset accuracy, or a detection accuracy among the top N, etc.

[0364] There can be multiple candidate defects. Therefore, multiple defect detection models can be obtained, each corresponding to one of the multiple candidate defects.

[0365] This embodiment can further select suitable defect detection models for transfer training based on test set detection, thereby improving the efficiency and effectiveness of transfer training.

[0366] 2. Optionally, for the defect detection model corresponding to the identified candidate defects, transfer training can be performed using the acquired new defective product images, which may include the following steps:

[0367] For the multiple defect detection models that correspond one-to-one with the identified multiple candidate defects, transfer training is performed using the acquired images of the new defective products to obtain multiple new candidate defect detection models.

[0368] For each candidate new defect detection model, the detection accuracy for the new defect is determined using a test set; the test set may include images of the new defective product.

[0369] The candidate new defect detection models whose detection accuracy meets the preset accuracy conditions are determined as new defect detection models.

[0370] This embodiment does not limit the specific form and content of the preset accuracy condition. Optionally, the preset accuracy condition can be the highest detection accuracy, the detection accuracy greater than the preset accuracy, or the detection accuracy ranking in the top N, etc.

[0371] This embodiment can improve the effect of transfer training by performing test detection on the test set, transferring the candidate defect detection model, calculating the detection accuracy through the test set, and selecting a suitable candidate defect detection model as the new defect detection model.

[0372] This method does not limit the specific transfer training method.

[0373] Optionally, fine-tuning or supervised training can be performed based on product image samples with new defects. Specifically, some parameters in the model to be transferred can be adjusted and updated.

[0374] Alternatively, a smaller learning rate can be used to tune the parameters of the model to be transferred, which helps to preserve the original knowledge of the model.

[0375] For ease of understanding, the present invention also provides a specific example.

[0376] like Figure 2 As shown, Figure 2 This is a schematic diagram illustrating the principle of defect detection model migration according to an embodiment of the present invention.

[0377] Among them, multiple defect detection models corresponding to multiple identified candidate defects can be combined into a backup model library.

[0378] Then, new defect image datasets can be collected and combined into a test set.

[0379] Then, the defect detection models in the backup model library are determined using the test set. The detection accuracy for the new defects can be specifically combined into a model-accuracy table. The test set can include images of new defective products with the new defects.

[0380] Then, a defect detection model whose detection accuracy meets the preset detection conditions can be selected, and the newly acquired defective product images can be used for transfer training, which facilitates the rapid deployment of the platform in the future.

[0381] For ease of understanding, this invention also provides an application embodiment.

[0382] In screen manufacturing or other industrial scenarios, process defects can appear at various stages of the production process. As the development of customized defect detection technologies and deep learning models accumulates, the number of defect detection models increases. When new defects emerge or similar defects appear on other production lines, it becomes difficult to develop rapid defect detection models. Furthermore, the lack of comparative analysis data for production line defects makes it difficult to generate a large number of annotation files.

[0383] To this end, this embodiment designs an unsupervised similarity analysis process and specific method for defects in screens or other industrial scenarios. It evaluates the similarity of defects appearing in different scenarios, and then develops and deploys a rapid model transfer learning model for defects with high similarity.

[0384] Image similarity analysis is a crucial problem in computer vision, applicable to tasks such as image retrieval, classification, and recognition. Deep learning, a key technology that has achieved significant success in computer vision in recent years, can automatically learn feature representations of images from massive datasets and has demonstrated excellent performance in tasks like image similarity analysis. Within deep learning, Convolutional Neural Networks (CNNs) are commonly used image processing networks. CNNs extract meaningful features from images, transforming them into high-dimensional feature vectors, and then calculate image similarity by comparing these feature vectors. Common methods for comparing feature vectors include calculating metrics such as Euclidean distance or cosine similarity. By calculating these metrics, a set of similarity scores is obtained, allowing images to be ranked based on their similarity.

[0385] Unsupervised learning is a method of machine learning that, unlike supervised learning, does not require pre-labeled data to learn from. Instead, it automatically learns patterns and features from unlabeled data. Unsupervised learning can be used in various fields, such as data mining, pattern recognition, image processing, and speech recognition. The goal of this method is to discover the inherent structure and patterns in data to better understand the data and subsequent tasks.

[0386] This embodiment designs a defect similarity analysis process and specific method for screens or other industrial scenarios. It evaluates the similarity of defects appearing in different scenarios, and then develops and deploys a rapid model transfer learning model for defects with high similarity.

[0387] This embodiment designs a defect similarity analysis process and specific method for screens or other industrial scenarios. It evaluates the similarity of defects appearing in different scenarios, and then develops and deploys a rapid model transfer learning model for defects with high similarity.

[0388] This includes similarity measurement models and processes for screens and other industrial defects; methods and processes for rapid transfer learning based on existing models after model matching; and management and storage methods for managing existing models and existing defect example images based on similarity measurement.

[0389] This embodiment can greatly reduce repetitive model development work in industrial production processes, improve the efficiency and accuracy of industrial defect detection development, reduce the data cost of industrial defect detection, and shorten the development cycle when new defects appear in different scenarios.

[0390] This embodiment designs an unsupervised similarity analysis process and specific method for defects in screens or other industrial scenarios.

[0391] Specifically, a screen defect feature extraction model can be trained using both supervised and unsupervised training methods. This model can then be used to extract different known defect features from different screen product images with similar known defects. Specifically, these can be known defect features with a similarity greater than a first preset similarity threshold.

[0392] Subsequently, for new screen defects, a screen defect feature extraction model can be used to identify target known defects similar to the new defects. Then, based on the defect detection model of the target known defects, transfer training can be performed to obtain a screen defect detection model for detecting new screen defects.

[0393] For details, please refer to the explanation of the above method embodiments.

[0394] Furthermore, the similarity between different defect features can be determined based on the defect feature extraction model. This defect feature similarity can then be used to define the similarity between defect detection models, which can be displayed to users for easy viewing and selection.

[0395] Given existing defect detection model similarity information (i.e., the similarity between defects), this embodiment can use the following steps to manage different models with similarity using a relational database:

[0396] 1. Database Schema Design: First, design a relational database schema to store the model's metadata (e.g., model name, type, creation date, performance metrics, etc.) and similarity information. The database schema includes the following tables:

[0397] Model table: Stores basic information about the model, such as model ID, name, type, etc.

[0398] Similarity table: Stores similarity information between models, including model ID, reference model ID, and similarity value.

[0399] 2. Import similarity information: Import existing model similarity information into the similarity table. Ensure that each record contains the model ID, reference model ID, and similarity value.

[0400] 3. Implement query functionality: Based on requirements, implement a query function based on model similarity. For example, the following functionality can be implemented:

[0401] Query the model most similar to a given model; query the similarity values ​​between multiple models; query similar models within a given threshold.

[0402] 4. Model Organization and Management: Based on similarity information, similar models can be grouped together for easier management. This can be achieved through the following methods:

[0403] Add a field (e.g., group_id) to the model table to assign models with similarity to the same group.

[0404] Create a separate model group table to store information about model groups (such as group ID, group name, description, etc.). In the model table, add a field (e.g., group_id) for each model to indicate the model group it belongs to.

[0405] 5. Update and maintenance: Implement functions for updating and maintaining model similarity information and model metadata as needed.

[0406] Update model metadata;

[0407] Update the similarity information between models;

[0408] Add a new model and import similarity information.

[0409] By following the steps described above, this embodiment can use a relational database to manage different models that have similarity.

[0410] To visualize the model information of the similarity management database on a web page, the following steps can be taken:

[0411] 1. Choose a visualization library: Select a visualization library suitable for the web, such as D3.js, Plotly.js, Chart.js, ECharts, etc.

[0412] 2. Create a Web Application: Create a basic web application so that it can be visualized in a browser.

[0413] 3. Connect to the database: On the backend of the web application (e.g., using Node.js, Flask, Django, etc.), connect to a relational database to query model and similarity information. Implement an API interface to pass data from the database to the frontend.

[0414] 4. Data Acquisition: On the front end, use AJAX or the Fetch API to retrieve model and similarity information from the back end API. Process the data to make it suitable for visualization.

[0415] 5. Create visualizations: Use the selected visualization library to plot the acquired data into charts. For example:

[0416] Scatter plot: Displays the similarity relationship between models. In a scatter plot, each model is represented by a point, and the x-axis and y-axis can represent the main features of the model. Color, shape, or size can be used to represent the similarity between models. Models with high similarity are closer together on the plot. Hovering over a point displays detailed information about the model.

[0417] Matrix diagram: Displays the similarity values ​​between models. A matrix diagram is a two-dimensional table where rows and columns represent models. The color of a cell indicates the similarity value between models. The darker the color, the higher the similarity. Hovering over a cell displays the specific similarity value.

[0418] Hierarchical Graph: Displays model clustering based on similarity, for example, using a dendrogram or pie chart to show hierarchical relationships. The root node of the tree represents the entire model set, child nodes represent clusters, and leaf nodes represent models. Models with high similarity are located on the same branch in the dendrogram. Hovering over a node displays detailed information about the model or cluster.

[0419] Force-directed graph: Displays the similarity relationships between models. In the force-directed graph, models are represented as nodes, and similarity is represented as the weight of the edges. The distance between nodes reflects the similarity relationship; models with high similarity are closer together in the graph. Hovering over a node displays detailed information about the model; hovering over an edge displays the similarity value.

[0420] 6. Interactive Features: Add interactive features as needed, such as zooming, dragging, hover tooltips, and click events. This will make the visualizations more engaging and help users better understand the model information.

[0421] 7. Responsive Design: To ensure that visualizations display correctly on different devices (such as desktops, tablets, and mobile phones), apply responsive design. Use CSS and JavaScript to adapt to different screen sizes and device types.

[0422] Through the above steps, this embodiment can visualize the model information of the similarity management database on a web page.

[0423] In addition, this invention also discloses another method for transferring a defect detection model.

[0424] like Figure 3 As shown, Figure 3 This is a flowchart illustrating another method for migrating a defect detection model according to an embodiment of the present invention.

[0425] The embodiments of this invention do not limit the executing entity of this method. Optionally, the executing entity can be any computing device. For example, a server or client used to detect product defects.

[0426] The method may include the following steps.

[0427] S201: For a product image with a new defect, use feature extraction to identify known defects similar to the new defect as candidate defects.

[0428] S202: Obtain a candidate defect detection model for detecting whether the input product image has a candidate defect.

[0429] S203: For the acquired candidate defect detection models, transfer training is performed using the new defective product image to obtain a new defect detection model for detecting whether the input product image has a new defect.

[0430] This method does not limit the specific feature extraction method. Optionally, an image recognition model can be used to extract defect features, or the aforementioned defect feature extraction model can be used to extract defect features.

[0431] Optionally, the method for extracting new defect features from images of new defective products can be the same as the method for extracting known defect features.

[0432] Optionally, for a product image with a new defect, a known defect similar to the new defect is identified as a candidate defect through feature extraction. Specifically, this can be done by: extracting new defect features from the product image with the new defect using a preset method; and identifying known defect features whose similarity to the new defect features meets a preset similarity condition as candidate defects. The known defect features are obtained through a preset method.

[0433] This method is not limited to known defects that are similar to the new defect.

[0434] Optionally, a known defect similar to the new defect can specifically be a known defect feature whose similarity to the new defect feature meets a preset similarity condition, and further, a known defect corresponding to that feature. This embodiment does not limit the preset similarity condition; optionally, the preset similarity condition can be the highest similarity or a similarity greater than a preset similarity threshold.

[0435] Optionally, a known defect similar to the new defect can specifically be a known defect feature whose similarity to the new defect feature is greater than a first preset similarity, and a further known defect corresponding to it.

[0436] This method does not limit the specific method for identifying known defects that are similar to the new defect.

[0437] Alternatively, the determination can be based on the similarity between defect features, or it can be based on the similarity between a new defective product image and a known defective product image.

[0438] This method does not limit the method for obtaining alternative defect detection models.

[0439] Optionally, a candidate defect detection model can be directly selected from a pre-defined set of different defect detection models that correspond one-to-one with different known defects to detect whether the input product image has a candidate defect. For specific selection methods, please refer to the above method embodiments.

[0440] Alternatively, since known defects can generally accumulate a large number of samples, image samples of alternative defects can also be used for real-time training to obtain an alternative defect detection model for detecting whether an input product image has alternative defects.

[0441] Optionally, alternative defect detection models can be selected and transferred to train based on their recognition accuracy for images of new defective products.

[0442] This method does not limit the specific way of performing transfer training; please refer to the above method implementation examples for details.

[0443] Furthermore, for a detailed explanation of this method, please refer to the above-described method embodiments.

[0444] The above uses a defect detection scenario as an example to explain the specific model transfer method.

[0445] This invention also provides model transfer methods for other scenarios.

[0446] For example, a similar transfer method can be used to quickly obtain a new product type detection model for a product type detection model. Specifically, for newly developed products, the new product type detection model can be used for detection during machine sorting, facilitating subsequent sorting.

[0447] For example, a similar transfer method can be used for the detection model of item type to quickly obtain a new item type detection model for the new item type.

[0448] For example, a similar transfer method can be used for image type detection models to quickly obtain new image type detection models for new image types.

[0449] The embodiments of the present invention are not limited to specific image types.

[0450] Optionally, the image type may include at least one of the following: whether the product in the image has a certain defect, whether the product in the image belongs to a certain product type, whether the item in the image belongs to a certain item type, whether the person in the image belongs to a certain group of people, etc.

[0451] like Figure 4 As shown, Figure 4 This is a flowchart illustrating a type detection model transfer method according to an embodiment of the present invention.

[0452] The embodiments of the present invention do not limit the executing entity of this method flow. Optionally, the executing entity can be any computing device. For example, a server or client for detecting types.

[0453] At least two known types can be pre-set, as well as different type detection models corresponding to different known types; the type detection model can be used to detect whether the input image conforms to the corresponding known type.

[0454] The method may include the following steps.

[0455] S301: Obtain a pre-trained type feature extraction model and a set of type features.

[0456] Among them, the type feature extraction model can be used to extract different known type features with a similarity greater than a first preset similarity from different images of similar known types.

[0457] The type feature set can include known type features extracted by the type feature extraction model for images that conform to a known type.

[0458] S302: Extract the new type of features from the image input type feature extraction model that conforms to the new type.

[0459] S303: In the set of type features, for known type features whose similarity to new type features meets the preset similarity conditions, the corresponding known type is determined as a candidate type.

[0460] S304: For the type detection model corresponding to the determined candidate type, use the image that conforms to the new type for transfer training to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0461] The above method identifies known types similar to the new type based on the similarity between type features, and performs transfer training based on the type detection model that detects the identified known types, which can improve the generation efficiency of the new type detection model for detecting new types.

[0462] The embodiments of the present invention do not limit the specific type of image correspondence.

[0463] Optionally, the type may include at least one of the following: whether the product in the image has a certain defect, whether the product in the image belongs to a certain product type, whether the item in the image belongs to a certain item type, whether the person in the image belongs to a certain group of people, etc.

[0464] Optionally, the training methods for the type feature extraction model include:

[0465] Repeat the following training steps until the preset training stopping condition is met:

[0466] Obtain multiple sets of different images that conform to the same known type, and determine each set of images as images that conform to a similar known type;

[0467] The current type feature extraction model is trained using multiple sets of acquired images to improve the similarity of known type features extracted between images within a group and / or improve the difference of known type features extracted between images from different groups.

[0468] Optionally, the training steps may also include:

[0469] Get images that match the known types;

[0470] Based on the current type feature extraction model, known type features are extracted from the acquired image;

[0471] Cluster the extracted known type features to obtain at least two similar type feature groups, and label the similar type feature group to which the corresponding known type feature belongs for each acquired image;

[0472] The current type feature extraction model is trained based on the labeled images to improve the similarity of known type features extracted between images with the same label, and / or improve the difference of known type features extracted between images with different labels.

[0473] Optionally, the current type feature extraction model is trained based on the labeled image, including:

[0474] Repeat the following steps until the first training stopping condition is met:

[0475] Based on the current type feature extraction model, known type features are extracted from the labeled image to obtain the first feature set;

[0476] Based on a preset strategy, a known type feature is determined based on the known type features of each corresponding annotation, thus obtaining a second feature set;

[0477] The loss is calculated based on the obtained first and second feature sets, and the current type feature extraction model is updated.

[0478] Optionally, the current type feature extraction model is trained based on the labeled image, including:

[0479] From the known type features corresponding to each label, randomly select one known type feature and add it to the initial feature set; then add each feature in the initial feature set to the edge policy feature set and the average policy feature set, respectively.

[0480] Repeat the following steps until the second training stopping condition is met:

[0481] Based on the current type feature extraction model, known type features are extracted from the labeled image to obtain the full feature set;

[0482] The edge policy feature set is updated based on the following steps:

[0483] For the target type features corresponding to the target annotation in the edge strategy feature set, among the known type features corresponding to the target annotation in the full feature set, the known type feature with the lowest similarity to the target type feature is identified as the edge feature; the target type feature is updated based on the edge feature.

[0484] The average policy feature set is updated based on the following steps:

[0485] For the target type features corresponding to the target labels in the average strategy feature set, calculate the mean for each known type feature corresponding to the target labels in the full feature set, and update the target type features based on the calculated mean.

[0486] Update the current initial feature set based on the updated edge policy feature set and the updated average policy feature set;

[0487] The loss is calculated based on the updated initial feature set and the full feature set, and the current type feature extraction model is updated.

[0488] Optionally, for the type detection model corresponding to the determined candidate types, transfer training is performed using the acquired new type images, including:

[0489] For the multiple type detection models that correspond one-to-one with the identified multiple candidate types, the detection accuracy for the new type is determined using a test set; the test set contains images of the new type that conform to the new type.

[0490] Select a type detection model whose detection accuracy meets the preset detection conditions, and perform transfer training using the acquired new type images.

[0491] Optionally, for the type detection model corresponding to the determined candidate types, transfer training is performed using the acquired new type images, including:

[0492] For the multiple type detection models that correspond one-to-one with the identified multiple candidate types, transfer training is performed using the acquired new type images to obtain multiple candidate new type detection models;

[0493] For each candidate new type detection model, the detection accuracy for the new type is determined using a test set; the test set contains new type images that conform to the new type.

[0494] The candidate new type of detection model whose detection accuracy meets the preset accuracy conditions is determined as the new type of detection model.

[0495] For a detailed explanation of this method, please refer to the above method implementation examples.

[0496] In addition, embodiments of the present invention also disclose a transfer method for another type of detection model.

[0497] like Figure 5 As shown, Figure 5 This is a flowchart illustrating a transfer method for another type of detection model according to an embodiment of the present invention.

[0498] The embodiments of the present invention do not limit the executing entity of this method flow. Optionally, the executing entity can be any computing device. For example, a server or client for detecting image types.

[0499] The method may include the following steps.

[0500] S401: For images that conform to the new type, known types similar to the new type are identified as candidate types through feature extraction;

[0501] S402: Obtain a candidate type detection model for detecting whether the input image conforms to the candidate type;

[0502] S403: For the obtained candidate type detection models, use images that conform to the new type for transfer training to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0503] This method does not limit the specific feature extraction method. Optionally, an image recognition model can be used to extract type features, or the type feature extraction model described above can be used to extract type features.

[0504] Optionally, the method for extracting new type features from new type images can be the same as the method for extracting features of known types.

[0505] Optionally, for a new type image that conforms to the new type, known types similar to the new type are identified as candidate types through feature extraction. Specifically, this can be done by: extracting new type features for images that conform to the new type using a preset method; and identifying known type features whose similarity to the new type features meets a preset similarity condition as candidate types. The known type features are obtained through the preset method.

[0506] This method is not limited to known types that are similar to the new type.

[0507] Optionally, the known type similar to the new type can specifically be a known type feature whose similarity to the new type's features meets a preset similarity condition, and further, the corresponding known type. This embodiment does not limit the preset similarity condition; optionally, the preset similarity condition can be the highest similarity or a similarity greater than a preset similarity threshold.

[0508] Optionally, the known type similar to the new type can specifically be a known type feature whose similarity to the new type feature is greater than a first preset similarity, and further corresponding to the known type.

[0509] This method does not limit the specific method for determining known types that are similar to the new type.

[0510] Alternatively, the determination can be based on the similarity between type features, or it can be based on the similarity between the new type image and known type images.

[0511] This method does not limit the method for obtaining alternative type detection models.

[0512] Optionally, a candidate type detection model can be directly selected from a pre-defined set of different type detection models that correspond one-to-one with different known types to determine whether the input image conforms to the candidate type. For specific selection methods, please refer to the above method embodiments.

[0513] Alternatively, since known types can generally accumulate a large number of samples, image samples of alternative types can also be used for real-time training to obtain an alternative type detection model for detecting whether an input image conforms to the alternative type.

[0514] Optionally, alternative type detection models can be combined to select and transfer training based on the recognition accuracy of new type images.

[0515] This method does not limit the specific way of performing transfer training; please refer to the above method implementation examples for details.

[0516] Furthermore, for a detailed explanation of this method, please refer to the above-described method embodiments.

[0517] Corresponding to the above method embodiments, this invention also provides an apparatus embodiment.

[0518] like Figure 6 As shown, Figure 6 This is a schematic diagram of a defect detection model transfer device according to an embodiment of the present invention. At least two known defects are pre-set, and different defect detection models correspond one-to-one with each of the different known defects; the defect detection model is used to detect whether an input product image has the corresponding known defect.

[0519] The device may include the following units.

[0520] The acquisition unit 501 is used to acquire a pre-trained defect feature extraction model and a defect feature set;

[0521] The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects; the defect feature set includes the known defect features extracted by the defect feature extraction model from known defect product images with known defects;

[0522] The similarity unit 502 is used to input the image of the new defective product with the new defect into the defect feature extraction model to obtain the new defect feature; in the defect feature set, for the known defect features whose similarity with the new defect feature meets the preset similarity condition, the corresponding known defect is determined as the candidate defect;

[0523] The transfer unit 503 is used to transfer and train the defect detection model corresponding to the determined candidate defects using the acquired new defect product image to obtain a new defect detection model for detecting whether the input product image has a new defect.

[0524] Optionally, the training method for the defect feature extraction model includes:

[0525] Repeat the following training steps until the preset training stopping condition is met:

[0526] Acquire multiple sets of different product images with the same known defect, and identify each set of product images as product images with similar known defects;

[0527] The current defect feature extraction model is trained using the acquired multiple sets of product images to improve the similarity of known defect features extracted between product images within the same group, and / or improve the difference of known defect features extracted between product images from different groups.

[0528] Optionally, the training steps may also include:

[0529] Obtain product images with known defects;

[0530] Based on the current defect feature extraction model, known defect features are extracted from the acquired product images;

[0531] Cluster the extracted known defect features to obtain at least two similar defect feature groups, and label the similar defect feature group to which the corresponding known defect feature belongs for each product image acquired.

[0532] The current defect feature extraction model is trained based on the labeled product images to improve the similarity of known defect features extracted between product images with the same label, and / or improve the difference of known defect features extracted between product images with different labels.

[0533] Optionally, the current defect feature extraction model is trained based on the labeled product images, including:

[0534] Repeat the following steps until the first training stopping condition is met:

[0535] Based on the current defect feature extraction model, known defect features are extracted from the labeled product image to obtain the first feature set;

[0536] Based on a preset strategy, a known defect feature is determined based on the known defect features of each corresponding annotation, thus obtaining a second feature set;

[0537] The loss is calculated based on the obtained first and second feature sets, and the current defect feature extraction model is updated.

[0538] Optionally, the current defect feature extraction model is trained based on the labeled product images, including:

[0539] From the known defect features corresponding to each annotation, randomly select one known defect feature and add it to the initial feature set; then add each feature in the initial feature set to the edge policy feature set and the average policy feature set, respectively.

[0540] Repeat the following steps until the second training stopping condition is met:

[0541] Based on the current defect feature extraction model, known defect features are extracted from the labeled product images to obtain the full feature set;

[0542] The edge policy feature set is updated based on the following steps:

[0543] For the target defect features corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect features with the lowest similarity to the target defect features are identified as edge features; the target defect features are updated based on the edge features.

[0544] The average policy feature set is updated based on the following steps:

[0545] For the target defect features corresponding to the target annotation in the average strategy feature set, calculate the mean for each known defect feature corresponding to the target annotation in the full feature set, and update the target defect features based on the calculated mean.

[0546] Update the current initial feature set based on the updated edge policy feature set and the updated average policy feature set;

[0547] The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

[0548] Optionally, the migration unit 503 is used for:

[0549] For the multiple defect detection models corresponding to the identified candidate defects, the detection accuracy for the new defects is determined using a test set; the test set contains images of the new defective products with the new defects.

[0550] Select a defect detection model whose detection accuracy meets the preset detection conditions, and perform transfer training using the acquired new defective product images.

[0551] Optionally, the migration unit 503 is used for:

[0552] For the multiple defect detection models corresponding to the identified candidate defects, transfer training is performed using the acquired images of the new defective products to obtain multiple new candidate defect detection models.

[0553] For each candidate new defect detection model, the detection accuracy for the new defect is determined using a test set; the test set contains images of the new defective product.

[0554] The candidate new defect detection models whose detection accuracy meets the preset accuracy conditions are determined as new defect detection models.

[0555] For a detailed explanation, please refer to the above method implementation examples.

[0556] like Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of a migration device for another defect detection model according to an embodiment of the present invention.

[0557] The device may include the following units.

[0558] The feature extraction unit 601 is used to identify known defects similar to the new defect as candidate defects for a new defect product image with a new defect by means of feature extraction.

[0559] The model acquisition unit 602 is used to acquire a candidate defect detection model for detecting whether the input product image has the candidate defect;

[0560] The transfer training unit 603 is used to transfer train the acquired candidate defect detection model using the new defect product image to obtain a new defect detection model for detecting whether the input product image has a new defect.

[0561] For a detailed explanation of this device embodiment, please refer to the above method embodiment.

[0562] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a migration device for a type detection model according to an embodiment of the present invention.

[0563] At least two known types are pre-set, and different type detection models are pre-defined for each of the different known types; the type detection models are used to detect whether the input image conforms to the corresponding known type.

[0564] The device may include the following units.

[0565] Type acquisition unit 701 is used to acquire a pre-trained type feature extraction model and a set of type features;

[0566] The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity from different images of similar known types; the type feature set includes known type features extracted by the type feature extraction model for images that conform to known types;

[0567] The new type unit 702 is used to input an image that conforms to the new type into the type feature extraction model to obtain a new type feature; in the type feature set, for known type features whose similarity to the new type feature meets the preset similarity condition, the corresponding known type is determined as a candidate type;

[0568] Training unit 703 is used to transfer train the type detection model corresponding to the determined candidate type using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0569] The embodiments of the present invention do not limit the specific type of image correspondence.

[0570] Optionally, the type may include at least one of the following: whether the product in the image has a certain defect, whether the product in the image belongs to a certain product type, whether the item in the image belongs to a certain item type, whether the person in the image belongs to a certain group of people, etc.

[0571] Optionally, the training methods for the type feature extraction model include:

[0572] Repeat the following training steps until the preset training stopping condition is met:

[0573] Obtain multiple sets of different images that conform to the same known type, and determine each set of images as images that conform to a similar known type;

[0574] The current type feature extraction model is trained using multiple sets of acquired images to improve the similarity of known type features extracted between images within a group and / or improve the difference of known type features extracted between images from different groups.

[0575] Optionally, the training steps may also include:

[0576] Get images that match the known types;

[0577] Based on the current type feature extraction model, known type features are extracted from the acquired image;

[0578] Cluster the extracted known type features to obtain at least two similar type feature groups, and label the similar type feature group to which the corresponding known type feature belongs for each acquired image;

[0579] The current type feature extraction model is trained based on the labeled images to improve the similarity of known type features extracted between images with the same label, and / or improve the difference of known type features extracted between images with different labels.

[0580] Optionally, the current type feature extraction model is trained based on the labeled image, including:

[0581] Repeat the following steps until the first training stopping condition is met:

[0582] Based on the current type feature extraction model, known type features are extracted from the labeled image to obtain the first feature set;

[0583] Based on a preset strategy, a known type feature is determined based on the known type features of each corresponding annotation, thus obtaining a second feature set;

[0584] The loss is calculated based on the obtained first and second feature sets, and the current type feature extraction model is updated.

[0585] Optionally, the current type feature extraction model is trained based on the labeled image, including:

[0586] From the known type features corresponding to each label, randomly select one known type feature and add it to the initial feature set; then add each feature in the initial feature set to the edge policy feature set and the average policy feature set, respectively.

[0587] Repeat the following steps until the second training stopping condition is met:

[0588] Based on the current type feature extraction model, known type features are extracted from the labeled image to obtain the full feature set;

[0589] The edge policy feature set is updated based on the following steps:

[0590] For the target type features corresponding to the target annotation in the edge strategy feature set, among the known type features corresponding to the target annotation in the full feature set, the known type feature with the lowest similarity to the target type feature is identified as the edge feature; the target type feature is updated based on the edge feature.

[0591] The average policy feature set is updated based on the following steps:

[0592] For the target type features corresponding to the target labels in the average strategy feature set, calculate the mean for each known type feature corresponding to the target labels in the full feature set, and update the target type features based on the calculated mean.

[0593] Update the current initial feature set based on the updated edge policy feature set and the updated average policy feature set;

[0594] The loss is calculated based on the updated initial feature set and the full feature set, and the current type feature extraction model is updated.

[0595] Optionally, training unit 703 is used for:

[0596] For the multiple type detection models that correspond one-to-one with the identified multiple candidate types, the detection accuracy for the new type is determined using a test set; the test set contains images of the new type that conform to the new type.

[0597] Select a type detection model whose detection accuracy meets the preset detection conditions, and perform transfer training using the acquired new type images.

[0598] Optionally, training unit 703 is used for:

[0599] For the multiple type detection models that correspond one-to-one with the identified multiple candidate types, transfer training is performed using the acquired new type images to obtain multiple candidate new type detection models;

[0600] For each candidate new type detection model, the detection accuracy for the new type is determined using a test set; the test set contains new type images that conform to the new type.

[0601] The candidate new type of detection model whose detection accuracy meets the preset accuracy conditions is determined as the new type of detection model.

[0602] For a detailed explanation of this device embodiment, please refer to the above method embodiment.

[0603] like Figure 9 As shown, Figure 9 This is a schematic diagram of a migration device for another type of detection model according to an embodiment of the present invention. The device may include the following units.

[0604] The candidate type unit 801 is used to determine known types similar to the new type as candidate types for images that conform to the new type by means of feature extraction;

[0605] The alternative model unit 802 is used to obtain an alternative type detection model for detecting whether the input image conforms to the alternative type;

[0606] Result unit 803 is used to perform transfer training on the acquired candidate type detection model using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

[0607] For a detailed explanation of this device embodiment, please refer to the above method embodiment.

[0608] This invention also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described method embodiments.

[0609] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above method embodiments.

[0610] Figure 10 This is a schematic diagram of the hardware structure of a computer device configured according to an embodiment of the present invention. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are internally connected to each other via the bus 1050.

[0611] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0612] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of the present invention are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0613] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0614] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0615] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0616] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of the present invention, and not necessarily all the components shown in the figures.

[0617] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described method embodiments.

[0618] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described method embodiments.

[0619] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0620] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0621] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0622] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. In implementing the embodiments of the present invention, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0623] The above description is merely a specific implementation of the embodiments of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the embodiments of the present invention, and these improvements and modifications should also be considered as protection of the embodiments of the present invention.

[0624] In this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0625] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. The invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0626] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for transferring a defect detection model, characterized in that, There are at least two known defects pre-set, as well as different defect detection models corresponding to each of the different known defects; The defect detection model is used to detect whether the input product image has a corresponding known defect; The method includes: Obtain a pre-trained defect feature extraction model and a set of defect features; The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects; the defect feature set includes known defect features extracted by the defect feature extraction model from known defect product images with known defects. The image of the product with the new defect is input into the defect feature extraction model to obtain the new defect features; In the set of defect features, for known defect features whose similarity to the new defect feature meets the preset similarity condition, the corresponding known defects are determined as candidate defects; For the defect detection model corresponding to the identified candidate defects, the new defect product image is used for transfer training to obtain a new defect detection model for detecting whether the input product image has a new defect.

2. The method according to claim 1, characterized in that, The training method for the defect feature extraction model includes: Repeat the following training steps until the preset training stopping condition is met: Acquire multiple sets of different product images with the same known defect, and identify each set of product images as product images with similar known defects; The current defect feature extraction model is trained using the acquired multiple sets of product images to improve the similarity of known defect features extracted between product images within the same group, and / or improve the difference of known defect features extracted between product images from different groups.

3. The method according to claim 2, characterized in that, The training steps also include: Obtain product images with known defects; Based on the current defect feature extraction model, known defect features are extracted from the acquired product images; Cluster the extracted known defect features to obtain at least two similar defect feature groups, and label the similar defect feature group to which the corresponding known defect feature belongs for each product image acquired. The current defect feature extraction model is trained based on the labeled product images to improve the similarity of known defect features extracted between product images with the same label, and / or improve the difference of known defect features extracted between product images with different labels.

4. The method according to claim 3, characterized in that, The process of training the current defect feature extraction model based on the labeled product images includes: Repeat the following steps until the first training stopping condition is met: Based on the current defect feature extraction model, known defect features are extracted from the labeled product image to obtain the first feature set; Based on a preset strategy, a known defect feature is determined based on the known defect features of each corresponding annotation, thus obtaining a second feature set; The loss is calculated based on the obtained first and second feature sets, and the current defect feature extraction model is updated.

5. The method according to claim 3, characterized in that, The process of training the current defect feature extraction model based on the labeled product images includes: From the known defect features corresponding to each annotation, randomly select one known defect feature and add it to the initial feature set; then add each feature in the initial feature set to the edge policy feature set and the average policy feature set, respectively. Repeat the following steps until the second training stopping condition is met: Based on the current defect feature extraction model, known defect features are extracted from the labeled product images to obtain the full feature set; The edge policy feature set is updated based on the following steps: For the target defect features corresponding to the target annotation in the edge strategy feature set, among the known defect features corresponding to the target annotation in the full feature set, the known defect feature with the lowest similarity to the target defect feature is determined as the edge feature; the target defect feature is updated based on the edge feature; The average policy feature set is updated based on the following steps: For the target defect features corresponding to the target annotation in the average strategy feature set, calculate the mean for each known defect feature corresponding to the target annotation in the full feature set, and update the target defect features based on the calculated mean. Update the current initial feature set based on the updated edge policy feature set and the updated average policy feature set; The loss is calculated based on the updated initial feature set and the full feature set, and the current defect feature extraction model is updated.

6. The method according to claim 1, characterized in that, The defect detection model corresponding to the identified candidate defects is trained using the acquired new defective product images, including: For each of the identified candidate defects, a test set is used to determine the detection accuracy for the new defect; the test set includes images of the new defective product. Select a defect detection model whose detection accuracy meets the preset detection conditions, and perform transfer training using the acquired new defective product images.

7. The method according to claim 1, characterized in that, The defect detection model corresponding to the identified candidate defects is trained using the acquired new defective product images, including: For the multiple defect detection models that correspond one-to-one with the identified multiple candidate defects, transfer training is performed using the acquired images of the new defective products to obtain multiple new candidate defect detection models. For each candidate new defect detection model, the detection accuracy for the new defect is determined using a test set; the test set contains images of the new defective product. The candidate new defect detection models whose detection accuracy meets the preset accuracy conditions are determined as new defect detection models.

8. A method for transferring a defect detection model, characterized in that, There are at least two known defects pre-set, as well as different defect detection models corresponding to each of the different known defects; The defect detection model is used to detect whether the input product image has a corresponding known defect; the method includes: For a product image with a new defect, a feature extraction method is used to identify known defects similar to the new defect as candidate defects. Obtain a candidate defect detection model for detecting whether the input product image has the candidate defects; For the acquired candidate defect detection models, transfer training is performed using new defective product images to obtain a new defect detection model for detecting whether the input product image has new defects.

9. The method according to claim 8, characterized in that, The method for identifying a new defective product image with a novel defect involves using feature extraction to determine known defects similar to the novel defect as candidate defects, including: For images of products with new defects, extract the features of the new defects using a preset method. For known defect features whose similarity to the new defect features meets a preset similarity condition, the corresponding known defects are identified as candidate defects; The known defect features are extracted using the preset method.

10. A transfer learning method for a type detection model, characterized in that, At least two known types are pre-set, and different type detection models are pre-defined for each of the different known types; the type detection models are used to detect whether the input image conforms to the corresponding known type. The method includes: Obtain a pre-trained type feature extraction model and a set of type features; The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity from different images of similar known types; the type feature set includes known type features extracted by the type feature extraction model for images that conform to known types; The image that matches the new type is input into the type feature extraction model to obtain the new type features; In the set of type features, for known type features whose similarity to the new type feature meets the preset similarity condition, the corresponding known type is determined as a candidate type; For the type detection model corresponding to the identified candidate types, transfer training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

11. A transfer learning method for a type detection model, characterized in that, There are at least two known defects pre-set, as well as different defect detection models corresponding to each of the different known defects; The defect detection model is used to detect whether the input product image has a corresponding known defect; the method includes: For images that conform to the new type, known types similar to the new type are identified as candidate types through feature extraction. Obtain a candidate type detection model for detecting whether the input image conforms to the candidate type; For the obtained candidate type detection models, transfer training is performed using images that conform to the new type to obtain a new type detection model for detecting whether the input image conforms to the new type.

12. The method according to claim 11, characterized in that, For images conforming to the new type, the method of feature extraction identifies known types similar to the new type as candidate types, including: For images that conform to the new type, extract the new type features using a preset method; For known type features whose similarity to the new type features meets the preset similarity conditions, the corresponding known types are determined as candidate types; Among them, the known type features are extracted through the preset method.

13. A transfer device for a defect detection model, characterized in that, There are at least two known defects pre-set, as well as different defect detection models corresponding to each of the different known defects; The defect detection model is used to detect whether the input product image has a corresponding known defect; The device includes: The acquisition unit is used to acquire the pre-trained defect feature extraction model and the defect feature set; The defect feature extraction model is used to extract different known defect features with a similarity greater than a first preset similarity from different product images with similar known defects; the defect feature set includes known defect features extracted by the defect feature extraction model from known defect product images with known defects. The similarity unit is used to input the image of the product with the new defect into the defect feature extraction model to obtain the new defect feature; in the defect feature set, for the known defect features whose similarity with the new defect feature meets the preset similarity condition, the corresponding known defects are determined as candidate defects; The transfer unit is used to train the defect detection model corresponding to the identified candidate defects using the acquired new defect product image, so as to obtain a new defect detection model for detecting whether the input product image has a new defect.

14. A transfer device for a defect detection model, characterized in that, There are at least two known defects pre-set, as well as different defect detection models corresponding to each of the different known defects; The defect detection model is used to detect whether the input product image has a corresponding known defect; the device includes: The feature extraction unit is used to identify known defects similar to the new defect as candidate defects for a new defect product image with a new defect by feature extraction. The model acquisition unit is used to acquire a candidate defect detection model for detecting whether the input product image has the candidate defect; The transfer training unit is used to transfer train the acquired candidate defect detection models using images of new defective products, so as to obtain a new defect detection model for detecting whether the input product image has a new defect.

15. A transfer apparatus for a type detection model, characterized in that, At least two known types are pre-set, and different type detection models are pre-defined for each of the different known types; the type detection models are used to detect whether the input image conforms to the corresponding known type. The device includes: The type acquisition unit is used to acquire a pre-trained type feature extraction model and a set of type features; The type feature extraction model is used to extract different known type features with a similarity greater than a first preset similarity from different images of similar known types; the type feature set includes known type features extracted by the type feature extraction model for images that conform to known types; The new type unit is used to input images that conform to the new type into the type feature extraction model to obtain new type features; in the type feature set, for known type features whose similarity to the new type features meets the preset similarity conditions, the corresponding known types are determined as candidate types; The training unit is used to transfer train the type detection model corresponding to the determined candidate types using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

16. A transfer apparatus for a type detection model, characterized in that, There are at least two known defects pre-set, as well as different defect detection models corresponding to each of the different known defects; The defect detection model is used to detect whether the input product image has a corresponding known defect; the device includes: The candidate type unit is used to identify known types similar to the new type as candidate types for images that conform to the new type through feature extraction. The alternative model unit is used to obtain an alternative type detection model for detecting whether the input image conforms to the alternative type. The result unit is used to transfer train the acquired candidate type detection model using images that conform to the new type, so as to obtain a new type detection model for detecting whether the input image conforms to the new type.

17. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 12.

18. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1 to 12.

Images