Automatic defect labeling method and device, electronic equipment and readable storage medium

Abstract

The invention provides an automatic defect labeling method and device, electronic equipment and a readable storage medium, and the method comprises the steps: segmenting a specified defect for a target image based on a first model for an input reference image and an example of a reference image mask, and obtaining a target image prediction mask; and judging whether the prediction mask of the target image belongs to a real defect or not through a multi-layer perceptron, and reserving the part belonging to the real defect, thereby realizing automatic marking of the defect in the target image. Wherein the first model is obtained by adding a feature encoder in a context reference segmentation algorithm DINOV model, and the feature encoder is used for extracting the similarity of defects in the reference image and the target image. According to the method, the defects of the target image can be labeled in different industrial scenes, the problem of background over-killing in defect labeling is reduced, and meanwhile, the time required for subsequent manual defect labeling inspection is shortened.

Description

Technical Field

[0001] The present application relates to the field of automated labeling technology, and in particular to an automated defect labeling method, device, electronic device, and readable storage medium. Background Art

[0002] Traditional defect detection needs to be done manually, which is inefficient and prone to problems such as poor consistency in defect detection and high labor costs due to human factors. It is difficult to meet the requirements of modern industrial production.

[0003] With the development of industrial automation and the drive to reduce costs, automated defect detection technology has emerged. Automated defect detection technology follows the development of computer vision algorithms and mainly relies on rule-based, pattern recognition, and feature engineering. With the help of deep learning technology, deep learning models represented by convolutional neural networks have promoted breakthroughs in automated detection capabilities.

[0004] However, deep learning models need to rely on a large amount of labeled data to seek more robust generalization capabilities. Although the introduction of new technologies such as adaptive learning and small sample learning has reduced the dependence of deep models on data, there is still a long way to go before they can be used in practice. Therefore, how to quickly obtain accurate labeling information for large-scale data is still a practical problem that needs to be explored in the industrial field. Summary of the invention

[0005] The embodiments of the present application provide an automated defect annotation method, device, electronic device, and readable storage medium, which realizes automated annotation of defects in target images based on a modified context-referenced segmentation algorithm DINOV model and a multi-layer perceptron. The method can annotate defects in target images in different industrial scenarios, reduce the problem of background overkill in defect annotation, and reduce the time required for subsequent manual inspection of defect annotation.

[0006] In a first aspect, an automated defect annotation method is provided, which is applied to a first model, and the method includes:

[0007] Acquire a target image, a reference image, and a reference image mask, where the reference image mask is used to indicate a real defect in the reference image;

[0008] The target image, the reference image and the reference image mask are input into the first model for processing, and a target image prediction mask is output; wherein the target image prediction mask includes background overkill and real defects in the target image, and the target image prediction mask is determined based on the similarity between the target image and the real defects in the reference image, and the background overkill is a portion marked as a real defect in the image background;

[0009] Input the target image prediction mask into a multi-layer perceptron (MLP), and retain the target image prediction mask that the MLP determines to be a true defect, to obtain the true mask of the target image.

[0010] In a second aspect, an automated defect annotation device is provided, including:

[0011] An acquisition module, configured to acquire a target image, a reference image, and a reference image mask, where the reference image mask is used to indicate true defects in the reference image;

[0012] A prediction mask module, configured to input the target image, the reference image, and the reference image mask into a first model for processing, and output a target image prediction mask; wherein, the target image prediction mask includes background overkill and true defects in the target image, the target image prediction mask is determined based on the similarity between the target image and the true defects in the reference image, and the background overkill is the part marked as a true defect in the image background;

[0013] A screening mask module, configured to input the target image prediction mask into a multi-layer perceptron (MLP), and retain the target image prediction mask that the MLP determines to be a true defect, to obtain the true mask of the target image.

[0014] In a third aspect, an electronic device is provided, which includes one or more memories and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as shown in the foregoing first aspect.

[0015] In a fourth aspect, a computer-readable storage medium is provided, on which computer instructions are stored, and when the computer instructions are executed by a processor, the method as shown in the foregoing first aspect is implemented.

[0016] In a fifth aspect, a chip is provided, on which a computer program is stored, and when the computer program is executed by the chip, the method as shown in the foregoing first aspect is implemented.

[0017] The above-mentioned automated defect annotation method, device, electronic device and readable storage medium. The first model is based on a reference image and a reference image mask to segment defects in the target image, obtaining a predicted mask for the target image. Then, an MLP is used to determine whether the predicted mask of the target image belongs to a real defect, and the part belonging to the real defect is retained, realizing the automated annotation of defects in the target image. Among them, the first model is obtained by adding a feature encoder to the DINOV model, and this feature encoder is used to extract the similarity of defects in the reference image and the target image. The MLP is trained by distinguishing the features of background overkill and the features of the real defect, and this MLP supports distinguishing the real defect and the background overkill. In this way, the automated defect annotation method can annotate defects in the target image in different industrial scenarios, reduce the problem of background overkill in defect annotation, and at the same time reduce the time required for subsequent manual inspection of defect annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Schematic diagram of a DINOV model provided by an embodiment of the present application;

[0019] Figure 2 Schematic diagram of a similarity learning method provided by an embodiment of the present application;

[0020] Figure 3 Schematic diagram of a feature encoder network structure provided by an embodiment of the present application;

[0021] Figure 4 Flowchart of a method for training the first model to predict similarity provided by an embodiment of the present application;

[0022] Figure 5 Schematic diagram of a label discrimination process provided by an embodiment of the present application;

[0023] Figure 6 Schematic diagram of a multi-layer perceptron structure provided by an embodiment of the present application;

[0024] Figure 7 Flowchart of an automated defect annotation method provided by an embodiment of the present application;

[0025] Figure 8 Schematic diagram of a multi-layer perceptron training process provided by an embodiment of the present application;

[0026] Figure 9 Block diagram of the structure of an automated defect annotation device provided by an embodiment of the present application;

[0027] Figure 10 Schematic diagram of an electronic device provided by an embodiment of the present application;

[0028] Figure 11Internal structure diagram of a computer-readable storage medium provided by an embodiment of the present application. Detailed implementation manners

[0029] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The technical solutions in the embodiments of the present application will be clearly and exhaustively described below with reference to the drawings. Among them, in the description of the embodiments of the present application, unless otherwise specified, " / " means "or", for example, A / B may mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0030] Hereinafter, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as implying or suggesting relative importance or implicitly indicating the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0031] In the embodiments of the present application, the context reference segmentation algorithm (DETR with Improved deNoising AnchOr boxes V2, DINOV) is a technology that utilizes context information in the field of image segmentation to expand the model's capabilities and improve the segmentation accuracy. Among them, the context information includes the shallow texture appearance information, deep semantic information, position information of the object extracted in the image segmentation, and the environmental information around the target. Through the context information of the object, it can be judged whether the categories of the same object are consistent, and the context correlation within the same category of objects and between different categories of objects can also be learned to achieve a reliable feature representation of the object.

[0032] The goal of context reference segmentation is to enable the electronic device to understand different reference descriptions and segment the specified specific object from the image. Among them, the reference format includes text information and visual reference. For example, if it is desired that the model realizes the defect segmentation of the object in the image, the text reference of the defect can be provided to the model, or the image of the defect and the segmentation mask of the defect in the image can be provided. Although their forms are different, they both belong to the references that can be provided to the model. The former is a text reference, and the latter is a visual reference. The reference format in the embodiments of the present application specifically refers to the visual reference.

[0033] In the embodiments of the present application, the model adopting the DINOV algorithm is called the DINOV model. The DINOV model segments specific objects in an image or video based on reference information, and the reference information comes from specific objects and their corresponding segmentation masks in a single image or multiple images. The DINOV model is usually applied to segment specific objects in natural scenes, and often directly determines the state and position of the object through class consistency. However, in industrial scenarios, it is often difficult to directly determine the location of defects through class consistency. Specifically, for the same type of defect in different positions of the same object, the appearance may vary greatly. Therefore, the context information defined by the class consistency method is not applicable to industrial scenarios, and a large number of over-kill problems will occur in the DINOV model trained relying on this method.

[0034] It should be noted that in different industrial scenarios, the background of the target object in the image varies greatly, and the texture appearance in its background may be very similar to the real defect of the target object, and the two can often only be distinguished by prior art. The DINOV model can learn the consistency of context information, but there is no prior information to distinguish the background appearance from the real defect, so it is unable to distinguish the texture appearance in the background from the real defect, which affects the annotation ability of the DINOV model for real defects.

[0035] Based on the problems of the DINOV model when annotating defects, the embodiments of the present application propose an automated defect annotation method, device, electronic device and readable storage medium. This method inputs a reference image and an example of the mask of a specified defect in the image (i.e., the reference image mask) through a first model. The first model can segment the specified defect in the target image, that is, the first model outputs the predicted defect segmentation mask in the target image (i.e., the target image prediction mask). Then, a Multilayer Perceptron (MLP) is used to determine whether the target image prediction mask belongs to a real defect or background over-kill, and retain the part belonging to the real defect (i.e., the real mask of the target image), so as to realize the automated annotation of defects in the target image. Among them, background over-kill refers to the part in the background of the target image that is similar to the real defect and is mislabeled as a real defect. The first model is an improved context reference segmentation algorithm DINOV model, which adds a feature encoder on the basis of the DINOV model. The feature encoder is used to extract the similarity between the defects in the reference image and the defects in the target image, and set thresholds in different scenarios based on this similarity, so as to reduce the over-kill problem caused by loose judgment of the defects in the reference image and the defects in the target image. This method can annotate the defects of the target image in different industrial scenarios, reduce the problem of background over-kill in the defect annotation of the target image, and at the same time reduce the time required for subsequent manual inspection of defect annotations.

[0036] The following is through Figures 1 to 11To introduce an automated defect annotation method, apparatus, electronic device, and readable storage medium provided by an embodiment of the present application.

[0037] Embodiment 1

[0038] Figure 1 Shows a DINOV model. The DINOV model includes a visual feature encoder 110, a reference feature encoder 120, and a decoder 130. The visual feature encoder 110 receives the target image 101 and outputs the target image features corresponding to the target image 101; receives the reference image 102 and outputs the reference image features corresponding to the reference image 102. The reference feature encoder 120 receives the reference image mask 103 and the reference image features and outputs reference information, which is used to indicate the defects to be segmented in the image features. The decoder 130 outputs the target image prediction mask 104 based on the target image features and the reference information. Among them, the target image prediction mask 104 annotates the defects in the target image 101, including overkill in the background and real defects in the target image 101.

[0039] It should be noted that there are two reasons why the DINOV model cannot be directly used for on-site defect detection:

[0040] 1. The DINOV model is a general context segmentation large model based on the transformer learning framework. Its inference stage requires certain computing resources, and it is difficult to ensure sufficient computing resources in the actual work site. In addition, the inference speed of the DINOV model often cannot meet the requirements of the site. Therefore, the DINOV model is more suitable to be used as a pre-annotation model to annotate the defects in the on-site target images, reducing the annotation cost.

[0041] 2. The defect morphology in the actual work site often varies greatly from the defect images used for training the DINOV model. In this case, it is also difficult for the DINOV model to directly obtain inference results that meet the customer's requirements. However, by placing the inference process of the DINOV model before manual annotation and using the DINOV model as a pre-annotation model for defects, the time cost of subsequent annotation can be reduced while ensuring accuracy.

[0042] It should be noted that the goal of the embodiment of the present application is to pre-annotate industrial data through the improved DINOV model, reduce the annotation cost of manually annotated data. There are the following problems with directly using the DINOV model for defect annotation in industrial scenarios:

[0043] 1. In natural scenes, the reference image and the target image contain the same type of objects, and the same type of objects have similar morphological features, so the DINOV model can be directly trained. However, in industrial scenes, the same type of defects does not mean that they have similar morphological features. If only the same category is used as the training condition for the DINOV model, the context reference information of the DINOV model will be too broad. That is, in the face of two different types of defects with large morphological differences, training the DINOV model with the same category as the condition will make the DINOV model think that these two defects are similar enough, resulting in the DINOV model labeling defects that originally do not belong to the same category, or even parts that are not defects, as the same category of defects, that is, over-kill labeling occurs.

[0044] 2. For the trained DINOV model, it is difficult to adjust the reference standard of defects during the actual detection process because the parameters in the trained DINOV model remain unchanged, and the DINOV model cannot make adaptive adjustments to specific scenarios.

[0045] In summary, it is difficult to apply the DINOV model directly to defect labeling in industrial scenes. The embodiments of this application propose several improvements to apply the DINOV applied in natural scenes to industrial scenes, reducing the over-kill labeling problem of the DINOV model. The specific improvements are shown in Improvement 1 and Improvement 2 below. Among them, Improvement 1 shows an improvement scheme for the DINOV model, and Improvement 2 shows a scheme for reclassifying the mask output by the model.

[0046] Improvement 1

[0047] The embodiments of this application propose to add a feature encoder to the DINOV model framework. The feature encoder is used to obtain the similarity of context reference information. Among them, the similarity of context reference information refers to the similarity between the defects in the reference image and the defects in the target image. After the DINOV model learns to obtain this similarity, defects that are not similar enough can be filtered by a post-processing threshold. For example, defect labels with a similarity greater than the post-processing threshold are retained, and labeled defects with a similarity not greater than the post-processing threshold are removed. The post-processing threshold can be adjusted according to different requirements for defect labeling to reduce the over-kill labeling of the DINOV model. For example, in the case of strict requirements for context reference information, the processing threshold can be increased. Among them, the DINOV model with the added feature encoder can be called the first model, the post-processing threshold can be called the first threshold, the value range of the similarity includes 0 to 1, the value range of the first threshold includes 0-1, and the first threshold is used to filter the defects in the target image.

[0048] Figure 2A similarity learning method is shown. For the newly added feature encoder 140 in the first model, the feature encoder 140 calculates new features through learnable embedded features and input multi-scale image features via Masked Cross Attention calculation. Among them, the learnable embedded features are obtained by the feature encoder 140 during training, the multi-scale image features are sampled from the image features at different scales, and the Masked Cross Attention algorithm is improved from the Cross Attention algorithm. Cross Attention is a form of the Attention Mechanism, which is used to fuse features or information from two different sources. Its calculation formula is shown as follows:

[0049]

[0050] Among them, Q is the Query matrix, indicating the information of the target to be queried. K is the Key, and V is the Value, which are generally presented in the form of a Key-Value Pair. The Key-Value Pair indicates the information used to provide context reference. d k represents the matrix dimension of K. In the embodiments of the present application, Q is the learnable embedded feature, and the input image features obtain K and V through a linear layer.

[0051] Masked Cross Attention dynamically adjusts the Key-Value Pairs that can be accessed by the Query by introducing a masking matrix M, thereby restricting the attention range of the feature encoder 140. In the embodiments of the present application, M is an image segmentation mask, which is used to restrict the feature encoder 140 to focus on the real defects in the image and ignore the image features in the background. The calculation formula of Masked Cross Attention is shown as follows:

[0052]

[0053] In the embodiments of the present application, the feature encoder 140 extracts multi-scale image features through a backbone network to obtain deep features. The feature encoder 140 obtains K and V from the deep features through a linear layer, samples M to the same scale as the deep features, uses the learnable embedded features as Q, and substitutes the above parameters into the calculation formula of Masked Cross Attention to obtain the result. This result is used as Q of the new layer network, the image features in the new layer network and the corresponding M of the image features are obtained, and then substituted into the above formula for calculation again, repeating N times. Among them, N represents N-layer image features. For example, 3-layer image features need to be calculated 3 times to obtain image similarity features.

[0054] In the embodiment of the present application, the feature encoder 140 inputs the reference image feature and the reference image mask 103, and outputs the reference image similarity feature, which is used to indicate the defect of the reference image. The feature encoder 140 inputs the target image feature and the predicted mask 104 of the target image, and outputs the target image similarity feature, which is used to indicate the defect of the target image. The similarity (such as cosine similarity) between the reference image similarity feature and the target image similarity feature is calculated. Based on this similarity, the first model can change the strictness of the context reference through a first threshold, and adjust the target image mask output under different scenario requirements. For example, based on this similarity, the predicted mask 104 of the target image is determined, and the target image mask with a similarity greater than the first threshold is used as the predicted mask 104 of the target image, thereby effectively alleviating the overkill problem caused by overly loose context reference.

[0055] In the embodiment of the present application, the calculation methods of similarity include various methods for calculating feature similarity such as Euclidean distance and cosine similarity. Among them, using cosine similarity is easy for the first model to converge during training. Specifically, the embodiment of the present application uses cosine similarity as the result of calculating the similarity feature, and its calculation formula is as follows:

[0056]

[0057] Where A represents the reference image similarity feature, B represents the target image similarity feature, both A and B are feature matrices of N*C, and AB T is the matrix product, ||A|| represents the norm of the feature matrix A, which is a matrix of N*1, and cosine(A,B) represents a matrix of N*N. N and C indicate the dimensions of the matrix and are both positive integers.

[0058] In some embodiments, the calculation of similarity can be implemented by using the method of calculating feature similarity with Euclidean distance, and the embodiment of the present application does not limit this.

[0059] Based on Figure 2 shows the feature encoder 140, Figure 3 shows the network structure of the feature encoder 140.

[0060] In the embodiment of the present application, Figure 2The network structures of the shown feature encoder 140 and the reference feature encoder 120 are the same, including multiple layers of Masked Cross Attention. The first layer of Masked Cross Attention obtains image features and an image mask, and inputs the calculated result into the second layer of Masked Cross Attention to repeat the above calculation. By repeating the above calculation multiple times, image similarity features are obtained. Among them, the number of layers of Masked Cross Attention can be the same as the number of layers of the image features. For example, 3 layers of image features need to go through 3 Masked Cross Attention calculations to obtain image similarity features. The specific calculation method of Masked Cross Attention can be referred to the foregoing Figure 2 , which will not be elaborated here.

[0061] In the embodiment of the present application, before the first model obtains an accurate similarity, the model needs to be trained. The predicted similarity is obtained through the first model, and the loss function is calculated with the true similarity (Ground Truth). The result of the loss function is backpropagated to the first model and optimized until the similarity output by the first model reaches the accurate similarity. Among them, the specific method of training the first model to predict similarity is as Figure 4 shown.

[0062] Figure 4 shows the flow of a method for training the first model to predict similarity. The method includes the following steps:

[0063] S101. Calculate the cosine similarity between the reference image similarity feature and the target image similarity feature.

[0064] Among them, the reference image similarity feature and the target image similarity feature are output by the feature encoder. The calculation method of the cosine similarity between the two can be referred to the foregoing Figure 2 specific description, which will not be elaborated here.

[0065] S102. Based on the pre-trained encoder, calculate the true similarity between the reference image similarity feature and the target image similarity feature.

[0066] Specifically, the pre-trained encoder can be understood as an additional image encoder trained on a large-scale industrial data. Through this encoder, the reference image similarity is extracted from the same reference image, the target image similarity is extracted from the same target image, and the cosine similarity of the above two similarities is the true similarity. This true similarity is the target for training the first model to predict similarity. When the predicted similarity of the first model reaches this true similarity, this predicted similarity is the accurate similarity.

[0067] S103. Calculate the mean square error loss function between the cosine similarity and the true similarity.

[0068] Among them, the loss functions for calculating the cosine similarity and the true similarity include the Mean Absolute Error (MAE) loss function (also known as the L1 loss function), the Mean Square Error (MSE) loss function (also known as the L2 loss function), and the weighted cross-entropy loss function. The loss function adopted in the embodiments of this application is the MSE loss function, and its calculation formula is as follows:

[0069]

[0070] Among them, y represents the true similarity, represents the cosine similarity predicted by the first model, and N is a positive integer.

[0071] In some embodiments, the loss function can adopt calculation methods such as the MAE loss function and the weighted cross-entropy loss function, and the embodiments of this application do not limit this.

[0072] S104. Based on the result of the mean square error loss function, optimize the parameters of the first model until the cosine similarity between the defects in the reference image and the defects in the target image can be accurately predicted.

[0073] Among them, optimizing the parameters of the first model includes the parameters of the entire first model (i.e., including the visual feature encoder, reference feature encoder, decoder, feature encoder, etc.). The first model updates the parameters as the gradient propagates backward according to the result of the MSE loss function.

[0074] By the above steps S101 - S104, the first model can be trained to obtain an accurate cosine similarity, which is used to handle the situation where the same type of defect has different morphological features. Specifically, different first thresholds are set in different industrial scenarios, and the first model retains and outputs the target image mask with a cosine similarity greater than the first threshold, effectively reducing the overkill problem caused by loose context reference information.

[0075] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless specifically stated herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps or other steps.

[0076] In the embodiments of the present application, when there are texture regions in the background of the image that are similar to real defects, the model will label these texture regions in the background as real defects, which is called background overkill. To distinguish between background overkill and real defects, it is often necessary to rely on the prior information of the current image. For example, defects usually appear in certain regions of the material in the image, and the texture regions in the background that are similar to real defects belong to overkill. Prior information is usually bound to the scenario of the project where the image is located, and different projects have different prior information. Among them, the first model is a general model trained by mixing data from different projects, and prior information cannot be incorporated into the training stage of the model. Therefore, the embodiments of the present application propose a labeling discrimination stage to automatically distinguish the labels generated by the first model to reduce the problem of background overkill, as specifically shown in Improvement 2.

[0077] Improvement 2

[0078] Figure 5 Shows a labeling discrimination process. The context reference information of the defects in the reference example pair is extracted by the first model 100, and a prediction mask for the input target image 101 is generated based on this information. A multi-layer perceptron 150 is added outside the framework of the first model 100, and the multi-layer perceptron 150 distinguishes between real defects and background overkill. Specifically, the multi-layer perceptron 150 receives the prediction mask of the target image 101 output by the first model, retains the prediction mask determined to be a real defect, excludes the prediction mask determined to be background overkill, and the mask output by the multi-layer perceptron 150 is the real mask 106 of the target image. Among them, the reference example pair includes a reference image 102 and a reference image mask 103. For the specific description of the first model 100, reference can be made to the foregoing Figure 2 , which will not be elaborated here.

[0079] It should be noted that the multi-layer perceptron 150 needs to run with the support of a central processing unit (CPU) to distinguish real defects from background overkill, and the first model 100 needs to run with the support of a graphics processing unit (GPU) to label the defects in the target image 101. Therefore, the newly added multi-layer perceptron 150 classifies the prediction mask output by the first model 100, without additionally occupying the computing resources of the first model 100, and at the same time can effectively reduce the problem of background overkill.

[0080] Figure 6 Shows a multi-layer perceptron structure. As Figure 6 shown, the multi-layer perceptron includes an input layer, an output layer, and two fully connected layers. Among them, the nodes x1 and x2 in the input layer represent the features in the mask image. The nodes y1 and y2 in the output layer represent the judgment results of the features in the mask image. The parameters of the fully connected layer include weights and biases. The bias can be a preset parameter, and the weight is a trainable parameter. Taking the fully connected layer 1 as an example, the calculation expression is y = WX + b. Where W is the weight, b is the bias, and X is the feature in the mask image, including x1 and x2. X is a feature matrix of N * C1, N is the number of features in X, C1 is the dimension of the feature in the mask image, and the output y is a feature matrix of N * C2, C2 is the intermediate feature dimension, and the parameter of the fully connected layer 1 is C1 * C2. The parameter of the fully connected layer 2 is C2 * C3, and C3 is the dimension for distinguishing real defects from background overkill.

[0081] In the embodiment of the present application, the bias is set to 0, that is, b is 0, and the calculation expression can be y = WX. The intermediate feature dimension C2 is set to 256, the dimension for distinguishing real defects from background overkill by the dichotomy method, C3 is set to 2, and the fully connected layer 2 is set to a parameter of 256 * 2.

[0082] In the output layer, each node represents the judgment result of the feature in the output mask image. The fully connected hidden layer 1 and hidden layer 2 are used to perform non-linear calculations on the input image mask to learn to distinguish background overkill and real defects in the mask image.

[0083] As Figure 7 shown, the embodiment of the present application provides an automated defect annotation method, taking this method applied to Figure 5 the first model 100 in Figures 5 to 6 and the multi-layer perceptron 150 in

[0084] S201. Obtain a target image, a reference image, and a reference image mask, where the reference image mask is used to indicate real defects in the reference image.

[0085] In an embodiment of the present application, the automated defect annotation device may store a target image, a reference image, and a reference image mask, or may receive the input target image, reference image, and reference image mask and store them in the acquisition module of the automated defect annotation device.

[0086] S202. Input the target image, the reference image, and the reference image mask into the first model for processing, and output a target image prediction mask; wherein, the target image prediction mask includes overkill in the background and real defects in the target image, and the target image prediction mask is determined based on the similarity between the target image and the real defects in the reference image, and the overkill is the part marked as a real defect in the image background.

[0087] In an embodiment of the present application, the first model segments specified defects in the target image based on examples of the input reference image and the reference image mask, and obtains a target image prediction mask. Among them, the first model is obtained by adding a feature encoder to the DINOV model, and the feature encoder is used to extract the similarity between the defects in the reference image and the target image.

[0088] In some embodiments, the first model calculates the similarity between the reference image similarity feature and the target image similarity feature, calculates the loss function of the similarity and the real similarity, and performs backpropagation based on the loss function to optimize the first model until the first model can accurately predict the similarity between the real defects in the target image and the reference image. Among them, the real similarity is obtained by extracting features from a trained feature encoder and calculating the similarity between the reference image similarity feature and the target image similarity feature based on the extracted features; the first model is run by a graphics processing unit GPU.

[0089] It can be seen that in an embodiment of the present application, the first model can annotate the defects of the target image in different industrial scenarios, reducing the problem of overkill in the background during defect annotation.

[0090] S203. Input the target image prediction mask into the MLP, and retain the target image prediction mask determined by the MLP as the real defect to obtain the real mask of the target image.

[0091] In an embodiment of the present application, the MLP determines whether the target image prediction mask belongs to a real defect, and retains the part belonging to the real defect to realize the automated annotation of the defects in the target image.

[0092] In some embodiments, the MLP is trained by sampling the features of the overkill in the background and the features of the real defects and inputting them into the MLP until the MLP supports distinguishing the real defect from the overkill in the background; wherein, the MLP is constructed by two fully connected layers and is run by a central processing unit CPU.

[0093] It can be seen that in the embodiments of the present application, the method of combining the first model with the MLP to obtain the true mask of the target image can further reduce the problem of over-killing the background in defect annotation, and at the same time reduce the time required for subsequent manual inspection of defect annotation.

[0094] In some embodiments, the first model includes a visual feature encoder, a reference feature encoder, a feature encoder, and a decoder. The target image, the reference image, and the reference image mask are input into the first model for processing, and a target image prediction mask is output, including:

[0095] Based on the target image and the reference image, the visual feature encoder respectively obtains a target image feature and a reference image feature;

[0096] Based on the reference image mask and the reference image feature, the reference feature encoder obtains reference information;

[0097] Based on the target image feature and the reference information, the decoder obtains a target image prediction mask;

[0098] Based on the reference image feature and the reference image mask, the feature encoder obtains a reference image similarity feature; the reference image similarity feature is used to indicate the defects of the reference image;

[0099] Based on the target image feature and the target image prediction mask, the feature encoder obtains a target image similarity feature; the target image similarity feature is used to indicate the defects of the target image;

[0100] Calculate the similarity between the reference image similarity feature and the target image similarity feature, and retain the target image prediction mask whose similarity is greater than the first threshold; the first threshold is used to filter the defects in the target image.

[0101] In some embodiments, the network structures of the feature encoder and the reference feature encoder are the same; the feature encoder obtains the similarity feature of the image based on the masked cross-attention algorithm through the learnable embedding feature and the image feature.

[0102] It should be understood that although the steps in the flowcharts involved in the above embodiments are sequentially shown according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0103] Based on Figure 7 the multi-layer perceptron training method shown in step S203 Figure 8 shows a multi-layer perceptron training process. During the training process of the multi-layer perceptron 150, the context reference information of the defects in the reference example pair is extracted by using the first model 100 (that is, the reference image 102 and the reference image mask 103 are input into the first model), and based on this information, a reference image prediction mask 105 is generated for the re-input reference image 102 (that is, the reference image 102 is also input into the first model again as the target image to obtain the reference image prediction mask 105). Among them, the reference example pair includes the reference image 102 and the reference image mask 103, and the reference image prediction mask 105 includes the background overkill and the real defects in the reference image 102. By comparing the reference image prediction mask 105 with the reference image mask 103, the real defects and the background overkill in the reference image prediction mask 105 are distinguished, and the features of the real defects and the features of the background overkill are sampled and input into the multi-layer perceptron 150 for training until the multi-layer perceptron 150 can correctly distinguish the real defects and the background overkill.

[0104] It should be noted that the training duration of the multi-layer perceptron 150 is basically controlled within 10s. Compared with the first model 100 with a training duration of several hours, the computing power cost and time cost of training the multi-layer perceptron 150 are less, but it can effectively reduce the problem of background overkill.

[0105] In the embodiments of the present application, multiple industrial project data are labeled, and the overkill index, missed detection index, and labeling time cost of the scheme of using the DINOV model alone, improvement 1 scheme (the scheme of improving the DINOV model into the first model), and improvement 1 plus improvement 2 scheme (the scheme of using the first model and multi-layer perceptron classification) are compared. The specific detected values are shown in Table 1.

[0106] Table 1

[0107]

[0108]

[0109] In Table 1, the overkill (or missed detection) metric is the percentage of the number of overkill (or missed detection) instances in the total number of predicted instances. The total number of instances is different from the total number of images. The former refers to the total number of all defects present in all images. The annotation duration is the time cost, including the time for automatic annotation and the time for subsequent manual correction of automatic annotation errors. In the table, h represents hours.

[0110] As can be observed from Table 1, both the improvement 1 scheme and the improvement 2 scheme proposed in the embodiments of the present application can effectively reduce the overkill of the DINOV model in annotating target images and maintain a low level of missed detection. At the same time, implementing the improvement 1 plus improvement 2 scheme can maximize the avoidance of the overkill problem of the DINOV model. Although the final output results of the scheme inevitably have overkill and missed detection, through the annotation of the first model and the MLP, it saves a lot of energy and time for subsequent manual inspection and annotation modification. Finally, compared with the method of using only the DINOV model, implementing the improvement 1 and improvement 2 schemes simultaneously can reduce the annotation duration by nearly 70%, especially in some scenarios where defects appear densely, more defect annotation time can be saved.

[0111] In summary, the scheme for the DINOV model to learn the context reference information similarity and the scheme for distinguishing the output annotation of the first model proposed in the embodiments of the present application overcome the two problems of the relatively high overkill of the DINOV model in annotating defects in industrial scenarios and the relatively long annotation time.

[0112] In some other embodiments, the DINOV model and the improvement 1 scheme can be used alone to annotate defects, or the DINOV model and the improvement 2 scheme can be used alone to annotate defects. The embodiments of the present application do not limit the specific implementation manners of the above improvement 1 and improvement 2.

[0113] Embodiment 2

[0114] As Figure 9 shown, an automated defect annotation device 900 provided by an embodiment of the present application includes:

[0115] An acquisition module 901, configured to acquire a target image, a reference image, and a reference image mask, where the reference image mask is used to indicate true defects in the reference image;

[0116] A prediction mask module 902 is configured to process the target image, the reference image, and the reference image mask through a first model, and output a target image prediction mask. The target image prediction mask includes background overkill and real defects in the target image. The target image prediction mask is determined based on the similarity between the target image and the real defects in the reference image. The background overkill refers to the parts marked as real defects in the image background.

[0117] A screening mask module 903 is configured to input the target image prediction mask into a multi-layer perceptron (MLP), retain the target image prediction mask determined by the MLP as the real defect, and obtain the real mask of the target image.

[0118] In some embodiments, the first model includes a visual feature encoder, a reference feature encoder, a feature encoder, and a decoder. In processing the target image, the reference image, and the reference image mask through the first model to output a target image prediction mask, the prediction mask module 902 is specifically configured to:

[0119] Based on the target image and the reference image, obtain a target image feature and a reference image feature respectively through the visual feature encoder.

[0120] Based on the reference image mask and the reference image feature, obtain reference information through the reference feature encoder.

[0121] Based on the target image feature and the reference information, obtain a target image prediction mask through the decoder.

[0122] Based on the reference image feature and the reference image mask, obtain a reference image similarity feature through the feature encoder. The reference image similarity feature is used to indicate the defects of the reference image.

[0123] Based on the target image feature and the target image prediction mask, obtain a target image similarity feature through the feature encoder. The target image similarity feature is used to indicate the defects of the target image.

[0124] Calculate the similarity between the reference image similarity feature and the target image similarity feature, and retain the target image prediction mask with the similarity greater than a first threshold. The first threshold is used to filter the defects in the target image.

[0125] In some embodiments, before the screening mask module 903 inputs the target image prediction mask output by the prediction mask module 902 into the multi-layer perceptron (MLP), retains the target image prediction mask determined by the MLP as the real defect, and obtains the real mask of the target image, the screening mask module 903 needs to train the MLP. In the training process of the MLP, the screening mask module 903 is specifically configured to:

[0126] Input the reference image and the reference image mask into the first model. The reference image is also input into the first model again as the target image to obtain a reference image prediction mask. The reference image prediction mask includes background overkill and real defects in the reference image.

[0127] Compare the reference image mask and the reference image prediction mask to distinguish the background overkill and the real defects.

[0128] Sample the features of the background overkill and the features of the real defects respectively and input them into the MLP for training until the MLP can support distinguishing the real defects and the background overkill. Among them, the MLP is constructed by two fully connected layers and runs through the central processing unit CPU.

[0129] Each module in the above-mentioned automatic defect annotation device 900 can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in the processor in the computer device in the form of hardware or be independent of it, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0130] Embodiment 3

[0131] As Figure 10 shows the structure of an electronic device provided by an embodiment of the present application. The electronic device 1000 includes: a processor 1001, a memory 1002, and a computer program 1003 stored in the above-mentioned memory 1002 and executable on the above-mentioned processor 1001. When the above-mentioned processor 1001 executes the above-mentioned computer program 1003, the method shown in the above Figures 1 - 9 is implemented.

[0132] Exemplarily, the above-mentioned computer program 1003 can be divided into one or more units / modules. The above-mentioned one or more units / modules are stored in the above-mentioned memory 1002 and executed by the above-mentioned processor 1001 to complete the present application. The above-mentioned one or more units / modules can be a series of computer program instruction segments capable of completing specific functions, and these instruction segments are used to describe the execution process of the above-mentioned computer program 1003 in the above-mentioned electronic device 1000.

[0133] For example, the above-mentioned computer program 1003 can be used to execute the method for calculating similarity by the feature encoder in the foregoing Figure 2 and can also be used to execute the method for training the feature encoder shown in steps S101 - S104 in the foregoing Figure 4 . The specific functions or mechanisms have been described in the above-mentioned embodiments and will not be elaborated here.

[0134] The above-mentioned electronic device 1000 may include, but is not limited to, a processor 1001 and a memory 1002. Those skilled in the art can understand that Figure 10 These are merely examples of the electronic device 1000 and do not constitute a limitation on the electronic device 1000. It may include more or fewer components than those shown in the figure, or combine certain components, or different components. For example, the above-mentioned electronic device 1000 may also include input / output devices, network access devices, a bus, etc.

[0135] The so-called processor 1001 may be a CPU, or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0136] The above-mentioned memory 1002 may be an internal storage unit of the electronic device 1000, such as the hard disk or memory of the electronic device 1000. The above-mentioned memory 1002 may also include both the internal storage unit of the above-mentioned electronic device 1000 and external storage devices.

[0137] The above-mentioned memory 1002 is used to store the above-mentioned computer program and other programs and data required by the above-mentioned electronic device 1000. The above-mentioned memory 1002 may also be used to temporarily store data that has been output or will be output, such as image data (including target images, reference images, reference image masks, target image masks, etc.) and the target image annotation results output by the MLP, etc.

[0138] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above-mentioned division of each functional unit and module is used as an example. In actual applications, the above-mentioned functions may be allocated to different functional units and modules according to needs, that is, the internal structure of the above-mentioned device is divided into different functional units or modules to complete all or part of the functions described above.

[0139] Each functional unit and module in the embodiments may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0140] In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction and are not used to limit the protection scope of this application.

[0141] Those skilled in the art can understand that Figure 10 the structure shown in is only a block diagram of some structures related to the solution of this application and does not constitute a limitation on the electronic device to which the solution of this application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0142] It should be understood that each step in the above method embodiments provided by this application can be completed by the integrated logic circuit of the hardware in the processor or the instructions in the form of software. The method steps disclosed in combination with the embodiments of this application can be directly embodied as being executed and completed by the hardware processor, or executed and completed by the combination of the hardware and software modules in the processor.

[0143] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in any one of the foregoing embodiments are implemented.

[0144] This application also provides a computer program product. The computer program product includes: a computer program (which can also be referred to as code or instructions). When the computer program is run, the computer is enabled to execute the method in any one of the foregoing embodiments.

[0145] As Figure 11 shown, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program (which can also be referred to as code or instructions). When the computer program is run, the computer is enabled to execute the method in any one of the foregoing embodiments.

[0146] This application also provides a chip system. The chip system includes at least one processor for implementing the functions involved in the method executed by the electronic device in any one of the foregoing embodiments.

[0147] In a possible design, the chip system further includes a memory for storing program instructions and data. The memory is located inside or outside the processor.

[0148] The chip system can be composed of chips or can include chips and other discrete devices.

[0149] In some embodiments, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor that implements by reading software code stored in a memory.

[0150] In some embodiments, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately provided from the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transitory processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or may be separately provided on different chips. The embodiments of the present application do not specifically limit the type of the memory and the setting manner of the memory and the processor.

[0151] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processing circuit (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0152] The various embodiments of the present application can be combined arbitrarily to achieve different technical effects. For the sake of concise description, not all possible combinations of the various technical features in the above embodiments are described. However, as long as the combinations of these technical features do not conflict, they should be considered to be within the scope described in this specification.

[0153] Those of ordinary skill in the art can understand all or part of the processes in the foregoing method embodiments. These processes can be completed by relevant hardware instructed by a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the foregoing method embodiments. The foregoing storage medium includes: various media such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store program codes.

[0154] In summary, the above description is only an embodiment of the technical solution of this application and is not intended to limit the protection scope of this application. Any modifications, equivalent replacements, improvements, etc. made in accordance with the disclosure of this application shall be included within the protection scope of this application.

Claims

1. An automated defect marking method, characterized in that: Applied to the first model, the method comprises: Acquire a target image, a reference image, and a reference image mask, wherein the reference image mask is used to indicate a real defect in the reference image; Input the target image, the reference image and the reference image mask into the first model for processing, and output a target image prediction mask; wherein the target image prediction mask includes background overkill and real defects in the target image, and the target image prediction mask is determined based on the similarity between the target image and the real defects in the reference image, and the background overkill is the part marked as a real defect in the image background; The target image prediction mask is input into a multi-layer perceptron MLP, and the target image prediction mask judged by the MLP as the real defect is retained to obtain the real mask of the target image.

2. The method according to claim 1, characterized in that: The first model includes a visual feature encoder, a reference feature encoder, a feature encoder and a decoder, and the target image, the reference image and the reference image mask are input into the first model for processing, and the target image prediction mask is output, including: Based on the target image and the reference image, obtaining target image features and reference image features respectively through the visual feature encoder; Based on the reference image mask and the reference image feature, obtaining reference information through the reference feature encoder; Based on the target image features and the reference information, obtaining a target image prediction mask through the decoder; Based on the reference image feature and the reference image mask, obtaining a reference image similarity feature through the feature encoder; the reference image similarity feature is used to indicate a defect of the reference image; Based on the target image feature and the target image prediction mask, a target image similarity feature is obtained through the feature encoder; the target image similarity feature is used to indicate a defect of the target image; The similarity between the reference image similarity feature and the target image similarity feature is calculated, and the target image prediction mask whose similarity is greater than a first threshold is retained; the first threshold is used to filter defects in the target image.

3. The method according to claim 2, characterized in that: The feature encoder and the reference feature encoder have the same network structure; the feature encoder obtains image similarity features based on a masked cross attention algorithm through learnable embedded features and image features.

4. The method according to claim 2, characterized in that: After calculating the similarity between the reference image similarity feature and the target image similarity feature and retaining the target image prediction mask having the similarity greater than a first threshold, the method further includes: Calculating a loss function of the similarity and the true similarity, performing back propagation based on the loss function, and optimizing the first model until the first model can accurately predict the similarity of the true defects in the target image and the reference image; The true similarity is obtained by extracting features from a trained feature encoder, and calculating the similarity between the reference image similarity feature and the target image similarity feature based on the extracted features; and the first model is run by a graphics processor GPU.

5. The method according to claim 4, characterized in that: The loss function includes a mean square error (MSE) loss function, and the MSE loss function is calculated using the following formula: Wherein, y is the true similarity, is the similarity predicted by the first model, and N is a positive integer.

6. The method according to claim 2, characterized in that: The similarity includes cosine similarity, and the cosine similarity is calculated using the following formula: Among them, A is the similarity feature of the reference image, and B is the similarity feature of the target image.

7. The method according to any one of claims 1 to 6, characterized in that: Before inputting the target image prediction mask into a multi-layer perceptron MLP, retaining the target image prediction mask judged by the MLP as the real defect, and obtaining the real mask of the target image, the method further includes: Inputting the reference image and the reference image mask into the first model, and inputting the reference image into the first model again as the target image to obtain a reference image prediction mask; the reference image prediction mask includes background overkill and real defects in the reference image; Comparing the reference image mask with the reference image prediction mask to distinguish the background overkill from the real defect; The features of the background overshoot and the features of the real defects are sampled respectively and input into the MLP training until the MLP supports distinguishing the real defects from the background overshoot; wherein the MLP is constructed by two fully connected layers and runs through a central processing unit CPU.

8. An automated defect marking device, characterized in that: include: An acquisition module, used for acquiring a target image, a reference image and a reference image mask, wherein the reference image mask is used for indicating a real defect in the reference image; A prediction mask module, configured to input the target image, the reference image and the reference image mask into a first model for processing, and output a target image prediction mask; wherein the target image prediction mask includes background overkill and real defects in the target image, and the target image prediction mask is determined based on the similarity between the target image and the real defects in the reference image, and the background overkill is a portion marked as a real defect in the image background; The screening mask module is used to input the target image prediction mask into a multi-layer perceptron MLP, retain the target image prediction mask judged by the MLP as the real defect, and obtain the real mask of the target image.

9. An electronic device, comprising one or more memories and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions so that the electronic device performs the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images