Surface defect anomaly detection method and device and electronic equipment

Abstract

The invention relates to a surface defect anomaly detection method and device and electronic equipment, and the method comprises the steps: obtaining an object image of a to-be-detected object; the object image is input into a surface defect anomaly detection model, the surface defect anomaly detection model comprises a shared feature extraction network and a sub-model set, the shared feature extraction network is used for extracting image features of the object image, and the sub-model set comprises a plurality of sub-models, each sub-model is used for processing the image features to obtain corresponding abnormal confidence; acquiring a plurality of abnormal confidence coefficients output by the surface defect anomaly detection model, and processing the plurality of abnormal confidence coefficients in a weighted voting mode to obtain defect anomaly confidence coefficients; and detecting whether the surface of the to-be-detected object has defects or not based on the defect anomaly confidence so as to improve the generalization ability of the model.

Description

Technical Field

[0001] This application relates to the field of anomaly detection technology, and in particular to a method, apparatus and electronic device for detecting surface defects, which can be applied to the detection of surface defects in industrial products. Background Technology

[0002] In related technologies, deep learning-based detection techniques are widely used for industrial surface defect detection. After a deep learning model is deployed to a production line, its performance degrades due to environmental changes, equipment aging, and process fluctuations. Continuous online and offline adjustments and iterations are needed to improve the model's fitting and generalization capabilities. However, existing methods for improving model fitting and generalization capabilities have limitations, such as being only applicable to small-scale adjustments and having difficulty controlling the accuracy of these adjustments. Summary of the Invention

[0003] This application provides a surface defect anomaly detection method, apparatus, and electronic device to improve the generalization ability of the model.

[0004] In a first aspect, embodiments of this application provide a method for detecting surface defect anomalies, including:

[0005] Obtain the object image of the object to be detected;

[0006] The object image is input into a surface defect anomaly detection model, which includes a shared feature extraction network and a sub-model set. The shared feature extraction network is used to extract image features from the object image, and the sub-model set includes multiple sub-models. Each sub-model is used to process the image features to obtain a corresponding anomaly confidence level.

[0007] Multiple anomaly confidence scores output by the surface defect anomaly detection model are obtained, and the multiple anomaly confidence scores are processed by a weighted voting method to obtain the defect anomaly confidence score.

[0008] Based on the defect anomaly confidence level, the presence of defects on the surface of the object to be tested is detected.

[0009] Secondly, embodiments of this application provide a surface defect anomaly detection device, comprising:

[0010] The image acquisition module is used to acquire the object image of the object to be detected;

[0011] The model detection module is used to input the object image into the surface defect anomaly detection model. The surface defect anomaly detection model includes a shared feature extraction network and a sub-model set. The shared feature extraction network is used to extract image features of the object image. The sub-model set includes multiple sub-models. Each sub-model is used to process the image features to obtain the corresponding anomaly confidence.

[0012] The weighted voting module is used to obtain multiple anomaly confidence scores output by the surface defect anomaly detection model, and to process the multiple anomaly confidence scores through a weighted voting method to obtain the defect anomaly confidence score.

[0013] The result output module is used to detect whether there are defects on the surface of the object to be tested based on the defect anomaly confidence level.

[0014] Thirdly, embodiments of this application provide an electronic device, including:

[0015] One or more processors;

[0016] The processor is used to invoke instructions to cause the electronic device to perform the method described in the first aspect above.

[0017] Fourthly, embodiments of this application provide a storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect above.

[0018] Fifthly, embodiments of this application provide a program product, including at least one of a program and instructions, wherein when the program and instructions are executed by an electronic device, they implement the steps of the method described in the first aspect.

[0019] According to the technical solution of this application, a surface defect anomaly detection model is constructed by a shared feature extraction network and multiple sub-models. The generalization ability of the model can be improved by multiple sub-models. Multiple sub-models share a shared feature extraction network, which can reduce the computational power requirement of the surface defect anomaly detection model. The detection accuracy can be improved by combining the weighted voting of the outputs of multiple sub-models.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1This is a flowchart illustrating a surface defect anomaly detection method according to an exemplary embodiment.

[0023] Figure 2 This is a block diagram illustrating a surface defect anomaly detection model according to an exemplary embodiment.

[0024] Figure 3 This is a flowchart illustrating a surface defect anomaly detection method according to another exemplary embodiment.

[0025] Figure 4 This is a flowchart illustrating a surface defect anomaly detection method according to yet another exemplary embodiment.

[0026] Figure 5 This is a flowchart illustrating a surface defect anomaly detection method according to yet another exemplary embodiment.

[0027] Figure 6 This is a block diagram of a surface defect anomaly detection device according to an exemplary embodiment.

[0028] Figure 7 This is a block diagram illustrating an apparatus for detecting surface defect anomalies according to an exemplary embodiment. Detailed Implementation

[0029] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0030] The embodiments described in this application are not exhaustive, but merely illustrative of some embodiments, and are not intended to limit the scope of protection of this application. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined with each other. For example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0031] In each embodiment of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0032] It should be noted that the acquisition, transmission, storage, use, and processing of data in this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0033] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0034] It is worth noting that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0035] Among related technologies, manual intervention is currently the most widely adopted online model adjustment solution. This involves manually adjusting the model's discrimination threshold and other parameters to restore or improve the model's detection accuracy. Another approach is online model adjustment based on adaptive dynamic adjustment. This utilizes dynamic thresholding technology to dynamically adjust model confidence and discrimination thresholds based on current data features without retraining the model, achieving dynamic model adaptation. Furthermore, when the model's detection bias is too large, iterative offline model updates are used. This involves collecting newly produced data from the production line, manually labeling it, and then retraining the model to improve its performance in new scenarios. The generalization ability of the iterated model is generally further enhanced.

[0036] While the aforementioned online model adjustment schemes can quickly and easily respond to production changes, the adaptability of current manual and adaptive adjustment methods is limited, failing to adapt to scenarios with large deviations beyond the model's generalization capabilities. Furthermore, threshold adjustment methods require high precision, which is difficult for general production line workers to perform, while adaptive adjustment lacks effective evaluation methods, potentially leading to further performance degradation due to adjustment errors. Offline iterative model methods offer stronger generalization capabilities for fitting new data, but cannot respond to real-time production line changes, and require relatively high data, computing power, and technical costs. In iterative training methods, full training using historical data suffers from data storage and training efficiency bottlenecks, while continuous learning and online learning techniques without historical data carry the risk of catastrophic forgetting due to overfitting to new data.

[0037] Based on this, embodiments of this application propose a surface defect anomaly detection method, apparatus, and electronic device, which can improve the generalization ability of the model, increase the detection accuracy, and enable rapid and accurate adjustment of the model over a wider range.

[0038] The following description, with reference to the accompanying drawings, describes a surface defect anomaly detection method, apparatus, and electronic device according to embodiments of this application.

[0039] It should be noted that the execution subject of the surface defect anomaly detection method in this application embodiment can be a surface defect anomaly detection device. The device can be implemented by software and / or hardware. The device can be configured in an electronic device, which may include, but is not limited to, a terminal, a server, etc.

[0040] Figure 1 This is a flowchart illustrating a surface defect anomaly detection method according to an exemplary embodiment. For example... Figure 1 As shown, the surface defect anomaly detection method may include, but is not limited to, the following steps 101-104.

[0041] In step 101, the object image of the object to be detected is obtained.

[0042] In some embodiments, an image acquisition device can be used to acquire an image of the object to be inspected on the production line. As an example, the image acquisition device is a camera, which transmits the acquired object image to the surface defect anomaly detection system.

[0043] In step 102, the object image is input into the surface defect anomaly detection model. The surface defect anomaly detection model includes a shared feature extraction network and a sub-model set. The shared feature extraction network is used to extract image features from the object image. The sub-model set includes multiple sub-models, each of which is used to process the image features to obtain the corresponding anomaly confidence.

[0044] In some embodiments, the shared feature extraction network may be a pre-trained deep neural network, such as an optional transformer network, convolutional neural network, etc.

[0045] In some embodiments, such as Figure 3 As shown, the sub-model includes an attention module and an anomaly detection module. The attention module extracts key features from the image, while the anomaly detection module performs anomaly detection based on these key features to obtain the corresponding anomaly confidence score. As an example, the attention module can be a Squeeze-Excitation network module, a CBAM network module, etc., and the anomaly detection module can be implemented using a Normalizing Flow method. Thus, the object image is processed by a shared feature extraction network to obtain image features. These image features are then processed by the attention module to obtain key features Fkey, increasing the weight of high-value features and maintaining dimensionality consistency between the image features and the key features Fkey. The key features Fkey are then processed by the anomaly detection module to calculate the anomaly confidence score for each region, thereby achieving surface defect anomaly detection based on this confidence score.

[0046] In some embodiments, an object image is input into a surface defect anomaly detection model to obtain the anomaly confidence level of each pixel in the object image.

[0047] In this embodiment, the anomaly confidence level is used to indicate the probability of an anomaly.

[0048] In some embodiments, the number of layers in the sub-model is much smaller than the number of layers in the shared feature extraction network, which can effectively reduce the storage and computing power pressure in multi-model scenarios.

[0049] In step 103, multiple anomaly confidence scores output by the surface defect anomaly detection model are obtained, and the multiple anomaly confidence scores are processed by weighted voting to obtain the defect anomaly confidence score.

[0050] In some embodiments, the weights of each sub-model in the sub-model set are obtained. Based on the weights of each sub-model, multiple anomaly confidence scores are processed using a weighted voting method to obtain the defect anomaly confidence score. As an example, the weighted voting can be implemented using a soft voting weighted average mechanism.

[0051] In some embodiments, the weights of each sub-model can be determined when determining the set of sub-models, such as based on the detection results of each sub-model on the first object.

[0052] In some embodiments, multiple outlier confidence scores correspond to the outlier confidence scores of each pixel in the object image output by multiple sub-models. The multiple outlier confidence scores are processed by a weighted voting method to obtain the final outlier confidence scores of each pixel in the object image.

[0053] In step 104, based on the defect anomaly confidence level, the presence of defects on the surface of the object to be inspected is detected.

[0054] In some embodiments, the anomaly confidence level of each pixel in the object image is compared with a preset anomaly confidence level threshold. If there is at least one pixel greater than the anomaly confidence level threshold, it is determined that the object to be detected has a surface defect, and the anomaly region of the object to be detected is obtained based on the at least one pixel greater than the anomaly confidence level threshold. If there is no pixel greater than the anomaly confidence level threshold, it is determined that the object to be detected does not have a surface defect.

[0055] In the above embodiments, a surface defect anomaly detection model is constructed by a shared feature extraction network and multiple sub-models. The generalization ability of the model can be improved by using multiple sub-models. Multiple sub-models share a shared feature extraction network, which can reduce the computational power requirements of the surface defect anomaly detection model. The detection accuracy can be improved by combining weighted voting on the outputs of multiple sub-models. By using sub-models with a network layer number much smaller than that of the shared feature extraction network, the storage and computational power pressure in multi-model scenarios can be effectively reduced.

[0056] To enable rapid and accurate adjustments to the surface defect anomaly detection model deployed on the production line, this application trains a sub-model library comprising multiple sub-models. This allows for timely and rapid selection of a sub-model set for production line deployment from the sub-model library based on detection results, and subsequent adjustments to the current sub-model set. The following provides a detailed explanation of how the first-piece inspection stage selects a sub-model set from the sub-model library for production line deployment to obtain the surface defect anomaly detection model. In some embodiments, based on any one or more combined embodiments described above, such as... Figure 3 As shown, the surface defect anomaly detection method may include, but is not limited to, the following steps 301-305.

[0057] In step 301, the detection results of the first object by the human and the detection results of the first object by each sub-model in the sub-model library are obtained.

[0058] One approach is to perform a manual first-piece inspection after each shift starts or changes. This involves manually inspecting multiple first-piece objects for surface defects. If defects are found, the defect area can be identified, resulting in a manual inspection result. This result is then input into the surface defect anomaly detection system. In other words, the manual inspection result for the first-piece object can include both the presence of surface defects and the location of defects.

[0059] As one implementation method, object images of products of different models and batches, along with corresponding surface defect detection results, are used as training samples to train a set of detection models. Each detection model includes a feature extraction network and a sub-model. The feature extraction network uses a pre-trained deep neural network, which is frozen during training; the model parameters of the deep neural network are not optimized, only the sub-models participate in training. Based on the trained set of detection models, a set of trained sub-models is obtained, forming a sub-model library. In this embodiment, the obtained set of detection models is used to inspect the first object, obtaining the anomaly confidence level output by each sub-model in the sub-model library. The detection results of each sub-model for the first object can be obtained from the anomaly confidence level output by each sub-model and the stored decision boundaries of each sub-model. The method for obtaining the decision boundaries of each sub-model will be described in detail in later steps.

[0060] In step 302, based on the comparison between the detection results of the first object by the human and the detection results of each sub-model in the sub-model library, the first candidate sub-model set with consistent detection results is selected from the sub-model library.

[0061] In this embodiment, the detection results of each sub-model on the first object are compared with the detection results of a human on the first object, and the first candidate model set with consistent detection results is selected.

[0062] In some embodiments, there are multiple first objects. For all first objects, the detection results by the human operator and the detection results of the sub-model must be consistent before the sub-model can be added to the first candidate model set.

[0063] In step 303, for each sub-model in the first candidate sub-model set, the first-piece evaluation information obtained by the sub-model when processing the image features of the first object is acquired; and the similarity between the first-piece evaluation information of the sub-model and its initial evaluation information is calculated to obtain the similarity of the sub-model; wherein, the initial evaluation information of the sub-model is obtained by processing multiple training samples after the sub-model is trained.

[0064] In some embodiments, the initial evaluation information for the sub-model includes a sparse feature set and a first confidence distribution.

[0065] In some embodiments, feature images corresponding to multiple training samples are input into the attention module of the trained sub-model to obtain multiple key features; feature aggregation sampling is performed on the multiple key features to obtain a sparse feature set of the sub-model.

[0066] In some embodiments, feature images corresponding to multiple training samples are input into a trained sub-model to obtain the abnormal confidence scores of each pixel in the object images of the multiple training samples; based on the abnormal confidence scores of each pixel in the object images of each training sample, a confidence distribution corresponding to each training sample is obtained; the confidence distributions of multiple training samples are concatenated to obtain the first confidence distribution of the sub-model. The first confidence distribution includes a first normal confidence distribution and a first abnormal confidence distribution.

[0067] For example, after training the sub-model, the entire historical training data is input into the sub-model to obtain the key feature groups {F} corresponding to all training data. key}, for the key feature group {F key Feature aggregation sampling is performed, and a core set based on class centers is calculated using k-means clustering, which constitutes the sparse feature set {F} of the sub-model. core The data is then stored. Feature aggregation sampling can be implemented using the Greedy Coreset Sample method. The confidence distributions of multiple training samples are concatenated to obtain the first normal confidence distribution and the first abnormal confidence distribution of the sub-model.

[0068] One possible implementation is to visualize the anomaly confidence scores of each pixel in an object image as a heatmap-like format, using the heatmap to determine normal and anomaly regions. The confidence scores of the normal and anomaly regions are then statistically analyzed to obtain a first normal confidence score distribution and a first anomaly confidence score distribution. Based on these distributions, a confidence score distribution curve can be formed, with the horizontal axis representing confidence score and the vertical axis representing the number of pixels. This curve includes both normal and anomaly confidence score distributions. Furthermore, the normal distribution boundary T2 and the anomaly distribution boundary T1 can be determined based on these curves. These boundaries are then used as the normal and anomaly decision boundaries for the sub-model, and similarity scores can be obtained based on these boundaries.

[0069] It should be noted that the confidence level of normal regions is low, while that of abnormal regions is high. Normal and abnormal regions generally overlap (because the model used to divide normal and abnormal regions has errors, making it difficult to distinguish between them). When the model is well optimized or the data is relatively simple, T2 will be less than T1, making it easier to distinguish between normal and abnormal regions.

[0070] In some embodiments, the first-item evaluation information obtained by the sub-model when processing the image features of the first-item object includes a first key feature and a second confidence distribution. The second confidence distribution includes a second normal confidence distribution and a second abnormal confidence distribution.

[0071] In this embodiment, the first key feature obtained by each sub-model in the first candidate sub-model set when processing the first object is statistically analyzed, along with the second normal confidence distribution and the second abnormal confidence distribution.

[0072] In some embodiments, a first cosine similarity is calculated between the first key feature and the core key features in the sparse feature set; the relative entropy between the second confidence distribution and the first confidence distribution is calculated; and the first cosine similarity and the relative entropy are weighted and summed to obtain the similarity of the sub-models. Here, the core key features in the sparse feature set are the most and most core key features during clustering for feature aggregation sampling.

[0073] In some embodiments, the cosine similarity between the first key feature and each key feature in the sparse feature set is calculated, and the average of the multiple cosine similarities is calculated to obtain the first cosine similarity.

[0074] In some embodiments, the first object detected in the first object detection step is multiple objects, and multiple objects correspond to multiple first key features. Multiple cosine similarities are obtained between the multiple first key features and the same number of key features in the sparse feature set. The average of the multiple cosine similarities is calculated to obtain the first cosine similarity.

[0075] In some embodiments, calculating the relative entropy between the second confidence distribution and the first confidence distribution includes calculating the first relative entropy between the second normal confidence distribution and the first normal confidence distribution, calculating the second relative entropy between the second abnormal confidence distribution and the first abnormal confidence distribution, and weighted summing the first cosine similarity, the first relative entropy and the second relative entropy to obtain the similarity of the sub-models.

[0076] As an example, similarity is calculated using the following formula:

[0077] S=α1·KL(P normal ||P′ normal )+α2·KL(P abnormal ||P′ abnormal )+α3·CosSim(F core ||F′ key )

[0078] Where S is the similarity, KL(·) is the KL divergence, also known as relative entropy, used to calculate the similarity between distributions; CosSim(·) is the cosine similarity, used to calculate the similarity between feature vectors; α1, α2, and α3 are the corresponding weights; F core P normal P anomaly These represent the core key features, the first normal confidence distribution, and the first abnormal confidence distribution in the sparse feature set, respectively; F′key 、P′ normal 、P′ abnormal These are the first key feature, the second normal confidence distribution, and the second abnormal confidence distribution, respectively.

[0079] In step 304, based on the similarity of each sub-model in the first candidate sub-model set, the sub-model set and the voting weight of each sub-model in the sub-model set are obtained from the first candidate sub-model set.

[0080] In some embodiments, the similarity of the sub-models in the first candidate sub-model set is sorted, and the top k sub-models in terms of similarity are selected to form a sub-model set, which is used as the production line model set for this batch of product testing.

[0081] In some embodiments, the normalized similarity of each sub-model in the sub-model set is calculated, and this normalized similarity is used as the weight for weighted voting. For example, the weight of the i-th sub-model...

[0082] In step 305, the pre-trained shared feature extraction network and sub-model set are combined to obtain a surface defect anomaly detection model.

[0083] In this embodiment, a pre-trained shared feature extraction network and a set of sub-models selected based on similarity are combined to obtain a surface defect anomaly detection model.

[0084] In other words, when deploying the surface defect anomaly detection model to the production line, multiple sub-models share the same feature extraction network, and when updating the surface defect anomaly detection model, only the corresponding sub-model needs to be loaded or replaced.

[0085] In the above embodiments, this solution utilizes the first-piece inspection results obtained manually, compares the consistency of the inspection results, determines the candidate sub-model set from the sub-model library, and obtains the sub-model set from the candidate sub-model set based on the similarity calculated based on key features and confidence distribution to construct a surface defect anomaly detection model. This can quickly and accurately determine the surface defect anomaly detection model that is accurately adapted to the current batch of products, thereby improving the accuracy of detection. This solution obtains weighted voting weights based on similarity, which can obtain a more accurate anomaly confidence level.

[0086] To achieve faster adjustments over a wider range without model training, the sub-model set of the current production line can be updated based on production line re-inspection feedback. In some embodiments, based on any one or more combined embodiments described above, such as... Figure 4 As shown, the surface defect anomaly detection method may include, but is not limited to, the following steps 401-405.

[0087] In step 401, the detection results of the human on the erroneous detection object and the detection results of each sub-model in the sub-model set on the erroneous detection object are obtained.

[0088] In this embodiment, during the production line operation, the products after the model quality inspection are manually sampled and re-inspected. When a missed inspection or false inspection occurs, the sub-model set of the current production line is updated using the data of the erroneous detection object.

[0089] It should be noted that the specific implementation method of step 401 can be found in step 301 of the above embodiment, and the principle is the same, so it will not be repeated here.

[0090] In step 402, based on the comparison between the detection results of the erroneous detection objects by the human and the detection results of the erroneous detection objects by each sub-model in the sub-model set, a second candidate sub-model set with consistent detection results is selected from the first candidate sub-model set.

[0091] The difference between this step and the first article inspection step in the above embodiment is that this embodiment filters the first candidate sub-model set obtained based on the first article object inspection to maintain consistency with the first article inspection data, retains the sub-models that are correctly detected in the existing production line sub-model set, and only removes the sub-models that predict incorrectly.

[0092] It should be noted that the specific implementation method of step 402 can be found in step 302 of the above embodiment, and the principle is the same, so it will not be repeated here.

[0093] In step 403, for each sub-model in the second candidate sub-model set, the first evaluation information obtained by the sub-model when processing the image features of the erroneous detection object is obtained; and the similarity between the first evaluation information of the sub-model and its initial evaluation information is calculated to obtain the similarity of the sub-model.

[0094] It should be noted that the specific implementation method of step 403 can be found in step 303 of the above embodiment, and the principle is the same, so it will not be repeated here.

[0095] In step 404, based on the similarity of each sub-model in the second candidate sub-model set, the sub-model set and the voting weight of each sub-model in the sub-model set are obtained from the second candidate sub-model set.

[0096] In this embodiment, the corresponding number of sub-models and the weight of the weighted vote of each sub-model are obtained from the second candidate sub-model set according to the similarity ranking of each sub-model in the newly calculated second candidate sub-model set.

[0097] It should be noted that the specific implementation method of step 404 can be found in step 304 of the above embodiment, and the principle is the same, so it will not be repeated here.

[0098] In step 405, the surface defect anomaly detection model is updated based on the combination of the pre-trained shared feature extraction network and the sub-model set.

[0099] In some embodiments, based on the sub-model set obtained from the second candidate sub-model set, the current sub-model set of the production line is supplemented and replaced. That is, the sub-models in the current sub-model set of the production line that are the same as the sub-models in the second candidate sub-model set are retained, and the sub-models in the current sub-model set of the production line other than the retained sub-models are replaced with the remaining sub-models in the second candidate sub-model set, so as to obtain the updated sub-model set of the production line.

[0100] In the above embodiments, this solution utilizes the inspection results of production line re-inspection and determines the second candidate sub-model set from the first candidate sub-model set by comparing the consistency of the inspection results, so as to maintain consistency with the first piece inspection data; this solution obtains the sub-model set from the second candidate sub-model set based on the similarity calculated by key features and confidence distribution to update the surface defect anomaly detection model, which can realize a wider range of rapid adjustment of the surface defect anomaly detection model without model training; that is, this solution utilizes the feedback from production line re-inspection to dynamically and on a large scale adjust the sub-model set of the surface defect anomaly detection model and update the corresponding weighted weights in real time.

[0101] After completing a batch of product object inspection tasks, if a sufficient number of incorrectly identified product data are accumulated, the sub-models in the sub-model library can be updated through an offline model iteration scheme. This provides more suitable sub-models or more sub-models that can improve detection accuracy for surface defect anomaly detection on the production line. In some embodiments, based on any one or more combined embodiments described above, such as... Figure 5 As shown, the surface defect anomaly detection method may include, but is not limited to, the following steps 501-509.

[0102] In step 501, multiple second key features corresponding to multiple incorrectly identified objects are obtained; the second key features are obtained through the attention module in the sub-model.

[0103] In this embodiment, the offline model iteration scheme freezes the shared feature extraction network during model training or iterative updates, meaning that the model parameters of the shared feature extraction network do not participate in the iterative updates.

[0104] In some embodiments, image features of multiple incorrectly identified objects are input into the attention module of the corresponding sub-model for detecting errors to obtain multiple second key features.

[0105] In step 502, for each of the multiple second key features, the second cosine similarity between the second key feature and the core key feature in the sparse feature set of each sub-model in the sub-model library is calculated.

[0106] It should be noted that the specific implementation method of step 502 can be found in step 303 of the above embodiment, and the principle is the same, so it will not be repeated here.

[0107] In step 503, if the second cosine similarity is less than the first threshold, the sub-model with the largest second cosine similarity in the sub-model library is determined as the first target sub-model.

[0108] In this embodiment, if the second cosine similarity is less than the first threshold, the sub-model corresponding to the largest second cosine similarity is selected as the first target sub-model.

[0109] In step 504, the image features corresponding to the second key feature are added to the training image feature sample combination of the first target sub-model, and the training image feature sample combination of the first target sub-model is updated; and the first target sub-model is trained based on the updated training image feature sample combination of the first target sub-model to obtain the iteratively updated first target sub-model.

[0110] In this embodiment, after determining the first target sub-model, the image features corresponding to the incorrectly identified data are added to the training data of the first target sub-model, and the first target sub-model is updated and iterated.

[0111] In step 505, if the second cosine similarity is greater than or equal to the first threshold, multiple core key features that meet the requirements of the second cosine similarity are obtained.

[0112] In this embodiment, if the second cosine similarity is greater than or equal to the first threshold, then a set of multiple core key features with high similarity are selected.

[0113] In step 506, the image features corresponding to the second key feature and the image features corresponding to multiple core key features are combined to form a new image feature training set; and a new sub-model is trained based on the new image feature training set.

[0114] In this embodiment, the parameter update of the attention module is trained by the degree of difference between the new key features and the key features in the previously obtained sparse feature set, and its loss function uses cosine similarity (Loss); the parameter update of the anomaly detection module combines the key features in the sparse feature set and the newly added key features, and its loss function uses maximum likelihood (Loss).

[0115] In this embodiment, a new training set is formed by combining multiple core key features with high similarity and image features corresponding to the misidentified data, and a new sub-model is trained using this new training set.

[0116] In step 507, the new sub-model is added to the sub-model library to obtain the updated sub-model library.

[0117] In this embodiment, the new sub-model is added to the sub-model library.

[0118] In the above embodiments, this scheme combines the cosine similarity loss and the maximum likelihood loss between historical key features and the key features corresponding to newly acquired misidentified objects. It can iteratively update existing sub-models with high similarity and train new sub-models. It can ensure that the new sub-model has sufficient differences from the sub-models in the sub-model library, while maintaining the recognition performance of historical data and avoiding the risk of catastrophic forgetting.

[0119] Figure 6 This is a block diagram illustrating a surface defect anomaly detection apparatus according to an exemplary embodiment. Figure 6 As shown, the surface defect anomaly detection device may include: an image acquisition module 610, a model detection module 620, a weighted voting module 630, and a result output module 640.

[0120] The image acquisition module 610 is used to acquire the object image of the object to be detected.

[0121] The model detection module 620 is used to input the object image into the surface defect anomaly detection model. The surface defect anomaly detection model includes a shared feature extraction network and a sub-model set. The shared feature extraction network is used to extract image features of the object image. The sub-model set includes multiple sub-models. Each sub-model is used to process the image features to obtain the corresponding anomaly confidence.

[0122] The weighted voting module 630 is used to obtain multiple anomaly confidence scores output by the surface defect anomaly detection model, and to process the multiple anomaly confidence scores through weighted voting to obtain the defect anomaly confidence score.

[0123] The result output module 640 is used to detect whether there are defects on the surface of the object to be inspected based on the defect anomaly confidence level.

[0124] In some embodiments, the apparatus further includes a model screening module 650, for:

[0125] Obtain the detection results of the first object by humans and the detection results of each sub-model in the sub-model library on the first object;

[0126] Based on the comparison between the detection results of the first object by human inspection and the detection results of each sub-model in the sub-model library, the first candidate sub-model set with consistent detection results is selected from the sub-model library;

[0127] For each sub-model in the first candidate sub-model set, the first-piece evaluation information obtained by the sub-model when processing the image features of the first object is obtained; and the similarity between the first-piece evaluation information of the sub-model and its initial evaluation information is calculated to obtain the similarity of the sub-model; wherein, the initial evaluation information of the sub-model is obtained by processing multiple training samples after the sub-model is trained;

[0128] Based on the similarity of each sub-model in the first candidate sub-model set, obtain the sub-model set and the voting weight of each sub-model in the sub-model set from the first candidate sub-model set;

[0129] By combining a pre-trained shared feature extraction network and a set of sub-models, a surface defect anomaly detection model is obtained.

[0130] In some embodiments, the sub-model includes an attention module for extracting key features of the image; the initial evaluation information of the sub-model includes a sparse feature set and a first confidence distribution.

[0131] When obtaining the initial evaluation information of the sub-models, the model selection module 650 is used for:

[0132] Input the feature images corresponding to multiple training samples into the attention module of the trained sub-model to obtain multiple key features;

[0133] Feature aggregation sampling is performed on multiple key features to obtain a sparse feature set for the sub-model;

[0134] Input the feature images corresponding to multiple training samples into the trained sub-model to obtain the anomaly confidence of each pixel in the object image of multiple training samples.

[0135] Based on the anomaly confidence of each pixel in the object image of each training sample, the confidence distribution corresponding to each training sample is obtained.

[0136] By concatenating the confidence distributions of multiple training samples, the first confidence distribution of the sub-model is obtained.

[0137] In some embodiments, the first-piece evaluation information includes a first key feature and a second confidence distribution; the model screening module 650, when calculating the similarity between the first-piece evaluation information and the initial evaluation information of the sub-model to obtain the similarity of the sub-model, is specifically used for:

[0138] Calculate the first cosine similarity between the first key feature and the core key features in the sparse feature set;

[0139] Calculate the relative entropy between the second confidence distribution and the first confidence distribution;

[0140] The similarity of the sub-models is obtained by weighted summation of the first cosine similarity and the relative entropy.

[0141] In some embodiments, the model filtering module 650 is further configured to:

[0142] Obtain the detection results of human inspectors on the erroneous detection objects and the detection results of each sub-model in the sub-model set on the erroneous detection objects;

[0143] Based on the comparison between the detection results of the erroneous objects by the human and the detection results of each sub-model in the sub-model set, a second candidate sub-model set with consistent detection results is selected from the first candidate sub-model set;

[0144] For each sub-model in the second candidate sub-model set, the first evaluation information obtained by the sub-model when processing the image features of the erroneous detection object is obtained; and the similarity between the first evaluation information of the sub-model and its initial evaluation information is calculated to obtain the similarity of the sub-model.

[0145] Based on the similarity of each sub-model in the second candidate sub-model set, the sub-model set and the voting weight of each sub-model in the sub-model set are obtained from the second candidate sub-model set;

[0146] The surface defect anomaly detection model is updated based on a combination of a pre-trained shared feature extraction network and a sub-model set.

[0147] In some embodiments, the apparatus further includes a model training module 660, configured to:

[0148] Multiple second key features corresponding to multiple incorrectly identified objects are obtained; the second key features are obtained through the attention module in the sub-model.

[0149] For each of the multiple second key features, calculate the second cosine similarity between the second key feature and the core key feature in the sparse feature set of each sub-model in the sub-model library;

[0150] If the second cosine similarity is less than the first threshold, the sub-model with the largest second cosine similarity in the sub-model library is determined as the first target sub-model;

[0151] Add the image features corresponding to the second key feature to the training image feature sample combination of the first target sub-model, and update the training image feature sample combination of the first target sub-model;

[0152] The first target sub-model is trained by combining the training image feature samples of the updated first target sub-model, and the updated first target sub-model is obtained iteratively.

[0153] In some embodiments, the model training module 660 is further configured to:

[0154] If the second cosine similarity is greater than or equal to the first threshold, obtain multiple core key features that meet the requirements of the second cosine similarity.

[0155] The image features corresponding to the second key feature are combined with the image features corresponding to multiple core key features to form a new image feature training set;

[0156] A new sub-model is obtained by training a new set of image features;

[0157] Add the new sub-model to the sub-model library to obtain the updated sub-model library.

[0158] In some embodiments, the number of layers in the sub-model is less than the number of layers in the shared feature extraction network.

[0159] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0160] Figure 7 This is a block diagram illustrating an apparatus 700 for detecting surface defect anomalies according to an exemplary embodiment. For example, apparatus 700 may be an electronic device, such as a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, wearable display device (such as a head-up display device), etc.

[0161] Reference Figure 7 The device 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 77, an input / output (I / O) interface 712, a sensor component 714, and a communication component 716.

[0162] Processing component 702 typically controls the overall operation of device 700, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 702 may include one or more modules to facilitate interaction between processing component 702 and other components. For example, processing component 702 may include a multimedia module to facilitate interaction between multimedia component 708 and processing component 702.

[0163] Memory 704 is configured to store various types of data to support the operation of device 700. Examples of this data include instructions for any application or method operating on device 700, contact data, phonebook data, messages, pictures, videos, etc. Memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0164] The power supply component 706 provides power to the various components of the device 700. The power supply component 706 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 700.

[0165] Multimedia component 708 includes a screen that provides an output interface between device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 708 includes a front-facing camera and / or a rear-facing camera. When device 700 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0166] Audio component 77 is configured to output and / or input audio signals. For example, audio component 77 includes a microphone (MIC) configured to receive external audio signals when device 700 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 704 or transmitted via communication component 716. In some embodiments, audio component 77 also includes a speaker for outputting audio signals.

[0167] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0168] Sensor assembly 714 includes one or more sensors for providing state assessments of various aspects of device 700. For example, sensor assembly 714 may detect the on / off state of device 700, the relative positioning of components such as the display and keypad of device 700, changes in the position of device 700 or a component of device 700, the presence or absence of user contact with device 700, the orientation or acceleration / deceleration of device 700, and temperature changes of device 700. Sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 714 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0169] Communication component 716 is configured to facilitate wired or wireless communication between device 700 and other devices. Device 700 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 716 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 716 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0170] In an exemplary embodiment, the apparatus 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0171] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, which can be executed by a processor 720 of the device 700 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0172] In an exemplary embodiment, a program product is also provided, including at least one of a program and instructions, wherein the program and instructions, when executed by the processor 720 of the device 700, implement the steps of the above method.

[0173] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0174] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for detecting surface defect anomalies, characterized in that, include: Obtain the object image of the object to be detected; The object image is input into a surface defect anomaly detection model, which includes a shared feature extraction network and a sub-model set. The shared feature extraction network is used to extract image features from the object image, and the sub-model set includes multiple sub-models. Each sub-model is used to process the image features to obtain a corresponding anomaly confidence level. Multiple anomaly confidence scores output by the surface defect anomaly detection model are obtained, and the multiple anomaly confidence scores are processed by a weighted voting method to obtain the defect anomaly confidence score. Based on the defect anomaly confidence level, the presence of defects on the surface of the object to be tested is detected.

2. The method according to claim 1, characterized in that, Obtaining the surface defect anomaly detection model includes: Obtain the detection results of the first object by human inspection and the detection results of each sub-model in the sub-model library on the first object; Based on the comparison between the detection results of the first object by the human and the detection results of each sub-model in the sub-model library, a first candidate sub-model set with consistent detection results is selected from the sub-model library; For each sub-model in the first candidate sub-model set, the first-piece evaluation information obtained by the sub-model when processing the image features of the first object is acquired; and the similarity between the first-piece evaluation information of the sub-model and its initial evaluation information is calculated to obtain the similarity of the sub-model; wherein, the initial evaluation information of the sub-model is obtained by processing multiple training samples after the sub-model is trained; Based on the similarity of each sub-model in the first candidate sub-model set, a sub-model set and the voting weight of each sub-model in the sub-model set are obtained from the first candidate sub-model set; The pre-trained shared feature extraction network and the sub-model set are combined to obtain the surface defect anomaly detection model.

3. The method according to claim 2, characterized in that, The sub-model includes an attention module, which is used to extract key features of the image features; The initial evaluation information of the sub-model includes a sparse feature set and a first confidence distribution; This includes obtaining the initial evaluation information for the sub-model, including: The feature images corresponding to multiple training samples are input into the attention module in the trained sub-model to obtain multiple key features; Feature aggregation sampling is performed on the multiple key features to obtain the sparse feature set of the sub-model; Input the feature images corresponding to multiple training samples into the trained sub-model to obtain the anomaly confidence of each pixel in the object image of multiple training samples. Based on the anomaly confidence of each pixel in the object image of each training sample, the confidence distribution corresponding to each training sample is obtained. The confidence distributions of the multiple training samples are concatenated to obtain the first confidence distribution of the sub-model.

4. The method according to claim 3, characterized in that, The initial evaluation information includes a first key feature and a second confidence distribution; calculating the similarity between the initial evaluation information of the sub-model and its initial evaluation information to obtain the similarity of the sub-model includes: Calculate the first cosine similarity between the first key feature and the core key features in the sparse feature set; Calculate the relative entropy between the second confidence distribution and the first confidence distribution; The similarity of the sub-model is obtained by weighted summation of the first cosine similarity and the relative entropy.

5. The method as described in claim 2, characterized in that, The method further includes: Obtain the detection results of the erroneous detection objects by humans and the detection results of each sub-model in the sub-model set for the erroneous detection objects; Based on the comparison between the detection results of the human on the erroneous detection objects and the detection results of each sub-model in the sub-model set on the erroneous detection objects, a second candidate sub-model set with consistent detection results is selected from the first candidate sub-model set; For each sub-model in the second candidate sub-model set, obtain the first evaluation information obtained by the sub-model when processing the image features of the error detection object; and calculate the similarity between the first evaluation information of the sub-model and its initial evaluation information to obtain the similarity of the sub-model. Based on the similarity of each sub-model in the second candidate sub-model set, a sub-model set and the voting weight of each sub-model in the sub-model set are obtained from the second candidate sub-model set; The surface defect anomaly detection model is updated based on the combination of the pre-trained shared feature extraction network and the sub-model set.

6. The method as described in claim 1, characterized in that, The method further includes: Multiple second key features corresponding to multiple incorrectly identified objects are obtained; the second key features are obtained through the attention module in the sub-model. For each of the plurality of second key features, calculate the second cosine similarity between the second key feature and the core key features in the sparse feature set of each sub-model in the sub-model library; If the second cosine similarity is less than the first threshold, the sub-model with the largest second cosine similarity in the sub-model library is determined as the first target sub-model; The image features corresponding to the second key feature are added to the training image feature sample combination of the first target sub-model to update the training image feature sample combination of the first target sub-model; and the first target sub-model is trained based on the updated training image feature sample combination of the first target sub-model to obtain the iteratively updated first target sub-model.

7. The method as described in claim 6, characterized in that, The method further includes: If the second cosine similarity is greater than or equal to the first threshold, obtain multiple core key features that meet the requirements of the second cosine similarity; The image features corresponding to the second key feature are combined with the image features corresponding to the multiple core key features to form a new image feature training set; and a new sub-model is trained based on the new image feature training set. The new sub-model is added to the sub-model library to obtain the updated sub-model library.

8. The method according to any one of claims 1-7, characterized in that, The number of layers in the sub-model is less than the number of layers in the shared feature extraction network.

9. A surface defect anomaly detection device, characterized in that, include: The image acquisition module is used to acquire the object image of the object to be detected; The model detection module is used to input the object image into the surface defect anomaly detection model. The surface defect anomaly detection model includes a shared feature extraction network and a sub-model set. The shared feature extraction network is used to extract image features of the object image. The sub-model set includes multiple sub-models. Each sub-model is used to process the image features to obtain the corresponding anomaly confidence. The weighted voting module is used to obtain multiple anomaly confidence scores output by the surface defect anomaly detection model, and to process the multiple anomaly confidence scores through a weighted voting method to obtain the defect anomaly confidence score. The result output module is used to detect whether there are defects on the surface of the object to be tested based on the defect anomaly confidence level.

10. An electronic device, characterized in that, include: One or more processors; The processor is configured to invoke instructions to cause the electronic device to perform the method of any one of claims 1-8.

11. A storage medium storing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method of any one of claims 1-8.

12. A program product comprising at least one of a program and instructions, characterized in that, When at least one of the program or instructions is executed by an electronic device, it implements the steps of the method according to any one of claims 1-8.

Images