Metal surface detection method, system, electronic device, and storage medium

Abstract

The application discloses a metal surface detection method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring image information of a metal surface to be detected; performing local feature extraction on the image information to obtain extracted features; performing metric learning on the extracted features and features in a memory library, and obtaining an abnormal score according to the metric learning result; the memory library is established according to nominal positive samples; and performing flaw segmentation on the metal surface to be detected according to the abnormal score to obtain a detection result. The application determines whether the metal surface is abnormal by performing metric learning on the local features of the metal surface and the features in the memory library representing the nominal positive samples. The embodiment of the application is beneficial to improving the accuracy of metal surface detection. The application can be widely applied in the technical field of Internet of Things.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Things, and in particular to a metal surface detection method and system, an electronic device, and a storage medium. BACKGROUND

[0002] In various sheet metal manufacturing, intelligent metal surface defect detection technology can identify the appearance defects of various industrial products to ensure product quality and maintain production stability. For example, in the traditional sheet metal profile manufacturing industry, the defect detection method usually relies on manual sampling inspection, which is difficult to meet the large-scale quality inspection demand. With the continuous progress of industrial imaging, computer vision and deep learning technologies, visual-based metal surface defect detection technology has made significant progress and has become an effective product appearance quality detection solution.

[0003] In related technologies, the defects on the metal surface are detected by artificial intelligence. However, the types of surface defects are various, the number of defect pictures is small and difficult to obtain, which leads to low prediction accuracy of the model and further affects the accuracy of metal surface detection. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a high-accuracy metal surface detection method, system, electronic device and storage medium.

[0005] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application provides a metal surface detection method, which comprises: acquiring image information of a metal surface to be detected; performing local feature extraction on the image information to obtain extracted features; performing metric learning on the extracted features and features in a memory library, and obtaining an abnormal score according to the metric learning result; the memory library is established according to nominal positive samples; and performing defect segmentation on the metal surface to be detected according to the abnormal score to obtain a detection result. The embodiments of the present application determine whether there is an abnormality on the metal surface by performing metric learning on the local features of the metal surface and the features in the memory library representing nominal positive samples. The embodiments of the present application are beneficial to improving the accuracy of metal surface detection.

[0006] In some embodiments, the memory library is determined by the following steps:

[0007] Acquiring image sample information and performing local feature extraction on the image sample information to obtain extracted sample features;

[0008] Performing neighborhood feature fusion processing on the extracted sample features, and storing the positive sample features after the fusion processing to obtain a memory library;

[0009] If the data quantity in the memory library is greater than a preset data quantity threshold, the memory library is reduced by a feature library kernel set reduction algorithm, and the memory library is updated.

[0010] In some embodiments, the method provided by the embodiments of the present application comprises:

[0011] The sample features are extended in a neighborhood, and the multi-layer neighborhood features obtained are dimensionally matched to obtain neighborhood matching features.

[0012] The neighborhood matching features at different levels are pooled, and the features of different scales are spliced to obtain multi-scale representation features.

[0013] According to the multi-scale representation features, positive sample features are determined to obtain a memory library.

[0014] In some embodiments, the method provided by the embodiments of the present application comprises:

[0015] A plurality of feature vector blocks are randomly selected from the memory library.

[0016] The distances between all vectors in the memory library and each of the feature vector blocks are calculated to obtain a first distance set.

[0017] According to the first distance set, a first feature vector block is determined, and the first feature vector block is placed in a new memory library.

[0018] The distances between all vectors in the memory library and the first feature vector block are calculated to obtain a second distance set.

[0019] The values in the first distance set and the second distance set are subjected to a bit operation, and according to the bit operation result, a third distance set is determined.

[0020] According to the third distance set, a second feature vector block is determined, and the second feature vector block is placed in a new memory library.

[0021] The second distance set is taken as a new first distance set, the second feature vector block is taken as a new first feature vector block, and the step of calculating the distances between all vectors in the memory library and the first feature vector block to obtain a second distance set is returned until the number of cycles is greater than or equal to a preset number threshold. The new memory library obtained is taken as an updated memory library.

[0022] In some embodiments, the method provided by the embodiments of the present application comprises the following steps:

[0023] obtaining the extracted feature and an extracted neighborhood feature of the extracted feature;

[0024] determining a maximum value in distances between the extracted feature, the extracted neighborhood feature and each feature in the memory library, to obtain a maximum distance;

[0025] obtaining a weighted feature block in the memory library closest to the extracted feature;

[0026] performing weighted processing on the maximum distance according to the weighted feature block and the extracted feature, to obtain an anomaly score.

[0027] In some embodiments, the method provided by the embodiments of the present application comprises the following steps:

[0028] performing interpolation processing on the anomaly scores at different positions of the metal surface to be detected, and covering the interpolation result with original image information, to obtain a heat map;

[0029] performing defect position segmentation on the heat map according to an anomaly value threshold, to obtain a detection result.

[0030] In some embodiments, the method provided by the embodiments of the present application comprises the following steps:

[0031] performing inference on the image sample information through a pre-trained network, to obtain multi-layer feature encoding;

[0032] extracting a middle-layer feature encoding as an extracted sample feature.

[0033] To achieve the above object, another aspect of the embodiments of the present application provides a metal surface detection system, which comprises:

[0034] a first module configured to obtain image information of a metal surface to be detected;

[0035] a second module configured to extract a local feature from the image information, to obtain an extracted feature;

[0036] a third module configured to perform metric learning on the extracted feature and a feature in a memory library, and obtain an anomaly score according to a metric learning result; the memory library is established according to a nominal positive sample;

[0037] A fourth module is configured to perform flaw segmentation on the metal surface to be detected according to the abnormal score, and obtain a detection result.

[0038] To achieve the above object, another aspect of the embodiments of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0039] To achieve the above object, another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0040] The embodiments of the present application at least have the following beneficial effects: the method provided by the embodiments of the present application comprises: obtaining image information of a metal surface to be detected; performing local feature extraction on the image information to obtain extracted features; performing metric learning on the extracted features and features in a memory library, and obtaining an abnormal score according to a metric learning result; the memory library is established according to nominal positive samples; performing flaw segmentation on the metal surface to be detected according to the abnormal score to obtain a detection result. The embodiments of the present application determine whether there is an abnormality in the metal surface by performing metric learning on the local features of the metal surface and the features in the memory library representing the nominal positive samples. The embodiments of the present application are beneficial to improving the accuracy of metal surface detection. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of one embodiment of the metal surface detection method provided by the present application;

[0042] Figure 2 is a flowchart of one embodiment of the establishment and reasoning of the memory library provided by the present application;

[0043] Figure 3 is a flowchart of one embodiment of the determination process of the memory library provided by the present application;

[0044] Figure 4 is a flowchart of one embodiment of the preliminary establishment of the memory library provided by the present application;

[0045] Figure 5 is a flowchart of one embodiment of the determination process of the abnormal score provided by the present application;

[0046] Figure 6 is a flowchart of one embodiment of the flaw segmentation process provided by the present application;

[0047] Figure 7 is a flowchart of another embodiment of the establishment of the memory library provided by the present application;

[0048] Figure 8 is a flow chart of an embodiment of the detection inference process provided by the present application;

[0049] Figure 9 is a structural schematic diagram of a metal surface detection system provided by an embodiment of the present application;

[0050] Figure 10 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all the implementations consistent with embodiments of the present application, but are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0052] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0053] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by a person skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0055] Before the embodiments of the present application are described in detail, first, some nouns and terms involved in the embodiments of the present application are described, and the nouns and terms involved in the embodiments of the present application are applicable to the following explanations.

[0056] Metal surface flaw detection: Metal surface flaw detection is an important part of industrial quality control, which involves the use of various techniques to identify and classify defects or abnormalities on the surface of metal products. These flaws can affect the appearance, durability or functionality of the product, so detecting and removing or repairing these flaws is crucial to ensure product quality and safety. Including but not limited to scratches, dents, cracks, holes, rust, discoloration, foreign matter inclusions, etc.

[0057] Local block features: Global features refer to the overall properties of an image. Common global features include color features, texture features, and shape features, such as intensity histograms. Since they are low-level visual features at the pixel level, global features have good invariance, simple calculation, and intuitive representation. However, their high feature dimension and large computational load are their fatal weaknesses. Compared with line features, texture features, and structural features, local image features have the following characteristics: a large number of features are contained in the image, the correlation between features is small, and in the case of occlusion, the disappearance of part of the feature will not affect the detection and matching of other features.

[0058] Perception aggregation: Perception aggregation is a feature fusion strategy that aims to combine multi-scale and multi-level features in deep neural networks to obtain richer and more robust feature representations. In computer vision tasks such as image classification, object detection, and semantic segmentation, perception aggregation enables models to better understand image content, thereby improving performance. Deep convolutional neural networks abstract image information layer by layer through multiple convolutional layers, each layer capturing perceptual fields of different sizes. Lower levels usually capture details and texture information, while higher levels capture more abstract semantic information. Perception aggregation combines these features at different levels, allowing the model to consider both local details and global context. In feature pyramid networks (FPN) or similar architectures, perception aggregation involves merging feature maps at different depths. This is usually achieved through top-down paths and lateral connections, allowing information to flow and enhance across different scales.

[0059] In the field of 3C sheet metal manufacturing, intelligent metal surface defect detection technology can identify various types of industrial product appearance defects to ensure product quality and maintain production stability. For example, in traditional sheet metal manufacturing, defect detection methods usually rely on manual sampling, which is difficult to meet the needs of large-scale quality inspection. In recent years, with the continuous progress of industrial imaging, computer vision and deep learning, visual-based metal surface defect detection technology has made significant progress and has become an effective product appearance quality detection solution. In actual business scenarios, defect detection usually faces three problems. First, in many industrial scenarios, it is relatively easy to obtain normal image samples, but it is costly and complex to fully specify expected defect changes. Second, there are many types of defects, including but not limited to scratches, dents, cracks, holes, rust, discoloration, and foreign matter inclusions, and errors can range from subtle changes such as small scratches to larger structural defects such as missing components. Third, there is an unknown nature, and defects in parts are usually unpredictable, and traditional machine learning methods may not accurately predict actual production defects. In addition, the high cost of training and the failure of old defect detection methods due to product changes are the main factors that make it difficult for metal surface defect detection AI quality inspection to quickly form a scale effect. Therefore, there is an urgent need for a flexible and versatile metal surface defect detection method to meet the large-scale quality inspection needs of the metal manufacturing industry.

[0060] To this end, the present application provides a metal surface defect detection scheme based on local block feature perception aggregation. In the sheet metal processing and manufacturing industry, it is relatively easy to obtain normal image samples for the final metal material product, but it is costly and complex to fully specify expected defect changes. And traditional visual detection methods usually have difficulty in obtaining good detection results for defects outside the training sample distribution. The out-of-distribution defect problem of metal surface defect detection is particularly difficult to enumerate because of the large number of defect types. The metal surface defect detection method proposed in the present application is a computer vision-based detection method based on image pixel level. The core idea is to train a detection model using commonly used normal samples, which can obtain the local features of any pixel point in the metal surface image. By calculating the geometric distance of the local features of the pixel points in the detection image and the normal samples, the defect location and confidence of the detection image are determined.

[0061] The metal surface detection method provided in the embodiments of the present application relates to the technical field of Internet of Things. The metal surface detection method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server, and can also be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, and the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform, and the server can also be a node server in a blockchain network; and the software can be an application for implementing the metal surface detection method, and the like, but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0063] It should be noted that in each specific embodiment of the present application, when relevant processing needs to be performed according to user information, user behavior data, user historical data, and user location information and other data related to the identity or characteristics of the user, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for enabling the embodiments of the present application to normally operate will be obtained.

[0064] Figure 1 is an optional flowchart of the metal surface detection method provided in the embodiments of the present application; Figure 1The method in the first aspect can include but is not limited to comprising steps S100-S400.

[0065] At step S100, image information of a metal surface to be detected is acquired.

[0066] At step S200, local feature extraction is performed on the image information to obtain extracted features.

[0067] At step S300, metric learning is performed on the extracted features and features in a memory library, and an anomaly score is obtained according to a result of the metric learning; the memory library is established according to nominal positive samples.

[0068] At step S400, according to the anomaly score, a flaw is segmented from the metal surface to be detected to obtain a detection result.

[0069] It can be understood that metric learning (Metric Learning) is also known as similarity learning (Similarity Learning), which is a machine learning method for measuring the similarity between samples by learning a distance function (metric). The memory library in the embodiments of the present application can be established based on nominal positive samples, i.e., metals that appear to be flawless on the surface. The present application compares the local feature data of the metal surface to be detected with a large number of feature data of positive samples, determines an anomaly score according to the comparison, and further determines the flaw position.

[0070] The metal surface flaw detection technical solution proposed in the present application is based on an innovative unsupervised learning method. The core hypothesis is that a large number of flawless (nominal) sample images can be obtained during model training. These images are used as training data to construct a representative feature memory library. In industrial applications, such images can be easily collected from the defect-free areas of products. In addition, considering that in actual applications, samples with flaws are often difficult to obtain, especially before model deployment. Therefore, the present application does not rely on the availability of samples with flaws, but focuses on the detection of unknown abnormal patterns. The algorithm realizes anomaly detection by constructing a memory library containing rich nominal features. Each feature vector in the memory library represents the visual characteristics of a local region in the image. The algorithm uses these feature vectors to identify abnormal regions by calculating the similarity between each local region in the test image and the feature vectors in the memory library. The advantage of this method is that it can effectively extract and utilize contextual information from nominal samples, while maintaining high sensitivity to unknown abnormal patterns.

[0071] Reference Figure 2As shown, the content involved in the present scheme mainly includes two processes of local feature library training construction (i.e. the establishment of the memory library) and reasoning (i.e. metal surface detection), the local feature library construction process includes local feature extraction, neighborhood feature perception aggregation, local feature library kernel set reduction; the metal defect model reasoning detection mainly includes: local feature extraction of the sample to be tested, local block feature metric learning, defect segmentation based on abnormal value. The steps are briefly introduced below.

[0072] Local feature extraction: the local feature block feature extraction of the metal surface defect detection mainly uses the selected intermediate feature layer in the deep neural network pre-training model as the main feature of the defect detection, the purpose is to retain the local texture information and surface details. For the task of metal surface defect detection which needs to detect local details, too deep level will cause the feature to be too abstract, thereby losing the local feature information that should be retained.

[0073] Aggregation of neighborhood features: in order to increase the size of the receptive field and the robustness to small spatial deviations, and without losing the spatial resolution or the availability of the feature map, the present patent further aggregates the neighborhood features. This is to consider the multi-scale correspondence relationship of the template positive samples and the defect samples, on the one hand, the correlation of the neighborhood pixels in the image is considered, on the other hand, the multi-scale difference of different images under different resolutions is considered, to ensure that the high resolution and high precision can be maintained while improving the robustness to spatial changes and noise, so as to realize more effective defect detection and positioning.

[0074] Local feature library kernel set reduction: as the number of positive samples in the training set increases, the feature library also increases, thereby significantly increasing the reasoning time and the required storage space of the new test data, therefore, the present application proposes to find a subset Mc (updated memory library) of M (original feature library / memory library), so that Mc can maintain a good representation of the original memory library, while greatly reducing its storage requirements and reasoning time. It is usually necessary to reduce the dimensionality of the feature library, and to save the encoding features of good data as much as possible. The present application proposes a kernel set sampling mechanism to reduce the original feature library, reduce the consumption of system resources in the detection process, and improve the detection speed.

[0075] Local feature extraction of the sample to be tested: this step mainly obtains the aggregated feature map of different positions of the sample to be detected through the pre-training model and the neighborhood feature aggregation means, so as to judge the defect position.

[0076] Local block feature metric learning: using the feature block memory library M obtained by training, we estimate the image level anomaly score of the test image x_test at position (w, h) through the maximum distance score s between the feature blocks in the memory library and the test feature blocks, and weight the abnormal value through the distance between (w, h) and the adjacent sample inspection.

[0077] Based on the outlier-based defect segmentation: through the outliers in different positions (w, h), in order to match the original input resolution, the application further optimizes and smooths the results through bilinear interpolation upsampling and Gaussian filtering, covers the original test image to generate a heat map. The defect position is segmented by the outlier threshold.

[0078] In some embodiments, with reference to Figure 3 The method provided by the embodiment of the application determines the memory bank through the following steps:

[0079] Step S500, acquiring image sample information and performing local feature extraction on the image sample information to obtain extracted sample features;

[0080] Step S600, performing neighborhood feature fusion processing on the extracted sample features, and storing the positive sample features after the fusion processing to obtain a memory bank;

[0081] Step S700, if the data quantity in the memory bank is greater than a preset data quantity threshold, performing reduction processing on the memory bank through a feature bank kernel set reduction algorithm, and updating the memory bank.

[0082] In some embodiments, with reference to Figure 4 The method provided by the embodiment of the application performs neighborhood feature fusion processing on the extracted sample features, and stores the positive sample features after the fusion processing to obtain a memory bank, including:

[0083] Step S610, performing neighborhood expansion on the extracted sample features, and performing dimension matching on the obtained multi-layer neighborhood features to obtain neighborhood matching features;

[0084] Step S620, performing pooling processing on the neighborhood matching features at different levels, and performing feature splicing on features of different scales to obtain multi-scale representation features;

[0085] Step S630, determining positive sample features according to the multi-scale representation features to obtain a memory bank.

[0086] Specifically, neighborhood feature fusion: using a feature map φ i,j (h,w) represents a feature slice at position (h, w), so that the feature can cope with local spatial changes, cover a large enough receptive field, and at the same time not lose spatial resolution or feature availability.

[0087] The specific steps are as follows:

[0088] Step S21: neighborhood expansion of the feature slice is performed through the following formula:

[0089]

[0090] where N p (h,w) denotes a neighborhood of size p x p around position (h, w), a denotes the extension of the feature in the vertical direction at h, w, b denotes the extension in the horizontal direction, N p (h,w) denotes a neighborhood of size p x p around position (h, w), a denotes the extension of the feature in the vertical direction at h, w, b denotes the extension in the horizontal direction, N

[0091] Step S22: Dimension matching of multi-layer neighborhood features is performed using the following formula:

[0092]

[0093] By calculating P s,p (φ i,j+n ) and aggregating it with the corresponding feature block at the highest resolution, T i denotes the selection of the down-sampling or up-sampling method for the feature map, the present application can use Bilinear interpolation to adjust the dimensions of P s,p (φ i,j+1 ) to P s,p (φ i,j+n ) so as to match the dimensions of |P s,p (φ i,j )|.

[0094] Step S23: An adaptive average pooling function is used to further control the feature dimensions at different scale levels using the following formula:

[0095]

[0096] where φ i,j denotes the neighborhood feature, which describes the local perception feature at (h, w): f agg is an aggregation function of the feature vector in the neighborhood, which is the adaptive average pooling function mentioned above, then for a single feature tensor φ i,j , the set of local perception feature blocks is:

[0097]

[0098] where s is a stride parameter, indicating the stride of feature aggregation, the smaller s is, the denser the generated local feature blocks are, and p denotes the size of the neighborhood.

[0099] Step S24: Feature block splicing. In order to save parameters and computational cost, the features of different layers are spliced along the channel dimension, and the present application can use the Contact operator to perform feature splicing on feature blocks of different scales to obtain a multi-scale feature representation.

[0100] Step S25: data storage is performed for all positive example training samples to form a memory library, which can be defined as:

[0101]

[0102] In some embodiments, the method provided by the embodiments of the present application includes, if the amount of data in the memory library is greater than the preset data amount threshold, reducing the memory library by using a feature library core set reduction algorithm, and updating the memory library, including:

[0103] arbitrarily selecting a plurality of feature vector blocks from the memory library;

[0104] calculating the distance between all vectors in the memory library and each feature vector block to obtain a first distance set;

[0105] determining a first feature vector block according to the first distance set; and placing the first feature vector block into a new memory library;

[0106] calculating the distance between all vectors in the memory library and the first feature vector block to obtain a second distance set;

[0107] performing a bit operation on the values in the first distance set and the second distance set, and determining a third distance set according to the bit operation result;

[0108] determining a second feature vector block according to the third distance set; and placing the second feature vector block into the new memory library;

[0109] taking the second distance set as a new first distance set, taking the second feature vector block as a new first feature vector block, and returning to the step of calculating the distance between all vectors in the memory library and the first feature vector block to obtain a second distance set, until the number of loops is greater than or equal to a preset number threshold; and taking the obtained new memory library as an updated memory library.

[0110] In some possible implementations, the first distance set can be an average of a plurality of distances, and the first feature vector block can be a feature vector corresponding to the maximum / minimum distance in the first distance set.

[0111] Local feature library core set reduction: as the number of positive samples in the training set increases, the feature library also increases, thereby significantly increasing the inference time and required storage space of new test data. Therefore, it is necessary to simplify the resolution and quantity of the memory library M while retaining a large amount of encoding information of M. The feature library core set reduction algorithm is proposed to abstract the memory library M reduction problem as a facility location problem to ensure that the core set M C The coverage of the feature space substantially remains consistent with the original memory library M:

[0112]

[0113] The specific algorithm is as follows:

[0114] Input: pre-trained Positive sample data X to be trained N , the number of core set targets L, and the random linear projection ψ.

[0115] Output: Memory bank Mc.

[0116] (1) Initialize the memory bank M<-{},

[0117] (2) Traverse all xi∈X N , M<-M∪P s,p (φ j (xi)),

[0118] (3) Initialize Mc to be a random L feature vector block in the M memory bank,

[0119] (4) Traverse any feature block Pi (i∈[0,…,L-1]) in Mc, calculate the distance from all vectors in the memory bank M to Pi, and obtain the vector distance Di, update Pi=Pi+Di,

[0120] (5) Loop until Pi is no longer updated, and obtain the simplified local feature memory library M C .

[0121] In some embodiments, reference Figure 5 As shown, the method provided in the embodiment of the present application performs metric learning on the extracted features and the features in the memory library, and obtains an anomaly score based on the metric learning results, including:

[0122] Step S310, obtaining extracted features and extracted neighborhood features of the extracted features;

[0123] Step S320, determining the maximum value of the distances between the extracted feature, the extracted neighborhood feature, and each feature in the memory library to obtain the maximum distance;

[0124] Step S330, obtaining the weighted feature block closest to the extracted feature in the memory bank;

[0125] Step S340: weighting the maximum distance according to the weighted feature blocks and the extracted features to obtain an anomaly score.

[0126] In some possible implementations, local block feature metric learning: By using the feature library Mc trained with positive samples, this application proposes to calculate the maximum distance score s *to estimate the image-level anomaly score s of the test image x test, which represents the maximum distance between the patch features in the test patch feature set and their respective nearest neighbors m *

[0127]

[0128] the maximum distance score s * (i.e., the maximum distance in this application) is used to measure the most mismatched patch feature in the test image, thereby identifying the abnormal region that may exist.

[0129] In order to obtain the optimized anomaly score s, the present application uses scaling of s * to consider the behavior of adjacent patches: if the feature m test,* closest to the anomaly candidate m * in the memory bank has a large distance between its adjacent samples, it is already a rare normal case, and the anomaly score is increased using the following formula:

[0130]

[0131] where N b is the b feature blocks (i.e., weighted feature blocks) in the memory bank M that are closest in spatial distance to the test feature block, and this reweighting method increases the detection ability of rare normal cases, making the model more robust when facing edge cases.

[0132] In some embodiments, as shown in FIG. 13, the method provided by the embodiments of the present application includes the following steps: Figure 6

[0133] In step S410, the anomaly scores at different positions of the metal surface to be detected are interpolated, and the interpolation result is overlaid with the original image information to obtain a heat map.

[0134] In step S420, the heat map is segmented according to the anomaly value threshold to obtain a detection result.

[0135] In some possible implementations, the anomaly value-based flaw segmentation: through the anomaly values at different positions (w, h), in order to match the original input resolution, the present application further optimizes and smooths the result through bilinear interpolation upsampling and Gaussian filtering, and overlays the result on the original test image to generate a heat map. The flaw position is segmented through the anomaly value threshold.

[0136] In some embodiments, the method provided by the embodiments of the present application extracts local features from the image sample information to obtain extracted sample features, including:

[0137] ​​Infer the image sample information through the pre-trained network to obtain multi-layer feature encoding;

[0138] Extract the middle-layer feature encoding as the extracted sample feature.

[0139] Regarding local feature extraction: use X N to represent all positive sample images available during training (for all x e X N , y x = 0, and correspondingly, define X T to represent the sample set provided during testing, for all x e X T , y x e {0, 1} represents whether the image x is a normal sample (0) or a defective sample (1). The pre-trained network used is denoted as φ. Use φ i,j to represent the final output feature of image xi through the pre-trained network j-level corresponding spatial resolution block. In order to preserve more local texture feature information of the metal surface, the present application extracts features containing middle-level features to form a memory library M for training as an unsupervised feature library.

[0140] For local feature extraction of the sample to be tested: first, the image needs to be preprocessed into the format of the pre-trained model input, and then the local feature extraction and neighborhood feature aggregation method is used to obtain the neighborhood aggregated feature map of the reduced dimension of the image to be tested.

[0141] Next, combined with specific application examples, the scheme of the embodiments of the present application is described and explained in detail with reference to Figure 7 and Figure 8 , taking an industrial quality inspection task as an example:

[0142] Step S31, using resnet50 as a pre-trained model, 100 normal metal workpiece samples are collected from industrial actual production, the pictures are provided by the factory customers, preprocessed to generate 224x224 size square photos, resnet50 is used for inference to obtain 4-layer feature encoding.

[0143] Step S32, the (100, 512, 28, 28) dimensional feature encoding output by layer2 is aggregated to obtain feature values more suitable for defect detection, where 100 is the number of training pictures. Using the torch.nn.Unfold function in pytorch, the feature encoding output by layer2 is aggregated, patch_size = 3, stride = 1, padding = 1, to obtain (100, 784, 512, 3, 3) dimensional feature encoding, and the first two dimensions are merged to obtain (78400, 512, 3, 3) dimensional feature.

[0144] Step S33, encode the features of the layer3 output with the same method, and obtain the (100, 196, 1024, 3, 3) dimension defect detection feature code. Interpolate the layer3 output to match the layer2, and finally obtain the feature dimension of (78400, 1024, 3, 3).

[0145] Step S34, stack the two layer feature codes after adaptive mean pooling, obtain the (78400, 2, 1024) feature code, and further adaptive mean pooling to obtain the (78400, 1024) feature library with the dimension of 1024 and the number of 78400.

[0146] Step S35, simplify the feature library, simplify the dimension 1024 to 128 through a fully connected layer, and downsample the feature library through the following method. In the initial stage, randomly select 10 indexes, calculate the Euclidean distance of all feature points with respect to the ten points through the following code to obtain the set D(78400, 10). Then take the mean value to obtain the average distance set d1(78400, 1). Select the value with the largest value as a point x of Mc. Then calculate the Euclidean distance d2(78400, 1) of x with respect to all points. Perform corresponding bit operation on d1 and d2, and retain the smaller value to obtain d3. Finally, take the largest value in d3 and add it to Mc to become a new x. Repeat the above steps 78400*0.1 times. Finally, the simplified feature library size is (7840, 128).

[0147] Step S36, in the inference stage, preprocess the image collected by the industrial camera on the client side, and crop it into a square picture with a size of 224x224. Inference through the resnet pre-training model to obtain the feature value sequence output by the four residual blocks. Output the feature value sequence and the unique number corresponding to the image.

[0148] Step S37, fuse the feature value sequence, and obtain the (28, 28, 128) dimension feature value for Euclidean distance calculation. The (28, 28) dimension distance matrix is used as the abnormal value judgment of this position. Then, through the 2D interpolation method, the distance matrix is expanded to a (224, 224) dimension matrix to obtain the abnormal value mask.

[0149] Step S38, connect the abnormal detection judgment result with the image unique code, and feed back to the factory client. The client links the factory hardware to process the abnormal workpiece, such as stopping the workpiece, sorting the unloading mechanism, flashing the warning light, and counting the defect workpiece.

[0150] Therefore, the data acquisition cost of the embodiments of the present application is reduced: in many industrial scenarios, especially in industries with extremely high requirements for product quality, acquiring a large number of defect samples can be costly because they require a large amount of manual annotation or can only be captured under specific conditions. The metal surface defect detection technology provided by the patent can be effectively trained with only a small number of samples, thereby reducing the cost of data acquisition and annotation.

[0151] The embodiments of the present application can quickly adapt to new product lines: factories may need to frequently adjust or replace production lines to adapt to market changes. The metal surface defect detection technology provided by the patent enables the quality inspection system to quickly adapt to the detection needs of new products.

[0152] The embodiments of the present application are beneficial to improve production efficiency: traditional defect detection methods may require a large number of samples to train the model, which consumes a lot of time and resources. The metal surface defect detection technology provided by the patent can establish an effective detection model in a short period of time, thereby speeding up the production process and improving the overall production efficiency.

[0153] Please refer to Figure 9 The embodiments of the present application also provide a metal surface detection system, which can implement the above metal surface detection method. The system comprises:

[0154] The first module 810 is configured to acquire image information of a metal surface to be detected.

[0155] The second module 820 is configured to perform local feature extraction on the image information to obtain extracted features.

[0156] The third module 830 is configured to perform metric learning on the extracted features and the features in the memory bank, and obtain an abnormal score according to the metric learning result. The memory bank is established according to the nominal positive samples.

[0157] The fourth module 840 is configured to perform defect segmentation on the metal surface to be detected according to the abnormal score to obtain a detection result.

[0158] It can be understood that the contents in the above method embodiments are applicable to the system embodiments, the system embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0159] The embodiments of the present application also provide an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the above metal surface detection method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0160] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiments, the present device embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0161] Please refer to Figure 10 , Figure 10 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0162] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the present application.

[0163] The memory 902 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 902 can store an operating system and other application programs. When the technical solutions provided by the present application are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the metal surface detection method of the present application.

[0164] The input / output interface 903 is used to realize information input and output.

[0165] The communication interface 904 is used to realize the communication interaction between the present device and other devices. The communication can be realized by a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.).

[0166] The bus 905 is used to transmit information between various components (such as the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0167] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between them in the device.

[0168] The present application also provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above metal surface detection method is realized.

[0169] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiments, the present storage medium embodiments specifically implement the functions same as those of the above method embodiments, and achieve the same beneficial effects as those of the above method embodiments.

[0170] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory disposed remotely relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0171] The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than those shown in the figures, or combine certain steps, or different steps.

[0173] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0174] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0175] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of this application, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of orderly or chronological mundane operation, reverse order operation, based on circuitry availability, based on stated preference or the like, and that "default" or other orderings are thus permissible. Further, the terms "comprise", "comprising", "include", "including", and the like, are specifically intended to be open-ended. That is, references to individual steps and the like do not suhstantially exclude the presence of two or more of a given step or its integral presence in the process, method, system, article, or apparatus having been made with a wider scope. The use of notation such as "first", "second", "third", etc. does not generally limit the areas, but can be used for clarity, and merely establishes the order of the steps or placement of components. Moreover, singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise.

[0176] It should be understood that, in the application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0177] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the above-described device embodiments are only illustrative, for example, the division of the above-mentioned units is only a logical functional division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0178] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0179] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0180] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in part, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions used to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store programs.

[0181] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A metal surface detection method, characterized in that: The method comprises: Acquire image information of the metal surface to be detected; Performing local feature extraction on the image information to obtain extracted features; Performing metric learning on the extracted features and features in a memory database, and obtaining an anomaly score based on the metric learning result; the memory database is established based on nominal positive samples; According to the abnormality score, the metal surface to be detected is segmented to obtain a detection result; The extracting features are subjected to metric learning with the features in the memory library, and an anomaly score is obtained based on the metric learning result, including: Obtaining the extracted features and extracted neighborhood features of the extracted features; Determine the maximum value among the distances between the extracted feature, the extracted neighborhood feature and each feature in the memory library to obtain a maximum distance; Obtaining a weighted feature block in the memory bank that is closest to the extracted feature; The maximum distance is weighted according to the weighted feature block and the extracted feature to obtain an anomaly score.

2. The method according to claim 1, characterized in that The memory bank is determined by the following steps: Acquiring image sample information, and performing local feature extraction on the image sample information to obtain extracted sample features; Performing neighborhood feature fusion processing on the extracted sample features, and storing the fused positive sample features to obtain a memory library; If the amount of data in the memory bank is greater than a preset data amount threshold, the memory bank is reduced by a feature library core set reduction algorithm to update the memory bank.

3. The method according to claim 2, characterized in that The performing neighborhood feature fusion processing on the extracted sample features and storing the fused positive sample features to obtain a memory library includes: Performing neighborhood expansion on the extracted sample features, and performing dimension matching on the obtained multi-layer neighborhood features to obtain neighborhood matching features; Pooling is performed on neighborhood matching features at different levels, and features at different scales are concatenated to obtain multi-scale representation features; According to the multi-scale representation features, positive sample features are determined to obtain a memory library.

4. The method according to claim 2, characterized in that If the amount of data in the memory bank is greater than a preset data amount threshold, reducing the memory bank by using a feature library core set reduction algorithm, and updating the memory bank, including: arbitrarily selecting a number of feature vector blocks from the memory bank; Calculating the distance between all vectors in the memory bank and each of the feature vector blocks to obtain a first distance set; Determine a first feature vector block according to the first distance set; and put the first feature vector block into a new memory bank; Calculating the distances between all vectors in the memory bank and the first feature vector block to obtain a second distance set; performing a bitwise operation on the values ​​in the first distance set and the second distance set, and determining a third distance set according to the bitwise operation result; Determine a second feature vector block according to the third distance set; and place the second feature vector block into a new memory bank; The second distance set is used as a new first distance set, the second feature vector block is used as a new first feature vector block, and the step of calculating the distance between all vectors in the memory bank and the first feature vector block to obtain the second distance set is returned until the number of loops is greater than or equal to a preset number threshold; the obtained new memory bank is used as the updated memory bank.

5. The method according to claim 1, wherein The defect segmentation of the metal surface to be detected is performed according to the abnormality score to obtain a detection result, including: Performing interpolation processing on the anomaly scores at different positions on the metal surface to be detected, and overlaying the interpolation results with the original image information to obtain a heat map; According to the outlier threshold, the heat map is segmented into defect positions to obtain a detection result.

6. The method according to claim 2, characterized in that The extracting local features of the image sample information to obtain extracted sample features includes: Inferring the image sample information through a pre-trained network to obtain multi-layer feature encoding; Extract the mid-level feature code as the extracted sample feature.

7. A metal surface detection system, characterized in that: The system comprises: The first module is used to obtain image information of the metal surface to be detected; The second module is used to extract local features of the image information to obtain extracted features; The third module is used to perform metric learning on the extracted features and the features in the memory bank, and obtain an anomaly score based on the metric learning results; the memory bank is established based on nominal positive samples; A fourth module is configured to segment the metal surface to be inspected according to the anomaly score to obtain a detection result; The third module is used to: Obtaining the extracted features and extracted neighborhood features of the extracted features; Determine the maximum value among the distances between the extracted feature, the extracted neighborhood feature and each feature in the memory library to obtain a maximum distance; Obtaining a weighted feature block in the memory bank that is closest to the extracted feature; The maximum distance is weighted according to the weighted feature block and the extracted feature to obtain an anomaly score.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images