Metal surface detection method and system, electronic equipment and storage medium

Abstract

The invention discloses a metal surface detection method and system, electronic equipment and a storage medium. The method comprises the steps that image information of a to-be-detected metal surface is acquired; performing local feature extraction on the image information to obtain extracted features; performing metric learning on the extracted features and features in a memory bank, and obtaining an abnormal score according to a metric learning result; the memory bank is established according to a nominal positive sample; and according to the abnormal score, performing flaw segmentation on the metal surface to be detected to obtain a detection result. According to the method, the local features of the metal surface and the features in the memory bank representing the nominal positive samples are subjected to metric learning, and whether the metal surface is abnormal or not is determined. The embodiment of the invention is beneficial to improving the accuracy of metal surface detection. The method can be widely applied to the technical field of Internet of Things.

Description

Technical Field

[0001] The present application relates to the technical field of Internet of Things, and in particular to a metal surface detection method, system, electronic device and storage medium. Background Art

[0002] In various sheet metal manufacturing, intelligent metal surface defect detection technology can identify the appearance defects of various industrial products to ensure product quality and maintain production stability. For example, the defect detection method in the traditional sheet metal profile manufacturing industry usually relies on manual sampling, which is difficult to meet the needs of large-scale quality inspection. With the continuous advancement of technologies such as industrial imaging, computer vision and deep learning, vision-based metal surface defect detection technology has made significant progress and has become an effective product appearance quality inspection solution.

[0003] In the related art, metal surface defects are detected by artificial intelligence. However, there are many types of surface defects, and the number of defect images is small and difficult to obtain, which leads to low prediction accuracy of the model, thus affecting the accuracy of metal surface detection. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a high-accuracy metal surface detection method, system, electronic device and storage medium.

[0005] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application proposes a metal surface detection method, the method comprising: obtaining image information of a metal surface to be detected; extracting local features of the image information to obtain extracted features; performing metric learning on the extracted features and features in a memory library, and obtaining anomaly scores based on the metric learning results; the memory library is established based on nominal positive samples; based on the anomaly scores, defect segmentation is performed on the metal surface to be detected to obtain detection results. An embodiment of the present application determines whether there is an abnormality on the metal surface by performing metric learning on the local features of the metal surface and features in a memory library representing nominal positive samples. An embodiment of the present application is conducive to improving the accuracy of metal surface detection.

[0006] In some embodiments, in the method provided by the embodiments of the present application, the memory bank is determined by the following steps:

[0007] Acquire image sample information, and perform local feature extraction on the image sample information to obtain extracted sample features;

[0008] Performing neighborhood feature fusion processing on the extracted sample features, and storing the fused positive sample features to obtain a memory library;

[0009] If the amount of data in the memory library is greater than a preset data amount threshold, the memory library is reduced by a feature library core set reduction algorithm to update the memory library.

[0010] In some embodiments, the method provided in the embodiments of the present application, wherein the extracted sample features are subjected to neighborhood feature fusion processing, and the fused positive sample features are stored to obtain a memory library, comprises:

[0011] Performing neighborhood expansion on the extracted sample features, and performing dimension matching on the obtained multi-layer neighborhood features to obtain neighborhood matching features;

[0012] The neighborhood matching features at different levels are pooled, and the features of different scales are concatenated to obtain multi-scale representation features;

[0013] According to the multi-scale representation features, positive sample features are determined to obtain a memory library.

[0014] In some embodiments, the method provided in the embodiments of the present application, if the amount of data in the memory bank is greater than a preset data amount threshold, the memory bank is reduced by a feature library core set reduction algorithm to update the memory bank, including:

[0015] arbitrarily selecting a number of feature vector blocks from the memory bank;

[0016] Calculating the distance between all vectors in the memory bank and each of the feature vector blocks to obtain a first distance set;

[0017] Determine a first feature vector block according to the first distance set; and put the first feature vector block into a new memory bank;

[0018] Calculating the distances between all vectors in the memory bank and the first feature vector block to obtain a second distance set;

[0019] Performing a bit operation on the values ​​in the first distance set and the second distance set, and determining a third distance set according to the bit operation result;

[0020] Determine a second feature vector block according to the third distance set; and put the second feature vector block into a new memory bank;

[0021] The second distance set is used as a new first distance set, the second feature vector block is used as a new first feature vector block, and the step of calculating the distance between all vectors in the memory bank and the first feature vector block to obtain the second distance set is returned until the number of loops is greater than or equal to a preset number threshold; the obtained new memory bank is used as an updated memory bank.

[0022] In some embodiments, the method provided in the embodiments of the present application, wherein the extracted features are subjected to metric learning with the features in the memory library, and an abnormality score is obtained according to the metric learning result, including:

[0023] Acquire the extracted features and extracted neighborhood features of the extracted features;

[0024] Determine the maximum value of the distances between the extracted feature, the extracted neighborhood feature and each feature in the memory library to obtain a maximum distance;

[0025] Obtaining a weighted feature block in the memory bank that is closest to the extracted feature;

[0026] The maximum distance is weighted according to the weighted feature block and the extracted feature to obtain an anomaly score.

[0027] In some embodiments, the method provided in the embodiments of the present application, performing defect segmentation on the metal surface to be detected according to the abnormality score to obtain the detection result includes:

[0028] Performing interpolation processing on the abnormality scores at different positions of the metal surface to be detected, and overlaying the interpolation result with the original image information to obtain a thermal map;

[0029] According to the outlier threshold, the thermal map is segmented into defect positions to obtain a detection result.

[0030] In some embodiments, the method provided by the embodiments of the present application, extracting local features from the image sample information to obtain extracted sample features, includes:

[0031] Inferring the image sample information through a pre-trained network to obtain multi-layer feature encoding;

[0032] The middle-level feature codes are extracted as the extracted sample features.

[0033] To achieve the above object, another aspect of the embodiment of the present application provides a metal surface detection system, the system comprising:

[0034] The first module is used to obtain image information of the metal surface to be detected;

[0035] The second module is used to extract local features of the image information to obtain extracted features;

[0036] The third module is used to perform metric learning on the extracted features and the features in the memory bank, and obtain an abnormality score according to the metric learning result; the memory bank is established according to the nominal positive sample;

[0037] The fourth module is used to perform defect segmentation on the metal surface to be detected according to the abnormality score to obtain a detection result.

[0038] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0039] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0040] The embodiments of the present application include at least the following beneficial effects: The method provided by the embodiments of the present application includes: acquiring image information of the metal surface to be detected; extracting local features of the image information to obtain extracted features; performing metric learning on the extracted features and features in a memory library, and obtaining anomaly scores based on the metric learning results; the memory library is established based on nominal positive samples; based on the anomaly scores, defect segmentation is performed on the metal surface to be detected to obtain detection results. The embodiments of the present application determine whether there is an abnormality on the metal surface by performing metric learning on the local features of the metal surface and the features in the memory library representing the nominal positive samples. The embodiments of the present application are conducive to improving the accuracy of metal surface detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of an embodiment of the metal surface detection method provided by the present application;

[0042] Figure 2 It is a flowchart of an embodiment of the establishment and reasoning of the memory bank provided by the present application;

[0043] Figure 3 is a flow chart of an embodiment of a process for determining a memory bank provided by the present application;

[0044] Figure 4 It is a flow chart of an embodiment of the preliminary establishment of a memory bank provided by the present application;

[0045] Figure 5 is a flowchart of an embodiment of a process for determining an abnormality score provided by the present application;

[0046] Figure 6 is a flow chart of an embodiment of the defect segmentation process provided by the present application;

[0047] Figure 7 is a flow chart of another embodiment of establishing a memory bank provided by the present application;

[0048] Figure 8 is a flow chart of an embodiment of the detection reasoning process provided by the present application;

[0049] Fig. 9 is a schematic diagram of the structure of a metal surface detection system provided in an embodiment of the present application;

[0050] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0052] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0053] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0055] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0056] Metal surface defect detection: Metal surface defect detection is an important part of industrial quality control. It involves the use of various technologies to identify and classify defects or anomalies on the surface of metal products. These defects may affect the appearance, durability or functionality of the product. Therefore, detecting and removing or repairing these defects is crucial to ensure product quality and safety. Including but not limited to scratches, dents, cracks, holes, rust, discoloration, foreign matter inclusions, etc.

[0057] Local block features: Global features refer to the overall properties of an image. Common global features include color features, texture features, and shape features, such as intensity histograms. Since they are low-level visual features at the pixel level, global features have good invariance, simple calculations, and intuitive representations. However, their fatal weaknesses are high feature dimensions and large amounts of calculations. Compared with global image features such as line features, texture features, and structural features, local image features are rich in number, have low correlation between features, and will not affect the detection and matching of other features due to the disappearance of some features in the case of occlusion.

[0058] Perceptual aggregation: Perceptual aggregation is a feature fusion strategy that aims to obtain richer and more robust feature representations by combining multi-scale and multi-level features in deep neural networks. In computer vision tasks such as image classification, object detection, and semantic segmentation, perceptual aggregation enables the model to better understand the image content, thereby improving performance. Deep convolutional neural networks abstract image information layer by layer through multiple convolutional layers, each of which captures perceptual fields of different sizes. Lower layers usually capture details and texture information, while higher layers capture more abstract semantic information. Perceptual aggregation enables the model to consider both local details and global context by fusing features from these different levels. In Feature Pyramid Network (FPN) or similar architectures, perceptual aggregation involves merging feature maps of different depths. This is usually achieved through top-down paths and lateral connections, allowing information to flow and enhance at different scales.

[0059] In the field of 3C sheet metal manufacturing, intelligent metal surface defect detection technology can identify the appearance defects of various industrial products to ensure product quality and maintain production stability. For example, in the traditional sheet metal profile manufacturing industry, defect detection methods usually rely on manual sampling, which is difficult to meet the needs of large-scale quality inspection. In recent years, with the continuous progress in fields such as industrial imaging, computer vision and deep learning, vision-based metal surface defect detection technology has made significant progress and has become an effective product appearance quality inspection solution. In actual business scenarios, defect detection usually faces three problems. First, in many industrial scenarios, it is relatively easy to obtain image samples of normal conditions, but it is costly and complicated to fully specify the expected defect changes; second, there are many types of defects, including but not limited to scratches, dents, cracks, holes, rust, discoloration, foreign body inclusions, etc. Errors may range from subtle changes, such as small scratches, to larger structural defects, such as missing components; third, there is unknown, and the defects of parts are usually unpredictable. Traditional machine learning methods may not be able to accurately predict the defects of actual production. In addition, the conversion of production has caused the old defect detection methods to become ineffective, and the high training cost is also the main reason why AI quality inspection for metal surface defect detection is difficult to quickly form a scale effect. Therefore, there is an urgent need for a flexible and versatile metal surface defect detection method to meet the large-scale quality inspection needs of the metal manufacturing industry.

[0060] In this regard, the present application provides a metal surface defect detection solution based on local block feature perception aggregation. In the sheet metal profile processing and manufacturing industry, it is relatively easy to obtain normal image samples for the final metal material products, but it is costly and complicated to fully specify the expected defect changes. In addition, traditional visual inspection methods usually have difficulty in obtaining good detection results for defects outside the distribution of training samples. The classification of defects outside the distribution of metal surface defect detection is particularly difficult because there are many types of defects and they are difficult to enumerate. The metal surface defect detection method proposed in this application is a detection method based on computer vision, that is, the image pixel level. The core idea is to use normal samples commonly used in industry to train a detection model. The model can obtain the local features of any pixel point of the metal surface image, and judge the defect position and confidence of the detection image by calculating the geometric distance between the local features of the pixel points in the detection image and the normal sample.

[0061] The metal surface detection method provided in the embodiment of the present application relates to the field of Internet of Things technology. The metal surface detection method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements the metal surface detection method, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0064] Figure 1 is an optional flow chart of the metal surface detection method provided in the embodiment of the present application; Figure 1The method may include but is not limited to steps S100 to S400.

[0065] Step S100, obtaining image information of the metal surface to be detected;

[0066] Step S200, extracting local features from the image information to obtain extracted features;

[0067] Step S300, performing metric learning on the extracted features and the features in the memory library, and obtaining an abnormality score based on the metric learning result; the memory library is established based on the nominal positive samples;

[0068] Step S400: performing defect segmentation on the metal surface to be inspected according to the abnormality score to obtain the inspection result.

[0069] It is understandable that metric learning is also called similarity learning, which is a machine learning method that measures the similarity between samples by learning a distance function (metric). The memory bank in the embodiment of the present application can be established based on a nominal positive sample, that is, a metal that appears to be flawless on the surface. The present application compares the local feature data of the metal surface to be tested with the feature data of a large number of positive samples, determines the abnormality score based on the comparison, and then determines the defect location.

[0070] The metal surface defect detection technology proposed in this application is based on an innovative unsupervised learning method. The core assumption is that a large number of flawless (nominal) sample images can be obtained during the model training phase. These images are used as training data to build a representative feature memory library. In industrial applications, such images can usually be easily collected from defect-free areas of the product. In addition, considering that in practical applications, defective samples are often difficult to obtain, especially before the model is deployed. Therefore, this application does not rely on the availability of defective samples, but focuses on the detection of abnormal patterns that have never been seen. The algorithm achieves anomaly detection by constructing a memory library containing rich nominal features. Each feature vector in the memory library represents the visual characteristics of a local area in the image. The algorithm uses these feature vectors to identify abnormal areas by calculating the similarity between each local area in the test image and the feature vector in the memory library. The advantage of this method is that it can effectively extract and utilize contextual information from nominal samples while being highly sensitive to unknown abnormal patterns.

[0071] Reference Figure 2As shown in the figure, the content involved in this solution mainly includes two processes: local feature library training and construction (i.e., memory library establishment) and reasoning (i.e., metal surface detection). The local feature library construction process includes local feature extraction, neighborhood feature perception aggregation, and local feature library core set reduction; metal defect model reasoning detection mainly includes: local feature extraction of samples to be tested, local block feature metric learning, and defect segmentation based on outliers. The following is a brief introduction to each step.

[0072] Local feature extraction: The local feature block feature extraction of metal surface defect detection mainly uses the middle feature layer selected from the deep neural network pre-training model as the main feature of defect detection, with the purpose of retaining local texture information and surface details. For tasks such as metal surface defect detection that require local and detailed detection, too deep a layer will lead to over-abstract features, thus losing the local feature information that should have been retained.

[0073] Aggregation of neighborhood features: In order to increase the size of the receptive field and the robustness to small spatial deviations without losing the spatial resolution or the availability of the feature map, this patent further aggregates the neighborhood features. This is to consider the multi-scale correspondence between the template positive sample and the defect sample, on the one hand, taking into account the correlation of the neighborhood pixels in the image, and on the other hand, taking into account the multi-scale differences of different images at multiple resolutions, to ensure that while maintaining high resolution and high accuracy, it can improve the robustness to spatial changes and noise, thereby achieving more effective defect detection and positioning.

[0074] Local feature library core set reduction: As the number of positive samples in the training set increases, the feature library is also increasing, which leads to a significant increase in the inference time and required storage space of new test data. Therefore, this application proposes to find a subset Mc (updated memory library) of M (original feature library / memory library) so that Mc can maintain a good representation of the original memory library while greatly reducing its storage requirements and inference time. It is usually necessary to reduce the dimension of the feature library and preserve the encoding features of good data as much as possible. This application proposes a core set sampling mechanism to reduce the original feature library, reduce the consumption of system resources during the detection process, and improve the detection speed.

[0075] Extraction of local features of the sample to be tested: This step mainly uses the pre-trained model and neighborhood feature aggregation to obtain the aggregated feature map of different positions of the sample to be tested, so as to determine the defect location.

[0076] Local block feature metric learning: Using the trained feature block memory M, we estimate the image-level anomaly score of the test image x_test at position (w, h) by the maximum distance score s between the feature block in its memory and the test feature block, and perform weighted maintenance on the outliers by the distance between (w, h) and the neighboring sample quality inspection.

[0077] Defect segmentation based on outliers: Through the outliers at different positions (w, h), in order to match the original input resolution, this application further optimizes and smoothes the results through bilinear interpolation upsampling and Gaussian filtering, overlays them on the original test image, and generates a heat map. The defect position is segmented by the outlier threshold.

[0078] In some embodiments, reference Figure 3 In the method provided in the embodiment of the present application, the memory bank is determined by the following steps:

[0079] Step S500, acquiring image sample information, and performing local feature extraction on the image sample information to obtain extracted sample features;

[0080] Step S600, performing neighborhood feature fusion processing on the extracted sample features, and storing the fused positive sample features to obtain a memory library;

[0081] Step S700: If the amount of data in the memory is greater than a preset data amount threshold, the memory is reduced by a feature library core set reduction algorithm to update the memory.

[0082] In some embodiments, reference Figure 4 The method provided in the embodiment of the present application performs neighborhood feature fusion processing on the extracted sample features, and stores the fused positive sample features to obtain a memory library, including:

[0083] Step S610, performing neighborhood expansion on the extracted sample features, and performing dimension matching on the obtained multi-layer neighborhood features to obtain neighborhood matching features;

[0084] Step S620, performing pooling processing on neighborhood matching features at different levels, and performing feature splicing on features at different scales to obtain multi-scale representation features;

[0085] Step S630, determining positive sample features according to the multi-scale representation features, and obtaining a memory library.

[0086] Specifically, neighborhood feature fusion: using feature map φ i,j (h,w) represents the feature slice at position (h,w), so that the feature can cope with local spatial changes and cover a large enough receptive field without losing spatial resolution or feature availability.

[0087] The specific steps are as follows:

[0088] Step S21: The feature slice is expanded in the neighborhood by the following formula:

[0089]

[0090] Where N p (h,w) represents a neighborhood of size p×p near the position (h, w), a represents the vertical expansion of the feature at h, w, b represents the horizontal expansion, N p (h,w) Contains the features of all pixels in the area of ​​size p×p near (h, w).

[0091] Step S22: Dimension matching of multi-layer neighborhood features is performed using the following formula:

[0092]

[0093] By calculating P s,p (φ i,j+n ) and aggregate it with the corresponding feature block at the highest resolution, T i Indicates the choice of downsampling or upsampling method for the feature map. This application can use Bilinear interpolation to convert P s,p (φ i,j+1 ) to P s,p (φ i,j+n ) to adjust the dimension so that it is consistent with |P s,p (φ i,j )|Dimension matching.

[0094] Step S23: Use the adaptive average pooling function to further control the feature dimensions at different scale levels using the following formula:

[0095]

[0096] where φ i,j Represents the neighborhood feature, which describes the local perceptual feature at (h, w): agg is the aggregation function of the feature vectors in the neighborhood, here is the adaptive average pooling function mentioned above, then for a single feature tensor φ i,j , its local perception feature block set is:

[0097]

[0098] Where s is the stride parameter, which indicates the stride of feature aggregation. The smaller s is, the denser the generated local feature blocks are. p indicates the size of the neighborhood.

[0099] Step S24: feature block splicing. In order to save parameters and calculation amount, the features of different layers are spliced ​​along the channel dimension. The present application can use the Contact operator to splice the feature blocks of different scales to obtain a multi-scale feature representation.

[0100] Step S25: Store data for all positive training samples to form a memory bank. The memory bank M can be defined as:

[0101]

[0102] In some embodiments, the method provided in the embodiments of the present application, if the amount of data in the memory library is greater than a preset data amount threshold, the memory library is reduced by a feature library core set reduction algorithm to update the memory library, including:

[0103] Randomly select several feature vector blocks from the memory bank;

[0104] Calculate the distance between all vectors in the memory bank and each feature vector block to obtain a first distance set;

[0105] Determine a first feature vector block according to the first distance set; put the first feature vector block into a new memory bank;

[0106] Calculate the distances between all vectors in the memory bank and the first feature vector block to obtain a second distance set;

[0107] Performing bitwise operations on the values ​​in the first distance set and the second distance set, and determining a third distance set according to the bitwise operation results;

[0108] Determine a second feature vector block according to the third distance set; put the second feature vector block into a new memory bank;

[0109] The second distance set is used as the new first distance set, the second feature vector block is used as the new first feature vector block, and the distance between all vectors in the memory bank and the first feature vector block is calculated back to obtain the second distance set. This step is repeated until the number of loops is greater than or equal to a preset number threshold; the obtained new memory bank is used as the updated memory bank.

[0110] In some possible implementations, the first distance set may be an average value of multiple distances, and the first feature vector block may be a feature vector corresponding to the maximum / minimum distance in the first distance set.

[0111] Local feature library core set reduction: As the number of positive samples in the training set increases, the feature library also continues to grow, resulting in a significant increase in the inference time and storage space required for new test data. Therefore, it is necessary to simplify the resolution and number of M while retaining a large amount of encoded information in the memory library M. This application proposes a feature library core set reduction algorithm, which abstracts the memory library M reduction problem into a facility location problem to ensure that the core set M selected by sampling is C The coverage of the feature space is roughly consistent with the original memory library M:

[0112]

[0113] The specific algorithm is as follows:

[0114] Input: Pre-trained Positive sample data to be trained X N , the number of core set targets L, and the random linear projection ψ.

[0115] Output: Memory bank Mc.

[0116] (1) Initialize the memory bank M<-{},

[0117] (2) Traverse all xi∈X N , M<-M∪P s,p (φ j (xi)),

[0118] (3) Initialize Mc to be a random L feature vector block in the M memory bank,

[0119] (4) Traverse any feature block Pi (i∈[0,…,L-1]) in Mc, calculate the distance from all vectors in the memory bank M to Pi, and obtain the vector distance Di, update Pi=Pi+Di,

[0120] (5) Loop until Pi is no longer updated, and obtain the simplified local feature memory library M C .

[0121] In some embodiments, reference Figure 5 As shown, the method provided in the embodiment of the present application performs metric learning on the extracted features and the features in the memory library, and obtains an abnormality score based on the metric learning result, including:

[0122] Step S310, obtaining extracted features and extracted neighborhood features of the extracted features;

[0123] Step S320, determining the maximum value of the distances between the extracted feature, the extracted neighborhood feature and each feature in the memory library to obtain the maximum distance;

[0124] Step S330, obtaining the weighted feature block in the memory bank that is closest to the extracted feature;

[0125] Step S340, weighting the maximum distance according to the weighted feature blocks and the extracted features to obtain an anomaly score.

[0126] In some possible implementations, local block feature metric learning: By using the feature library Mc trained with positive samples, this application proposes to calculate the maximum distance score s *To estimate the image-level anomaly score s of the test image x_test, the score represents the relationship between the block feature in the test block feature set and its respective nearest neighbor feature m in * Maximum distance between:

[0127]

[0128] Maximum distance score * (ie, the maximum distance in this application) is used to measure the most mismatched block features in the test image, thereby identifying possible abnormal areas.

[0129] In order to obtain the optimized anomaly score s, this patent uses s * The scaling of m to consider the behavior of neighboring blocks: if the memory bank is related to the abnormal candidate m test,* The closest feature m * If the distance between it and its neighboring samples is large and therefore it is already a rare normal case, the anomaly score is increased using the following formula:

[0130]

[0131] Where N b The b feature blocks (i.e., weighted feature blocks) in the memory library M that are closest to the test feature block in space are used. This re-weighting method increases the detection capability of rare normal situations and makes the model more robust in the face of edge cases.

[0132] In some embodiments, reference Figure 6 As shown, the method provided in the embodiment of the present application performs defect segmentation on the metal surface to be detected according to the abnormality score to obtain the detection result, including:

[0133] Step S410, interpolating the abnormality scores at different positions of the metal surface to be detected, and overlaying the interpolation result with the original image information to obtain a heat map;

[0134] Step S420, segmenting the heat map according to the outlier threshold to obtain a detection result.

[0135] In some possible implementations, defect segmentation based on outliers: Through outliers at different positions (w, h), in order to match the original input resolution, this application further optimizes and smoothes the results through bilinear interpolation upsampling and Gaussian filtering, overlays them on the original test image, and generates a heat map. The defect position is segmented by an outlier threshold.

[0136] In some embodiments, the method provided in the embodiments of the present application extracts local features of image sample information to obtain extracted sample features, including:

[0137] The image sample information is inferred through the pre-trained network to obtain multi-layer feature encoding;

[0138] The middle-level feature codes are extracted as the extracted sample features.

[0139] About local feature extraction: Use X N represents all positive sample images available during training (for all x∈X N ,y x = 0, accordingly, define X T is the sample set provided for testing, for all x∈X T ,y x ∈{0,1} indicates whether the image x is a normal sample (0) or a defective sample (1). The pre-trained network used is denoted as φ. i,j It represents the final output features of the spatial resolution block corresponding to the pre-trained network j level. In order to retain more local texture feature information of the metal surface, the present application extracts features containing mid-level features to form a memory library M for training as an unsupervised feature library.

[0140] For the local feature extraction of the samples to be tested: For the samples to be tested, this application first needs to preprocess the image into the format of the pre-trained model input, and then use the method of local feature extraction and aggregation of neighborhood features to obtain the neighborhood aggregation feature map after the dimension of the image to be tested is reduced.

[0141] The following is a detailed description of the embodiments of the present invention in conjunction with specific application examples. Figure 7 and Figure 8 As shown, take an industrial quality inspection task as an example:

[0142] Step S31, using resnet50 as an example of a pre-training model, 100 normal metal workpiece samples are collected from actual industrial production. The pictures are provided by factory customers. After preprocessing, square photos of 224x224 size are generated. Resnet50 is used for inference to obtain 4-layer feature coding.

[0143] Step S32, the feature encoding of the (100, 512, 28, 28) dimension output by layer2 is locally aggregated to obtain a feature value more suitable for defect detection, where 100 is the number of training images. Use the torch.nn.Unfold function in pytorch to aggregate the feature encoding output by layer2, with patch_size=3, stride=1, padding=1, to obtain the feature encoding of the (100, 784, 512, 3, 3) dimension, and merge the first two dimensions to obtain the (78400, 512, 3, 3) dimensional feature.

[0144] In step S33, the feature coding of the dimension (100, 1024, 14, 14) output by layer3 is locally aggregated using the same method to obtain the defect detection feature coding of the dimension (100, 196, 1024, 3, 3). In order to merge it with the layer2 feature, the output of layer3 is interpolated to match it with layer2, and the final feature dimension is (78400, 1024, 3, 3).

[0145] In step S34, the two layers of feature coding are adaptively mean pooled and stacked to obtain feature coding of (78400, 2, 1024), and further adaptive mean pooling is performed to obtain a feature library of (78400, 1024), whose dimension is 1024 and the number is 78400.

[0146] Step S35, simplify the feature library, simplify the dimension 1024 to 128 through a fully connected layer, and downsample the feature library by the following method. In the initial stage, randomly select 10 indexes, calculate the Euclidean distances of all feature points about these ten points through the following code, and obtain the set D(78400,10), then take the average to obtain the average distance set d1(78400,1), select the value with the largest value as a point x of Mc, then calculate the Euclidean distance d2(78400,1) of x about all points, perform corresponding bit operations on d1 and d2, retain the smaller value, and thus obtain d3, finally take the largest value in d3 and add it to Mc, and become the new x, and repeat 78400*0.1 times. The final simplified feature library size is (7840,128).

[0147] Step S36: In the inference phase, on the client side, the image captured by the industrial camera is preprocessed and cropped into a square image of 224x224. Inference is performed through the resnet pre-trained model to obtain the eigenvalue sequence output by the four residual blocks. The eigenvalue sequence and the unique number corresponding to the image are output.

[0148] Step S37, after fusing the eigenvalue sequence, the obtained (28, 28, 128) dimensional eigenvalues ​​are subjected to Euclidean distance calculation, and the calculated (28, 28) dimensional distance matrix is ​​used as the outlier judgment of the position, and then the distance matrix is ​​expanded to a (224, 224) dimensional matrix through a 2D interpolation method to obtain the mask of the outlier.

[0149] Step S38, the abnormal detection judgment result is connected to the unique image code and fed back to the factory client. The client connects to the factory hardware to process abnormal workpieces, such as stopping work for re-judgment, sorting by unloading mechanism, flashing warning lights, and statistics of defective workpieces.

[0150] Therefore, the data acquisition cost of the embodiment of the present application is reduced: In many industrial scenarios, especially those industries with extremely high product quality requirements, it may be costly to obtain a large number of defect samples because they require a lot of manual annotation or can only be captured under specific conditions. The metal surface defect detection technology provided by this patent can be effectively trained with only a small number of samples, thereby reducing the cost of data acquisition and annotation.

[0151] The embodiments of the present application can quickly adapt to new product lines: factories may need to frequently adjust or replace production lines to adapt to market changes. The metal surface defect detection technology provided by this patent enables the quality inspection system to quickly adapt to the detection needs of new products.

[0152] The embodiments of the present application are conducive to improving production efficiency: traditional defect detection methods may require a large number of samples to train the model, which consumes a lot of time and resources. The metal surface defect detection technology provided by this patent can establish an effective detection model in a short time, thereby speeding up the production process and improving overall production efficiency.

[0153] See also Fig. 9 The embodiment of the present application further provides a metal surface detection system, which can implement the above metal surface detection method, and the system includes:

[0154] The first module 810 is used to obtain image information of the metal surface to be detected;

[0155] The second module 820 is used to extract local features from the image information to obtain extracted features;

[0156] The third module 830 is used to perform metric learning on the extracted features and the features in the memory bank, and obtain an abnormality score according to the metric learning result; the memory bank is established according to the nominal positive sample;

[0157] The fourth module 840 is used to perform defect segmentation on the metal surface to be inspected according to the abnormality score to obtain the inspection result.

[0158] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0159] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above metal surface detection method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0160] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0161] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0162] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0163] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the metal surface detection method of the embodiment of the present application;

[0164] Input / output interface 903, used to implement information input and output;

[0165] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0166] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0167] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0168] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned metal surface detection method is implemented.

[0169] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0170] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0175] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0181] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A metal surface detection method, characterized in that: The method comprises: Acquire image information of the metal surface to be detected; Extracting local features from the image information to obtain extracted features; Performing metric learning on the extracted features and the features in the memory library, and obtaining an anomaly score according to the metric learning result; the memory library is established according to the nominal positive sample; According to the abnormality score, the defect segmentation is performed on the metal surface to be detected to obtain a detection result.

2. The method according to claim 1, characterized in that The memory bank is determined by the following steps: Acquire image sample information, and perform local feature extraction on the image sample information to obtain extracted sample features; Performing neighborhood feature fusion processing on the extracted sample features, and storing the fused positive sample features to obtain a memory library; If the amount of data in the memory library is greater than a preset data amount threshold, the memory library is reduced by a feature library core set reduction algorithm to update the memory library.

3. The method according to claim 2, characterized in that The step of performing neighborhood feature fusion processing on the extracted sample features and storing the fused positive sample features to obtain a memory library includes: Performing neighborhood expansion on the extracted sample features, and performing dimension matching on the obtained multi-layer neighborhood features to obtain neighborhood matching features; The neighborhood matching features at different levels are pooled, and the features of different scales are concatenated to obtain multi-scale representation features; According to the multi-scale representation features, positive sample features are determined to obtain a memory library.

4. The method according to claim 2, characterized in that: If the amount of data in the memory is greater than a preset data amount threshold, the memory is reduced by a feature library core set reduction algorithm to update the memory, including: arbitrarily selecting a number of feature vector blocks from the memory bank; Calculating the distance between all vectors in the memory bank and each of the feature vector blocks to obtain a first distance set; Determine a first feature vector block according to the first distance set; and put the first feature vector block into a new memory bank; Calculating the distances between all vectors in the memory bank and the first feature vector block to obtain a second distance set; Performing a bit operation on the values ​​in the first distance set and the second distance set, and determining a third distance set according to the bit operation result; Determine a second feature vector block according to the third distance set; and put the second feature vector block into a new memory bank; The second distance set is used as a new first distance set, the second feature vector block is used as a new first feature vector block, and the step of calculating the distance between all vectors in the memory bank and the first feature vector block to obtain the second distance set is returned until the number of loops is greater than or equal to a preset number threshold; the obtained new memory bank is used as an updated memory bank.

5. The method according to claim 1, characterized in that The extracting features are subjected to metric learning with the features in the memory library, and an abnormality score is obtained according to the metric learning result, including: Acquire the extracted features and extracted neighborhood features of the extracted features; Determine the maximum value of the distances between the extracted feature, the extracted neighborhood feature and each feature in the memory library to obtain a maximum distance; Obtaining a weighted feature block in the memory bank that is closest to the extracted feature; The maximum distance is weighted according to the weighted feature block and the extracted feature to obtain an anomaly score.

6. The method according to claim 1, characterized in that The defect segmentation of the metal surface to be detected is performed according to the abnormality score to obtain the detection result, including: Performing interpolation processing on the abnormality scores at different positions of the metal surface to be detected, and overlaying the interpolation result with the original image information to obtain a thermal map; According to the outlier threshold, the thermal map is segmented into defect positions to obtain a detection result.

7. The method according to claim 2, characterized in that: The extracting local features of the image sample information to obtain extracted sample features includes: Inferring the image sample information through a pre-trained network to obtain multi-layer feature encoding; The middle-level feature codes are extracted as the extracted sample features.

8. A metal surface detection system, characterized in that: The system comprises: The first module is used to obtain image information of the metal surface to be detected; The second module is used to extract local features of the image information to obtain extracted features; The third module is used to perform metric learning on the extracted features and the features in the memory bank, and obtain an abnormality score according to the metric learning result; the memory bank is established according to the nominal positive sample; The fourth module is used to perform defect segmentation on the metal surface to be detected according to the abnormality score to obtain a detection result.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images