Surface defect detection method, device and system

Abstract

The present application discloses a surface defect detection method, device, and system, which belong to the field of deep learning technology. In the present application, if the neural network defect segmentation model is used to detect and determine that there are no surface defects of known defect categories in the image to be detected, then the neural network defect feature extraction model is used to extract image features of the image to be detected to obtain features to be compared. If the similarity between the features to be compared and the features represented by normal data is small, it is determined that there are surface defects of unknown defect categories in the image to be detected. In other words, this solution can detect surface defects of unknown defect categories, thereby improving the accuracy of detection.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of deep learning technology, and in particular to a surface defect detection method, device, and system. Background Art

[0002] Surface defects refer to flaws on the appearance of an object. Surface defects are characterized by a wide variety of types, varied shapes, unstable positions, and diverse background textures. Surface defect detection is an important part of quality control in the industrial field, and the higher the accuracy of surface defect detection, the better.

[0003] The deep learning neural network in related technologies can only detect surface defects of known defect categories, and cannot effectively detect new categories of defects encountered in actual inspections, that is, surface defects of unknown defect categories. Summary of the Invention

[0004] The embodiments of the present application provide a surface defect detection method, device, and system that can detect surface defects of unknown defect types and improve detection accuracy. The technical solution is as follows:

[0005] In one aspect, a surface defect detection method is provided, the method comprising:

[0006] Obtain the image to be detected;

[0007] Inputting the image to be inspected into a neural network defect segmentation model and outputting a first inspection result, wherein the neural network defect segmentation model is used to detect surface defects of known defect categories;

[0008] If the first detection result is that the image to be detected does not have the surface defects of the known defect category, inputting the image to be detected into a neural network defect feature extraction model and outputting features to be compared, where the features to be compared are image features of the image to be detected;

[0009] If the similarity between the feature to be compared and the feature representing normal data is less than a similarity threshold, it is determined that a surface defect of an unknown defect category exists in the image to be detected, and the feature representing normal data is generated based on the image feature of an image without surface defects.

[0010] Optionally, after determining that the image to be inspected has a surface defect of an unknown defect category, the method further includes:

[0011] The defect position of the surface defect in the image to be detected is determined based on the similarity between the feature to be compared and the feature represented by the normal data, and the mapping relationship between the image feature matrix and the image pixel matrix.

[0012] Optionally, before inputting the image to be detected into the neural network defect segmentation model and outputting the first detection result, the method further includes:

[0013] Acquire a first data set, the first data set including defect images of known defect categories and corresponding first annotation information, wherein the first annotation information is annotation information indicating the defect categories; the first data set also including images without surface defects and corresponding second annotation information, wherein the second annotation information is annotation information indicating the absence of surface defects;

[0014] The neural network defect segmentation model is trained based on the first data set.

[0015] Optionally, before inputting the image to be detected into a neural network defect feature extraction model and outputting features to be compared, the method further includes:

[0016] Acquire a second data set, the second data set including images of known object categories and corresponding third annotation information, wherein the third annotation information is annotation information indicating the object categories;

[0017] The neural network defect feature extraction model is trained based on the second data set.

[0018] Optionally, before determining that the image to be inspected has a surface defect of an unknown defect category if the similarity between the feature to be compared and the feature represented by the normal data is less than a similarity threshold, the method further includes:

[0019] Acquire at least one first sample image, wherein the at least one first sample image has no surface defects;

[0020] Inputting the at least one first sample image into the neural network defect feature extraction model, and outputting image features of the at least one first sample image;

[0021] The image features of the at least one first sample image are used as the normal data representation features.

[0022] Optionally, before determining that the image to be inspected has a surface defect of an unknown defect category if the similarity between the feature to be compared and the feature represented by the normal data is less than a similarity threshold, the method further includes:

[0023] acquiring a plurality of second sample images, wherein the plurality of second sample images have no surface defects;

[0024] Inputting the plurality of second sample images into the neural network defect feature extraction model, and outputting image features of the plurality of second sample images;

[0025] Clustering the image features of the plurality of second sample images to obtain a plurality of groups of normal data features;

[0026] At least one image feature is selected from each group of normal data features in the multiple groups of normal data features to obtain the normal data representative feature.

[0027] Optionally, after inputting the image to be detected into a neural network defect feature extraction model and outputting features to be compared, the method further includes:

[0028] If the similarity between the feature to be compared and the feature representing the normal data is greater than or equal to the similarity threshold, it is determined that the image to be inspected has no surface defects.

[0029] In another aspect, a surface defect detection device is provided, comprising:

[0030] A first acquisition module is used to acquire an image to be detected;

[0031] a detection module, configured to input the image to be detected into a neural network defect segmentation model and output a first detection result, wherein the neural network defect segmentation model is used to detect surface defects of known defect categories;

[0032] a first processing module, configured to input the image to be detected into a neural network defect feature extraction model and output a feature to be compared if the first detection result indicates that the image to be detected does not have surface defects of the known defect category, wherein the feature to be compared is an image feature of the image to be detected;

[0033] The first determination module is configured to determine that a surface defect of an unknown defect category exists in the image to be detected if the similarity between the feature to be compared and a feature representing normal data is less than a similarity threshold, wherein the feature representing normal data is generated based on image features of an image without surface defects.

[0034] Optionally, the device further comprises:

[0035] The second determination module is used to determine the defect position of the surface defect existing in the image to be detected based on the similarity between the feature to be compared and the feature represented by the normal data, and the mapping relationship between the image feature matrix and the image pixel matrix.

[0036] Optionally, the device further comprises:

[0037] a second acquisition module, configured to acquire a first data set, wherein the first data set includes defect images of known defect categories and corresponding first annotation information, wherein the first annotation information is annotation information indicating the defect categories; and further includes images without surface defects and corresponding second annotation information, wherein the second annotation information is defect information indicating the absence of surface defects.

[0038] The first training module is used to train the neural network defect segmentation model based on the first data set.

[0039] Optionally, the device further comprises:

[0040] a third acquisition module, configured to acquire a second data set, wherein the second data set includes images of known object categories and corresponding third annotation information, wherein the third annotation information is annotation information indicating the object categories;

[0041] The second training module is used to train the neural network defect feature extraction model based on the second data set.

[0042] Optionally, the device further comprises:

[0043] A fourth acquisition module is configured to acquire at least one first sample image, where the at least one first sample image has no surface defects;

[0044] The second processing module is further configured to input the at least one first sample image into the neural network defect feature extraction model and output image features of the at least one first sample image;

[0045] The third determining module is configured to use the image features of the at least one first sample image as the normal data representation features.

[0046] Optionally, the device further comprises:

[0047] A fifth acquisition module is configured to acquire a plurality of second sample images, wherein the plurality of second sample images have no surface defects;

[0048] a third processing module, configured to input the plurality of second sample images into the neural network defect feature extraction model, and output image features of the plurality of second sample images;

[0049] a clustering module, configured to cluster the image features of the plurality of second sample images to obtain a plurality of groups of normal data features;

[0050] The fourth determination module is configured to select at least one image feature from each group of normal data features in the multiple groups of normal data features to obtain the normal data representative feature.

[0051] Optionally, the device further comprises:

[0052] The fifth determining module is configured to determine that the image to be inspected has no surface defects if the similarity between the feature to be compared and the feature representing the normal data is greater than or equal to the similarity threshold.

[0053] In another aspect, a surface defect detection system is provided, the surface defect detection system comprising a camera and at least one processor;

[0054] The camera is used to capture at least one surface of the object to be detected as an image to be detected;

[0055] The at least one processor is used to obtain the image to be detected and implement the steps of the above-mentioned surface defect detection method.

[0056] Optionally, the surface defect detection system further comprises a conveying device, wherein the conveying device is used to transport the object to be detected;

[0057] The camera is used to photograph the object to be detected during the process of the conveying device transporting the object to be detected.

[0058] On the other hand, a computer device is provided, which includes a processor, a communication interface, a memory and a communication bus. The processor, the communication interface and the memory communicate with each other through the communication bus. The memory is used to store computer programs, and the processor is used to execute the programs stored in the memory to implement the steps of the above-mentioned surface defect detection method.

[0059] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the surface defect detection method described above are implemented.

[0060] On the other hand, a computer program product comprising instructions is provided, which, when executed on a computer, causes the computer to perform the steps of the surface defect detection method described above.

[0061] The technical solutions provided in the embodiments of the present application can at least bring the following beneficial effects:

[0062] In an embodiment of the present application, if the neural network defect segmentation model is used to detect that there are no surface defects of a known defect category in the image to be detected, then the image features of the image to be detected are extracted by the neural network defect feature extraction model to obtain the features to be compared. If the similarity between the features to be compared and the features represented by the normal data is small, it is determined that there are surface defects of an unknown defect category in the image to be detected. That is, this solution can detect surface defects of an unknown defect category, thereby improving the accuracy of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] Figure 1 This is a system architecture diagram of a surface defect detection system provided by an embodiment of the present application;

[0065] Figure 2 This is a flow chart of a surface defect detection method provided by an embodiment of the present application;

[0066] Figure 3 is a flow chart of another surface defect detection method provided in an embodiment of the present application;

[0067] Figure 4 This is a schematic diagram of a method for training a detection model provided in an embodiment of the present application;

[0068] Figure 5 This is a schematic diagram of a method for modeling normal data features provided in an embodiment of the present application;

[0069] Figure 6 This is a schematic structural diagram of a surface defect detection device provided in an embodiment of the present application;

[0070] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0071] Figure 8 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0072] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0073] First, the system architecture involved in the surface defect detection method provided in the embodiment of the present application is introduced.

[0074] Figure 1 This is a system architecture diagram of a surface defect detection system provided in an embodiment of the present application. The surface defect detection system is used to implement the surface defect detection method provided in an embodiment of the present application. Figure 1 , the system includes a camera 101 and at least one processor 102.

[0075] The at least one processor 102 is a processor in a computer device. The camera 101 and the at least one processor 102 in the computer device are connected wirelessly or wired for communication. The camera 101 is used to capture an image of the object to be inspected and send it to the at least one processor 102 as the device to be inspected. The computer device, as the device to be inspected, performs surface defect inspection via the at least one processor 102.

[0076] In some embodiments, the at least one processor 102 is a processor in the camera 101. The camera 101 serves as a detection device, captures an image of the object to be detected as the image to be detected, and performs surface defect detection.

[0077] Optionally, the system further includes a conveyor device for transporting the object to be inspected. In one scenario, while the conveyor device is transporting the object to be inspected, the camera 101 photographs the object to be inspected, and the inspection equipment detects surface defects. The conveyor device may be a conveyor belt, a transport vehicle, or a conveyor machine, and the object to be inspected may be factory-produced metal parts, glassware, paper, or other products. In this way, while the object to be inspected is being transported, objects with surface defects can be detected, thereby identifying defective products and ensuring product quality.

[0078] Next, the surface defect detection method provided in the embodiment of the present application is explained in detail.

[0079] Figure 2 This is a flow chart of a surface defect detection method provided in an embodiment of the present application, which is introduced by taking the method applied to a surface defect detection device (hereinafter referred to as the detection device) as an example. Figure 2 , the method includes the following steps.

[0080] Step 201: Acquire an image to be detected.

[0081] In the embodiment of the present application, the inspection device obtains an image to be inspected, which refers to an image of an object to be inspected for surface defects, such as an image of an industrial tool, paper, or other object.

[0082] Exemplarily, the detection device stores the image to be detected, or the detection device receives the image to be detected sent by other devices. For example, an industrial camera sends the image of the object it captures to the detection device, and the detection device uses the received image as the image to be detected.

[0083] Step 202: Input the image to be detected into a neural network defect segmentation model and output a first detection result. The neural network defect segmentation model is used to detect surface defects of known defect categories.

[0084] In an embodiment of the present application, a neural network defect segmentation model is deployed in the inspection device. The neural network defect segmentation model is a deep learning model used to detect surface defects of known defect categories. The inspection device inputs the image to be inspected into the neural network defect segmentation model and outputs a first inspection result.

[0085] For example, assuming that the image to be detected has a surface defect of a known defect category, the first detection result is that the image to be detected has a surface defect of the known defect category. assuming that the image to be detected does not have a surface defect of the known defect category, the first detection result is that the image to be detected does not have a surface defect of the known defect category.

[0086] Step 203: If the first detection result is that the image to be detected does not have surface defects of known defect categories, the image to be detected is input into the neural network defect feature extraction model, and the features to be compared are output, where the features to be compared are image features of the image to be detected.

[0087] In an embodiment of the present application, the inspection device also deploys a neural network defect feature extraction model, a deep learning model used to extract image features. If the first inspection result indicates that the image to be inspected does not contain surface defects of a known defect category, the inspection device inputs the image to be inspected into the neural network defect feature extraction model and outputs features to be compared, which are image features of the image to be inspected.

[0088] Step 204: If the similarity between the feature to be compared and the feature representing the normal data is less than the similarity threshold, it is determined that the image to be detected has a surface defect of an unknown defect category, and the feature representing the normal data is generated based on the image feature of the image without surface defects.

[0089] In an embodiment of the present application, the inspection device stores normal data representation features, which are generated based on image features of an image without surface defects. For example, the normal data representation features include image features obtained by processing the image without surface defects using a neural network defect feature extraction model. After the inspection device extracts the features to be compared using the neural network defect feature extraction model, it calculates the similarity between the features to be compared and the features representing the normal data. If the similarity between the features to be compared and the features representing the normal data is less than a similarity threshold, the inspection device determines that a surface defect of an unknown defect category exists in the image to be inspected.

[0090] Among them, there are many methods for calculating the similarity between the features to be compared and the features represented by normal data, such as the Euclidean distance similarity calculation method between features, the Mahalanobis distance similarity calculation method between features, the cosine distance similarity calculation method between features, etc., which are not limited in the embodiments of the present application.

[0091] In an embodiment of the present application, the normal data representation feature includes one or more image features of an image without surface defects. The detection device calculates the similarity between the feature to be compared and each image feature included in the normal data representation feature, obtaining one or more similarities. If at least one of the one or more similarities is less than a first similarity threshold, the detection device determines that a surface defect of an unknown defect category exists in the image to be detected. Alternatively, if the average of the one or more similarities is less than a second similarity threshold, the detection device determines that a surface defect of an unknown defect category exists in the image to be detected. The first similarity threshold and the second similarity threshold may be the same or different.

[0092] In the embodiment of the present application, if the similarity between the feature to be compared and the feature represented by the normal data is greater than or equal to the similarity threshold, the detection device determines that the image to be detected has no surface defects.

[0093] Exemplarily, the inspection device calculates the similarity between the feature to be compared and each image feature included in the normal data representation feature to obtain one or more similarities. If the one or more similarities are all greater than or equal to a first similarity threshold, the inspection device determines that the image to be inspected has no surface defects. Alternatively, if the average of the one or more similarities is greater than or equal to a second similarity threshold, the inspection device determines that the image to be inspected has no surface defects.

[0094] Optionally, in an embodiment of the present application, if there is a surface defect of an unknown defect category in the image to be detected, the detection device can also detect the position of the surface defect of the unknown defect category in the image to be detected, that is, detect and determine the defect position of the surface defect in the image to be detected.

[0095] In an embodiment of the present application, the inspection device determines the location of a defect in the image to be inspected by comparing the features to be compared with the features represented by normal data. For example, the inspection device determines the location of a surface defect in the image to be inspected based on the similarity between the features to be compared and the features represented by normal data, as well as the mapping relationship between the image feature matrix and the image pixel matrix.

[0096] In one implementation, there is a certain mapping relationship between the image feature matrix and the image pixel matrix. The image pixel matrix refers to a matrix composed of pixel values ​​of the image. The image feature matrix is ​​a matrix obtained by downsampling the image pixel matrix through the neural network defect feature extraction model, that is, the image features extracted by the neural network defect feature extraction model are represented in matrix form as an image feature matrix.

[0097] For example, it is assumed that the neural network defect feature extraction model in the embodiment of the present application is constructed based on a convolutional neural network, the extracted image features include features of C channels, the image pixel matrix is ​​a 100*100 matrix, and the image pixel matrix is ​​four times downsampled by the neural network defect feature extraction model to obtain an image feature matrix with a feature array of 25*25 C-dimensional features. The feature array is a third-order tensor, which can be understood as each position of the 25*25 image feature matrix is ​​a C-dimensional vector, and each position corresponds to a 4*4 area in the image pixel matrix. For example, the first position in the 25*25 image feature matrix corresponds to the first 4*4 area of ​​the 100*100 image pixel matrix. Then, the similarity between the C-dimensional vector of each position in the 25*25 feature matrix to be compared and the C-dimensional vector of the corresponding position in the 25*25 normal data feature matrix determines whether there is a surface defect in the corresponding 4*4 area in the image pixel matrix.

[0098] The detection device calculates the similarity between the elements at the same position in the feature matrix to be compared and the normal data feature matrix. If the similarity between the elements at the same position is less than the similarity threshold, it means that the detection device has detected the defect feature position. The defect feature position refers to the position in the feature matrix to be compared where the feature similarity is less than the similarity threshold. Then, based on the mapping relationship between the image feature matrix and the image pixel matrix, the detection device determines the image pixel position corresponding to the defect feature position from the image pixel matrix corresponding to the image to be compared, as the defect position of the surface defect detected in the image to be detected. Among them, the feature matrix to be compared refers to the image feature matrix corresponding to the image to be detected, and the normal data feature matrix refers to the image feature matrix corresponding to the characteristics represented by the normal data.

[0099] For example, assuming that the size of the image pixel matrix corresponding to the image to be detected is 100*100, and the feature matrix to be compared and the feature matrix of the normal data are both third-order tensors with 25*25 C-dimensional features, the detection device calculates the similarity between the C-dimensional vector at the first position in the feature matrix to be compared and the C-dimensional vector at the first position in the normal data feature matrix. If the similarity is less than the similarity threshold, the detection device determines that the image position of the first 4*4 area in the image pixel matrix corresponding to the image to be detected is the defect position based on the mapping relationship between the image feature matrix and the image pixel matrix.

[0100] Figure 3 This is a flow chart of another surface defect detection method provided by an embodiment of the present application. Figure 3 Taking the image to be inspected as a real-time image captured by an industrial camera as an example, the inspection equipment has two functions: known defect detection and unknown defect detection. The inspection equipment deploys trained models, including a neural network defect segmentation model and a neural network defect feature extraction model. The neural network defect segmentation model is used to implement known defect detection, that is, to detect surface defects of known defect categories. The inspection equipment also stores normal data representation features. The neural network defect feature extraction model and normal data representation features are combined to implement unknown defect detection, that is, to detect surface defects of unknown defect categories.

[0101] The known defect detection process involves the inspection equipment feeding images captured in real time by an industrial camera into a neural network defect segmentation model. If a surface defect is detected, the inspection equipment outputs a known defect detection result, specifically the defect category of the surface defect present in the image being inspected. If no surface defect is detected, the inspection proceeds to unknown defect detection.

[0102] The unknown defect detection process involves the following: the detection device inputs the image captured in real time by the industrial camera into a neural network defect feature extraction model, outputting real-time image features, that is, outputting the image features of the image to be detected. The detection device then compares the real-time image features with the features representing normal data for feature similarity and outputs the unknown defect detection results. For example, if the similarity between the real-time image features and the features representing normal data is less than a similarity threshold, the unknown defect detection result is that the image contains a surface defect of an unknown defect category, and the defect location is marked on the image. If the similarity between the real-time image features and the features representing normal data is greater than or equal to the similarity threshold, the unknown defect detection result is that the image has no surface defects, that is, the image is a normal image without surface defects, and the object corresponding to the image is a normal object.

[0103] From the above, it can be seen that in the embodiment of the present application, the detection equipment can preliminarily detect surface defects of known defect categories through the neural network defect segmentation model, and further detect whether there are surface defects of unknown defect categories in the image to be detected through the neural network defect feature extraction model and normal data representation features. That is, this solution can effectively detect surface defects of unknown defect categories, thereby improving the accuracy of detection.

[0104] The above describes the process by which a detection device detects surface defects based on a detection model and features representing normal data. The detection model includes a neural network defect segmentation model and a neural network defect feature extraction model. It should be noted that in the embodiments of this application, the neural network defect segmentation model deployed in the detection device is a trained deep learning model. The following describes an implementation method for training a neural network defect segmentation model.

[0105] In an embodiment of the present application, a detection device acquires a first data set and trains a neural network defect segmentation model based on the first data set. The first data set includes defect images of known defect categories and corresponding first annotation information, which indicates the defect category. Defect images are images with surface defects. The first data set also includes images without surface defects and corresponding second annotation information, which indicates the absence of surface defects.

[0106] That is, a part of defective images with known defect categories is obtained, and another part of normal images without surface defects is obtained. Normal images refer to images without surface defects. The known defect categories in the defective images are marked to obtain the first marking information corresponding to the defective images. The first marking information is the corresponding defect category, and the normal images are marked as having no surface defects.

[0107] For example, assuming that the image size in the embodiment of the present application is 100*100 pixels, the annotation information corresponding to the defect image and the normal image is also represented by a 100*100 matrix. For example, a 100*100 all-zero matrix indicates the absence of surface defects, that is, the annotation information corresponding to the normal image is a 100*100 all-zero matrix, and a 100*100 non-all-zero matrix indicates the presence of surface defects, that is, the annotation information corresponding to the defect image is a 100*100 non-all-zero matrix. That is, in the embodiment of the present application, an all-zero matrix with the same size as the image is used as the annotation information for the normal image, and a non-all-zero matrix with the same size as the image is used as the annotation information for the defect image.

[0108] In the non-zero matrix, different values ​​represent different defect categories. For example, a value of 1 represents a first-category known defect, a value of 2 represents a second-category known defect, and a value of 3 represents a third-category known defect. Furthermore, the locations of the non-zero elements in the non-zero matrix correspond to the local locations in the image where surface defects exist. For example, if the first row of elements in the non-zero matrix is ​​non-zero, it indicates that a surface defect exists at the location of the first row of pixels in the image. If the values ​​of the first three elements in the first row of the non-zero matrix are 1, it indicates that the first three pixels in the first row of pixels in the image contain a first-category known defect.

[0109] Based on the above example, optionally, the first detection result is represented by a matrix of the same size as the image, and the first detection result can be referred to as a result matrix. After the detection device inputs the image to be detected into the neural network defect feature extraction model, it outputs a result matrix of the same size as the image. The detection device can determine whether there are surface defects of known defect categories in the image to be detected, as well as the defect locations, based on the result matrix. For example, if the result matrix is ​​a non-zero matrix, the detection device determines that there are surface defects of known defect categories in the image to be detected, and determines the defect category and defect location of the surface defects in the image to be detected based on the positions and values ​​of the non-zero elements in the result matrix.

[0110] In an embodiment of the present application, the neural network defect feature extraction model deployed in the detection device is also a trained deep learning model. Next, an implementation method of training the neural network defect feature extraction model is introduced.

[0111] In an embodiment of the present application, the detection device acquires a second data set and trains a neural network defect feature extraction model based on the second data set. The second data set includes images of known object categories and corresponding third annotation information, where the third annotation information is annotation information indicating the object category.

[0112] Exemplarily, the detection device obtains a large number of public data sets with object category annotations as the second data set, and trains a neural network defect feature extraction model based on the second data set including a large amount of data. The richer the data included in the second data set, the better the effect of the trained neural network defect feature extraction model in extracting image features.

[0113] One implementation method for training a neural network defect feature extraction model obtained by the detection device is as follows: First, the detection device constructs a neural network classification model based on a convolutional neural network. The constructed neural network classification model includes a convolutional layer, a pooling layer, a fully connected layer, a softmax layer, etc. The detection device trains the neural network classification model based on a second data set, and the output of the trained neural network classification model is the object category. Since the neural network classification model calculates and extracts image features layer by layer during the forward reasoning process and finally outputs the object category through the fully connected layer and the softmax layer, in order to obtain a neural network defect feature extraction model capable of outputting image features, the detection device removes the last fully connected layer and softmax layer of the trained neural network classification model, thereby obtaining a neural network defect feature extraction model capable of outputting image features.

[0114] It should be noted that the embodiments of the present application do not limit the deep learning technology, deep learning framework, etc. used to construct the neural network defect feature extraction model.

[0115] Figure 4 Schematic diagram of a method for training a detection model provided in an embodiment of the present application, wherein the detection model includes a neural network segmentation model and a neural network defect feature extraction model. Figure 4 ,The training process of the detection model includes collecting ,training data, building and training the neural network.

[0116] The training data collection process includes: the industrial camera sends the captured image data to a computer device, which acts as a detection device. The detection device labels the image data of defective objects with defect labels. In other words, it labels defective images with known defect categories and obtains labeling information corresponding to the defective images. The labeling information is the corresponding defect category. The detection device also labels the image data of normal objects as having no surface defects. In other words, it labels normal images without surface defects and obtains labeling information corresponding to the normal images. The labeling information is no surface defects.

[0117] The process of building and training a neural network is as follows: A segmentation model (an initialized neural network defect segmentation model) is constructed using the inspection equipment, and the segmentation model is trained based on training data collected by industrial cameras to obtain a trained neural network defect segmentation model. A classification model (an initialized neural network classification model) is constructed using the inspection equipment, and the classification model is trained based on a large number of publicly labeled datasets to obtain a trained neural network defect feature extraction model.

[0118] Next, two implementation methods of determining normal data representation characteristics by the detection device are introduced.

[0119] The first implementation method: the detection device obtains at least one first sample image, and the at least one first sample image has no surface defects. The detection device inputs the at least one first sample image into the neural network defect feature extraction model, outputs the image features of the at least one first sample image, and the detection device uses the image features of the at least one first sample image as normal data representation features.

[0120] That is, the detection equipment obtains image data of some normal objects, extracts image features of these image data through the neural network defect feature extraction model, and uses the extracted image features as normal data representation features.

[0121] For example, as can be seen from the foregoing, if the first dataset includes normal images without surface defects, the testing device can obtain at least one normal image from the first dataset as at least one first sample image. For example, the testing device can randomly select a certain proportion or a certain number of images from the normal images included in the first dataset as the at least one first sample image, or the testing device can use all normal images included in the first dataset as first sample images.

[0122] That is, the detection device first selects a normal image whose image features need to be extracted from the first data set, and then extracts image features from the selected normal image as normal data representation features.

[0123] Optionally, the detection device inputs all normal images included in the first data set into a neural network defect feature extraction model and outputs image features of all normal images included in the first data set. The detection device then randomly selects a certain proportion or a certain number of image features from the image features of all normal images included in the first data set as features representing normal data, or the detection device uses the image features of all normal images included in the first data set as features representing normal data.

[0124] That is, the detection device first extracts image features from all normal images included in the first data set, and then randomly selects a certain proportion or all of the extracted image features as normal data representation features.

[0125] In a second implementation, the testing device acquires multiple second sample images, each of which is free of surface defects. The testing device inputs these multiple second sample images into a neural network defect feature extraction model, outputting image features of these multiple second sample images. The testing device clusters the image features of these multiple second sample images to obtain multiple groups of normal data features. The testing device then selects at least one image feature from each of these multiple groups of normal data features to obtain a normal data representation feature.

[0126] Specifically, the inspection equipment acquires image data of normal objects, extracts features from these images using a neural network defect feature extraction model, clusters these features to generate multiple sets of normal data features, and then selects a certain proportion or number of features from each set as the normal data representation features. This allows the inspection equipment to obtain a richer and more representative set of normal data representation features.

[0127] For example, as mentioned above, if the first dataset includes normal images without surface defects, the inspection device can use all the normal images in the first dataset as multiple second sample images, extract image features from these multiple second sample images, and cluster all the extracted image features to obtain multiple groups of normal data representation features. The inspection device then selects a certain number or a certain proportion of image features from each group of normal data features as normal data representation features.

[0128] Of course, the detection device may also first select a certain proportion or a certain number of normal images from the first data set as multiple second sample images, and then obtain normal data representation features through feature extraction, clustering and screening.

[0129] As can be seen from the aforementioned description of the first and second implementations of determining features representative of normal data, assuming that the detection device determines features representative of normal data based on normal images included in the first data set, the detection device can first extract image features from all normal images included in the first data set, and then, based on a screening mechanism, screen all extracted image features to determine features representative of normal data. The screening mechanism can include randomly selecting a certain proportion or number of image features, selecting all image features, or clustering all image features and selecting a certain proportion or number of image features from each category of image features.

[0130] It should be noted that there are many clustering algorithms used by the detection device to cluster image features, such as K-means clustering algorithm, mean shift clustering algorithm, density-based clustering algorithm, hierarchical-based clustering algorithm, etc., and the embodiments of this application do not limit this.

[0131] In the embodiment of the present application, the process of the detection device determining the normal data representation characteristics can be understood as the process of normal data feature modeling. Figure 5 This is a schematic diagram of a method for modeling normal data features provided by an embodiment of the present application. Figure 5 This process is explained again.

[0132] See also Figure 5 The inspection device inputs image data of normal objects into a neural network defect feature extraction model, which outputs normal data features. For example, all normal images included in the first data set are input into the neural network defect feature extraction model, which outputs image features of all normal images. The inspection device then selects a screening mechanism to filter the normal data features to obtain normal data representation features.

[0133] To sum up, in an embodiment of the present application, if the neural network defect segmentation model is used to detect that there are no surface defects of a known defect category in the image to be detected, then the neural network defect feature extraction model is used to extract the image features of the image to be detected to obtain the features to be compared. If the similarity between the features to be compared and the features represented by the normal data is small, it is determined that there are surface defects of an unknown defect category in the image to be detected. That is, this scheme can detect surface defects of an unknown defect category, thereby improving the accuracy of detection.

[0134] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and the embodiments of the present application will not be described in detail one by one.

[0135] Figure 6This is a schematic diagram of the structure of a surface defect detection device 600 provided in an embodiment of the present application. The surface defect detection device 600 can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be the detection device in the above embodiment. Please refer to Figure 6 The device 600 includes: a first acquisition module 601, a detection module 602, a first processing module 603 and a first determination module 604.

[0136] A first acquisition module 601 is used to acquire an image to be detected;

[0137] A detection module 602 is configured to input an image to be detected into a neural network defect segmentation model and output a first detection result, wherein the neural network defect segmentation model is configured to detect surface defects of a known defect category;

[0138] A first processing module 603 is configured to input the image to be inspected into a neural network defect feature extraction model if the first inspection result indicates that the image to be inspected does not have surface defects of a known defect category, and output features to be compared, where the features to be compared are image features of the image to be inspected;

[0139] The first determination module 604 is configured to determine that a surface defect of an unknown defect category exists in the image to be detected if the similarity between the feature to be compared and the feature representing normal data is less than a similarity threshold, wherein the feature representing normal data is generated based on image features of an image without surface defects.

[0140] Optionally, the apparatus 600 further includes:

[0141] The second determination module is used to determine the defect position of the surface defect in the image to be detected based on the similarity between the feature to be compared and the feature represented by the normal data, and the mapping relationship between the image feature matrix and the image pixel matrix.

[0142] Optionally, the apparatus 600 further includes:

[0143] a second acquisition module, configured to acquire a first data set, the first data set including defect images of known defect categories and corresponding first annotation information, the first annotation information being annotation information indicating the defect categories; and the first data set also including images without surface defects and corresponding second annotation information, the second annotation information being annotation information indicating the absence of surface defects.

[0144] The first training module is used to train a neural network defect segmentation model based on the first data set.

[0145] Optionally, the device further comprises:

[0146] A third acquisition module is used to acquire a second data set, where the second data set includes images of known object categories and corresponding third annotation information, where the third annotation information is annotation information indicating the object category;

[0147] The second training module is used to train a neural network defect feature extraction model based on the second data set.

[0148] Optionally, the apparatus 600 further includes:

[0149] A fourth acquisition module is used to acquire at least one first sample image, where the at least one first sample image has no surface defects;

[0150] The second processing module is further configured to input the at least one first sample image into the neural network defect feature extraction model and output an image feature of the at least one first sample image;

[0151] The third determining module is configured to use an image feature of at least one first sample image as a normal data representation feature.

[0152] Optionally, the apparatus 600 further includes:

[0153] A fifth acquisition module is used to acquire a plurality of second sample images, wherein the plurality of second sample images have no surface defects;

[0154] a third processing module, configured to input the plurality of second sample images into the neural network defect feature extraction model and output image features of the plurality of second sample images;

[0155] A clustering module, configured to cluster the image features of the plurality of second sample images to obtain a plurality of groups of normal data features;

[0156] The fourth determination module is configured to select at least one image feature from each group of normal data features in the plurality of groups of normal data features to obtain a normal data representation feature.

[0157] Optionally, the apparatus 600 further includes:

[0158] The fifth determination module is configured to determine that the image to be inspected has no surface defects if the similarity between the feature to be compared and the feature representing the normal data is greater than or equal to a similarity threshold.

[0159] In an embodiment of the present application, if the neural network defect segmentation model is used to detect and determine that there are no surface defects of known defect categories in the image to be detected, then the neural network defect feature extraction model is used to extract the image features of the image to be detected to obtain the features to be compared. If the similarity between the features to be compared and the features represented by normal data is small, it is determined that there are surface defects of unknown defect categories in the image to be detected. That is, this scheme can detect surface defects of unknown defect categories, thereby improving the accuracy of detection.

[0160] It should be noted that the surface defect detection device provided in the above embodiment is only illustrated by the division of the above functional modules when detecting surface defects. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the surface defect detection device provided in the above embodiment and the surface defect detection method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0161] Figure 7 The following is a block diagram of a terminal 700 according to an exemplary embodiment of the present application. Terminal 700 may be a smartphone, tablet computer, laptop computer, or desktop computer. Terminal 700 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names. Optionally, terminal 700 is the detection device described in the above embodiments.

[0162] Typically, the terminal 700 includes a processor 701 and a memory 702 .

[0163] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0164] Memory 702 may include one or more computer-readable storage media, which may be non-transitory. Memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more magnetic disk storage devices or flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 702 is used to store at least one instruction, which is executed by processor 701 to implement the surface defect detection method provided in the method embodiment of the present application.

[0165] In some embodiments, terminal 700 may optionally include a peripheral device interface 703 and at least one peripheral device. Processor 701, memory 702, and peripheral device interface 703 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 703 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708, and a power supply 709.

[0166] The peripheral device interface 703 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0167] The RF circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 704 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 704 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 704 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 704 may also include circuits related to Near Field Communication (NFC), which is not limited in this application.

[0168] Display screen 705 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 705 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 705. These touch signals can be input as control signals to processor 701 for processing. Display screen 705 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 705, located on the front panel of terminal 700. In other embodiments, there can be at least two display screens 705, located on different surfaces of terminal 700 or in a foldable design. In other embodiments, display screen 705 can be a flexible display, located on a curved or foldable surface of terminal 700. Display screen 705 can also be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 705 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0169] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0170] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 701 for processing, or input into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the terminal 700. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0171] Positioning component 708 is used to locate the current geographic location of terminal 700 to implement navigation or LBS (Location Based Service). Positioning component 708 can be based on the US GPS (Global Positioning System), China's Beidou system, Russia's Greninja system, or the European Union's Galileo system.

[0172] Power supply 709 is used to power various components in terminal 700. Power supply 709 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 709 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0173] In some embodiments, the terminal 700 further includes one or more sensors 710 , including but not limited to: an acceleration sensor 711 , a gyroscope sensor 712 , a pressure sensor 713 , a fingerprint sensor 714 , an optical sensor 715 , and a proximity sensor 716 .

[0174] The accelerometer 711 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 700. For example, the accelerometer 711 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 701 can control the display screen 705 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 711. The accelerometer 711 can also be used to collect game or user motion data.

[0175] The gyroscope sensor 712 can detect the orientation and rotation angle of the terminal 700. It can work with the accelerometer 711 to collect the user's 3D movements on the terminal 700. Based on the data collected by the gyroscope sensor 712, the processor 701 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0176] The pressure sensor 713 can be set on the side frame of the terminal 700 and / or the lower layer of the display screen 705. When the pressure sensor 713 is set on the side frame of the terminal 700, it can detect the user's grip signal of the terminal 700, and the processor 701 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 713. When the pressure sensor 713 is set on the lower layer of the display screen 705, the processor 701 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 705. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0177] The fingerprint sensor 714 is used to collect the user's fingerprint. The processor 701 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 714, or the fingerprint sensor 714 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 701 authorizes the user to perform relevant sensitive operations, such as unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 714 can be set on the front, back, or side of the terminal 700. When a physical button or manufacturer logo is set on the terminal 700, the fingerprint sensor 714 can be integrated with the physical button or manufacturer logo.

[0178] The optical sensor 715 is used to detect ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the display screen 705 based on the ambient light intensity detected by the optical sensor 715. Specifically, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is decreased. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 based on the ambient light intensity detected by the optical sensor 715.

[0179] Proximity sensor 716, also known as a distance sensor, is typically located on the front panel of terminal 700. Proximity sensor 716 is used to detect the distance between the user and the front of terminal 700. In one embodiment, when proximity sensor 716 detects that the distance between the user and the front of terminal 700 is gradually decreasing, processor 701 controls display screen 705 to switch from the screen-on state to the screen-off state. When proximity sensor 716 detects that the distance between the user and the front of terminal 700 is gradually increasing, processor 701 controls display screen 705 to switch from the screen-off state to the screen-on state.

[0180] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the terminal 700, and the terminal 700 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0181] Figure 8 8 is a schematic diagram of a server structure of a surface defect detection device according to an exemplary embodiment. The server 800 may be a server in a backend server cluster, and the server 800 may be the detection device in the above embodiment. Specifically:

[0182] The server 800 includes a central processing unit (CPU) 801, a system memory 804 including a random access memory (RAM) 802 and a read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the central processing unit 801. The server 800 also includes a basic input / output system (I / O system) 806 that facilitates information transfer between various components within the computer, and a mass storage device 807 for storing an operating system 813, application programs 814, and other program modules 815.

[0183] The basic input / output system 806 includes a display 808 for displaying information and an input device 809, such as a mouse and keyboard, for user input. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include an input / output controller 810 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0184] The mass storage device 807 is connected to the central processing unit 801 through a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable media provide non-volatile storage for the server 800. That is, the mass storage device 807 may include computer-readable media (not shown) such as a hard disk or CD-ROM drive.

[0185] Without loss of generality, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that computer storage media is not limited to the aforementioned types. The above-mentioned system memory 804 and mass storage device 807 may be collectively referred to as memory.

[0186] According to various embodiments of the present application, the server 800 may also be connected to a remote computer on a network such as the Internet for operation. That is, the server 800 may be connected to the network 812 via the network interface unit 811 connected to the system bus 805, or the network interface unit 811 may be used to connect to other types of networks or remote computer systems (not shown).

[0187] The memory further includes one or more programs, which are stored in the memory and configured to be executed by the CPU. The one or more programs include instructions for performing the surface defect detection method provided in the embodiment of the present application.

[0188] In some embodiments, a computer-readable storage medium is further provided, the storage medium storing a computer program that, when executed by a processor, implements the steps of the surface defect detection method described in the above embodiments. For example, the computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0189] It is worth noting that the computer-readable storage medium mentioned in the embodiments of the present application may be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0190] It should be understood that all or part of the steps for implementing the above embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the steps may be implemented in the form of a computer program product. The computer program product may include one or more computer instructions. The computer instructions may be stored in the computer-readable storage medium.

[0191] That is, in some embodiments, a computer program product comprising instructions is further provided, which, when executed on a computer, enables the computer to execute the steps of the surface defect detection method described above.

[0192] It should be understood that the "at least one" mentioned herein refers to one or more, and "a plurality of" refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0193] The above description is an embodiment provided for this application and is not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A surface defect detection method, characterized in that: The method comprises: acquiring a plurality of second sample images, wherein the plurality of second sample images have no surface defects; Inputting the plurality of second sample images into a neural network defect feature extraction model, and outputting image features of the plurality of second sample images; Clustering the image features of the plurality of second sample images to obtain a plurality of groups of normal data features; selecting at least one image feature from each group of normal data features in the plurality of groups of normal data features to obtain a normal data representation feature, wherein the normal data representation feature is represented by a normal data feature matrix; Obtain the image to be detected; Inputting the image to be inspected into a neural network defect segmentation model and outputting a first detection result, wherein the neural network defect segmentation model is used to detect surface defects of known defect categories, and the first detection result is represented by a matrix of the same size as the image to be inspected; If the first detection result is a non-all-zero matrix, determining that a surface defect of a known defect category exists in the image to be detected, and determining the defect category and defect location of the surface defect existing in the image to be detected based on the position and value of the non-zero elements in the first detection result; If the first detection result is an all-zero matrix, it is determined that the image to be detected does not have the surface defects of the known defect category, the image to be detected is input into the neural network defect feature extraction model, and features to be compared are output, where the features to be compared are image features of the image to be detected and are represented by a feature matrix to be compared; Determine the similarity between the feature to be compared and one or more image features included in the normal data representation feature to obtain one or more similarities; If at least one of the one or more similarities is less than a similarity threshold, or if an average of the one or more similarities is less than a similarity threshold, determining that a surface defect of an unknown defect category exists in the image to be inspected; Determining a defect feature position, where the defect feature position refers to an element position in the feature matrix to be compared whose similarity with an element at the same position in the normal data feature matrix is ​​less than a similarity threshold; According to the mapping relationship between the image feature matrix and the image pixel matrix, the image pixel position corresponding to the defect feature position is determined from the image pixel matrix corresponding to the image to be detected as the defect position of the surface defect existing in the image to be detected.

2. The method according to claim 1, characterized in that Before inputting the image to be detected into the neural network defect segmentation model and outputting the first detection result, the method further includes: Acquire a first data set, the first data set including defect images of known defect categories and corresponding first annotation information, wherein the first annotation information is annotation information indicating the defect categories; the first data set also including images without surface defects and corresponding second annotation information, wherein the second annotation information is annotation information indicating the absence of surface defects; The neural network defect segmentation model is trained based on the first data set.

3. The method according to claim 1, characterized in that Before inputting the image to be detected into the neural network defect feature extraction model and outputting the features to be compared, the method further includes: Acquire a second data set, the second data set including images of known object categories and corresponding third annotation information, wherein the third annotation information is annotation information indicating the object categories; The neural network defect feature extraction model is trained based on the second data set.

4. The method according to any one of claims 1 to 3, characterized in that: After inputting the image to be detected into the neural network defect feature extraction model and outputting the features to be compared, the method further includes: If the one or more similarities are all greater than or equal to the similarity threshold, or if an average value of the one or more similarities is greater than or equal to the similarity threshold, it is determined that the image to be inspected has no surface defects.

5. A surface defect detection device, characterized in that: The device comprises: A fifth acquisition module is configured to acquire a plurality of second sample images, wherein the plurality of second sample images have no surface defects; a third processing module, configured to input the plurality of second sample images into a neural network defect feature extraction model, and output image features of the plurality of second sample images; a clustering module, configured to cluster the image features of the plurality of second sample images to obtain a plurality of groups of normal data features; a fourth determining module, configured to select at least one image feature from each group of normal data features in the plurality of groups of normal data features to obtain a normal data representation feature, wherein the normal data representation feature is represented by a normal data feature matrix; A first acquisition module is used to acquire an image to be detected; a detection module, configured to input the image to be detected into a neural network defect segmentation model and output a first detection result, wherein the neural network defect segmentation model is used to detect surface defects of known defect categories, and the first detection result is represented by a matrix of the same size as the image to be detected; a first processing module, configured to, if the first detection result is an all-zero matrix, determine that the image to be detected does not have surface defects of the known defect category, input the image to be detected into the neural network defect feature extraction model, and output features to be compared, where the features to be compared are image features of the image to be detected and are represented by a feature matrix to be compared; a first determining module, configured to determine that a surface defect of an unknown defect category exists in the image to be inspected if at least one of the one or more similarities is less than a similarity threshold, or if an average of the one or more similarities is less than the similarity threshold; a second determining module, configured to determine a defect characteristic position, wherein the defect characteristic position refers to an element position in the feature matrix to be compared whose similarity with an element in phase position in the normal data feature matrix is ​​less than a similarity threshold, and determine, from the image pixel matrix corresponding to the image to be detected, an image pixel position corresponding to the defect characteristic position based on a mapping relationship between the image feature matrix and the image pixel matrix, as the defect position of the surface defect present in the image to be detected; The device further includes a module for performing the following operations: If the first detection result is a non-all-zero matrix, determining that a surface defect of a known defect category exists in the image to be detected, and determining the defect category and defect location of the surface defect existing in the image to be detected based on the position and value of the non-zero elements in the first detection result; Determine the similarity between the feature to be compared and one or more image features included in the normal data representation feature to obtain the one or more similarities.

6. The device according to claim 5, characterized in that The device further comprises: a second acquisition module, configured to acquire a first data set, the first data set including defect images of known defect categories and corresponding first annotation information, the first annotation information being annotation information indicating the defect categories; the first data set also including images without surface defects and corresponding second annotation information, the annotation information corresponding to the images without surface defects included in the first data set being "no surface defects", and the second annotation information being annotation information indicating no surface defects; The first training module is used to train the neural network defect segmentation model based on the first data set.

7. A surface defect detection system, characterized in that: The surface defect detection system includes a camera and at least one processor; The camera is used to capture at least one surface of the object to be detected as an image to be detected; The at least one processor is used to obtain the image to be detected and implement the steps of any one of the methods of claims 1-4.

8. The system according to claim 7, characterized in that The surface defect detection system further includes a conveying device, which is used to transport the object to be detected; The camera is used to photograph the object to be detected during the process of the conveying device transporting the object to be detected.

Images