A method and system for detecting surface defects of objects based on convolutional neural network

Abstract

The present invention belongs to the field of deep learning and artificial intelligence, and provides a method and system for detecting surface defects of objects based on convolutional neural networks. The method includes obtaining a surface image of the object to be detected; based on the surface image of the object to be detected, using a Gaussian mixture background model to extract the features of the surface of the object to be detected; using a trained neural network model to compare the features of the surface of the object to be detected with a feature library to detect whether there are defects; if so, obtaining the type and location of the defect.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and artificial intelligence, and in particular relates to a method and system for detecting surface defects of an object based on a convolutional neural network. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Traditional industrial production and manufacturing, due to the limitations of science and technology, still mainly uses manual inspection methods to detect surface defects of products. Due to manual limitations and backward technology, this method is not only slow and inefficient in product inspection, but also prone to errors during the inspection process, resulting in inaccurate inspection results.

[0004] Since the use of manual inspection methods has long been unable to meet the needs of production and modern process manufacturing, improving the efficiency and accuracy of product surface defect detection is still a technical problem that needs to be solved urgently. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a method and system for detecting surface defects of objects based on convolutional neural networks, which can quickly realize the detection of surface defects of objects and accurately identify the type and location of defects.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] The first aspect of the present invention provides a method for detecting surface defects of an object based on a convolutional neural network.

[0008] A method for detecting surface defects of an object based on a convolutional neural network, comprising:

[0009] Acquire a surface image of the object to be measured;

[0010] Based on the surface image of the object to be measured, a Gaussian mixture background model is used to extract the features of the surface of the object to be measured;

[0011] The trained neural network model is used to compare the surface features of the object to be tested with the feature library to detect whether there are defects; if so, the type and location of the defects are obtained.

[0012] A second aspect of the present invention provides a system for detecting surface defects of an object based on a convolutional neural network.

[0013] A convolutional neural network-based object surface defect detection system, comprising:

[0014] An acquisition module is configured to: acquire a surface image of the object to be measured;

[0015] A feature extraction module is configured to: extract features of the surface of the object to be measured based on the surface image of the object to be measured and using a Gaussian mixture background model;

[0016] The result detection module is configured to: use the trained neural network model to compare the features of the surface of the object to be tested with the feature library to detect whether there are defects; if so, obtain the type and location of the defects.

[0017] A third aspect of the present invention provides a computer-readable storage medium.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the object surface defect detection method based on a convolutional neural network as described in the first aspect above.

[0019] A fourth aspect of the present invention provides a computer device.

[0020] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for detecting surface defects of objects based on a convolutional neural network as described in the first aspect above are implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention greatly improves the efficiency of production operations, avoids the accuracy of detection results affected by operating conditions, subjective judgment, etc., can perform surface defect detection better and more accurately, and identify product surface defects more quickly.

[0023] The present invention uses a camera to capture the object to be tested based on a Gaussian mixture background model and transmits it to the cloud for detection. The model has high speed, low video memory occupancy, high accuracy, can reduce labor costs, and is more suitable for application in production engineering.

[0024] The present invention performs pruning on the basis of YOLOv3 and uses gIOU loss for model training, so that the model size is smaller and the generalization is stronger.

[0025] The technology adopted by the present invention is fast, accurate and real-time, and is suitable for production line quality inspection and engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 is a flow chart of a method for detecting surface defects of an object based on a convolutional neural network according to an embodiment of the present invention;

[0028] FIG. 2( a ) is a simplified diagram of a method for capturing a detection image using a fixed camera according to an embodiment of the present invention;

[0029] FIG2( b ) is a schematic diagram of a second method of capturing a detection image using a fixed camera according to an embodiment of the present invention;

[0030] FIG2( c ) is a third simplified diagram of a method of capturing a detection image using a fixed camera according to an embodiment of the present invention;

[0031] FIG. 3( a ) is a partial result of defect detection shown in an embodiment of the present invention. Figure 1 ,The rectangular box is the detected defect area;

[0032] FIG3( b ) is a second partial result diagram of defect detection obtained according to an embodiment of the present invention, wherein the rectangular frame is the detected defect area;

[0033] FIG3( c ) is a third partial result diagram of defect detection obtained according to an embodiment of the present invention, wherein the rectangular frame is the detected defect area;

[0034] Figure 4 1 is a schematic diagram of the test accuracy results shown in an embodiment of the present invention, where class0-class5 are 6 defect detection categories. DETAILED DESCRIPTION

[0035] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0036] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0039] Embodiment 1

[0040] like Figure 1 As shown, this embodiment provides a method for detecting surface defects of objects based on a convolutional neural network. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communications, middleware services, domain name services, security services CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application. In this embodiment, the method includes the following steps:

[0041] Acquire a surface image of the object to be measured;

[0042] Based on the surface image of the object to be measured, a Gaussian mixture background model is used to extract the features of the surface of the object to be measured;

[0043] The trained neural network model is used to compare the surface features of the object to be tested with the feature library to detect whether there are defects; if so, the type and location of the defects are obtained.

[0044] As one or more implementation modes, the Gaussian mixture background model obtains the foreground position according to the surface image of the object to be measured, and extracts the features of the surface of the object to be measured.

[0045] Specifically, in order to make the image of the detected object as clear and complete as possible and reduce missed detections, a fixed camera is used to monitor the objects on the production line, and the foreground position is obtained and the features of the object to be inspected are extracted based on Gaussian mixture background modeling to extract the foreground and background.

[0046] The model is initialized. Each pixel value of the first frame image is taken as the mean, and the variance w is set to establish q Gaussian mixture models. The default values ​​of q and w are 1 and 1.0.

[0047] Model learning, compare the pixels of the current frame with the pixels corresponding to the existing Gaussian model,

[0048] If |X is satisfied k -u (q,k) |>2.5σ (q,k) , the weight needs to be updated. The weight update method is as follows:

[0049] w k,t =(1-a)*w k,t-1 +a*M k,t

[0050] Among them, w is the weight, a is the learning rate, t represents the t-th pixel value, k represents the current k-th pattern, and for the matching pattern, w k,t =1, otherwise w k,t =0;

[0051] If |X is satisfied k -u (q,k) |≤2.5σ (q,k) , judge the background and foreground;

[0052] Where X K is the new pixel value, u (q.k) is the mean, and is σ (q.k) Standard deviation, q, k is the kth Gaussian component at time q.

[0053] The judgment formula is:

[0054]

[0055] Among them, w q is the result after weight normalization, B is the number of selected background patterns, and T represents the proportion of the background.

[0056] The detected foreground is analyzed, and when the object moves to the center of the camera, an image is captured, and the image to be tested is labeled and transmitted to the cloud for defect detection. Figure 2(a)-Figure 2(c) As shown. Figure 2(a) is a simplified diagram of the first method of capturing the detection image using a fixed camera, the top represents the camera, the rectangle represents the conveyor belt, the five-pointed star represents the object to be detected, and the arrow represents the direction of transmission to the moving direction; Figure 2(b) is a simplified diagram of the second method of capturing the detection image using a fixed camera, the simplified diagram is a top view, the five-pointed star represents the object to be detected, the rectangular frame represents the camera observation area, and at this time the object to be detected enters the camera observation area; Figure 2(c) is a simplified diagram of the third method of capturing the detection image using a fixed camera, the simplified diagram is a top view, the five-pointed star represents the object to be detected, the rectangular frame represents the camera observation area, and at this time the object to be detected has not entered the camera observation area.

[0057] As one or more implementation modes, the process of training the neural network model includes:

[0058] Acquire a historical data set of surface defects and divide it into a training set and a test set; the historical data set of surface defects includes surface defect images and a real defect type corresponding to each surface defect image;

[0059] The surface defect images in the training set are used as the input of the neural network model, combined with YOLO to predict the bounding box, and the defect type and defect features are output;

[0060] Compare the actual defect type with the defect type, combine the loss function, and optimize the neural network model;

[0061] The test set is used to input the optimized neural network model until the difference between the output value of the neural network model and the true value meets the set threshold, and a trained neural network model is obtained.

[0062] As one or more implementation modes, the loss function is:

[0063]

[0064] Among them, A is the labeled target detection box, B is the target detection box output by the neural network, and C is the minimum enclosing rectangle of A∪B.

[0065] As one or more implementation modes, the feature library includes a plurality of defect features.

[0066] Specifically, for different usage scenarios, we need to select appropriate data sets for training, so that the network model is more practical for specific scenarios and the accuracy of specific defects is improved. We perform pruning on the basis of YoloV3 and add IOU Loss as part of the network loss, as shown below:

[0067] In terms of bounding boxes, YoloV3 is the same as YoloV2: dimension clusters are used to find prior anchor boxes, and then bounding boxes are predicted through anchor boxes.

[0068] YOLO uses logistic regression to predict the objectness score of each bounding box. If the currently predicted bounding box overlaps best with the ground truth, its score is 1. If the current prediction is not the best, but it overlaps with the groundtruth object above a certain threshold, the network will ignore this prediction. The system only assigns one bounding box to each ground truth object. If the bounding box is not assigned to the corresponding object, it will not cause loss of coordinates or class predictions, but only target loss. During training, the sum of squared error losses is used, and gIOU Loss is added on this basis, and L1 regularization is used to reduce the degree of overfitting, thereby improving the accuracy of a certain class on the test set.

[0069]

[0070] IoU has the advantage of scale invariance, but it cannot be used directly as a loss function due to some problems:

[0071] If the two targets do not overlap, the IoU will be 0 and will not reflect the distance between the two targets. In this case of non-overlapping targets, if IoU is used as a loss function, the gradient is 0 and cannot be optimized.

[0072] IoU cannot distinguish between different alignments between two objects. More precisely, the IoU of two overlapping objects with the same intersection level in different directions will be exactly equal.

[0073] Therefore, this embodiment uses gIoU Loss for training

[0074]

[0075] GIoU can effectively solve the problems existing in IoU.

[0076] YOLOv3 does not use softmax to classify each box in class prediction, but uses an independent logistic classifier for prediction. During training, binary cross entropy loss is used for class prediction.

[0077] After channel pruning of YOLOv3, the number of model parameters and model size are reduced by 80%, FLOPs are reduced by 70%, and the forward inference speed can reach 200% of the original, while keeping mAP basically unchanged.

[0078] The detected defect information is marked and a test report is sent, indicating the object number and defect location to facilitate manual re-inspection later. Figure 3(a)-Figure 2(c) As shown, the rectangular box is the detected defect area.

[0079] Manual re-inspection is carried out based on the test report to find products that may have quality problems.

[0080] Figure 4 1 is a schematic diagram of the accuracy test results of the method described in this embodiment, and class0-class5 are 6 defect detection categories.

[0081] Embodiment 2

[0082] This embodiment provides an object surface defect detection system based on convolutional neural network.

[0083] A convolutional neural network-based object surface defect detection system, comprising:

[0084] An acquisition module is configured to: acquire a surface image of the object to be measured;

[0085] A feature extraction module is configured to: extract features of the surface of the object to be measured based on the surface image of the object to be measured and using a Gaussian mixture background model;

[0086] The result detection module is configured to: use the trained neural network model to compare the features of the surface of the object to be tested with the feature library to detect whether there are defects; if so, obtain the type and location of the defects.

[0087] It should be noted that the examples and application scenarios implemented by the acquisition module, feature extraction module and result detection module are the same as those in the steps of Embodiment 1, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above modules as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0088] Embodiment 3

[0089] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the object surface defect detection method based on a convolutional neural network as described in the above-mentioned embodiment 1 are implemented.

[0090] Embodiment 4

[0091] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for detecting surface defects of objects based on a convolutional neural network as described in the first embodiment are implemented.

[0092] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0093] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0097] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting surface defects of an object based on a convolutional neural network, characterized in that: include: Acquire a surface image of the object to be measured; Based on the surface image of the object to be measured, a Gaussian mixture background model is used to extract the features of the surface of the object to be measured; The Gaussian mixture background model obtains the foreground position according to the surface image of the object to be measured, and extracts the features of the surface of the object to be measured; Specifically, in order to make the image of the detected object as clear and complete as possible and reduce missed detection, a fixed camera is used to monitor the objects on the production line, and the foreground position is obtained based on Gaussian mixture background modeling and the features of the object to be detected are extracted to extract the foreground and background; Initialize the model, take each pixel value of the first frame image as the mean, set the variance w, and build q Gaussian mixture models, where q defaults to 1 and w defaults to 1.0; Model learning, compare the pixels of the current frame with the pixels corresponding to the existing Gaussian model, If |X is satisfied k -u (q,k) |>2.5σ (q,k) , the weight needs to be updated. The weight update method is as follows: w k,t =(1-a)*w k,t-1 +a*M k,t Among them, w is the weight, a is the learning rate, t represents the t-th pixel value, k represents the current k-th pattern, and for the matching pattern, w k,t =1, otherwise w k,t =0; If |X is satisfied k -u (q,k) |≤2.5σ (q,k) , judge the background and foreground; Where X K is the new pixel value, u (q.k) is the mean, and is σ (q.k) Standard deviation, q, k is the kth Gaussian component at time q; The judgment formula is: Among them, w q is the result after weight normalization, B is the number of selected background patterns, and T represents the proportion of the background; Analyze the detected foreground, capture the image when the object moves to the center of the camera, and label the image to be tested and transmit it to the cloud for defect detection; Using the trained neural network model, the features of the surface of the object to be tested are compared with the feature library to detect whether there are defects; if so, the type and location of the defects are obtained; The process of training the neural network model includes: Acquire a historical data set of surface defects and divide it into a training set and a test set; the historical data set of surface defects includes surface defect images and a real defect type corresponding to each surface defect image; The surface defect images in the training set are used as the input of the neural network model, combined with YOLO to predict the bounding box, and the defect type and defect features are output; Compare the actual defect type with the defect type, combine the loss function, and optimize the neural network model; The test set is used to input the optimized neural network model until the difference between the output value of the neural network model and the true value meets the set threshold, thereby obtaining a trained neural network model; The loss function is: Among them, A is the labeled target detection box, B is the target detection box output by the neural network, and C is the minimum enclosing rectangle of A∪B. The method is applied to a server and / or a terminal; the server includes: a physical server, a server cluster, a distributed system and a cloud server; the terminal includes: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker and a smart watch.

2. The object surface defect detection method based on convolutional neural network according to claim 1 is characterized in that: The feature library includes a plurality of defect features.

3. A surface defect detection system based on convolutional neural network, characterized in that: include: An acquisition module is configured to: acquire a surface image of the object to be measured; A feature extraction module is configured to: extract features of the surface of the object to be measured based on the surface image of the object to be measured and using a Gaussian mixture background model; The Gaussian mixture background model obtains the foreground position according to the surface image of the object to be measured, and extracts the features of the surface of the object to be measured; Specifically, in order to make the image of the detected object as clear and complete as possible and reduce missed detection, a fixed camera is used to monitor the objects on the production line, and the foreground position is obtained based on Gaussian mixture background modeling and the features of the object to be detected are extracted to extract the foreground and background; Initialize the model, take each pixel value of the first frame image as the mean, set the variance w, and build q Gaussian mixture models, where q defaults to 1 and w defaults to 1.0; Model learning, compare the pixels of the current frame with the pixels corresponding to the existing Gaussian model, If |X is satisfied k -u (q,k) |>2.5σ (q,k) , the weight needs to be updated. The weight update method is as follows: w k,t =(1-a)*w k,t-1 +a*M k,t Among them, w is the weight, a is the learning rate, t represents the t-th pixel value, k represents the current k-th pattern, and for the matching pattern, w k,t =1, otherwise w k,t =0; If |X is satisfied k -u (q,k) |≤2.5σ (q,k) , judge the background and foreground; Where X K is the new pixel value, u (q.k) is the mean, and is σ (q.k) Standard deviation, q, k is the kth Gaussian component at time q; The judgment formula is: Among them, w q is the result after weight normalization, B is the number of selected background patterns, and T represents the proportion of the background; Analyze the detected foreground, capture the image when the object moves to the center of the camera, and label the image to be tested and transmit it to the cloud for defect detection; The result detection module is configured to: use the trained neural network model to compare the surface features of the object to be tested with the feature library to detect whether there are defects; if so, obtain the type and location of the defects; The process of training the neural network model includes: Acquire a historical data set of surface defects and divide it into a training set and a test set; the historical data set of surface defects includes surface defect images and a real defect type corresponding to each surface defect image; The surface defect images in the training set are used as the input of the neural network model, combined with YOLO to predict the bounding box, and the defect type and defect features are output; Compare the actual defect type with the defect type, combine the loss function, and optimize the neural network model; The test set is used to input the optimized neural network model until the difference between the output value of the neural network model and the true value meets the set threshold, thereby obtaining a trained neural network model; The loss function is: Among them, A is the labeled target detection box, B is the target detection box output by the neural network, and C is the minimum enclosing rectangle of A∪B. The method is applied to a server and / or a terminal; the server includes: a physical server, a server cluster, a distributed system and a cloud server; the terminal includes: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker and a smart watch.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the object surface defect detection method based on a convolutional neural network as described in any one of claims 1-2 are implemented.

5. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the object surface defect detection method based on convolutional neural network as described in any one of claims 1-2 are implemented.

Images