Network model training method, metal surface defect detection method and electronic equipment

Abstract

The present invention discloses a network model training method, a metal surface defect detection method and an electronic device, which belongs to the technical field related to defect detection. The training method includes: obtaining metal surface defect data, and annotating the metal surface defect data to obtain training samples; using an oversampling method to perform data enhancement on the training samples to obtain a training set; using a lightweight network to replace the target detection model backbone feature extraction network to obtain a lightweight target detection network; using the NEU-DET data set as a pre-training data set, and sending it to the lightweight target detection network to obtain an improved lightweight target detection network; inputting the training set into the improved lightweight target detection network for training to obtain a lightweight target detection network model. The present invention can effectively realize the automatic detection of small defects on large-scale metal surfaces, has a high accuracy rate, and has the characteristics of lightweight, high-precision image real-time monitoring, etc., and can be used in industrial scenarios.

Description

Technical Field

[0001] The present application belongs to the technical field related to defect detection, and in particular relates to a network model training method, a metal surface defect detection method and an electronic device. Background Art

[0002] In the actual production process of metal, due to the influence of various factors, scratches, grooves and other defects will appear on the metal surface, which will seriously affect the quality of the metal. In order to ensure product quality, manual visual inspection is required. However, the surface of the metal itself will contain lines, holes, oil stains, etc., which are not easily distinguished from defects. Traditional manual visual inspection is very laborious and cannot accurately determine surface defects in a timely manner, making it difficult to control the efficiency of quality inspection.

[0003] In recent years, algorithms based on deep convolutional networks have had a huge impact. In the field of target detection, there are two-stage detectors represented by FasterR-CNN, and single-stage detectors represented by SSD and YOLO. Among them, the YOLO series is faster and more accurate, and YOLOv4 is widely used in industrial links.

[0004] However, in the process of implementing this application, the inventors found that the related technology has at least the following problems:

[0005] Current target detection algorithms build deeper and more complex networks while improving performance, but these networks do not meet the requirements of embedded and mobile devices in terms of scale and speed, and have limited application in the industrial field.

[0006] Current target detection algorithms require a large amount of data for training, but it is difficult to collect sufficient defect target data in actual industrial scenarios.

[0007] In industrial scenarios, the metal surface itself contains lines, holes, oil stains, etc., which are not easily distinguished from defects. In addition, the groove defects on the metal surface are small, and the scratches are thin and shallow, resulting in low accuracy of most detection methods and prone to missed detections. Summary of the invention

[0008] The purpose of the embodiments of the present application is to provide a network model training method, a metal surface defect detection method and an electronic device, which can be used to solve the problems in the related technology.

[0009] According to a first aspect of an embodiment of the present application, a method for training a network model is provided, comprising:

[0010] Acquire metal surface defect data, and annotate the metal surface defect data to obtain training samples;

[0011] Performing data enhancement on the training samples by using an oversampling method to obtain a training set;

[0012] The backbone feature extraction network of the target detection model is replaced by a lightweight network to obtain a lightweight target detection network.

[0013] Using the NEU-DET dataset as a pre-training dataset and feeding it into the lightweight object detection network to obtain an improved lightweight object detection network;

[0014] The training set is input into the improved lightweight object detection network for training to obtain a lightweight object detection network model.

[0015] Furthermore, metal surface defect data is obtained and the metal surface defect data is annotated, including:

[0016] Get images of metal surfaces at different angles and lighting conditions;

[0017] Marking key point positions of defects in the image;

[0018] The categories of defects in the image are labeled.

[0019] Furthermore, after the metal surface defect data is labeled, it also includes:

[0020] The annotated metal surface defect data are cropped and used as training samples.

[0021] Furthermore, an oversampling method is used to perform data enhancement on the training samples, including:

[0022] The images marked as defective parts in the training samples are deducted and copied to the images without defects through OpenCV to generate new images.

[0023] Furthermore, the backbone feature extraction network of the target detection network is replaced by a lightweight network to obtain a lightweight target detection network, including:

[0024] The backbone feature extraction network of the target detection network is changed to a lightweight network. After obtaining the effective feature layer, it is inserted into the target detection network. The ordinary convolution block is replaced by a lightweight module, and the activation function uses the hard sigmoid function.

[0025] According to a second aspect of an embodiment of the present application, a metal surface defect detection method is provided, comprising:

[0026] Acquire metal surface defect data to be detected;

[0027] The metal surface defect data to be detected is detected using a lightweight target detection network model trained based on the method described in the first aspect.

[0028] According to a third aspect of an embodiment of the present application, a training device for a metal surface defect detection model is provided, comprising:

[0029] The first acquisition module is used to acquire metal surface defect data and annotate the metal surface defect data to obtain training samples;

[0030] An enhancement module, used for performing data enhancement on the training samples by using an oversampling method to obtain a training set;

[0031] A replacement module, used to replace the backbone feature extraction network of the target detection network with a lightweight network to obtain a lightweight target detection network;

[0032] A first training module is used to use the NEU-DET dataset as a pre-training dataset and send it into the lightweight object detection network to obtain an improved lightweight object detection network;

[0033] The second training module is used to input the training set into the improved lightweight target detection network for training to obtain a metal surface pit detection model.

[0034] According to a fourth aspect of an embodiment of the present application, a metal surface defect detection device is provided, comprising:

[0035] The second acquisition module is used to acquire the metal surface defect data to be detected;

[0036] A detection module is used to detect the metal surface defect data to be detected using a lightweight target detection network model trained based on the method described in the first aspect.

[0037] According to a fifth aspect of an embodiment of the present application, there is provided an electronic device, including:

[0038] one or more processors;

[0039] A memory for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the metal surface defect detection model as described in the first aspect, or the metal surface defect detection method as described in the second aspect.

[0041] According to the sixth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the training method of the metal surface defect detection model as described in the first aspect, or the metal surface defect detection method as described in the second aspect is implemented.

[0042] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0043] It can be seen from the above embodiments that the present application uses the oversampling data enhancement method to effectively solve the problem of small data volume in the core data set and the detection accuracy of small defects.

[0044] The backbone feature extraction network of the target detection network is changed to a lightweight network. After obtaining the effective feature layer, it is inserted into the target detection network as a modified network, which reduces the parameters required by the target detection network. On the one hand, it reduces most of the calculation amount so that the network can run on the terminal, and on the other hand, it improves the detection accuracy.

[0045] Using the NEU-DET dataset as the pre-training dataset effectively solves the problem of small data volume in the core dataset.

[0046] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0048] Figure 1 The figure is a flowchart of a method for training a network model according to an exemplary embodiment.

[0049] Figure 2 A comprehensive metal surface picture of 5472*3648 pixels after collection and annotation is shown according to an exemplary embodiment.

[0050] Figure 3 The original metal surface image and the oversampled image are shown according to an exemplary embodiment, wherein (a) is a partial metal surface image with a size of 1024*1024 pixels in the embodiment, and (b) is a data set image after the image is enhanced by oversampling data.

[0051] Figure 4 It is a YOLOv4 backbone feature extraction network CSPDarkNet53 structure shown according to an exemplary embodiment.

[0052] Figure 5 FIG. 4 is a Ghost Module structure shown according to an exemplary embodiment.

[0053] Figure 6 is an image output by a pre-trained network according to an exemplary embodiment.

[0054] Figure 7 The figure is a flow chart of a metal surface defect detection method according to an exemplary embodiment.

[0055] Figure 8 is an image of a network output according to an exemplary embodiment.

[0056] Fig. 9 1 is an accuracy, recall and AP curve according to an exemplary embodiment, wherein (a), (b) and (c) are the accuracy, recall and AP curves obtained by GhostNet-YOLOv4, respectively.

[0057] Fig.10 It is a block diagram of a network model training device according to an exemplary embodiment.

[0058] Fig.11 The figure is a block diagram of a metal surface defect detection device according to an exemplary embodiment. DETAILED DESCRIPTION

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0060] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0061] Figure 1 is a flowchart of a method for training a network model according to an exemplary embodiment. Figure 1 As shown, the following steps may be included:

[0062] Step S11, acquiring metal surface defect data, and annotating the metal surface defect data to obtain training samples;

[0063] Step S12, performing data enhancement on the training samples by using an oversampling method to obtain a training set;

[0064] Step S13, replacing the backbone feature extraction network of the target detection model with a lightweight network to obtain a lightweight target detection network;

[0065] Step S14, using the NEU-DET dataset as a pre-training dataset and sending it into the lightweight object detection network to obtain an improved lightweight object detection network;

[0066] Step S15, inputting the training set into the improved lightweight object detection network for training to obtain a lightweight object detection network model.

[0067] It can be seen from the above embodiments that the present application uses the oversampling data enhancement method to effectively solve the problem of the small amount of data in the core data set and the detection accuracy of small defects. The backbone feature extraction network of the target detection network is changed to a lightweight network, and after obtaining the effective feature layer, it is inserted into the target detection network as a change network, which reduces the parameters required by the target detection network. On the one hand, it reduces most of the calculation amount, allowing the network to run on the terminal, and on the other hand, it improves the detection accuracy. Using the NEU-DET dataset as a pre-training dataset effectively solves the problem of the small amount of data in the core dataset.

[0068] In the specific implementation of step S11, metal surface defect data is obtained, and the metal surface defect data is annotated to obtain training samples; this step may include the following sub-steps:

[0069] Step S111, acquiring images of the metal surface at different angles and illuminations;

[0070] Specifically, place the metal on the table and use a high-precision camera to collect high-precision, large-scale images of the metal surface at different angles and lighting conditions. The specific size is 5472*3648, such as Figure 2 shown.

[0071] Step S112, marking the key point positions of the defects in the image;

[0072] Specifically, LabelImg is used on the original image to mark various defects, and the marking format is the horizontal and vertical coordinate values ​​of the key points.

[0073] Step S113, marking the categories of defects in the image.

[0074] Specifically, the defect category labels are marked, and the marking format is the names of different defect categories, specifically groove and scratch;

[0075] After the metal surface defect data is labeled, it also includes:

[0076] Step S114, cropping the annotated metal surface defect data, and using the cropped data as training samples.

[0077] Specifically, the large-scale metal surface image is cropped into a 1024*1024 size picture according to the defect location, and then an XML format file containing the metal surface defect information can be generated.

[0078] In the specific implementation of step S12, an oversampling method is used to perform data enhancement on the training samples to obtain a training set;

[0079] Specifically, the images marked as defective parts in the training samples are deducted and copied to images without defects through opencv to generate new images, and all images are divided to obtain training sets. More specific steps are as follows:

[0080] (1) Traverse all defect files. If the pixel size of the defect is less than or equal to 32 pixels, it is considered a small defect target and is cropped out of the image and saved.

[0081] (2) The cropped images are randomly inverted, noisy, etc., and then evenly pasted on the defect-free images. Figure 8 new defects that do not overlap with other defects, such as Figure 3 shown.

[0082] In the specific implementation of step S13, a lightweight network is used to replace the backbone feature extraction network of the target detection model to obtain a lightweight target detection network;

[0083] Conventional CNN reasoning requires a lot of computing power, so it is difficult to apply it to resource-constrained scenarios such as mobile terminals and the Internet of Things. Only through complex cropping and quantization can it be barely deployed on mobile terminals. Currently, the more mature lightweight networks include: Google's MobileNet series, EfficientNet Lite series, Kuangshi's ShuffleNet series, Huawei's GhostNet, etc. GhostNet is used as an example in this example. Object detection algorithms include two-stage detectors represented by Faster R-CNN, and single-stage detectors represented by SSD and YOLO. Among them, the YOLO series is faster and more accurate, and YOLOv4 is widely used in industrial links.

[0084] Specifically, the YOLOv4 backbone feature extraction network CSPDarkNet53 is changed to the GhostNet network, and three effective feature layers are obtained and inserted into YOLOv4, and the GhostModule method is used to replace the ordinary convolution block. Figure 5As shown in the figure, GhostModule divides ordinary convolution into three steps: conventional convolution to generate intrinsic feature map, Ghost generation and feature map concatenation. Specifically, first, the number of layers of the intrinsic feature map and the Ghost feature map is set to be the same, then Ghost generation uses 3*3 ordinary convolution to linearly transform the intrinsic feature map into the Ghost feature map, and finally concat is used to concatenate the intrinsic feature map and the Ghost feature map to generate the final output.

[0085] The structure of CSPDarkNet53 is as follows Figure 4 As shown in the figure, the network has five residual edges, and the last three residual edges will output three valid feature layers. When the input is 416*416*3, the third, fourth, and fifth residual edges will output feature maps of size 52*52, 26*26, and 13*13 respectively. The convolutional blocks of GhostNet in the example are shown in Table 1:

[0086]

[0087]

[0088] Table 1

[0089] Where k is the convolution kernel size, t is the dimensionality ratio, c is the number of output channels, SE is the SE structure, and s represents the compression multiple.

[0090] When the input is 416*416*3, the output after Stage3 is 52*52*40, the output after Stage4 is 26*26*112, and the output after Stage5 is 13*13*160, which replaces the valid feature layer output by CSPDarkNet53 for subsequent deep feature extraction.

[0091] The activation function uses the hard sigmoid function, specifically:

[0092] In the specific implementation of step S14, the NEU-DET dataset is used as a pre-training dataset and sent to the lightweight object detection network to obtain an improved lightweight object detection network;

[0093] The lightweight target detection network described in this example is GhostNet-YOLOv4 used by S13;

[0094] Specifically, when executing this step, the NEU-DET dataset is used as the input of the neural network, the neural network training batch size is set to 8, the training cycle epoch is set to 60, the learning rate is set to 0.001, and the network model with the smallest validation loss is found as the pre-training result. The pre-training output result is as follows Figure 6 shown.

[0095] In the specific implementation of step S15, the training set is input into the improved lightweight object detection network for adversarial training to obtain a lightweight object detection network model.

[0096] The lightweight target detection network described in this example is GhostNet-YOLOv4 used by S13.

[0097] Specifically, when executing this step, the metal surface defect dataset is used as the input of the neural network, the neural network training batch size is set to 8, the initial training cycle epoch is 20, and 280 layers of training are frozen. Freezing training can speed up the training speed and prevent the weights from being destroyed in the early stage of training. The learning rate is 0.001, and the change of learning rate adopts cosine annealing decay. After the training is completed, the training is thawed for 280 cycles, the neural network training batch size is set to 4, the learning rate is 0.0001, and the change of learning rate still adopts cosine annealing decay. After the training is completed, validation loss is used as the evaluation criterion for the model training effect.

[0098] Figure 7 is a flow chart of a metal surface defect detection method according to an exemplary embodiment. Figure 7 As shown, the following steps may be included:

[0099] Step S21, obtaining metal surface defect data to be detected;

[0100] Step S22, using the GhostNet-YOLOv4 network model trained by the above method to detect the metal surface defect data to be detected.

[0101] The specific processing of step S21 to obtain the metal surface defect data to be detected can refer to step S11, which will not be repeated here.

[0102] In the specific implementation of step S22, the GhostNet-YOLOv4 network model trained by the above method is used to detect the metal surface defect data to be detected.

[0103] Specifically, by inputting metal surface images and videos into the trained GhostNet-YOLOv4, defects can be quickly classified and located. This example uses AP, precision, and recall to evaluate defect detection results. The higher the AP, the better the network's detection results for metal surface defects. The calculation formulas for AP, precision, and recall are as follows:

[0104] Precision = N TP / (N TP +N FP )

[0105] Recall = N TP / (N TP +N FN )

[0106] AP is the area of ​​the curve enclosed by precision and recall. TP N is the positive sample classified correctly. FN Misclassified positive samples, N FP is a negative sample that is misclassified. The experimental results of this example are as follows Figure 8 shown.

[0107] The comparison between this example and the conventional YOLOv4 is shown in Table 2. As can be seen from the table. Compared with the YOLOv4 network, GhostNet-YOLOv4 has improved its AP from 66% to 94% in terms of groove defects, an increase of 28%, and mAP from; at the same time, the detection speed of large metal images (5472*3648) of YOLOv4 is 8.9fps, and the detection speed of GhostNet is 25fps, an increase of 180%. The accuracy, recall rate and AP values ​​are shown in the figure. Fig. 9 As shown in the figure, the number of parameters required for the operation is reduced from 64,040,001 to 11,428,545, a decrease of 17.8% of the original number. It can be seen that a huge improvement has been achieved compared to the original YOLOv4.

[0108] Network Model Groove AP Scratch AP mAP Detection speed Parameter quantity YOLOv4 66％ 80％ 72.94％ 8.9fps 64040001 GhostNet-YOLOv4 94％ 81％ 87.76％ 25fps 11428545

[0109] Table 2

[0110] Corresponding to the aforementioned embodiment of a method for training a metal surface defect detection model, the present application also provides an embodiment of a device for training a metal surface defect detection model.

[0111] Fig.10 is a block diagram of a training device for a metal surface defect detection model according to an exemplary embodiment. Fig.10 , the device comprises:

[0112] The first acquisition module 31 is used to acquire metal surface defect data and annotate the metal surface defect data to obtain training samples;

[0113] An enhancement module 32, configured to perform data enhancement on the training samples by using an oversampling method to obtain a training set;

[0114] A replacement module 33 is used to replace the YOLOv4 backbone feature extraction network with the GhostNet network to obtain a GhostNet-YOLOv4 network;

[0115] A first training module 34 is used to use the NEU-DET data set as a pre-training data set and send it into the GhostNet-YOLOv4 network to obtain an improved YOLOv4 network;

[0116] The second training module 35 is used to input the training set into the improved YOLOv4 network for adversarial training to obtain a metal surface pit detection model.

[0117] Corresponding to the aforementioned embodiment of a metal surface defect detection method, the present application also provides an embodiment of a metal surface defect detection device.

[0118] Fig.11 is a block diagram of a metal surface defect detection device according to an exemplary embodiment. Fig.11 , the device comprises:

[0119] The second acquisition module 41 is used to acquire the metal surface defect data to be detected;

[0120] The detection module 42 is used to detect the metal surface defect data to be detected by using the GhostNet-YOLOv4 network model trained by the above method.

[0121] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0122] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0123] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the metal surface defect detection model, or the metal surface defect detection method as described above.

[0124] Correspondingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, characterized in that when the instructions are executed by a processor, a training method for a metal surface defect detection model or a metal surface defect detection method as described above is implemented.

[0125] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0126] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A network model training method, characterized in that: include: Acquire metal surface defect data, and annotate the metal surface defect data to obtain training samples; Performing data enhancement on the training samples by using an oversampling method to obtain a training set; The backbone feature extraction network of the target detection model is replaced by a lightweight network to obtain a lightweight target detection network. Using the NEU-DET dataset as a pre-training dataset and feeding it into the lightweight object detection network to obtain an improved lightweight object detection network; Inputting the training set into the improved lightweight object detection network for training to obtain a lightweight object detection network model; Among them, a lightweight network is used to replace the target detection backbone feature extraction network to obtain a lightweight target detection network, including: The backbone feature extraction network of the target detection network is changed to a lightweight network. After obtaining three effective feature layers, they are inserted into the target detection network. The ordinary convolution block is replaced by a lightweight module, and the hard sigmoid function is used as the activation function.

2. The training method according to claim 1, characterized in that: Obtain metal surface defect data and mark the metal surface defect data, including: Get images of metal surfaces at different angles and lighting conditions; Marking key point positions of defects in the image; The categories of defects in the image are labeled.

3. The training method according to claim 1, characterized in that: After the metal surface defect data is labeled, it also includes: The annotated metal surface defect data are cropped and used as training samples.

4. The training method according to claim 1, characterized in that: The oversampling method is used to perform data enhancement on the training samples, including: The images marked as defective parts in the training samples are deducted and copied to the images without defects through OpenCV to generate new images.

5. A metal surface defect detection method, characterized in that: include: Acquire metal surface defect data to be detected; The metal surface defect data to be detected is detected using a lightweight target detection network model trained based on the method described in any one of claims 1 to 4.

6. A training device for a metal surface defect detection model, characterized in that: include: The first acquisition module is used to acquire metal surface defect data and annotate the metal surface defect data to obtain training samples; An enhancement module, used for performing data enhancement on the training samples by using an oversampling method to obtain a training set; A replacement module is used to replace the backbone feature extraction network of the target detection network model with a lightweight network to obtain a lightweight target detection network; A first training module is used to use the NEU-DET dataset as a pre-training dataset and send it into the lightweight object detection network to obtain an improved lightweight object detection network; A second training module is used to input the training set into the improved lightweight object detection network for training to obtain a metal surface pit detection model; Among them, a lightweight network is used to replace the target detection backbone feature extraction network to obtain a lightweight target detection network, including: The backbone feature extraction network of the target detection network is changed to a lightweight network. After obtaining three effective feature layers, they are inserted into the target detection network. The ordinary convolution block is replaced by a lightweight module, and the hard sigmoid function is used as the activation function.

7. A metal surface defect detection device, characterized in that: include: The second acquisition module is used to acquire the metal surface defect data to be detected; A detection module is used to detect the metal surface defect data to be detected using a lightweight target detection network model trained based on any one of the methods described in claims 1-4.

8. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the metal surface defect detection model as described in any one of claims 1 to 4, or the metal surface defect detection method as described in claim 5.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the training method of the metal surface defect detection model as described in any one of claims 1 to 4, or the metal surface defect detection method as described in claim 5 is implemented.

Images