A single-stage target detection neural network based on multi-scale fusion and an industrial product surface defect detection model

Abstract

The application provides a single-stage target detection neural network based on multi-scale fusion and an industrial product surface defect detection model, and belongs to the technical field of image processing and analysis. The application solves the problem of detection and identification of industrial product surface defects, and can improve the identification rate of defect detection and reduce the missed detection of small target defects. The neural network of the application is composed of an image preprocessing layer, a feature extraction network layer, a connection layer, a multi-scale fusion network layer and a prediction network layer, which respectively realize the functions of image data enhancement, image feature extraction, convolution connection, multi-scale fusion processing of image features and multi-sampling processing of the fusion features of the multi-scale fusion network layer. The application adopts the maximum pooling method for multi-scale fusion, increases the number of predictors, realizes the full use of the feature information of the shallow network by the deep network, and can detect and identify the industrial product surface defects by using the trained model.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing and image analysis, specifically relating to a single-stage target detection neural network based on multi-scale fusion and an industrial product surface defect detection model. Background Technology

[0002] In the actual production process of industrial products such as product packaging, printed circuit boards (PCBs), and glass bottles, minor defects such as scratches and bubbles can occur on the surface of industrial products due to factors such as vibration and manufacturing processes. These defects can seriously affect the quality of industrial products, thus requiring defect detection. Currently, most manufacturers still use manual visual inspection methods. However, defects on the surface of industrial products are often small and have low distinguishability, making manual inspection laborious, inefficient, and prone to failure. Using traditional image defect detection techniques to detect defects in images of industrial products such as PCBs requires pixel-level traversal of the image to be inspected, resulting in slow detection speed and insufficient detection recognition rate. In recent years, with the development of deep learning technology, deep learning, represented by convolutional neural networks (CNNs), has received widespread attention in defect detection. As a type of feedforward neural network that includes convolutional operations and has a deep structure, CNNs can learn defect features through forward propagation and adjust convolutional kernel parameters through backpropagation, intelligently detecting defect images and becoming a research hotspot in related fields. Summary of the Invention

[0003] In view of this, in order to improve the recognition rate of surface defects in industrial products, reduce the false negative rate of small target defects, ensure the quality inspection speed of industrial product surfaces, and achieve accurate identification of small target defects in relatively complex working environments and inspection backgrounds, this invention aims to propose a single-stage target detection neural network based on multi-scale fusion and an industrial product surface defect detection model.

[0004] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0005] Step 1: Establish a neural network for detecting surface defects in industrial products - a single-stage target detection neural network YOLO_4t based on multi-scale fusion, for detecting surface defects in industrial products;

[0006] Step 2: Obtain the image training set of the industrial products to be inspected, and standardize the length, width and height and image type of the acquired images;

[0007] Step 3: Divide the image training set obtained in Step 2 into training set D-1, validation set D-2, and test set D-3; training set D-1 is used to train the network model, validation set D-2 is used to prevent overfitting of the network model, and test set D-3 is used to test the performance of the network model.

[0008] Step 4: Label and classify the images in the training set D-1, validation set D-2 and test set D-3 mentioned in Step 3, and organize them into YOLO dataset format;

[0009] Step 5: Based on the features of the industrial product images to be detected in Step 2, preset the number of targets to be detected and obtain the number and size of detection prior boxes on the training set D-1 using the K-means clustering algorithm;

[0010] Step 6: Select the neural network YOLO_4t described in Step 1, and input the prior box data obtained in Step 5 into the cfg configuration file of the neural network YOLO_4t;

[0011] Step 7: Input the image data in the training set D-1 processed in Step 4 into the neural network YOLO_4t in sequence. Based on the outputs of predictor 1, predictor 2 and predictor 3 in the network YOLO_4t, optimize the series of convolution kernels in the neural network YOLO_4t using the loss function in the network YOLO_4t to obtain the optimized network model MM.

[0012] Step 8: Input the defect images of the validation set D-2 processed in Step 4 into the network model MM described in Step 7 to perform defect detection on the images and prevent overfitting of the network model MM.

[0013] Step 9: Input the images of the test set D-3 processed in Step 4 into the network model MM described in Step 8, and verify the output of the network model MM and the actual images to obtain the verification accuracy.

[0014] Step 10: When the verification accuracy described in Step 9 reaches the expected target, the final network model YOLO_4t_mod is obtained and used to perform defect detection on the surface image of the industrial product to be inspected; when the verification accuracy described in Step 9 is lower than the expected target, the parameters of the network model MM need to be adjusted, and Steps 7-10 are repeated until the verification accuracy reaches the expected target and the final network model YOLO_4t_mod is obtained.

[0015] The industrial product image dataset mentioned in step 2 is an image dataset of industrial products with surface defects, including but not limited to surface defects of industrial products such as printed circuit boards (PCBs), printed cigarette boxes, printed medicine boxes, and fabrics. The labeling types of defects include: spur, copper, pin-hole, mousebite, open, short, etc. Taking printed circuit boards as an example.

[0016] The YOLO_4t single-stage target detection neural network based on multi-scale fusion is a single-stage defect detection network for small targets across all scales.

[0017] The YOLO_4t single-stage target detection neural network based on multi-scale fusion is characterized by comprising an image preprocessing layer, a feature extraction network layer, a connection layer, a multi-scale fusion layer, and a prediction network layer. The image preprocessing layer has one input and one output, with the output connected to the input of the feature extraction network layer. The feature extraction network layer has one input and three outputs, two of which are intermediate outputs, and the third output is connected to the input of the multi-scale fusion layer. The connection layer has one input and one output, with the output connected to the input of the multi-scale fusion layer. The multi-scale fusion layer has one input and one output, which performs calculations based on the obtained inputs and optimizes the parameters of the fusion unit based on the calculation results. The prediction layer (prediction network layer) has three inputs and one output, with one of the three inputs connected to the output of the multi-scale fusion layer, and the other two inputs connected to the two intermediate outputs of the feature extraction layer (feature extraction network layer). The feature extraction network layer and the multi-scale fusion network layer transmit data through the connection layer, which is used to convolve the image output by the feature extraction network layer again.

[0018] The image preprocessing layer consists of a data augmentation layer, a convolutional layer A-1, an instance normalization layer-1, a Leaky ReLU activation function layer, a convolutional layer A-2, an instance normalization layer-2, and a Leaky ReLU activation function layer connected in series, and is used for data augmentation of image data.

[0019] The feature extraction network layer is composed of a feature extractor AA1, a max pooling layer M1, a feature extractor AA2, a max pooling layer M2, a feature extractor AA3, and a max pooling layer M3 connected in series, used to extract image features. Feature extractors AA2 and AA3 each contain an intermediate output, forming the two intermediate outputs of the feature extraction layer. Feature extractor AA1 is composed of a convolutional layer A1, an instance normalization layer 1, a Leaky ReLU activation function layer, an operational layer 1, a convolutional layer A2, an instance normalization layer 2, a Leaky ReLU activation function layer, a convolutional layer A3, an instance normalization layer 3, a Leaky ReLU activation function layer, an operational layer 2, a convolutional layer A4, an instance normalization layer 4, a Leaky ReLU activation function layer, and an operational layer 3 connected in series. Feature extractor AA2 is composed of a convolutional layer A5, an instance normalization layer 5, a Leaky ReLU activation function layer, an operational layer 4, a convolutional layer A6, an instance normalization layer 6, and a Leaky ReLU activation function layer 3 connected in series. The feature extractor AA2 is composed of a ReLU activation function layer, a convolutional layer A7, an instance normalization layer 7, a Leaky ReLU activation function layer, an operational layer 5, a convolutional layer A8, an instance normalization layer 8, a Leaky ReLU activation function layer, and an operational layer 6, connected in series. The output of the convolutional layer A8 of the feature extractor AA2, after passing through the instance normalization layer 8 and the Leaky ReLU activation function, is the intermediate output of the feature extractor AA2, which is connected to one input of the operational layer 15 within the predictor 3 of the prediction layer. The feature extractor AA3 consists of a convolutional layer A9, an instance normalization layer 9, a Leaky ReLU activation function layer, an operational layer 7, a convolutional layer A10, an instance normalization layer 10, a Leaky ReLU activation function layer, a convolutional layer A11, an instance normalization layer 11, a Leaky ReLU activation function layer, an operational layer 8, a convolutional layer A12, an instance normalization layer 12, and a Leaky ReLU activation function layer 6. The ReLU activation function layer and the computation layer 9 are connected in series. The output of the convolutional layer A12 of the feature extractor AA3, after passing through the instance normalization layer 12 and the Leaky ReLU activation function layer, is the intermediate output of the feature extractor AA3, which is connected to one input of the computation layer 13 in the predictor 2 of the prediction layer. The instance normalization layers -1, -2, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 have the same structure and function. The Leaky ReLU activation function layer is constructed using a modified linear unit function with leakage for activation processing.The first operational layer divides the feature map of convolutional layer A1 into two equal groups and uses the first group as the input to convolutional layer A2, employing the route algorithm for grouping. The second operational layer performs a summation operation between the outputs of convolutional layer A2 and convolutional layer A3, using the add algorithm. The third operational layer performs a summation operation between the outputs of convolutional layer A1 and convolutional layer A4, also using the add algorithm. The fourth operational layer divides the feature map of convolutional layer A5 into two equal groups and uses the first group as the input to convolutional layer A6, employing the route algorithm for grouping. The fifth operational layer performs a summation operation between the outputs of convolutional layer A6 and convolutional layer A2. The summation operation between the outputs of A7 uses the add algorithm; the operation layer 6 performs the summation operation between the outputs of convolutional layers A5 and A8 using the add algorithm; the operation layer 7 divides the feature map of convolutional layer A9 into two groups and takes the first group as the input of convolutional layer A10, using the route algorithm; the operation layer 8 performs the summation operation between the outputs of convolutional layers A10 and A11 using the add algorithm; the operation layer 9 performs the summation operation between the outputs of convolutional layers A9 and A12 using the add algorithm.

[0020] The connection layer is composed of a convolutional layer A13, a strength normalization layer 13, and a Leaky ReLU activation function layer connected in sequence;

[0021] The multi-scale fusion layer consists of a max pooling layer M4, a computation layer 10, a max pooling layer M5, and a computation layer 11. The computation layer 10 performs cross-layer concatenation of the inputs of the multi-scale fusion layer as the input of the max pooling layer M5. The computation layer 11 performs the summation operation of the inputs of the multi-scale fusion layer and the outputs of the max pooling layers M4 and M5, using the add algorithm. The input of the max pooling layer M4 is the input of the multi-scale fusion layer, and the max pooling layers M4 and M5 have exactly the same structure and function.

[0022] The prediction network layer contains three predictors: predictor 1, predictor 2, and predictor 3, which are used to process the fused features output by the multi-scale fusion network at different sampling rates. Predictor 1 is composed of convolutional layer A14, instance normalization layer 14, Leaky ReLU activation function layer, convolutional layer A15, instance normalization layer 15, Leaky ReLU activation function layer, convolutional layer A16, instance normalization layer 16, Leaky ReLU activation function layer, convolutional layer A17, Leaky ReLU activation function layer, and decoder Y1 connected in series. The input of predictor 1 is connected to the output of the multi-scale fusion layer, and the output of predictor 1 is the predicted bounding box of the feature map downsampled by 5 times. Predictor 2 is composed of computational layer 12, convolutional layer A18, instance normalization layer 17, Leaky ReLU activation function layer, upsampling layer U1, computational layer 13, convolutional layer A19, instance normalization layer 18, Leaky ReLU activation function layer, and Leaky ReLU activation function layer. The predictor 2 is constructed by sequentially connecting the ReLU activation function layer, convolutional layer A20, instance normalization layer 19, Leaky ReLU activation function layer, convolutional layer A21, Leaky ReLU activation function layer, and decoder Y2. The input of predictor 2 is connected to the output of the convolutional layer A16 of predictor 1, processed by instance normalization layer 16 and Leaky activation function layer. Predictor 2 outputs a predicted bounding box of the feature map with a 4x downsampling. One input of the computational layer 13 of predictor 2 is connected to the intermediate output of feature extractor AA3. Predictor 3 consists of computational layer 14, convolutional layer A22, instance normalization layer 20, Leaky ReLU activation function layer, upsampling layer U2, computational layer 15, convolutional layer A23, instance normalization layer 21, Leaky ReLU activation function layer, convolutional layer A24, and Leaky ReLU activation function layer Y2. The ReLU activation function layer and decoder Y3 are connected in series. The input of predictor 3 is connected to the output of the convolutional layer A20 of predictor 2, which is processed by the instance normalization layer 19 and the Leaky activation function layer. The output of predictor 3 is the predicted bounding box of the feature map downsampled by 3 times. One input of the operation layer 15 of predictor 3 is connected to the intermediate output of feature extractor AA2. The operation layer 12 implements the cross-layer connection between predictor 1 and predictor 2, and its output is the input of convolutional layer A18. The operation layer 13 implements the summation operation between the intermediate output of feature extractor AA3 and the output of upsampling layer U1, and the summation operation uses the add algorithm. The operation layer 14 implements the cross-layer connection between predictor 2 and predictor 3, and its output is connected to the input of convolutional layer A22. The operation layer 15 implements the summation operation between the intermediate output of feature extractor AA2 and the output of upsampling layer U2, and the summation operation uses the add algorithm.The decoding formulas for decoders Y1, Y2, and Y3 are the same as those in the YOLO-v4tiny network.

[0023] The convolutional layers A-1, A-2, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, A21, A22, A23, and A24 have identical structures, and their construction parameters include the number of convolutional kernels, kernel normalization method, kernel size, kernel stride, kernel padding method, kernel bias term, and initial kernel value.

[0024] The instance normalization layers 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, and 21 have the same structure and function, and are used to adjust the input data to a specific range.

[0025] The YOLO_4t single-stage target detection neural network based on multi-scale fusion is characterized in that the YOLO_4t network is built based on the Darknet deep learning framework, and the loss function of the YOLO_4t network is the same as the loss function of the YOLO-v4tiny network.

[0026] The YOLO_4t single-stage target detection neural network based on multi-scale fusion is characterized in that the padding method of the padding layer PA in the YOLO_4t network is same padding, the padding size is 1, and the padding method is symmetrical padding.

[0027] The YOLO_4t single-stage target detection neural network based on multi-scale fusion is characterized in that the padding method of the padding layer PB in the YOLO_4t network adopts the interpolation method, the padding size is 13, and the padding method is symmetrical padding.

[0028] The aforementioned industrial product surface defect detection model based on a multi-scale fusion single-stage target detection neural network is characterized in that, in step 2, preferably, image normalization processing is performed using Python and OpenCV to normalize the image size to 640*640 and perform image binarization processing; in step 4, preferably, the labeling software LabelImage is used to label defects in the image, and the generated TXT file is used for training the YOLO_4t network.

[0029] Compared with existing technologies, the single-stage target detection neural network and industrial product surface defect detection model based on multi-scale fusion described in this invention have the following advantages:

[0030] This invention discloses a single-stage target detection neural network and industrial product surface defect detection model based on multi-scale fusion. The feature extraction network layer and multi-scale fusion layer are constructed based on a deep convolutional neural network. The internal convolutional layers are cross-connected and fuse features, enabling the deep network to fully utilize the feature information from the shallow network. The multi-scale fusion layer employs 3×3 and 5×5 max pooling methods for multi-scale fusion, increasing the number of feature channels and improving the ability to extract defect features. Furthermore, by adding a predictor, the ability to identify small target defects is enhanced. A neural network model is obtained through training, which can be used to detect surface defects in industrial products, including PCBs. This invention effectively solves the problem of defect classification and localization in images of defective industrial products, including PCBs, and has great application potential and value in image content analysis, computer vision, and intelligent detection of image information quality in industrial products. Attached Figure Description

[0031] The accompanying drawings, which constitute the basis of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0032] Figure 1 This is a schematic diagram of the network structure of the single-stage target detection neural network based on multi-scale fusion described in this invention.

[0033] Figure 2 This is a structural diagram of the feature extraction network layer in the single-stage target detection neural network based on multi-scale fusion described in this invention;

[0034] Figure 3 This is a structural diagram of the feature extractor AA1 in the feature extraction network layer described in this invention;

[0035] Figure 4 This is a structural diagram of the feature extractor AA2 in the feature extraction network layer described in this invention;

[0036] Figure 5 This is a structural diagram of the feature extractor AA3 in the feature extraction network layer described in this invention;

[0037] Figure 6 This is a structural diagram of the multi-scale fusion layer in the single-stage target detection neural network based on multi-scale fusion described in this invention;

[0038] Figure 7 This is a structural diagram of predictor 1 in the prediction layer of the present invention;

[0039] Figure 8 This is a structural diagram of predictor 2 in the prediction layer of the present invention;

[0040] Figure 9 This is a structural diagram of the predictor 3 in the prediction layer of the present invention;

[0041] Figure 10 This is a schematic diagram of the neural network training and image defect detection process for surface defect detection of industrial products as described in this invention.

[0042] Figure 11 This is an image of a defective printed circuit board used for defect detection in Embodiment 1 of the present invention;

[0043] Figure 12 This is a detection diagram of the network model of a defective printed circuit board image in Embodiment 1 of the present invention. Detailed Implementation

[0044] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0045] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0046] Figure 1 This is a schematic diagram of the network structure of the single-stage target detection neural network based on multi-scale fusion described in this invention. Figure 2 This is a structural diagram of the feature extraction network layer in the single-stage target detection neural network based on multi-scale fusion described in this invention. Figure 3 This is a structural diagram of the feature extractor AA1 in the feature extraction network layer described in this invention. Figure 4 This is a structural diagram of the feature extractor AA2 described in this invention. Figure 5 This is a structural diagram of the feature extractor AA3 described in this invention; Figure 6 This is a structural diagram of the multi-scale fusion layer in the single-stage target detection neural network based on multi-scale fusion described in this invention; Figure 7This is a structural diagram of predictor 1 in the prediction layer of the present invention. Figure 8 This is a structural diagram of the predictor 2 described in this invention. Figure 9 10 is a structural diagram of the predictor 3 described in this invention; 10 is a schematic diagram of the neural network training and image defect detection process for surface defect detection of industrial products described in this invention. See also Figures 1-9 A single-stage target detection neural network based on multi-scale fusion includes: an image preprocessing layer, a feature extraction network layer, a connection layer, a multi-scale fusion layer, and a prediction network layer. The image preprocessing layer is used for image data augmentation, and the feature extraction network layer is used for extracting image features. The connection layer connects the feature extraction layer and the multi-scale fusion layer through a convolutional layer 13. The multi-scale fusion network is used to fuse the image features, and the prediction network layer is used to generate prediction boxes with various sampling rate features based on the fused features output by the multi-scale fusion network. Taking a defective industrial product surface image as an example, combined with the technical solution provided by this invention and the aforementioned industrial product surface defect detection model, image data acquisition, neural network construction, network training, and model building can be performed to address image defect detection problems in related fields, solving the defect detection problem of industrial product surface images. The network of this invention solves the problem of information loss of image feature information during network transmission due to excessive neural network layers, including problems such as surface defect classification, defect localization, and quality detection. The embodiments provided by this invention are implemented based on the structure diagram 1-10, the control flowchart, and the technical solution provided by this invention. Specific embodiments are as follows:

[0047] Example 1

[0048] See appendix Figures 1-10 And the technical solution provided by this invention.

[0049] Step 1: Construct a dataset that includes surface defects of industrial products to be inspected.

[0050] Taking the defect detection application of printed circuit board images as an example, a printed circuit board image dataset is obtained. The image dataset D of the multi-scale fusion type single-stage target detection neural network is constructed, then cropped into 640×640 sub-images, and the images are binarized to avoid lighting interference. Then, one random operation is performed on the images: "rotate counterclockwise by 90 degrees," "flip left and right," "flip up and down," "flip left and right and then rotate counterclockwise by 90 degrees," or "flip up and down and then rotate counterclockwise by 90 degrees," to obtain the image dataset. The image dataset is then divided (70% is randomly selected from the industrial product surface defect dataset as the training set, 10% as the test set, and 20% as the validation set), into training set D-1, validation set D-2, and test set D-3, and the defect image data is labeled. The training set is used to train the model, the validation set is used to prevent overfitting, and the test set is used to test the model performance. This embodiment utilizes Python and OpenCV to perform image normalization processing, normalizing the image size to 640*640 pixels and performing image binarization. Specifically, it uses the rot90, fliplr, and flipud algorithms and their combinations from the NumPy library in Python. For defect image annotation, LabelImage annotation software is used to annotate the images, and the generated TXT file is used for training the YOLO_4t network in this embodiment.

[0051] Step 2: Use the dataset to train and build the YOLO_4t neural network to obtain the optimized network model MM.

[0052] According to the technical solution of the present invention, a single-stage target detection neural network YOLO_4t based on multi-scale fusion is constructed.

[0053] Based on the features of the industrial product images to be detected, the number of targets to be detected is preset (the number of centroids in the YOLO-4t network model), and the number and size of the detection prior boxes are obtained on the training set D-1 using the K-means clustering algorithm; specifically, the process includes the following:

[0054] According to the K-means algorithm, the surface defect image dataset is divided into K groups, and K initial points are randomly selected as the centroids of the clusters. The calculation method is to calculate the IOU distance between each sample point and the centroid, and assign each sample point to the nearest cluster centroid. The centroid of each cluster and the assigned sample points form a cluster. The centroid of each cluster is recalculated based on the existing sample points in the cluster. The above process is repeated until the centroid no longer changes, thereby finally determining the category and centroid of each sample point.

[0055] The YOLO-4t network model has K=9 centroids, and the IOU is expressed as follows:

[0056]

[0057] The distance d between the sample and the centroid is

[0058] d(box,centroid)=1-IOU(box,centroid) (2)

[0059] In formula (1), area(candidate bound) represents the area of ​​the candidate bounding box generated by the grid, area(ground truth bound) represents the area of ​​the ground truth bounding box, box represents the bounding box, and centroid represents the cluster centroid.

[0060] Select the YOLO_4t neural network and input the prior box data obtained above into the YOLO_4t network cfg configuration file; the configuration of the cfg configuration file includes the following steps:

[0061] Step (1): Select the YOLO_4t network structure;

[0062] Step (2): Input the prior bounding box data of the defect dataset into the network selected in step (1), and set the relevant parameters of the selected network; preferably, the default parameters of the YOLO_4t network are set as follows: cumulative batch size 64, segment size 64, learning rate 0.00261, momentum parameter 0.9, weight regularization decay 0.0005, and number of iterations 30000;

[0063] Step (3): Evaluate the training of the network described in step (1) using the CIOU loss function of YOLO-tiny. When the CIOU loss function value is less than 0.1 and stabilizes, end the training of the network.

[0064] Step (4): After step (3) is completed, the network automatically saves the best weights during the training process, which are used to calculate the accuracy of the test set.

[0065] Then, the images in the training set D-1 are sequentially input into the neural network YOLO_4t. The convolutional kernels of the neural network are optimized based on the outputs of predictors 1, 2, and 3 to obtain the network model MM. A defective image PP-1 from the training set D-1 is input into the feature extraction network layer to obtain the output O-1 of feature extractor A. This output O-1 is then input into the convolutional layer A13 to obtain the input I-1 of the multi-scale fusion network layer. Input I-1 is then input into the multi-scale fusion layer to obtain the output O-2, which serves as the input I-2 of predictor 1. I-2 is then decoded by decoder 1 in predictor 1 using the size of the prior bounding box to obtain the network output O-3. Output O-3 and the true defect value are input into the loss function to optimize the backpropagation of the convolutional kernels of the object detection network model. Input I-1 is then input into the multi-scale fusion layer to obtain the output O-2, which serves as the input I-2 of predictor 1. I-2 is then decoded by the decoder 1 in predictor 1 using the size of the prior bounding box. The decoder 2 in predictor 2, combined with the size of the prior box, decodes to obtain the network output O-4; the output O-4 and the true value of the defect are input into the loss function to optimize the backpropagation of the convolution kernel of the target detection network model; repeat the above steps to train all corresponding images in the image data training set D-1, thus completing one training of the multi-scale fusion single-stage target detection neural network, and the training result is stored in the target detection neural network; perform multiple iterative training to obtain the multi-scale fusion single-stage target detection neural network model YOLO_4t_mod that can be used for printed circuit board defect detection; wherein predictor 1 can better detect larger target defects, predictor 3 can better detect smaller target defects, and predictor 2 can better detect medium-sized target defects.

[0066] The decoding formulas for predictor 1, predictor 2, and predictor 3 of the YOLO_4t network in this embodiment are as follows:

[0067]

[0068] Wherein, b in formula (3) x b y b is the center coordinate value of the predicted bounding box. w b h These are the width and height values ​​of the prediction box; c x c y The coordinates of the center point of the preset prior bounding box described in this invention are located in which cell of the feature map; δ is the Sigmoid function, which is the value of t. x t yCompressing to the [0, 1] interval ensures the target center is within the network unit performing the prediction, preventing excessive offset; e is the mathematical constant 2.718; t x t y t represents the center coordinate offset predicted by the predictor. w t h The width and height offsets predicted by the predictor; p w p h The width and height of the prior bounding box.

[0069] Finally, the printed circuit board images from the validation set D-2 are input into the optimized target detection neural network model MM to detect defects in the images, thus preventing overfitting of the network model MM.

[0070] Step 3: Acquire images of surface defects on industrial products to be detected, and train the YOLO_4t network to obtain the final network model YOLO_4t_mod.

[0071] Defective PCB images are selected from test set D-3. These images are then input into the network model MM. The output of the network model MM and the actual images are verified to obtain the verification accuracy. Obtaining the accuracy involves the following steps:

[0072] Step (1): After training, the MM model is used to verify the accuracy of the predicted test set samples as positive or negative samples. Samples belonging to a specific surface defect type of an industrial product are positive samples, and samples not belonging to the specified defect type are negative samples. TP represents the number of positive samples identified as positive samples, FP represents the number of negative samples identified as positive samples, and FN represents the number of positive samples identified as negative samples. After accuracy verification, the precision and recall of the test set predictions are calculated. The calculation formulas are as follows:

[0073]

[0074] Among them, precision represents the proportion of true positive samples out of the samples predicted as positive; recall represents the proportion of positive samples that are correctly predicted.

[0075] Step (2): Calculate the average accuracy (AP) of each type of sample. The average accuracy is the area enclosed by the precision-recall curve. The average AP of each type of sample is used to obtain the overall average accuracy (mAP) of the selected model.

[0076] When the verification accuracy reaches the expected target, the final network model YOLO_4t_mod is obtained, which is used to detect defects in the surface image of the industrial product to be inspected; when the verification accuracy is lower than the expected target, the parameters of the network model MM need to be adjusted, and step 3 is repeated until the verification accuracy reaches the expected target, and the final network model YOLO_4t_mod is obtained.

[0077] Step 4: Use the trained YOLO_4t_mod model to detect defects on the surface images of the industrial products to be inspected. See the image of the defective printed circuit board surface. Figure 11 Input into the YOLO_4t_mod model;

[0078] Step 5: Output Results. After processing by the YOLO_4t_mod model, the model can output all detected defect types, see... Figure 12 .

[0079] Example 2

[0080] See appendix Figures 1-10 And the technical solution provided by this invention.

[0081] Step 1: Construct a dataset that includes surface defects of industrial products to be inspected.

[0082] Taking the application of steel surface image defect detection as an example, an image dataset of steel surface defect images is obtained. An image dataset D based on a multi-scale fusion single-stage target detection neural network, as described in this invention, is constructed. All images are obtained from a linear scanning CCD. Then, the images are cropped into 640×640 sub-images, and binarized to avoid illumination interference. The cropped images are then subjected to one of the following random operations: "rotate counterclockwise by 90 degrees," "flip left and right," "flip up and down," "flip left and right and then rotate counterclockwise by 90 degrees," or "flip up and down and then rotate counterclockwise by 90 degrees," to obtain the image dataset. The image dataset is then divided (70% is randomly selected as the training set, 10% as the test set, and 20% as the validation set), resulting in training set D-1, validation set D-2, and test set D-3. The defect image data is labeled, where the training set is used to train the model, the validation set is used to prevent overfitting, and the test set is used to test the model's performance. This embodiment uses Python and OpenCV to implement image homogenization processing. Specifically, it uses the rot90, fliplr, and flipud algorithms and their combinations from the NumPy library in Python. The labeling of defective images is done using the LabelImage labeling software, and the generated TXT file is used for training the YOLO_4t network in this embodiment.

[0083] Step 2: Use the dataset to train and build the YOLO_4t neural network to obtain the optimized network model MM.

[0084] According to the technical solution of the present invention, a single-stage target detection neural network YOLO_4t based on multi-scale fusion is constructed.

[0085] Based on the characteristics of the industrial product images to be detected, the number of targets to be detected (the number of centroids of the YOLO-4t network model) is preset, and the number and size of the detection prior boxes are obtained on the training set D-1 by the K-means clustering algorithm; the specific process and operation involved are similar to the description of step 2 in Example 1.

[0086] Step 3: Acquire images of surface defects on industrial products to be detected, and train the YOLO_4t network to obtain the final network model YOLO_4t_mod.

[0087] Defective PCB images from test set D-3 are selected and input into the network model MM. The output of the network model MM and the actual images are then verified to obtain the verification accuracy. The method and steps for obtaining the accuracy are similar to those described in step 3 of Embodiment 1.

[0088] Step 4: Use the trained model YOLO_4t_mod to detect defects on the surface images of the industrial products to be inspected. Input the defect images of the steel surface into the YOLO_4t_mod model;

[0089] Step 5: Output Results. After processing by the YOLO_4t_mod model, the model can output all detected defect types.

[0090] This embodiment obtains a neural network model through training. Taking printed circuit board and steel surface defects as examples, this model can be used to detect defects in surface images of industrial products, including PCBs, but is not limited to printed circuit board and steel surface defects. This invention effectively solves the problem of defect classification and localization in industrial product surface defect images, and has great application potential and value in image content analysis, computer vision, and intelligent detection of image information quality of industrial products. The above description is only a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A single-stage target detection neural network based on multi-scale fusion, characterized in that, The network consists of an image preprocessing layer, a feature extraction network layer, a connection layer, a multi-scale fusion layer, and a prediction network layer. The image preprocessing layer has one input and one output, with the output connected to the input of the feature extraction network layer. The feature extraction network layer has one input and three outputs, with two of the three outputs being intermediate outputs and the other output connected to the input of the multi-scale fusion layer. The connection layer has one input and one output, with the output connected to the input of the multi-scale fusion layer. The multi-scale fusion layer, also with one input and one output, performs calculations based on the obtained input and optimizes the fusion network parameters based on the calculation results. The prediction network layer has three inputs and one output, with one input connected to the output of the multi-scale fusion layer and the other two inputs connected to the two intermediate outputs of the feature extraction network layer. The feature extraction network layer and the multi-scale fusion network layer transmit data through the connection layer, which is used to convolve the image output by the feature extraction network layer again. The image preprocessing layer consists of a data augmentation layer, a convolutional layer A-1, an instance normalization layer-1, a Leaky ReLU activation function layer, a convolutional layer A-2, an instance normalization layer-2, and a Leaky ReLU activation function layer connected in series, and is used for data augmentation of image data. The feature extraction network layer is composed of a feature extractor AA1, a max pooling layer M1, a feature extractor AA2, a max pooling layer M2, a feature extractor AA3, and a max pooling layer M3 connected in series, used to extract image features. Feature extractors AA2 and AA3 each contain an intermediate output, forming the two intermediate outputs of the feature extraction network layer. Feature extractor AA1 is composed of a convolutional layer A1, an instance normalization layer 1, a Leaky ReLU activation function layer, an operational layer 1, a convolutional layer A2, an instance normalization layer 2, a Leaky ReLU activation function layer, a convolutional layer A3, an instance normalization layer 3, a Leaky ReLU activation function layer, an operational layer 2, a convolutional layer A4, an instance normalization layer 4, a Leaky ReLU activation function layer, and an operational layer 3 connected in series. Feature extractor AA2 is composed of a convolutional layer A5, an instance normalization layer 5, a Leaky ReLU activation function layer, an operational layer 4, a convolutional layer A6, an instance normalization layer 6, and a Leaky ReLU activation function layer 3. The feature extractor AA2 is composed of a ReLU activation function layer, a convolutional layer A7, an instance normalization layer 7, a Leaky ReLU activation function layer, an operational layer 5, a convolutional layer A8, an instance normalization layer 8, a Leaky ReLU activation function layer, and an operational layer 6, connected in series. The output of the convolutional layer A8 of the feature extractor AA2, after passing through the instance normalization layer 8 and the Leaky ReLU activation function, is the intermediate output of the feature extractor AA2, which is connected to one input of the operational layer 15 within the predictor 3 of the prediction network layer. The feature extractor AA3 consists of a convolutional layer A9, an instance normalization layer 9, a Leaky ReLU activation function layer, an operational layer 7, a convolutional layer A10, an instance normalization layer 10, a Leaky ReLU activation function layer, a convolutional layer A11, an instance normalization layer 11, a Leaky ReLU activation function layer, an operational layer 8, a convolutional layer A12, an instance normalization layer 12, and a Leaky ReLU activation function layer 6. The ReLU activation function layer and the computation layer 9 are connected in series. The output of the convolutional layer A12 of the feature extractor AA3, after passing through the instance normalization layer 12 and the Leaky ReLU activation function layer, is the intermediate output of the feature extractor AA3, which is connected to one input of the computation layer 13 in the predictor 2 of the prediction network layer. The instance normalization layers -1, -2, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and 12 have the same structure and function. The Leaky ReLU activation function layer is constructed using a modified linear unit function with leakage for activation processing.The first operational layer divides the feature map of convolutional layer A1 into two equal groups and uses the first group as the input to convolutional layer A2, employing the route algorithm for grouping. The second operational layer performs a summation operation between the outputs of convolutional layer A2 and convolutional layer A3, using the add algorithm. The third operational layer performs a summation operation between the outputs of convolutional layer A1 and convolutional layer A4, also using the add algorithm. The fourth operational layer divides the feature map of convolutional layer A5 into two equal groups and uses the first group as the input to convolutional layer A6, employing the route algorithm for grouping. The fifth operational layer performs a summation operation between the outputs of convolutional layer A6 and convolutional layer A2. The summation operation between the outputs of A7 uses the add algorithm; the operation layer 6 performs the summation operation between the outputs of convolutional layers A5 and A8 using the add algorithm; the operation layer 7 divides the feature map of convolutional layer A9 into two groups and takes the first group as the input of convolutional layer A10, using the route algorithm; the operation layer 8 performs the summation operation between the outputs of convolutional layers A10 and A11 using the add algorithm; the operation layer 9 performs the summation operation between the outputs of convolutional layers A9 and A12 using the add algorithm. The connection layer is composed of a convolutional layer A13, a strength normalization layer 13, and a Leaky ReLU activation function layer connected in sequence; The multi-scale fusion layer consists of a max pooling layer M4, a computation layer 10, a max pooling layer M5, and a computation layer 11. The computation layer 10 performs cross-layer concatenation of the inputs of the multi-scale fusion layer as the input of the max pooling layer M5. The computation layer 11 performs the summation operation of the inputs of the multi-scale fusion layer and the outputs of the max pooling layers M4 and M5, using the add algorithm. The input of the max pooling layer M4 is the input of the multi-scale fusion layer, and the max pooling layers M4 and M5 have exactly the same structure and function. The prediction network layer contains three predictors: predictor 1, predictor 2, and predictor 3, which are used to process the fused features output by the multi-scale fusion network at different sampling rates. Predictor 1 is composed of convolutional layer A14, instance normalization layer 14, Leaky ReLU activation function layer, convolutional layer A15, instance normalization layer 15, Leaky ReLU activation function layer, convolutional layer A16, instance normalization layer 16, Leaky ReLU activation function layer, convolutional layer A17, Leaky ReLU activation function layer, and decoder Y1 connected in series. The input of predictor 1 is connected to the output of the multi-scale fusion layer, and the output of predictor 1 is a predicted bounding box of a 5x downsampled feature map. Predictor 2 is composed of computational layer 12, convolutional layer A18, instance normalization layer 17, Leaky ReLU activation function layer, and decoder Y1. The predictor 2 is constructed by sequentially connecting the ReLU activation function layer, upsampling layer U1, operation layer 13, convolutional layer A19, instance normalization layer 18, Leaky ReLU activation function layer, convolutional layer A20, instance normalization layer 19, Leaky ReLU activation function layer, convolutional layer A21, Leaky ReLU activation function layer, and decoder Y2. The input of predictor 2 is connected to the output of convolutional layer A16 of predictor 1, processed by instance normalization layer 16 and Leaky activation function layer. Predictor 2 outputs a predicted bounding box of the feature map with a 4x downsampling. One input of the operation layer 13 of predictor 2 is connected to the intermediate output of feature extractor AA3. Predictor 3 consists of operation layer 14, convolutional layer A22, instance normalization layer 20, Leaky ReLU activation function layer, upsampling layer U2, operation layer 15, convolutional layer A23, instance normalization layer 21, Leaky ReLU activation function layer, convolutional layer A24, and Leaky ReLU activation function layer. The ReLU activation function layer and decoder Y3 are connected in series. The input of predictor 3 is connected to the output of the convolutional layer A20 of predictor 2, which is processed by the instance normalization layer 19 and the Leaky activation function layer. The output of predictor 3 is the predicted bounding box of the feature map downsampled by 3 times. One input of the operation layer 15 of predictor 3 is connected to the intermediate output of feature extractor AA2. The operation layer 12 implements the cross-layer connection between predictor 1 and predictor 2, and its output is the input of convolutional layer A18. The operation layer 13 implements the summation operation between the intermediate output of feature extractor AA3 and the output of upsampling layer U1, and the summation operation uses the add algorithm. The operation layer 14 implements the cross-layer connection between predictor 2 and predictor 3, and its output is connected to the input of convolutional layer A22. The operation layer 15 implements the summation operation between the intermediate output of feature extractor AA2 and the output of upsampling layer U2, and the summation operation uses the add algorithm.The decoding formulas for decoders Y1, Y2, and Y3 are the same as those in the YOLO-v4 tiny network. The convolutional layers A-1, A-2, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, A21, A22, A23, and A24 have identical structures, and their construction parameters include the number of convolutional kernels, kernel normalization method, kernel size, kernel stride, kernel padding method, kernel bias term, and initial kernel value. The instance normalization layers 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, and 21 have the same structure and function, and are used to adjust the input data to a specific range.

2. The single-stage target detection neural network based on multi-scale fusion according to claim 1, characterized in that, The network described is built on the Darknet deep learning framework, and its loss function is the same as that of the YOLO-v4 tiny network.

3. The single-stage target detection neural network based on multi-scale fusion according to claim 2, characterized in that, The padding method of the padding layer PA in the neural network is same padding, the padding size is 1, and the padding method is symmetrical padding.

4. The single-stage target detection neural network based on multi-scale fusion according to claim 2, characterized in that, The filling method of the filling layer PB in the neural network adopts interpolation, with a filling size of 13 and a symmetrical padding method.

5. An industrial product surface defect detection model based on a single-stage target detection neural network with multi-scale fusion, characterized in that, It includes the following steps: Step 1: Establish the YOLO_4t single-stage target detection neural network based on multi-scale fusion as described in claim 1; Step 2: Obtain the image training set of the industrial products to be inspected, and standardize the length, width and height and image type of the acquired images; Step 3: Divide the image training set obtained in Step 2 into training set D-1, validation set D-2 and test set D-3; The training set D-1 is used to train the network model, the validation set D-2 is used to prevent the network model from overfitting, and the test set D-3 is used to test the performance of the network model. Step 4: Label and classify the images in the training set D-1, validation set D-2 and test set D-3 mentioned in Step 3, and organize them into YOLO dataset format; Step 5: Based on the features of the industrial product images to be detected in Step 2, preset the number of targets to be detected and obtain the number and size of detection prior boxes on the training set D-1 using the K-means clustering algorithm; Step 6: Select the neural network YOLO_4t described in Step 1, and input the prior box data obtained in Step 5 into the cfg configuration file of the neural network YOLO_4t; Step 7: Input the image data in the training set D-1 processed in Step 4 into the neural network YOLO_4t in sequence. Based on the outputs of predictor 1, predictor 2 and predictor 3 in the network YOLO_4t, optimize the series of convolution kernels in the neural network YOLO_4t using the loss function in the network YOLO_4t to obtain the optimized network model MM. Step 8: Input the defect images of the validation set D-2 processed in Step 4 into the network model MM described in Step 7 to perform defect detection on the images and prevent overfitting of the network model MM. Step 9: Input the images of the test set D-3 processed in Step 4 into the network model MM described in Step 8, and verify the output of the network model MM and the actual images to obtain the verification accuracy. Step 10: When the verification accuracy described in Step 9 reaches the expected target, the final network model YOLO_4t_mod is obtained and used to perform defect detection on the surface image of the industrial product to be inspected; when the verification accuracy described in Step 9 is lower than the expected target, the parameters of the network model MM need to be adjusted, and Steps 7-10 are repeated until the verification accuracy reaches the expected target and the final network model YOLO_4t_mod is obtained.

6. The industrial product surface defect detection model based on a multi-scale fusion single-stage target detection neural network according to claim 5, characterized in that, Step 2 uses Python and OpenCV to normalize the image size to 640*640 and perform image binarization. Step 4 uses LabelImage annotation software to annotate the image for defects, and the generated TXT file is used for training the YOLO_4t network.

Images