A single board defect recognition method and system based on deep learning

By adopting deep learning methods in veneer defect detection, combined with YOLO network and feature fusion module, the problem of low defect detection efficiency and accuracy in veneer production is solved, and efficient identification of small-scale and similar color defects is achieved, which improves the Detection accuracy.

CN117853783BActive Publication Date: 2024-11-12UNIV OF JINAN
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202311714362.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-12
Publication Date
2024-11-12
Estimated Expiration
2043-12-12

AI Technical Summary

Technical Problem

During the production process of veneer, defect detection efficiency and low accuracy, especially small-scale defects and defects with similar colors are difficult to accurately identify, resulting in frequent missed detection.

Method used

Using a single-board defect detection method based on deep learning, the YOLO network architecture combines the small-scale defect feature extraction module and the feature fusion module to increase the attention module for adaptive weighting, and combines the multi-scale Retinex theory for image enhancement, extract and fusion features. Improve model recognition capabilities.

Benefits of technology

It significantly improves the ability to identify small-scale defects, enhances the recognition accuracy of color similarity defects, reduces the leakage detection rate, and improves the accuracy and efficiency of surface defect detection of veneer boards.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117853783B_ABST
    Figure CN117853783B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of machine vision technology, and discloses a single board defect detection method and system based on deep learning, the method comprising: obtaining a single board image to be detected; performing defect detection on the single board image based on a pre-trained single board defect detection model; wherein the single board defect detection model is based on a YOLO network architecture, comprising a backbone network, a bottleneck network and a detection network connected in sequence; a small-scale defect feature extraction module is added to the input end of the backbone network, and a feature fusion module is added to the bottleneck network, which is used to fuse the output of the small-scale defect feature extraction module with the terminal output of the original bottleneck network, and output it to the detection network. The present invention can accurately identify multiple defects on single board images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology, and in particular relates to a single board defect recognition method and system based on deep learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Veneer is a thin wood chip made from logs by peeling or slicing, with a thickness of 1-3mm. Due to the special processing technology of veneer and the uncertainty factors in the growth process of trees, the surface quality of the veneer peeled on the veneer peeling machine is uneven, which in turn affects the quality of subsequent plywood products.

[0004] At present, in the veneer production process, although most companies have set up veneer defect detection links, most of them use manual visual inspection to complete veneer quality inspection. As a major branch in the field of deep learning, target detection algorithms can automatically detect the type and position of targets in images, providing new solutions for industrial defect detection. In the field of wood processing, there are detection schemes for defects such as knots and holes on the surface of fir boards, and cracks, scars, and mildew on red pine and larch wood. However, due to the special processing technology and material characteristics of veneers, the surface conditions are complex. The defects that affect the mechanical properties of veneers include live knots, semi-live knots, dead knots, needle knots, cracks, etc. The color similarity of various defects is high, which is not conducive to the automatic discrimination of the model. Moreover, the shapes and scales of various defects vary greatly. If the scale of the surface defects of the veneer is too small, the small defects contain less characteristic information, and the defects are easily integrated with the background, resulting in missed detection of defects. Summary of the invention

[0005] In order to solve the problems of low single board defect detection efficiency and low detection accuracy in the single board production process, the present invention provides a single board defect recognition method and system based on deep learning.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] A single board defect detection method based on deep learning, comprising:

[0008] Obtaining the image of the board to be detected;

[0009] Based on a pre-trained single board defect detection model, performing defect detection on the single board image;

[0010] Among them, the single board defect detection model is based on the YOLO network architecture, including a backbone network, a bottleneck network and a detection network connected in sequence; a small-scale defect feature extraction module is added to the input end of the backbone network, and a feature fusion module is added to the bottleneck network to fuse the output of the small-scale defect feature extraction module with the terminal output of the original bottleneck network, and output it to the detection network.

[0011] In some embodiments, the backbone network includes multiple layers of feature extraction modules, and an attention module is connected to the end of each layer of feature extraction modules.

[0012] In some embodiments, the small-scale defect feature extraction module includes two CBS modules, a Ghost-CSP module and an attention module connected in sequence.

[0013] In some embodiments, the multi-layer feature extraction module includes, from shallow to deep, a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module; in the bottleneck network, the output of the fourth feature extraction module is upsampled and fused with the output of the third feature extraction module to obtain a first fused feature, the first fused feature is processed and fused with the output of the second feature extraction module to obtain a second fused feature, the second fused feature is processed and fused with the output of the first feature extraction module to obtain a third fused feature, and the second fused feature and the third fused feature are both output to the detection network after processing.

[0014] In some embodiments, the second fusion feature is further fused with the first fusion feature after processing to obtain a fourth fusion feature, and the fourth fusion feature is output to the detection network after processing; the fourth fusion feature is fused after processing to obtain a fifth fusion feature, and the fifth fusion feature is output to the detection network after processing.

[0015] In some embodiments, after acquiring the image of the single board to be detected, image enhancement is also performed:

[0016] Convert the single-board image from RGB color space to HSV color space;

[0017] The V channel is decomposed into illumination component and reflection component with the help of multi-scale Retinex theory;

[0018] The exponential parameter value of the adaptive gamma correction is calculated according to the brightness of the illumination component, and the V channel and S channel of the image are processed.

[0019] A second aspect of the present invention provides a single board defect detection system based on deep learning, comprising:

[0020] An image acquisition module is configured to acquire an image of a board to be detected;

[0021] A defect detection module is configured to perform defect detection on the single board image based on a pre-trained single board defect detection model;

[0022] Among them, the single board defect detection model is based on the YOLO network architecture, including a backbone network, a bottleneck network and a detection network connected in sequence; a small-scale defect feature extraction module is added to the input end of the backbone network, and a feature fusion module is added to the bottleneck network to fuse the output of the small-scale defect feature extraction module with the terminal output of the original bottleneck network, and output it to the detection network.

[0023] The third aspect of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the single board defect detection method based on deep learning when executing the program.

[0024] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the single board defect detection method based on deep learning.

[0025] The fifth aspect of the present invention provides a single board defect detection system based on deep learning, including an image acquisition system and a host computer, wherein the image acquisition system is used to acquire single board images and send them to the host computer, and the host computer is configured to execute the single board defect detection method based on deep learning.

[0026] One or more of the above technical solutions have the following beneficial effects:

[0027] According to the scale differences of various defects of single plates, the present invention expands the feature extraction module for small-scale defects, and adds a related network design that is integrated with the output of the feature extraction module, effectively improving the model's recognition ability for small-scale defect features.

[0028] In order to solve the problem that the color differences of various defects on single boards are not obvious, attention modules are added to the feature detection modules at all levels to adaptively weight the red, green and blue channels of feature maps of different scales, fully exploring the color differences of various defects, making the color features more obvious, and providing a guarantee for the subsequent identification of defects with similar colors. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0030] Figure 1(a) is a schematic diagram of a dead knot defect in a single board;

[0031] Figure 1(b) is a schematic diagram of a half-slip knot defect in a single plate;

[0032] Figure 1(c) is a schematic diagram of a live joint defect;

[0033] Figure 1(d) is a schematic diagram of a crack defect;

[0034] Figure 1(e) is a schematic diagram of a needle node defect;

[0035] Figure 2 A diagram of the architecture of a single board defect detection model in one or more embodiments of the present invention;

[0036] Figure 3 It is an overall flow chart of single board defect detection in one or more embodiments of the present invention;

[0037] Figure 4 A single-board image enhancement flow chart in one or more embodiments of the present invention;

[0038] Figure 5 The figure is a comparison chart of mAP test results of one or more embodiments of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in one or more embodiments of the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0040] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0041] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0042] The size of the veneer involved in one or more embodiments of the present application is 1270mm×850mm. In order to clarify the type of surface defects and grade assessment standards of veneers, the veneer defect index is formulated with reference to the national standard for veneers (GB / T13010-2020). In the veneer grading process, in addition to requiring the surface defects of the veneers to meet some requirements such as plate grading and allowable plate defects in GB / T13010-2020, it is also necessary to accurately measure and analyze the surface defects of the veneers to ensure the mechanical properties and appearance quality of the veneers. In the veneer grade assessment work, the mechanical properties of the veneer are an important reference factor in the veneer grading. For the mechanical properties of the veneer, there are two main factors to consider, one is the number and area of ​​node defects, and the other is the cracking phenomenon that may occur in the veneer during processing and storage. Therefore, in one or more embodiments of the present application, live knots, semi-live knots, dead knots, needle knots, and cracks that affect the mechanical properties of the veneer are studied, and the defect image is shown in Figure 1.

[0043]

Establishment of single board surface defect dataset

[0044] Establish a veneer surface defect dataset. Collect veneer images, perform data enhancement and annotation on the veneer images, and produce a veneer surface defect dataset. According to actual production needs, the dataset contains five types of defects in veneers: live knots, dead knots, semi-live knots, cracks, and needle knots. After data cleaning, the collected veneer defect data totals 2,000. In order to improve the generalization ability of the model, the ImgAug data enhancement strategy is also adopted to expand the data to 6,000. Subsequently, the image dataset is divided into a training set, a validation set, and a test set at a ratio of 8:2:1. After the veneer image data is expanded, it is necessary to perform defect annotation on the veneer image. One or more embodiments of the present invention use LabelImg software to annotate the enhanced veneer image, select the defects in the image, and accurately annotate each type of defect. The annotation information mainly includes the location information and category information of the defect.

[0045]

Building a single board defect detection model

[0046] The single board defect detection model in one or more embodiments of the present invention is improved based on the YOLOv5s model.

[0047] like Figure 2 As shown, the single board defect detection model includes a backbone network, a bottleneck network and a detection network connected in sequence; a small-scale defect feature extraction module is added to the input end of the backbone network, and a feature fusion module is added to the bottleneck network to fuse the output of the small-scale defect feature extraction module with the terminal output of the original bottleneck network, and output it to the detection network.

[0048] Specifically, the backbone network includes multiple layers of feature extraction modules, and the feature extraction modules corresponding to each layer include at least one CBS module, CSP module and attention module connected in sequence; the deepest feature extraction module is also connected to the SPPF module between the CSP module and the attention module. Specifically, the CSP module adopts the Ghost-CSP module, that is, the convolution layer in the CSP module adopts Ghost convolution, which can achieve lightweight network. In the multi-layer feature extraction module, the output end of the Ghost-CSP module of the upper layer is connected to the input end of the CBS of the next layer.

[0049] Specifically, in order to improve the detection accuracy of small target defects on a single board, the size of the feature map output by the shallowest layer of the multi-layer feature extraction module is determined in combination with the scale characteristics of various defects, so as to avoid the loss of small-scale defect features caused by the reduction of the feature map during the downsampling process, and facilitate the subsequent capture of small target defects. As a specific implementation method, the feature extraction module includes 4 layers, the feature map size of the shallowest layer output is 160×160, and the other layers are 80×80, 40×40, and 20×20 respectively, which can effectively reduce the impact of defect scale changes on model performance. From shallow to deep, they are recorded as the first feature extraction module, the second feature extraction module, the third feature extraction module, and the fourth feature extraction module.

[0050] Among them, the first feature extraction module is an additional small-scale defect feature extraction module, which includes more than two CBS modules, Ghost-CSP modules and attention modules connected in sequence. In order to avoid the omission of small-scale defects, more than two CBS modules are set in the feature extraction module, and there are two in the figure. The purpose is to capture more subtle features through multiple convolutions and activations to ensure the recognition ability of small-scale defects.

[0051] For single board defects, the traditional YOLOv5s network architecture can only distinguish defects with large scale differences, and due to the similarity in color, it is difficult to distinguish defects with no obvious scale differences but similar colors, and related interference textures will inevitably appear on the image. In order to analyze the color characteristics of single board surface defects, the RGB pixel mean of live knots, dead knots, semi-live knots, cracks, and pin knots in the data set was extracted, and a three-dimensional scatter plot was drawn with the RGB parameter value of the image as the X-axis, Y-axis, and Z-axis to visualize the color characteristics of the defects. It was found that the RGB features of the five types of defects were highly similar. Specifically, the red and blue color features of live knots, cracks, and pin knots are highly similar; the red and green features of dead knots, semi-live knots, pin knots, and cracks are highly similar; the yellow-green features of these five types of defects are highly similar. In order to better distinguish the color features of defects on single boards, in some embodiments, an attention module is set at the end of each hierarchical feature detection module. Specifically, it is a CBAM attention module. The CBAM attention mechanism can learn the importance weights of different channels in the feature map by adaptively weighting the channel dimension, thereby improving the model's ability to distinguish color features and reducing the interference of noise and irrelevant information on the model.

[0052] As a specific implementation, the feature extraction module includes 4 layers, the first feature extraction module includes a first CBS module, a second CBS module, a first Ghost-CSP module and a first CBAM attention module connected in sequence, the second feature extraction module includes a third CBS module, a second Ghost-CSP module and a second CBAM attention module connected in sequence, the third feature extraction module includes a fourth CBS module, a third Ghost-CSP module and a third CBAM attention module connected in sequence, and the fourth feature extraction module includes a fifth CBS module, a fourth Ghost-CSP module, an SPPF module and a fourth CBAM attention module connected in sequence. And wherein, the output end of the first Ghost-CSP module is connected to the third CBS module, the output end of the second Ghost-CSP module is connected to the fourth CBS module, and the output end of the third Ghost-CSP module is connected to the fifth CBS module.

[0053] The bottleneck network includes multiple CBS modules, multiple CSP2_X modules, multiple upsampling modules (i.e., US in the figure) and multiple fusion modules, wherein the CSP2_X module includes 2*X CBL modules. The specific architecture is as follows: the output end of the fourth feature extraction module is sequentially connected to the sixth CBS module and the first upsampling module, the output of the first upsampling module is fused with the output of the third feature extraction module to obtain a first fused feature as the input of the first CSP2_X module, the output end of the first CSP2_X module is sequentially connected to the seventh CBS module and the second upsampling module, the output of the second upsampling module is fused with the output of the second feature extraction module to obtain a second fused feature, which is respectively used as the input of the third CSP2_X module and the fourth CSP2_X module, wherein the output end of the third CSP2_X module is sequentially connected to the eighth CBS module and the third upsampling layer, the output of the third upsampling layer is fused with the output of the first feature extraction module to obtain the first fused feature. The three fusion features are used as the input of the fifth CSP2_X module, and the output of the fifth CSP2_X module is connected to the detection network; the output end of the fourth CSP module is divided into two paths, one path is connected to the detection network, and the other path is connected to the ninth CBS module. The output of the ninth CBS module is fused with the first fusion feature to obtain the fourth fusion feature; the output end of the fourth fusion feature is connected to the fifth CSP2_X module, and the output end of the fifth CSP module is divided into two paths, one path is connected to the detection network, and the other path is connected to the tenth CBS module. The output of the tenth CBS module is fused to obtain the fifth fusion feature. The output end of the fifth fusion feature is connected to the sixth CSP2_X module, and the output end of the sixth CSP2_X module is connected to the detection network.

[0054] It is worth emphasizing that the output feature obtained by processing the fourth fusion feature through the tenth CBS module can be fused with the output of the fourth feature extraction module, or can be fused based on the output feature, instead of not fusing with other features. Based on this, the feature map of the largest scale obtained can also retain the characteristics of defects at various scales as much as possible, further improving the model's ability to recognize small-scale defects.

[0055] The fusion mentioned above can be achieved by concat splicing fusion.

[0056] At this point, the bottleneck network has four output ends connected to the detection network. The detection network processes the feature maps of the four scales output by the bottleneck network and uses the idea based on anchors to achieve the final defect location and classification. The detection network divides the feature map at each detection scale into S×S grids of the same size, and convolves the divided feature maps to obtain the predicted box and category information of the target. The predicted box contains the location information of the target and consists of three values: center coordinates, width, and height. It is obtained by decoding the offset and scale factor predicted for each position in the feature map; for each target box, the detection network also outputs a category probability vector, in which each element represents the probability of a category. Finally, based on the predicted location and size information and the predicted category probability, the confidence score of each target box is calculated. The product of the confidence score and the category probability score is usually used as the final score, and the target box with a score higher than a certain threshold is output as the final detection result.

[0057] Compared with the traditional YOLOv5s model, the hierarchical structure of the feature extraction module is expanded based on the scale characteristics of various defects, and a smaller-scale feature extraction module is added to better capture the characteristics of small defects; on the basis of expanding the feature extraction module, a corresponding upsampling channel is added at the end of the bottleneck network, the 80×80 feature map is upsampled to 160×160, and the expanded feature map is fused and spliced ​​with the shallow 160×160 feature map output by the backbone network. Finally, additional small-scale detection channels are added to the detection network to better capture the information of small targets. Through the above improvements, the characteristics of small defects can be effectively retained, and the newly added detection channel has a smaller receptive field, which can effectively improve the model's detection ability to identify small-scale defects.

[0058]

Single board defect detection model training

[0059] Aiming at the problem of uneven brightness of single-board images, a single-board surface enhancement method based on Retinex theory is proposed to enhance the single-board images in the data set. Figure 4 As shown, the method includes:

[0060] (1) Convert the single-board image from RGB color space to HSV color space;

[0061] (2) Decompose the V channel into illumination component and reflection component with the help of multi-scale Retinex theory;

[0062] (3) The exponential parameter value of the adaptive gamma correction is calculated according to the brightness of the illumination component, and the V channel and S channel of the image are processed to achieve single-board image enhancement processing.

[0063] The above-mentioned image enhancement method can enhance the dark area image in a targeted manner according to the actual situation of the single board surface, improve the brightness of the dark area image, restore the defect color information and detail information, and at the same time adaptively enhance the normal brightness area in the image to avoid affecting the detail information and color information of the image due to excessive brightness.

[0064] The training of deep learning models requires massive calculations, and the requirements for computers are relatively high. One or more embodiments of the present invention use GPU to accelerate the training of models for training and testing. The graphics card model is NVIDIA Tesla P40, and the video memory is 24GB. The CPU is Intel (R) Xeon (R) CPU E5-2650v4 @ 2.20GHz. The software platform used in the experiment is based on Python 3.8 and Pycharm, and the experimental model is built using the open source framework Pytorch.

[0065] In order to objectively evaluate the performance of the single board surface defect detection model, five evaluation indicators including precision, recall, average precision, parameter amount and inference speed are set to comprehensively evaluate the model performance.

[0066] (1) Precision: It indicates the ratio of true positive classes among all the classes judged as positive classes in target detection. The calculation formula of precision is as follows. Taking the surface defects of single board as an example, TP indicates the number of marked single board defects judged as single board defects, and FP indicates the number of non-single board defects judged as single board defects.

[0067]

[0068] (2) Recall (FN) is as follows, where FN represents the number of annotated board defects that have not been detected.

[0069]

[0070] (3) Mean of Average Precision (mAP): The usual target detection research involves the detection of multiple categories. For each category, the model will calculate its corresponding average precision separately. The average precision of all categories in the data set is added together to get the mean, which is the mean. The mean is an important indicator for evaluating the target detection effect. The higher the value, the better the detection effect of the model. mAP is as follows, where ClassPrecision represents the detection precision of each category, and N represents the total categories of all defects in the data set.

[0071]

[0072] (4) Parameter: It is used to evaluate the complexity and size of the neural network. It refers to the total number of adjustable parameters that need to be trained in the network model. The main factor affecting the number of model parameters is the structure of the neural network. Generally speaking, the higher the number of model parameters, the stronger the model's representation ability is, and the better the accuracy performance can be obtained; the number of parameters is inversely proportional to the model's inference speed.

[0073] (5) Inference speed (FPS): FPS is used to describe the inference speed of the defect model in actual work. In the single board surface defect detection task, FPS is the number of frames of images processed by the defect detection model per second. The higher the FPS, the higher the processing efficiency of the model.

[0074] Ablation experiment refers to the process of controlling or excluding certain conditions or variables in experimental research to verify the effect of a certain variable in the entire experimental system. One or more embodiments of the present invention use ablation experiments to verify the contribution of the CBAM module, multi-scale detection network and Ghost-CSP structure to the single-board surface defect detection model. In order to verify the effectiveness of the improvement strategy, one or more embodiments of the present invention conducted 6 groups of ablation experiments. The experimental results are shown in Table 1. One or more embodiments of the present invention conduct an in-depth analysis of the improvement strategy proposed in one or more embodiments of the present invention from two perspectives: the contribution of the detection accuracy and the contribution of the detection efficiency of the analysis model.

[0075] Table 1 Ablation experiment results

[0076]

[0077] According to the data in Table 1, we can analyze the impact of each module on the detection accuracy of the model. From Experiments 1 and 2, it can be concluded that after adding a multi-scale detection network to the YOLOv5s model alone, the precision, recall and mAP of the model increased by 0.5%, 1.7% and 1.9% respectively. From Experiments 1 and 3, it can be concluded that after adding the CBAM attention module to the YOLOv5s model, the precision, recall and mAP of the model increased by 2.7%, 2.6% and 1.8% respectively. From Experiments 1 and 4, it can be concluded that after the YOLOv5s model is lightweight, the precision, recall and mAP of the model do not show a significant decrease. From Experiments 1 and 5, it can be concluded that after adding the CBAM attention module and the multi-scale detection network to the YOLOv5s model, the precision, recall and mAP of the model increased by 2.1%, 6.5% and 4.0% respectively. It is concluded from Experiments 1 and 6 that after adding the CBAM attention module, the multi-scale detection network and replacing Ghost-CSP to the YOLOv5s model, the precision, recall and mAP of the model increased by 1.4%, 6.1% and 3.5% respectively.

[0078] According to the data in Table 1, we can analyze the impact of each module on the parameter scale of the model. From Experiments 1, 2, and 3, it can be concluded that after adding the multi-scale detection network and CBAM attention mechanism to the YOLOv5s model, the model parameters increased by 9.415% and 0.467% respectively. From Experiments 1 and 5, it can be concluded that after adding the multi-scale detection network and CBAM attention mechanism to the YOLOv5s model at the same time, the model parameters increased by 12.25%. From Experiments 1 and 4, it can be concluded that the parameters of the YOLOv5s model after lightweight processing decreased by 30.271%. From Experiments 5 and 6, it can be concluded that the model after adding the CBAM attention module and the multi-scale detection network is lightweight, and the parameters of the model after lightweighting are reduced by 29.421%, which is 20.775% lower than the original YOLOv5s model parameters.

[0079] In summary, the CBAM attention module and multi-scale detection network proposed in one or more embodiments of the present invention can effectively improve the precision, recall rate and mAP of the model in single-board surface defect detection; in order to address the problem of increased detection efficiency parameters after adding relevant modules, one or more embodiments of the present invention propose a lightweight strategy based on Ghost-CSP, which can help reduce the number of parameters of the single-board surface defect detection model by 29.421%; compared with the original YOLOv5s model, the number of parameters of the single-board surface defect detection model proposed in one or more embodiments of the present invention is reduced by 20.775%.

[0080] mAP is a comprehensive performance indicator of the defect detection model in multi-category defect detection tasks, which can be used to judge the comprehensive performance of the defect detection model for each category in multi-category detection tasks. In order to verify the impact of the module proposed in this chapter on the training process, based on the above experiments, the mAP curve in the model training process is output, and one or more embodiments of the present invention set the original YOLO model, the CBAM-YOLO model with the CBAM attention mechanism added, the Detect-YOLO model with the multi-scale detection channel added, the Detect-CYOLO with the CBAM attention mechanism added and the multi-scale detection channel added, and the single board surface defect detection model (Detect-GCYOLO) proposed in this application, and conduct control experiments.

[0081] The mAP image comparison of the model during training is as follows Figure 5As shown in the figure, after 50 rounds of training, the network models have reached the fitting state, among which the peak mAP of the YOLO network model is 0.928, and the peak mAPs of the CBAM-YOLO and Detect-YOLO network models are close, 0.946 and 0.947 respectively. The two improved schemes are 1.9% and 1.8% higher than the peak mAP of the YOLO network model respectively. The Detect-CYOLO network model performs best, with a peak mAP of 0.968, which is 4% higher than that of YOLO and about 1.8% higher than that of the CBAM-YOLO and Detect-YOLO network models; the peak accuracy of the Detect-GCYOLO model is 0.963, which is 3.5% higher than the peak accuracy of the YOLO model.

[0082] After analyzing the data generated during the model training process, it can be concluded that the Detect-GCYOLO model converges faster during the training process; at the same time, in the single board surface defect detection task, the Detect-GCYOLO model has a more stable accuracy performance than the original YOLOv5s model.

[0083] In order to verify the advantages of the single-board surface defect detection model, the current mainstream target detection model is used to conduct a comparative experiment with the single-board surface defect detection model. YOLOv4 was proposed by Alexey Bochkovskiy in 2020. YOLOv4 is currently widely used in the industrial field due to its high detection accuracy and detection efficiency; YOLOv5x is a version in the YOLOv5 series. Compared with YOLOv5s, it has a larger model scale and higher detection accuracy. One or more embodiments of the present invention use YOLOv4, YOLOv5x and the single-board defect detection model (Detect-GCYOLO) proposed in this application for comparative experiments, and comprehensively evaluate the model's mAP, FPS and detection accuracy for each category of defects. The three models are trained separately using the same training parameters and training environment, and the models are comprehensively evaluated.

[0084] The comparative experimental results are shown in Table 2. The comparative results show that the single board defect detection model has the highest detection accuracy. In terms of accuracy, the mAP of the single board defect detection model is 96.3%, which is 7.2% higher than the mAP of YOLOv4 and 0.5% higher than the mAP of YOLOv5x model. The single board surface defect detection model proposed in one or more embodiments of the present invention is a lightweight model. In terms of detection speed, the FPS of the single board surface defect detection model is 12.5, which is 75% higher than the FPS of YOLOv4 and about 438.793% higher than the FPS of YOLOv5x. At the same time, compared with the YOLOv4 and YOLOv5x models, the single board defect detection model has the highest overall detection accuracy for live joints, dead joints, semi-live joints, cracks, and pin joints, which are 96.6%, 96.8%, 96.1%, 95.8%, and 96.2% respectively.

[0085] Table 2 Model comparison experiment

[0086]

[0087]

[0088] The confidence score indicates the possibility that the predicted object box contains the target. The confidence value range is between 0 and 1. The closer the confidence is to 1, the better the model detection effect is; the closer the confidence is to 0, the more unstable the detection effect is. Experimental verification shows that the detection confidence of the YOLOv5x model for half-joint and pin-joint defects is 0.99 and 0.51 respectively; the detection confidence of the YOLOv4 model for half-joint and pin-joint defects is 0.97 and 0.18 respectively, and YOLOv4 has false detection phenomenon; the single board surface defect detection model proposed in one or more embodiments of the present invention has the highest confidence of 0.99 and 0.98 for half-joint and pin-joint defects. The confidence levels of the YOLOv5x model for half-joints and joint defects are 0.94 and 0.91 respectively; the detection confidence levels of the YOLOv4 model for half-joints and joint defects are 0.66 and 0.39 respectively; the detection confidence levels of the single board surface defect detection model for half-joints and joint defects are 0.97 and 0.95 respectively.

[0089] In summary, compared with other mainstream target detection models, the single board surface defect detection model proposed in one or more embodiments of the present invention performs better in detecting small single board surface defects and defects with similar color features.

[0090]

Application of single board defect detection model

[0091] Based on the above single board defect detection model, one or more embodiments of the present invention provide a single board defect detection method based on deep learning, comprising the following steps:

[0092] Step 1: Obtain the image of the board to be detected;

[0093] Step 2: Based on the pre-trained single board defect detection model, defect detection is performed on the single board image.

[0094] One or more embodiments of the present invention further provide a single board defect detection system based on deep learning, comprising:

[0095] A data acquisition module is configured to acquire an image of a board to be detected;

[0096] The defect detection module is configured to perform defect detection on the single board image based on a pre-trained single board defect detection model.

[0097] One or more embodiments of the present invention also provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a single board defect detection method based on deep learning when executing the program.

[0098] In some embodiments, the single board defect detection method based on deep learning can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program can be loaded and / or installed into an electronic device via a RAM and / or a ROM and / or a communication unit. When the computer program is loaded into the RAM and / or the ROM and executed by the electronic device, one or more steps of the method described above can be performed.

[0099] One or more embodiments of the present invention also provide a single board defect detection system based on deep learning, including an image acquisition system and a control system. The image acquisition system uses machine vision technology to collect single boards on the production line in real time; the control system uses automation technology to cooperate with the image acquisition system and the defect detection system to realize the automatic detection of single board surface defects, and uses a rule engine to realize the real-time evaluation of single board grade.

[0100] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A single board defect detection method based on deep learning, characterized in that: include: Obtaining the image of the board to be detected; Based on a pre-trained single board defect detection model, performing defect detection on the single board image; The single board defect detection model is based on the YOLO network architecture, including a backbone network, a bottleneck network and a detection network connected in sequence; a small-scale defect feature extraction module is added to the input end of the backbone network, and a feature fusion module is added to the bottleneck network to fuse the output of the small-scale defect feature extraction module with the terminal output of the original bottleneck network, and output it to the detection network; The backbone network includes multiple layers of feature extraction modules, and each layer of feature extraction modules is connected to an attention module at the end; The small-scale defect feature extraction module includes more than two CBS modules, a Ghost-CSP module and an attention module connected in sequence; The bottleneck network has four output terminals connected to the detection network. The detection network processes the feature maps of the four scales output by the bottleneck network and realizes the final defect location and classification based on the idea of ​​Anchor, specifically: The detection network divides the feature map at each detection scale into S×S grids of the same size, and convolves the divided feature map to obtain the predicted box and category information of the target; the predicted box contains the location information of the target, consisting of three values: center coordinates, width and height, and is obtained by decoding the offset and scale factor predicted for each position in the feature map; for each target box, the detection network also outputs a category probability vector, in which each element represents the probability of a category; finally, based on the predicted position and size information and the predicted category probability, the confidence score of each target box is calculated, and the product of the confidence score and the category probability score is used as the final score. The target box with a score higher than a certain threshold is output as the final detection result.

2. The single board defect detection method based on deep learning according to claim 1, characterized in that: The multi-layer feature extraction module includes, from shallow to deep, a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module; in the bottleneck network, the output of the fourth feature extraction module is upsampled and fused with the output of the third feature extraction module to obtain a first fused feature, the first fused feature is processed and fused with the output of the second feature extraction module to obtain a second fused feature, the second fused feature is processed and fused with the output of the first feature extraction module to obtain a third fused feature, and the second fused feature and the third fused feature are processed and output to the detection network.

3. The single board defect detection method based on deep learning according to claim 2, characterized in that: After being processed, the second fused feature is further fused with the first fused feature to obtain a fourth fused feature, and the fourth fused feature is output to the detection network after being processed; the fourth fused feature is fused after being processed to obtain a fifth fused feature, and the fifth fused feature is output to the detection network after being processed.

4. The single board defect detection method based on deep learning according to claim 1, characterized in that: After obtaining the image of the board to be detected, image enhancement is also performed: Convert the single-board image from RGB color space to HSV color space; The V channel is decomposed into illumination component and reflection component by using multi-scale Retinex theory; The exponential parameter value of the adaptive gamma correction is calculated according to the brightness of the illumination component, and the V channel and S channel of the image are processed.

5. A single board defect detection system based on deep learning, characterized in that: include: An image acquisition module is configured to acquire an image of a board to be detected; A defect detection module is configured to perform defect detection on the single board image based on a pre-trained single board defect detection model; The single board defect detection model is based on the YOLO network architecture, including a backbone network, a bottleneck network and a detection network connected in sequence; a small-scale defect feature extraction module is added to the input end of the backbone network, and a feature fusion module is added to the bottleneck network to fuse the output of the small-scale defect feature extraction module with the terminal output of the original bottleneck network, and output it to the detection network; The backbone network includes multiple layers of feature extraction modules, and each layer of feature extraction modules is connected to an attention module at the end; The small-scale defect feature extraction module includes more than two CBS modules, a Ghost-CSP module and an attention module connected in sequence; The bottleneck network has four output terminals connected to the detection network. The detection network processes the feature maps of the four scales output by the bottleneck network and realizes the final defect location and classification based on the idea of ​​Anchor, specifically: The detection network divides the feature map at each detection scale into S×S grids of the same size, and convolves the divided feature map to obtain the predicted box and category information of the target; the predicted box contains the location information of the target, consisting of three values: center coordinates, width and height, and is obtained by decoding the offset and scale factor predicted for each position in the feature map; for each target box, the detection network also outputs a category probability vector, in which each element represents the probability of a category; finally, based on the predicted position and size information and the predicted category probability, the confidence score of each target box is calculated, and the product of the confidence score and the category probability score is used as the final score. The target box with a score higher than a certain threshold is output as the final detection result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the single board defect detection method based on deep learning as described in any one of claims 1-4 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a single board defect detection method based on deep learning as described in any one of claims 1 to 4 is implemented.

8. A single board defect detection system based on deep learning, characterized in that: It includes an image acquisition system and a host computer, wherein the image acquisition system is used to acquire single board images and send them to the host computer, and the host computer is configured to execute the single board defect detection method based on deep learning as described in any one of claims 1-4.

Citation Information

Patent Citations

  • Single low-illumination image enhancement method and system based on Retinex and convolutional neural network

    CN113129236A

  • YOLOX-based steel plate surface defect detection method, device and equipment

    CN116071338A

  • PCB defect detection method based on improved YOLOv5 and storage medium

    CN116245817A

  • Storage battery defect detection method and system based on image deep learning

    CN116385402A

  • Steel surface defect detection method and system and computer equipment

    CN116664558A