A lightweight network backlight panel defect detection method for industrial applications

Abstract

This invention relates to a lightweight network-based method for backlight panel defect detection in industrial scenarios, belonging to the field of backlight panel defect detection. The method includes: proposing a lightweight object detection model, SH-YOLO-Lite, which is an improvement on the YOLOv5-lites benchmark model. The SH-YOLO-Lite model employs a lightweight backbone network RASNet on the backbone network side and an Efficient-RepGFPN-Lite feature fusion network on the feature fusion network side. The SH-YOLO-Lite model is trained using a backlight panel defect dataset to obtain a backlight panel defect detection model. Defect detection is then performed on backlight panel images in actual industrial scenarios, and the backlight panel defect detection results are output. This invention enables low-cost deployment and efficient operation of backlight panel defect detection on existing low-computing-power embedded platforms.

Description

Technical Field

[0001] The present invention belongs to the field of backlight panel defect detection, and in particular relates to a backlight panel defect detection method of a lightweight network for industrial scenarios. Background Art

[0002] For the existing detection of backlight product defects, the defect detection method based on deep learning has gradually replaced the traditional detection method that relies on manpower, and realized the automatic detection of product defects. However, the general target detection model commonly used in this technology has too high a calculation amount and parameter quantity, and has strict requirements on the performance of hardware equipment. In addition, the real-time detection performance is limited, which is difficult to meet actual industrial needs. Therefore, it is difficult to directly apply it to low-performance devices such as embedded platforms and mobile devices in actual industrial scenarios. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the prior art and provide a lightweight network backlight panel defect detection method for industrial scenarios, so as to achieve low-cost deployment and efficient operation of backlight panel defect detection on existing low-computing power embedded platforms.

[0004] To achieve the above purpose, the technical solution of the present invention is: a lightweight network backlight panel defect detection method for industrial scenarios, comprising:

[0005] A lightweight target detection model SH-YOLO-Lite is proposed. The lightweight target detection model SH-YOLO-Lite is improved based on the YOLOv5-lites benchmark model. The lightweight backbone network RASNet is used on the backbone network side, and the feature fusion network Efficient-RepGFPN-Lite is used on the feature fusion network side.

[0006] The lightweight target detection model SH-YOLO-Lite is trained using the backlight panel defect dataset to obtain the backlight panel defect detection model. The defect detection is then performed on the backlight panel images to be detected in actual industrial scenes, and the backlight panel defect detection results are output.

[0007] In one embodiment of the present invention, the lightweight backbone network RASNet is constructed as follows:

[0008] By introducing structural reparameterization technology and channel cross operation, the network module Shuffle Block of the original backbone network in YOLOv5-lites is improved to propose a lightweight network module RAS Block, and a lightweight backbone network RASNet is constructed by stacking the lightweight network module RAS Block and the downsampling module in the lightweight network ShuffleNetV2.

[0009] In one embodiment of the present invention, the network structure of the lightweight backbone network RASNet is as follows:

[0010] First, the CBRM layer is used as the first layer of the network. The CBRM layer is a combination operation layer of convolution, batch normalization, ReLU activation function, and maximum pooling.

[0011] Next, the lightweight network module RAS Block is used as the basic unit, and the network body is built by cross-repeatedly stacking RAS Block and the downsampling module in ShuffleNet V2.

[0012] In one embodiment of the present invention, the lightweight network module RAS Block is constructed as follows:

[0013] (1) The grouping ratio parameter is introduced, and the segmentation ratio r is used as a variable to adjust the number of feature channels input to the feature extraction branch. By setting the parameter r to 0.5 and 0.25 respectively, two lightweight target detection models SH-YOLO-Lite of different scales can be obtained: SH-YOLO-Litel and SH-YOLO-Lites.

[0014] (2) Introduce channel crossover operation and set it in the middle position of the lightweight network module RAS Block. By fusing the new feature map output by the feature extraction branch with part of the old feature map of the input module, establish the information interaction mechanism of the middle layer of the lightweight network module RAS Block;

[0015] (3) The 3×3 depth-wise convolution in the feature extraction branch is structurally reparameterized, that is, the 3×3 depth-wise convolution is replaced by a 3×3 Mobileone Block convolution module containing a 3×3 depth-wise convolution. The 3×3 Mobileone Block convolution module introduces a reparameterizable branch with only a jump connection with a BN layer and a branch containing a 1×1 convolution layer and a BN layer based on the depth-wise separable convolution structure, and introduces a hyperparameter K to adjust the number of branches of the depth-wise separable convolution branch, 1≤K≤5. During training of the 3×3 Mobileone Block convolution module, when the value of K is 2 or above, the 3×3 Mobileone Block convolution module will have multiple parallel and repeated depth-wise separable convolution branches. During inference of the 3×3 Mobileone Block convolution module, the various branches of the 3×3 Mobileone Block convolution module will be merged into one feature extraction branch.

[0016] In one embodiment of the present invention, in step (3), the specific process of structural reparameterization is as follows:

[0017] 1) Unify the branches of the 3×3 Mobileone Block convolution module into a unified 3×3 convolution form, as follows:

[0018] First, the 1×1 convolution in the branch containing the 1×1 convolution layer and the BN layer is reconstructed into an equivalent 3×3 convolution, that is, the 1×1 convolution is padded with zeros to expand the convolution kernel size to 3×3, thus obtaining a 3×3 convolution layer.

[0019] Next, an equivalent 3×3 convolution is reconstructed in the shortcut branch, i.e., the branch with only the skip connection of the BN layer, to obtain a 3×3 convolution layer.

[0020] Finally, the 3×3 convolutional layers and BN layers of each branch are fused to obtain the fused 3×3 convolutional layer; the weight calculation formula of the fused 3×3 convolutional layer is as follows:

[0021]

[0022] in, , are the weight parameters and bias parameters of the i-th channel of the input feature map, is the mean of the i-th channel of the input feature map, is the variance of the i-th channel of the input feature map, and are the learning parameters of the i-th channel of the input feature map, and It is obtained by statistics during the training process of the 3×3 Mobileone Block convolution module. and It is learned during the training process of the 3×3 Mobileone Block convolution module;

[0023] 2) Add the fused branches together to form a feature extraction branch.

[0024] In one embodiment of the present invention, the feature fusion network Efficient-RepGFPN-Lite is constructed as follows:

[0025] Introduce a lightweight convolution module GSConv that integrates ordinary convolution, depthwise separable convolution, and channel shuffle operations to replace the convolution module of the original feature fusion network in YOLOv5-lites;

[0026] A weighted feature fusion method is introduced to perform differentiated fusion of different input feature maps;

[0027] The self-designed ESP attention module is inserted near the input of the original feature fusion network Efficient-RepGFPN in YOLOv5-lites, and finally the feature fusion network Efficient-RepGFPN-Lite is constructed.

[0028] In one embodiment of the present invention, the self-designed ESP attention module is constructed by introducing the ESP attention mechanism of image segmentation and coordinate attention mechanism. The ESP attention mechanism includes an upper branch and a lower branch. The specific implementation steps are as follows:

[0029] (1) Segmentation: When the input feature map is input to the lower branch, the lower branch uses a sliding window of size K×K to segment the input feature map into K×K regional feature maps. At this time, the size of each regional feature map is C×K×K, where C is the channel of the input feature map and K is the size of the sliding window used to segment the feature map.

[0030] (2) Expansion and attention operation: When the input feature map is input to the upper branch, the 1×1 convolution will expand the channel of the input feature map to K 2 , generate attention filters through the coordinate attention mechanism;

[0031] (3) Dot product: Perform Hadamard product operation on the output of the upper branch and the lower branch, and broadcast C times; the dot product operation between the attention filter output by the upper branch and the regional feature map output by the lower branch is equivalent to firstly expanding the attention filter in an orderly manner in the channel dimension and dividing the regional feature map into C groups according to the channel order, and then multiplying the attention filter with the regional feature map of each group, that is, multiplying the elements in the same position of the two, and finally outputting the weighted feature map;

[0032] (4) Feature map addition: The size of the dot multiplication operation is K 2 ×C×H×W feature maps are added together, and then added to the input feature map to obtain an output feature map with the same shape as the input feature map.

[0033] In an embodiment of the present invention, the weighted feature fusion method is a fast normalization fusion method.

[0034] Compared with the prior art, the present invention has the following beneficial effects: the method of the present invention proposes a lightweight target detection model SH-YOLO-Lite, which is improved based on YOLOv5-lites as the benchmark model, adopts a lightweight backbone network RASNet on the backbone network side, and adopts a feature fusion network Efficient-RepGFPN-Lite on the feature fusion network side. Compared with the existing algorithm model, it has a lower backlight panel defect missed detection rate and better overall detection performance, and can realize low-cost deployment and efficient operation of backlight panel defect detection on the existing low-computing power embedded platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is the SH-YOLO-Lite structure of a lightweight target detection model proposed in an example of the present invention.

[0036] Figure 2 This is the network structure of RASNet, an example of the present invention.

[0037] Figure 3 This is an example RAS Block structure of the present invention.

[0038] Figure 4 The following is a flow chart of implementing channel crossover operation according to an example of the present invention.

[0039] Figure 5 This is a 3×3 Mobileone Block convolution module according to an example of the present invention.

[0040] Figure 6 This is a structural reparameterization process of an example of the present invention.

[0041] Figure 7 This is a schematic diagram of the reconstruction process of each branch of the 3×3 Mobileone Block convolution module in an example of the present invention; wherein (a) is the reconstruction of a 1×1 convolution into a 3×3 convolution; and (b) is the reconstruction of a shortcut branch into a 3×3 convolution.

[0042] Figure 8 This is an example of the ESP attention mechanism of the present invention.

[0043] Fig. 9 This is a dot product operation in the ESP attention mechanism of an example of the present invention.

[0044] Fig.10 This is the network structure of Efficient-RepGFPN-Lite, an example of the present invention.

[0045] Fig.11 This is an example GSConv of the present invention.

[0046] Fig.12This is the inference result of the self-built backlight panel defect dataset of an example of the present invention; where (a) is the label; (b) is the SH-YOLO-Lite l ; (c) SH-YOLO-Lite s .

[0047] Fig.13 This is a model visualization result of backlight panel defect detection in an example of the present invention; wherein (a) is a label; (b) is the optimized model No. 4; and (c) is the optimized model No. 8.

[0048] Fig.14 The structure of an industrial defect detection system based on the NVIDIA Jetson AGX Orin embedded platform is an example of the present invention.

[0049] Fig.15 An industrial defect detection system based on the NVIDIA Jetson AGX Orin embedded platform is an example of the present invention. DETAILED DESCRIPTION

[0050] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0051] The present invention proposes a lightweight network backlight panel defect detection method for industrial scenarios, comprising:

[0052] A lightweight target detection model SH-YOLO-Lite is proposed. The lightweight target detection model SH-YOLO-Lite is improved based on the YOLOv5-lites benchmark model. The lightweight backbone network RASNet is used on the backbone network side, and the feature fusion network Efficient-RepGFPN-Lite is used on the feature fusion network side.

[0053] The lightweight target detection model SH-YOLO-Lite is trained using the backlight panel defect dataset to obtain the backlight panel defect detection model. The defect detection is then performed on the backlight panel images to be detected in actual industrial scenes, and the backlight panel defect detection results are output.

[0054] The following is the specific implementation process of the present invention.

[0055] Aiming at the actual industrial scenarios, hardware resource configuration and operating environment, this paper proposes a new lightweight target detection algorithm for industrial scenarios, SH-YOLO-Lite, to achieve low-cost deployment and efficient operation of the target detection algorithm on the existing low-computing power embedded platform. SH-YOLO-Lite uses YOLOv5-lites as the benchmark model for algorithm improvement. Its algorithm structure is as follows: Figure 1As shown in the figure. On the backbone network side, SH-YOLO-Lite improves the network module Shuffle Block in the original backbone network by introducing structural reparameterization technology and channel crossover operation to propose a new lightweight network module - Reparameterized Augmented ShuffleBlock (RAS Block), and constructs a new lightweight backbone network - RASNet by stacking RAS Block and Shuffle Block as a downsampling module in ShuffleNetV2. On the feature fusion network side, SH-YOLO-Lite designs a new lightweight attention mechanism, the ESP attention mechanism, for the backlight panel defect detection task. This mechanism enables the model to fully capture the local effective features of the image by introducing image segmentation and coordinate attention (CA) mechanism, thereby improving the accuracy of the detection model; in addition, SH-YOLO-Lite introduces lightweight convolution GSconv, self-designed ESP attention module and weighted feature fusion method (Weighted Conact) to improve the Efficient-RepGFPN feature fusion network, thereby proposing a neck network that is both lightweight and feature-rich, namely Efficient-RepGFPN-Lite, which solves the problem of excessive computational complexity and parameters of the feature fusion network.

[0056] 1. Backbone network - RASNet

[0057] The backbone network is responsible for extracting features from the input image, converting it into high-dimensional abstract semantic information, and outputting feature maps of multiple scales to the feature fusion network for further processing. This paper introduces the structural reparameterization technology and proposes a new lightweight backbone network, RASNet, based on the new reparameterization module RAS Block. The network structure of RASNet is as follows: Figure 2 As shown in Figure 1, the network first uses CBRM as the first layer of the network to replace the Focus layer in the original algorithm, effectively reducing the computing power burden brought by the Slice operation contained in the Focus layer, thereby achieving fast downsampling of the image. CBRM is a combined operation layer of convolution, batch normalization (BatchNormalization, BN), ReLU activation function, and maximum pooling. Then RASNet uses RAS Block as the basic unit, and builds the network body by cross-repeatedly stacking RAS Block and the downsampling module in ShuffleNet V2. The specific structure and parameter configuration are shown in Table 1.

[0058] Table 1

[0059]

[0060] The present invention improves the network module Shuffle Block in ShuffleNetV2 and introduces the structure reparameterization technology, and designs a structure reparameterized network module based on channel shuffle (Channel Shuffle) and channel crossover (Channel Crossover) - Reparameterized Augmented Shuffle Block (RAS Block), the structure of which is as follows Figure 3 The advantages of RAS Block are as follows:

[0061] (1) Introducing the Split Ratio parameter. By using the split ratio r as a variable, the number of feature channels input to the feature extraction branch can be flexibly adjusted to further control the efficiency of the entire network module in terms of computational cost, parameter consumption, and inference speed. The present invention sets the parameter r to 0.5 and 0.25 respectively, thereby obtaining two models of different scales: SH-YOLO-Lite l with SH-YOLO-Lite s .

[0062] (2) Introduce the channel crossover operation. This operation is set in the middle of the module. By fusing the new feature map output by the feature extraction branch with part of the old feature map of the input module, channel-level expansion is achieved without computational cost, thereby establishing an information interaction mechanism in the middle layer of the module, further promoting information exchange between feature channels. Its implementation process is as follows: Figure 4 As shown, the output feature channels from two different branches are first divided into two groups, then the channels in each group are rearranged and merged, and finally the two groups of feature channels constructed by merging are output to different branches respectively.

[0063] (3) Reparameterize the structure of the 3×3 depth-wise convolution (3×3 DWConv) in the feature extraction branch, that is, replace the original 3×3 depth-wise convolution with a 3×3 Mobileone Block convolution module containing a 3×3 depth-wise convolution. Figure 3 As shown in the figure, the 3×3 Mobileone Block convolution module uses a multi-branch structure during training to obtain higher training accuracy to compensate for the loss of lightweight design of the module; during inference, all branches are fused into one feature extraction branch, thereby effectively reducing the amount of calculation and parameters of the model while maintaining the same accuracy, and improving the inference speed of the model.

[0064] The 3×3 Mobileone Block convolution module containing 3×3 depth-wise convolution is as follows Figure 5 As shown in the figure, it introduces a reparameterizable branch of jump connections with only BN layers and a branch containing 1×1 convolution layers and BN layers based on the depthwise separable convolution structure, and introduces a hyperparameter K to adjust the number of branches of the depthwise separable convolution branch (K can be taken from 1 to 5); when the 3×3 Mobileone Block convolution module is trained, when K is taken as 2 or above, the 3×3 Mobileone Block convolution module will have multiple parallel and repeated depthwise separable convolution branches; when the 3×3 Mobileone Block convolution module is inferring, the various branches of the 3×3 Mobileone Block convolution module will be fused into a feature extraction branch, thereby improving the reasoning efficiency of the model.

[0065] The specific process of structural reparameterization is as follows: Figure 6 As shown in , each branch is first reconstructed into a unified 3×3 convolution form in order to realize the subsequent addition and fusion of each branch. The specific steps are: First, the 1×1 convolution in the 1×1 convolution branch is reconstructed into an equivalent 3×3 convolution, that is, Figure 7 As shown in (a), the original 1×1 convolution is padded with zeros to expand the size of each convolution kernel to 3×3. Then, an equivalent 3×3 convolution is reconstructed in the shortcut branch, that is, the following is constructed: Figure 7 The convolution matrix shown in (b) makes the input feature map consistent with the output feature map. Finally, the 3×3 convolutional layers of each branch are fused with the BN layer. The BN layer mainly contains 4 parameters: (mean), (variance), and ,in and It is obtained by statistics during the training process. and It is obtained through training and learning. The original formula of the BN layer is shown in formula (1). M No. i The calculation formula for batch normalization (BN) of channels is shown in formula (2). Since both convolution and BN operations are essentially linear operations on the input feature map, the two operations can be fused into one operation, that is, the convolution layer and the BN layer immediately following it are fused into a new convolution layer. The weight calculation formula of the new convolution layer after conversion is shown in formula (3), where and are the new weight parameters and bias parameters.

[0066] (1)

[0067] (2)

[0068] (3)

[0069] in, is a small constant used to prevent the denominator from being zero.

[0070] 2. ESP Attention Mechanism

[0071] Attention mechanisms are often used to improve the accuracy of deep neural networks. However, most attention mechanisms bring non-negligible computational overhead while improving model accuracy, making them difficult to apply to lightweight models. In addition, in the backlight panel defect detection task, since the target objects to be detected usually include tiny targets such as white spots and black spots, these targets are often concentrated in a local area of ​​the detection image. However, most existing attention mechanisms usually process the entire image directly, trying to learn local semantic information from the global semantic information of the image, and often ignore the regional spatial information in each feature channel, resulting in their limited effect on defect detection tasks.

[0072] In this regard, the present invention proposes a lightweight attention mechanism that is more suitable for backlight panel defect detection tasks and can effectively capture regional spatial information - the Efficient Segmentation-and-Product (ESP) attention mechanism. Aiming at the characteristics of the backlight panel defect detection task that the target is concentrated in the local area, this attention mechanism introduces image segmentation operations, divides the input feature map into multiple regional feature maps, and then performs attention operations on each segmented feature map separately, so that the model can directly and effectively extract spatial local information, significantly improving the model's ability to extract target feature information from each region. Ultimately, this mechanism enhances the detection accuracy of the target detection model in defect tasks, which can enable the model to focus on the area where the detection target is located in the image, and effectively distinguish the target information from the background interference information. Figure 8 shown.

[0073] The ESP attention mechanism first divides the input feature map into multiple regional feature maps with different spatial positions, then assigns different weights to each regional feature map, thereby screening out important target features in different spatial positions, and finally outputs the result through the feature map addition operation. Figure 8For example, the ESP attention mechanism module inputs a feature map of size C×W×H. The parameters in the figure are set to width W=3, height H=3, number of channels C=3, K=3, G=1, where K represents the size of the sliding window used to segment the feature map, and G represents the number of groups, that is, the number of slices and groups of the input feature map in the channel dimension. The mechanism will perform the same attention operation on each group of segmented feature maps. The specific implementation steps of the attention mechanism are as follows:

[0074] (1) Segmentation: When the input feature map is input into the lower branch, the lower branch uses a sliding window of size K×K to segment the input feature map into K×K regional feature maps. At this time, the size of each regional feature map is C×K×K.

[0075] (2) Expand and Attention Operation: When the feature map is input to the upper branch, the 1×1 convolution in the upper branch will expand its channel to K 2 , and finally the attention filter will be generated through the coordinate attention (CA) mechanism module. The present invention applies the coordinate attention mechanism to the ESP attention mechanism, and realizes the attention operation of each regional feature map representing different spatial positions with low computational overhead, so that the model can effectively capture the defect feature information concentrated in the local area.

[0076] (3) Product: Perform Hadamard product operation on the output of the upper branch and the lower branch, and broadcast C times. The dot product operation is equivalent to orderly expanding the filters of the channel dimension, and then performing product operation on the elements at the same position in the feature map to be weighted with the same number and size of attention filters. Fig. 9 As shown in the figure, the dot product operation between the attention filter (The Attention Filter) output by the upper branch and the segmented feature map (The Segmented Feature Maps) output by the lower branch can be equivalent to firstly expanding the attention filter in an orderly manner in the channel dimension and dividing the segmented feature map into C groups according to the channel order, and then multiplying the attention filter with the segmented feature map of each group, that is, multiplying the elements in the same position of the two, and finally outputting the weighted feature map (The Weighted Feature Maps), thereby realizing the screening of the feature map representing the characteristics of each region by the attention filter through the dot product operation.

[0077] (4) Feature map addition (Add): The size of the feature map generated by the previous dot multiplication operation is C×K 2The feature maps of ×H×W are added for normalization, and finally added to the input feature map to obtain an output feature map with the same shape as the input feature map.

[0078] The computation and parameter capacity of the ESP attention mechanism are limited by hyperparameters, and the size of the model can be adjusted according to actual needs. The scale of the parameters and FLOPs (floating point operations) of the mechanism mainly depends on the hyperparameters G (number of groups) and K (sliding window size). In network design, by adjusting G, K and the insertion position of the attention module, the size of the attention module can be effectively changed, thereby better balancing the contradiction between accuracy improvement and computational burden. This attention mechanism can achieve comprehensive and effective extraction of local image features while balancing accuracy, parameter capacity and computational burden. Therefore, compared with most other attention mechanisms, it is more suitable for lightweight object detection models.

[0079] 3. Feature Fusion Network——Efficient-RepGFPN-Lite

[0080] In the target detection task, feature fusion network is widely used to improve the performance of target detection model. It is usually used as the bottleneck stage of the model to fuse feature maps of different levels to obtain rich semantic information and positioning information. This paper proposes a more efficient feature fusion network, Efficient-RepGFPN-Lite, as the bottleneck stage of the detection model to meet the needs of lightweight target detection networks in industrial scenarios for efficient feature fusion networks. The overall structure of the network is as follows: Fig.10 As shown. Efficient-RepGFPN-Lite advantages:

[0081] (1) A lightweight convolution module GSconv is introduced to replace the original ordinary convolution module, thereby realizing efficient extraction of image features of different resolutions. The lightweight convolution module GSConv introduced in the present invention combines ordinary convolution, depthwise separable convolution and channel shuffle operations to achieve a feature extraction effect close to that of ordinary convolution at a lower computational cost, thereby effectively reducing the model burden while maintaining model performance.

[0082] like Fig.11 As shown in the figure, when the input feature map is input into the GSConv convolution, the channel transformation is first performed through ordinary convolution to output a feature map with a channel number of 2 / c2. Then, the depth-wise convolution (DWConv) is used to output a feature map with a channel number of 2 / c2. Finally, the two feature maps with the same number of channels are spliced ​​together and the information fusion between the feature channels is realized through channel shuffling operation.

[0083] In the target detection model, the input image will undergo a process of gradual transmission of spatial information to the channel in the backbone network, which is accompanied by spatial compression and channel expansion of the feature map, which will inevitably lead to partial loss of semantic information. In this regard, GSConv convolution will retain the hidden connections between each channel as much as possible to reduce the loss of semantic information, but if GSConv convolution is used in all stages of the target detection model, the network layer of the model will be deepened, which will significantly increase the reasoning time of the model. Therefore, the present invention only uses GSConv convolution in the bottleneck stage, because the input feature map in the bottleneck stage has less redundant and repeated information, and no further spatial compression is required.

[0084] (2) Introducing a weighted feature fusion method (Weighted Concat) - Fastnormalized fusion, which enables the network to learn the importance of different input features more efficiently and perform differentiated fusion of different input features. As shown in the following formula, similar to the Softmax operation, Fast normalized fusion will scale the value range of each normalized weight to between [0, 1], but it has a faster training speed.

[0085]

[0086] in, represents the output feature layer, represents the input feature layer, Represents the learnable weight coefficient corresponding to the input feature layer, each The Relu function will be applied afterwards to ensure that it is greater than or equal to 0; Equal to 0.0001, a small constant to avoid numerical instability.

[0087] (3) A self-designed lightweight attention mechanism, ESP attention mechanism, is inserted near the input of the feature fusion network to achieve efficient fusion of the channel information and spatial information of the input feature map, thereby effectively compensating for the accuracy loss caused by the lightweight design of the network.

[0088] In addition, the Efficient-RepGFPN-Lite feature fusion network follows the basic structure of the original Efficient-RepGFPN fusion module (Fusion Block). This module introduces the structure reparameterization technology and the connection form of efficient layer aggregation networks (ELAN), which effectively improves the model accuracy with a lighter computational burden. The core of the Fusion Block module is the CSPStage Block improved based on the CSPNet (Cross Stage Partial Network) network module. It compresses the input feature map from two 1×1 convolution features into two parts through partial connections and outputs them to two branches respectively. This design will effectively reduce the computational complexity of the module. One of the branches of the feature map input is a skip connection branch, which is directly spliced ​​with the output of the other branch, while the other branch is a feature extraction branch, which is mainly composed of N 3×3 convolutions and the residual combination of the structure reparameterization module Rep-Block. It realizes the full interaction of input features between different stages through skip connections between different stages, thereby improving the reusability of features, which will help improve the feature extraction ability of the target detection model. Moreover, such a structural design fully considers the flow path of features in the network, which not only avoids the occurrence of gradient vanishing or explosion, but also improves the generalization ability of the model, so that the model has good performance on general data sets and various defect data sets. The structural reparameterization module Rep-Block embedded in the feature extraction branch combines structural reparameterization technology, obtains higher training accuracy in the form of dual branches in the training stage, and combines the two branches into one through reparameterization technology in the reasoning stage, thereby maintaining high detection accuracy while improving the reasoning efficiency of the model, and ultimately improving the balance between performance and complexity of the model.

[0089] 4. Experimental results verification

[0090] The present invention conducts a performance comparison experiment on the backlight panel defect dataset. The experimental results are shown in Table 2. The SH-YOLO-Lite proposed in the present invention achieves good performance. l with SH-YOLO-Lite s Compared with YOLOv5n of the same size, the mAP@0.5 and mAP@.5:0.95 accuracies are improved by 3.4%, 3.8% and 2.2%, 4.6% respectively.

[0091] Table 2

[0092]

[0093] Further statistics on SH-YOLO-Lite l SH-YOLO-Lite s The model’s detection performance for each type of target in the self-built backlight panel defect dataset is evaluated to achieve performance evaluation of the model on different detection targets. The experimental results are shown in Tables 3 and 4.

[0094] As shown in Table 3, SH-YOLO-lite l It demonstrates strong backlight panel surface defect detection capabilities. In particular, its detection accuracy for white clusters, which are scarce defect targets, is as high as 99.3%.

[0095] Table 3

[0096]

[0097] As shown in Table 4, SH-YOLO-lite s As a more lightweight algorithm, it is compared with SH-YOLO-Lite l , with a smaller model complexity, it shows higher AP@0.5 accuracy on three types of defect targets: white dots, black dots, and white clusters. The accuracy index has increased by 1.9%, 1%, and 0.2% respectively. s The AP@.5:0.95 accuracy of four types of defect targets, namely white dots, black dots, scratches, and white clusters, was improved by 3.8%, 1.3%, 2.2%, and 7.1%, respectively.

[0098] Table 4

[0099]

[0100] In order to intuitively demonstrate the defect detection capability of the proposed model in different application scenarios, the present invention conducted a visualization experiment on a self-built backlight panel defect dataset. The present invention randomly selected 16 images from the two datasets and used SH-YOLO-Lite l SH-YOLO-Lite s Two models are used for model inference.

[0101] The inference results of the self-built backlight panel defect dataset are as follows Fig.12As shown in the figure, the defect target in the selected image has the smallest size and the most dense distribution among the three defect data, so its detection is the most difficult. Compared with the original labels of the dataset, in this detection task, the three models all maintain good detection effects and can detect the main defect targets in the image well. l Thanks to its higher model complexity, it exhibits a lower missed detection rate and better overall detection performance compared to other algorithm models.

[0102] 5. Algorithm deployment and system design

[0103] In actual industrial detection scenarios, considering factors such as real-time performance and network transmission bandwidth, most industrial defect detection application scenarios require the target detection model to be deployed directly on embedded, mobile and edge devices to achieve local offline detection and edge processing. Therefore, the present invention uses the NVIDIA Jetson AGX Orin embedded platform for algorithm optimization and deployment to achieve edge detection of industrial defects, and evaluates the detection performance of the deployment platform. And design a corresponding industrial defect detection system.

[0104] In the algorithm optimization and deployment stage, the present invention will generate multiple different optimization models by selecting different deep learning reasoning frameworks and adjusting parameter settings. As shown in Table 5, these models can be divided into two groups, 1-4 and 5-8, representing SH-YOLO-Lite l SH-YOLO-Lite s The four optimized models generated under different frameworks and different parameter settings are the dynamic graph and static graph models output under the ONNX framework, and the models with weight precisions of FP32 and FP16 generated under the TensorRT framework.

[0105] As shown in Table 6, in the backlight defect detection task, the models 3, 4, 7, and 8 optimized by the TensorRT framework have significantly improved the detection speed while maintaining the same accuracy level compared to the models 1, 2, 5, and 6 optimized by the ONNX framework. Among them, the models 4 and 8 with a weight accuracy of FP16 in the optimization model using the TensorRT optimization framework have further improved the detection speed by 13.0% and 2.3% respectively while the accuracy remains basically unchanged compared to the models 3 and 7 with a weight accuracy of FP32. SH-YOLO-Lite with moderate model complexity l The algorithm’s optimized model No. 4 achieved the highest detection speed, reaching 68.5 FPS.

[0106] Table 5

[0107]

[0108] Table 6

[0109]

[0110] The model visualization results of backlight panel defect detection are as follows Fig.13 As shown in the figure, the overall detection task is quite challenging because the defective backlight panel to be detected is small in size and occupies a relatively low proportion of the image area. Compared with the original labels of the dataset, in this detection task, the models can basically accurately locate all target positions. Among them, the optimized model No. 8 shows better robustness and its missed detection rate is relatively lower.

[0111] After optimizing the lightweight object detection algorithm and deploying it on the NVIDIA Jetson AGX Orin embedded platform, this paper proposes a design scheme for an industrial defect detection system based on the NVIDIA Jetson AGX Orin embedded platform. The overall system structure is as follows: Fig.14 As shown, it can be divided into two parts: a defect collection unit and a defect processing unit.

[0112] The defect acquisition unit combines the light source system and the industrial camera, and together they undertake the key task of efficiently and accurately acquiring images of product surface defects. The design and configuration of the light source system is the core link to ensure the generation of high-quality, high-contrast images. It can eliminate the influence of shadows and glare by carefully adjusting the lighting conditions, and achieve sufficient illumination of the target area, thereby effectively highlighting the structure and characteristics of product surface defects. At the same time, the industrial camera, with its excellent image capture capabilities, records the surface condition of the product after optimized illumination by the light source system in real time, and transmits the captured defect images to the NVIDIA Jetson AGX Orin embedded platform for further processing. The collaborative operation between the two provides a solid and reliable data input source for subsequent image processing.

[0113] The defect detection unit is mainly composed of the NVIDIA Jetson AGX Orin embedded platform. Its task is to preprocess the images output by the industrial camera and complete the prediction of defect targets. First, the defect detection unit preprocesses the input image. It will use a single-scale adjustment strategy and combine the adaptive image scaling algorithm to adjust the input image to the scale required for detection. During the preprocessing process, when the industrial camera continuously collects multiple defect images, the defect detection unit can process multiple images at the same time, uniformly scale these images, and then input them in parallel into the detection model for detection. Subsequently, the defect detection unit inputs the preprocessed images into the detection model for processing. During the processing, the convolutional network on the backbone network side will extract features from the input image, convert the input image into high-dimensional abstract semantic information, and output feature maps of three scales to the feature fusion network side for further processing. Subsequently, the feature pyramid structure on the feature fusion side will perform feature fusion on the input feature map and also output feature maps of three scales. Finally, the output side of the detection model will perform target prediction on each feature map, and predict and output the confidence information, category information, and position (regression) information of the defect target respectively. The defect detection unit obtains the target information output by the detection model, which will use this information to draw a detection frame on the original input image, and feeds back the detection results and target information to the visualization terminal for staff to view.

[0114] As an important part of the detection system, the visualization terminal consists of input devices such as the main display and keyboard. It is directly connected to the industrial computer or embedded platform, and is designed to provide operators with an interactive interface for real-time observation of detection results and comprehensive control of the detection system. Specifically, the visualization terminal provides an intuitive interface that enables operators to easily view the results of defect detection and promptly discover and analyze possible problems. In addition, the visualization terminal is also responsible for summarizing and displaying the detection information obtained from each defect acquisition unit for comprehensive data analysis and decision making. At the same time, the visualization terminal also has the function of dynamically retrieving and replacing the model weights in the image processing unit to achieve flexible application and optimization updates of the system in different scenarios. Through these functions, the visualization terminal realizes effective monitoring and management of the entire system, providing staff with a convenient and efficient operation experience.

[0115] Based on the above solution, the present invention uses the existing backlight panel defect detection platform to build an industrial defect detection system based on the NVIDIA Jetson AGX Orin embedded platform. Fig.15As shown in the figure, the system is also divided into two parts: the defect collection unit and the defect processing unit. The defect collection unit consists of a backlight defect detection platform, which includes a turntable for placing the backlight and an industrial camera for collecting product images. In actual work, the backlight is first placed at position 2 of the turntable, then the turntable is rotated to position 3 for dust removal, and then to position 4 for image collection. During the collection process, the backlight is placed in a dark place that is opaque to light, and the image is collected by lighting the backlight. Finally, the backlight is transferred to position 4 for unloading. The defect processing unit is mainly composed of the NVIDIA Jetson AGX Orin embedded platform, which is responsible for processing and detecting product images taken by industrial cameras.

[0116] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A lightweight network-based backlight panel defect detection method for industrial scenarios, characterized in that: include: A lightweight target detection model SH-YOLO-Lite is proposed. The lightweight target detection model SH-YOLO-Lite is improved based on the YOLOv5-lites benchmark model. The lightweight backbone network RASNet is used on the backbone network side, and the feature fusion network Efficient-RepGFPN-Lite is used on the feature fusion network side. The lightweight target detection model SH-YOLO-Lite is trained using the backlight panel defect dataset to obtain a backlight panel defect detection model. The defect detection is then performed on the backlight panel images to be detected in actual industrial scenes, and the backlight panel defect detection results are output. The lightweight backbone network RASNet is constructed as follows: By introducing structural reparameterization technology and channel cross operation to improve the network module Shuffle Block of the original backbone network in YOLOv5-lites, a lightweight network module RAS Block is proposed, and a lightweight backbone network RASNet is constructed by stacking the lightweight network module RAS Block and the downsampling module in the lightweight network ShuffleNetV2; The network structure of the lightweight backbone network RASNet is as follows: First, the CBRM layer is used as the first layer of the network. The CBRM layer is a combination operation layer of convolution, batch normalization, ReLU activation function, and maximum pooling. Next, the lightweight network module RAS Block is used as the basic unit, and the network body is built by cross-repeatedly stacking RAS Block and the downsampling module in ShuffleNet V2; The lightweight network module RAS Block is constructed as follows: (1) The grouping ratio parameter is introduced, and the segmentation ratio r is used as a variable to adjust the number of feature channels input to the feature extraction branch. By setting the parameter r to 0.5 and 0.25 respectively, two lightweight target detection models SH-YOLO-Lite of different scales can be obtained: SH-YOLO-Litel and SH-YOLO-Lites. (2) Introduce channel crossover operation and set it in the middle position of the lightweight network module RAS Block. By fusing the new feature map output by the feature extraction branch with part of the old feature map of the input module, establish the information interaction mechanism of the middle layer of the lightweight network module RAS Block; (3) The 3×3 depth-wise convolution in the feature extraction branch is structurally reparameterized, that is, the 3×3 depth-wise convolution is replaced by a 3×3 Mobileone Block convolution module containing a 3×3 depth-wise convolution. The 3×3 MobileoneBlock convolution module introduces a reparameterizable branch with only a jump connection of a BN layer and a branch containing a 1×1 convolution layer and a BN layer based on the depth-wise separable convolution structure, and introduces a hyperparameter K' to adjust the number of branches of the depth-wise separable convolution branch, 1≤K'≤5. When training the 3×3 Mobileone Block convolution module, when K' is 2 or above, the 3×3 Mobileone Block convolution module will have multiple parallel and repeated depth-wise separable convolution branches. When inferring the 3×3 MobileoneBlock convolution module, the various branches of the 3×3 MobileoneBlock convolution module will be merged into one feature extraction branch. The feature fusion network Efficient-RepGFPN-Lite is constructed as follows: Introduce a lightweight convolution module GSConv that integrates ordinary convolution, depthwise separable convolution, and channel shuffle operations to replace the convolution module of the original feature fusion network in YOLOv5-lites; A weighted feature fusion method is introduced to perform differentiated fusion of different input feature maps; Insert the self-designed ESP attention module near the input of the original feature fusion network Efficient-RepGFPN in YOLOv5-lites, and finally construct the feature fusion network Efficient-RepGFPN-Lite; The self-designed ESP attention module is constructed by introducing the ESP attention mechanism of image segmentation and coordinate attention mechanism. The ESP attention mechanism includes an upper branch and a lower branch. The specific implementation steps are as follows: (1) Segmentation: When the input feature map is input to the lower branch, the lower branch uses a sliding window of size K×K to segment the input feature map into K×K regional feature maps. At this time, the size of each regional feature map is C×K×K, where C is the channel of the input feature map and K is the size of the sliding window used to segment the feature map. (2) Expansion and attention operation: When the input feature map is input to the upper branch, the 1×1 convolution will expand the channel of the input feature map to K 2 , generate attention filters through the coordinate attention mechanism; (3) Dot product: Perform Hadamard product operation on the output of the upper branch and the lower branch, and broadcast C times; the dot product operation between the attention filter output by the upper branch and the regional feature map output by the lower branch is equivalent to firstly expanding the attention filter in an orderly manner in the channel dimension and dividing the regional feature map into C groups according to the channel order, and then multiplying the attention filter with the regional feature map of each group, that is, multiplying the elements in the same position of the two, and finally outputting the weighted feature map; (4) Feature map addition: The size of the dot multiplication operation is K 2 The feature maps of ×C×H×W are added and then added to the input feature map to obtain an output feature map with the same shape as the input feature map.

2. According to the lightweight network backlight panel defect detection method for industrial scenarios according to claim 1, it is characterized in that: In step (3), the specific process of structural reparameterization is as follows: 1) Unify the branches of the 3×3 Mobileone Block convolution module into a 3×3 convolution form, as follows: First, the 1×1 convolution in the branch containing the 1×1 convolution layer and the BN layer is reconstructed into an equivalent 3×3 convolution, that is, the 1×1 convolution is padded with zeros to expand the convolution kernel size to 3×3, thus obtaining a 3×3 convolution layer. Next, an equivalent 3×3 convolution is reconstructed in the shortcut branch, i.e., the branch with only the skip connection of the BN layer, to obtain a 3×3 convolution layer. Finally, the 3×3 convolutional layers and BN layers of each branch are fused to obtain the fused 3×3 convolutional layer; the weight calculation formula of the fused 3×3 convolutional layer is as follows: ； in, , is the weight parameter and bias parameter of the i-th channel of the input feature map, is the mean of the i-th channel of the input feature map, is the variance of the i-th channel of the input feature map, and are the learning parameters of the i-th channel of the input feature map, and It is obtained by statistics during the training process of the 3×3 Mobileone Block convolution module. and It is learned during the training process of the 3×3 Mobileone Block convolution module; 2) Add the fused branches together to form a feature extraction branch.

3. According to the lightweight network backlight panel defect detection method for industrial scenarios according to claim 1, it is characterized in that: The weighted feature fusion method is a fast normalization fusion method.

Images