A method for surface defect detection of malleable cast iron pipe fittings oriented to edge computing

Abstract

The application relates to a kind of surface defect detection methods of edge computing-oriented malleable steel pipe fittings, which comprises the following steps: collecting multiple malleable steel pipe surface defect images; the defects on the collected malleable steel pipe surface defect images are labeled to obtain a dataset; 80% of the dataset is used as a training set, and 20% is used as a test set; an expanded compression residual bottleneck network is constructed, one of the feature extraction branches of the network adopts a convolution strategy of expanding first and then compressing, which is used to extract the features of the internal defects of the malleable steel pipe on the input feature map I and suppress the background noise information; the other feature extraction branch adopts a convolution strategy of compressing first and then expanding, which is used to extract the features of the edge contour of the malleable steel pipe on the input feature map I; after the two feature extraction branches output the feature maps extracted by features respectively, an attention mechanism is introduced to obtain an expanded compression feature extraction backbone network fused with the attention mechanism; the method has high detection accuracy, low computational complexity and low calculation cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automated defect detection, in particular to a method for surface defect detection of malleable iron pipe fittings based on edge computing. BACKGROUND

[0002] Malleable iron pipe fittings, also known as malleable iron pipe fittings, have been widely used in fire water supply, life heating and gas supply pipe network systems due to their excellent wear resistance, impact resistance and ductility. However, surface defects such as sand holes, depressions, defects, flash, ridge and spots will inevitably occur during the production process of malleable iron pipe fittings, which will affect the subsequent production and processing cycle. Even the missed detection of some small defect pipe fittings will cause unpredictable safety hazards to the actual pipe network system. Figure 1-1

[0003] In the actual pipe surface defect detection process, nearly three-quarters of the workers in the entire factory are employed to check product quality. Manual visual inspection is the most common detection method, but this method not only has low detection efficiency, high labor intensity, high false detection rate and high missed detection rate, but also has high labor cost and is easily affected by subjective factors of workers. It is worth noting that when the defect size of malleable iron pipe fittings is less than 0.5mm and there is no large optical deformation, the human eye cannot determine the defect characteristics, which is not suitable for the needs of large-scale malleable iron pipe production. At present, the defect detection method based on convolutional neural network has been widely studied in the academic field. This method can identify multiple defects and reduce labor costs after a certain amount of defect data sample debugging, but in actual application, this type of defect detection model has a complex network structure, large parameter quantity, high computational complexity, slow detection speed and high requirements for operation units. Therefore, it is of great significance to design a method that can detect surface defects of malleable iron pipe fittings in real time on edge devices.

[0004] Convolutional neural network (CNN) is widely used in complex industrial environments with great differences in object shape, size, texture, color, background, layout and imaging light due to its powerful feature extraction capability. Such network can directly calculate the category, positioning result and category confidence of the defect object in the input image, avoiding the complex process of manually designing feature extractor and parameter debugging in traditional detection algorithms.

[0005] ​Because of the heavy quality inspection quantity of malleable iron (MI) in the detection process, directly using the CNN network in the defect detection field as a feature extractor often cannot meet the requirements in detection speed. Therefore, many studies have made lightweight improvements to the CNN network structure for surface defect detection. The existing lightweight CNN network structure methods can be divided into three categories: detection methods based on lightweight modules, neural architecture search (NAS), and detection methods based on using various techniques to compress pre-trained models.

[0006] The detection method based on lightweight modules constructs the network through high-efficiency operation units such as point-by-point convolution, separable convolution, and group convolution, reduces the parameter quantity and computational complexity of the model, and thus reduces the detection delay of the model. However, this method focuses more on improving the efficient use of operation resources, i.e., time complexity and space complexity, and the detection accuracy cannot meet the requirements of actual production. The detection method based on neural architecture search introduces reinforcement learning (RL) to search for a lightweight CNN network architecture with high accuracy. However, the search space of this method mainly focuses on the structure at the unit level, and the same unit may be reused in all layers, which may cause the computational cost to increase exponentially. The detection method based on using various techniques to compress pre-trained models is a complementary work of the above two methods, which can further optimize the trained model through techniques such as quantization, pruning, and distillation. However, this method is essentially a strategy to improve network efficiency by reducing accuracy, and it requires complex processing steps. The relatively advanced detection method based on lightweight modules usually introduces an attention mechanism to let the network learn the weights of each channel in the feature map autonomously, and improves the detection accuracy through weighting.

[0007] In summary, the detection methods based on lightweight modules and the detection methods based on using various techniques to compress pre-trained models focus on using different strategies to reduce the complexity of the model, while the detection method based on neural architecture search focuses on improving the detection accuracy of the model under the premise of limited operation resources. In summary, the first two methods often do not consider the detection accuracy, while the last method does not consider the storage cost of the operation unit to the model. SUMMARY

[0008] The technical problem to be solved by the present application is to provide a malleable iron surface defect detection method for edge computing with high detection accuracy, low computational complexity, and low computational cost.

[0009] The technical solution adopted in this invention is a method for detecting surface defects in malleable iron pipe fittings based on edge computing, which includes the following steps:

[0010] S1. At a fixed acquisition height, multiple images of surface defects of malleable iron pipe fittings are acquired using industrial CCD cameras of different models and different ambient light angles. Each image of surface defects of malleable iron pipe fittings includes at least one surface defect of malleable iron pipe fittings.

[0011] S2. Label the defects on the multiple surface defect images of malleable iron pipes collected in step S1 to obtain the label file corresponding to each surface defect image of malleable iron pipes, and combine the label files into a dataset.

[0012] S3. Use 80% of the dataset obtained in step S2 as the training set and 20% as the test set;

[0013] S4. Construct an extended compressed residual bottleneck network, which includes two feature extraction branches. The input feature map is set as I. The input feature map I is projected into two feature extraction branches. One feature extraction branch uses a convolution strategy of first dilation and then compression to extract the features of internal defects of malleable iron pipes on the input feature map I and suppress background noise information. The other feature extraction branch uses a convolution strategy of first compression and then dilation to extract the features of the edge contour of malleable iron pipes on the input feature map I. The two feature extraction branches output feature maps after feature extraction. The feature maps output by the two feature extraction branches are fused along the channel direction by element-wise summation. The fused feature map is then output without recalibration.

[0014] S5. Based on the expanded compressed residual bottleneck network constructed in step S4, after the feature maps extracted by the feature extraction are output by the two feature extraction branches, an attention mechanism is introduced to obtain an expanded compressed feature extraction backbone network with an attention mechanism.

[0015] S6. Based on the attention-integrated extended compression feature extraction backbone network obtained in step S5, a surface defect detection model for malleable iron pipe fittings is obtained.

[0016] S7. Use the training set divided in step S3 to train the surface defect detection model of malleable iron pipes obtained in step S6, and obtain the trained surface defect detection model of malleable iron pipes.

[0017] S8. Use the test set divided in step S3 to test the trained malleable iron pipe surface defect detection model obtained in step S7, and adjust the model parameters to obtain the optimized malleable iron pipe surface defect detection model.

[0018] S9. Input the real-time detected images of malleable iron pipe fittings into the optimized surface defect detection model of malleable iron pipe fittings obtained in step S8, and perform real-time defect detection on the surface of malleable iron pipe fittings.

[0019] Preferably, in step S2, the defects on the multiple images of surface defects of malleable iron pipes collected in step S1 are labeled to obtain a label file. The specific process is as follows: each defect on the surface defect image of the malleable iron pipe is labeled with a bounding box and a class label, and the labeled image is saved as a label file in JSON format.

[0020] Preferably, in step S4, one of the feature extraction branches adopts a convolution strategy of first expanding and then compressing. The specific process for extracting features of internal defects of malleable iron pipe fittings on the input feature map I and suppressing background noise information includes the following steps:

[0021] S4-01. Map the input feature map I to a high-dimensional subspace through a 1×1 pointwise convolution, and denote the result as follows: The nonlinear activation output is denoted as Where e represents the expansion coefficient;

[0022] S4-02, Extracting using 3×3 depthwise convolution The feature information of each input channel is used to obtain the result, which is denoted as... The nonlinear activation output is denoted as

[0023] S4-03, then compress using 1×1 pointwise convolution. The number of channels, the result of which is denoted as right The result obtained by performing a linear activation operation is denoted as:

[0024] Preferably, in step S4, the other feature extraction branch adopts a convolution strategy of compression followed by expansion. The specific process for extracting the features of the malleable iron pipe edge contour on the input feature map I includes the following steps:

[0025] S4-11. Map the input feature map I to a low-dimensional subspace through a 1×1 pointwise convolution, and the result is denoted as... The nonlinear activation output is denoted as s represents the compression factor; S4-12, applying a 3×3 depthwise convolution to... The result obtained from each input channel is denoted as . The nonlinear activation output is denoted as

[0026] S4-13, Expanding by 1×1 pointwise convolution The number of channels, the result of which is denoted as right The Swish nonlinear activation operation is applied, and the result is denoted as...

[0027] Preferably, in step S4, the feature maps output by the two feature extraction branches are fused along the channel direction by element-wise summation. The specific process for outputting the fused feature map is as follows: the identity mappings between the output feature maps of the two feature extraction branches and their corresponding input feature maps are fused by element-wise summation, and the fused feature map is output. The output fused feature map is denoted as... in

[0028] Preferably, in step S5, based on the dilated compressed residual bottleneck network constructed in step S4, an attention mechanism is introduced after the feature maps extracted by the two feature extraction branches are output, resulting in a dilated compressed feature extraction backbone network with an attention mechanism. The specific process includes the following steps:

[0029] S5-1, Integration: Establishing an attention mechanism for the overall perception of each neuron in the fused feature map, i.e., integrating the features from various sources through element-wise summation. and The information is integrated into each neuron of the fusion feature map to obtain a new fusion feature map. The expression of the new fusion feature map is: in,

[0030] S5-2, Recalibration: a. Using a global average pooling layer Generate new fusion feature maps Quantitative statistical data information s, New fusion feature map The c-th channel counts the c-th element of s, and its expression is: b. Through a fully connected layer with an activation function Construct a shrinkage feature z, Its expression is Where β() represents the batch normalization layer, and σ() represents the Mish activation function. c. Adaptively recalibrate the output feature maps of the two feature extraction branches using the SoftMax operator. and The response of each channel, and the calibration weight of each branch channel is guided by the shrinkage feature z, which is expressed as: in, and They are respectively Feature map channel domain attention weight vector; specifically... Represents the c-th row of U, u c It is the c-th element of u;

[0031] S5-3, Reintegration: The channel domain attention weight vectors u and v calculated in S5-2 are used to reintegrate the feature maps. and The final recalibrated fused feature map O is obtained, and the feature map O of the c-th channel of the recalibrated fused feature map O is obtained. c The expression is:

[0032]

[0033] Among them, u c +v c =1, O=[O1,O2,…,O C ],

[0034] Compared with existing technologies, the beneficial effects of this invention are: compared with detection methods based on lightweight modules and detection methods based on compressing pre-trained models using various techniques, this invention has lower model complexity; compared with detection methods based on neural architecture search, this invention makes more efficient use of the storage cost of computing units; in addition, this invention has higher detection accuracy for surface defects of malleable iron pipe fittings, and the complexity of the model can be adjusted according to the computing power of edge devices. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the expanded compression residual bottleneck network in the method for detecting surface defects of malleable iron pipe fittings oriented towards edge computing according to the present invention.

[0036] Figure 2 This is a schematic diagram of the structure of the extended compression feature extraction backbone network that incorporates an attention mechanism in the edge computing-oriented method for detecting surface defects in malleable iron pipe fittings according to the present invention.

[0037] Figure 3 This is a schematic diagram illustrating the classification and labeling of surface defects in malleable iron pipe fittings in step S2 of this embodiment of the invention;

[0038] Figure 4 This is a schematic diagram illustrating the operation steps of Labelme in an embodiment of the present invention;

[0039] Figure 5 The image shows the test results obtained by testing all models using the same input resolution of 224×224 in this embodiment of the invention. Detailed Implementation

[0040] The invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can implement it based on the description. The scope of protection of the invention is not limited to these specific embodiments.

[0041] This invention provides a method for detecting surface defects in malleable iron pipe fittings based on edge computing. The method includes the following steps:

[0042] S1. At a fixed acquisition height (15cm), use industrial CCD cameras of different models (such as IMX 700CMOS, FA, SONY) and different ambient light angles (any light angle that ensures the defect is visible) to acquire multiple images of surface defects of malleable iron pipe fittings. Each image of surface defects of malleable iron pipe fittings includes at least one surface defect of malleable iron pipe fittings.

[0043] S2. Label the defects on the multiple surface defect images of malleable iron pipes collected in step S1 to obtain the label file corresponding to each surface defect image of malleable iron pipes, and combine the label files into a dataset.

[0044] S3. Use 80% of the dataset obtained in step S2 as the training set and 20% as the test set;

[0045] S4. Construct an Expandation and Shrinking Bottlenecks (ESNet) network, such as... Figure 1 As shown, the extended compressed residual bottleneck network includes two feature extraction branches, with the input feature map set as I. The input feature map I is projected into two feature extraction branches. One feature extraction branch uses a convolution strategy of first dilation and then compression to extract the features of internal defects of malleable iron pipes on the input feature map I and suppress background noise information. The other feature extraction branch uses a convolution strategy of first compression and then dilation to extract the features of the edge contour of malleable iron pipes on the input feature map I. The two feature extraction branches output feature maps after feature extraction. The feature maps output by the two feature extraction branches are fused along the channel direction by element-wise summation. The fused feature map is then output without recalibration.

[0046] S5. Based on the dilated compressed residual bottleneck network constructed in step S4, an attention mechanism is introduced after the feature maps extracted from the two feature extraction branches are output, resulting in a dilated compressed feature extraction backbone network with an attention mechanism, as shown below. Figure 2 As shown;

[0047] S6. Based on the attention-integrated extended compression feature extraction backbone network obtained in step S5, a surface defect detection model for malleable iron pipe fittings is obtained.

[0048] S7. Use the training set divided in step S3 to train the surface defect detection model of malleable iron pipes obtained in step S6, and obtain the trained surface defect detection model of malleable iron pipes.

[0049] S8. Use the test set divided in step S3 to test the trained malleable iron pipe surface defect detection model obtained in step S7, and adjust the model parameters to obtain the optimized malleable iron pipe surface defect detection model.

[0050] S9. Input the real-time detected images of malleable iron pipe fittings into the optimized surface defect detection model of malleable iron pipe fittings obtained in step S8, and perform real-time defect detection on the surface of malleable iron pipe fittings.

[0051] Preferably, in step S2, the defects on the multiple surface defect images of malleable iron pipes acquired in step S1 are labeled to obtain a label file. The specific process is as follows: each defect on the surface defect image of the malleable iron pipe is labeled using a bounding box and a class label, and the labeled image is saved as a JSON format label file; for example... Figure 3 The dataset shown contains 9 types of defects, and each defect image corresponds to a JSON-formatted label file.

[0052] Figure 3 In the process, each defect instance is labeled with a bounding box and a class label, resulting in a JSON-formatted label file for each defect image. Through discussions with quality control workers, the following labeling rules were established:

[0053] First, surface defects in malleable iron pipe fittings are classified into four main categories: The most common is the defective type, characterized by varying degrees of surface damage. Based on the size of the damaged area, these can be further categorized as sand holes, approximately 5% defects (Appr. 5%), approximately 15% defects (Appr. 15%), approximately 25% defects (Appr. 25%), and approximately 50% defects (Appr. 50%). The next most common defect is the convex defect, characterized by excess casting material on the surface. Based on the size of this excess casting material, these can be categorized as long pegs (long pegs) and seam ridges (Abn. pegs). Less common are concave defects, which involve settlement damage on the surface of the malleable iron pipe fitting, primarily including collapse. The least common are casting waste defects, which are defects where two casting molds connect during the pipe casting process, primarily including casting waste. Figure 3 Samples of the nine defects mentioned above are provided, with the bounding box of each defect marked with a different color.

[0054] Second, when multiple defect categories overlap at the same location, the defect with the largest area should be labeled first. In particular, concave defects and broken defects often appear together. When the bounding boxes of the two types of defects are in an inclusive relationship, the bounding box category is determined by the defect with the largest area; when the bounding boxes of the two types of defects are in an intersecting relationship, the bounding boxes should be as close as possible to their respective defect areas.

[0055] Third, for irregular, large-scale defects, the defect should be completely labeled by one large bounding box, rather than by multiple smaller bounding boxes.

[0056] Fourth, regarding the easily confused defects of sand holes and approximately 5% defects (Appr. 5%), the defect area of ​​sand holes accounts for between 0.25% and 2.5% of the defect image, while the defect area of ​​Appr. 5% accounts for between 2.5% and 7.5% of the defect image.

[0057] Guided by the above labeling rules, this article uses the open-source software Labelme tool, such as... Figure 4Annotate images of defects in malleable iron pipe fittings. The annotation process mainly includes the following steps: ① Start the software and open the folder containing the defect dataset (Open Dir); ② Click the Edit button and select Create Rectangle to select a rectangular bounding box for annotation; ③ Select the defect area with the left mouse button; ④ Enter the defect category in the edit box that appears according to the annotation rules; ⑤ Click the Confirm button to complete the annotation of one defect image; ⑥ Click Next Image to annotate the next image. Repeat steps ③-⑥ to complete the annotation of the entire dataset.

[0058] Preferably, in step S4, such as Figure 1 As shown, one of the feature extraction branches employs a convolution strategy of first dilation and then compression to extract features of internal defects in malleable iron pipe fittings from the input feature map I and to suppress background noise information. The specific process includes the following steps:

[0059] S4-01. Map the input feature map I to a high-dimensional subspace through a 1×1 pointwise convolution, and denote the result as... The nonlinear activation output is denoted as Where e represents the expansion coefficient;

[0060] S4-02, Extracting using 3×3 depthwise convolution The feature information of each input channel is used to obtain the result, which is denoted as... The nonlinear activation output is denoted as

[0061] S4-03, then compress using 1×1 pointwise convolution. The number of channels, the result of which is denoted as To reduce the loss of information within the feature map after compression, The result obtained by performing a linear activation operation is denoted as:

[0062] Preferably, in step S4, such as Figure 1 As shown, the other feature extraction branch adopts a convolution strategy of compression followed by dilation. The specific process for extracting the features of the edge contour of the malleable iron pipe fitting on the input feature map I includes the following steps:

[0063] S4-11. Map the input feature map I to a low-dimensional subspace through a 1×1 pointwise convolution, and the result is denoted as... The nonlinear activation output is denoted as s represents the compression factor; S4-12, applying a 3×3 depthwise convolution to... The result obtained from each input channel is denoted as . The nonlinear activation output is denoted as

[0064] S4-13, Expanding by 1×1 pointwise convolution The number of channels, the result of which is denoted as Unlike the second pointwise convolution in the dilated branch, for The Swish nonlinear activation operation is applied, and the result is denoted as...

[0065] Preferably, in step S4, the feature maps output by the two feature extraction branches are fused along the channel direction by element-wise summation. The specific process for outputting the fused feature map is as follows: the identity mappings between the output feature maps of the two feature extraction branches and their corresponding input feature maps are fused by element-wise summation, and the fused feature map is output. The output fused feature map is denoted as... in

[0066] Preferably, in step S5, based on the extended compressed residual bottleneck network constructed in step S4, an attention mechanism is introduced after the feature maps extracted by the two feature extraction branches are output, such as... Figure 2 As shown, the specific process of obtaining the extended compressed feature extraction backbone network with an attention mechanism includes the following steps:

[0067] S5-1, Integration: Establish an attention mechanism to perceive the overall structure of each neuron in the fused feature map (a neuron is the sum of the elements within each channel of the feature map), that is, to integrate the data from each neuron in the fused feature map through element-wise summation. and The information is integrated into each neuron of the fusion feature map to obtain a new fusion feature map. The expression of the new fusion feature map is:

[0068] S5-2, Recalibration: a. Using a global average pooling layer Generate new fusion feature maps Quantitative statistical data information s, New fusion feature map The c-th channel counts the c-th element of s, and its expression is: b. Through a fully connected layer with an activation function Construct a shrinkage feature z, Its expression is Where β() represents the batch normalization layer, and σ() represents the Mish activation function. The subsequent experiments also investigated the impact of the reduction coefficient r on model performance; c. The SoftMax operator was used to adaptively recalibrate the output feature maps of the two feature extraction branches respectively. and The response of each channel, and the calibration weight of each branch channel is guided by the shrinkage feature z, which is expressed as: in, and They are respectively Feature map channel domain attention weight vector; specifically... Represents the c-th row of U, u c It is the c-th element of u;

[0069] S5-3, Reintegration: The channel domain attention weight vectors u and v calculated in S5-2 are used to reintegrate the feature maps. and The final recalibrated fused feature map O is obtained, and the feature map O of the c-th channel of the recalibrated fused feature map O is obtained. c The expression is: Among them, u c +v c =1, O=[O1,O2,…,O C ],

[0070] The English name of the extended compressed feature extraction backbone network with integrated attention mechanism is ES-MobileNet*; the English name of the extended compressed feature extraction backbone network is ES-MobileNet;

[0071] Table 1 shows the detailed structure of ES-MobileNet*. In Table 1, E and S represent the number of channels in the output feature map of the expansion and compression branches after the first pointwise convolution, respectively; SK indicates whether there is a selective kernel in the operation unit block; NL indicates the type of non-linear activation function used; where S represents Swish and R represents ReLU; ESNet represents the expansion and compression bottleneck network with fused attention mechanism proposed in this invention; Conv2d represents the standard two-dimensional convolution operation; s, k, and t represent the stride of the convolution kernel, the kernel size, and the number of times the operation unit is repeated, respectively.

[0072] Table 1. Detailed structure of ES-MobileNet*:

[0073]

[0074] The dataset IIDD of this invention contains 4020 images of surface defects in malleable iron pipe fittings, comprising a total of 6313 defect instances. The entire dataset includes nine defect categories: Sand Hole, Long Peg, Appr. 5% (5% surface defect), Abn. Peg (seam ridge), Appr. 50% (50% surface defect), Collapse, Appr. 25% (25% surface defect), Appr. 15% (15% surface defect), and Cast Waste. Table 2 shows the number of instances and proportions for each defect category. Figure 3 Examples of various defect samples are provided; in the evaluation experiment, we randomly selected 80% of each category as the training set and 20% as the test set.

[0075] Table 2. Statistics on 9 common surface defects of malleable iron pipe fittings in IIDD:

[0076]

[0077] The ES-MobileNet* defined in Table 1 is based on the ESNet design. We use the proposed ES-MobileNet* as a feature extractor and perform defect detection on the Darknet framework. The default hyperparameters are as follows: the number of edges in the input image is adjusted to 224; the number of training steps is 20000; the batch size and mini-batch size are 64 and 16, respectively; a stepped decay learning rate scheduling strategy is adopted, with an initial learning rate of 0.001, multiplied by factors of 0.1, 10, 0.1, and 0.1 at steps 1100, 15000, and 18000, respectively; the warm-up steps are 1000; and the momentum and weight decay are set to 0.949 and 0.0005, respectively. Except for the last two layers, all our convolutional layers use batch-normalization. All experiments were trained and evaluated on a Windows PC using a GeForce RTX 3070 and 8GB of memory. In particular, we evaluated the FPS of all models on the Nvidia Jetson Nano, which has 128 Maxwell cores and 4GB of memory.

[0078] This paper evaluates the performance of the extended compressed residual bottleneck network by comparing mean accuracy (mAP), precision, recall, F1-score, FLOPs, number of parameters (#Params), and FPS (frames per second) at a 0.5 IoU threshold. Generally, mAP measures the accuracy across all classes in a dataset, FPS is used to evaluate inference speed, recall measures the coverage of predicted and ground truth boxes, F1-score comprehensively evaluates a model's performance, FLOPs quantifies the model's computational time complexity, and #Params quantifies the model's computational space complexity.

[0079] We compared YOLOv2 and SSD (SSD is a detection algorithm in the field of computer vision) as baselines, and also evaluated and compared the detection performance of a modified version of YOLOv2 called ES-YOLO (a model that replaces all standard convolutions in the prediction layer of YOLOv2 with ESNet); the results are shown in Table 3; the input resolution of both YOLOv2 and ES-YOLO is 416×416; in ES-YOLO, the e, s, and r of ESNet are 4, 2, and 16, respectively; compared with the original YOLOv2, ES-YOLO significantly reduces the number of parameters (1 / 8) and computational cost (1 / 10).

[0080] Table 3. Performance comparison of ES-YOLO with other large-scale networks on the IIDD dataset task of detecting surface defects in malleable iron pipe fittings:

[0081]

[0082] In the comparative experiments, the only difference between ES-MobileNet and ES-MobileNet* is that the former removes all attention mechanisms from the latter (i.e., removes SK from Table 1); the similarity is that both retain the extended and compressed branches of this invention. ES-MobileNet is called Extended Compressed Feature Extraction Backbone Network; ES-MobileNet* is called Extended Compressed Feature Extraction Backbone Network with Integrating Attention Mechanisms.

[0083] In this embodiment of the invention, we select small NAS-based networks (such as NasNet, MnasNet-A, EfficientNet-B0, and MobileNetV3-large) and hand-designed small networks (such as Tiny-YOLO, MobileNetV2, ShuffleNetV1, and ShuffleNetV2) as baselines and compare them with the proposed hand-designed small network ES-MobileNet. In experiments, all small networks were evaluated at a 224 input resolution within the YOLO framework. The ES bottleneck in our model has e, s, and r values ​​of 4, 2, and 16, respectively. We do not compare model performance under other frameworks, such as Faster-RCNN, because our focus is on mobile / real-time models. As shown in Table 4, ES-MobileNet achieves similar mAP to EfficientNet-B0 and MobileNetV2 but with better FPS, and fewer parameters and lower computational cost. Notably, the proposed network ES-MobileNet* achieves the best mAP with negligible computational complexity thanks to the Selective Kernel. For a more intuitive understanding, we... Figure 5 A detailed description is provided in the document.

[0084] To investigate the performance and computational cost tradeoffs of selective kernels, experiments were conducted using the ES-MobileNet* network with a range of different r values, fixing e=4 and s=2. As shown in Table 5, the model reaches its optimal state at r=8, and setting r=16 achieves a good balance between accuracy and complexity.

[0085] The embodiments of the present invention conducted experiments on ES-MobileNet with a series of different s values ​​with a fixed e=4. The comparison in Table 6 shows that increasing s does not monotonically improve mAP, while a larger s significantly reduces the computational complexity of the model. Setting s=2 can obtain the best accuracy and acceptable complexity.

[0086] The embodiments of this invention conducted experiments on ES-MobileNet with a fixed s=2 and a series of different e values. The comparisons in Table 7 show that increasing e does not monotonically improve mAP. It is worth noting that changes in the expansion factor have a greater impact on the model's space and computational complexity than changes in the contraction factor.

[0087] Table 4. Performance comparison of ES-MobileNet and other smaller networks on the IIDD dataset MI defect detection task, * indicates selective kernels:

[0088]

[0089] Table 5. The effect of selective kernels on ES-MobileNet* under different scaling factors r. Here, "original" refers to ES-MobileNet.

[0090]

[0091]

[0092] Table 6. Effects of ES-MobileNet on the model under different shrinkage factors s, with e = 4 fixed:

[0093] Models mAP #Params BFLOPs 4e0s 60.62 0.75M 0.612 4e1s 60.58 0.78M 0.738 4e2s 61.98 0.75M 0.675 4e3s 58.71 0.75M 0.654 4e4s 57.22 0.75M 0.644

[0094] Table 7. The impact of ES-MobileNet on the model under different expansion factors e, with s=2 fixed:

[0095] Models mAP #Params BFLOPs 0e2s 50.76 0.25M 0.176 1e2s 53.83 0.78M 0.301 2e2s 56.54 4.06M 0.426 3e2s 58.54 6.08M 0.551 4e2s 61.98 0.75M 0.675 5e2s 61.45 0.90M 0.800 6e2s 60.92 1.05M 0.925

Claims

1. A method for detecting surface defects in malleable iron pipe fittings based on edge computing, characterized in that: The method includes the following steps: S1. At a fixed acquisition height, multiple images of surface defects of malleable iron pipe fittings are acquired using industrial CCD cameras of different models and different ambient light angles. Each image of surface defects of malleable iron pipe fittings includes at least one surface defect of malleable iron pipe fittings. S2. Label the defects on the multiple surface defect images of malleable iron pipes collected in step S1 to obtain the label file corresponding to each surface defect image of malleable iron pipes, and combine the label files into a dataset. S3. Use 80% of the dataset obtained in step S2 as the training set and 20% as the test set; S4. Construct an extended compressed residual bottleneck network, which includes two feature extraction branches. The input feature map is set as I. The input feature map I is projected into two feature extraction branches. One feature extraction branch uses a convolution strategy of first dilation and then compression to extract the features of internal defects of malleable iron pipes on the input feature map I and suppress background noise information. The other feature extraction branch uses a convolution strategy of first compression and then dilation to extract the features of the edge contour of malleable iron pipes on the input feature map I. The two feature extraction branches output feature maps after feature extraction. The feature maps output by the two feature extraction branches are fused along the channel direction by element-wise summation. The fused feature map is then output without recalibration. S5. Based on the expanded compressed residual bottleneck network constructed in step S4, after the feature maps extracted by the feature extraction are output by the two feature extraction branches, an attention mechanism is introduced to obtain an expanded compressed feature extraction backbone network with an attention mechanism. S6. Based on the expanded and compressed feature extraction backbone network with attention mechanism obtained in step S5, a surface defect detection model for malleable iron pipe fittings is obtained. S7. Use the training set divided in step S3 to train the surface defect detection model of malleable iron pipes obtained in step S6, and obtain the trained surface defect detection model of malleable iron pipes. S8. Use the test set divided in step S3 to test the trained malleable iron pipe surface defect detection model obtained in step S7, and adjust the model parameters to obtain the optimized malleable iron pipe surface defect detection model. S9. Input the real-time detected images of malleable iron pipe fittings into the optimized surface defect detection model of malleable iron pipe fittings obtained in step S8, and perform real-time defect detection on the surface of malleable iron pipe fittings.

2. The method for detecting surface defects in malleable iron pipe fittings based on edge computing according to claim 1, characterized in that: In step S2, the defects on the multiple surface defect images of malleable iron pipes collected in step S1 are labeled to obtain a label file. The specific process is as follows: each defect on the surface defect image of the malleable iron pipe is labeled with a bounding box and a class label, and the labeled image is saved as a label file in JSON format.

3. A method for detecting surface defects in malleable iron pipe fittings based on edge computing, as described in claim 1 or 2, characterized in that: In step S4, one of the feature extraction branches adopts a convolution strategy of first dilation and then compression to extract features of internal defects in malleable iron pipe fittings on the input feature map I and to suppress background noise information. The specific process includes the following steps: S4-01. Map the input feature map I to a high-dimensional subspace through a 1×1 pointwise convolution, and denote the result as... The nonlinear activation output is denoted as Where e represents the expansion coefficient; S4-02, Extracting using 3×3 depthwise convolution The feature information of each input channel is used to obtain the result, which is denoted as... The nonlinear activation output is denoted as S4-03, then compress using 1×1 pointwise convolution. The number of channels, the result of which is denoted as right The result obtained by performing a linear activation operation is denoted as:

4. The method for detecting surface defects in malleable iron pipe fittings based on edge computing according to claim 3, characterized in that: In step S4, the other feature extraction branch adopts a convolution strategy of compression followed by dilation. The specific process for extracting the features of the edge contour of the malleable iron pipe fitting on the input feature map I includes the following steps: S4-11. Map the input feature map I to a low-dimensional subspace through a 1×1 pointwise convolution, and the result is denoted as... The nonlinear activation output is denoted as s represents the compression factor; S4-12, Applying 3×3 depthwise convolution to... The result obtained from each input channel is denoted as . The nonlinear activation output is denoted as S4-13, Expanding by 1×1 pointwise convolution The number of channels, the result of which is denoted as right The Swish nonlinear activation operation is applied, and the result is denoted as...

5. The method for detecting surface defects in malleable iron pipe fittings based on edge computing according to claim 4, characterized in that: In step S4, the feature maps extracted from the two feature extraction branches are fused along the channel direction by element-wise summation. The specific process for outputting the fused feature map is as follows: the identity mappings between the output feature maps of the two feature extraction branches and their corresponding input feature maps are fused by element-wise summation, and the fused feature map is output. The output fused feature map is denoted as... in 6. The method for detecting surface defects in malleable iron pipe fittings based on edge computing according to claim 5, characterized in that: In step S5, based on the dilated compressed residual bottleneck network constructed in step S4, an attention mechanism is introduced after the feature maps extracted by the two feature extraction branches are output, resulting in a dilated compressed feature extraction backbone network with an attention mechanism. The specific process includes the following steps: S5-1, Integration: Establishing an attention mechanism for the overall perception of each neuron in the fused feature map, i.e., integrating the features from various sources through element-wise summation. and The information is integrated into each neuron of the fusion feature map to obtain a new fusion feature map. The expression of the new fusion feature map is: in, S5-2, Recalibration: a. Using a global average pooling layer Generate new fusion feature maps Quantitative statistical data information s, New fusion feature map The c-th channel counts the c-th element of s, and its expression is: b. Through a fully connected layer with an activation function Construct a shrinkage feature z, Its expression is Where r represents the reduction rate, which is a non-zero positive integer; β() represents the batch normalization layer; and σ() represents the Mish activation function. c. Adaptively recalibrate the output feature maps of the two feature extraction branches using the SoftMax operator. and The response of each channel, and the calibration weight of each branch channel is guided by the shrinkage feature z, which is expressed as: Among them, U, and u, They are respectively Feature map channel domain attention weight vector; specifically... Represents the c-th row of U, u c V is the c-th element of the two-dimensional matrix u. c This represents the c-th row of V, v c It is the c-th element of the two-dimensional matrix v; S5-3, Reintegration: The channel domain attention weight vectors u and v calculated in S5-2 are used to reintegrate the feature maps. and The final recalibrated fused feature map O is obtained, and the feature map O of the c-th channel of the recalibrated fused feature map O is obtained. c The expression is: Among them, u c +v c =1,

Images