Turbine starter appearance defect detection method based on multi-scale feature extraction

Abstract

The invention relates to a turbine starter appearance defect detection method based on multi-scale feature extraction, and the method comprises the steps: collecting appearance defect data of a turbine starter, and defining defect types to obtain an appearance defect feature map; the image is input into a data enhancement module, parameters are adjusted according to image types, and a multi-attribute defect feature map is output; constructing a feature extraction module, extracting defect data features in the multi-attribute defect feature map, converting the defect data features into multilayer feature mapping, and generating a multi-scale defect feature map; inputting the multi-scale defect feature map into an enhanced feature extraction network, obtaining and aggregating high-dimensional multi-scale defect features, and generating a multi-scale fusion defect feature map; constructing a defect detection module, and detecting bounding box prediction and classification results of multi-scale defects; and carrying out training and weight updating on the model by adopting similarity and union set intersection loss to obtain a final defect detection model. According to the invention, accurate positioning and classification of defects of various targets with different sizes on the appearance of the turbine starter are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent manufacturing automated detection, and in particular relates to a method for detecting appearance defects of a turbine starter by extracting multi-scale features. Background Art

[0002] In industrial defect detection, the detection of targets of various scales has always been a major pain point and problem in the field of production quality control, especially the detection of appearance defects of turbine starters. Due to the large differences in image size and defect characteristics, it is difficult for common defect detection frameworks to learn the features of such targets of various scales and to locate and classify such targets with high precision.

[0003] With the rise of deep learning and artificial intelligence technology, deep learning-based methods no longer require artificial features, but instead use convolutional neural networks to obtain advanced features, making it easier to capture complex patterns and relationships from input data. Due to its high precision and strong real-time performance, the world's manufacturing powers have proposed applying artificial intelligence technology to their future development strategies for their manufacturing industries in order to foster a highly flexible, personalized, and intelligent production management model. The present invention addresses the current difficulty in turbine starter appearance defect detection, namely, defect detection of targets of multiple scales, and proposes a turbine starter appearance defect detection method using multi-scale feature extraction, which can detect defects of targets of multiple scales in the product with high precision. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention proposes a method for detecting defects on the appearance of a turbine starter based on multi-scale feature extraction. By improving the feature extraction and fusion in the traditional target detection model, the feature extraction and fusion of the multi-scale feature structure are performed by using the attention mechanism and spatial pyramid pooling, so that the neural network model can adapt to the variable sensory field of different defect features, so as to realize the positioning and classification of defects of targets of various different scales, and meet the task of detecting multi-scale defects on the appearance of the turbine starter.

[0005] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0006] A method for detecting appearance defects of a turbine starter by extracting multi-scale features, specifically comprising:

[0007] S1, collecting appearance defect data of a turbine starter, and defining image types of the collected appearance defects to obtain a turbine starter appearance defect feature image I;

[0008] S2. Construct a data enhancement module, input the turbine starter appearance defect feature map i into the data enhancement module IDE, adjust the parameters of the input turbine starter appearance defect image according to the type of image that the model is good at processing, and obtain the multi-attribute defect feature map Im ；

[0009] S3, build a feature extraction module, and transform the multi-attribute defect feature map I m Input the feature extraction network, extract the features of the data containing defects in the image, convert it into a multi-layer feature map, and generate a multi-scale defect feature map I containing defect features of different scales. MSPP ;

[0010] S4, build an enhanced feature fusion module to transform the multi-scale defect feature map I MSPP Input the enhanced feature extraction network to obtain higher-dimensional defect feature information, and aggregate multi-scale features through the feature fusion network to generate a multi-scale fusion defect feature map I M ;

[0011] S5. Construct a defect prediction module to fuse the multi-scale defect feature map I M Input into the defect detection network, the bounding box prediction results and classification results of multi-scale defects on the appearance of the turbine starter are detected;

[0012] S6. For the defect detection model of multi-scale defects, similarity and union intersection loss are used as loss functions to train and update the weights of the model, and the final defect detection model is obtained.

[0013] Furthermore, step S1 specifically includes:

[0014] S11. Use a two-dimensional industrial camera to collect images of the appearance of the defective turbine starter;

[0015] S12, defining the types of the collected images of the appearance defects of the turbine starter, specifically defining them into seven types: pits, bumps, edge damage, rust, breakage, scratches, and wrinkles;

[0016] S13. After the type definition is completed, data annotation is performed on the appearance defects to obtain a turbine starter appearance defect feature diagram I.

[0017] Furthermore, step S2 specifically includes:

[0018] S21, inputting the turbine starter appearance defect feature map I into the data enhancement module IDE, and performing a random judgment cycle of the feature map processing method;

[0019] S22, for the input turbine starter appearance defect feature image I, generate a random judgment parameter set {a i∈(0,1)|i=1,2,...,10}; each parameter in the set corresponds to a feature processing method, including: cropping, rotation, flipping, scaling, translation, brightness adjustment, contrast adjustment, grid distortion, adding Gaussian noise and adding ISO noise;

[0020] S23, in the random judgment cycle, set the cumulative parameter N, and give it an initial value of 0, and set the random judgment parameter a i The threshold α, when a i >α, the corresponding feature processing method is executed and the processed feature map is output, and the cumulative parameter N increases by 1; conversely, when a i When ≤α, the corresponding feature processing method is not executed and the accumulation parameter N is not changed;

[0021] S24, after a round of random feature processing, if the cumulative parameter N < 16, it is determined that the loop has not ended, and the random feature processing continues for the feature map I; until N ≥ 16, the loop ends and the 16 feature maps that have been processed before are spliced ​​into a multi-attribute defect feature map I after data enhancement. m And output.

[0022] Furthermore, step S3 specifically includes:

[0023] S31, multi-attribute defect feature map I m Input the focusing module to perform focusing operation and convert the high-resolution multi-attribute defect feature map I m The samples are cut into four low-resolution defect feature maps and spliced ​​along the channel direction to form the multi-attribute defect feature map I m The original RGB three channels are expanded to 12 channels, while the width W and height H are shortened to half of the original, and the focus defect feature map I is obtained. f ;

[0024] S32, focusing on defect feature map I f The input is a variable scale feature extraction network composed of DCBL deformable convolution module and CSP convolution residual module alternately, extracts variable scale defect feature information, and outputs variable scale defect feature map I v ;

[0025] S33, the variable scale defect feature map I v Input CBAM convolutional block attention module, use channel attention mechanism and spatial attention mechanism combined with convolutional neural network to enhance the input variable scale defect feature map I v Channel attention is used to enhance the information interaction between different channels and output the refined feature map I CBAM ;

[0026] S34, refining feature map I CBAMInput the MSPP multi-scale spatial pyramid pooling module for pooling operation. Using its spatial pyramid structure, it is compatible with feature maps of different sizes and scales, and extracts features of corresponding scales according to different scales of feature maps to improve recognition accuracy. Finally, it outputs a multi-scale defect feature map I containing defect features of different scales. MSPP .

[0027] More specifically, in step S32:

[0028] The DCBL module consists of a deformable convolution layer, a batch normalization BN layer, and an activation function connected in sequence; the input focus defect feature map I f After entering the DCBL module, the sampling position is adaptively adjusted through deformable convolution to extract the deformation features in the feature map, and then the internal covariate offset is reduced through batch normalization to avoid gradient explosion and disappearance. Finally, nonlinear factors are introduced by adding activation functions to enhance the fitting ability of the neural network model. The formula of the DCBL module is expressed as:

[0029] DCBL(I0)=Leaky ReLU(BN(DCN(I0)));

[0030] Where I0 represents the feature map of the input DCBL module, Leaky ReLU (·) is the leaky linear rectification function, i.e., the activation function; BN (·) is the batch normalization function, and DCN (·) is the deformable convolution operation;

[0031] CSP modules are divided into two categories, including four different architectures: CSP1_1, CSP1_2, CSP1_3, and CSP2_1;

[0032] The first type of CSP1_n (n=1,2,3) modules include three architectures: CSP1_1, CSP1_2, and CSP1_3, which are divided into two branches. In the first branch, the input feature map I1 is extracted through a sequential architecture consisting of a CBL ordinary convolution module, a residual module composed of n residual components, and an ordinary convolution layer. In the second branch, the input feature map I1 is directly extracted through an ordinary convolution layer, and then the feature maps output by the two branches are spliced, and then batch normalization operations, activation functions and a CBL module are added to finally output a feature map containing rich global and local information.

[0033] The architecture of the CBL module is similar to the DCBL module, except that CBL only uses ordinary convolution.

[0034] The residual component is: the input feature map I2 is passed through a feature extraction branch with two CBL modules connected in sequence and a direct edge branch, and the feature maps output by the two branches are spliced ​​and then output;

[0035] The overall architecture of the second type CSP2_n (n=1) module is similar to that of the CSP1_n module, except that the residual module in the first branch is replaced by 2n CBL modules.

[0036] More specifically, in step S34:

[0037] The MSPP module adopts a spatial pyramid structure with four maximum pooling layers. It stores the output of each maximum pooling layer after the feature map is passed and connects the outputs to capture features at different scales. At the same time, a CBL module is connected to the input and output of the MSPP module, and the output of the first and third maximum pooling layers is used as the input of the fourth layer. The formula is expressed as:

[0038] I MSPP =CBL(COncat[CBL(I CBAM ); P1; P2; P3; P4]);

[0039] Among them, the output of each pooling layer is P1=MaxPool(I CBAN ), P2=MaxPool(P1), P3=MaxPool(P2), P4=MaxPool(Concat[P1;P3]).

[0040] Furthermore, step S4 specifically includes:

[0041] S41, multi-scale defect feature map I MSPP The input is an enhanced feature fusion network composed of the CBL module, the CARAFE content-aware feature reorganization operator and the CSP2_1 module connected in sequence. The CARAFE operator is used to dynamically adjust the weight according to the content of the input feature map to perform efficient and accurate feature upsampling to improve the prediction ability of the neural network model for dense defects, and the enhanced feature defect feature map I is output. E , the specific calculation formula is:

[0042] I E =CSP2_1(CARAFE(CBL(I MSPP )));

[0043] The CARAFE operator first predicts the upsampling kernel from the input feature map using convolution operation through the kernel prediction module, and then uses the predicted upsampling kernel to perform weighted reorganization on the input feature map through the content-aware reorganization module to generate the upsampled feature map.

[0044] S42, enhancing the characteristic defect feature map I EThe input is a feature fusion network composed of a CBAM module and an SCB separable convolution bottleneck module. The CBAM module is used to further focus on the important feature channels and spatial position information in the feature map to obtain an enhanced refined defect feature map I' E , and then use the SCB module to extract and fuse the multi-scale feature structure, so that the neural network model can adapt to the variable sensory field of different defect features, and finally output the multi-scale fusion defect feature map I M .

[0045] More specifically, in step S42:

[0046] The SCB module combines separable convolution and spatial pyramid structure to achieve efficient feature extraction and fusion while reducing computational costs. The formula is expressed as:

[0047] I M =Leaky ReLU(BN(Concat[Maxpool(I′ E ); 2SConv(I′ E ); 2SConv(2SConv(I′ E ))]));

[0048] Among them, Leaky ReLU (·) is the leaky linear rectification function, that is, the activation function; BN (·) is the batch normalization function; 2SConv (I ′ E ) represents the enhanced refining defect characteristic diagram I′ E Perform two-step separable convolution, i.e., first perform depthwise convolution and then perform pointwise convolution; the depthwise convolution is to apply an independent convolution kernel to each input channel of the input feature map to extract the features of each channel one by one, and the gradual convolution is to recombine the features of each channel of the input feature map through a 1×1 convolution, perform feature fusion between channels, fuse these separated channel information again, and restore the relationship between global features;

[0049] Enhanced refining defect feature diagram I′ E Perform maximum pooling, one separable convolution and two separable convolutions, and fuse the output through batch normalization and activation function to output a multi-scale fusion defect feature map I M .

[0050] Furthermore, step S5 specifically includes:

[0051] S51, multi-scale fusion defect feature map I M Upsample through the CARAFE operator to generate the feature map I′ M, and input three defect prediction branches of different scales for calculation respectively; the three defect prediction branches are used for defect prediction of large, medium and small scales respectively;

[0052] S52, upsampled feature map I' M Different numbers of SCB modules are passed through the three branches respectively. The first branch does not contain an SCB module, the second branch passes through an SCB module, and the third branch also passes through an SCB module, and feature fusion is performed to adapt to defect prediction of different scales; then they are processed by the CSP2_1 module and the CBAM module respectively, focusing on defects of corresponding scales, and finally, a common convolutional layer is used to output the prediction results of defects of large, medium and small scales respectively; the prediction results include the predicted defect position rectangular bounding box, confidence and defect category classification probability.

[0053] Furthermore, step S6 specifically includes:

[0054] Adopt bounding box regression loss L IOU To supervise and optimize the defect detection training process, and adopt the classifier loss L CLA To optimize the defect classification performance and use the similarity loss L SIM To evaluate the detection accuracy of the object, that is, the confidence, the joint loss L reg It can be expressed as:

[0055] L reg =λ1L IOU +λ2L SIM +λ3L CLA ;

[0056] Among them, λ1, λ2 and λ3 represent the proportion of each loss function;

[0057] The bounding box regression loss L IOU The CIOU loss is used to evaluate the accuracy of the predicted defect location rectangular bounding box, and the formula is expressed as:

[0058]

[0059] Where IOU is the intersection-over-union ratio between the predicted box and the true box, d is the distance between the center point of the predicted box and the center point of the true box, c is the diagonal length of the smallest closed rectangle containing the predicted box and the true box, α is a weight parameter used to balance the impact of center point distance loss and aspect ratio loss, and v is a correction factor used to measure the difference in aspect ratio between the predicted box and the true box, and further adjust the loss function;

[0060] The classifier loss L CLA VFL loss is used to evaluate the classification accuracy of defect categories, and the formula is expressed as:

[0061] L CLA = -tlog(p)(1-p t ) γ ;

[0062] Among them, t is the true label of the target category, p is the probability predicted by the model as this category, and p t is the adjusted predicted probability, defined as in represents element-wise division, α′ is a balancing factor used to adjust the weight of negative samples, and γ is a focusing parameter used to reduce the weight of easy-to-classify samples;

[0063] The similarity loss L SIM Gaussian distribution is used to evaluate the detection accuracy of objects, that is, confidence, and the formula is expressed as:

[0064] L SIM =1-ln SIM(A,B);

[0065] Where A=(cx a ,cy a ,w a ,h a ), cx a ,cy a Represents the coordinates of the upper left corner of the rectangular bounding box A of the predicted defect location, w a ,h a represents the width and height of the bounding box A of the predicted defect, B = (cx b ,cy b ,w b ,h b ), cx b ,cy b represents the coordinates of the upper left corner of the actual defect bounding box B, w b ,h b represents the width and height of the actual defect bounding box B, SIM(·) can be expressed as:

[0066]

[0067] in, Represents the bi-norm of a vector.

[0068] Based on the above technical solution, the present invention has the following beneficial effects:

[0069] The present invention can improve the feature extraction and fusion in the traditional target detection model, use the attention mechanism and spatial pyramid pooling to extract and fuse the features of multi-scale feature structures, make the neural network model adapt to the variable sensory field of different defect features, and realize the accurate positioning and classification of defects of various scales of targets on the appearance of the turbine starter. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0071] Figure 1 It is a flow chart of a method for detecting appearance defects of a turbine starter by multi-scale feature extraction proposed by the present invention;

[0072] Figure 2 It is a multi-scale defect feature extraction and prediction network structure diagram proposed by the present invention;

[0073] Figure 3 It is a structural diagram of the IDE data enhancement module proposed by the present invention;

[0074] Figure 4 It is a structural diagram of the DCBL deformable convolution module proposed in the present invention;

[0075] Figure 5 It is a structural diagram of the CSP module proposed by the present invention;

[0076] Figure 6 It is the structure diagram of the MSPP multi-scale spatial pyramid pooling module proposed by the present invention;

[0077] Figure 7 It is a structural diagram of the SCB separable convolution bottleneck module proposed in the present invention;

[0078] Figure 8 This is the test result of the 550A model turbine starter appearance by the present invention. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0080] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0081] like Figure 1 As shown, the present invention proposes a method for detecting appearance defects of a turbine starter by extracting multi-scale features, which specifically includes the following steps:

[0082] S1, collecting appearance defect data of a turbine starter, and defining image types of the collected appearance defects to obtain a turbine starter appearance defect feature image I;

[0083] As a preferred implementation, step S1 specifically includes:

[0084] S11. Use a two-dimensional industrial camera to collect images of the appearance of the defective turbine starter;

[0085] S12, defining the types of the collected images of the appearance defects of the turbine starter, specifically defining them into seven types: pits, bumps, edge damage, rust, breakage, scratches, and wrinkles;

[0086] S13. After the type definition is completed, data annotation is performed on the appearance defects to obtain a turbine starter appearance defect feature diagram I.

[0087] S2. Construct a data enhancement module, input the turbine starter appearance defect feature map I into the data enhancement module IDE, adjust the parameters of the input turbine starter appearance defect image according to the type of image that the model is good at processing, and obtain the multi-attribute defect feature map I. m ;

[0088] As a preferred implementation, step S2 specifically includes:

[0089] S21, inputting the turbine starter appearance defect feature map I into the data enhancement module IDE, and performing a random judgment cycle of the feature map processing method;

[0090] S22, such as Figure 3 As shown, for the input turbine starter appearance defect feature map I, a random judgment parameter set {a i ∈(0,1)|i=1,2,...,10}; each parameter in the set corresponds to a feature processing method, including: cropping, rotation, flipping, scaling, translation, brightness adjustment, contrast adjustment, grid distortion, adding Gaussian noise and adding ISO noise;

[0091] S23, in the random judgment cycle, set the cumulative parameter N, and give it an initial value of 0, and set the random judgment parameter a i The threshold α, when a i >α, the corresponding feature processing method is executed and the processed feature map is output, and the cumulative parameter N increases by 1; conversely, when a i When ≤α, the corresponding feature processing method is not executed and the accumulation parameter N is not changed;

[0092] S24, after a round of random feature processing, if the cumulative parameter N < 16, it is determined that the loop has not ended, and the random feature processing continues for the feature map I; until N ≥ 16, the loop ends and the 16 feature maps that have been processed before are spliced ​​into a multi-attribute defect feature map I after data enhancement. m And output.

[0093] In this embodiment, it should be noted that the 16 images obtained are processed by a random feature processing method, which does not necessarily include all of the above processing methods.

[0094] S3, build a feature extraction module, and transform the multi-attribute defect feature map I m Input the feature extraction network, extract the features of the data containing defects in the image, convert it into a multi-layer feature map, and generate a multi-scale defect feature map I containing defect features of different scales. MSPP ;

[0095] As a preferred embodiment, Figure 2 As shown, step S3 specifically includes:

[0096] S31, multi-attribute defect feature map I m Input focusing module, perform focusing operation, similar to downsampling operation, to convert high-resolution multi-attribute defect feature map I m The samples are cut into four low-resolution defect feature maps and spliced ​​along the channel direction to form the multi-attribute defect feature map I m The original RGB three channels are expanded to 12 channels, while the width W and height H are shortened to half of the original, and the focus defect feature map I is obtained. f ;

[0097] S32, focusing on defect feature map I f The input is a variable scale feature extraction network composed of DCBL deformable convolution module and CSP convolution residual module alternately, extracts variable scale defect feature information, and outputs variable scale defect feature map I v ;

[0098] In this embodiment, Figure 4 As shown:

[0099] The DCBL module consists of a deformable convolution layer, a batch normalization BN layer, and an activation function connected in sequence; the input focus defect feature map I f After entering the DCBL module, the sampling position is first adaptively adjusted through deformable convolution to extract deformation features in the feature map, so that the neural network model can capture defect features of shape and scale changes; then the internal covariate offset is reduced through batch normalization operation to avoid gradient explosion and disappearance, thereby improving the training speed and stability of the neural network model; finally, nonlinear factors are introduced by adding an activation function (this embodiment uses a leaky linear rectifier function Leaky ReLU as the activation function) to enhance the fitting ability of the neural network model; the formula of the DCBL module is expressed as:

[0100] DCBL(I0)=Leaky ReLU(BN(DCN(I0)));

[0101] Where I0 represents the feature map of the input DCBL module, Leaky ReLU(·) is the leaky linear rectification function, i.e., the activation function; BN(·) is the batch normalization function;

[0102] DCN(·) is a deformable convolution operation, that is, an offset is introduced in the ordinary convolution operation. Specifically, the feature map is subjected to ordinary convolution operation, and an offset generation network is used to process the feature map to generate an offset feature map with 2N channels, which contains the offset information in the x and y directions of each position of the feature map. The final result is then added to the ordinary convolution result. The calculation formula is as follows:

[0103]

[0104] Among them, p0 represents the position of the output feature map, δ n represents an offset of the convolution kernel (for example, a 3×3 convolution kernel has 9 offsets), w(δ n ) is the convolution kernel at offset δ n The weight at I0(p0+δ n ) indicates that the input feature map I0 is at position p0+δ n The pixel value of The offsets are learned by the offset generation network (a normal convolution layer containing 2N convolution kernels). These offsets allow the convolution kernels to be flexibly adjusted, allowing the network to adapt to the shape and structure of the input. is the pixel value of the input feature map at the adjusted position, calculated by bilinear interpolation to cope with non-integer sampling positions;

[0105] In addition, the calculation formula of the leaky linear rectification function Leaky ReLU is as follows:

[0106]

[0107] Where x represents the input and α is a small constant, usually set to 0.01 or 0.1, which controls the slope of negative inputs.

[0108] The CSP modules are divided into two categories, including four different architectures: CSP1_1, CSP1_2, CSP1_3 and CSP2_1. The CSP modules enhance the feature learning ability of the neural network by introducing residual connections and cross-stage feature fusion, which is suitable for the small target detection problem of small-sized defects in turbine starters.

[0109] like Figure 5 As shown in the figure, the first type of CSP1_n (n=1,2,3) modules include three architectures, CSP1_1, CSP1_2, and CSP1_3, which are divided into two branches; in the first branch, the input feature map I1 is extracted through a CBL module (ordinary convolution module), a residual module composed of n residual components, and a sequential architecture composed of an ordinary convolution layer; in the second branch, the input feature map I1 is directly extracted through an ordinary convolution layer, and then the feature maps output by the two branches are spliced, and then batch normalization operations, activation functions and a CBL module are added, and finally a feature map containing rich global and local information is output;

[0110] The architecture of the CBL module is similar to that of the DCBL module, and is considered to be the basic version of the DCBL module. The difference is that CBL only uses ordinary convolution CN(.), and the formula is expressed as:

[0111] CBL(I1)=Leaky ReLU(BN(CN(I0)));

[0112] CN(I1)=∑w(δ n )·I0(p0+δ n );

[0113] The residual component is: the input feature map I2 passes through a feature extraction branch with two CBL modules connected in sequence and a direct edge branch, and the feature maps output by the two branches are spliced ​​and then output; the design of such a residual structure can increase the gradient value of back propagation between layers, avoid the gradient disappearance caused by deepening, so that more fine-grained features can be extracted without worrying about network degradation. The calculation formula of the residual component is as follows:

[0114] Res(I2)=concat(I2,CBL(CBL(I2)))

[0115] Among them, Res(·) represents the residual component, and concat(·,·) represents the concatenation operation;

[0116] The overall architecture of the second type CSP2_n (n=1) module is similar to that of the CSP1_n module, except that the residual module in the first branch is replaced by 2n CBL modules;

[0117] S33, the variable scale defect feature map I v Input CBAM convolutional block attention module, use channel attention mechanism and spatial attention mechanism combined with convolutional neural network to enhance the input variable scale defect feature map I v Channel attention is used to enhance the information interaction between different channels and output the refined feature map I CBAM ;

[0118] The CBAM module uses a dual attention mechanism to make the neural network model pay more attention to important feature channels and spatial positions, thereby fully capturing key information in the features and enhancing the feature representation ability of the neural network model. The specific calculation formula is as follows:

[0119]

[0120] Among them, I CBAM is the refined feature map output by the CBAM module, Represents element-by-element multiplication, M(I v )=M ca (I v )+M sa (I v ) represents the final attention map obtained by adding the results of channel attention and spatial attention; in this embodiment:

[0121] Channel Attention M ca (I v ) is used to determine the importance of different channels of the input feature map. The specific calculation formula is as follows:

[0122] M ca (I v )=σ(MLP(AvgPool(I v ))+MLP(MaxPool(I v )));

[0123] Among them, MaxPool(I v ) and MaxPool(I v ) represent the v Perform global average pooling and global maximum pooling operations. MLP(·) represents a multi-layer perceptron, which consists of two fully connected layers and is used to learn the nonlinear relationship between channels. σ(·) represents the Sigmoid activation function, which is used to convert the output into a weight between 0 and 1. Its calculation formula is

[0124] The spatial attention M sa (I v ) is used to determine the importance of the input feature map in the spatial dimension. The specific calculation formula is as follows:

[0125] M sa (I v )=σ(f 7×7 (Concat[AvgPool(I v );MaxPool(I v )]))

[0126] Among them, f 7×7 Represents a 7×7 convolution kernel, which is used to learn the spatial attention weights of the input feature map. Concat[AvgPool(I v );MaxPool(I v )] indicates that I v The global average pooling and maximum pooling results are spliced ​​together to form a new feature map which is input into the convolutional layer;

[0127] S34, refining feature map I CBAM Input the MSPP multi-scale spatial pyramid pooling module for pooling operation. Using its spatial pyramid structure, it is compatible with feature maps of different sizes and scales, and extracts features of corresponding scales according to different scales of feature maps to improve recognition accuracy. Finally, it outputs a multi-scale defect feature map I containing defect features of different scales. MSPP ;

[0128] More specifically, Figure 6 As shown, in this embodiment, the MSPP module adopts a spatial pyramid structure of four maximum pooling layers, stores the output of the feature map after each maximum pooling layer, and connects the outputs to capture features at different scales; at the same time, a CBL module is connected to the input and output of the MSPP module, and the output of the first and third maximum pooling layers is used as the input of the fourth layer to pay more attention to large and fine targets, so that the neural network model can better adapt to the defect feature recognition of high noise and uneven light; the formula of this module is expressed as:

[0129] I MSPP =CBL(Concat[CVL(I CBAM ); P1; P2; P3; P4]);

[0130] Among them, the output of each pooling layer is P1=MaxPool(I CBAM), P2=MaxPool(P1), P3=MaxPool(P2), P4=MaxPool(Concat[P1;P3]).

[0131] S4, build an enhanced feature fusion module to transform the multi-scale defect feature map I MSPP Input the enhanced feature extraction network to obtain higher-dimensional defect feature information, and aggregate multi-scale features through the feature fusion network to generate a multi-scale fusion defect feature map I M ;

[0132] As a preferred implementation, step S4 specifically includes:

[0133] S41, multi-scale defect feature map I MSPP The input is an enhanced feature fusion network composed of the CBL module, the CARAFE content-aware feature reorganization operator and the CSP2_1 module connected in sequence. The CARAFE operator is used to dynamically adjust the weight according to the content of the input feature map to perform efficient and accurate feature upsampling to improve the prediction ability of the neural network model for dense defects, and the enhanced feature defect feature map I is output. E , the specific calculation formula is:

[0134] I E =CSP2_1(CARAFE(CBL(I MSPP )));

[0135] The CARAFE operator first predicts the upsampling kernel from the input feature map using convolution operation through the kernel prediction module, and then uses the predicted upsampling kernel to perform weighted reorganization on the input feature map through the content-aware reorganization module to generate the upsampled feature map.

[0136] In this embodiment, the specific operation steps of the kernel prediction module in the CARAFE operator are as follows: First, a 1×1 convolution C1 is used to reduce the channel dimension C of the input feature map of shape (N, C, H, W) to N is the batch size, indicating that there are N independent feature maps in this batch. H and W are the height and width, respectively, indicating the number of pixels in each feature map in the vertical and horizontal directions. Then, a K×K convolution c2 is used to map the feature map to S. 2 ×K 2 channel, S is the upsampling factor; then pixel shuffle is performed to rearrange the pixels in the feature map so that the pixels of each low-resolution feature image block are recombined into a larger area of ​​the high-resolution feature image, whose shape is Finally, the softmax function is applied for normalization to obtain the final upsampling kernel K P ;

[0137] The specific operation steps of the content-aware reconstruction module are as follows: zero-fill the input feature map to change its shape to Then expand it to the shape of (N,C,H,W,K 2 ), and then with the upsampling kernel K P Perform matrix multiplication to obtain feature map I O , whose shape is (N,H,W,C), and finally I O Perform pixel shuffling to obtain the final upsampled feature map I' O ;

[0138] S42, enhancing the characteristic defect feature map I E The input is a feature fusion network composed of a CBAM module and an SCB separable convolution bottleneck module. The CBAM module is used to further focus on the important feature channels and spatial position information in the feature map to obtain an enhanced refined defect feature map I' E , and then use the SCB module to extract and fuse the multi-scale feature structure, so that the neural network model can adapt to the variable sensory field of different defect features, and finally output the multi-scale fusion defect feature map I M ;

[0139] More specifically, in step S42, if Figure 7 As shown in Figure 2, the SCB module combines separable convolution and spatial pyramid structure to achieve efficient feature extraction and fusion while reducing computational costs. The formula is: M =Leaky ReLU(BN(Concat[Maxpool(I′ E ); 2SConv(I′ E ); 2SConv(2SConv(I′ E ))]));

[0140] Among them, Leaky ReLU (·) is the leaky linear rectification function, that is, the activation function; BN (·) is the batch normalization function; 2SConv (I ′ E ) represents the enhanced refining defect characteristic diagram I′ E Perform two-step separable convolution, i.e., first perform depthwise convolution and then perform pointwise convolution; the depthwise convolution is to apply an independent convolution kernel to each input channel of the input feature map to extract the features of each channel one by one, and the gradual convolution is to recombine the features of each channel of the input feature map through a 1×1 convolution, perform feature fusion between channels, fuse these separated channel information again, and restore the relationship between global features;

[0141] In this embodiment, the depth convolution does not need to perform global operations between all channels, so the amount of calculation is significantly reduced. The specific operation process is: for a feature map with an input shape of (N, C, H, W), a feature map with a shape of (K H ,K W ,C,M) convolution kernel (size is K H ×K W ) performs convolution calculation, and the output shape is feature map, where P is padding and S is the stride;

[0142] For point-by-point convolution, the features of each channel of the input feature map are recombined through a 1×1 convolution to fuse the features between channels. The specific operation process is as follows: for a feature map with an input shape of (N, M, H', W'), A convolution kernel with a shape of (1,1,M,C') is used for convolution calculation, and a feature map with an output shape of (N,C',H',W') is output;

[0143] Through the above process, step S42 sequentially enhances and refines the defect feature map I′ E Perform maximum pooling, one separable convolution and two separable convolutions, and fuse the output through batch normalization and activation function to output a multi-scale fusion defect feature map I M .

[0144] S5. Construct a defect prediction module to fuse the multi-scale defect feature map I M Input into the defect detection network, the bounding box prediction results and classification results of multi-scale defects on the appearance of the turbine starter are detected;

[0145] As a preferred implementation, step S5 specifically includes:

[0146] S51, multi-scale fusion defect feature map I M Upsample through the CARAFE operator to generate the feature map I′ M , and input three defect prediction branches of different scales for calculation respectively; the three defect prediction branches are used for defect prediction of large, medium and small scales respectively;

[0147] S52, upsampled feature map I′ MDifferent numbers of SCB modules are passed through the three branches respectively. The first branch does not contain an SCB module, the second branch passes through an SCB module, and the third branch also passes through an SCB module, and feature fusion is performed to adapt to defect prediction of different scales; then they are processed by the CSP2_1 module and the CBAM module respectively, focusing on defects of corresponding scales, and finally, a common convolutional layer is used to output the prediction results of defects of large, medium and small scales respectively; the prediction results include the predicted defect position rectangular bounding box, confidence and defect category classification probability.

[0148] S6. For the defect detection model of multi-scale defects, similarity and union intersection loss are used as loss functions to train and update the weights of the model, and the final defect detection model is obtained;

[0149] As a preferred implementation, step S6 specifically includes:

[0150] Adopt bounding box regression loss L IOU To supervise and optimize the defect detection training process, and adopt the classifier loss L CLA To optimize the defect classification performance and use the similarity loss L SIM To evaluate the detection accuracy of the object, that is, the confidence, the joint loss L reg It can be expressed as:

[0151] L reg =λ1L IOU +λ2L SIM +λ3L CLA ;

[0152] Among them, λ1, λ2 and λ3 represent the proportion of each loss function;

[0153] The bounding box regression loss L IOU The CIOU loss is used to evaluate the prediction accuracy of the predicted defect position rectangular bounding box, and the formula is expressed as:

[0154]

[0155] Where IOU is the intersection-over-union ratio between the predicted box and the true box, d is the distance between the center point of the predicted box and the center point of the true box, c is the diagonal length of the smallest closed rectangle containing the predicted box and the true box, α is a weight parameter used to balance the impact of center point distance loss and aspect ratio loss, and v is a correction factor used to measure the difference in aspect ratio between the predicted box and the true box, and further adjust the loss function;

[0156] The classifier loss L CLA VFL loss is used to evaluate the classification accuracy of defect categories, and the formula is expressed as:

[0157] LCLA = -t log(p)(1-p t ) γ ;

[0158] Among them, t is the true label of the target category, p is the probability predicted by the model as this category, and p t is the adjusted predicted probability, defined as in represents element-wise division, α′ is a balancing factor used to adjust the weight of negative samples, and γ is a focusing parameter used to reduce the weight of easy-to-classify samples;

[0159] The similarity loss L SIM Gaussian distribution is used to evaluate the detection accuracy of objects, that is, confidence, and the formula is expressed as:

[0160] L SIM =1-ln SIM(A,B);

[0161] Where A=(cx a ,cy a ,w a ,h a ), cx a ,cy a Represents the coordinates of the upper left corner of the rectangular bounding box A of the predicted defect location, w a ,h a represents the width and height of the bounding box A of the predicted defect, B = (cx b ,cy b ,w b ,h b ), cx b ,cy b represents the coordinates of the upper left corner of the actual defect bounding box B, w b ,h b represents the width and height of the actual defect bounding box B, SIM(·) can be expressed as:

[0162]

[0163] in, Represents the bi-norm of a vector.

[0164] In summary, the construction and training of multi-scale defect feature extraction and prediction network have been realized. The present invention can improve the feature extraction and fusion in the traditional target detection model, use the attention mechanism and spatial pyramid pooling to extract and fuse the features of multi-scale feature structures, make the neural network model adapt to the variable sensory field of different defect features, and realize the accurate positioning and classification of defects of various scales of targets on the appearance of turbine starters. The final result is as follows: Figure 8shown.

[0165] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0166] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for detecting appearance defects of a turbine starter by extracting multi-scale features, characterized in that: Specifically include: S1, collecting appearance defect data of a turbine starter, and defining image types of the collected appearance defects to obtain a turbine starter appearance defect feature image I; S2. Construct a data enhancement module, input the turbine starter appearance defect feature map into the data enhancement module IDE, adjust the parameters of the input turbine starter appearance defect image according to the image type that the model is good at processing, and obtain the multi-attribute defect feature map I m ; S3, build a feature extraction module, and transform the multi-attribute defect feature map I m Input the feature extraction network, extract the features of the data containing defects in the image, convert it into a multi-layer feature map, and generate a multi-scale defect feature map I containing defect features of different scales. MSPP ; S4, build an enhanced feature fusion module to transform the multi-scale defect feature map I MSPP Input the enhanced feature extraction network to obtain higher-dimensional defect feature information, and aggregate multi-scale features through the feature fusion network to generate a multi-scale fusion defect feature map I M ; S5. Construct a defect prediction module to fuse the multi-scale defect feature map I M Input into the defect detection network, the bounding box prediction results and classification results of multi-scale defects on the appearance of the turbine starter are detected; S6. For the defect detection model of multi-scale defects, similarity and union intersection loss are used as loss functions to train and update the weights of the model, and the final defect detection model is obtained.

2. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 1, characterized in that: Step S1 specifically includes: S11. Use a two-dimensional industrial camera to collect images of the appearance of the defective turbine starter; S12, defining the types of the collected images of the appearance defects of the turbine starter, specifically defining them into seven types: pits, bumps, edge damage, rust, breakage, scratches, and wrinkles; S13. After the type definition is completed, data annotation is performed on the appearance defects to obtain a turbine starter appearance defect feature diagram I.

3. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 1, characterized in that: Step S2 specifically includes: S21, inputting the turbine starter appearance defect feature map into the data enhancement module IDE, and performing a random judgment cycle of the feature map processing method; S22, for the input turbine starter appearance defect feature image I, generate a random judgment parameter set {a i ∈(0, 1)|i=1, 2, ..., 10}; each parameter in the set corresponds to a feature processing method, including: cropping, rotation, flipping, scaling, translation, brightness adjustment, contrast adjustment, grid distortion, adding Gaussian noise and adding ISO noise; S23, in the random judgment cycle, set the cumulative parameter N, and give it an initial value of 0, and set the random judgment parameter a i The threshold α, when a i >α, the corresponding feature processing method is executed and the processed feature map is output, and the cumulative parameter N increases by 1; conversely, when a i When ≤α, the corresponding feature processing method is not executed and the accumulation parameter N is not changed; S24, after a round of random feature processing, if the cumulative parameter N < 16, it is determined that the loop has not ended, and the random feature processing continues for the feature map I; until N ≥ 16, the loop ends and the 16 feature maps that have been processed before are spliced ​​into a multi-attribute defect feature map I after data enhancement. m And output.

4. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 1, characterized in that: Step S3 specifically includes: S31, multi-attribute defect feature map I m Input the focusing module to perform focusing operation and convert the high-resolution multi-attribute defect feature map I m The samples are cut into four low-resolution defect feature maps and spliced ​​along the channel direction to form the multi-attribute defect feature map I m The original RGB three channels are expanded to 12 channels, while the width W and height H are shortened to half of the original, and the focus defect feature map I is obtained. f ; S32, focusing on defect feature map I f The input is a variable scale feature extraction network composed of DCBL deformable convolution module and CSP convolution residual module alternately, extracts variable scale defect feature information, and outputs variable scale defect feature map I v ; S33, the variable scale defect feature map I v Input CBAM convolutional block attention module, use channel attention mechanism and spatial attention mechanism combined with convolutional neural network to enhance the input variable scale defect feature map I v Channel attention is used to enhance the information interaction between different channels and output the refined feature map I CBAM ; S34, refining feature map I CBAM Input the MSPP multi-scale spatial pyramid pooling module for pooling operation. Using its spatial pyramid structure, it is compatible with feature maps of different sizes and scales, and extracts features of corresponding scales according to different scales of feature maps to improve recognition accuracy. Finally, it outputs a multi-scale defect feature map I containing defect features of different scales. MSPP .

5. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 4, characterized in that: In step S32: The DCBL module consists of a deformable convolution layer, a batch normalization BN layer, and an activation function connected in sequence; the input focus defect feature map I f After entering the DCBL module, the sampling position is adaptively adjusted through deformable convolution to extract the deformation features in the feature map, and then the internal covariate offset is reduced through batch normalization to avoid gradient explosion and disappearance. Finally, nonlinear factors are introduced by adding activation functions to enhance the fitting ability of the neural network model. The formula of the DCBL module is expressed as: DCBL(I0)=Leaky ReLU(BN(DCN(I0))); Among them, 10 represents the feature map of the input DCBL module, Leaky ReLU (·) is the leaky linear rectification function, that is, the activation function; BN (·) is the batch normalization function, and DCN (·) is the deformable convolution operation; CSP modules are divided into two categories, including four different architectures: CSP1_1, CSP1_2, CSP1_3, and CSP2_1; The first type of CSP1_n (n=1, 2, 3) modules includes three architectures, CSP1_1, CSP1_2, and CSP1_3, which are divided into two branches. In the first branch, the input feature map I1 is extracted through a CBL ordinary convolution module, a residual module composed of n residual components, and a sequential architecture composed of an ordinary convolution layer. In the second branch, the input feature map I1 is directly extracted through an ordinary convolution layer, and then the feature maps output by the two branches are spliced, and then batch normalization operations, activation functions and a CBL module are added to finally output a feature map containing rich global and local information. The architecture of the CBL module is similar to that of the DCBL module, except that the CBL module only uses ordinary convolution; The residual component is: the input feature map I2 is passed through a feature extraction branch with two CBL modules connected in sequence and a direct edge branch, and the feature maps output by the two branches are spliced ​​and then output; The overall architecture of the second type CSP2_n (n=1) module is similar to that of the CSP1_n module, except that the residual module in the first branch is replaced by 2n CBL modules.

6. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 4, characterized in that: In step S34: The MSPP module adopts a spatial pyramid structure with four maximum pooling layers. It stores the output of each maximum pooling layer after the feature map is passed and connects the outputs to capture features at different scales. At the same time, a CBL module is connected to the input and output of the MSPP module, and the output of the first and third maximum pooling layers is used as the input of the fourth layer. The formula is expressed as: I MSPP =CBL(Concat[CBL(I CBAM );P1;P2;P3;P4]); Among them, the output of each pooling layer is P1=MaxPool(I CBAM ), P2=MaxPool(P1), P3=MaxPool(P2), P4=MaxPool(Concat[P1;P3]).

7. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 1, characterized in that: Step S4 specifically includes: S41, multi-scale defect feature map I MSPP The input is an enhanced feature fusion network composed of the CBL module, the CARAFE content-aware feature reorganization operator and the CSP2_1 module connected in sequence. The CARAFE operator is used to dynamically adjust the weight according to the content of the input feature map to perform efficient and accurate feature upsampling to improve the prediction ability of the neural network model for dense defects, and the enhanced feature defect feature map I is output. E , the specific calculation formula is: I E =CSP2_1(CARAFE(CBL(I MSPP ))); The CARAFE operator first predicts the upsampling kernel from the input feature map using convolution operation through the kernel prediction module, and then uses the predicted upsampling kernel to perform weighted reorganization on the input feature map through the content-aware reorganization module to generate the upsampled feature map. S42, enhancing the characteristic defect feature map I E The input is a feature fusion network composed of a CBAM module and an SCB separable convolution bottleneck module. The CBAM module is used to further focus on the important feature channels and spatial position information in the feature map to obtain an enhanced refined defect feature map I′. E , and then use the SCB module to extract and fuse the multi-scale feature structure, so that the neural network model can adapt to the variable sensory field of different defect features, and finally output the multi-scale fusion defect feature map I M .

8. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 7, characterized in that: In step S42: The SCB module combines separable convolution and spatial pyramid structure to achieve efficient feature extraction and fusion while reducing computational costs. The formula is expressed as: I M =Leaky ReLU(BN(Concat[Maxpool(I′ E );2SConv(I′ E );2SConv(2SConv(I′ E ))])); Among them, Leaky ReLU (·) is the leaky linear rectification function, that is, the activation function; BN (·) is the batch normalization function; 2SConv (I ′ E ) represents the enhanced refining defect characteristic diagram I′ E Perform two-step separable convolution, i.e., first perform depthwise convolution and then perform pointwise convolution; the depthwise convolution is to apply an independent convolution kernel to each input channel of the input feature map to extract the features of each channel one by one, and the gradual convolution is to recombine the features of each channel of the input feature map through a 1×1 convolution, perform feature fusion between channels, fuse these separated channel information again, and restore the relationship between global features; Enhanced refining defect feature diagram I′ E Perform maximum pooling, one separable convolution and two separable convolutions, and fuse the output through batch normalization and activation function to output a multi-scale fusion defect feature map I M .

9. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 1, characterized in that: Step S5 specifically includes: S51, multi-scale fusion defect feature map I M Upsample through the CARAFE operator to generate the feature map I′ M , and input three defect prediction branches of different scales for calculation respectively; the three defect prediction branches are used for defect prediction of large, medium and small scales respectively; S52, upsampled feature map I' M Different numbers of SCB modules are passed through the three branches respectively. The first branch does not contain an SCB module, the second branch passes through an SCB module, and the third branch also passes through an SCB module, and feature fusion is performed to adapt to defect prediction of different scales; then they are processed by the CSP2_1 module and the CBAM module respectively, focusing on defects of corresponding scales, and finally, a common convolutional layer is used to output the prediction results of defects of large, medium and small scales respectively; the prediction results include the predicted defect position rectangular bounding box, confidence and defect category classification probability.

10. The method for detecting appearance defects of a turbine starter by multi-scale feature extraction according to claim 9, characterized in that: Step S6 specifically includes: Adopt bounding box regression loss L IOU To supervise and optimize the defect detection training process, and adopt the classifier loss L CLA To optimize the defect classification performance and use the similarity loss L SIM To evaluate the detection accuracy of the object, that is, the confidence, the joint loss L reg It can be expressed as: L reg =λ1L IOU +λ2L SIM +λ3L CLA ; Among them, λ1, λ2 and λ3 represent the proportion of each loss function; The bounding box regression loss L IOU The CIOU loss is used to evaluate the accuracy of the predicted defect location rectangular bounding box, and the formula is expressed as: Where IOU is the intersection-over-union ratio between the predicted box and the true box, d is the distance between the center point of the predicted box and the center point of the true box, c is the diagonal length of the smallest closed rectangle containing the predicted box and the true box, α is a weight parameter used to balance the impact of center point distance loss and aspect ratio loss, and v is a correction factor used to measure the difference in aspect ratio between the predicted box and the true box, and further adjust the loss function; The classifier loss L CLA VFL loss is used to evaluate the classification accuracy of defect categories, and the formula is expressed as: L CLA n-leave(p)(1-p t )γ ( Among them, t is the true label of the target category, p is the probability predicted by the model as this category, and pt is the adjusted predicted probability, defined as in represents element-wise division, α′ is a balancing factor used to adjust the weight of negative samples, and γ is a focusing parameter used to reduce the weight of easy-to-classify samples; The similarity loss L SIM Gaussian distribution is used to evaluate the detection accuracy of objects, that is, confidence, and the formula is expressed as: L SIM =1-lnSIM(A,B); Where A=(cx a ,cy a , w a ,h a ), cx a ,cy a Represents the coordinates of the upper left corner of the rectangular bounding box A of the predicted defect location, w a ,h a represents the width and height of the bounding box A of the predicted defect, B = (cx b ,cy b , w b ,h b ), cx b ,cy b represents the coordinates of the upper left corner of the actual defect bounding box B, w b ,h b represents the width and height of the actual defect bounding box B, SIM(·) can be expressed as: in, Represents the bi-norm of a vector.

Images