Commutator inner side image defect detection method based on fusible feature pyramid

By adopting a structure based on a fusion feature pyramid in the image defect detection of the commutator, fusing multi-level and multi-scale features, the problem of limited detection accuracy and speed in the prior art is solved, and more efficient feature extraction and Detection effect.

CN116152226BActive Publication Date: 2024-11-22DONGGUAN POLYTECHNIC +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310350973.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-04
Publication Date
2024-11-22
Estimated Expiration
2043-04-04

AI Technical Summary

Technical Problem

The existing feature pyramids cannot effectively fuse multi-level features and multi-scale features in the image defect detection on the inner side of the commutator, resulting in limited detection accuracy and speed.

Method used

Using a fusion-based feature pyramid structure, multi-level features are extracted through the feature extraction network and feature splicing module of the multi-scale feature pyramid network is input to perform feature splicing and encoding and decoding, and combining feature aggregation module for multi-level and multi-scale features. Form a fusible feature pyramid and add a convolutional layer to the convolutional neural network for detection.

Benefits of technology

It improves detection accuracy and speed, obtains richer and more effective multi-level feature information, breaks through the limitations of single-level feature representation after convolution of feature pyramids, and is suitable for image defect detection on the inner side of the commutator.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116152226B_ABST
    Figure CN116152226B_ABST
Patent Text Reader

Abstract

The present invention relates to a commutator inner side image defect detection method based on a fusible feature pyramid, comprising: step S1, a feature extraction network inputs the extracted multi-level features into a feature splicing module of a multi-scale feature pyramid network; step S2, the feature splicing module splices the input multi-level features, and inputs the spliced ​​multi-level features into a codec module in the multi-scale feature pyramid network; step S3, the codec module generates features with multiple scales according to the features input by the feature splicing module; step S4, the feature aggregation module of the multi-scale feature pyramid network aggregates the multi-level features in step S1 and the multi-scale features in step S3 into a fusible feature pyramid, and the detection network obtains a convolutional neural network by adding several convolutional layers after the fusible feature pyramid; step S5, training the convolutional neural network to obtain a detection result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of target detection, and in particular to a method for detecting defects in an inner side image of a commutator based on a fusible feature pyramid. Background Art

[0002] In recent years, feature pyramids of different structures have been proposed and their functions have been continuously improved. However, the feature pyramids currently used only construct feature pyramid modules based on feature extraction networks with multi-scale pyramid structures. The multi-scale of these feature pyramids is mainly designed for classification tasks, which is not very suitable for detecting defective images in the field of commutator inner side image defect detection. It has limitations for detection tasks and cannot fuse multi-level features and multi-scale features to obtain richer feature information and improve the accuracy and speed of commutator inner side image defect detection.

[0003] Chinese patent ZL201910242976.2 discloses a feature-enhanced target detection method based on a feature pyramid network. Its technical feature is to add a top-down module TDM, a bottom-up module BUM and a fusion extension module FEM to the backbone network model used before sending it to the target detector, thereby forming a new enhanced feature pyramid network eFPN. However, it only takes into account the features output by the backbone network, and the feature information obtained is less and not accurate enough. Summary of the invention

[0004] To this end, the present invention provides a commutator inner side image defect detection method based on a fusible feature pyramid, which can solve the technical problem that multi-level features and multi-scale features cannot be fused to solve the limitations of the feature pyramid in commutator inner side image defect detection.

[0005] To achieve the above object, the present invention provides a method for detecting defects in an image of a commutator inner side based on a fusible feature pyramid, comprising:

[0006] Step S1, the feature extraction network extracts multi-level features of the inner image of the commutator to be detected, and inputs the extracted multi-level features into the feature splicing module of the multi-scale feature pyramid network;

[0007] Step S2, the feature splicing module splices the input multi-level features, and inputs the spliced ​​multi-level features into the encoding and decoding module in the multi-scale feature pyramid network;

[0008] Step S3, the encoding and decoding module generates features with multiple scales according to the features input by the feature splicing module;

[0009] Step S4, the feature aggregation module of the multi-scale feature pyramid network aggregates the multi-level features in step S1 and the multi-scale features in step S3 into a fusible feature pyramid, and the detection network obtains a convolutional neural network by adding several convolution layers after the fusible feature pyramid, wherein the aggregation process of the multi-level features and the multi-scale features by the feature aggregation module includes:

[0010] Step S41, feature-connecting the multi-scale features and the multi-level features along the channel dimensions of equal proportions to form a feature block;

[0011] Step S42, compressing the global spatial information into a channel description, and performing global average pooling on the feature blocks;

[0012] Step S43, comprehensively capturing channel dependencies to utilize information gathered in the compression operation;

[0013] Step S44, re-weighting the feature blocks and outputting them;

[0014] Step S5, training the convolutional neural network, and inputting the picture set of the inner side image of the commutator to be detected into the trained convolutional neural network in sequence to obtain a set of prediction boxes, marking the coordinate values ​​of the prediction boxes on the pictures, and obtaining the detection results.

[0015] Furthermore, in step S42, the feature aggregation module compresses the global spatial information into a channel description through a Squeeze operation, and performs global average pooling on the feature blocks. Formally, the statistics z∈R C is generated by shrinking U in the spatial dimension W×H, where the cth element of z is,

[0016]

[0017] Among them, u c (i, j) represents the statistical information of the cth element at the (i, j)th position in the spatial dimension W×H.

[0018] Furthermore, in step S43, the feature aggregation module fully captures channel dependencies through the Excitation operation, that is, aggregates features.

[0019] s=F ex (z, W) = σ(W2δ(W1z))

[0020] Among them, σ represents the ReLU function, δ represents the sigmoid function, R represents the reduction ratio, and the bottleneck of two fully connected (FC) layers is used to parameterize the gate mechanism, that is, the dimension reduction layer parameter is W1, the dimension reduction ratio is r, and W2 is the dimension increase layer to re-weight the output X of U.

[0021] Furthermore, in step S44, the feature aggregation module re-weights and outputs the feature blocks, wherein:

[0022]

[0023] Among them, X represents the final output of the SE block, Representative fusion pyramid features With s c The product of .

[0024] Furthermore, in step S1, the feature extraction network is constructed by stacking a Darknet-53 network using a shuttle residual structure, and a downsampling operation is performed using a Mobilenet structure with a step size of 2, wherein the shuttle residual structure is formed by connecting the Darknet structure in YOLOv3 with the residual structure.

[0025] Furthermore, in step S2, the feature concatenation module performs an upsampling operation on the input multi-level features, and scales the deep features to the same scale before performing the connection operation.

[0026] Furthermore, in step S3, the encoding and decoding module includes an encoder and a decoder, and uses continuous 3×3 convolutional layers to perform convolution downsampling operations on the features input by the feature splicing module, wherein the feature output of each layer of the encoder is used as a reference feature set, and the decoder is a series of 3×3 convolutional layers with a step size of 1, and the decoder branch includes an upsampling layer and a pixel-by-pixel summation operation.

[0027] Furthermore, in step S5, the training of the convolutional neural network includes:

[0028] Step S51, the detection network obtains a number of three-dimensional tensor-encoded prediction boxes, objects, and class predictions, and performs box regression and class regression in combination with a loss function;

[0029] In step S52, the Soft-NMS method is used to filter the highly overlapping prediction boxes to obtain the final prediction bounding box.

[0030] Furthermore, in step S51, the detection network normalizes the parameters x, y, w and h of the prediction box, where:

[0031]

[0032]

[0033] The detection network divides the commutator image to be detected into a grid of s×s, where S=s, (x0, y0) is the predicted coordinate of the center point of the bounding box, the grid coordinates of the center point are (row, col), and width img is the width of the commutator image to be detected, height img The height of the commutator image to be detected, width box is the width of the predicted bounding box, height box is the width of the predicted bounding box, where the predicted bounding box contains four parameters: x, y, w, and h. x and y refer to the X and Y coordinates of the center point of the predicted box, w and h refer to the width and height of the predicted box. The point (x0, y0) is the center point of the bounding box, and its grid coordinates are (row, col). img and height img Represents the width and height of the entire image, width box and height box Represents the width and height of the bounding box.

[0034] Furthermore, in step S52, the Soft-NMS algorithm is:

[0035]

[0036] Among them, B represents the initial list; S represents the detection score, and the function f(IOU(M,b i ))for:

[0037]

[0038] Among them, M represents the detection box with the highest score, b i Represents the processed box, N t represents the threshold and a represents the parameter.

[0039] Compared with the prior art, the beneficial effect of the present invention lies in that the present invention provides a multi-scale feature pyramid network, which can process the multi-level features extracted by the feature extraction network to generate multi-scale features, and fuse the multi-scale features with the multi-level features to form a fusible feature pyramid. The fusible feature pyramid structure not only takes into account the features output by the backbone network, but also adopts a new feature pyramid structure based on a U-shaped codec. In addition, the present invention updates and transforms the traditional residual block and proposes a new shuttle-shaped residual block to add a depth-separable convolution and maintain the feature expression ability of the residual block while reducing the amount of calculation, which is more suitable for the detection of image defects on the inner side of the commutator and improves the detection accuracy and speed.

[0040] In particular, the feature aggregation module forms a fusible feature pyramid by fusing multi-level features and multi-scale features. The feature extraction of the fusible feature pyramid is more standardized, and richer and more effective multi-level feature information can be further obtained, breaking through the limitation that the feature map after the convolution of the feature pyramid is mainly composed of feature representation of a single level, and is more suitable for the detection of image defects on the inner side of the commutator.

[0041] In particular, the shuttle residual structure is a fusion of the traditional convolution module in MobileNet, which is designed as a deep convolution structure and a point convolution structure. The feature extraction network is constructed by stacking the Darknet-53 network using the shuttle residual structure, which makes the feature extraction more comprehensive and can improve the detection efficiency by reducing the amount of calculation while keeping the accuracy basically unchanged.

[0042] In particular, the purpose of adding upsampling layers and pixel-by-pixel summation operations on the decoder branches is to keep the feature map size the same, enhance learning capabilities and maintain feature smoothness.

[0043] In particular, the detection network normalizes the prediction box parameters, which can reduce the impact of singular sample data and speed up network learning.

[0044] In particular, the NMS algorithm directly sets the IOU of the largest box with a score greater than a certain threshold to zero, and the algorithm is relatively rough. Therefore, the detection network uses the Soft-NMS algorithm. The detection network sets a threshold and removes the prediction box with a category confidence lower than the threshold to obtain the final prediction box, thereby improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a schematic diagram of the structure of a commutator inner side image defect detection system based on a fusible feature pyramid according to an embodiment of the invention;

[0046] Figure 2 This is a flow chart of a method for detecting defects in an inner side image of a commutator based on a fusible feature pyramid according to an embodiment of the invention;

[0047] Figure 3 This is a schematic diagram of the specific structure of the feature aggregation module of an embodiment of the invention;

[0048] Figure 4 This is a schematic diagram of the shuttle residual structure of an embodiment of the invention;

[0049] Figure 5 This is a schematic diagram of the specific structure of the splicing module of the embodiment of the invention;

[0050] Figure 6 This is a schematic diagram of the specific structure of the encoding and decoding module of the embodiment of the invention;

[0051] Figure 7 The test results of the LSVM+DPM algorithm and the RCNN algorithm in this paper are shown in the embodiment of the invention;

[0052] Figure 8 The test results of the Faster R-CNN algorithm and the Faster R-CNN_ResNet_101 algorithm in this paper are presented in the embodiments of the invention;

[0053] Fig. 9 The test results of the Multi-task CNN algorithm and the YOLOv2 algorithm in this paper are presented;

[0054] Fig.10 The test results of the Aston-EAS algorithm and the F-ConvNet algorithm in this paper are presented as examples of the invention;

[0055] Fig.11 The test results of the MonoGRNet algorithm and the LTN algorithm in this paper are presented as examples of the invention;

[0056] Fig.12 The test results of the YOLOv3 algorithm of the invention embodiment and the algorithm of this article are shown in this article. DETAILED DESCRIPTION

[0057] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0059] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0060] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] See also Figure 1 As shown, it is a schematic diagram of the structure of a commutator inner side image defect detection system based on a fusible feature pyramid according to an embodiment of the present invention, including:

[0062] A feature extraction network, which is used to extract multi-level features of the inner side image of the commutator to be inspected;

[0063] A multi-scale feature pyramid network, which is connected to the feature extraction network, and includes a feature splicing module, a codec module and a feature aggregation module, wherein the feature splicing module is used to splice features from multiple different levels of the feature extraction network, the codec module is used to generate multi-scale features according to the features input by the feature splicing module, and the feature aggregation module is used to fuse the multi-level features and the multi-scale features into a fusible feature pyramid;

[0064] A detection network, which is connected to the multi-scale feature pyramid network, is used to obtain a prediction frame of the image defect on the inner side of the commutator to be detected.

[0065] Please read Figure 2 As shown, it is a flow chart of a method for detecting defects of an inner side image of a commutator based on a fusible feature pyramid according to an embodiment of the present invention, comprising:

[0066] Step S1, the feature extraction network extracts multi-level features of the inner image of the commutator to be detected, and inputs the extracted multi-level features into the feature splicing module of the multi-scale feature pyramid network;

[0067] Step S2, the feature splicing module splices the input multi-level features, and inputs the spliced ​​multi-level features into the encoding and decoding module in the multi-scale feature pyramid network;

[0068] Step S3, the encoding and decoding module generates features with multiple scales according to the features input by the feature splicing module;

[0069] Step S4, the feature aggregation module of the multi-scale feature pyramid network aggregates the multi-level features in step S1 and the multi-scale features in step S3 into a fusible feature pyramid, and the detection network obtains a convolutional neural network by adding several convolution layers after the fusible feature pyramid, wherein the aggregation process of the multi-level features and the multi-scale features by the feature aggregation module includes:

[0070] Step S41, feature-connecting the multi-scale features and the multi-level features along the channel dimensions of equal proportions to form a feature block;

[0071] Step S42, compressing the global spatial information into a channel description, and performing global average pooling on the feature blocks;

[0072] Step S43, comprehensively capturing channel dependencies to utilize information gathered in the compression operation;

[0073] Step S44, re-weighting the feature blocks and outputting them;

[0074] Step S5, training the convolutional neural network, and inputting the picture set of the inner side image of the commutator to be detected into the trained convolutional neural network in sequence to obtain a set of prediction boxes, marking the coordinate values ​​of the prediction boxes on the pictures, and obtaining the detection results.

[0075] In step S42, the feature aggregation module compresses the global spatial information into a channel description through a Squeeze operation, and performs global average pooling on the feature blocks. Formally, the statistics z∈R C is generated by shrinking U in the spatial dimension W×H, where the cth element of z is,

[0076]

[0077] Among them, u c (i, j) represents the statistical information of the cth element at the (i, j)th position in the spatial dimension W×H.

[0078] In step S43, the feature aggregation module fully captures channel dependencies through the Excitation operation, that is, aggregates features.

[0079] s=F ex (z, W) = σ(W2δ(W1z))

[0080] Among them, σ represents the ReLU function, δ represents the sigmoid function, R represents the reduction ratio, and the bottleneck of two fully connected (FC) layers is used to parameterize the gate mechanism, that is, the dimension reduction layer parameter is W1, the dimension reduction ratio is r, and W2 is the dimension increase layer parameter, so as to re-weight the output X of U:

[0081] In step S44, the feature aggregation module re-weights and outputs the feature blocks, wherein:

[0082]

[0083] Among them, X represents the final output of the SE block, Representative fusion pyramid features With s c The product of .

[0084] Specifically, the feature aggregation module forms a fusible feature pyramid by fusing multi-level features and multi-scale features. The feature extraction of the fusible feature pyramid is more standardized, and richer and more effective multi-level feature information can be further obtained, breaking through the limitation that the feature map after the convolution of the feature pyramid is mainly composed of feature representation of a single level, and is more suitable for the detection of image defects on the inside of the commutator.

[0085] Specifically, the present invention does not specifically limit the structure of the feature aggregation module, as long as it can meet its functional requirements. The embodiment of the present invention provides a preferred specific structure of the feature aggregation module, please refer to Figure 3 As shown, it is a schematic diagram of the specific structure of the feature aggregation module of an embodiment of the present invention.

[0086] In step S1, the feature extraction network is constructed by stacking a Darknet-53 network using a shuttle residual structure, and a downsampling operation is performed using a Mobilenet structure with a step size of 2, wherein the shuttle residual structure is formed by connecting the Darknet structure in YOLOv3 with the residual structure.

[0087] Specifically, the shuttle residual structure is a fusion of the traditional convolution module in MobileNet, which is designed as a deep convolution structure and a point convolution structure. The feature extraction network is constructed by stacking the shuttle residual structure using the Darknet-53 network, which makes the feature extraction more comprehensive and can improve the detection efficiency by reducing the amount of calculation while keeping the accuracy basically unchanged.

[0088] Specifically, see Figure 4 As shown, it is a schematic diagram of the shuttle residual structure of an embodiment of the present invention. The present invention does not limit the shuttle residual structure, as long as it can meet the feature extraction requirements. The embodiment of the present invention provides a preferred embodiment, including, first setting the number of input channels to Ci, using a 1×1 convolution block to upgrade the dimension to t×Ci, then filtering through a 3×3 filter, and finally reducing the dimension to Ci through a 1×1 convolution block.

[0089] In step S2, the feature concatenation module performs an upsampling operation on the input multi-level features and scales the deep features to the same scale before performing the concatenation operation.

[0090] Specifically, the present invention does not specifically limit the structure of the feature splicing module, as long as it can splice features of different levels. The embodiment of the present invention provides a preferred specific structure of the feature splicing module, please refer to Figure 5 As shown, it is a schematic diagram of the specific structure of the feature splicing module of an embodiment of the present invention, wherein F1, F2, and F3 are input features of the feature splicing module, and 896, 1×1, and 256 mean that the input channel is 896, the convolution kernel size is 1×1, and the output channel is 256.

[0091] In step S3, the encoding and decoding module includes an encoder and a decoder, and uses continuous 3×3 convolutional layers to perform convolution downsampling operations on the features input by the feature splicing module, wherein the feature output of each layer of the encoder is used as a reference feature set, and the decoder is a series of 3×3 convolutional layers with a step size of 1, and the decoder branch includes an upsampling layer and a pixel-by-pixel summation operation.

[0092] Specifically, the purpose of adding upsampling layers and pixel-by-pixel summation operations on the decoder branches is to keep the feature map size the same, enhance learning capabilities and maintain feature smoothness.

[0093] Specifically, the present invention does not specifically limit the structure of the encoding and decoding module, as long as it can achieve its functions. The embodiment of the present invention provides a preferred specific structure of the encoding and decoding module, please refer to Figure 6 As shown, it is a schematic diagram of the specific structure of the encoding and decoding module of an embodiment of the present invention, wherein 256, 1, 256 means: the input channel is 256, the step size is 1, the output channel is 256, and the symbol ⊕ represents pixel-by-pixel addition.

[0094] In step S5, the training of the convolutional neural network includes:

[0095] Step S51, the detection network obtains a number of three-dimensional tensor-encoded prediction boxes, objects, and class predictions, and performs box regression and class regression in combination with a loss function;

[0096] In step S52, the Soft-NMS method is used to filter the highly overlapping prediction boxes to obtain the final prediction bounding box.

[0097] In step S51, the detection network normalizes the parameters x, y, w and h of the prediction box. The predicted bounding box contains four parameters x, y, w and h. x and y refer to the X and Y coordinates of the center point of the prediction box, and w and h refer to the width and height of the prediction box. We set S = 7 and divide the image into a 7×7 grid. The dotted area represents the position of the bounding box. The point (x0, y0) is the center point of the bounding box, and its grid coordinates are (row, col), and width img and height img Represents the width and height of the entire image, width box and height box represents the width and height of the bounding box. Therefore, the normalization process is:

[0098]

[0099]

[0100] The detection network divides the commutator image to be detected into a grid of s×s, where S=s, (x0, y0) is the predicted coordinate of the center point of the bounding box, the grid coordinates of the center point are (row, col), and width img is the width of the commutator image to be detected, height img The height of the commutator image to be detected, width box is the width of the predicted bounding box, height box is the width of the predicted bounding box.

[0101] Specifically, the detection network uses K-Means clustering to determine the prediction box prior.

[0102] Specifically, the detection network normalizes the prediction box parameters, which can reduce the impact of singular sample data and speed up network learning.

[0103] In step S52, the Soft-NMS algorithm is:

[0104]

[0105] Among them, B represents the initial list; S represents the detection score, and the function f(IOU(M,b i ))for

[0106]

[0107] Among them, M represents the detection box with the highest score, b i Represents the processed box. N trepresents the threshold and a represents the parameter. If the overlap of bounding boxes is high, then a lower score is assigned, otherwise, the original detection score is kept.

[0108] Specifically, the NMS algorithm directly sets the IOU of the largest box with a score greater than a certain threshold to zero. The algorithm is relatively rough. Therefore, the detection network uses the Soft-NMS algorithm. The detection network sets a threshold and removes the prediction box with a category confidence lower than the threshold to obtain the final prediction box, thereby improving the detection accuracy.

[0109] Specifically, the present invention does not specifically limit the training process of the convolutional neural network, as long as the training can be completed. The embodiment of the present invention provides a preferred embodiment, including:

[0110] Step s1, capturing a set of image defect data of the internal test image of the traffic switch in a real scene, recorded as X: [X1, X2, X3...Xn];

[0111] Step s2, unify the image size to 416 pixels in length and 416 pixels in width by cropping, scaling and other operations, and annotate the image set X by using the dataset annotation software Labelimg to obtain the commutator surface defect dataset called SD_data;

[0112] Step s3, pre-training the feature extraction network on the Imagenet dataset to obtain a pre-trained feature extraction network, and training the overall model by combining SD_data with the loss function;

[0113] In step s4, the training image is passed through the feature extraction network to obtain a convolution feature map, and then through a series of convolution and upsampling processes, N×N (N is the length and width of the convolution feature map) three-dimensional tensor encodings of bounding boxes, objects, and class predictions are obtained. Combined with the loss function, border regression and category regression are performed, and finally the Soft-NMS method is used to filter highly overlapping areas.

[0114] Specifically, the present invention does not specifically limit the implementation of the detection method, as long as the detection result meets the preset standard. The embodiment of the present invention provides a preferred implementation scheme, including:

[0115] Step S10, obtaining a picture set X.

[0116] Step S20, the image X is sequentially input into the trained convolutional neural network to obtain a set of prediction boxes. Soft-NMS is used to screen the prediction boxes with high overlap, and the final result obtained by the neural network is output: the coordinates (x, y) of the upper left corner of the prediction box, the width W and height H of the prediction box.

[0117] Step S30, marking the coordinate values ​​on the image to obtain the final prediction box.

[0118] Specifically, the embodiment of the present invention provides a test result and analysis, including comparative tests of the algorithm in this paper and the YOLOv3 algorithm under different brightness and different road congestion conditions. In the experiment, it is set that the defect of the labeled commutator inner measurement image is detected only when the IOU of the detected commutator inner measurement image defect area and the actual labeled commutator inner measurement image defect area is greater than 70%. Then the recall rate and accuracy are calculated to obtain the F value. Each experiment is repeated 3 times for the comparative test, and the average of the 3 F values ​​is taken as the evaluation result.

[0119] Comparison test under different light intensities: In actual image detection, different light intensities will affect the detection effect. Therefore, three conditions are set in the comparison test, namely: large scale, medium scale and small scale, in order to compare the detection performance of the two algorithms under different environments. The test first randomly selects several images from the original test set, classifies them according to the three conditions, and randomly selects again until there are 100 images in each of the three condition categories. In each comparison test, 50 images are randomly selected from each comparison condition as the test set of the experiment. The images in the test set are detected using the two algorithms. The detection results are shown in Table 1.

[0120] Table 1 Image comparison test results between YOLOv3 and the proposed algorithm under different environmental conditions

[0121]

[0122] The comprehensive results in Table 1 are calculated by summing up the total number of commutator internal measurement image defects, the number of detected commutator internal measurement image defects, and the number of correct commutator internal measurement image defects detected in each comparative test data set, and then calculating the overall recall rate, accuracy rate, and F value using the same method as each test. From the results in Table 1, it can be seen that compared with YOLOv3, this method improves the detection performance under different lighting conditions. Among them, the detection accuracy of large scale is improved by 5.64%, the detection of medium-scale defects is improved by 5.17%, and the detection of small scale defects is improved by 7.09%. Compared with YOLOv3, this method can better adapt to changes in lighting.

[0123] The datasets in this paper are all images captured on the commutator. The proposed method is trained and tested on the dataset in this paper. The method in this paper is comprehensively evaluated and compared with classic target detection methods and recent commutator internal image defect detection methods such as LSVM+DPM, RCNN, Faster R-CNN, Faster R-CNN (ResNet version), Multi-task CNN, Aston-EAS, F-ConverNet, MonoGRNet, LTN, YOLOv2, YOLOv3, etc. Figure 6-12 The PR graphs of the above methods are shown in Figure 2. The AP values ​​of each algorithm in three different difficulty levels under the dataset in this paper are listed in Table 2.

[0124] Table 2 Average accuracy of three different difficulty levels in this paper's dataset

[0125]

[0126] The datasets in this paper are all real images taken by on-board cameras on the road. The experiments on the commutator internal image defect detection algorithm in this paper are more in line with the actual application scenarios. It can be seen from the PR graphs of each algorithm and Table 2 that the AP of the algorithm in this paper under three different standards (Easy / Moderate / Hard) in this paper's dataset are 95.29%, 92.42%, and 82.32% respectively. Compared with the base algorithm YOLOv3 target detection model, the AP at each difficulty level has been improved, which are: 0.29%, 3.50%, and 4.63% respectively. Here, the reason for the improvement in accuracy has a great relationship with the addition of the feature pyramid module, because in the YOLOv3 target detection model, it only upsamples the bottom convolution feature map by 2 times, and then concatenates the features with the upper stage to obtain the feature map of this stage; in the commutator internal test image defect detection model of this paper, the three-scale feature map generated from the backbone network is first fused to form a basic feature map that is passed into a U-shaped codec to generate a higher-order multi-scale feature map. Then we fuse the multi-scale feature map generated in the U-shaped codec with the three-scale feature map generated in the original backbone network and add a feature attention module to improve the feature expression ability. Compared with the FPN-like multi-scale detection method used in the YOLOv3 detection model, the feature pyramid of this model has a higher feature expression ability, can better detect small targets in the detection image, and can generate more semantic information, thereby further improving the overall detection effect. Similarly, this is one of the reasons for the significant improvement under the Hard difficulty.

[0127] According to the results of the comparison with the R-CNN series algorithms in Table 2, compared with the two-stage target detection algorithms such as R-CNN and Faster R-CNN, the algorithm in this paper is improved based on the single-stage target detection algorithm of YOLOv3. The commutator internal test image defect detection process is regarded as a regression problem. Unlike the RCNN series, a large number of candidate frames are generated by the regional recommendation network (RPN) before the commutator internal test image defect recognition is performed. As a result, the speed will be greatly improved. It can be seen from the table that the running time of the R-CNN series detection algorithm is more than 1000m, while the running time (ms) of the algorithm in this paper and the one-stage algorithm such as the YOLO series does not exceed 2 digits; and the accuracy is also improved compared with the R-CNN series, mainly because it fully combines the shallow spatial information and the deep semantic information (except the YOLOv2 algorithm). The speed of the YOLOv2 algorithm is the fastest commutator internal test image defect detection method among the algorithms listed in the table, but this algorithm does not take into account the multi-scale detection problem of the image. Compared with the currently popular commutator internal test image defect detection algorithms, the proposed algorithm has also improved in accuracy to a certain extent compared with the Multi-task CNN algorithm, Aston-AES algorithm, MonoGRNet algorithm, and LTN algorithm, and its speed is also prioritized. Compared with the F-ConverNet algorithm, although the accuracy of the proposed algorithm is not as good as the former at the Easy difficulty, its speed is several times faster than that of the F-ConverNet algorithm. We believe that the proposed algorithm is more effective in handling commutator internal test image defect detection in actual situations.

[0128] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A commutator inner side image defect detection method based on a fusible feature pyramid, characterized in that: include: Step S1, the feature extraction network extracts multi-level features of the inner image of the commutator to be detected, and inputs the extracted multi-level features into the feature splicing module of the multi-scale feature pyramid network; Step S2, the feature splicing module splices the input multi-level features, and inputs the spliced ​​multi-level features into the encoding and decoding module in the multi-scale feature pyramid network; Step S3, the encoding and decoding module generates features with multiple scales according to the features input by the feature splicing module; Step S4, the feature aggregation module of the multi-scale feature pyramid network aggregates the multi-level features in step S1 and the multi-scale features in step S3 into a fusible feature pyramid, and the detection network obtains a convolutional neural network by adding several convolution layers after the fusible feature pyramid, wherein the aggregation process of the multi-level features and the multi-scale features by the feature aggregation module includes: Step S41, feature-connecting the multi-scale features and the multi-level features along the channel dimensions of equal proportions to form a feature block; Step S42, compressing the global spatial information into a channel description, and performing global average pooling on the feature blocks; Step S43, comprehensively capturing channel dependencies to utilize information gathered in the compression operation; Step S44, re-weighting the feature blocks and outputting them; Step S5, training the convolutional neural network, and inputting the picture set of the inner side image of the commutator to be detected into the trained convolutional neural network in sequence to obtain a set of prediction boxes, marking the coordinate values ​​of the prediction boxes on the pictures, and obtaining the detection results.

2. The method for detecting defects of the inner side image of a commutator based on a fusible feature pyramid according to claim 1, characterized in that: In step S42, the feature aggregation module compresses the global spatial information into a channel description through a Squeeze operation, and performs global average pooling on the feature blocks. Formally, the statistics z∈R C is generated by shrinking U in the spatial dimension W×H, where the cth element of z is, Among them, u c (i, j) represents the statistical information of the cth element at the (i, j)th position in the spatial dimension W×H.

3. The commutator inner side image defect detection method based on fusible feature pyramid according to claim 2, characterized in that: In step S43, the feature aggregation module fully captures channel dependencies through the Excitation operation, that is, aggregates features. s=F ex (z, W)=σ(W2δ(W1z)) Among them, σ represents the ReLU function, δ represents the sigmoid function, R represents the reduction ratio, and the bottleneck of two fully connected layers is used to parameterize the gate mechanism, that is, the dimension reduction layer parameter is W1, the dimension reduction ratio is r, and W2 is the dimension increase layer parameter to re-weight U to output X.

4. The method for detecting defects of the inner side image of a commutator based on a fusible feature pyramid according to claim 3, characterized in that: In step S44, the feature aggregation module re-weights and outputs the feature blocks, wherein: Among them, X represents the final output of the SE block, Representative fusion pyramid features With s c The product of .

5. The method for detecting defects of the inner side image of a commutator based on a fusible feature pyramid according to claim 4, characterized in that: In step S1, the feature extraction network is constructed by stacking a Darknet-53 network using a shuttle residual structure, and a downsampling operation is performed using a Mobilenet structure with a step size of 2, wherein the shuttle residual structure is formed by connecting the Darknet structure in YOLOv3 with the residual structure.

6. The method for detecting defects of the inner side image of a commutator based on a fusible feature pyramid according to claim 5, characterized in that: In step S2, the feature concatenation module performs an upsampling operation on the input multi-level features and scales the deep features to the same scale before performing the concatenation operation.

7. The method for detecting defects of the inner side image of a commutator based on a fusible feature pyramid according to claim 6, characterized in that: In step S3, the encoding and decoding module includes an encoder and a decoder, and uses continuous 3×3 convolutional layers to perform convolution downsampling operations on the features input by the feature splicing module, wherein the feature output of each layer of the encoder is used as a reference feature set, and the decoder is a series of 3×3 convolutional layers with a step size of 1, and the decoder branch includes an upsampling layer and a pixel-by-pixel summation operation.

8. The method for detecting defects of the inner side image of a commutator based on a fusible feature pyramid according to claim 7, characterized in that: In step S5, the training of the convolutional neural network includes: Step S51, the detection network obtains a number of three-dimensional tensor-encoded prediction boxes, objects, and class predictions, and performs box regression and class regression in combination with a loss function; In step S52, the Soft-NMS method is used to filter the highly overlapping prediction boxes to obtain the final prediction bounding box.

9. The method for detecting defects of the inner side image of a commutator based on a fusible feature pyramid according to claim 8, characterized in that: In step S51, the detection network normalizes the parameters x, y, w and h of the prediction box, where: The detection network divides the commutator image to be detected into a grid of s×s, where S=s, (x0, y0) is the predicted coordinate of the center point of the bounding box, the grid coordinate of the center point is (row, col), and width img is the width of the commutator image to be detected, height img The height of the commutator image to be detected, width box is the width of the predicted bounding box, height box The predicted bounding box contains four parameters: x, y, w, and h. x and y refer to the X and Y coordinates of the center point of the predicted box, and w and h refer to the width and height of the predicted box.