Steel rail surface defect detection method, device, medium and equipment

Abstract

The invention discloses a steel rail surface defect detection method and device, a medium and equipment, and relates to the technical field of image processing. Comprising the following steps: replacing a standard convolutional layer of a Resnet50 module in an original Faster R-CNN network with a deformable convolutional layer; a feature pyramid network is added to the output end of the Resnet50 module; the ROI Pooling module is replaced by the ROI-Align module, so that the network is improved. The deformable convolution adaptively adjusts the sampling point position of the convolution kernel, the sampling strategy of the convolution kernel can be dynamically adjusted according to the input image, and it is ensured that the defect geometric feature map is transmitted more effectively; the feature pyramid network enhances a defect geometric feature map through continuous down-sampling and up-sampling operations, so that steel rail surface defect shape information is more stable and prominent, a more accurate defect candidate area and a more accurate defect feature vector can be obtained, and a more accurate detection result can be output.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, medium and equipment for detecting rail surface defects. Background Art

[0002] During the long-term use of railway tracks, various defects such as cracks, fractures, and wear will inevitably occur. If these defects are not discovered and handled in time, they may lead to serious railway accidents. With the rapid development of computer technology and deep learning, object detection is used for defect detection on the surface of railway tracks.

[0003] At present, most scholars adjust and optimize the existing detection framework to meet the requirements of practical applications. He et al. proposed a strip surface defect detection network based on Faster R-CNN. The innovation of this network lies in combining the multi-level feature maps in the backbone network into a multi-scale feature map. Wang Yanshu proposed a hot-rolled strip surface defect detection algorithm called adaptive global positioning network, which combines residual network and feature pyramid network to extract features, and realizes high-precision defect positioning through adaptive tree candidate box extraction network and global positioning regression algorithm.

[0004] However, the types and shapes of rail surface defects are diverse, and their scales vary greatly, which makes it difficult to locate the defect positions using the above method, and the defect detection accuracy obtained needs to be improved. Summary of the invention

[0005] Based on this, it is necessary to provide a rail surface defect detection method, device, medium and equipment to address the above technical problems.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a method for detecting rail surface defects, comprising:

[0008] Replace the standard convolutional layer of the Resnet50 module in the original Faster R-CNN network with a deformable convolutional layer; add a feature pyramid network at the output of the Resnet50 module in the original Faster R-CNN network to construct an improved Resnet50 module; replace the ROIPooling module of the original Faster R-CNN network with a ROI-Align module; construct an improved Faster R-CNN network including the improved Resnet50 module, the region proposal module of the original Faster R-CNN network, the ROI-Align module, and the target classification and bounding box prediction module connected in sequence;

[0009] Collect rail surface defect images to build a data set, use the data set to train the improved Faster R-CNN network, and obtain a detection model for detecting rail surface defects.

[0010] The surface image of the rail to be inspected is input into the detection model. The sampling points of the convolution kernel are adaptively adjusted by improving the deformable convolution layer in the Resnet50 module to capture the irregular geometric features of the rail surface defects and obtain the defect geometric feature map. The defect geometric feature map is enhanced at different scales by improving the multi-scale upsampling and downsampling operations of the feature pyramid network in the Resnet50 module to obtain a multi-scale defect enhancement feature map. The multi-scale defect enhancement feature map is detected by the region proposal module to generate several defect candidate regions. The defect feature vectors corresponding to several defect candidate regions are extracted from the multi-scale defect enhancement feature map by the ROI-Align module. The defect feature vector is mapped into a detection result containing the rail surface defect category and the rail surface defect bounding box by the target classification and bounding box prediction module.

[0011] Furthermore, the defect geometric feature maps of different scales are enhanced by correspondingly connecting each downsampling layer in the feature pyramid network with the output end of each residual module in the Resnet50 module to enhance the defect geometric feature maps of different scales extracted by each residual module.

[0012] Furthermore, the enhancement of the defect geometric feature map at different scales is to connect the first downsampling layer in the feature pyramid network with the output end of the last residual module of the Resnet50 module to enhance the defect geometric feature map of the minimum scale output by the Resnet50 module.

[0013] Furthermore, the defect feature vectors corresponding to several defect candidate regions are extracted from the multi-scale defect enhancement feature map by the ROI-Align module, specifically including:

[0014] The multi-scale defect enhancement feature map and multiple ROI bounding boxes are used as the input of the ROI-Align module;

[0015] Each ROI bounding box is divided into a grid of fixed size, and four sampling points are evenly placed in each grid unit. The eigenvalues ​​corresponding to the four sampling points are calculated using bilinear interpolation.

[0016] In each grid cell, the feature values ​​of the four sampling points are subjected to a maximum pooling operation to obtain the feature value of the corresponding grid cell, and the feature value is used as a pixel value in the feature map output by the ROI-Align module;

[0017] Based on the feature values ​​of all grid cells in each ROI bounding box, each ROI bounding box generates a fixed-size feature map as the corresponding defect feature vector.

[0018] Furthermore, the improved Faster R-CNN network is trained using the data set, and the loss function used is the SIoU loss function; the SIoU loss function calculates the gap between the predicted bounding box and the true bounding box by introducing angle loss, distance loss, shape loss and IoU loss, so as to guide the weight coefficient of the improved Faster R-CNN network to be updated in the direction in which the gap between the predicted bounding box and the true bounding box is minimized.

[0019] The present invention provides a rail surface defect detection device, comprising:

[0020] A model building module for replacing the standard convolutional layer of the Resnet50 module in the original Faster R-CNN network with a deformable convolutional layer; adding a feature pyramid network to the output of the Resnet50 module in the original Faster R-CNN network to build an improved Resnet50 module; replacing the ROIPooling module of the original Faster R-CNN network with a ROI-Align module; and building an improved Faster R-CNN network including the improved Resnet50 module, the region proposal module of the original Faster R-CNN network, the ROI-Align module, and the object classification and bounding box prediction module connected in sequence;

[0021] The model training module is used to collect rail surface defect images to construct a data set, and use the data set to train the improved FasterR-CNN network to obtain a detection model for detecting rail surface defects;

[0022] The defect detection module is used to input the surface image of the rail to be detected into the detection model, and adaptively adjust the sampling points of the convolution kernel by improving the deformable convolution layer in the Resnet50 module to capture the irregular geometric features of the rail surface defects and obtain a defect geometric feature map; the defect geometric feature map is enhanced at different scales by improving the multi-scale upsampling and downsampling operations of the feature pyramid network in the Resnet50 module to obtain a multi-scale defect enhancement feature map; the multi-scale defect enhancement feature map is detected by the region proposal module to generate several defect candidate regions; the defect feature vectors corresponding to several defect candidate regions are extracted from the multi-scale defect enhancement feature map by the ROI-Align module; the defect feature vector is mapped into a detection result containing the rail surface defect category and the rail surface defect bounding box by the target classification and bounding box prediction module.

[0023] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned rail surface defect detection method is implemented.

[0024] The present invention provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned rail surface defect detection method when executing the program.

[0025] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0026] The present invention improves the original Faster R-CNN network by replacing the standard convolution layer of the Resnet50 module in the original Faster R-CNN network with a deformable convolution layer, adding a feature pyramid network at the output end of the Resnet50 module, and replacing the ROI Pooling module with an ROI-Align module, thereby enhancing the accuracy of the Faster R-CNN network in detecting rail surface defects. Specifically, the deformable convolution adaptively adjusts the sampling point position of the convolution kernel, so that each downsampling layer can dynamically adjust the sampling strategy of the convolution kernel according to the specific input rail surface defects, ensuring that the defect geometric feature map can convey more effective rail surface defect shape information; the feature pyramid network enhances the defect geometric feature map through continuous downsampling and upsampling operations, making the rail surface defect shape information more robust and prominent, which is conducive to obtaining more accurate defect candidate areas and more accurate defect feature vectors, thereby outputting more accurate detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0028] Figure 1 A schematic flow chart of a rail surface defect detection method provided by the present invention;

[0029] Figure 2 A schematic diagram of the structure of the improved Faster R-CNN network provided by the present invention;

[0030] Figure 3 A schematic diagram of a deformable convolution structure provided by the present invention;

[0031] Figure 4 A schematic diagram of the FPN network structure provided by the present invention;

[0032] Figure 5The RoI-Pooling principle diagram provided by the present invention;

[0033] Figure 6 The RoI-Align principle diagram provided by the present invention;

[0034] Figure 7 Schematic diagram of the SIoU loss function provided by the present invention;

[0035] Figure 8 Schematic diagram of different types of defects provided by the present invention, Figure 8 (a) is a crack defect. Figure 8 (b) is an inclusion defect. Figure 8 (c) in the figure is a plaque defect. Figure 8 (d) in the figure is a pit defect. Figure 8 (e) is the iron oxide scale indentation defect. Figure 8 (f) in the figure is a scratch defect. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present application will be clearly and completely described below in combination with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] Railways play a vital role in a country's economy and development, and directly affect people's transportation quality. With the rapid development of my country's economy, railway transportation has gradually developed towards heavy load and high speed, so the safe operation of railways faces new challenges. Railway tracks are a key component of modern transportation systems, and their safety is directly related to the reliability of train operation and the safety of passengers' lives and property. However, in the long-term use of railway tracks, various defects such as cracks, fractures, and wear will inevitably occur. If these defects are not discovered and handled in time, they may cause serious railway accidents. Traditional track detection methods mainly rely on manual visual inspection and regular maintenance, which is not only inefficient, but also difficult to achieve comprehensive and continuous track status monitoring. With the development of computer vision and image processing technology, it is particularly important to propose an automated and high-precision track surface defect detection algorithm. Such an algorithm can greatly improve the speed and accuracy of detection, reduce the interference of human factors, and realize real-time monitoring and early warning, thereby ensuring the safety and efficiency of railway transportation. Therefore, it is not only of great theoretical value to study the ability to quickly and accurately identify defects on the rails and improve the efficiency of rail defect detection, but also has broad application prospects.

[0038] Traditional methods for detecting rail surface defects include visual inspection, mechanical inspection, magnetic particle inspection, ultrasonic inspection, and infrared thermal imaging inspection. Visual inspection mainly relies on manual observation. Due to the limitations of manpower and subjective factors, this method generally has problems such as poor real-time performance and low detection efficiency. Mechanical inspection uses sensors to detect physical changes on the surface of the rails, providing more accurate results than visual inspection. Magnetic particle inspection and ultrasonic inspection can detect cracks and surface defects, while infrared thermal imaging inspection relies on temperature anomalies to identify defects. These traditional methods can detect defects in railway tracks to a certain extent, but there are problems such as low efficiency, dependence on manual labor, and low accuracy. In recent years, computer vision-based technologies have begun to be applied to railway track surface defect detection. Laser scanning detection is an early automated non-contact detection technology. With the development of semiconductor and imaging technology, a detection method based on CCD camera devices has been developed. Compared with traditional detection methods, computer vision has improved detection accuracy and reduced detection costs to a certain extent.

[0039] With the rapid development of computer technology and deep learning, the above challenges faced by traditional railway transportation detection methods have begun to be gradually resolved. Object detection is an important task in computer vision. Its goal is to detect and locate objects in images or videos, usually accompanied by the classification of object categories. The deep learning-based defect detection task is a practice of applying object detection algorithms to industrial production. Its core task is to find defects in images and accurately mark their size, location and category. There are two main categories of deep learning detection algorithms: one-stage detection and two-stage detection, which have some important differences in design and execution. YOLO is a typical one-stage detector that can simultaneously complete the tasks of object detection and classification. This method is usually simpler and more efficient. Because it only needs one forward propagation to generate detection results, the YOLO algorithm has a faster inference speed and higher detection accuracy. Faster R-CNN is the most representative two-stage detector, which was proposed by Ren et al. in 2015 based on Fast-RCNN. It introduces the RPN (Region Proposal Network) network, which can quickly generate candidate regions without relying on external methods (such as Selective Search), thereby achieving end-to-end target detection, which is faster and more accurate. This method uses two independent stages to perform target detection. The first stage is usually the process of extracting candidate regions (Region Proposal), which decomposes the original image into multiple candidate regions that may produce targets, and the second stage is to classify and accurately locate these candidate regions through classification and regression loss functions. In actual industrial applications, it is necessary not only to locate the defects on the extracted images, but also to classify them. Therefore, most scholars adjust and optimize the existing detection framework to meet the requirements of practical applications. Zhang et al. combined Faster R-CNN with CBAM to improve the response between features and enhance the detection effect of small targets. He et al. proposed a strip surface defect detection network based on Faster R-CNN. The innovation of this network lies in combining the multi-level feature maps in the backbone network into a multi-scale feature map. The proposed method achieves good detection results on NEU-DET. Wang Yanshu proposed a hot-rolled strip surface defect detection algorithm called Adaptive Global Localization Network (AGLNet). The algorithm combines the residual network (ResNet) and the feature pyramid network (FPN) to extract features, and achieves high-precision defect localization through the adaptive tree candidate box extraction network (AT-RPN) and the global localization regression algorithm. Deng Hui et al. proposed a hot-rolled strip surface defect detection method based on the improved FasterR-CNN.The innovation of this method is to fuse the underlying details with high-level semantic features by feature addition, use Precise ROIPooling to obtain fixed feature vectors, and cluster the data set through the mean shift algorithm to optimize the prior box size. Zhang Jiaxiong proposed an improved YOLOv7 algorithm for surface defect detection of steel plates. By adding CBAM to the feature fusion network, a new efficient layer aggregation network was constructed to improve the learning and generalization capabilities of the model. The article also introduced a decoupled head to more effectively handle complex defect detection tasks, and used the Wise-IoU function to optimize the training process and reduce the impact of low-quality samples.

[0040] Although these methods have achieved certain results in their respective application scenarios, they also have some shortcomings: first, the FasterR-CNN network has weak detection capabilities for small targets, because small targets account for a small proportion of the feature map and are easily ignored or confused with the background. Second, the FasterR-CNN network is not sensitive enough to the shape and posture changes of the target, which may cause some areas of some targets to be blocked or missed. Since the surface defects of the rails are irregular in shape, have large scale variations, and are difficult to locate, the above algorithms are used directly for detection, and the defect detection accuracy is not high and the amount of calculation is large.

[0041] In view of these problems, the present invention proposes a rail surface defect detection algorithm based on the improved Faster R-CNN. First, for the feature extraction network, on the basis of the original feature extraction network Resnet50, deformable convolution is introduced to increase the network's feature extraction capability for various types of defects; then, the FPN network is constructed to fully integrate the deep semantic information and the shallow geometric information, so as to solve the problem that the rail surface defect targets are small and difficult to detect; then, the ROI-Align structure is used to replace the ROIPooling to reduce the positioning error generated in the candidate region extraction process; finally, the SIoU loss function is used to replace the traditional Smooth L1loss to improve the training accuracy and the overall performance of the algorithm.

[0042] Example 1

[0043] Figure 1 The process of the rail surface defect detection method according to the embodiment of the present disclosure is shown. Figure 1 The method is described in detail and specifically comprises the following steps:

[0044] S1: Replace the standard convolutional layer of the Resnet50 module in the original FasterR-CNN network with a deformable convolutional layer; add a feature pyramid network to the output of the Resnet50 module in the original FasterR-CNN network to construct an improved Resnet50 module; replace the ROIPooling module of the original FasterR-CNN network with a ROI-Align module; construct an improved FasterR-CNN network including the improved Resnet50 module, the region proposal module of the original FasterR-CNN network, the ROI-Align module, and the object classification and bounding box prediction module connected in sequence.

[0045] Improve the network structure of Faster R-CNN network Figure 2 As shown in the figure, the model consists of the following four parts: ① Feature Extraction Network: The classic ResNet50 is used to perform convolution operations on the original image to extract the shallow visual features and deep semantic features of the image; and the feature pyramid (FPN) connected to the output of ResNet50. ② Region Proposal Network (RPN): This network replaces the Selective Search network to identify and extract the proposed region. RPN receives the feature map from the feature extraction network and then generates multiple candidate regions, which are considered to contain targets. RPN uses anchor boxes as reference boxes to generate candidate regions, and then screens and accurately locates these candidate regions through classification and regression operations. ③ ROIAlign Network: This network is used to extract fixed-size feature vectors corresponding to the candidate regions from the feature map of the feature extraction network. These feature vectors will be used for subsequent target classification and bounding box regression. ④ Target Classification and Bounding Box Prediction Network: The feature vector obtained from the RoIAlign layer will be input into two independent fully connected layers, one for target classification (determining the object category in each RoI) and the other for bounding box regression (accurately locating the bounding box of the object).

[0046] S101: ResNet50 network based on deformable convolution.

[0047] Track surface defects are irregular and have variable shapes. Using the original 3×3 convolution sampling will introduce too much defect background, making it difficult to extract defect features. Therefore, deformable convolution

[16] is used to replace the ordinary 3×3 convolution in the Faster R-CNN feature extraction network ResNet50. The deformable convolution operation is an extension of the standard convolution operation. For planar graphics, the deformable convolution adds the learned offset to the network sampling position in the standard convolution, causing the sampling position of the standard convolution to deform and focus more on the region of interest (ROI). In this way, the sampling points of the convolution kernel will change position according to the shape of the input image, forming the ability to adapt to changes and having better geometric invariance.

[0048] For a standard 3×3 convolution R={(-1,-1),(-1,0),...,(0,1),(1,1)}, for each position P0 on the output feature map, we have:

[0049]

[0050] Among them, P n enumerates the positions in R. In deformable convolution, the regular network increases {ΔP n |n=1,...,N}, where N=|R|, equation (1) becomes:

[0051]

[0052] So far, sampling is done at irregular and offset positions P n +ΔP n Because ΔP n is usually a decimal, equation (2) can be implemented by bilinear interpolation:

[0053] x(P)=∑ q G(q,p)·x(q) (3)

[0054] Where p represents any (decimal) position (P0+P in formula (2) n +ΔP n ), q enumerates all the global spatial locations in the feature map, G(·,·) is the kernel of bilinear interpolation, G is two-dimensional, it is divided into two one-dimensional kernels:

[0055] G(q,p)=g(q x , p x )·g(q y ·p y ) (4)

[0056] Where G(q,b)=max(0,1-|ab|), equation (3) can be calculated quickly because G(q,P) is non-zero only for some q.

[0057] like Figure 3 As shown in , the offset is obtained by applying a convolutional layer on the same input feature map. The convolution kernel has the same spatial resolution and dilation as the current convolutional layer (e.g., Figure 2 , which also has a 3×3 convolution with dilation 1). The output offset domain has the same spatial resolution as the input feature map, and the channel dimension corresponds to the N 2D offsets. During training, the convolution kernel and the offsets used to generate the output features are learned simultaneously. To learn the offsets, the gradients are back-propagated through the bilinear operation in equations (3) and (4).

[0058] S102: Feature Pyramid Network FPN.

[0059] Since the original FasterR-CNN only uses the last layer of features in the feature extraction network for classification, this method has the advantages of fast speed, less memory required, and better detection effect for large targets, but the effect of detecting small targets such as rail defects is relatively poor. Therefore, the FPN network is constructed to enhance the detection effect of small targets.

[0060] The structure of FPN is as follows Figure 4 As shown, it contains three structures: a bottom-up path, a lateral connection, and a top-down path. The bottom-up path starts from the bottom of the backbone network and gradually extracts features upward. Typically, this involves extracting features from different layers of the backbone network, each with different resolutions. The output of the bottom-up path usually includes a series of feature maps, where the bottom layer has the highest resolution and the top layer has the lowest resolution. The top-down path starts from the top of the bottom-up path and gradually propagates information downward. In this path, the resolution of the feature map gradually decreases, while the semantic information gradually increases. The top-down path usually includes an upsampling operation to map the low-resolution feature map to a high resolution in order to combine it with the feature map of the bottom-up path. At the level where the top-down and bottom-up paths intersect, feature fusion is performed. Typically, this includes a simple element-by-element addition operation to combine the bottom-up and top-down feature maps. This makes the feature map have multi-scale and high semantic information. Finally, FPN generates a multi-scale feature pyramid, where each level contains feature maps of different scales, and each feature map incorporates high semantic information. This feature pyramid usually includes high-resolution features for processing small objects and low-resolution features for processing large objects.

[0061] The connection method between the feature pyramid network and the Resnet50 network can be as follows: each downsampling layer in the feature pyramid network can be connected to the output end of each residual module in the Resnet50 network, that is, each layer of the feature pyramid network corresponds to a defect geometric feature map of one scale, so as to enhance the defect geometric feature maps of different scales extracted by each residual module.

[0062] The connection method between the feature pyramid network and the Resnet50 network can also be: the first downsampling layer in the feature pyramid network is connected to the output end of the last residual module of the Resnet50 network, that is, the feature pyramid network only uses the minimum scale defect geometric feature map output by the network as input to enhance the minimum scale defect geometric feature map output by the Resnet50 network.

[0063] The Faster R-CNN network with FPN added can fuse high-semantic features with the detail features of shallow feature maps at multiple scales, enabling the model to obtain the details and overall information of surface defects at different scales. At the same time, it improves the detection accuracy and can better adapt to surface defects of different shapes and sizes, thereby improving the detection precision and accuracy.

[0064] S103: Pooling method based on ROI-Align.

[0065] The main purpose of the RoI pooling layer is to map regions of interest of different sizes to a fixed-size feature vector for subsequent classification or regression tasks. The traditional Faster R-CNN uses ROI-Pooling to fix the feature map. The problem with ROI-Pooling is that it is not accurate enough for the size and position of RoIs, which may lead to information loss. The specific principle is as follows Figure 5 as shown.

[0066] Taking the input surface defect image as an example, a 256×256 image is input. Assume that the RoI of the defect in the image is 150×150. Since the downsampling rate is 16, the image is reduced to 1 / 16 of the original image. The corresponding size of the defect after the RoI pooling layer is 150 / 16=9.375. At this time, RoI-Pooling performs the first quantization on it, and the RoI feature map of the defect is rounded to 9×9, such as Figure 4As shown in the third step. The obtained area is then divided into 7×7 units on average, and the boundary of each unit is quantized. After the above two quantizations, the candidate box at this time has a certain deviation from the position initially regressed. The author of the paper summarizes it as a "misalignment problem". Especially for various small-sized defects on the surface of the rail, slight deviations will have a greater impact on the accuracy of model detection. In order to solve the above-mentioned shortcomings of RoI-Pooling, the present invention replaces RoI-Pooling with the RoI-Align structure. First, RoI-Align maps the RoI to the corresponding input feature map based on the input image and the predicted coordinates of the region of interest. Then for each region of interest, the bilinear interpolation method is used to remove the features of the corresponding position from the feature map. Finally, the features in each region of interest are bilinearly interpolated and mapped to an output feature vector of a fixed size. As shown Figure 6 As shown in the figure, RoI-Align uses bilinear interpolation to accurately align features in the region of interest, avoiding the information loss problem that may exist in RoI-Pooling. Therefore, it can provide more accurate feature representation and improve the model's ability to understand the target position and shape. Especially in scenarios where more precise positioning of the target is required, the advantages are more significant.

[0067] S2: Collect Faster R-CNN network images to build a dataset, use the dataset to train the improved Faster R-CNN network, and obtain a detection model for detecting rail surface defects.

[0068] S201: Set the loss function.

[0069] The loss of Faster R-CNN includes classification loss (cls loss) and regression loss (bbox regression loss). Its expression is as follows:

[0070]

[0071] Where P i The probability of predicting the i-th anchor as the true target; Indicates whether there is an object in the target binary classification, 1 if there is an object, otherwise 0; t i ={t x ,t y ,t w ,t h}, is a four-point vector representing the position coordinates of the bounding box regression as well as its width and height; is and t iA vector of the same dimension, representing the position coordinates and width and height of the anchor box detected as a positive sample. L in the loss function (5) reg As shown in formula (6):

[0072]

[0073] Where R is the Smooth L1 function, and its expression is:

[0074]

[0075] Faster R-CNN uses SmoothL1 as the bounding box regression loss function, but Smooth L1 Loss penalizes small objects too strongly, which may cause the model's performance in small object detection to deteriorate. Because the loss function of SmoothL1 Loss decreases slowly when the prediction deviation is small, especially for small objects such as rail surface defects, which are more difficult to predict. Secondly, Smooth L1 Loss is more sensitive to outliers (or outliers), that is, the loss decreases slowly for those prediction boxes that deviate far from the true value. This may cause the model to have poor generalization ability for some special cases. Finally, Smooth L1 Loss calculates the loss for each object separately, without considering the correlation between objects. In some cases, there may be certain correlations between objects, such as mutual occlusion or dependency between objects, which may affect the calculation of the loss function and the training effect of the model. Therefore, in order to further improve the performance of the Faster R-CNN model, it is an effective method to use the IoU

[17] (Intersection over Union) loss function to improve the positioning loss in Faster R-CNN. It can better measure the overlap between the predicted box and the true box, thereby improving the accuracy of target positioning. The traditional IoU loss function mainly reflects the quality of the prediction result through the IoU value of the predicted box and the true box, without considering the scale difference of the target, which may cause the model to have unstable performance when dealing with small-scale targets. In addition, the traditional IoU loss function also does not take into account the shape difference of the target. For defects of different shapes on the surface of the rail, the same IoU value may represent different detection effects, resulting in the model's insufficient detection ability for targets of some special shapes. Therefore, the present invention uses the SIoU loss function to improve the loss function of Faster R-CNN. The SIoU loss introduces a more complex bounding box regression method by incorporating angle considerations and scale sensitivity, thereby solving the limitations of previous loss functions. The SIoU loss function includes four components: angle loss, distance loss, shape loss and IoU loss. The schematic diagram of the SIoU loss function is shown in the figure. Figure 7 The formula for angle loss is as follows:

[0076]

[0077] Where C h is the height difference between the center points of the real box and the candidate box; σ is the distance between the center points of the real box and the predicted box.

[0078] The formula for distance loss is as follows:

[0079]

[0080] in:

[0081]

[0082]

[0083] γ=2-Λ (12)

[0084] Where: is the position coordinate of the center point of the real frame, It is the position coordinate of the center point of the prediction box.

[0085] The formula for shape loss is as follows:

[0086]

[0087] in:

[0088]

[0089]

[0090] Where: w, h are the width and height of the prediction box, w gt ,h gt are the width and height of the true box, and θ represents the degree of attention paid to shape loss.

[0091] In summary, the SIoU loss function is defined as follows:

[0092]

[0093] Compared with the original loss function, SIoU has certain advantages in scale invariance, positioning accuracy, adaptability and category imbalance processing, and can better meet the needs of target detection tasks. In theory, it can improve the performance and generalization ability of the model after replacement.

[0094] S202: Experimental dataset.

[0095] The data set used in the present invention is based on the NEU-DET data set provided by the Northeastern University Strip Steel Inspection Database. The data set includes images of six typical defects on the surface of the strip steel, namely crazing, inclusions, patches, pitted-surface, rolled-in-scale and scratches. Each defect type has 300 grayscale images with a size of 200×200 pixels, and the entire data set has 1800 grayscale images. In the experiment, the 1800 images were randomly divided into a training set and a test set. The training set has 1200 images, including 200 images of each type of defect, and the test set has 600 images, including 100 images of each type of defect. The six defect type images are as follows: Figure 8 shown.

[0096] S203: Experimental environment.

[0097] Configuration used in the experiment: Windows 11 operating system, CPU: Intel Core i5, GPU: NVIDIA GTX1650, memory: 16G, programming tool: Python 3.8, deep learning framework: pyTorch 2.0.1, CUDA: 11.8.

[0098] S204: Evaluation indicators.

[0099] Four commonly used evaluation indicators are used to test the performance of the model: Precision (P), Recall (R), mAP and AP. According to the actual category of the defect test sample and the model prediction category, it is divided into true positive examples (TP), false positive examples (FP), true negative examples (TN), and false negative examples (FN).

[0100] Accuracy (Precision, P): Accuracy refers to the ratio of the number of samples correctly predicted as positive by the model to the number of samples predicted as positive. It measures the classification accuracy of the model for the positive class. Accuracy can be expressed by the following formula:

[0101]

[0102] Recall (R): Recall refers to the ratio of the number of samples correctly predicted as positive by the model to the number of all true positive samples. It measures the ability of the model to discover positive categories. Recall can be expressed by the following formula:

[0103]

[0104] mAP (Mean Average Precision): mAP is a performance metric for multi-category classification or object detection tasks. It first calculates the average precision (AP) of each category, and then takes the average of the APs of all categories to get mAP. mAP measures the performance of the model in multi-category tasks, especially for detection tasks. The formula is as follows:

[0105]

[0106] AP (Average Precision): AP is mainly used to measure the performance of the model on different categories or detection targets. It is calculated based on the Precision-Recall Curve. The formula is as follows:

[0107]

[0108] S205: Experimental data analysis.

[0109] (1) Comparison of experimental results of different backbone networks.

[0110] In the target detection algorithm, the backbone network is used to extract the features of the target and is a core component. In order to verify the detection performance of Faster R-CNN under different feature extraction networks, this experiment selected four feature extraction networks: Resnet50, VGG16, MobileNet-V3, and EfficientNet for comparison. After the same number of iterations, the experimental results are shown in Table 1.

[0111] Table 1. Comparison of experimental results of different backbone networks

[0112]

[0113] According to the results shown in Table 1, MobileNet-V3 and EfficientNet, as lightweight feature extraction networks, have mAP values ​​of 34.9% and 38.6% respectively, and their ability to extract the three defects of cracks, pits and scale indentation is very poor. VGG16 is a classic deep convolutional network. Although it has a high accuracy rate, it is relatively slow and resource-intensive, with an mAP value of 53.4%. Using the unmodified ResNet50 as the backbone network shows a relatively high mAP value of 58.1%, highlighting the characteristics of ResNet as a powerful network structure, which successfully solves the degradation problem that deep networks are prone to through residual connections. These results reveal the performance differences of different backbone networks in the task of rail surface defect detection, and provide important references and inspirations for selecting appropriate network structures.

[0114] (2) Comparison of experimental results of different target detection algorithms.

[0115] The detection results of the improved model of the present invention on the NEU-DET dataset are compared with other mainstream network models RetinaNe, SSD and YoloV5. The experimental results are shown in Table 2.

[0116] Table 2. Comparison of experimental results of different target detection algorithms

[0117]

[0118] According to the data in Table 2, the mAP of this model on this dataset reached 80.2%, which is better than other one-stage detection models (such as RetinaNet, SSD, YoloV5, etc.). It is worth noting that for defects with blurred boundaries and difficult features to extract, such as cracks and iron oxide indentations, this model achieved the best AP value and is more effective in extracting features of these types of defects. At the same time, in the face of obvious defect features, such as inclusions, scratches, etc., this model also shows better results. These results show the advantages of this model in extracting and identifying various types of defects.

[0119] (3) Network improvement ablation experiment.

[0120] In order to compare the detection effects of the improved Faster R-CNN algorithm and the Faster R-CNN algorithm on the same data set, the present invention designed four groups of experiments. Experiment 1: Using ResNet50 as the Faster R-CNN algorithm for feature extraction network; Experiment 2: Adding FPN network module on the basis of Experiment 1; Experiment 3: Replacing the pooling method from RoI-Pooling to RoI-Align on the basis of Experiment 2; Experiment 4: Replacing the ordinary 3×3 convolution in the feature extraction network with deformable convolution on the basis of Experiment 3; Experiment 5: Replacing the original SmoothL1 loss function with SIoU loss function on the basis of Experiment 4. The experimental results are shown in Table 3.

[0121] Table 3. Ablation experiment results

[0122]

[0123] According to the data in Table 3, the traditional Faster R-CNN model has some limitations in defect recognition, especially in terms of cracks and iron oxide scale indentation, the recognition effect is relatively poor, and the AP value is only 22.3% and 43.1%. The Faster R-CNN model that introduced the FPN module in Experiment 2 improved the mAP by 17.2% compared with the traditional Faster R-CNN model. The recognition effect of defects that are easy to miss and difficult to detect is significantly enhanced, showing a more superior performance. After improving the pooling method in Experiment 3, although the detection effect of plaques and pits decreased slightly, the overall mAP increased by 1.4%. In Experiment 4, deformation convolution was introduced. Although the detection effect of scratches decreased slightly, the detection effect of cracks and plaques was more obvious, and the detection effect of other types of defects was also slightly improved. The overall mAP increased by 1.3% compared with Experiment 3. The SIoU used in Experiment 5 replaced the original SmoothL1 loss function, and the detection effect of various defects was further improved. Although the detection effect of scratches did not reach the optimal level, the highest mAP value was achieved overall, further proving the effectiveness of the improved model proposed in this invention in surface defect detection.

[0124] The present invention proposes an improved Faster R-CNN rail defect detection algorithm, which aims to achieve rapid detection and classification of rail surface defects. The algorithm is based on Faster R-CNN and improves the network detection performance by improving the local network and adding modules. The deformation convolution is introduced into the feature extraction network to improve the ability to extract irregular defect features, and the FPN module is introduced to strike a balance between details and semantics, enhance the ability to extract small target features, and improve the accuracy of rail defect detection. The pooling method of the original algorithm is improved to reduce the positioning error in the candidate area extraction process and improve the positioning accuracy of defects. The loss function of the original algorithm is replaced to improve the defect accuracy of defect targets and make the handling of category imbalance more effective. The improved model has achieved higher accuracy in experiments on the NEU-DET dataset, but there is still more room for improvement in the detection effect of cracks and iron oxide scale indentations. Future work will further study the detection method for small target defects to improve the detection performance of the algorithm.

[0125] S3: The surface image of the rail to be inspected is input into the detection model. The sampling points of the convolution kernel are adaptively adjusted by improving the deformable convolution layer in the Resnet50 module to capture the irregular geometric features of the rail surface defects and obtain a defect geometric feature map. The defect geometric feature map is enhanced at different scales by improving the multi-scale upsampling and downsampling operations of the feature pyramid network in the Resnet50 module to obtain a multi-scale defect enhancement feature map. The multi-scale defect enhancement feature map is detected by the region proposal module to generate several defect candidate regions. The defect feature vectors corresponding to several defect candidate regions are extracted from the multi-scale defect enhancement feature map by the ROI-Align module. The defect feature vector is mapped into a detection result containing the rail surface defect category and the rail surface defect bounding box by the target classification and bounding box prediction module.

[0126] The above is a rail surface defect detection method provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding rail surface defect detection device, including:

[0127] A model building module is used to replace the standard convolutional layer of the Resnet50 module in the original Faster R-CNN network with a deformable convolutional layer; and to add a feature pyramid network to the output end of the Resnet50 module in the original Faster R-CNN network to build an improved Resnet50 module; replace the ROIPooling module of the original Faster R-CNN network with a ROI-Align module; and build an improved Faster R-CNN network including an improved Resnet50 module, a region proposal module of the original Faster R-CNN network, a ROI-Align module, and an object classification and bounding box prediction module connected in sequence.

[0128] The model training module is used to collect rail surface defect images to build a data set, and use the data set to train the improved Faster R-CNN network to obtain a detection model for detecting rail surface defects.

[0129] The defect detection module is used to input the surface image of the rail to be detected into the detection model, and adaptively adjust the sampling points of the convolution kernel by improving the deformable convolution layer in the Resnet50 module to capture the irregular geometric features of the rail surface defects and obtain a defect geometric feature map; the defect geometric feature map is enhanced at different scales by improving the multi-scale upsampling and downsampling operations of the feature pyramid network in the Resnet50 module to obtain a multi-scale defect enhancement feature map; the multi-scale defect enhancement feature map is detected by the region proposal module to generate several defect candidate regions; the defect feature vectors corresponding to several defect candidate regions are extracted from the multi-scale defect enhancement feature map by the ROI-Align module; the defect feature vector is mapped into a detection result containing the rail surface defect category and the rail surface defect bounding box by the target classification and bounding box prediction module.

[0130] The specific definition of the rail surface defect detection device can be found in the definition of the rail surface defect detection method above, which will not be repeated here. Each module in the above-mentioned rail surface defect detection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0131] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A rail surface defect detection method is provided.

[0132] The present invention also provides a computer device. At the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 A rail surface defect detection method is provided.

[0133] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0134] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A rail surface defect detection method, characterized in that: include: Replace the standard convolutional layer of the Resnet50 module in the original Faster R-CNN network with a deformable convolutional layer; add a feature pyramid network at the output of the Resnet50 module in the original Faster R-CNN network to construct an improved Resnet50 module; replace the ROIPooling module of the original Faster R-CNN network with a ROI-Align module; construct an improved Faster R-CNN network including the improved Resnet50 module, the region proposal module of the original Faster R-CNN network, the ROI-Align module, and the target classification and bounding box prediction module connected in sequence; Collect Faster R-CNN network images to build a data set, use the data set to train the improved Faster R-CNN network, and obtain a detection model for detecting rail surface defects; The surface image of the rail to be inspected is input into the inspection model. The sampling points of the convolution kernel are adaptively adjusted by improving the deformable convolution layer in the Resnet50 module to capture the irregular geometric features of the rail surface defects and obtain the defect geometric feature map. By improving the multi-scale upsampling and downsampling operations of the feature pyramid network in the Resnet50 module, the defect geometric feature map is enhanced at different scales to obtain a multi-scale defect enhancement feature map; the multi-scale defect enhancement feature map is detected through the region proposal module to generate several defect candidate regions; The defect feature vectors corresponding to several defect candidate regions are extracted from the multi-scale defect enhancement feature map through the ROI-Align module; the defect feature vectors are mapped into detection results containing rail surface defect categories and rail surface defect bounding boxes through the target classification and bounding box prediction module.

2. The rail surface defect detection method according to claim 1, characterized in that: The defect geometric feature map is enhanced at different scales by correspondingly connecting each downsampling layer in the feature pyramid network with the output end of each residual module in the Resnet50 module to enhance the defect geometric feature maps of different scales extracted by each residual module.

3. The rail surface defect detection method according to claim 1, characterized in that: The defect geometric feature map is enhanced at different scales by connecting the first downsampling layer in the feature pyramid network to the output end of the last residual module of the Resnet50 module to enhance the defect geometric feature map of the minimum scale output by the Resnet50 module.

4. The rail surface defect detection method according to claim 1, characterized in that: The defect feature vectors corresponding to several defect candidate regions are extracted from the multi-scale defect enhancement feature map by using the ROI-Align module, specifically including: The multi-scale defect enhancement feature map and multiple ROI bounding boxes are used as the input of the ROI-Align module; Each ROI bounding box is divided into a grid of fixed size, and four sampling points are evenly placed in each grid unit. The eigenvalues ​​corresponding to the four sampling points are calculated using bilinear interpolation. In each grid cell, the feature values ​​of the four sampling points are subjected to a maximum pooling operation to obtain the feature value of the corresponding grid cell, and the feature value is used as a pixel value in the feature map output by the ROI-Align module; Based on the feature values ​​of all grid cells in each ROI bounding box, each ROI bounding box generates a fixed-size feature map as the corresponding defect feature vector.

5. The rail surface defect detection method according to claim 1, characterized in that: The improved FasterR-CNN network is trained using the data set, and the loss function used is the SIoU loss function; the SIoU loss function calculates the gap between the predicted bounding box and the true bounding box by introducing angle loss, distance loss, shape loss and IoU loss, so as to guide the weight coefficient of the improved FasterR-CNN network to be updated in the direction of minimizing the gap between the predicted bounding box and the true bounding box.

6. A rail surface defect detection device, characterized in that: include: A model building module for replacing the standard convolutional layer of the Resnet50 module in the original Faster R-CNN network with a deformable convolutional layer; adding a feature pyramid network to the output of the Resnet50 module in the original Faster R-CNN network to build an improved Resnet50 module; replacing the ROIPooling module of the original Faster R-CNN network with a ROI-Align module; and building an improved Faster R-CNN network including the improved Resnet50 module, the region proposal module of the original Faster R-CNN network, the ROI-Align module, and the object classification and bounding box prediction module connected in sequence; The model training module is used to collect rail surface defect images to construct a data set, and use the data set to train the improved FasterR-CNN network to obtain a detection model for detecting rail surface defects; The defect detection module is used to input the surface image of the rail to be inspected into the detection model, and adaptively adjust the sampling points of the convolution kernel by improving the deformable convolution layer in the Resnet50 module to capture the irregular geometric features of the rail surface defects and obtain the defect geometric feature map; By improving the multi-scale upsampling and downsampling operations of the feature pyramid network in the Resnet50 module, the defect geometric feature map is enhanced at different scales to obtain a multi-scale defect enhancement feature map; the multi-scale defect enhancement feature map is detected through the region proposal module to generate several defect candidate regions; The defect feature vectors corresponding to several defect candidate regions are extracted from the multi-scale defect enhancement feature map through the ROI-Align module; the defect feature vectors are mapped into detection results containing rail surface defect categories and rail surface defect bounding boxes through the target classification and bounding box prediction module.

7. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of claims 1 to 5 is implemented.

Images