Rail damage detection method and device

Abstract

This paper relates to the field of rail damage detection and provides a rail damage detection method and device. The method comprises obtaining a rail image to be inspected; constructing input data, wherein the input data includes the rail image to be inspected and supporting images for each rail damage category, where the supporting images are labeled with the damage category and damage location; inputting the input data into a rail damage detection model to obtain the damage category and damage location; and the rail damage detection model is used to fuse the feature similarity between the rail damage image to be inspected and the supporting images in the rail damage image to obtain an aggregated feature map, and then identifying the aggregated feature map to obtain the damage category and damage location. This paper can improve the accuracy of damage category detection and reduce the rail damage detection model's sensitivity to negligible damage.

Description

Technical Field

[0001] This article relates to the field of rail damage detection, and in particular to a rail damage detection method and device. Background Art

[0002] Rails are a key component of railway tracks and a fundamental part of railway transportation infrastructure. They guide the wheels of trains, directly bearing the forces exerted by the train wheels and transmitting them to the sleepers. The condition of the rails directly impacts the safety of train operations. Due to long-term exposure to the elements, including wind, frost, rain, snow, high temperatures, and low temperatures, as well as the pressure of train operation, rails inevitably suffer various types of damage, posing a variety of safety risks.

[0003] The main defects on the rail surface include: rail surface abrasions, peeling blocks, cracks, rail head crushing, and tread wave wear. Different defects have different impacts on train operation safety. Rail surface defect detection is one of the most important links in the inspection and maintenance of track safety facilities.

[0004] In the existing technology, rail damage detection is mainly based on rail images. Rail images are usually obtained by mounting a camera on the bottom of the inspection vehicle and continuously imaging the rail surface during inspection to obtain rail surface images. Rail surface defects are then identified using rail surface defect image recognition technology. Compared with general identification tasks, rail surface defect detection and identification has its own special difficulties and challenges. The environment where the railway passes is complex, and various natural conditions, such as light, wind, frost, rain and snow, make the quality of the collected on-site images inconsistent, which greatly increases the difficulty of defect identification. Finally, the pattern of rail surface defects is also relatively complex. For example, some defects are relatively subtle and only occupy a small part compared to the rails, which also increases the difficulty of detection and identification to a certain extent. Rail surface defect image recognition technologies mainly include the following two types:

[0005] 1. Machine Learning

[0006] Based on rail images captured by cameras, a deep learning target localization algorithm is used to detect rail surface defects. First, a large number of rail surface defect images are manually collected and labeled. A rail surface defect detection model is trained based on the labeled data. The trained rail surface defect detection model is then used to detect rail surface defects in the rail images.

[0007] The method of rail damage detection based on machine learning has the following defects: a large number of rail defect image samples are required (hundreds of megabytes, gigabytes, or even terabytes of data). In fact, there is no way to obtain a large number of image samples and label them; the detectable defects must be defects of the category that have appeared in the samples; for small sample sizes, rail surface defect technology is prone to missed detection in the absence of samples.

[0008] 2. Few-shot Learning

[0009] A deep neural network and a deep forest model are used to model rail surface scratches. After image preprocessing, the trained model is used to detect rail surface defects in the image. Both methods specifically address the small sample size problem.

[0010] The rail damage detection method based on small sample learning has the following defects: there is a lack of design of specific modules for rail damage detection, and similarity information is not fully utilized. Therefore, compared with the supervised method using a large number of annotations, there are certain deficiencies in detection accuracy and it is easy to generate a large number of false alarms. Summary of the Invention

[0011] This paper aims to solve the problems in the existing technology of rail damage detection methods based on machine learning, such as low detection accuracy, easy missed detection, and the need to label a large number of rail defect images; and the problem of rail damage detection methods based on small sample learning, which easily detects minor damage and generates a large number of false alarms, thereby increasing the review work.

[0012] In order to solve the above technical problems, a first aspect of this invention provides a rail damage detection method, comprising:

[0013] Acquire the image of the rail to be inspected;

[0014] Constructing input data, wherein the input data includes an image of a rail to be inspected and supporting images of each rail damage category, wherein the supporting images are images with damage categories and damage locations marked;

[0015] Inputting the input data into a rail damage detection model to obtain damage type and damage location;

[0016] The rail damage detection model is used to fuse the feature similarity between the rail damage image to be detected and the support image in the rail damage image to obtain an aggregated feature map, and identify the aggregated feature map to obtain the damage category and damage location.

[0017] As a further embodiment of this article, the rail damage detection model includes: a feature extraction module, a relationship fusion module and a classification module;

[0018] Utilizing the feature extraction module to extract a query feature map from the rail damage image to be detected and a supporting feature map from each supporting image;

[0019] The query feature graph and the support feature graph are fused using the relationship fusion module as follows: similarity between the query feature graph and each support feature graph is calculated; and the query feature graph is weighted using the calculated similarity to obtain an aggregated feature graph, wherein the aggregated feature graph has the same size as the query feature graph;

[0020] The classification module is used to identify the aggregated feature map and obtain the damage category and damage location.

[0021] As a further embodiment of this invention, the relationship fusion module calculates the similarity between the query feature graph and each supporting feature graph, and uses the calculated similarity to perform weighted processing on the query feature graph to obtain an aggregated feature graph including:

[0022] Encoding the query feature map to obtain a query value feature map and a query key feature map;

[0023] Encoding multiple support feature maps to obtain a support value feature map and a support key feature map;

[0024] Perform linear transformation on the query key feature graph and the support key feature graph respectively, and perform product processing on the transformed feature graphs to obtain the key feature graph of relationship similarity;

[0025] Normalize the key feature map of the relationship similarity, multiply the normalized feature map with the support value feature map to obtain the weighted support value feature map;

[0026] The query value feature map and the weighted support value feature map are concatenated to obtain an aggregated feature map.

[0027] As a further embodiment of the present invention, the first dimension of the query value feature map is the product of the width and height of the query value feature map, and the second dimension of the query value feature map is the number of channels after halving;

[0028] The first dimension of the support value feature map is the product of the number of support feature maps, width and height, and the second dimension of the support value feature map is the number of channels after halving;

[0029] The first dimension of the query key feature graph is the product of the width and height of the query key feature graph, and the second dimension of the query key feature graph is 1 / 8 of the number of channels;

[0030] The first dimension of the support key feature map is 1 / 8 of the number of channels, and the second dimension of the support key feature map is the product of the number of support feature maps, width and height.

[0031] As a further embodiment of this article, the rail damage detection model further includes: a multi-scale network module and a sampling splicing module;

[0032] Down-sampling the query feature graph and the support feature graph respectively using the multi-scale network module to obtain a first feature subgraph and a second feature subgraph of M scales;

[0033] The first feature subgraph and the second feature subgraph of the smallest scale are fused by the relationship fusion module to obtain an aggregate feature graph of the Mth scale; the feature graph output by the sampling splicing module is fused to obtain an aggregate feature graph of the first M-1 scales;

[0034] The sampling and splicing module is used to upsample the aggregate feature map of the mth scale, and the first feature submap and the second feature submap of the m-1th scale are spliced ​​with the upsampled aggregate feature map of the mth scale, where the value range of m is 2 to M.

[0035] As a further embodiment of this invention, the rail damage detection model training process includes:

[0036] Determine a plurality of batch samples, wherein the batch samples include: a query image and supporting images of each rail damage category; the query image is an original rail damage image;

[0037] Input the batch samples into the rail damage detection model to predict the damage type and damage location of the query image in the batch samples;

[0038] Calculate the loss function based on the predicted damage type and damage location, as well as the damage type and damage location annotated in the query image;

[0039] The parameters in the rail damage detection model are trained according to the loss function.

[0040] As a further embodiment of this invention, before determining the plurality of batches of samples, the method further includes:

[0041] The parameters of the rail damage detection model are pre-trained using the object detection dataset.

[0042] As a further embodiment of this invention, the loss function is calculated using the following formula:

[0043] loss=l box +l cls +l obj ;

[0044]

[0045]

[0046]

[0047] Among them, loss is the loss function, l box is the damage location frame regression error, l cls is the classification error, l obj is the target detection error, λ coord ,λ class ,λ noobj ,λ obj is the weight, S 2 The number of times each image in the input data is divided into S×S grids for detection traversal, Is there a target of interest in the (i, j)th damage location candidate box, ω i 、h i 、x i 、y i are the width, height, horizontal coordinate and vertical coordinate of the predicted damage location candidate box, are the width, height, horizontal coordinate and vertical coordinate of the damage location box, B is the number of candidate boxes for the damage location, and classes is the number of rail damage categories;

[0048] p i (c) is the probability that the object in grid i belongs to category c, is the true probability that the object belongs to category c;

[0049] Is there any target of interest in the (i, j)th damage location candidate box? i is the predicted damage category, The damage category is marked.

[0050] A second aspect of the present invention provides a rail damage detection device, comprising:

[0051] An acquisition module is used to acquire the image of the rail to be detected;

[0052] A preprocessing module is used to construct input data, wherein the input data includes the image of the rail to be inspected and supporting images of each rail damage category, wherein the supporting images are images with damage categories and damage locations marked;

[0053] A detection module, configured to input the input data into a rail damage detection model to obtain damage categories and damage locations;

[0054] The rail damage detection model is used to fuse the feature similarity between the rail damage image to be detected and the support image in the rail damage image to obtain an aggregated feature map, and identify the aggregated feature map to obtain the damage category and damage location.

[0055] A third aspect of this document provides a computer device comprising a memory, a processor, and a computer program and a rail damage detection model stored in the memory, wherein when the computer program is run by the processor, the computer program executes instructions of the method according to any of the aforementioned embodiments.

[0056] A fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor of a computer device, the computer program executes instructions of the method described in any of the aforementioned embodiments.

[0057] The rail damage detection method and device provided in this article pre-establish a rail damage detection model. The model is used to fuse the feature similarities between the image of the rail damage to be detected and the supporting image (an image with damage classification and damage location labeled) to obtain an aggregated feature map. The aggregated feature map is then identified to obtain the damage classification and damage location. This method can achieve the following two technical effects:

[0058] (1) Improve the detection accuracy of the rail damage detection model for known damage categories. Fully utilize the supporting images of each rail damage category so that the image of the rail to be detected can be integrated with the feature similarity between the image of the rail to be detected and all supporting images, and the recognition results can be made closer to the known rail damage categories, thereby improving the discrimination and detection accuracy of the damage categories.

[0059] (2) The rail damage detection model’s sensitivity to negligible damage is reduced. By calculating the feature similarity between the rail damage image to be detected and the supporting image, the similarity calculated for negligible damage (small area, shallow damage, etc.) is generally low. The similarity is integrated into the rail damage image to be detected, which reduces the confidence level of the detection results for negligible damage. This lowers the confidence level of the rail damage detection model for these negligible damages, avoids detecting negligible minor damage, and thus saves valuable subsequent review work and improves detection efficiency.

[0060] In order to make the above and other purposes, features and advantages of this article more obvious and easy to understand, the following specifically cites preferred embodiments and provides detailed descriptions in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of this article or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of this article. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 A flow chart showing the rail damage detection model training process according to the embodiment of this article is shown;

[0063] Figure 2 A flow chart showing a batch sample determination process according to an embodiment of the present invention is shown;

[0064] Figure 3 A first structural diagram of the rail damage detection model according to an embodiment of this invention is shown;

[0065] Figure 4 The structure diagram of the relationship fusion model of the embodiment of this article is shown;

[0066] Figure 5 A second structural diagram of the rail damage detection model according to an embodiment of this invention is shown;

[0067] Figure 6 A flow chart of a rail damage detection method according to an embodiment of the present invention is shown;

[0068] Figure 7 A flowchart showing a similarity calculation process between a query feature graph and each supporting feature graph according to an embodiment of this invention is shown;

[0069] Figure 8 The structure diagram of the rail damage detection system according to the embodiment of this invention is shown;

[0070] Figure 9 The structure diagram of the rail damage detection device according to the embodiment of this invention is shown;

[0071] Figure 10 The diagram shows the structure of the computer device according to the embodiment of this article.

[0072] Description of the accompanying symbols:

[0073] 210, Dataset;

[0074] 220, support set;

[0075] 230, query set;

[0076] 221, 231, images;

[0077] 240, batch sample;

[0078] 301. Feature extraction module;

[0079] 302, Relationship Fusion Module;

[0080] 303, classification module;

[0081] 304, multi-scale network module;

[0082] 3041, first characteristic subgraph;

[0083] 3042, second characteristic subgraph;

[0084] 305. Sampling and splicing module;

[0085] 401, encoder;

[0086] 402, encoder;

[0087] 403. A first linear transformation unit;

[0088] 404. Second linear transformation unit;

[0089] 405, first multiplication operation unit;

[0090] 406, normalization unit;

[0091] 407, second multiplication operation unit;

[0092] 408, splicing operation unit;

[0093] 801, database;

[0094] 802. Computing equipment;

[0095] 901. Get module;

[0096] 902, preprocessing module;

[0097] 903, detection module;

[0098] 1002. Computer equipment;

[0099] 1004, processor;

[0100] 1006. Memory;

[0101] 1008, driving mechanism;

[0102] 1010, input / output module;

[0103] 1012. Input device;

[0104] 1014. Output device;

[0105] 1016. Presentation equipment;

[0106] 1018. Graphical user interface;

[0107] 1020, network interface;

[0108] 1022, communication link;

[0109] 1024. Communication bus. DETAILED DESCRIPTION

[0110] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of this document. Obviously, the embodiments described are only part of the embodiments of this document, not all of the embodiments. Based on the embodiments of this document, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this document.

[0111] It should be noted that the terms "first," "second," and the like in the specification and claims herein and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or devices.

[0112] This specification provides method operation steps as described in the embodiments or flowcharts, but more or fewer operation steps may be included based on routine or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many orderings and does not represent the only execution order. When a system or device product is actually executed, the method can be executed in the order shown in the embodiments or the drawings or in parallel.

[0113] It should be noted that the data involved in this application (including but not limited to data used for detection, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties.

[0114] In the existing technology, the rail damage detection method based on machine learning has the problems of low detection accuracy, easy missed detection and the need to label a large number of rail defect images. The rail damage detection method based on small sample learning has the problem of easily detecting minor damage and generating a large number of false alarms, thereby increasing the review work.

[0115] To solve the above technical problems, this paper constructs and trains a rail damage detection model. The rail damage detection model is used to receive input data, where the input data includes a query image and supporting images of each rail damage category. The feature similarity between the query image and the supporting images of each rail damage category is fused in the query image to obtain an aggregated feature map. The aggregated feature map is identified to obtain the damage category and damage location.

[0116] The internal structure of the rail damage detection model can be configured based on the aforementioned functions. The query image is an image with unidentified rail damage categories, and the supporting images for each rail damage category include images with the rail damage category and damage location labeled for each rail damage category. The rail damage categories described herein can refer to either major rail damage categories or minor damage categories, including but not limited to breakage, cracks, and other conditions that affect and limit rail performance during use. A rail crack refers to the separation of part of the metal on or within the rail. Other rail damage refers to damage other than cracks and breaks that affects rail performance, such as wear, crushing, depression (or dents), wave wear, bending deformation, surface defects, and external corrosion. The rail damage state refers to the macroscopic morphology (or shape) characteristics of the rail damage, which directly records and reflects the development process and outcome of the damage. The rail damage location can be represented by the width, height, abscissa, and ordinate of the damage location box. During implementation, the supporting image may be represented by an original damage image and a damage location image, wherein the damage location in the damage location image is white, and the remaining locations are black or gray.

[0117] like Figure 1 As shown in Figure 2, the rail damage detection model training process includes:

[0118] Step 110 , determining a plurality of batch samples, the batch samples including: a query image and supporting images of each rail damage category, the query image being the original rail damage image;

[0119] Step 120: Input the batch samples into the rail damage detection model to predict the damage type and damage location of the query image in the batch samples;

[0120] Step 130 , calculating a loss function based on the predicted damage type and damage location, and the damage type and damage location annotated in the query image;

[0121] Step 140: training parameters in the rail damage detection model according to the loss function.

[0122] In some embodiments, Figure 2 As shown, the batch samples in step 110 are determined as follows:

[0123] (1) Obtain a data set 210, wherein the data set includes a plurality of original rail damage images.

[0124] (2) Label the categories and damage locations of some original rail damage images in the dataset to obtain support images for each rail damage category. The support images constitute a support set 220, and the unlabeled original rail damage images in the dataset constitute a query set 230. When implementing this step, first determine N types of typical rail damage. For each damage category, select the most representative K images as support images, where K is generally 1 or 2. 1 image can ensure higher model convergence stability, but slightly lower accuracy. 2 images can ensure the generalization ability, higher accuracy, and stronger adaptability of the model, but slightly lower stability. Of course, in specific implementation, K can also be greater than 2, because each category is equivalent to setting too many support images, which makes the model distracted and does not know which image can be used as a typical example of the category. Therefore, when K is greater than 2, the performance of rail damage will decrease.

[0125] (3) Extract an image 231 from the query set and N images 221 from the support set, that is, extract a support image from each rail damage category in the support set (if the support set includes multiple support images for each rail damage category, then randomly extract one from the multiple support images), and the extracted N+1 images constitute a batch sample 240.

[0126] During implementation, in order to improve the model's adaptability to interference in the data, the supporting images in the batch samples are also masked according to the annotation information.

[0127] During model training, in order to accelerate convergence, each graphic in the batch sample can be cropped in advance. When step 130 is implemented, the loss function is calculated using the following formula:

[0128] loss=l box +l cls +l obj ;

[0129]

[0130]

[0131]

[0132] Among them, loss is the loss function, l box is the damage location frame regression error, l cls is the classification error, l obj is the target detection error, λ coord ,λ class ,λ noobj ,λ obj is the weight, S 2 The number of times each image in the input data is divided into S×S grids for detection traversal, Is there a target of interest (i.e., damage) in the (i, j)th damage location candidate box? i ,h i ,x i ,y i are respectively the width, height, abscissa and ordinate (i.e. the coordinates of the center point in the candidate box) of the predicted damage location candidate box. are the width, height, abscissa, and ordinate of the damage location box (i.e., the coordinates of the center point in the candidate box), B is the number of candidate boxes for the damage location, and classes is the number of rail damage categories.

[0133] p i (c) is the probability that the object in grid i belongs to category c, is the true probability that the object belongs to category c;

[0134] Is there any target of interest in the (i, j)th damage location candidate box? i is the predicted damage category, The damage category is marked.

[0135] When step 140 is implemented, the loss function is differentiated and the derivative is back-propagated to each parameter of the rail damage detection model, thereby completing the parameter update.

[0136] During specific training, a training cycle (e.g., 10 training cycles) may be specified in advance. After the training cycle is completed, the model training is completed. Each training cycle means that all batches of samples are fed into the rail damage detection model once.

[0137] In this paper, in order to improve the model training speed, Figure 1 Before executing the training process, the following steps are performed: pre-training the parameters of the rail damage detection model using an object detection dataset. The object detection dataset described herein is a pre-annotated image dataset, such as the Microsoft COCO dataset. Fine-tuning the parameters of the pre-trained rail damage detection model is performed using batches of rail damage samples.

[0138] In one embodiment of this paper, in order to improve the accuracy of model training, the batch samples are divided into training set, validation set and test set according to a certain ratio (such as 7:2:1). The entire training process of the rail damage detection model includes:

[0139] First, the training phase: The training process from step 120 to step 140 is performed using the batch samples in the training set.

[0140] Second, during the validation phase, batch samples from the validation set are fed into the rail damage detection model to calculate precision and recall. The calculated precision and recall are then used to determine whether to adjust the model parameters. Specifically, if the precision and recall do not meet the pre-set conditions (defined by the standard precision and recall values), the model parameters are retrained.

[0141] Precision is the ratio of correctly predicted positive results to all predicted positive results. Recall is the ratio of correctly predicted positive results to all actual positive results. The calculation process is as follows:

[0142] a) Set the IOU threshold to 0.5, where the IOU threshold reflects the intersection-over-union standard value between the model output detection box and the support image annotation box;

[0143] b) Let TP be the number of positive samples predicted as positive (True Positive), that is, the detection box detects the object (damage) and the IOU (Intersection-over-Union ratio, that is, the intersection-over-Union ratio of the detection box output by the model and the annotation box of the support image, the detection box output by the model is the location information of the rail damage) is greater than the threshold, indicating a successful detection;

[0144] c) Let FP be the number of false positives, i.e., the detection box detects the object but the IOU is less than the threshold, indicating a false detection.

[0145] d) Let FN be the number of false negative samples, i.e., the detection box fails to detect the object but actually contains the object, indicating missed detection.

[0146] e) Let TN be the number of negative samples predicted as positive (True Negative), i.e., the detection box failed to detect the object and actually did not contain the object. This cannot be calculated and does not need to be calculated;

[0147] f) Accuracy

[0148] g) Recall rate

[0149] Among them, TP, FP, FN, and TN are obtained by manually comparing the model output data and the labeled data.

[0150] Third, after the precision and recall meet the preset conditions, the mean average precision (mAP) of each rail damage detection model trained in step (2) is calculated. The mAP calculation process includes:

[0151] a) Gradually increase the IOU threshold from 0 to 1;

[0152] b) For each IOU threshold, a set of precision P and recall R can be calculated;

[0153] c) With P as the vertical coordinate and R as the horizontal coordinate, a two-dimensional curve can be drawn, namely the PR curve;

[0154] d) The area under the curve is AP (Average Precision);

[0155] e) Repeat the above operation for all categories to obtain the AP of each category, and calculate the average to get the mAP.

[0156] Fourth, testing phase. The rail damage detection model with the best performance (i.e., the highest mAP) in the validation set is selected and the mAP is calculated in the test set. If the calculated mAP meets the preset threshold, the model training is considered complete. Otherwise, the above step (1) is repeated.

[0157] In one embodiment of this invention, Figure 3 As shown, the rail damage detection model includes: a feature extraction module 301, a relationship fusion module 302 and a classification module 303.

[0158] The input of the feature extraction module 301 is the input of the rail damage detection model, which is used to receive input data, extract a query feature map from the rail damage image to be detected, and extract a supporting feature map from each supporting image.

[0159] The relationship fusion module 302 is used to perform the following fusion processing on the query feature graph and the support feature graph: calculate the similarity between the query feature graph and each support feature graph; and use the calculated similarity to weight the query feature graph to obtain an aggregated feature graph, which has the same size as the query feature graph.

[0160] The classification module 303 is used to identify the aggregated feature map and obtain the damage category and damage location.

[0161] During implementation, the feature extraction module 301 may utilize a multi-layer convolutional network to identify the rail damage image to be detected to obtain a query feature map, and identify the support image to obtain a support feature map.

[0162] The classification module 303 can be designed using an existing classifier architecture, and this document does not limit its specific internal structure.

[0163] like Figure 4As shown, the relationship fusion module 302 includes: an encoder 401, an encoder 402, a first linear transformation unit 403, a second linear transformation unit 404, a first multiplication operation unit 405, a normalization unit 406, a second multiplication operation unit 407 and a splicing operation unit 408.

[0164] The encoder 401 encodes the query feature map to obtain a query two-dimensional feature map, wherein the query two-dimensional feature map includes a query value feature map and a query key feature map;

[0165] The encoder 402 encodes the plurality of support feature maps to obtain a support two-dimensional feature map, wherein the support two-dimensional feature map includes a support value feature map and a support key feature map;

[0166] The first linear transformation unit 403 performs linear transformation on the query key feature graph;

[0167] The second linear transformation unit 404 performs linear transformation on the support key feature map;

[0168] The first multiplication operation unit 405 performs a product process on the query key feature graph after the linear transformation and the support key feature graph after the linear transformation to obtain a relation similarity key feature;

[0169] Normalization unit 406 (softmax layer) normalizes the relationship similarity key features;

[0170] The second multiplication operation unit 407 multiplies the normalized feature and the support value feature map to obtain a weighted support value feature map;

[0171] The splicing operation unit 408 performs splicing processing on the query value feature map and the weighted support value feature map to obtain an aggregated feature map.

[0172] In one specific embodiment, the first dimension of the query value feature map is the product of the width W and height H of the query feature map, that is, HW, and the second dimension of the query value feature map is the number of channels after halving, that is, C / 2;

[0173] The first dimension of the support value feature map is the product of the number of support feature maps N, width W and height H, that is, NHW, and the second dimension of the support value feature map is the number of channels after halving, that is, C / 2;

[0174] The first dimension of the query key feature map is the product of the width W and height H of the query feature map, that is, HW, and the second dimension of the query key feature map is 1 / 8 of the number of channels, that is, C / 8;

[0175] The first dimension of the support key feature map is 1 / 8 of the number of channels, that is, C / 8, and the second dimension of the support key feature map is the product of the number N of support feature maps, width W and height H, that is, NHW.

[0176] Figure 4 The relationship fusion module shown can retain a part of the feature information of the query feature graph and fuse the relationship features between the query feature graph and all supporting feature graphs. Figure 4 The relationship fusion module shown can also make the detection results closer to the known N support categories, including ignoring unlabeled minor damage (not belonging to the N categories of known features), and suppressing other features that have never been seen (also not belonging to the N categories of known features) such as background noise, thereby improving recognition accuracy.

[0177] In one embodiment of this invention, Figure 5 As shown, the rail damage detection model further includes: a multi-scale network module 304 and a sampling splicing module 305.

[0178] The multi-scale network module 304 downsamples the query feature graph and the support feature graph output by the feature extraction module 301 to obtain a first feature subgraph 3041 and a second feature subgraph 3042 of M scales, where M is a positive integer and can be set according to actual needs.

[0179] The relationship fusion module 302 fuses the first feature subgraph and the second feature subgraph of the smallest scale to obtain an aggregate feature graph of the Mth scale; and fuses the feature graph output by the sampling splicing module to obtain an aggregate feature graph of the first M-1 scales.

[0180] The sampling and splicing module 305 upsamples the aggregate feature map of the mth scale, and splices the first feature submap and the second feature submap of the m-1th scale with the upsampled aggregate feature map of the mth scale, where the value of m ranges from 2 to M.

[0181] The relationship fusion module 302 will output M aggregated feature maps, and the classification module 303 will identify the M aggregated feature maps respectively to obtain M damage categories and damage location recognition results. The M damage categories and damage location recognition results are comprehensively analyzed to obtain the final damage category and damage location.

[0182] In one embodiment of this invention, a rail damage detection method is also provided. This method can be run on a third-party system with independent data sources, smart terminals, including smart phones, tablet computers, desktop computers, etc. It can also be a separate application, a small program embedded in other programs, or a web page. This invention does not limit the specific implementation method. Specifically, Figure 6 As shown in FIG, the rail damage detection method includes:

[0183] Step 610, obtaining an image of the rail to be inspected;

[0184] Step 620: construct input data, wherein the input data includes the image of the rail to be inspected and supporting images of each rail damage category, and the supporting images can be obtained from the support library;

[0185] Step 630: input the input data into a rail damage detection model to obtain damage type and damage location;

[0186] Among them, the rail damage detection model can be trained using the method described in any of the above embodiments, and is used to fuse the feature similarity between the rail damage image to be detected and the support image in the rail damage image to obtain an aggregated feature map, and identify the aggregated feature map to obtain the damage category and damage location.

[0187] This embodiment can achieve the following two technical effects:

[0188] (1) Improve the detection accuracy of the rail damage detection model for known damage categories. Fully utilize the supporting images of each rail damage category so that the image of the rail to be detected can be integrated with the feature similarity between the image of the rail to be detected and all supporting images, and the recognition results can be made closer to the known rail damage categories, thereby improving the discrimination and detection accuracy of the damage categories.

[0189] (2) The rail damage detection model’s sensitivity to negligible damage is reduced. By calculating the feature similarity between the rail damage image to be detected and the supporting image, the similarity calculated for negligible damage (small area, shallow damage, etc.) is generally low. The similarity is integrated into the rail damage image to be detected, which reduces the confidence level of the detection results for negligible damage. This lowers the confidence level of the rail damage detection model for these negligible damages, avoids detecting negligible minor damage, and thus saves valuable subsequent review work and improves detection efficiency.

[0190] Specifically, refer to Figure 3 The rail damage detection model shown in FIG. 1 includes the following steps to identify the input data:

[0191] Using a feature extraction module to extract a query feature map from the rail damage image to be detected and a supporting feature map from each supporting image;

[0192] The query feature graph and the support feature graph are fused using the relationship fusion module as follows: the similarity between the query feature graph and each support feature graph is calculated; and the query feature graph is weighted using the calculated similarity to obtain an aggregated feature graph, which has the same size as the query feature graph;

[0193] The classification module is used to identify the aggregated feature map and obtain the damage category and damage location.

[0194] Further, such as Figure 7 As shown in FIG, the relationship fusion module calculates the similarity between the query feature graph and each supporting feature graph; and uses the calculated similarity to perform weighted processing on the query feature graph to obtain the aggregated feature graph. The process includes:

[0195] Step 710: Encode the query feature map to obtain a query value feature map and a query key feature map, wherein the first dimension of the query value feature map is the product of the width and height of the query feature map, and the second dimension of the query value feature map is the number of channels after halving. The first dimension of the query key feature map is the product of the width and height of the query feature map, and the second dimension of the query key feature map is 1 / 8 of the number of channels.

[0196] Step 720: Encode the multiple support feature maps to obtain a support value feature map and a support key feature map, wherein the first dimension of the support value feature map is the product of the number of support feature maps, width, and height, and the second dimension of the support value feature map is the number of channels after halving; the first dimension of the support key feature map is 1 / 8 of the number of channels, and the second dimension of the support key feature map is the product of the number of support feature maps, width, and height;

[0197] Step 730: perform linear transformation on the query key feature graph and the support key feature graph respectively, and perform product processing on the transformed feature graphs to obtain the key features of the relationship similarity;

[0198] Step 740: normalize the relationship similarity key features, and multiply the normalized features with the support value feature map to obtain a weighted support value feature map;

[0199] Step 750 : Concatenate the query value feature map and the weighted support value feature map to obtain an aggregated feature map.

[0200] refer to Figure 5 The rail damage detection model shown in FIG. 1 includes the following steps to identify the input data:

[0201] Using a feature extraction module to extract a query feature map from the rail damage image to be detected and a supporting feature map from each supporting image;

[0202] Down-sampling the query feature graph and the support feature graph respectively using a multi-scale network module to obtain a first feature subgraph and a second feature subgraph of N scales;

[0203] The first feature subgraph and the second feature subgraph of the smallest scale are fused by the relationship fusion module to obtain an aggregate feature graph of the Nth scale; the feature graph output by the sampling splicing module is fused to obtain an aggregate feature graph of the first N-1 scales;

[0204] The sampling and splicing module is used to upsample the aggregate feature map of the nth scale, and the first feature sub-map and the second feature sub-map of the n-1th scale are spliced ​​with the upsampled aggregate feature map of the nth scale, where the value of n ranges from 2 to N.

[0205] The classification module is used to identify the aggregated feature map output by the relationship fusion module to obtain the damage category and damage location.

[0206] In one embodiment of the present invention, a rail damage detection system is also provided. Figure 8 As shown, it includes: a database 801 and a computing device 802.

[0207] The database 801 is used to store the support set including the support images of each rail damage category, the structure and parameters of the rail damage detection model. The process of determining the support set and the parameters of the rail damage detection model is referred to the above embodiment and will not be described in detail here.

[0208] The computing device 802 is used to receive the rail image to be inspected uploaded by the client, construct input data, where the input data includes the rail image to be inspected and supporting images of each rail damage category; input the input data into the rail damage detection model to obtain the damage category and damage location.

[0209] In some embodiments of this specification, the computing device 802 described herein includes, but is not limited to, a server and a computer device, and the client device may be a desktop computer, a tablet computer, a laptop computer, a smartphone, a digital assistant, a smart wearable device, etc. Among these, smart wearable devices may include smart bracelets, smart watches, smart glasses, smart helmets, etc. Of course, the client device is not limited to the aforementioned electronic devices with a certain physical form, and may also be software running on the aforementioned electronic devices.

[0210] In some embodiments of this specification, the computing device 802 is also used to train the parameters in the rail damage detection model. The specific training process refers to the aforementioned embodiment. In specific implementation, the computing device for training and the computing device for detection can be the same or different devices, and this document does not limit this. The rail damage detection model obtained through training in this document can achieve the following technical effects: First, it improves the detection accuracy of the rail damage detection model for known damage categories. By making full use of the labeled known rail damage category images, the similarity is calculated through the relationship aggregation module, so that the similarity information between the query image and the known image can be provided to supplement, thereby improving the discrimination and detection accuracy of the damage category. Second, it reduces the detection sensitivity of the rail damage detection model to negligible damage. The similarity calculated by the relationship aggregation module for negligible damage (small area, shallow damage, etc.) is generally low. After the similarity information is spliced ​​with the detection feature information, the confidence of the detection result will be reduced, thereby lowering the detection confidence of the detection model for these negligible damages, avoiding the detection of negligible minor damages. Since minor damages are no longer detected, damage detection at a certain location on a certain road section will not be reported, so there is no need for manual inspection and repair, thereby improving the efficiency of detection work.

[0211] Based on the same inventive concept, this article also provides a rail damage detection device, as described in the following embodiments. Since the principle of solving the problem of the rail damage detection device is similar to that of the rail damage detection method, the implementation of the device can refer to the rail damage detection method, and the repeated parts will not be repeated. Specifically, Figure 9 As shown, the rail damage detection device includes:

[0212] An acquisition module 901 is used to acquire an image of a rail to be inspected;

[0213] A pre-processing module 902 is used to construct input data, wherein the input data includes the image of the rail to be inspected and supporting images of each rail damage category, wherein the supporting images are images with damage categories and damage locations marked;

[0214] Detection module 903, used to input the input data into the rail damage detection model to obtain damage type and damage location;

[0215] The rail damage detection model is used to fuse the feature similarity between the rail damage image to be detected and the support image in the rail damage image to obtain an aggregated feature map, and identify the aggregated feature map to obtain the damage category and damage location.

[0216] In one embodiment of the present invention, a computer device is further provided for executing the steps of the method described in any of the above embodiments, such as Figure 10As shown, computer device 1002 may include one or more processors 1004, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. Computer device 1002 may also include any memory 1006 for storing any type of information, such as code, settings, data, and the like. For example, and without limitation, memory 1006 may include any one or more combinations of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, and the like. More generally, any memory may use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of computer device 1002. In one embodiment, when processor 1004 executes associated instructions stored in any memory or combination of memories, computer device 1002 may perform any operation of the associated instructions. Computer device 1002 also includes one or more drive mechanisms 1008, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like, for interacting with any memory.

[0217] Computer device 1002 may also include an input / output module 1010 (I / O) for receiving various inputs (via input device 1012) and providing various outputs (via output device 1014). A specific output mechanism may include a presentation device 1016 and an associated graphical user interface 1018 (GUI). In other embodiments, input / output module 1010 (I / O), input device 1012, and output device 1014 may not be included, and the computer device 1002 may simply function as a computer device on a network. Computer device 1002 may also include one or more network interfaces 1020 for exchanging data with other devices via one or more communication links 1022. One or more communication buses 1024 couple the components described above together.

[0218] The communication link 1022 may be implemented in any manner, for example, via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 1022 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0219] Corresponding to Figure 1 、 Figure 6 、 Figure 7 The embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which executes the steps of the above method when executed by a processor.

[0220] The embodiment of the present invention also provides a computer readable instruction, wherein when the processor executes the instruction, the program causes the processor to execute the following Figure 1 、 Figure 6 、 Figure 7 The method shown.

[0221] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0222] It should also be understood that in the embodiments herein, the term "and / or" merely describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" could represent: A alone, A and B simultaneously, or B alone. Furthermore, the character " / " in this document generally indicates an "or" relationship between the associated objects.

[0223] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0224] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0225] In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices, or units, or can be an electrical, mechanical, or other form of connection.

[0226] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments herein.

[0227] In addition, the functional units in the various embodiments herein may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0228] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0229] This article uses specific embodiments to illustrate the principles and implementation methods of this article. The description of the above embodiments is only used to help understand the methods and core ideas of this article. At the same time, for those skilled in the art, based on the ideas of this article, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation to this article.

Claims

1. A rail damage detection method, characterized in that: include: Acquire the image of the rail to be inspected; Constructing input data, wherein the input data includes an image of a rail to be inspected and supporting images of each rail damage category, wherein the supporting images are images with damage categories and damage locations marked; Inputting the input data into a rail damage detection model to obtain damage type and damage location; The rail damage detection model is used to determine an aggregated feature map, and identify the aggregated feature map to obtain damage categories and damage locations; The process of determining the aggregated feature map includes: Extracting a query feature map from the rail damage image to be detected and extracting a supporting feature map from each supporting image; Encoding the query feature map to obtain a query value feature map and a query key feature map; Encoding multiple support feature maps to obtain a support value feature map and a support key feature map; Perform linear transformation on the query key feature graph and the support key feature graph respectively, and perform product processing on the transformed feature graphs to obtain the key feature graph of relationship similarity; Normalize the key feature map of the relationship similarity, multiply the normalized feature map with the support value feature map to obtain the weighted support value feature map; The query value feature map and the weighted support value feature map are spliced ​​to obtain an aggregated feature map; The first dimension of the query value feature map is the product of the width and height of the query value feature map, and the second dimension of the query value feature map is the number of channels after halving. The first dimension of the support value feature map is the product of the number of support feature maps, width and height, and the second dimension of the support value feature map is the number of channels after halving; The first dimension of the query key feature graph is the product of the width and height of the query key feature graph, and the second dimension of the query key feature graph is 1 / 8 of the number of channels; The first dimension of the support key feature map is 1 / 8 of the number of channels, and the second dimension of the support key feature map is the product of the number of support feature maps, width and height.

2. The method according to claim 1, wherein The rail damage detection model includes: a feature extraction module, a relationship fusion module and a classification module; Utilizing the feature extraction module to extract a query feature map from the rail damage image to be detected and a supporting feature map from each supporting image; Using the relationship fusion module to fuse the query feature graph and the support feature graph to obtain an aggregated feature graph, wherein the aggregated feature graph has the same size as the query feature graph; The classification module is used to identify the aggregated feature map and obtain the damage category and damage location.

3. The method according to claim 2, wherein The rail damage detection model also includes: a multi-scale network module and a sampling splicing module; Down-sampling the query feature graph and the support feature graph respectively using the multi-scale network module to obtain a first feature subgraph and a second feature subgraph of M scales; The first feature subgraph and the second feature subgraph of the smallest scale are fused by the relationship fusion module to obtain an aggregate feature graph of the Mth scale; the feature graph output by the sampling splicing module is fused to obtain an aggregate feature graph of the first M-1 scales; The sampling and splicing module is used to upsample the aggregate feature map of the mth scale, and the first feature submap and the second feature submap of the m-1th scale are spliced ​​with the upsampled aggregate feature map of the mth scale, where the value range of m is 2 to M.

4. The method according to claim 3, wherein The rail damage detection model training process includes: Determine a plurality of batch samples, wherein the batch samples include: a query image and supporting images of each rail damage category; the query image is an original rail damage image; Input the batch samples into the rail damage detection model to predict the damage type and damage location of the query image in the batch samples; Calculate the loss function based on the predicted damage type and damage location, as well as the damage type and damage location annotated in the query image; The parameters in the rail damage detection model are trained according to the loss function.

5. The method according to claim 4, wherein Before determining multiple batches of samples, it also includes: The parameters of the rail damage detection model are pre-trained using the object detection dataset.

6. The method according to claim 4, wherein The loss function is calculated using the following formula: 1oss=l box +l cls +l obj ; Among them, loss is the loss function, l box is the damage location frame regression error, l cls is the classification error, l obj is the target detection error, l coord 、l class 、l noobj 、l obj is the weight, S 2 The number of times each image in the input data is divided into S×S grids for detection traversal, Is there a target of interest in the (i, j)th damage location candidate box, w i 、h i 、x i 、y i are the width, height, horizontal coordinate and vertical coordinate of the predicted damage location candidate box, are the width, height, horizontal coordinate and vertical coordinate of the damage location box, B is the number of candidate boxes for the damage location, and classes is the number of rail damage categories; p i (c) is the probability that the object in grid i belongs to category c, is the true probability that the object belongs to category c; Is there any target of interest in the (i, j)th damage location candidate box? i is the predicted damage category, The damage category is marked.

7. A rail damage detection device, characterized in that: include: An acquisition module is used to acquire the image of the rail to be detected; A preprocessing module is used to construct input data, wherein the input data includes the image of the rail to be inspected and supporting images of each rail damage category, wherein the supporting images are images with damage categories and damage locations marked; A detection module, configured to input the input data into a rail damage detection model to obtain damage categories and damage locations; The rail damage detection model is used to determine an aggregated feature map, and identify the aggregated feature map to obtain damage categories and damage locations; The process of determining the aggregated feature map includes: Extracting a query feature map from the rail damage image to be detected and extracting a supporting feature map from each supporting image; Encoding the query feature map to obtain a query value feature map and a query key feature map; Encoding multiple support feature maps to obtain a support value feature map and a support key feature map; Perform linear transformation on the query key feature graph and the support key feature graph respectively, and perform product processing on the transformed feature graphs to obtain the key feature graph of relationship similarity; Normalize the key feature map of the relationship similarity, multiply the normalized feature map with the support value feature map to obtain the weighted support value feature map; The query value feature map and the weighted support value feature map are spliced ​​to obtain an aggregated feature map; The first dimension of the query value feature map is the product of the width and height of the query value feature map, and the second dimension of the query value feature map is the number of channels after halving. The first dimension of the support value feature map is the product of the number of support feature maps, width and height, and the second dimension of the support value feature map is the number of channels after halving; The first dimension of the query key feature graph is the product of the width and height of the query key feature graph, and the second dimension of the query key feature graph is 1 / 8 of the number of channels; The first dimension of the support key feature map is 1 / 8 of the number of channels, and the second dimension of the support key feature map is the product of the number of support feature maps, width and height.

8. A computer device comprising a memory, a processor, and a computer program and a rail damage detection model stored in the memory, wherein: When the computer program is executed by the processor, the computer program executes the instructions of the method according to any one of claims 1 to 6.

9. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor of a computer device, the computer program executes the instructions of the method according to any one of claims 1 to 6.

Images