A rail anomaly detection method and system based on deep feature comparison
Through the railway track abnormality detection method based on deep feature comparison, the twin network and multi-scale fusion network extract feature differences are solved, and the problem of the detection time of edge equipment and insufficient detection ability of unknown foreign objects is achieved. The real-time and efficient railway track abnormality detection is achieved and versatility, suitable for complex railway scenarios.
Patent Information
- Application Number
- CN202411027695.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-30
AI Technical Summary
When transplanting to edge equipment, existing railway track abnormality detection methods have problems such as long detection time, insufficient detection capability for unknown foreign objects, poor anti-interference capability and insufficient universality, especially in complex railway scenarios, which are difficult to achieve real-time and efficient detection. .
Using the rail anomaly detection method based on deep feature comparison, the simulated rail anomaly samples are generated in the training stage and the normal samples are featured and fused. The feature differences are extracted using twin networks and multi-scale fusion networks, and logical judgments are made in real time during the test stage. Realize rail abnormality detection.
Real-time detection of railway tracks on edge equipment is realized, with high anti-interference ability and versatility, and can effectively locate unknown foreign objects, reduce detection time and manual labeling costs, and is suitable for diverse railway track scenarios.
Smart Images

Figure CN118552919B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a rail anomaly detection method and system based on depth feature comparison. Background Art
[0002] Public transportation or transport vehicles that need to travel on specific tracks in natural scenarios may pose certain safety risks when facing foreign object intrusion. However, limitations such as complex track scenarios and uncertain anomaly types bring many challenges to current anomaly intrusion detection methods.
[0003] The initial method mainly relied on railway inspectors to check for abnormal conditions on the rails 24 hours a day, but this method consumed a lot of manpower and relied too much on the subjective judgment and physical condition of the inspectors. It was inefficient and had many unstable factors. In recent years, thanks to the rapid development of drones and robots, anomaly detection on rails is mainly based on high-quality images collected by edge smart devices combined with visual algorithms for identification. These smart devices have gradually become the main choice for current railway safety inspections. However, different geographical environments also bring many complex railway scenes, and a large number of unknown categories of anomalies may also appear in a single scene. In addition, there are many conditions such as detection accuracy, real-time performance, and edge deployment, which may require more anomaly detection algorithms to adapt to different scenes. The present invention mainly proposes a general anomaly detection algorithm that can adapt to most railway scenes, and can be deployed on an embedded platform to efficiently and in real time detect unknown anomalies on railway tracks.
[0004] Patent document 1 (CN108550139A) proposes a method for detecting foreign objects on rails based on multi-level deep features. First, a frame of video is extracted from a preset position in the rail video taken by a drone as the image to be processed, and then N sub-images are obtained by randomly cropping different areas of the image (the shooting problem causes the foreign object target to be too small, and the size of the foreign object target in the sub-image is increased by dividing the sub-image), and the union of all sub-images must contain all areas of the image. Subsequently, the features of all sub-images are extracted through a fully convolutional network, and the features extracted from different depth levels of the network are used as multi-layer deep features. Finally, the multi-level deep features of the N sub-images of the image to be processed are used for classification (the possible foreign object targets need to be determined in advance) to achieve foreign object detection on the rails of the image.
[0005] This solution has the following disadvantages:
[0006] Disadvantage 1: By extracting video frames as images to be processed and dividing the processed images into blocks, this operation may increase the overall detection time to a certain extent.
[0007] Disadvantage 2: The detection of foreign objects depends on pre-defined foreign object targets, otherwise the classification network cannot distinguish and detect unknown foreign object targets. Therefore, this patent has certain disadvantages for intrusion detection of unknown foreign objects.
[0008] Patent document 2 (CN114926456A) proposes a rail foreign body detection method based on semi-automatic labeling and improved deep learning, which includes the following steps:
[0009] Step 1: Arrange thermal infrared cameras and visible light cameras in a scene within 25 degrees Celsius and collect videos. Then use the OSTU algorithm to quickly segment the human body area in the thermal infrared image and map it to the visible light image. Automatically complete the human body labeling, manually correct a small number of images that may have incorrect segmentation, realize semi-automatic acquisition of human body label information, and build a training set.
[0010] Step 2: Migrate the ImageNet pre-trained model weights to ResNet 50, replace the activation function ReLu with the LReLu function, and complete the scene classification.
[0011] Step 3: Based on the results of scene recognition, YOLOv5 is improved. The convolutional attention mechanism is added to the feature extraction network part, and the target box regression loss function CIOU_Loss is changed to GIOU_Loss in the prediction network part, and the h-swish function is introduced. Finally, the tracking module of the target tracking algorithm deepsort is added to perform classification training of intrusion foreign objects.
[0012] Step 4: Improve the railway track contour extraction algorithm based on traditional algorithm and deep learning, and take the common part of the contour information extracted by the two algorithms as the result.
[0013] Step 5: According to step 3 and step 4, the coordinate information of the foreign object and the contour information of the track are obtained, and a logic judgment function is added to divide the alarm level according to the relative position of the foreign object on the rail.
[0014] This solution has the following disadvantages:
[0015] Disadvantage 1: The rail foreign object detection method proposed in this invention includes several modules such as scene classification network, foreign object detection network, rail contour extraction algorithm, etc., which may consume a lot of detection time and affect the real-time performance of detection.
[0016] Disadvantage 2: The semi-automatic annotated dataset only contains label information for human body segmentation. The category of foreign body intrusion detection is relatively limited. It can only be used for specific scenarios and depends on the correctness of the overall dataset labels.
[0017] Disadvantage 3: The foreign object detection network uses the supervised algorithm YOLOv5, which requires the provision of annotated label information and a limited number of labels, and lacks universality for foreign objects of unknown categories.
[0018] Patent document 3 (CN116612457A) proposes a method for detecting foreign objects in track ballast based on deep unsupervised learning, which includes the following steps:
[0019] Step 1: Collect the track bed image, scale the image, randomly rotate it, and crop it with the center point of the rotated image as the origin. Then convert the cropped image data into a tensor and normalize it.
[0020] Step 2: Use a convolutional neural network to extract features from the image preprocessed in step 1, obtain high-dimensional feature information, and perform normalization operations.
[0021] Step 3: Construct a high-dimensional feature anomaly estimation module based on the standardized flow structure, which mainly includes an affine transformation layer, a reversible convolution layer, and an affine coupling layer. Input the normal track image into the module for training and learning the distribution of images around the normal track to distinguish the distribution of the roadbed image with foreign objects.
[0022] Step 4: Input the test image and use the trained model to generate high-dimensional feature tensors of different dimensions. Then upsample and concatenate them to generate the anomaly score map of the input image, and finally obtain the location information of the foreign object.
[0023] This solution has the following disadvantages:
[0024] Disadvantage 1: This invention mainly learns the feature distribution around the rails in normal track images in an unsupervised manner, and uses the feature differences around the tracks where foreign objects exist to locate foreign objects. It can detect any unknown foreign objects, but the distribution of trained normal image scenes is limited. The same trained model may have certain limitations for different track scenes and is not universal.
[0025] Disadvantage 2: The invention mainly relies on training image data, namely normal track scenes, and has general anti-interference ability. For example, for the same railway scene, if the lens for collecting video images has a large offset, the method may fail.
[0026] Patent document 4 (CN116597394A) proposes a railway foreign body intrusion detection system and method based on deep learning, including the following steps:
[0027] Step 1: Place cameras around the rails to collect video images, use the semantic segmentation model to mark the boundary area, and only detect intrusions in the boundary area.
[0028] Step 2: Use the target detection algorithm to detect foreign objects of known categories, and cooperate with the change detection algorithm to detect unknown categories (no need to retrain the model after switching scenes, and it is more versatile). The specific method of change detection is to uniformly sample 20 frames of images in the first 10 seconds and calculate the pixel average to obtain the recent historical frame, and uniformly sample 60 frames of images in the first hour and calculate the pixel average to obtain the early historical frame. The data processing module calculates the change characteristics of the recent historical frames and the early historical frames to detect foreign objects and track the targets of foreign objects.
[0029] Step 3: Suppose that the abnormal alarm information is processed through a certain filtering mechanism, such as by tracking the moving target ID in the first 10 frames. If the abnormality of the same ID is less than 5 frames, no alarm will be issued, which can reduce false alarms to a certain extent.
[0030] This solution has the following disadvantages: the invention detects foreign targets of known categories through a target detection algorithm and detects foreign targets of position categories through a change detection algorithm based on deep learning. This can achieve universality for different scenarios and generalization for unknown foreign objects, but the method is highly complex and may take a long time to detect when ported to embedded devices.
[0031] The shortcomings of the technical solutions provided by the above four patent documents can be summarized as follows:
[0032] 1. Some methods need to collect foreign objects and manually mark their locations and categories for supervised training. Some foreign objects may have a low frequency of occurrence and are difficult to collect, which can easily lead to a long-tail distribution of the training data set. In addition, they lack the ability to detect foreign objects of unknown categories, requiring additional incremental learning.
[0033] 2. Some methods have good detection capabilities for unknown foreign objects, but lack versatility and anti-interference. Foreign object detection in different scenes may fail, and some interference in the same scene, such as large lens shake, may also fail;
[0034] 3. Some methods have good versatility and the ability to detect unknown foreign objects, but the overall method is relatively complex, and it may be time-consuming to transplant it to edge devices, and it cannot meet real-time requirements. Summary of the invention
[0035] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method for detecting rail anomalies based on deep feature comparison; taking into account the various limitations of existing foreign object detection methods based on railway tracks when transplanted to edge applications, the present invention proposes a method that can be transplanted to edge devices and perform real-time detection of unknown anomalies on railway tracks. At the same time, the method has certain anti-interference capabilities and general detection capabilities for different scenarios.
[0036] Another object of the present invention is to provide a rail anomaly detection system based on depth feature comparison.
[0037] The purpose of the present invention is achieved through the following technical solutions:
[0038] A rail anomaly detection method based on deep feature comparison includes the following steps:
[0039] Training phase:
[0040] Obtain normal samples and perform overlapping block processing, and then generate rail abnormal samples;
[0041] The abnormal rail samples and normal rail samples form a training sample pair for preprocessing, and the encoding module and decoding module extract, fuse and decode the features of the preprocessed data to complete the training;
[0042] Testing phase:
[0043] First, fix the detection scene, obtain the next frame of the current rail scene without foreign objects as the normal rail sample, and the rail to be inspected sample is the real-time lens scene image;
[0044] Then, the normal rail sample and the rail sample to be inspected are treated as a test sample pair and preprocessed;
[0045] The trained encoding module and decoding module are used to output the corresponding prediction results. Finally, logical judgment is made on the decoded results to issue real-time alarms for abnormal rail conditions.
[0046] During the training phase, the method for generating the foreign body image in the rail abnormality sample is as follows:
[0047] (1) Specify the image of the foreign body generation area;
[0048] (2) Simulate unknown rail foreign objects of different sizes and colors that may appear in a single scene:
[0049] Use public segmentation task datasets to obtain foreground images of objects of different categories;
[0050] The resolution of the foreground image is scaled, rotated and the color space transformed at different ratios according to the preset probability to simulate foreign objects of different sizes and colors and obtain the foreign object image on the rail.
[0051] During the training phase, the method for generating the foreign body image in the rail abnormality sample is as follows:
[0052] A. Specify the image of the foreign body generation area ;
[0053] B. Use a few rail scene images to simulate rail foreign bodies in most scenarios:
[0054] a. Generate regional image of foreign matter The brightness, saturation and hue of the image are randomly adjusted, and then the adjusted image is divided into grids of equal proportions, and then the grids are rearranged to obtain a noise image simulating structural abnormalities. ;
[0055] b. Using the size of the region of interest or the size of the rail area, randomly generate a two-dimensional Perlin noise of the same size , and set the threshold to perform binarization processing to obtain the processing result ;
[0056] c. Obtain the noise foreground image with structural abnormality through the following formula , simulating foreign objects that may appear in different rail scenes, and noise foreground images with abnormal structures As an image of foreign matter on a railroad track:
[0057] ;
[0058] in, The transparency factor introduced is to better balance the image of the foreign body generation area and noisy image The fusion of the two images makes the fused image closer to the real anomaly.
[0059] During the training phase, the foreign body images in the rail abnormality samples After generation, generate the abnormal area image , the corresponding formula is:
[0060] ;
[0061] in, Generates an area image for the foreign object, To process the results Invert to get; use the size of the region of interest or the size of the rail area to randomly generate a two-dimensional Perlin noise of the same area size , and set the threshold to perform binarization processing to obtain the processing result ;
[0062] The abnormal area image Image of the foreign matter generation area replacing the normal sample , that is, spliced to the region of interest or rail area corresponding to the normal sample to obtain the final rail abnormal sample.
[0063] The encoding module uses two feature extraction networks with the same weight to form a twin network, processes the preprocessed abnormal samples and normal samples, and outputs three different levels of features:
[0064] Feature map of normal samples , and ;
[0065] And the characteristic map of abnormal samples , and ;
[0066] Among them, the first dimension N represents the number of samples input to the network in one iterative training, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width of the feature map;
[0067] Then, the feature maps of three different levels corresponding to normal samples and abnormal samples are spliced according to the channel dimension:
[0068] ;
[0069] Among them, i=1,2,3, corresponding to the three different levels in the middle of the feature extraction network, It represents the tensor channel concatenation operation, and the final result is recorded as , and ;
[0070] Finally, the differences between the three different levels of feature maps corresponding to normal samples and abnormal samples are calculated respectively. The operation is as follows:
[0071] ;
[0072] Among them, i=1,2,3, corresponding to the three different levels in the middle of the feature extraction network, It represents the mean square error of each position of the two input tensors, and the result is recorded as , and .
[0073] The encoding module uses a multi-scale fusion network to obtain the feature f obtained by channel splicing. 1_concat 、f 2_concat and f 3_concat Perform feature fusion:
[0074] First, a channel number is halved. Convolution and a constant number of channels Convolution on f 1_concat 、f 2_concat and f 3_concatExtract features and then use a Convolution increases the nonlinearity of the network, and we get , and ;
[0075] Remember f 3_msff = f 3_conv , the feature map f 3_msff Upsampled twice After the size is connected, the number of channels is halved Convolution operation obtains feature map , then f _temp With f 2_conv Add element by element to get ;
[0076] Similarly for f 2_msff Upsampling by two times and halving the number of channels After the convolution operation, 1_conv Add element by element to get ;
[0077] Finally, the multi-scale fusion features are output , and .
[0078] The decoding module is composed of multiple upsampling convolution operations combined The convolution operation is repeated, starting from the deep features, and finally using a The convolution operation reduces the channel dimension to 2 dimensions.
[0079] The training process of the decoding module is as follows:
[0080] The output f of the encoding module 1_encode 、f 2_encode and f 3_encode As the input of the decoding module, let f 3_decode =f 3_encode ,right Upsample twice to Size followed by one The convolution operation reduces the channel dimension to obtain the feature map Finally, f _temp1 With f 2_encode Concatenate in the channel dimension ;
[0081] Similarly, for f 2_decode Connect one after double upsampling Convolution operation obtains feature map , and then the feature map f _tepm2 With f1_encode Concatenate in the channel dimension ;
[0082] In order to further reduce the amount of calculation of the model, we continue to 1_decode After upsampling twice, connect a The convolution operation reduces the channel dimension to obtain the feature map ;
[0083] f 0_decoder Continue to upsample twice and connect a channel dimension unchanged The convolution operation obtains the same size as the input sample Feature map ;
[0084] Finally, use a The convolution operation is performed on f _up2mask Extract features and reduce the channel dimension to 2 dimensions to obtain , where the first dimension of the channel dimension is the foreground output and the second dimension is the background output.
[0085] Another object of the present invention is achieved by the following technical solutions:
[0086] A rail anomaly detection system based on deep feature comparison includes a sample input module, a data preprocessing module, a coding module, a decoding module and an anomaly alarm module; wherein the sample input module inputs rail anomaly samples and normal rail samples in the training phase, and inputs normal rail samples and rail samples to be inspected in the testing phase; the data preprocessing module, the coding module and the decoding module work simultaneously in the training phase; and the anomaly alarm module in the testing phase performs real-time rail anomaly alarms in the testing phase.
[0087] The working process of the abnormal alarm module is as follows:
[0088] The first dimension foreground channel of the feature map f_mask finally output by the decoding module is C1, and the second dimension background channel is C2. In order to enhance the output of the target foreground and weaken the influence of the background, the threshold is set to s to filter out some interfering pixels. The specific operations are as follows:
[0089] The pixels in C1 with values greater than s are set to 1, and the pixels in C2 with values less than s are set to 0;
[0090] By comparing the pixel values of C1 and C2 pixel by pixel, if C1 is greater than C2, it is set to 1, otherwise it is set to 0, and the comparison result is saved as the final mask map of the abnormal area .
[0091] At the same time, the present invention provides:
[0092] A server comprises a processor and a memory, wherein the memory stores at least one program, and the program is loaded and executed by the processor to implement the above-mentioned rail anomaly detection method based on deep feature comparison.
[0093] A computer-readable storage medium stores at least one program, which is loaded and executed by a processor to implement the above-mentioned rail anomaly detection method based on deep feature comparison.
[0094] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0095] 1. The present invention proposes a method for simulating and generating foreign objects in a railway track scenario, which can be applied to unknown foreign objects and different railway track scenarios. Compared with the existing supervised foreign object detection methods for railway tracks, such as the target detection algorithm YOLOv5, the present invention does not need to consider the problem of incomplete collection and small number of foreign objects, and also greatly avoids the cost of manual labeling under supervised learning.
[0096] 2. The present invention proposes a rail foreign body generation module that can simulate unknown foreign bodies in different scenarios. Compared with the unsupervised method, the present invention uses the generated foreign bodies to perform self-supervised training of the model, which has a certain anti-interference ability and is universal for most similar rail scenes. As for the supervised method, the method of the present invention can reduce the cost of manual labeling to a certain extent and solve the problem that foreign body data is difficult to collect.
[0097] 3. The present invention uses a fully convolutional neural network to extract features at different depth levels between normal samples and test samples for comparison, thereby locating the position of foreign objects. The method is simple to implement, has a small amount of calculation, and can detect unknown foreign objects on edge devices in real time. And combined with the rail foreign object generation module proposed in the present invention, it has a universal detection capability for most rail scenes. However, the existing methods are relatively complex in the implementation of unknown foreign object detection and scene versatility. Most of them require a supervised target detection algorithm to detect known foreign objects and combine methods similar to the present invention to detect unknown anomalies. It is time-consuming to detect and cannot be well transplanted to the edge platform for real-time detection.
[0098] 4. The present invention combines a multi-scale feature fusion network to effectively reduce the redundancy of feature calculation to a certain extent, reduce the amount of calculation, and proposes a feature attention mechanism network, which can more efficiently locate the optimal difference position between normal samples and test samples and accelerate the convergence of model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0099] Figure 1This is an overall flow chart of the rail anomaly detection method based on deep feature comparison described in the present invention.
[0100] Figure 2 This is a flow chart of the training phase of the rail anomaly detection method based on deep feature comparison described in the present invention.
[0101] Figure 3 This is a flow chart of the testing phase of the rail anomaly detection method based on deep feature comparison described in the present invention.
[0102] Figure 4 This is a workflow diagram of the encoding module described in the present invention.
[0103] Figure 5 The following is a flowchart of the abnormal alarm module of the present invention. DETAILED DESCRIPTION
[0104] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.
[0105] The present invention mainly utilizes the difference between the multi-level features of normal samples and abnormal samples in the same scene to locate foreign objects. The overall process of the rail anomaly detection method based on deep feature comparison is as follows: Figure 1 As shown, the corresponding rail anomaly detection system mainly includes a sample input module, a data preprocessing module, an encoding module, a decoding module and an anomaly alarm module. Due to the diversity and complexity of rail scenes, it is difficult to extract relatively common normal sample features to compare with abnormal sample features. Therefore, the present invention uses a single normal sample in the scene to be tested to compare with the abnormal sample, and updates the normal sample through a certain strategy to ensure the effectiveness of its detection.
[0106] The overall process is divided into two stages: training and testing. For details, refer to Figure 2 and Figure 3 In the training phase, abnormal samples are mainly simulated by two abnormal synthesis strategies in different scenarios. In the test phase, the abnormal samples are simply replaced with the samples to be tested. The encoding module and the decoding module are common parts. The parameters of the module network will be updated in the training phase, and the parameters will be fixed in the test phase for network reasoning. Finally, in the test phase, there will be an additional abnormal alarm module to update normal samples and make logical judgments on abnormal alarms.
[0107] In the training phase, a normal sample is first input, and then divided into four equal parts with overlapping areas. The blocks are passed through the rail foreign body generation module to generate abnormal samples in the corresponding scene. Then, a training sample is formed with the normal sample to perform data preprocessing operations such as scaling, color mode conversion, standardization and normalization. Finally, the encoder and decoder are used to extract, fuse and decode the features of the preprocessed data.
[0108] Unlike the training phase, the test phase does not require the participation of the foreign body generation module, and a foreign body alarm module is added. First, the detection scene is fixed, and the image of the next frame without foreign bodies in the scene is obtained as a normal sample, and the sample to be tested is a real-time lens scene image. Then, the normal sample and the sample to be tested are used as a test sample for the same data preprocessing operations such as scaling, color mode conversion, standardization and normalization. The trained decoder and encoder are used to output the corresponding prediction results. Finally, the abnormal alarm module performs logical judgment on the decoded results, and abnormal situation alarms are performed in real time. The normal samples in the scene are continuously updated, which can reduce misjudgment to a certain extent.
[0109] The specific operation steps of each module in the two stages are as follows:
[0110] Step 1: Obtaining input samples
[0111] The method proposed in the present invention locates foreign objects based on the difference between the features of normal samples and abnormal samples in the same scene, so the input of the model requires a normal sample and an abnormal sample. Since there are certain differences in the definition and acquisition of abnormal samples in the training phase and the test phase, the present invention hereinafter refers to the abnormal samples in the test phase as test samples.
[0112] During the training phase, normal samples are mainly obtained through fixed lenses in different scenes with a resolution of Considering that some foreign objects are relatively small in size, in order to better improve the recall rate of small foreign object detection, the present invention adopts a block strategy on normal samples, that is, the original normal sample is divided into four equal parts with overlapping areas, and the overlap rate is 20%. The four image blocks of equal size are the input normal samples. The acquisition of abnormal samples is mainly generated by simulating normal samples through the rail foreign object generation module proposed by the present invention.
[0113] In the test phase, due to the relative fixedness of the detection scene, the present invention selects a frame of image without foreign objects under the fixed lens of the scene to be detected as a normal sample. It should be noted that, unlike the training phase, the normal sample input in the test phase is of original size and is not processed in blocks. The model must provide a frame of normal sample image in the scene for the first run. The test sample is a real-time video image frame in the corresponding scene.
[0114] Step 2: Rail Foreign Object Generation
[0115] The present invention completes self-supervised learning by introducing abnormal samples in the training stage, and proposes a method for simulating the generation of unknown foreign objects on rails that appear randomly. The specific operation is as follows:
[0116] (1) Specifying the area where foreign objects are generated. Given a normal sample, i.e., a rail image without foreign objects, artificially specify the area of interest where foreign objects are generated on the normal sample or obtain the specified rail area through the rail contour extraction algorithm to obtain the area image where foreign objects are generated. ;
[0117] (2) Generation of foreign objects. Due to the diversity of rail scenes and the uncertainty of foreign objects, and the difficulty in collecting training data for some scenes with foreign objects on the rails, previous work usually generates simulated abnormal rail scene samples from normal rail scene samples. Common generation methods include image generation networks, copy and paste, and image fusion, but these methods have poor migration capabilities on complex rail scenes and general anti-interference capabilities. For example, in the same scene, the performance difference between the algorithms during the day and at night may be large. Based on this, the present invention proposes two foreign object simulation strategies to generate rail foreign objects.
[0118] Method 1: Simulate unknown rail foreign objects of different sizes and colors that may appear in a single scene
[0119] Considering the diversity and uncertainty of foreign objects on the rails, the present invention mainly uses the segmentation foreground image of common objects to simulate common foreign objects on the rails. The specific operation is as follows:
[0120] a. Use the public segmentation task dataset to obtain foreground images of objects of different categories;
[0121] b. Scaling, rotating, and color space transformation are performed on the foreground image at different resolutions with a certain probability to simulate foreign objects of different sizes and colors.
[0122] Method 2: Using a few rail scene images to simulate rail foreign bodies in most scenarios
[0123] Method 1 only generates abnormal samples in a single scene or a small number of scenes, which has certain limitations. This method can only identify different foreign objects in a specific single scene or a small number of scenes. In order to achieve the same detection effect on more untrained and difficult to collect data rail scenes, the present invention proposes method 2 combined with method 1 to adapt to more untrained complex rail scenes. The specific operation of the method is as follows:
[0124] a. Image of the foreign body generation area obtained in step (1) The brightness, saturation and hue of the image are randomly adjusted, and then the adjusted image is divided into grids of equal proportions, and then the grids are rearranged to obtain a noise image simulating structural abnormalities. ;
[0125] b. Using the size of the region of interest or the size of the rail area obtained in step (1), randomly generate a two-dimensional Perlin noise of the same size , and set the threshold to perform binarization processing to obtain ;
[0126] c. The noise foreground image with structural abnormality is obtained by the following formula , simulating possible foreign objects in different rail scenarios, including The transparency factor introduced is to better balance the image of the foreign body generation area and noisy image The fusion of Closer to the real anomaly, the noise foreground image of the structural anomaly As an image of a foreign object on a railroad track.
[0127] ;
[0128] (3) The foreign body image generated by method 1 or method 2 in step (2) is , obtained by inverting M After the foreign body generated area image Multiply element by element and then add it to the foreign body image , generate the final abnormal area image , refer to the following formula for specific operations:
[0129] ;
[0130] (4) Finally, the abnormal area image Replace the foreign matter generation area image of the normal sample in step (1) , and spliced to the region of interest or rail area corresponding to the normal sample to obtain the final abnormal sample.
[0131] (5) Select method 1 or method 2 with a certain probability to simulate the generation of foreign matter on the rails.
[0132] Step 3: Data Preprocessing
[0133] (1) Read the two input samples required by the model. In order to reduce the computational complexity of model training and testing and improve the detection speed, the input data of the present invention is scaled proportionally in resolution and filled to a resolution of Size. Specifically, let the original sample width be , Gao Wei , the target width is scaled to , Gao Wei .calculate and The minimum value of the two is taken as the scaling ratio . use Calculate the dimensions and size to be filled. If , then the dimension to be filled is high, and the filling size is Otherwise, the dimension to be filled is width, and the filling size is ;
[0134] (2) Convert the color mode of the scaled input sample to RGB format, then normalize the image data and convert it into a tensor format supported by the model training code framework. The present invention uses the Pytorch framework tensor;
[0135] (3) The tensor data in step (2) is standardized, and the mean of each channel is (0.485, 0.456, 0.406) and the variance is (0.229, 0.224, 0.225).
[0136] Step 4: Encoding module training
[0137] The encoding module consists of three networks: feature extraction network, feature multi-scale fusion combined with spatial attention mechanism. In order to better extract the features of normal samples and abnormal samples for comparison, the present invention uses two feature extraction networks with the same weights to form a twin network. The feature extraction network uses a convolutional neural network trained on the public dataset ImageNet, and the present invention uses ResNet18. The overall encoding module data flow is as follows Figure 4 As shown, the specific training process is:
[0138] (1) Freeze the parameters of the pre-trained feature extraction network and input the pre-processed normal samples and abnormal samples. In order to retain more detailed information and ensure that the extracted information is not too abstract and biased towards ImageNet data, the present invention selects three different levels of features in the middle of ResNet18 as output, which are: , and , where the first dimension N represents the number of samples input to the network in one iterative training, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width of the feature map. Then perform the following operations:
[0139] a. Input normal samples and abnormal samples into ResNet18 respectively, and obtain three different levels of feature maps of the corresponding samples. , and is the feature map of normal samples, , and It is the feature map of abnormal samples.
[0140] b. According to the following formula, the feature maps of three different levels corresponding to normal samples and abnormal samples are spliced according to the channel dimension:
[0141] ;
[0142] Among them, i=1, 2, 3, corresponding to the three different levels in the middle of ResNet18, It represents the tensor channel concatenation operation, and the final result is recorded as , and .
[0143] c. Calculate the differences between the three different levels of feature maps corresponding to normal samples and abnormal samples respectively. The operation is as follows:
[0144] ;
[0145] Among them, i=1, 2, 3, corresponding to the three different levels in the middle of ResNet18, In the present invention, it represents the mean square error of each position of the two input tensors, and the result is recorded as , and .
[0146] (2) In order to avoid feature redundancy and low efficiency caused by subsequent multi-layer feature calculation, the present invention concatenates the features f obtained by channel in step (1) 1_concat 、f 2_concat and f 3_concat Send it to the multi-scale fusion network for feature fusion. Specific steps:
[0147] a. First, use a method that reduces the number of channels by half. Convolution and a constant number of channels Convolution on f 1_concat 、f 2_concat and f 3_concat Extract features and then use a Convolution increases the nonlinearity of the network, and we get , and .
[0148] b. Remember f 3_msff = f 3_conv , the feature map f 3_msff Upsampled twice After the size is connected, the number of channels is halved Convolution operation obtains feature map , then f _temp With f2_conv Add element by element to get .
[0149] c. Similarly for f 2_msff Double upsampling and halving the number of channels After the convolution operation, 1_conv Add element by element to get .
[0150] d. Finally, output the multi-scale fused features , and .
[0151] (3) Subsequently, in order to further improve the algorithm's attention to the difference information of abnormal regions, the present invention proposes a spatial attention mechanism network.
[0152] a. First, the difference between the three-level features of the normal sample and the abnormal sample obtained in step (1) is 1_diff 、f 2_diff and f 3_diff Channel average pooling is performed separately to output feature maps of three sizes, which are recorded as , and .
[0153] b. Then remember f 3_SA =f 3_AvgPool , upsample m3 twice to Size and f 2_AvgPool Multiply element by element to get .
[0154] c. Similarly, f 2_SA Upsampled twice Size and f 1_AvgPool Multiply element by element to get .
[0155] d. 1_SA 、f 2_SA and f 3_SA As the output of the spatial attention mechanism network.
[0156] (4) Finally, the multi-scale fusion feature f 1_msff 、f 2_msff and f 3_msff After spatial attention calculation in step (3), we get f 1_SA 、f 2_SA and f 3_SA Perform element-by-element multiplication respectively. Refer to the following formula for the operation:
[0157] ;
[0158] where i=1, 2, 3, Represents the multiplication of the corresponding position elements of the two input tensors, and the result is recorded as , and And output.
[0159] Step 5: Decoding module training
[0160] The decoding module is mainly composed of multiple upsampling convolution operations. The convolution operation is repeated, starting from the deep features (the smaller the feature map size, the deeper the level), and finally using a The convolution operation reduces the channel dimension to 2 dimensions. The specific operation is as follows:
[0161] (1) The output f of the encoding module 1_encode 、f 2_encode and f 3_encode As the input of the decoding module, let f 3_decode =f 3_encode ,right Upsample twice to Size followed by one The convolution operation reduces the channel dimension to obtain the feature map Finally, f _temp1 With f 2_encode Concatenate in the channel dimension ;
[0162] (2) Similarly, for f 2_decode Connect one after double upsampling Convolution operation obtains feature map , and then the feature map f _tepm2 With f 1_encode Concatenate in the channel dimension ;
[0163] (3) In order to further reduce the amount of calculation of the model, we continue to 1_decode After upsampling twice, connect a The convolution operation reduces the channel dimension to obtain the feature map .
[0164] (4) f 0_decoder Continue to upsample twice and connect a channel dimension unchanged The convolution operation obtains the same size as the input sample Feature map .
[0165] (5) Finally, use a The convolution operation is performed on f _up2mask Extract features and reduce the channel dimension to 2 dimensions to obtain , where the first dimension of the channel dimension is the foreground output and the second dimension is the background output.
[0166] Step 6: Abnormal alarm
[0167] This module is only executed in the testing phase, and mainly performs certain logical judgments and operations on the output of the trained decoding module. Figure 5 , the specific operations are as follows:
[0168] (1) The first dimension foreground channel of the feature map f_mask finally output by the decoding module is recorded as C1, and the second dimension background channel is recorded as C2. In order to enhance the output of the target foreground and weaken the influence of the background, the present invention sets the threshold value s to filter out some interfering pixels. The specific operation is as follows:
[0169] a. The pixels in C1 with values greater than s are set to 1, and the pixels in C2 with values less than s are set to 0;
[0170] b. Compare the pixel values of C1 and C2 pixel by pixel. If C1 is greater than C2, it is set to 1, otherwise it is set to 0. The comparison result is saved as the final mask image of the abnormal area. .
[0171] (2) In the actual scene test, if there are abnormalities in more than 20 consecutive input video frames, an abnormality alarm is triggered. If there are no abnormalities in more than 50 consecutive frames, the last frame without abnormalities is used to update and replace the input normal sample.
[0172] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.
Claims
1. A rail anomaly detection method based on deep feature comparison, characterized in that: The following steps are involved: Training phase: Obtain normal samples and perform overlapping block processing, and then generate rail abnormal samples; The abnormal rail samples and normal rail samples form a training sample pair for preprocessing, and the encoding module and decoding module extract, fuse and decode the features of the preprocessed data to complete the training; Testing phase: First, fix the detection scene, obtain the next frame of the current rail scene without foreign objects as the normal rail sample, and the rail to be inspected sample is the real-time lens scene image; Then, the normal rail sample and the rail sample to be inspected are treated as a test sample pair and preprocessed; Use the trained encoding module and decoding module to output the corresponding prediction results, and finally make logical judgments on the decoded results to issue real-time rail abnormality alarms; The encoding module uses two feature extraction networks with the same weight to form a twin network, processes the preprocessed abnormal samples and normal samples, and outputs three different levels of features: Feature map of normal samples , and ; And the characteristic map of abnormal samples , and ; Among them, the first dimension N represents the number of samples input to the network in one iterative training, the second dimension represents the number of channels, and the third and fourth dimensions represent the height and width of the feature map; Then, the feature maps of three different levels corresponding to normal samples and abnormal samples are spliced according to the channel dimension: ; Among them, i=1,2,3, corresponding to the three different levels in the middle of the feature extraction network, It represents the tensor channel concatenation operation, and the final result is recorded as , and ; Finally, the differences between the three different levels of feature maps corresponding to normal samples and abnormal samples are calculated respectively. The operation is as follows: ; Among them, i=1,2,3, corresponding to the three different levels in the middle of the feature extraction network, It represents the mean square error of each position of the two input tensors, and the result is recorded as , and ; The encoding module uses a multi-scale fusion network to obtain the feature f obtained by channel splicing. 1_concat 、f 2_concat and f 3_concat Perform feature fusion: First, a channel number is halved. Convolution and a constant number of channels Convolution on f 1_concat 、f 2_concat and f 3_concat Extract features and then use a Convolution increases the nonlinearity of the network, and we get , and ; Remember f 3_msff = f 3_conv , the feature map f 3_msff Upsampled twice After the size is connected, the number of channels is halved Convolution operation obtains feature map , then f _temp With f 2_conv Add element by element to get ; Similarly for f 2_msff Upsampling by two times and halving the number of channels After the convolution operation, 1_conv Add element by element to get ; Finally, the multi-scale fusion features are output , and ; The spatial attention network then proceeds as follows: a. First, the difference between the three levels of features of normal samples and abnormal samples is f 1_diff 、f 2_diff and f 3_diff Channel average pooling is performed separately to output feature maps of three sizes, which are recorded as , and ; b. Then remember f 3_SA =f 3_AvgPool , upsample m3 twice to Size and f 2_AvgPool Multiply element by element to get ; c. Similarly, f 2_SA Upsampled twice Size and f 1_AvgPool Multiply element by element to get ; d. 1_SA 、f 2_SA and f 3_SA As the output of the spatial attention mechanism network; Finally, f 1_msff 、f 2_msf、 f 3_msff With f 1_SA 、f 2_SA and f 3_SA Perform element-by-element multiplication respectively. Refer to the following formula for the operation: ; where i=1, 2, 3, Represents the multiplication of the corresponding position elements of the two input tensors, and the result is recorded as , and And output; The decoding module is composed of multiple upsampling convolution operations combined The convolution operation is repeated, starting from the deep features, and finally using a The convolution operation reduces the channel dimension to 2 dimensions; The training process of the decoding module is as follows: The output f of the encoding module 1_encode 、f 2_encode and f 3_encode As the input of the decoding module, let f 3_decode =f 3_encode ,right Upsample twice to Size followed by one The convolution operation reduces the channel dimension to obtain the feature map Finally, f _temp1 With f 2_encode Concatenate in the channel dimension ; Similarly, for f 2_decode Connect one after double upsampling Convolution operation obtains feature map , and then the feature map f _tepm2 With f 1_encode Concatenate in the channel dimension ; In order to further reduce the amount of calculation of the model, we continue to 1_decode After upsampling twice, connect a The convolution operation reduces the channel dimension to obtain the feature map ; f 0_decoder Continue to upsample twice and connect a channel dimension unchanged The convolution operation obtains the same size as the input sample Feature map ; Finally, use a The convolution operation is performed on f _up2mask Extract features and reduce the channel dimension to 2 dimensions to obtain , where the first dimension of the channel dimension is the foreground output and the second dimension is the background output.
2. The rail anomaly detection method based on deep feature comparison according to claim 1 is characterized in that: During the training phase, the foreign body images in the rail abnormality samples After generation, generate the abnormal area image , the corresponding formula is: ; in, Generates an area image for the foreign object, To process the results Invert to get; use the size of the region of interest or the size of the rail area to randomly generate a two-dimensional Perlin noise of the same area size , and set the threshold to perform binarization processing to obtain the processing result ; The abnormal area image Image of the foreign matter generation area replacing the normal sample , that is, spliced to the region of interest or rail area corresponding to the normal sample to obtain the final rail abnormal sample.
3. The rail anomaly detection method based on deep feature comparison according to claim 1 is characterized in that: During the training phase, the method for generating the foreign body image in the rail abnormality sample is as follows: (1) Specify the image of the foreign body generation area; (2) Simulate unknown rail foreign objects of different sizes and colors that may appear in a single scene: Use public segmentation task datasets to obtain foreground images of objects of different categories; The resolution of the foreground image is scaled, rotated and the color space transformed at different ratios according to the preset probability to simulate foreign objects of different sizes and colors and obtain the foreign object image on the rail.
4. The rail anomaly detection method based on deep feature comparison according to claim 1 is characterized in that: During the training phase, the method for generating the foreign body image in the rail abnormality sample is as follows: A. Specify the image of the foreign body generation area ; B. Use a few rail scene images to simulate rail foreign bodies in most scenarios: a. Generate regional image of foreign matter The brightness, saturation and hue of the image are randomly adjusted, and then the adjusted image is divided into grids of equal proportions, and then the grids are rearranged to obtain a noise image simulating structural abnormalities. ; b. Using the size of the region of interest or the size of the rail area, randomly generate a two-dimensional Perlin noise of the same size , and set the threshold to perform binarization processing to obtain the processing result ; c. Obtain the noise foreground image with structural abnormality through the following formula , simulating foreign objects that may appear in different rail scenes, and noise foreground images with abnormal structures As an image of foreign matter on a railroad track: ; in, The transparency factor introduced is to better balance the image of the foreign body generation area and noisy image The fusion of the two images makes the fused image closer to the real anomaly.
5. A rail anomaly detection system based on deep feature comparison, characterized in that: A method for detecting anomalies of rails as claimed in any one of claims 1 to 4; comprising a sample input module, a data preprocessing module, a coding module, a decoding module and an anomaly alarm module; wherein the sample input module inputs anomaly samples of rails and normal samples of rails in the training phase, and inputs normal samples of rails and samples to be inspected of rails in the testing phase; the data preprocessing module, the coding module and the decoding module work simultaneously in the training phase; and the anomaly alarm module in the testing phase performs an anomaly alarm of rails in real time in the testing phase.
6. The rail anomaly detection system based on deep feature comparison according to claim 5 is characterized in that: The working process of the abnormal alarm module is as follows: The first dimension foreground channel of the feature map f_mask finally output by the decoding module is C1, and the second dimension background channel is C2. In order to enhance the output of the target foreground and weaken the influence of the background, the threshold is set to s to filter out some interfering pixels. The specific operations are as follows: The pixels in C1 with values greater than s are set to 1, and the pixels in C2 with values less than s are set to 0; By comparing the pixel values of C1 and C2 pixel by pixel, if C1 is greater than C2, it is set to 1, otherwise it is set to 0, and the comparison result is saved as the final mask map of the abnormal area .
7. A server, comprising a processor and a memory, wherein at least one program is stored in the memory, wherein: The program is loaded and executed by the processor to implement the rail anomaly detection method according to any one of claims 1 to 4.
8. A computer-readable storage medium, wherein at least one program is stored in the storage medium, characterized in that: The program is loaded and executed by a processor to implement the rail anomaly detection method according to any one of claims 1 to 4.
