An intelligent detection method for abnormal rail fasteners based on multi-source information fusion

Abstract

An intelligent detection method for abnormal rail fasteners by multi-source information fusion, including equipment preparation; data collection; correction of left and right oblique angle image data; target object detection frame selection; target frame selection alignment; image processing; foreign body detection; surface position data interception and mapping; data integration; feature extraction using multi-source feature fusion intelligent detection method; lightweight deployment; real-time warning mechanism. The present invention utilizes multi-view data collected by multi-view acquisition equipment for rail fasteners, and applies an intelligent algorithm of multi-feature fusion to detect abnormal fasteners. Multi-source information fusion improves the accuracy of abnormal fastener detection, and establishes a real-time warning mechanism for abnormal situations, so that maintenance workers can be notified in time for emergency treatment to ensure the safety of railway driving.

Description

Technical Field

[0001] The present invention relates to track structure anomaly detection, image processing and multi-source information fusion technology, which is applied to the intelligent detection of abnormal track fasteners, and can efficiently and timely detect abnormal railway fasteners to ensure the safety of railway driving. Background Art

[0002] The existing track fastener detection technology usually uses the linear array camera on the detection equipment to take pictures of the fasteners from directly above the rails, and after image stitching and image preprocessing, the target detection algorithm is used to detect abnormal fasteners. However, in the image data, fasteners are often blocked by foreign objects such as ballast and dust, and it is difficult to assess whether the fasteners are deformed through photos from a single angle. As a result, the existing track fastener detection is prone to miss or misjudge when identifying the above abnormal fasteners, which greatly affects the safety of railway driving.

[0003] As in the prior art, CN116433840A discloses a method for synthesizing specific damaged images of ballastless track slab fasteners, including obtaining real ballastless track slab fastener image data; constructing a 3D BIM model of the ballastless track slab; performing simulation rendering processing on the model to construct a virtual space of the railway scene; collecting and outputting virtual fastener damaged image data through the virtual space; constructing a cyclic adversarial generative network to perform real stylized migration processing on real image data and virtual damaged image data to achieve image data synthesis and construct a synthetic dataset of specific damaged images of ballastless track slab fasteners. The present invention also discloses a detection method including the method for synthesizing specific damaged images of ballastless track slab fasteners. CN110796643A discloses a method and system for detecting defects in rail fasteners. The method is applied to a rail fastener defect detection system, the system comprising a fastener area positioning module and a defect classification module; the method comprises: inputting the first rail image into the fastener area positioning module; locating a target frame where the fastener is located in the first rail image according to the output of the fastener area positioning module; segmenting a fastener image from the first rail image according to the target frame; preprocessing the fastener image to obtain a first fastener image; inputting the first fastener image into the defect classification module, outputting a fastener defect detection result; and selecting the target frame corresponding to the defective fastener in the first rail image. CN116757990A discloses a method for online detection and identification of rail fastener defects based on machine vision, belonging to the field of railway technology. The method includes the following steps: system initialization, setting the camera trigger mechanism, which collects a two-dimensional image of the high-speed rail fastener area, sets the trigger condition for the next camera photo according to the prior value, and then performs edge detection on the next frame of the image collected, performs template matching with the standard image after translation, finds the best pixel, and corrects the preset photo trigger condition. The mechanism can ensure that the fastener to be inspected is located in the optimal imaging area; image preprocessing, preprocessing the captured two-dimensional image, first performing median filtering, and then performing bilateral filtering on the median filtered image to improve the imaging effect; fastener defect detection, binarization image segmentation of the preprocessed image, and then template matching to determine whether the fastener to be inspected has defects; fastener defect recognition. A lightweight neural network is used to identify the fastener image that is judged to be defective.CN117496349A discloses a method and system for detecting abnormalities of rail fasteners, which belongs to the technical field of computer-based abnormality detection of rail fasteners. A high-precision three-dimensional line scanning laser acquisition device is used to scan rail fasteners to acquire and register paired fastener depth maps and grayscale images; a pixel-level dimensional complementary image fusion method is used to efficiently fuse the depth map and the corresponding grayscale image to construct a rail fastener fusion data set; a feature fusion-decoupling module is constructed to realize multi-dimensional feature fusion and task decoupling of a backbone network; a loss function re-weighting method guided by a detection accuracy index is used to continuously adjust the category weight matrix during the training process, thereby realizing abnormality detection of rail fasteners. CN107576666A discloses a dual-spectrum imaging rail and fastener anomaly detection method, which belongs to the technical field of railway traffic safety detection and is used to solve the problem that the prior art is difficult to realize rail and fastener anomaly detection. The present invention adopts a visible light camera and an infrared camera to form a dual-spectrum imaging device, performs dual-spectrum imaging on the rail and fastener area, obtains rail and fastener texture images and infrared thermal images respectively, uses texture images to locate rails and fasteners in infrared thermal images, uses background model comparison to detect fastener missing, looseness, and fracture anomalies, and uses texture images and infrared thermal images together to detect rail peeling, abrasions, corrugation, and cracks.

[0004] Although the above-mentioned prior art discloses a method for detecting abnormalities in rail fasteners, the above-mentioned technical solution does not use pictures collected by multi-view imaging technology to comprehensively detect abnormal conditions of fasteners, nor does it detect foreign objects on rail fasteners. The image processing of the monitoring target is not perfect, which leads to erroneous judgments. Summary of the invention

[0005] In view of the shortcomings of existing detection methods, an intelligent detection method for abnormal fasteners based on multi-source information fusion is proposed. The images collected by multi-view image acquisition technology are used to comprehensively detect the abnormal status of fasteners, and a real-time detection and alarm system is built to remind maintenance workers to repair abnormal fasteners in time.

[0006] An intelligent detection method for abnormal rail fasteners by multi-source information fusion, characterized by:

[0007] Step S1, equipment preparation: including binocular camera, electronic detection equipment and line laser scanning equipment;

[0008] Step S2, data collection: multi-view image data, electromagnetic data, line laser position distance data;

[0009] Step S3, correcting the left and right oblique view image data: Since the images are taken at 45° relative to the plane of the two rails, there will be a certain degree of distortion. The distortion of the left and right oblique view images is eliminated through Hough correction transformation to convert them into left view images and right view images;

[0010] Step S4, target object detection frame selection: perform rail fastener detection on the front view image, the left view image and the right view image respectively to obtain the front view rail fastener target frame, the left view rail fastener target frame and the right view rail fastener target frame;

[0011] Step S5, target object frame selection and alignment: Since the camera position and angle are fixed, there are overlapping parts when the cameras at different angles shoot the same object. The pixels of the overlapping parts in the two perspective images are aligned. Combined with the rail fastener target frames of different perspectives, the target frames are aligned twice; for the orthographic image, the left-perspective rail fastener target frame and the right-perspective rail fastener target frame are mapped to the orthographic image through pixel alignment to obtain the orthographic left-mapped rail fastener target frame and the orthographic right-mapped rail fastener target frame, and then add the existing orthographic rail fastener target frame to take the average of the three frames as the orthographic rail fastener alignment target frame; similarly, the left-perspective rail fastener alignment target frame and the right-perspective rail fastener alignment target frame are obtained;

[0012] Step S6, image processing: crop the images of the three viewing angles, namely, the front and left viewing angles, according to the rail fastener alignment target frame of each viewing angle, and remove irrelevant background; in order to facilitate feature extraction, the cropped images are uniformly scaled to a size of 416*416 pixels to obtain the front viewing angle target image image1_norm, the left viewing angle target image image1_left, and the right viewing angle target image image1_right;

[0013] Step S7, foreign body detection: foreign body detection is performed in the normal view target image, the left view target image and the right view target image respectively, the foreign body is selected by a frame, and the pixels outside the foreign body frame are set to 0, so as to obtain the normal view foreign body image image2_norm, the left view foreign body image image2_left and the right view foreign body image image2_right, and the image size is also 416*416;

[0014] Step S8, electromagnetic data interception and mapping: The electromagnetic signal data range is consistent with the range illuminated by the front-view image. According to the position ratio of the front-view rail fastener target frame relative to the front-view image, the electromagnetic data of the corresponding length is intercepted to obtain the electromagnetic signal of the rail fastener, and the electromagnetic signal is mapped into data of the same size as the image through signal processing and deep learning network. The specific steps are as follows:

[0015] Step S9, surface position data interception and mapping: high-precision surface position data of the rail fastener is obtained by a line laser scanning device, the range of which is consistent with the range of the front-view image. According to the position ratio of the front-view rail fastener target frame relative to the front-view image, the corresponding rail fastener surface position data is intercepted and mapped into data of the same size as the image through a two-layer feedforward network;

[0016] Step S10, data integration: All data from steps S6-S9 are integrated together according to the third dimension as input X of the multi-source feature fusion intelligent detection method.

[0017] X=[image1_norm,image1_left,image1_right,image2_norm,image2_left,image2_right,En,Rn];

[0018] Where X is the input of the multi-source feature fusion intelligent detection method, the dimension is [416,416,C], all images are 3-channel images, and C is the sum of the third dimension of all features;

[0019] Step S11, extracting features by using a multi-source feature fusion intelligent detection method;

[0020] Step S12, lightweight deployment: In order to improve the efficiency of real-time monitoring, the model is quantized to int8, and the storage space and computational complexity of the model are significantly reduced while ensuring a certain initial inspection rate, so as to achieve real-time monitoring and anomaly detection of fastener status;

[0021] Step S13, real-time warning mechanism: according to the model detection results, establish a warning mechanism for abnormal situations, set the corresponding threshold p, and when the model detects that the rail is abnormal, promptly notify relevant personnel for emergency processing.

[0022] The invention also discloses an intelligent detection system for multi-source information fusion of abnormal rail fasteners.

[0023] The present invention also discloses a non-volatile storage medium, characterized in that the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above method.

[0024] The present invention also discloses an electronic device, characterized in that it comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the above method when they are executed. Beneficial Effects

[0025] By utilizing the multi-perspective data collected by the multi-perspective acquisition equipment of rail fasteners and applying an intelligent algorithm with multi-feature fusion, abnormal fasteners are detected. Multi-source information fusion improves the accuracy of abnormal fastener detection and establishes a real-time early warning mechanism for abnormal situations, so that maintenance workers can be notified in time for emergency processing and ensure the safety of railway operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1The present invention is a flow chart of the intelligent detection method of abnormal rail fasteners by multi-source information fusion. DETAILED DESCRIPTION Example 1

[0027] An intelligent detection method for abnormal rail fasteners by multi-source information fusion includes the following steps:

[0028] Step S1, equipment preparation: mainly including binocular camera, electronic detection equipment and line laser scanning equipment

[0029] Step S11, binocular camera multi-viewing angle acquisition: The railway automatic inspection vehicle sets up a sampling camera directly above each rail, and sets up a sampling camera at about 45° relative to the plane of the two rails, so as to achieve multi-viewing angle coverage of the rail fasteners.

[0030] Step S12, electromagnetic detection equipment: install the electromagnetic detection equipment above the railway automatic inspection trolley to ensure that it can effectively interact with the fasteners when passing through the rails. The electromagnetic detection equipment is used to detect internal damage or looseness of the fasteners.

[0031] Step S13, line laser scanning equipment: Install the line laser scanning equipment above the railway automatic inspection vehicle to ensure that the entire surface of the fastener can be covered, including parts that are difficult to detect visually. Use the line laser scanning equipment to obtain high-precision surface position information of the rail fastener.

[0032] Step S2, data collection: multi-view image data, electromagnetic data, line laser position distance data

[0033] Step S21, multi-view binocular camera: capturing image data of the rail fastener from three different angles, including a front view image, a left oblique view image, and a right oblique view image.

[0034] Step S22, electromagnetic detection equipment: detect the electromagnetic characteristics of the rail fasteners, identify metal fatigue or cracks, and obtain electromagnetic signal data, the range of which is consistent with the range of the positive viewing angle image.

[0035] Step S23, line laser scanning equipment: obtain high-precision surface position data of the rail fastener, the range of which is consistent with the range of the orthographic image illumination.

[0036] Step S24: The collected data is transmitted to the central processing unit via the network, and subsequent calculations are performed in the central processing unit.

[0037] Step S3, correcting the left and right oblique view image data: Since the images are taken at 45° relative to the plane of the two rails, there will be a certain degree of distortion. The distortion of the left and right oblique view images is eliminated through the Hough correction transform to convert them into left view images and right view images.

[0038] Step S4, target object detection frame selection: perform rail fastener detection on the front view image, the left view image and the right view image respectively to obtain the front view rail fastener target frame, the left view rail fastener target frame and the right view rail fastener target frame.

[0039] Step S5, target object frame selection and alignment: Since the camera position and angle are fixed, there are overlapping parts when the cameras at different angles shoot the same object. The pixel points of the overlapping parts in the two perspective images can be aligned. Combined with the rail fastener target frames of different perspectives, the target frames are aligned twice. For the positive perspective image, the left perspective rail fastener target frame and the right perspective rail fastener target frame are mapped to the positive perspective image through pixel alignment to obtain the positive perspective left mapped rail fastener target frame and the positive perspective right mapped rail fastener target frame, and then add the existing positive perspective rail fastener target frame, and take the average value of the three frames as the positive perspective rail fastener alignment target frame; similarly, the left perspective rail fastener alignment target frame and the right perspective rail fastener alignment target frame are obtained.

[0040] Step S6, image processing: crop the images of the three perspectives, image1_norm, image1_left and image1_right, according to the target frame aligned with the rail fasteners of each perspective, and remove irrelevant background; in order to facilitate feature extraction, the cropped images are uniformly scaled to 416*416 pixels to obtain the normal perspective target image image1_norm, the left perspective target image image1_left and the right perspective target image image1_right.

[0041] Step S7, foreign object detection: perform foreign object detection in the front-view target image, the left-view target image and the right-view target image respectively, select the foreign objects by box, and set the pixels outside the foreign object box to 0, to obtain the front-view foreign object image image2_norm, the left-view foreign object image image2_left and the right-view foreign object image image2_right, and the image size is also 416*416.

[0042] Step S8, electromagnetic data interception and mapping: The electromagnetic signal data range is consistent with the range illuminated by the front-view image. According to the position ratio of the front-view rail fastener target frame relative to the front-view image, the electromagnetic data of the corresponding length is intercepted to obtain the electromagnetic signal of the rail fastener, and the electromagnetic signal is mapped into data of the same size as the image through signal processing and deep learning network. The specific steps are as follows:

[0043] Step S81, electromagnetic signal data interception: according to the position of the rail fastener target frame in the positive viewing angle, a data segment corresponding to the image is intercepted from the electromagnetic signal data;

[0044] Step S82, data preprocessing: preprocessing the intercepted electromagnetic signal, including filtering to remove noise and possible baseline correction;

[0045] Step S83, signal processing: performing wavelet transform on the preprocessed electromagnetic signal to extract electromagnetic features E, the dimension is [E1, E2, E3];

[0046] Step S84, feature mapping: Use a deep learning network to map the electromagnetic feature E to the same size as the image. The formula is as follows:

[0047] Er=Flatten12(E)

[0048] Where Er is the feature after the first and second dimensions of E are flattened, the dimension is [E1×E2,E3], and Flatten12 represents the flattening operation;

[0049] En=act(WEr)

[0050] Where En is the electromagnetic feature after mapping, with a dimension of [416, 416, E3], act() is the activation function, and W is the three-dimensional mapping matrix with a dimension of [416, 416, E1×E2].

[0051] Step S9, surface position data interception and mapping: high-precision surface position data of the rail fastener is obtained by a line laser scanning device, the range of which is consistent with the range of the front-view image. According to the position ratio of the front-view rail fastener target frame relative to the front-view image, the corresponding rail fastener surface position data is intercepted and mapped into data of the same size as the image through a two-layer feedforward network; the specific steps are as follows:

[0052] Step S91, surface position data capture: based on the position and size of the rail fastener target frame in the positive viewing angle, the corresponding surface position data segment is captured from the original data acquired by the line laser scanning device.

[0053] Step S92, data normalization: normalize the intercepted surface position data to make its value range suitable for subsequent network mapping, and obtain the processed surface position data R with a dimension of [R1, R2].

[0054] Step S93, feature mapping: Use a deep learning network to map the surface position data R to the same size as the image. The formula is as follows:

[0055] Rr=Flatten12(R)

[0056] Where Rr is the feature after the first and second dimensions of R are flattened, with a dimension of [R1×R2], and Flatten12 represents the flattening operation;

[0057] Rn=act(MRr)

[0058] Where Rn is the surface position feature after mapping, with a dimension of [416, 416], act() is the activation function, and M is the three-dimensional mapping matrix with a dimension of [416, 416, R1×R2].

[0059] Step S10, data integration: All the data in steps S6-S9 are integrated together according to the third dimension as input X of the multi-source feature fusion intelligent detection method.

[0060] X=[image1_norm,image1_left,image1_right,image2_norm,image2_left,image2_right,En,Rn]

[0061] Where X is the input of the multi-source feature fusion intelligent detection method, the dimension is [416, 416, C], all images are 3-channel images, and C is the sum of the third dimension of all features.

[0062] Step S11: Propose a multi-source feature fusion intelligent detection method to extract features:

[0063] Step S111, local feature extraction: extract local features from the input X using local convolution, the formula is as follows:

[0064] F_LOCAL=local_conv(X,K=5,D=1)

[0065] Among them, F_LOCAL is the local feature, local_conv represents the local convolution operation, the convolution scale K=5, the expansion rate D=1, and local feature extraction is performed.

[0066] Step S112, global feature extraction: Based on the local feature extraction, global feature extraction is performed using global convolution, and the formula is as follows:

[0067] F_GLOBAL=global_conv(F_LOCAL,K=5,D=3)

[0068] Among them, F_GLOBAL is the global feature, global_conv represents the local convolution operation, the convolution scale K=5, and the dilation rate D=3. Increasing the dilation rate can sense a wider range.

[0069] Step S113, dimensional feature extraction: Based on the global feature extraction, dimensional feature extraction is performed using 1*1 convolution, and the formula is as follows:

[0070] F_DIM=dim_conv(F_GLOBAL,K=1,D=1)

[0071] Among them, F_DIM is the dimensional feature, dim_conv represents the dimensional convolution operation, the convolution scale K=1, the expansion rate D=1, feature extraction is performed on the dimension, and multi-source features are integrated.

[0072] Step S114, feature fusion: perform a dot multiplication operation on the extracted feature F_DIM and the original feature X to obtain the final multi-source feature F, which integrates the rail fastener target, foreign body target, electromagnetic signal and line position information.

[0073] Step S115, abnormal state classification: multi-source features F are classified through a two-layer feedforward neural network, and the output is a label for judging abnormal state. The normalized label range is [0,1], and the threshold is p. If it exceeds p, it means the rail is abnormal, and if it is lower than p, it means the rail is normal. The threshold is obtained from experience and initialized to 0.5, which can be adjusted according to actual conditions.

[0074] Step S116, incremental training: In practical applications, for images whose abnormal state label is in [p-0.1, p+0.1], manual annotation judgment is performed. If the actual abnormal state is inconsistent with the abnormal state predicted by the model, this data is included in the incremental training set. After a period of collection, incremental training is performed to improve the accuracy of the model.

[0075] Step S12, lightweight deployment: In order to improve the efficiency of real-time monitoring, the model is quantized to int8. While ensuring a certain initial inspection rate, the storage space and computational complexity of the model can be significantly reduced, and real-time monitoring and anomaly detection of fastener status can be achieved.

[0076] Step S13, real-time warning mechanism: according to the model detection results, establish a warning mechanism for abnormal situations, set the corresponding threshold p, and when the model detects that the rail is abnormal, promptly notify relevant personnel for emergency processing. Example 2

[0077] The intelligent detection system of abnormal rail fasteners with multi-source information fusion is characterized by:

[0078] Binocular camera multi-view acquisition module: The railway automatic inspection vehicle sets up a sampling camera directly above each rail, and sets up a sampling camera at 45° to the plane of the two rails, so as to achieve multi-view coverage of rail fasteners;

[0079] Electromagnetic detection equipment module: The electromagnetic detection equipment is installed above the railway automatic inspection trolley to ensure that it can effectively interact with the fasteners when passing through the rails. The electromagnetic detection equipment is used to detect internal damage or looseness of the fasteners;

[0080] Line laser scanning equipment module: Install the line laser scanning equipment above the railway automatic inspection trolley to ensure that the entire surface of the fastener can be covered; use the line laser scanning equipment to obtain high-precision surface position information of the rail fastener;

[0081] Data collection module: used to collect multi-view image data, electromagnetic data, and line laser position distance data;

[0082] Left and right oblique view image data correction module: The distortion of the left and right oblique view images is eliminated through Hough correction transformation, and the images are converted into left view images and right view images;

[0083] Target object detection and selection module: perform rail fastener detection on the front view image, left view image and right view image respectively, and obtain the front view rail fastener target frame, left view rail fastener target frame and right view rail fastener target frame;

[0084] Marking frame selection and alignment module: Combine the rail fastener target frames of different perspectives and perform secondary alignment on the target frames; for the positive perspective image, map the left-perspective rail fastener target frame and the right-perspective rail fastener target frame to the positive perspective image through pixel alignment to obtain the positive perspective left-mapped rail fastener target frame and the positive perspective right-mapped rail fastener target frame, and add the existing positive perspective rail fastener target frame to take the average of the three frames as the positive perspective rail fastener alignment target frame; similarly, obtain the left-perspective rail fastener alignment target frame and the right-perspective rail fastener alignment target frame;

[0085] Image processing module: crop the images of the three perspectives according to the rail fastener alignment target frame of each perspective, and remove irrelevant background; scale the cropped images to a uniform size of 416*416 pixels to obtain the normal perspective target image image1_norm, the left perspective target image image1_left and the right perspective target image image1_right;

[0086] Foreign object detection module: foreign object detection is performed in the normal view target image, the left view target image and the right view target image respectively, the foreign objects are selected by a box, and the pixels outside the foreign object box are set to 0, so as to obtain the normal view foreign object image image2_norm, the left view foreign object image image2_left and the right view foreign object image image2_right, and the image size is also 416*416;

[0087] Electromagnetic data interception and mapping module: According to the position ratio of the rail fastener target frame in the front view relative to the front view image, the electromagnetic data of the corresponding length is intercepted to obtain the electromagnetic signal of the rail fastener, and then mapped into data of the same size as the image through signal processing and deep learning network;

[0088] Surface position data interception and mapping module: The high-precision surface position data of the rail fastener is obtained through a line laser scanning device. The range is consistent with the range illuminated by the orthographic image. According to the position ratio of the orthographic rail fastener target frame relative to the orthographic image, the corresponding surface position data of the rail fastener is intercepted and mapped into data of the same size as the image through a two-layer feedforward network;

[0089] Data integration module: Integrate the data according to the third dimension as input X of the multi-source feature fusion intelligent detection method.

[0090] X=[image1_norm,image1_left,image1_right,image2_norm,image2_left,image2_right,En,Rn];

[0091] Where X is the input of the multi-source feature fusion intelligent detection method, the dimension is [416,416,C], all images are 3-channel images, and C is the sum of the third dimension of all features;

[0092] Lightweight deployment module: quantize the model into int8 to achieve real-time monitoring of fastener status and anomaly detection;

[0093] Real-time warning mechanism module: According to the model detection results, establish a warning mechanism for abnormal situations and set the corresponding threshold p. When the model detects abnormalities in the rails, promptly notify relevant personnel for emergency processing.

[0094] The present invention adopts multi-view fastener image acquisition technology: a new multi-view railway abnormal fastener intelligent detection method including a binocular camera is proposed. The railway automatic inspection vehicle sets up a sampling camera just above each rail and sets up a sampling camera at about 45° relative to the plane of the two rails. At the same time, multi-view images are collected for the fasteners on both sides of the two rails to ensure that the all-round status of the fasteners is covered.

[0095] The present invention adopts a multi-source feature fusion intelligent detection method: using a convolutional neural network algorithm based on deep learning, the acquired fastener images are analyzed and identified to achieve automatic detection and classification of fasteners. Subsequently, the image data information from multiple sources at the same detection position is fused, combined with the image recognition results, and integrated analysis is performed using a fusion algorithm to improve the accuracy and reliability of abnormal fastener detection.

[0096] The present invention provides a real-time detection module for abnormal fasteners: a lightweight algorithm model is designed while maintaining a certain detection rate, the data collected by the sensor is processed and analyzed in real time, and data processing technology is used in the monitoring center in combination with a deep learning algorithm to achieve real-time monitoring of the fastener status and abnormality detection.

[0097] The present invention sets up a real-time early warning mechanism: according to the model detection results, an early warning mechanism for abnormal situations is established, and corresponding thresholds and rules are set, so as to guide relevant personnel to carry out emergency processing.

[0098] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. An intelligent detection method for abnormal rail fasteners by multi-source information fusion, characterized by: Step S1, equipment preparation: including binocular camera, electromagnetic detection equipment and line laser scanning equipment: Step S11, binocular camera multi-view acquisition: the railway automatic inspection vehicle sets up a sampling camera directly above each rail, and sets up a sampling camera at 45° to the plane of the two rails, so as to achieve multi-view coverage of the rail fasteners; Step S12, electromagnetic detection equipment: installing the electromagnetic detection equipment above the railway automatic inspection trolley to ensure that it can effectively interact with the fasteners when passing through the rails, so as to detect the defect characteristics of the fasteners; Step S13, line laser scanning equipment: the line laser scanning equipment is installed above the railway automatic inspection vehicle to ensure that the entire surface of the fastener can be covered; the line laser scanning equipment is used to obtain high-precision surface position information of the rail fastener; Step S2, data collection: multi-view image data, electromagnetic data, line laser position distance data: Step S21, capturing image data of the rail fastener from three different angles using a multi-view binocular camera, including a front view image, a left oblique view image, and a right oblique view image; Step S22: Detect the electromagnetic characteristics of the rail fasteners by electromagnetic detection equipment, identify the defect characteristics of the fasteners to obtain electromagnetic signal data, and ensure that the detection range is consistent with the range of the positive viewing angle image illumination; Step S23, obtaining high-precision surface position data of the rail fastener by a line laser scanning device, wherein the obtained surface position range is consistent with the range illuminated by the orthographic image; Step S24, transmitting the collected data to the central processor via the network for data processing; Step S3, correcting the left and right oblique view image data: Since the images are taken at 45° relative to the plane of the two rails, there will be a certain degree of distortion. The distortion of the left and right oblique view images is eliminated through Hough correction transformation to convert them into left view images and right view images; Step S4, target object detection frame selection: perform rail fastener detection on the front view image, the left view image and the right view image respectively to obtain the front view rail fastener target frame, the left view rail fastener target frame and the right view rail fastener target frame; Step S5, target object frame selection and alignment: Since the camera position and angle are fixed, there are overlapping parts when the cameras at different angles shoot the same object. The pixels of the overlapping parts in the two perspective images are aligned. Combined with the rail fastener target frames of different perspectives, the target frames are aligned twice. For the positive-view image, the left-view rail fastener target frame and the right-view rail fastener target frame are mapped to the positive-view image through pixel alignment to obtain the positive-view left-mapped rail fastener target frame and the positive-view right-mapped rail fastener target frame, and then the existing positive-view rail fastener target frame is added, and the average value of the three frames is taken as the positive-view rail fastener alignment target frame; similarly, the left-view rail fastener alignment target frame and the right-view rail fastener alignment target frame are obtained; Step S6, image processing: crop the images of the three viewing angles, namely, the front and left viewing angles, according to the rail fastener alignment target frame of each viewing angle, and remove irrelevant background; in order to facilitate feature extraction, the cropped images are uniformly scaled to a size of 416*416 pixels to obtain the front viewing angle target image image1_norm, the left viewing angle target image image1_left, and the right viewing angle target image image1_right; Step S7, foreign body detection: foreign body detection is performed in the normal view target image, the left view target image and the right view target image respectively, the foreign body is selected by a frame, and the pixels outside the foreign body frame are set to 0, so as to obtain the normal view foreign body image image2_norm, the left view foreign body image image2_left and the right view foreign body image image2_right, and the image size is also 416*416; Step S8, electromagnetic data interception and mapping: The electromagnetic signal data range is consistent with the range illuminated by the front-view image. According to the position ratio of the front-view rail fastener target frame relative to the front-view image, the electromagnetic data of the corresponding length is intercepted to obtain the electromagnetic signal of the rail fastener, and the electromagnetic signal is mapped into data of the same size as the image through signal processing and deep learning network. The specific steps are as follows: Step S81, electromagnetic signal data interception: according to the position of the rail fastener target frame in the positive viewing angle, a data segment corresponding to the image is intercepted from the electromagnetic signal data; Step S82, data preprocessing: preprocessing the intercepted electromagnetic signal, including filtering to remove noise and baseline correction; Step S83, signal processing: performing wavelet transform on the preprocessed electromagnetic signal to extract electromagnetic features E, the dimension is [E1, E2, E3]; Step S84, feature mapping: Use a deep learning network to map the electromagnetic feature E to the same size as the image. The formula is as follows: Er=Flatten12(E) Where Er is the feature after the first and second dimensions of E are flattened, the dimension is [E1×E2,E3], and Flatten12 represents the flattening operation; En=act(W·Er) Where En is the electromagnetic feature after mapping, the dimension is [416,416,E3], act() is the activation function, W is the three-dimensional mapping matrix, the dimension is [416,416,E1×E2], W is used to transform Er into a three-dimensional matrix, · represents matrix multiplication; Step S9, surface position data interception and mapping: high-precision surface position data of the rail fastener is obtained by a line laser scanning device, the range of which is consistent with the range of the front-view image. According to the position ratio of the front-view rail fastener target frame relative to the front-view image, the corresponding rail fastener surface position data is intercepted and mapped into data of the same size as the image through a two-layer feedforward network. The specific steps are as follows: Step S91, surface position data interception: according to the position and size of the rail fastener target frame in the positive viewing angle, the corresponding surface position data segment is intercepted from the original data obtained by the line laser scanning device; Step S92, data normalization: normalize the intercepted surface position data to make its value range suitable for subsequent network mapping, and obtain processed surface position data R with a dimension of [R1, R2]; Step S93, feature mapping: Use a deep learning network to map the surface position data R to the same size as the image. The formula is as follows: Rr=Flatten12(R) Where Rr is the feature after the first and second dimensions of R are flattened, with a dimension of [R1×R2], and Flatten12 represents the flattening operation; Rn=act(M·Rr) Where Rn is the surface position feature after mapping, with a dimension of [416, 416], act() is the activation function, M is the three-dimensional mapping matrix, with a dimension of [416, 416, R1×R2], M is used to transform Rr into a three-dimensional matrix, and · represents matrix multiplication; Step S10, data integration: All data from steps S6-S9 are integrated together according to the third dimension as input X of the multi-source feature fusion intelligent detection method. X=[image1_norm,image1_left,image1_right,image2_norm,image2_left,image2_right,En,Rn]; Where X is the input of the multi-source feature fusion intelligent detection method, the dimension is [416,416,C], all images are 3-channel images, and C is the sum of the third dimension of all features; Step S11, extracting features by using a multi-source feature fusion intelligent detection method; Step S12, lightweight deployment: In order to improve the efficiency of real-time monitoring, the model is quantized to int8, and the storage space and computational complexity of the model are significantly reduced while ensuring a certain initial inspection rate, so as to achieve real-time monitoring and anomaly detection of fastener status; Step S13, real-time warning mechanism: according to the model detection results, establish a warning mechanism for abnormal situations, set the corresponding threshold p, and when the model detects that the rail is abnormal, promptly notify relevant personnel for emergency processing.

2. The intelligent detection method of abnormal rail fasteners by multi-source information fusion according to claim 1 is characterized by: The step S11 further includes the following contents: Step S111, local feature extraction: extract local features from the input X using local convolution, the formula is as follows: F_LOCAL=local_conv(X,K=5,D=1) Among them, F_LOCAL is the local feature, local_conv represents the local convolution operation, the convolution scale K=5, the expansion rate D=1, and local feature extraction; Step S112, global feature extraction: Based on the local feature extraction, global feature extraction is performed using global convolution, and the formula is as follows: F_GLOBAL=global_conv(F_LOCAL,K=5,D=3) Among them, F_GLOBAL is the global feature, global_conv represents the global convolution operation, the convolution scale K=5, and the dilation rate D=3. Increasing the dilation rate can sense a wider range; Step S113, dimensional feature extraction: Based on the global feature extraction, dimensional feature extraction is performed using 1*1 convolution, and the formula is as follows: F_DIM=dim_conv(F_GLOBAL,K=1,D=1) Among them, F_DIM is the dimensional feature, dim_conv represents the dimensional convolution operation, the convolution scale K=1, the dilation rate D=1, feature extraction is performed on the dimension, and multi-source features are integrated; Step S114, feature fusion: perform a dot multiplication operation on the extracted feature F_DIM and the original feature X to obtain the final multi-source feature F, which integrates the rail fastener target, foreign body target, electromagnetic signal and line position information; Step S115, abnormal state classification: the multi-source feature F is classified through a two-layer feedforward neural network, and the output is a label for judging the abnormal state. The normalized label range is [0, 1], and the threshold is p. If it exceeds p, it means that the rail is abnormal, and if it is lower than p, it means that the rail is normal. The threshold is obtained by experience and is initialized to 0.

5. Step S116, incremental training: For images with abnormal state labels in [p-0.1, p+0.1], manual annotation judgment is performed. If the actual abnormal state is inconsistent with the abnormal state predicted by the model, this data is included in the incremental training set. After a period of collection, incremental training is performed to improve the accuracy of the model.

3. An intelligent detection system for abnormal rail fasteners by multi-source information fusion, the system adopts the intelligent detection method for abnormal rail fasteners by multi-source information fusion as claimed in claim 1, and is characterized by: include: Binocular camera multi-view acquisition module: The railway automatic inspection vehicle sets up a sampling camera directly above each rail, and sets up a sampling camera at 45° to the plane of the two rails, so as to achieve multi-view coverage of the rail fasteners; Electromagnetic detection equipment module: The electromagnetic detection equipment is installed above the railway automatic inspection trolley to ensure that it can effectively interact with the fasteners when passing through the rails. The electromagnetic detection equipment is used to detect internal damage or looseness of the fasteners; Line laser scanning equipment module: Install the line laser scanning equipment above the railway automatic inspection trolley to ensure that the entire surface of the fastener can be covered; use the line laser scanning equipment to obtain high-precision surface position information of the rail fastener; Data collection module: used to collect multi-view image data, electromagnetic data, and line laser position distance data; Left and right oblique view image data correction module: The distortion of the left and right oblique view images is eliminated through Hough correction transformation, and the images are converted into left view images and right view images; Target object detection and selection module: perform rail fastener detection on the front view image, left view image and right view image respectively, and obtain the front view rail fastener target frame, left view rail fastener target frame and right view rail fastener target frame; Marking frame selection and alignment module: Combine rail fastener target frames from different perspectives and perform secondary alignment on the target frames; For the orthographic image, the left-view rail fastener target frame and the right-view rail fastener target frame are mapped to the orthographic image by pixel alignment to obtain the orthographic left-view mapped rail fastener target frame and the orthographic right-view mapped rail fastener target frame, and then the existing orthographic rail fastener target frame is added, and the average value of the three frames is taken as the orthographic rail fastener alignment target frame; Similarly, the left-view rail fastener alignment target frame and the right-view rail fastener alignment target frame are obtained; Image processing module: crop the images of the three perspectives according to the rail fastener alignment target frame of each perspective, and remove irrelevant background; scale the cropped images to a uniform size of 416*416 pixels to obtain the normal perspective target image image1_norm, the left perspective target image image1_left and the right perspective target image image1_right; Foreign object detection module: foreign object detection is performed in the normal view target image, the left view target image and the right view target image respectively, the foreign objects are selected by a box, and the pixels outside the foreign object box are set to 0, so as to obtain the normal view foreign object image image2_norm, the left view foreign object image image2_left and the right view foreign object image image2_right, and the image size is also 416*416; Electromagnetic data interception and mapping module: According to the position ratio of the rail fastener target frame in the front view relative to the front view image, the electromagnetic data of the corresponding length is intercepted to obtain the electromagnetic signal of the rail fastener, and then mapped into data of the same size as the image through signal processing and deep learning network; Surface position data interception and mapping module: The high-precision surface position data of the rail fastener is obtained through a line laser scanning device. The range is consistent with the range illuminated by the orthographic image. According to the position ratio of the orthographic rail fastener target frame relative to the orthographic image, the corresponding surface position data of the rail fastener is intercepted and mapped into data of the same size as the image through a two-layer feedforward network; Data integration module: Integrate the data according to the third dimension as input X of the multi-source feature fusion intelligent detection method. X=[image1_norm,image1_left,image1_right,image2_norm,image2_left,image2_right,En,Rn]; Where X is the input of the multi-source feature fusion intelligent detection method, the dimension is [416,416,C], all images are 3-channel images, and C is the sum of the third dimension of all features; Lightweight deployment module: quantize the model into int8 to achieve real-time monitoring of fastener status and anomaly detection; Real-time warning mechanism module: According to the model detection results, establish a warning mechanism for abnormal situations and set the corresponding threshold p. When the model detects abnormalities in the rails, promptly notify relevant personnel for emergency processing.

4. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute the method according to claim 1.

5. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method according to claim 1 when executed.

Images