Track fastener state identification method based on three-section parallel network architecture

Abstract

The invention provides a track fastener state recognition method based on a three-section type parallel network architecture. The method comprises the steps that a track inspection image is acquired; constructing a convolutional neural network (CNN) backbone network, and performing multi-scale feature extraction on the image of the track fastener; performing convolution processing and up-sampling processing on the fastener features by using the local feature branches to obtain local fastener features; performing global average pooling and convolution processing on the fastener features by using the global feature branches; the method comprises the following steps of: processing the characteristics of the fastener by using a mask generation branch of a framework based on Vision Transform, and generating a prototype mask of the track fastener; local, global and mask fastener features are fused; performing non-maximum suppression and mask generation on the fastener feature map; and analyzing the health state of the track fastener. According to the method, the CNN backbone network and the Vision Transform are combined, a new three-section parallel network architecture is provided, accurate identification and state analysis of the track fastener are realized through instance segmentation, and the identification accuracy in a complex environment is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a rail fastener state recognition method based on a three-segment parallel network architecture. Background Art

[0002] Fasteners are key connecting components in the rail system and play a vital role in ensuring the stability, reliability and safety of the rails. The main function of fasteners is to fix the rails to the sleepers to prevent the rails from moving in the lateral or longitudinal direction, while maintaining a stable spacing between the rails and reducing the impact force between the rails and the sleepers. With the rapid expansion of my country's railway network and the increase in train speeds, higher requirements are placed on the safety and integrity of the fastener system. If a large area of ​​fasteners is missing or damaged, it may cause the rails to deform or collapse, increase the risk of train derailment, and seriously threaten the safety of railway operations. Therefore, accurately and efficiently identifying the status of track fasteners is of great significance to ensuring the safe operation of railway lines.

[0003] Existing deep learning-based rail fastener status recognition algorithms usually use target detection. Although they can solve the real-time problem of rail fastener inspection to a certain extent, they cannot provide a fine-grained description of the fastener status, and it is difficult to distinguish between fasteners and interference objects in the environment in complex environments, resulting in missed detection or false detection. In addition, the existing methods have weak generalization capabilities in different rail scenarios, and the models are prone to failure when switching between scenarios, which cannot meet the needs of efficient recognition in large-scale and complex environments.

[0004] The track fastener status recognition method disclosed in the patent (CN116721263A, a track fastener status recognition method based on real-time instance segmentation) uses a track fastener real-time instance segmentation model based on YOLACT, and identifies the track fastener status through Res2Net backbone network feature extraction and feature fusion. However, the patent solution has the following problems:

[0005] 1. The YOLACT algorithm is not suitable for small object detection. The effect of small object detection is not precise. Small objects will be ignored by the YOLACT algorithm. In addition, the YOLACT algorithm cannot establish the association between objects according to the scene. Take the rail as an example. The structure of the rail is symmetrical, but the YOLACT algorithm cannot take into account the potential connection between objects, resulting in inaccurate detection.

[0006] 2. The structure of the Res2Net backbone network is complex and the overall computational complexity is large. Summary of the invention

[0007] The object of the present invention is to solve at least one of the technical drawbacks.

[0008] To this end, an object of the present invention is to propose a rail fastener status identification method based on a three-segment parallel network architecture to solve the problems mentioned in the background technology and overcome the deficiencies in the prior art.

[0009] In order to achieve the above object, an embodiment of the present invention provides a rail fastener state identification method based on a three-segment parallel network architecture, comprising:

[0010] Step S1, acquiring a track inspection image, and preprocessing the track inspection image, including: contrast enhancement of a track fastener image in the track inspection image;

[0011] Step S2, constructing a convolutional neural network (CNN) backbone network, using the CNN backbone network to perform multi-scale feature extraction on the images of the rail fasteners, and extracting low-level fastener features, mid-scale fastener features, mid-high-level fastener features, and high-level fastener features of the rail fasteners in stages;

[0012] Step S3, using the local feature branch in the multi-level feature processing network to perform convolution processing and up-sampling processing on the mesoscale fastener features and mid-high-level fastener features of the rail fastener, extract the detailed information of the rail fastener, and obtain the local fastener feature F local-final ;

[0013] Step S4, using the global feature branch in the multi-level feature processing network to perform global average pooling and convolution processing on the high-level fastener features of the rail fastener, extract the global semantic information of the rail fastener, and obtain the global fastener feature F global-final ;

[0014] Step S5, using the mask generation branch of the architecture based on Vision Transformer to process the mid- and high-level fastener features and the high-level fastener features of the rail fastener, generate a prototype mask of the rail fastener, and obtain a mask fastener feature M(x, y, c);

[0015] Step S6, the local fastener feature F local-final , Global fastener feature F global-final Perform three-stage parallel feature fusion with the mask fastener feature M(x, y, c), and perform convolution processing on the fused fastener feature map;

[0016] Step S7, non-maximum suppression and mask generation are performed on the generated fastener feature map to achieve instance segmentation and recognition of rail fasteners, and the mask score of each instance is calculated and filtered. The overlapping area is eliminated by non-maximum suppression, and the mask instance M with the highest score is retained. NMS (x,y);

[0017] Step S8, according to the mask instance M NMS (x, y) Analyze the health status of the rail fastener and generate a health report of the rail fastener, wherein the health report of the rail fastener includes: the damaged area and damage degree of the rail fastener.

[0018] Preferably, in any of the above solutions, in step S1, the contrast enhancement of the track fastener image in the track inspection image comprises: defining a local window W i,j , the local window W i,j Used to calculate the local contrast enhancement parameter and calculate the average pixel value μ of the local window i,j for:

[0019]

[0020] Where I(x,y) is the track inspection image, (x,y) is the pixel coordinate in the image, |W i,j | indicates the number of track fastener pixels within the window;

[0021] Calculate the standard deviation σ of the local window of the scanning track fastener i,j ;

[0022] For each track fastener pixel I(x,y), define an adaptive contrast enhancement coefficient C i,j ;

[0023] The rail fastener enhanced image I′(x,y) is calculated as:

[0024] I'(x,y)=C i,j ·(I(x,y)-μ i,j )+μ global ;

[0025] Among them, μ global is the average pixel value of the global image.

[0026] Preferably, in step S2, four stages are adopted to extract low-level fastener features, mid-scale fastener features, mid-high-level fastener features and high-level fastener features of the rail fasteners in stages, wherein:

[0027] Low-level stage: Three 3×3 convolutional layers and one pooling layer with a stride of 1 are used to extract low-level fastener features:

[0028] Mid-level stage: four 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract mid-scale fastener features;

[0029] Mid- and high-level stages: Four 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract mid- and high-level fastener features;

[0030] High-level stage: Six 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract high-level fastener features.

[0031] Preferably, from any of the above schemes, the low-level fastener features include edge information of the track fastener; and the high-level fastener features are global fastener features, including the fastener shape and position of the track fastener.

[0032] Preferably, in any of the above schemes, in step S3,

[0033] Convolution processing is performed on the mesoscale fastener features and the mid- and high-level fastener features respectively;

[0034] The high- and medium-level fastener features after convolution are adjusted to the same resolution as the mid-scale fastener features after convolution through upsampling operation;

[0035] Fuse the mid-scale fastener features after convolution with the mid- and high-level fastener features after upsampling;

[0036] The fused fastener features are compressed through convolution to generate local fastener features F local-final .

[0037] Preferably, in any of the above schemes, in step S4,

[0038] First, the high-level fastener feature F4(x, y, c) is globally averaged pooled to obtain the globally pooled fastener feature F global for:

[0039]

[0040] Where H and W are the height and width of the fastener feature graph, respectively;

[0041] Then the globally pooled fastener feature F global After processing through two convolutional layers, the global fastener feature F is generated. global-final .

[0042] Preferably, in any of the above schemes, in step S5,

[0043] The fastener feature map of the middle and high-level fastener features and the high-level fastener features is divided into image blocks of fixed size. Assuming that the size of each block is p×p, the fastener feature map F(x, y, c) is divided into blocks, each block is flattened and mapped to a vector

[0044]

[0045] Each vector is input into the Transformer layer and the attention weight is calculated through the self-attention mechanism:

[0046]

[0047] Among them, Q, K, and V are query matrix, key matrix, and value matrix respectively. k is the dimension of the key;

[0048] After the Transformer decoder layer, the fastener features used to generate the mask are output:

[0049]

[0050] The above fastener features are then remapped back to the spatial dimension through the Transformer decoder layer:

[0051]

[0052] Finally, the rail fastener mask predicts the fastener feature map M(x, y, c).

[0053] Preferably, in any of the above schemes, in step S6,

[0054] First, the local fastener feature F local-final and the global fastener feature F global-final Dimensionality reduction through 1×1 convolution:

[0055] F local-reduced =ReLU(W9*F local-final +b9)

[0056] F global-reduced =ReLU(W 10 *F global-final +b 10 )

[0057] Then the reduced-dimensional fastener features are weightedly fused with the fastener feature map of the mask generation branch M(x, y, c):

[0058] F fusion =α·F local-reduced +β·F global-reduced +γ·M(x,y,c)

[0059] Among them, α, β and γ are fusion weight coefficients;

[0060] Finally, the fused feature map is processed through a 3×3 convolutional layer:

[0061] F final =ReLU(W 11 *F fusion +b11 ).

[0062] Preferably, in any of the above solutions, in step S7, the mask score of each instance is calculated and filtered by a threshold value τ:

[0063]

[0064] Eliminate overlapping regions through non-maximum suppression and retain the instances with the highest scores:

[0065] M NMS (x,y)=NMS(M filtered (x,y)).

[0066] Preferably, in any of the above schemes, in step S8,

[0067] Calculate the area A of each rail fastener:

[0068]

[0069] According to the shape, area and position information of the mask, the health status of the rail fastener is judged:

[0070] Health Status=f(A,Shape,Position)

[0071] Among them, Shape is the shape of the rail fastener, and Position is the area of ​​the rail fastener.

[0072] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0073] 1. Accurate instance segmentation and fine-grained recognition: The existing technologies mostly use target detection methods, which can only roughly identify the status of track fasteners and cannot provide fine-grained analysis. The present invention combines CNN and ViT. Through instance segmentation technology, it can not only accurately distinguish track fasteners from environmental interferences and identify the location of track fasteners, but also realize fine-grained recognition of fastener status, and can effectively distinguish fasteners from interferences in the environment, thereby significantly improving the accuracy of recognition, especially in complex environments, and can effectively solve the problems of false detection and missed detection in the existing technologies. This method has stronger robustness and generalization ability in complex environments.

[0074] The present invention uses instance segmentation technology to describe the fastener status in a fine-grained manner, and can distinguish the fastener status of different damage degrees and locations. The innovation of this fine-grained analysis method ensures the accuracy of inspection. The present invention combines convolutional neural network (CNN) and Vision Transformer (ViT) to propose a new three-stage parallel network architecture, which realizes accurate identification and status analysis of rail fasteners through instance segmentation, and improves the recognition accuracy in complex environments.

[0075] 2. Multi-branch feature extraction and fusion enhance the robustness of the model: Traditional technologies are prone to detection failure or insufficient generalization in different scenarios or environments. The present invention adopts a multi-branch architecture design, including local feature branches, global feature branches and mask generation branches, and effectively integrates feature information at different levels through a feature fusion mechanism. This technology not only improves the robustness and generalization ability of the model, but also ensures efficient recognition in complex track scenarios, solving the defect of unstable performance of existing technologies when the environment changes.

[0076] The present invention generates a mask through ViT and fuses it with local and global features to improve the segmentation accuracy of the model. This multi-branch feature fusion mechanism enhances the generalization ability of the model in different scenarios, enabling the model to adapt to different track environments. It can work stably in both simple and complex scenarios, significantly improving the accuracy and robustness of rail fastener recognition.

[0077] 3. Strong real-time processing capability to meet large-scale inspection needs: The processing efficiency of the existing technology is low and it is difficult to meet the real-time requirements. The present invention optimizes the reasoning speed of the model by combining a lightweight CNN backbone network with ViT, and can achieve real-time recognition of the status of rail fasteners while ensuring recognition accuracy, adapting to large-scale inspection tasks, thereby significantly solving the shortcomings of the existing technology in terms of processing speed and real-time performance.

[0078] Compared with traditional deep learning models, the present invention can realize real-time status recognition of track inspection, greatly improving the inspection efficiency without sacrificing recognition accuracy.

[0079] 4. The present invention can provide a more detailed analysis report of the fastener status, especially in the scenario where the fastener is damaged or missing, and can accurately identify the specific location and extent of the problem, greatly improving the accuracy of track inspection and maintenance efficiency.

[0080] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0082] Figure 1 It is a flow chart of a rail fastener state identification method based on a three-segment parallel network architecture implemented according to the present invention;

[0083] Figure 2 A schematic diagram of the model structure of a rail fastener state identification method based on a three-segment parallel network architecture implemented according to the present invention;

[0084] Figure 3 A schematic diagram of fastener mask generation according to an embodiment of the present invention;

[0085] Figure 4 A schematic diagram of local feature fusion implemented according to the present invention;

[0086] Figure 5 A schematic diagram of global feature fusion implemented according to the present invention;

[0087] Figure 6 A schematic diagram of multi-scale feature fusion implemented according to the present invention;

[0088] Figure 7a and Figure 7b The following is a test example provided according to the present invention. DETAILED DESCRIPTION

[0089] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0090] The following first describes the technical terms involved in the present invention:

[0091] Inference speed: Measured by the number of frames processed per second (FPS), that is, the average number of model iterations per second, it can show the speed at which the model processes input. The higher this value, the faster the inference speed and the better the model performance.

[0092] like Figure 1 and Figure 2 As shown, the rail fastener state identification method based on the three-segment parallel network architecture of the embodiment of the present invention includes:

[0093] Step S1, acquiring a track inspection image, and preprocessing the track inspection image, including: contrast enhancement of a track fastener image in the track inspection image.

[0094] Specifically, a track fastener adaptive contrast enhancement algorithm (TFACE) is designed, which is used to adaptively adjust the contrast according to the local standard deviation of the image to enhance the contrast of the acquired track inspection images.

[0095] Contrast enhancement of rail fastener images in rail inspection images, including:

[0096] First, define a local window W i,j , the local window W i,j Used to calculate the local contrast enhancement parameters. The window size is m×m. Calculate the average pixel value μ of the local window i,j for:

[0097]

[0098] Where I(x,y) is the track inspection image, (x,y) is the pixel coordinate in the image, |W i,j |Indicates the number of track fastener pixels within the window.

[0099] Then, the standard deviation σ of the local window of the scanning track fastener is calculated i,j :

[0100]

[0101] Secondly, for each track fastener pixel I(x,y), define an adaptive contrast enhancement coefficient C i,j as follows:

[0102]

[0103] Finally, the track fastener enhanced image I′(x,y) is calculated as:

[0104] I′(x,y)=C i,j ·(I(x,y)-μ i,j )+μ global (4)

[0105] Among them, μ global is the average pixel value of the global image, which is used to maintain the consistency of global brightness.

[0106] Step S2, constructing a convolutional neural network (CNN) backbone network, using the CNN backbone network to perform multi-scale feature extraction on the images of rail fasteners, and extracting low-level fastener features, mid-scale fastener features, mid-high-level fastener features, and high-level fastener features of the rail fasteners in stages.

[0107] Specifically, a convolutional neural network (CNN) is used to construct a backbone network, and four stages are used to extract low-level fastener features, mid-scale fastener features, mid-to-high-level fastener features, and high-level fastener features of rail fasteners in stages. Each stage consists of several convolutional layers and pooling layers, and the network structure gradually extracts from low-level fastener features to high-level fastener features.

[0108] (1) Low-level stage: Three 3×3 convolutional layers and one pooling layer with a stride of 1 are used to extract low-level fastener features, which include the edge information of the track fasteners.

[0109] (2) Mid-level stage: Four 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract mid-scale fastener features.

[0110] (3) Mid- and high-level stage: Four 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract mid- and high-level fastener features.

[0111] (4) High-level stage: Six 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract high-level fastener features. The high-level fastener features are global fastener features, including the fastener shape and position of the rail fasteners.

[0112] Step S3, using the local feature branch in the multi-level feature processing network to perform convolution and up-sampling processing on the mesoscale fastener features and mid- and high-level fastener features of the rail fasteners, extracting the detailed information of the rail fasteners and obtaining the local fastener features F local-final .

[0113] Specifically, Figure 4 As shown in the figure, the local feature branch receives the fastener feature map of the mesoscale fastener features and the mid- and high-level fastener features, and extracts the detailed information of the track fasteners through multiple convolution operations. The specific convolution operation process is as follows:

[0114] First, convolution processing is performed on the mesoscale fastener features and mid- and high-level fastener features respectively.

[0115] The fastener feature map F2(x, y, c) of the mesoscale fastener features passes through three convolutional layers, and the output of each layer is:

[0116]

[0117] Among them, * represents the convolution operation, W i and b i are the weight and bias of the i-th layer respectively.

[0118] The fastener feature map F3(x,y,c) of the middle and high-level fastener features is passed through two convolutional layers:

[0119]

[0120] Secondly, the high-level fastener features after convolution are adjusted to the same resolution as the mid-scale fastener features after convolution through upsampling operations as follows:

[0121]

[0122] Then, the convolutional mid-scale fastener features and the up-sampled mid- and high-level fastener features are fused as follows:

[0123]

[0124] Finally, the fused fastener features are channel compressed through 1×1 convolution to generate local fastener features F local-final ,as follows:

[0125] F local-final =ReLU(W6*F local +b6)(9)

[0126] Step S4: The global fastener feature branch is used to extract the global semantic information of the rail fastener. The global feature branch in the multi-level feature processing network is used to perform global average pooling (GAP) and convolution processing on the high-level fastener features of the rail fastener to extract the global semantic information of the rail fastener and obtain the global fastener feature F. global-final .

[0127] like Figure 5 As shown, firstly, the fastener feature map F4(x, y, c) of the high-level fastener feature is globally averaged pooled to obtain the fastener feature F after global pooling. global for:

[0128]

[0129] Where H and W are the height and width of the fastener feature map, respectively.

[0130] Then, the globally pooled fastener features F global After processing through two convolutional layers, the global fastener feature F is generated. global-final for:

[0131]

[0132] Step S5, using the mask generation branch of the architecture based on Vision Transformer (ViT) to process the mid- and high-level fastener features and high-level fastener features of the rail fastener, generate a prototype mask of the rail fastener, and obtain the mask fastener feature M(x, y, c).

[0133] Specifically, Figure 3 As shown, the mask generation branch adopts the Vision Transformer (ViT) architecture and takes the fastener feature maps of mid- and high-level fastener features and high-level fastener features as input.

[0134] First, the input fastener feature map is divided into fixed-size image patches. Assuming that the size of each patch is p×p, the fastener feature map F(x, y, c) is divided into blocks, each block is flattened and mapped to a vector

[0135]

[0136] The above vector is then input into the Transformer layer, and the attention weight is calculated through the Transformer self-attention mechanism:

[0137]

[0138] Among them, Q, K, and V are query matrix, key matrix, and value matrix respectively. k The dimension of the key.

[0139] It then passes through the Transformer decoder layer and outputs the fastener features used to generate the mask:

[0140]

[0141] Secondly, the above fastener features are remapped back to the spatial dimension through the Transformer decoder layer:

[0142]

[0143] Finally, the rail fastener mask predicts the fastener feature map M(x, y, c).

[0144] Step S6: local fastener feature F local-final , Global fastener feature F global-final A three-stage parallel feature fusion is performed with the mask fastener feature M(x, y, c) to complete the fastener feature fusion operation that can refine the rail fastener identification, and the fused fastener feature map is convolved.

[0145] like Figure 6 As shown, first, the local fastener feature F local-final and the global fastener feature F global-final Dimensionality reduction through 1×1 convolution:

[0146]

[0147] Then the reduced-dimensional fastener features are weightedly fused with the fastener feature map of the mask generation branch M(x, y, c):

[0148] F fusion =α·F local-reduced +β·F global-reduced +γ·M(x,y,c)(17)

[0149] Among them, α, β and γ are fusion weight coefficients.

[0150] Finally, the fused feature map is processed through a 3×3 convolutional layer as follows:

[0151] F final =ReLU(W 11 *F fusion +b 11 )(18)

[0152] The present invention adopts a multi-branch architecture, in which the local feature branch extracts the edge details of the fastener, the global feature branch extracts the overall semantic information of the fastener, and the mask generation branch accurately locates the fastener area through ViT. Finally, the three are weightedly fused through feature fusion to ensure the accuracy and robustness of recognition.

[0153] Step S7, non-maximum suppression and mask generation are performed on the generated fastener feature map to achieve instance segmentation and recognition of rail fasteners, and the mask score of each instance is calculated and filtered. The overlapping area is eliminated by non-maximum suppression, and the mask instance M with the highest score is retained. NMS (x,y).

[0154] Specifically, after the mask is generated, non-maximum suppression (NMS) is used to process overlapping detection results. The mask score of each instance is calculated and filtered by a threshold τ:

[0155]

[0156] Eliminate overlapping regions through non-maximum suppression and retain the instances with the highest scores:

[0157] M NMS (x,y)=NMS(M filtered (x,y))(20)

[0158] The present invention adopts a model architecture that combines a custom convolutional neural network (CNN) backbone network with a Vision Transformer (ViT). CNN is used to extract multi-scale features of rail fasteners, and extracts and fuses local details, global semantic information, and precise masks of fasteners through local feature branches, global feature branches, and mask generation branches of a parallel architecture. At the same time, ViT is used to generate precise masks of fasteners through a self-attention mechanism, which enhances segmentation accuracy in complex environments.

[0159] Step S8: according to the mask instance M NMS (x, y) analyzes the health status of the track fastener and generates a health report of the track fastener, wherein the health report of the track fastener includes: the damaged area and damage degree of the track fastener.

[0160] In this step, the rail fasteners are analyzed and their health status is determined based on their area, shape, and location information.

[0161] Specifically, based on the generated mask M NMS (x,y), analyze the status of the track fasteners. Calculate the area A of each fastener:

[0162] A=∑ x,y M NMS (x,y)(21)

[0163] Based on the shape, area, and position of the mask and prior knowledge, the health status of the fastener is determined:

[0164] Health Status=f(A,Shape,Position)(22)

[0165] Among them, Shape is the shape of the rail fastener, and Position is the area of ​​the rail fastener.

[0166] The result is a health report of the rail fastenings, indicating possible areas and extent of damage.

[0167] The present invention performs fine-grained segmentation and analysis of fastener states through instance segmentation. The model combines local feature branches with global feature branches to accurately identify fastener states such as damage, missing, and deformation, and outputs detailed information such as the shape and position of the fastener.

[0168] In addition, the present invention can also realize the feature extraction of rail fasteners in the following manner:

[0169] (1) The CNN backbone network is used to extract the original state feature map and the changed state feature map from the acquired rail fastener images.

[0170] (2) Then, feature extraction is performed on the original state feature map and the changed state feature map through semantic labeling to obtain the original state semantic label and the changed state semantic label, respectively.

[0171] (3) The original semantic label and the changed semantic label are converted and encoded with local features to obtain semantic labels with local feature information.

[0172] (4) The semantic label with local feature information is divided into multiple local labels, and the local feature labels are decoded by using the corresponding decoding method to obtain the decoded local feature labels, and the multiple local feature labels are weighted fused to obtain the local feature information of the rail fastener.

[0173] (5) The original semantic label and the changed semantic label are converted and encoded with global features to obtain semantic labels with global feature information.

[0174] (6) The semantic label with global feature information is divided into multiple global labels, and the global feature labels are decoded using the corresponding decoding method to obtain the decoded global feature labels, and the multiple global feature labels are weightedly fused to obtain the global feature information of the track fastener.

[0175] (7) A mask generation branch based on a Vision Transformer (ViT) architecture is used to process fastener features of rail fasteners, generate prototype masks of rail fasteners, and obtain mask feature information.

[0176] (8) The health status of rail fasteners can be identified by fusing local feature information, global feature information, and mask feature information.

[0177] Figure 7a and Figure 7b Table 1 shows the accuracy and speed performance of the solution provided by the present invention and the advanced model on the test set.

[0178] Table 1 Accuracy and speed performance of the present invention and advanced models on the test set

[0179]

[0180]

[0181] The rail fastener status recognition method based on a three-stage parallel network architecture proposed in the present invention realizes accurate recognition and status analysis of rail fasteners by combining convolutional neural network (CNN) and Vision Transformer (ViT) technology. This method can not only effectively distinguish rail fasteners from interference objects in the environment, but also perform fine-grained segmentation and recognition of fastener status. By adopting a customized CNN backbone network and multiple feature extraction branches, the model can automatically learn the local and global features of rail fasteners and enhance the robustness of the model. At the same time, with the help of Vision Transformer, accurate fastener masks are generated to ensure that the status of the fastener can still be accurately identified in complex environments. In addition, the present invention has good real-time performance and can efficiently process a large number of track inspection images without affecting the recognition accuracy, meeting the needs of real-time monitoring of fastener status in railway operations.

[0182] The following is an explanation of the technical effects of the convolutional neural network (CNN) and the Vision Transformer (ViT) technology in the present invention:

[0183] 1. Global context capture capability

[0184] Transformer's self-attention mechanism can handle global features in fastener detection tasks, especially for long-distance, continuous track scenarios. Traditional CNNs may only detect the status of fasteners in a local area, and it is difficult to effectively capture the distribution and status of fasteners at a long distance. Transformer can learn global context, thereby improving the overall understanding of fastener status.

[0185] This capability is particularly important when detecting fastener problems such as wear, looseness, and misalignment, as these problems not only depend on the local state of the fastener, but may also be related to the overall stress distribution and dynamic environment of the track.

[0186] 2. Complex background and occlusion processing

[0187] Railway environments are usually complex, and fasteners are subject to various noise interferences, such as deformation of the rails themselves, debris, shadows, etc. CNN+Transformer combines the local detail capture of CNN with the global information processing of Transformer to better distinguish fasteners in complex backgrounds, especially when there are occlusions or changes in perspective. Transformer can ensure that the detection results are not affected by noise and background interference by modeling longer context information.

[0188] 3. Multi-scale feature fusion

[0189] Transformer can capture features of different scales, while CNN is good at extracting detailed information locally. Railway fasteners are usually of different sizes and may show different scale characteristics under different shooting angles. The combination of CNN+Transformer can process these different scale features at the same time, effectively cope with the changes in fastener size, shape and perspective, thereby improving the robustness and accuracy of detection.

[0190] This multi-scale processing capability is very important for detecting subtle damage to fasteners (such as cracks and loose bolts), ensuring that small defects are not missed.

[0191] 4. Modeling long-term dependencies

[0192] Transformer's long-range dependency modeling capability can capture the relationship between fasteners and their surroundings and other components. This means that the system can not only detect the status of a single fastener, but also understand the role and impact of the fastener on the entire rail system. For example, when some fasteners are loose, it may cause abnormalities in other fasteners. Modeling these upstream and downstream relationships is critical to detecting abnormal patterns.

[0193] In fastener inspection, this global view helps to identify systemic problems rather than just localized failures.

[0194] 5. Adapt to dynamically changing scenarios

[0195] The railway environment is subject to many dynamic changes, such as vibrations caused by passing trains and weather changes, which may affect the results of fastener detection. CNN+Transformer can better adapt to these changes through the dynamic feature modeling capabilities of Transformer, thereby maintaining high detection performance in different environments. The CNN part is responsible for extracting local static details to ensure that key information is not lost when there are minor changes.

[0196] This ability to adapt to dynamic environments improves the robustness of the model in actual scenarios, enabling it to cope with complex external conditions.

[0197] 6. Processing large amounts of data

[0198] Railway fastener inspection usually involves large-scale image data, which may include continuous track images. Transformer has a natural advantage in processing large-scale data and long sequence data. It can use the self-attention mechanism to effectively filter out key information and improve detection efficiency. In addition, CNN can quickly extract local features, combined with the global optimization ability of Transformer, making the system more efficient when processing large-scale images.

[0199] 7. High detection accuracy

[0200] The combination of CNN and Transformer can accurately locate the edge details of fasteners, which is especially important for detecting subtle problems such as damage and cracks in fasteners. Compared with traditional convolutional networks, the addition of Transformer gives the model better context perception and can more accurately distinguish the different states of fasteners.

[0201] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0202] It is not difficult for those skilled in the art to understand that the present invention includes any combination of the invention content and specific implementation methods of the above specification and the various parts shown in the drawings. Due to the limited space and to make the specification concise, the various schemes composed of these combinations are not described one by one. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0203] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A rail fastener status recognition method based on a three-stage parallel network architecture, characterized in that: The steps include: Step S1, acquiring a track inspection image, and preprocessing the track inspection image, including: contrast enhancement of a track fastener image in the track inspection image; Step S2, constructing a convolutional neural network (CNN) backbone network, using the CNN backbone network to perform multi-scale feature extraction on the images of the rail fasteners, and extracting low-level fastener features, mid-scale fastener features, mid-high-level fastener features, and high-level fastener features of the rail fasteners in stages; Step S3, using the local feature branch in the multi-level feature processing network to perform convolution processing and up-sampling processing on the mesoscale fastener features and mid-high-level fastener features of the rail fastener, extract the detailed information of the rail fastener, and obtain the local fastener feature F local-final ; Step S4, using the global feature branch in the multi-level feature processing network to perform global average pooling and convolution processing on the high-level fastener features of the rail fastener, extract the global semantic information of the rail fastener, and obtain the global fastener feature F global-final ; Step S5, using the mask generation branch of the architecture based on Vision Transformer to process the mid- and high-level fastener features and the high-level fastener features of the rail fastener, generate a prototype mask of the rail fastener, and obtain a mask fastener feature M(x, y, c); Step S6, the local fastener feature F local-final , Global fastener feature F global-final Perform three-stage parallel feature fusion with the mask fastener feature M(x, y, c), and perform convolution processing on the fused fastener feature map; Step S7, non-maximum suppression and mask generation are performed on the generated fastener feature map to achieve instance segmentation and recognition of rail fasteners, and the mask score of each instance is calculated and filtered. The overlapping area is eliminated by non-maximum suppression, and the mask instance M with the highest score is retained. NMS (x,y); Step S8, according to the mask instance M NMS (x, y) Analyze the health status of the rail fastener and generate a health report of the rail fastener, wherein the health report of the rail fastener includes: the damaged area and damage degree of the rail fastener.

2. The rail fastener state identification method based on a three-segment parallel network architecture according to claim 1, characterized in that: In the step S1, the contrast enhancement of the track fastener image in the track inspection image includes: defining a local window W i,j , the local window W i,j Used to calculate the local contrast enhancement parameter and calculate the average pixel value μ of the local window i,j for: Where I(x,y) is the track inspection image, (x,y) is the pixel coordinate in the image, |W i,j | indicates the number of track fastener pixels within the window; Calculate the standard deviation σ of the local window of the scanning track fastener i,j ; For each track fastener pixel I(x,y), define an adaptive contrast enhancement coefficient C i,j ; The rail fastener enhanced image I′(x,y) is calculated as: I′(x,y)=C i,j ·(I(x,y)-μ i,j )+μ global ; Among them, μ global is the average pixel value of the global image.

3. The rail fastener state identification method based on a three-segment parallel network architecture according to claim 1, characterized in that: In step S2, four stages are adopted to extract the low-level fastener features, the mid-scale fastener features, the mid-high-level fastener features and the high-level fastener features of the rail fasteners in stages, wherein: Low-level stage: Three 3×3 convolutional layers and one pooling layer with a stride of 1 are used to extract low-level fastener features: Mid-level stage: four 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract mid-scale fastener features; Mid- and high-level stages: Four 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract mid- and high-level fastener features; High-level stage: Six 3×3 convolutional layers and one pooling layer with a stride of 2 are used to extract high-level fastener features.

4. The rail fastener state identification method based on a three-segment parallel network architecture as claimed in claim 3, characterized in that: The low-level fastener features include edge information of the track fastener; the high-level fastener features are global fastener features, including the fastener shape and position of the track fastener.

5. The rail fastener state identification method based on a three-segment parallel network architecture according to claim 1, characterized in that: In step S3, Convolution processing is performed on the mesoscale fastener features and the mid- and high-level fastener features respectively; The high- and medium-level fastener features after convolution are adjusted to the same resolution as the mid-scale fastener features after convolution through upsampling operation; Fuse the mid-scale fastener features after convolution with the mid- and high-level fastener features after upsampling; The fused fastener features are compressed through convolution to generate local fastener features F local-final .

6. The rail fastener state identification method based on a three-segment parallel network architecture according to claim 1, characterized in that: In step S4, First, the high-level fastener feature F4(x, y, c) is globally averaged pooled to obtain the globally pooled fastener feature F global for: Where H and W are the height and width of the fastener feature graph, respectively; Then the globally pooled fastener feature F global After processing through two convolutional layers, the global fastener feature F is generated. global-final .

7. The rail fastener state identification method based on a three-stage parallel network architecture according to claim 1, characterized in that: In step S5, The fastener feature map of the middle and high-level fastener features and the high-level fastener features is divided into image blocks of fixed size. Assuming that the size of each block is p×p, the fastener feature map F(x, y, c) is divided into blocks, each block is flattened and mapped to a vector Each vector is input into the Transformer layer and the attention weight is calculated through the self-attention mechanism: Among them, Q, K, and V are query matrix, key matrix, and value matrix respectively. k is the dimension of the key; After the Transformer decoder layer, the fastener features used to generate the mask are output: The above fastener features are then remapped back to the spatial dimension through the Transformer decoder layer: Finally, the rail fastener mask predicts the fastener feature map M(x, y, c).

8. The rail fastener state identification method based on a three-segment parallel network architecture according to claim 1, characterized in that: In step S6, First, the local fastener feature F local-final and the global fastener feature F global-final Dimensionality reduction through 1×1 convolution: <h2 style=";text-align:left;direction:ltr">F<h2 style=";text-align:left;direction:ltr"> local-reduced <h2 style=";text-align:left;direction:ltr"> =ReLU(W9*F<h2 style=";text-align:left;direction:ltr"> local-final <h2 style=";text-align:left;direction:ltr"> +b9) F global-reduced =ReLU(W 10 *F global-final +b 10 ) Then the reduced-dimensional fastener features are weightedly fused with the fastener feature map of the mask generation branch M(x, y, c): F fusion =α·F local-reduced +β·F global-reduced +γ·M(x,y,c) Among them, α, β and γ are fusion weight coefficients; Finally, the fused feature map is processed through a 3×3 convolutional layer: F final =ReLU(W 11 *F fusion +b 11 )。 9. The rail fastener state identification method based on a three-stage parallel network architecture according to claim 1, characterized in that: In step S7, the mask score of each instance is calculated and filtered by a threshold τ: Eliminate overlapping regions through non-maximum suppression and retain the instances with the highest scores: M NMS (x,y)=NMS(M filtered (x,y))。 10. The rail fastener state identification method based on a three-stage parallel network architecture according to claim 1, characterized in that: In step S8, Calculate the area A of each rail fastener: According to the shape, area and position information of the mask, the health status of the rail fastener is judged: Health Status=f(A,Shape,Position) Among them, Shape is the shape of the rail fastener, and Position is the area of ​​the rail fastener.

Images