Rail fastener anomaly detection method and system

Abstract

The present invention provides a method and system for detecting rail fastener anomalies, belonging to the technical field of computer-based rail fastener anomaly detection. The system uses a high-precision three-dimensional line-scanning laser acquisition device to scan rail fasteners and collect and register paired depth and grayscale images of the fasteners. A pixel-level dimensional complementary image fusion method is used to efficiently fuse the depth image with the corresponding grayscale image to construct a fused rail fastener dataset. A feature fusion-decoupling module is constructed to achieve multi-dimensional feature fusion and task decoupling within the backbone network. A loss function reweighting method guided by a detection accuracy metric is employed to continuously adjust the category weight matrix during the training process, thereby enabling rail fastener anomaly detection. The present invention provides a method for detecting rail fastener anomalies based on multi-source data fusion. This method effectively utilizes multi-source data information, addressing the problem of misjudgment caused by relying solely on two-dimensional vision, which is susceptible to interference from surface rust and stains, and achieving accurate detection of rail fastener anomalies.

Description

Technical Field

[0001] The present invention relates to the technical field of computer-based rail fastener anomaly detection, and in particular to a rail fastener anomaly detection method and system. Background Art

[0002] Track infrastructure equipment is fundamental to ensuring the normal operation of trains, and rail fasteners are a crucial component. Fasteners primarily consist of spring clips, spiral spikes, gauge plates, and pads. Faults in various components can pose different risks. Promptly inspecting the health of each fastener component helps ensure safe and normal train operation and precise maintenance by maintenance departments. Traditional inspection methods rely on manual patrols, which not only wastes manpower and resources but also struggles to ensure real-time performance and accuracy. Therefore, the development of advanced and intelligent algorithms for detecting anomalies in railway fasteners is crucial.

[0003] Most existing anomaly detection methods rely on two-dimensional visual image information, using image processing and deep learning algorithms to identify and locate anomalies. However, in engineering applications, input algorithms that lack three-dimensional information are prone to misjudgment due to the influence of light, shadows, and surface stains. Three-dimensional information acquisition equipment represented by lidar, depth cameras, and stereo vision are gradually emerging, each with its own advantages and disadvantages. In view of the small size of fasteners and the fineness of defects, depth cameras are widely used due to their advantages of close range and high precision. Currently, the commonly used multi-source data fusion methods are feature layer fusion and result layer fusion. The network model constructed by feature layer fusion is complex and has the disadvantage of occupying a large amount of memory. The result layer fusion of sequential structure does not fully utilize the advantages of multi-source data, and the degree of fusion is not high. Summary of the Invention

[0004] The object of the present invention is to provide a method and system for detecting abnormalities in rail fasteners, so as to solve at least one technical problem existing in the above-mentioned background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In one aspect, the present invention provides a method for detecting anomalies in rail fasteners, comprising:

[0007] Acquire rail fastener images;

[0008] The acquired rail fastener images are processed using a pre-trained anomaly detection model to obtain rail fastener anomaly detection results. Training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale image to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to achieve multi-dimensional feature fusion and task decoupling of the backbone network; and using a loss function reweighting method guided by a detection accuracy indicator to continuously adjust the category weight matrix during the training process to achieve rail fastener anomaly detection.

[0009] Furthermore, a pixel-level dimensional complementary image fusion method is used to fuse the depth map and the corresponding grayscale image to construct a rail fastener fusion dataset, including:

[0010] The grayscale scaling coefficient c is 1.0, and the adjustment constant γ is 0.7. The GAMMA operator is used to enhance the brightness and contrast of the grayscale image. Gray is the grayscale pixel matrix, Img Gray ' is the enhanced grayscale pixel matrix;

[0011] Img Gray ′=c·Img Gray γ ;

[0012] Normalize the depth map to the same order of magnitude as the grayscale image, Img Depth is the depth pixel matrix, Img Depth ' is the normalized depth pixel matrix, min is the minimum value of the matrix, max is the maximum value of the matrix, (i, j) represents the i-th row and j-th column of the pixel matrix;

[0013]

[0014] Perform a dot multiplication operation on the two-dimensional pixel matrix at the pixel level and the third-dimensional depth matrix, Img Multiply is the product pixel matrix;

[0015] Img Multiply =Img Gray ′(i,j)·Img Depth ′(i,j)

[0016] Overflow processing is performed by normalizing with 0 as the lower limit to obtain the railway fastener fusion image, Img Fused is the fused pixel matrix;

[0017]

[0018] Furthermore, a novel feature fusion-decoupling module is constructed to realize multi-dimensional feature fusion and task decoupling of the backbone network, including: the backbone network consists of convolutional layers, pooling layers and three types of Blocks. Block1 includes a convolutional normalized activation function layer and multiple C3_1 modules, Block2 includes a convolutional normalized activation function layer and an upsampling layer, and Block3 includes three C3_2 modules and a convolutional normalized activation function layer. The three modules are alternately and repeatedly used to form the backbone network.

[0019] Furthermore, the multi-dimensional feature fusion method is to fuse the feature maps extracted from the shallow and deep blocks of the backbone network three times. The three sizes are 80×80×256, 40×40×512, and 40×40×256 respectively. The shallow feature maps F1, F2, and F4 extracted by the backbone network are connected with the deep feature maps F3, F5, and F6 through the connection function based on PyTorch. The connection function is as follows:

[0020] torch.cat(F1,F5)

[0021] torch.cat(F2,F3)

[0022] torch.cat(F4,F6);

[0023] Task decoupling is completed by Block4, which includes multiple convolutional normalization activation function layers and 2D convolution layers.

[0024] Furthermore, feature maps F6 and F7 are input into a convolutional normalized activation function layer and then branched into two paths, which are respectively input into two convolutional normalized activation function layers. One branch is input into the 2D convolution layer and outputs the predicted category result, and the other branch is branched again and input into the 2D convolution layer and outputs the predicted position information and prediction confidence information respectively.

[0025] Furthermore, a loss function reweighting method guided by the detection accuracy index is adopted to continuously adjust the category weight matrix during the training process, thereby realizing rail fastener anomaly detection, including:

[0026] Select the detection accuracy index AP and the IoU threshold as [0.5:0.95];

[0027] Before each round of training, the AP matrix of the validation set of all previous rounds is weighted and calculated, where m is the number of training rounds, c is the category code, and k = 1, 2, ..., m-1;

[0028]

[0029] Perform maximum normalization operation on the weighted AP matrix aveAP;

[0030] Add constraints to modify the above matrix to obtain the weight w of the category loss function based on the cross entropy function c m , thereby achieving dynamic changes in network learning tendencies;

[0031]

[0032] In a second aspect, the present invention provides a rail fastener anomaly detection system, comprising:

[0033] An acquisition module, used for acquiring rail fastener images;

[0034] A detection module is configured to process the acquired rail fastener images using a pre-trained anomaly detection model to obtain rail fastener anomaly detection results. Training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale map to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to achieve multi-dimensional feature fusion and task decoupling of the backbone network; and using a loss function reweighting method guided by a detection accuracy indicator to continuously adjust the category weight matrix during the training process to achieve rail fastener anomaly detection.

[0035] In a third aspect, the present invention provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the rail fastener anomaly detection method as described above is implemented.

[0036] In a fourth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the rail fastener anomaly detection method as described above.

[0037] In a fifth aspect, the present invention provides an electronic device comprising: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes instructions for implementing the rail fastener anomaly detection method as described above.

[0038] Beneficial effects of the present invention: Using a multi-source data fusion method based on a data layer to construct a railway fastener fusion data set can achieve multi-dimensional information complementarity, and can also retain the original information to the greatest extent, solving the problem of misjudgment caused by the interference of light, shadows and surface stains in two-dimensional anomaly detection due to the lack of third-dimensional depth information, and realizing accurate anomaly detection of fasteners. Compared with the fusion method of the feature layer and the result layer, the fastener anomaly detection network constructed by the present invention through data layer fusion is more efficient and convenient, has a higher degree of fusion, and requires less computational effort. The loss function reweighting method guided by the detection accuracy index can solve the problem of imbalance in sample types, while improving the detection accuracy of difficult sample categories.

[0039] Additional advantages of the present invention will be more clearly given in the following description or learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is a grayscale image of the track fastener described in an embodiment of the present invention.

[0042] Figure 2 This is a depth diagram of the rail fastener according to an embodiment of the present invention.

[0043] Figure 3 This is a fusion diagram of the track fastener described in an embodiment of the present invention.

[0044] Figure 4 This is a framework diagram of the feature fusion-decoupling module described in an embodiment of the present invention.

[0045] Figure 5 This is a diagram of the structural blocks in the feature fusion-decoupling module described in an embodiment of the present invention.

[0046] Figure 6 This is a diagram showing the effect of detecting abnormalities in railway fasteners using the method described in an embodiment of the present invention.

[0047] Figure 7 This is a flow chart of the rail fastener anomaly detection method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention.

[0049] Those skilled in the art will understand that unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs.

[0050] It should also be understood that terms, such as those defined in commonly used dictionaries, should be understood to have a meaning consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless as defined herein.

[0051] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or groups thereof.

[0052] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless otherwise inconsistent.

[0053] To facilitate understanding of the present invention, the present invention is further explained below with reference to specific embodiments in conjunction with the accompanying drawings. However, the specific embodiments do not constitute a limitation on the embodiments of the present invention.

[0054] Those skilled in the art should understand that the drawings are merely schematic diagrams of embodiments, and the components in the drawings are not necessarily necessary for implementing the present invention.

[0055] Example 1

[0056] In this embodiment 1, a rail fastener anomaly detection system is first provided, including: an acquisition module for acquiring rail fastener images; a detection module for processing the acquired rail fastener images using a pre-trained anomaly detection model to obtain rail fastener anomaly detection results; wherein, training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale map to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to realize multi-dimensional feature fusion and task decoupling of the backbone network; and using a loss function reweighting method guided by a detection accuracy indicator to continuously adjust the category weight matrix during the training process to realize rail fastener anomaly detection.

[0057] In this embodiment 1, the above-mentioned system is used to implement a rail fastener anomaly detection method, including: using an acquisition module to acquire a rail fastener image; using a detection module to process the acquired rail fastener image based on a pre-trained anomaly detection model to obtain a rail fastener anomaly detection result; wherein, training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale map to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to realize multi-dimensional feature fusion and task decoupling of the backbone network; and using a loss function reweighting method guided by a detection accuracy indicator to continuously adjust the category weight matrix during the training process, thereby realizing rail fastener anomaly detection.

[0058] Among them, a pixel-level dimensional complementary image fusion method is used to fuse the depth map and the corresponding grayscale image to construct a rail fastener fusion dataset, including:

[0059] The grayscale scaling coefficient c is 1.0, and the adjustment constant γ is 0.7. The GAMMA operator is used to enhance the brightness and contrast of the grayscale image. Gray is the grayscale pixel matrix, Img Gray ' is the enhanced grayscale pixel matrix;

[0060] Img Gray ′=c·Img Gray γ ;

[0061] Normalize the depth map to the same order of magnitude as the grayscale image, Img Depth is the depth pixel matrix, Img Depth ' is the normalized depth pixel matrix, min is the minimum value of the matrix, max is the maximum value of the matrix, (i, j) represents the i-th row and j-th column of the pixel matrix;

[0062]

[0063] Perform a dot multiplication operation on the two-dimensional pixel matrix at the pixel level and the third-dimensional depth matrix, Img Multiply is the product pixel matrix;

[0064] Img Multiply =Img Gray ′(i,j)·Img Depth ′(i,j)

[0065] Overflow processing is performed by normalizing with 0 as the lower limit to obtain the railway fastener fusion image, Img Fused is the fused pixel matrix;

[0066]

[0067] A novel feature fusion-decoupling module is constructed to realize multi-dimensional feature fusion and task decoupling of the backbone network, including: the backbone network consists of convolutional layers, pooling layers and three types of Blocks. Block1 includes a convolutional normalized activation function layer and multiple C3_1 modules, Block2 includes a convolutional normalized activation function layer and an upsampling layer, and Block3 includes three C3_2 modules and a convolutional normalized activation function layer. The three modules are alternately and repeatedly formed into the backbone network.

[0068] The multi-dimensional feature fusion method is to fuse the feature maps extracted from the shallow and deep blocks of the backbone network three times. The three sizes are 80×80×256, 40×40×512, and 40×40×256 respectively. The shallow feature maps F1, F2, and F4 extracted by the backbone network are connected with the deep feature maps F3, F5, and F6 through the connection function based on PyTorch. The connection function is as follows:

[0069] torch.cat(F1,F5)

[0070] torch.cat(F2,F3)

[0071] torch.cat(F4,F6);

[0072] Task decoupling is completed by Block4, which includes multiple convolutional normalization activation function layers and 2D convolution layers.

[0073] Feature maps F6 and F7 are input into a convolutional normalized activation function layer and then branched into two paths, which are respectively input into two convolutional normalized activation function layers. One branch is input into the 2D convolution layer and outputs the predicted category result. The other branch branches again and inputs the 2D convolution layer and outputs the predicted position information and prediction confidence information respectively.

[0074] Using a loss function reweighting method guided by the detection accuracy indicator, the category weight matrix is ​​continuously adjusted during the training process to achieve rail fastener anomaly detection, including:

[0075] Select the detection accuracy index AP and the IoU threshold as [0.5:0.95];

[0076] Before each round of training, the AP matrix of the validation set of all previous rounds is weighted and calculated, where m is the number of training rounds, c is the category code, and k = 1, 2, ..., m-1;

[0077]

[0078] Perform maximum normalization operation on the weighted AP matrix aveAP;

[0079] Add constraints to modify the above matrix to obtain the weight w of the category loss function based on the cross entropy function c m , thereby achieving dynamic changes in network learning tendencies

[0080]

[0081] Example 2

[0082] like Figure 7 As shown, in this embodiment 2, a method for detecting anomalies of rail fasteners is provided. To address the problem that two-dimensional vision is easily disturbed by planar factors, a line structured light camera is used to collect and align paired grayscale images and depth images of rail fasteners. A pixel-level dimensional complementary image fusion method is used to efficiently fuse the depth image and the corresponding grayscale image, thereby constructing a rail fastener fusion dataset. A novel feature fusion-decoupling module is constructed to effectively fuse the semantic features of the grayscale image at low resolution and the spatial features of the depth image at high resolution. The classification and regression tasks are decoupled in the detection head, and the fused dataset is trained in conjunction with the feature extraction backbone network of target detection. Taking into account the imbalance of sample categories in the fused dataset, a loss function reweighting method guided by the detection accuracy indicator is adopted. During the training process, the weights of each category in the category loss function are continuously adjusted according to the global validation set AP, thereby achieving accurate detection of rail fastener anomalies.

[0083] The rail fastener anomaly detection method described in this embodiment 2 includes the following steps:

[0084] In step 1, a self-developed track inspection vehicle travels back and forth on the ballastless track to collect and register paired fastener depth and grayscale images. The vehicle is equipped with high-precision 3D line-scan laser acquisition equipment, primarily consisting of a structured light camera and a laser transmitter. The use of multiple sensors increases the dimensionality of the collected data, enabling the merging of redundant information and the combination of complementary information.

[0085] Step 2: Use the pixel-level dimensional complementary image fusion method to efficiently fuse the depth map and the corresponding grayscale image to construct a rail fastener fusion dataset. Without increasing the network computational load, the two-dimensional plane information and the three-dimensional depth information are efficiently and quickly fused. The addition of depth information can effectively improve the influence of light and surface stains on rail fastener data. Figure 1 As shown, the difference between the grayscale image and the fusion image is quite obvious. The brightness of the rail is significantly improved, and the light and dark contrast of the fastener is enhanced.

[0086] The step 2 specifically includes the following steps:

[0087] Step 2.1, the grayscale image pixel matrix represents the color, with a value in [0, 255]. The grayscale image is nonlinearly transformed using the GAMMA operator with a grayscale scaling coefficient c of 1.0 and a constant γ of 0.7 to enhance the brightness and contrast of the grayscale image. Img Gray is the grayscale pixel matrix, Img Gray ' is the enhanced grayscale pixel matrix;

[0088] Img Gray ′=c·Img Gray γ

[0089] Step 2.2, the depth map pixel matrix represents the relative height, the value is [5000, 10000], the depth map is normalized to the same order of magnitude as the grayscale image, Img Depth is the depth pixel matrix, Img Depth ' is the normalized depth pixel matrix, min is the minimum value of the matrix, max is the maximum value of the matrix, (i, j) represents the i-th row and j-th column of the pixel matrix;

[0090]

[0091] Step 2.3, perform a dot multiplication operation on the two-dimensional pixel matrix at the pixel level and the third-dimensional depth matrix. The two pixel matrix elements are multiplied one-to-one to obtain the product pixel matrix, Img Multiply is the product pixel matrix;

[0092] Img Multiply =Img Gray ′(i,j)·Img Depth ′(i,j)

[0093] Step 2.4: To solve the problem of large-scale whitening of the image caused by saturated elements in the product pixel matrix, overflow processing is performed by normalizing with 0 as the lower limit to obtain the railway fastener fusion image, Img Fused is the fused pixel matrix;

[0094]

[0095] Step 3: Build a novel feature fusion-decoupling module to achieve multi-dimensional feature fusion and task decoupling of the backbone network, effectively extract multi-dimensional feature information of rail fasteners, and improve detection accuracy;

[0096] The step 3 specifically includes the following steps:

[0097] In step 3.1, the backbone network mainly consists of convolutional layers, pooling layers, and three types of blocks. Block 1 includes a convolutional normalization activation function layer and multiple C3_1 modules. Block 2 includes a convolutional normalization activation function layer and an upsampling layer. Block 3 includes three C3_2 modules and a convolutional normalization activation function layer. The three modules are alternately and repeatedly used to form the backbone network.

[0098] In step 3.2, the multi-dimensional feature fusion method is to fuse the feature maps extracted from the shallow and deep blocks of the backbone network three times. The three sizes are 80×80×256, 40×40×512, and 40×40×256 respectively. The shallow feature maps F1, F2, and F4 extracted by the backbone network are connected with the deep feature maps F3, F5, and F6 through the connection function based on PyTorch. The connection function is as follows:

[0099] torch.cat(F1,F5)

[0100] torch.cat(F2,F3)

[0101] torch.cat(F4,F6);

[0102] In step 3.3, task decoupling is completed by Block 4, which includes multiple convolutional normalization activation function layers and 2D convolution layers;

[0103] In step 3.4, feature maps F6 and F7 are input into a convolutional normalized activation function layer and then branched into two paths, which are respectively input into two convolutional normalized activation function layers. One path is input into a 2D convolution layer and outputs the predicted category result. The other branch branches again and inputs into the 2D convolution layer and outputs the predicted position information and prediction confidence information respectively.

[0104] Step 4: Using a loss function reweighting method guided by the detection accuracy metric, the class weight matrix is ​​continuously adjusted during training to achieve rail fastener anomaly detection. This step can effectively improve sample imbalance and reasonably improve the detection accuracy of the small sample class.

[0105] The step 4 specifically includes the following steps:

[0106] Step 4.1: Select the detection accuracy index AP and the IoU threshold as [0.5:0.95];

[0107] Step 4.2: Before each round of training, perform a weighted calculation on the AP matrix of the validation set of all previous rounds, where m is the number of training rounds, c is the category code, and k = 1, 2, ..., m-1;

[0108]

[0109] Step 4.3, perform maximum normalization operation on the weighted AP matrix aveAP;

[0110] In step 4.4, taking into account the gradient explosion and disappearance caused by weight magnitude overflow and weight zeroing, additional constraints are added to modify the above matrix to obtain the weight w of the category loss function based on the cross entropy function c m , thereby achieving dynamic changes in network learning tendencies.

[0111]

[0112] In summary, in Example 2, the construction of a railway fastener fusion dataset using a data layer-based multi-source data fusion approach can achieve multi-dimensional information complementarity and retain the original information to the greatest extent possible. This addresses the problem of misjudgment caused by two-dimensional anomaly detection due to the lack of third-dimensional depth information, which is susceptible to interference from light, shadows, and surface stains, thereby achieving accurate anomaly detection of fasteners. Compared to the fusion of feature layers and result layers, the fastener anomaly detection network constructed through data layer fusion is more efficient and convenient, with a higher degree of fusion and lower computational requirements. The loss function reweighting method guided by the detection accuracy index can address the problem of sample type imbalance and improve the detection accuracy of difficult sample categories.

[0113] Example 3

[0114] In this embodiment 3, a method for detecting abnormalities in rail fasteners is provided, comprising the following steps:

[0115] Step 1: Use high-precision 3D line-scanning laser data acquisition equipment to scan railway ballastless track fasteners and collect and register paired fastener depth and grayscale images. The use of multi-source sensors can increase the dimensionality of the collected data, enabling the merging of redundant information and the combination of complementary information.

[0116] The data acquisition device mainly includes a 3D structured light camera, a laser transmitter, a cooling fan, and a data adapter board. The data acquisition method is to drive a small car back and forth on the ballastless track at a speed of 1.0m / s in the experimental base. The simulated track length is 30m;

[0117] Step 2: Use the pixel-level dimensional complementary image fusion method to efficiently fuse the depth map and the corresponding grayscale image to construct a rail fastener fusion dataset. Without increasing the network computational load, the two-dimensional plane information and the three-dimensional depth information are efficiently and quickly fused. The addition of depth information can effectively improve the influence of light and surface stains on rail fastener data. Figure 1 As shown, the difference between the grayscale image and the fusion image is quite obvious. The brightness of the rail is significantly improved, and the light and dark contrast of the fastener is enhanced.

[0118] The step 2 specifically includes the following steps:

[0119] Step 2.1, the grayscale image pixel matrix represents the color, the value is [0, 255], the GAMMA operator is used to perform nonlinear transformation on the grayscale image to enhance the brightness and contrast of the grayscale image, the grayscale scaling factor c is set to 1.0, the adjustment constant γ is set to 0.7, Img Gray is the grayscale pixel matrix, Img Gray ' is the enhanced grayscale pixel matrix;

[0120] Img Gray ′=c·Img Gray γ

[0121] Step 2.2, the depth map pixel matrix represents the relative height, the value is [5000, 10000], the depth map is normalized to the same order of magnitude as the grayscale image, Img Depth is the depth pixel matrix, Img Depth ' is the normalized depth pixel matrix, min is the minimum value of the matrix, max is the maximum value of the matrix, (i, j) represents the i-th row and j-th column of the pixel matrix;

[0122]

[0123] Step 2.3, perform a dot multiplication operation on the two-dimensional pixel matrix at the pixel level and the third-dimensional depth matrix. The two pixel matrix elements are multiplied one-to-one to obtain the product pixel matrix, Img Multiply is the product pixel matrix;

[0124] Img Multiply =Img Gray ′(i,j)·Img Depth ′(i,j)

[0125] Step 2.4: To solve the problem of large-scale whitening of the image caused by saturated elements in the product pixel matrix, overflow processing is performed by normalizing with 0 as the lower limit, and finally the railway fastener fusion image Img is obtained. Fusedis the fused pixel matrix;

[0126]

[0127] Depend on Figures 1 to 3 It can be seen that a fusion graph is obtained after step 2.

[0128] Step 3: Build a novel feature fusion-decoupling module to achieve multi-dimensional feature fusion and task decoupling of the backbone network, effectively extract multi-dimensional feature information of rail fasteners, and improve detection accuracy;

[0129] The step 3 specifically includes the following steps:

[0130] Step 3.1: Construct the backbone network, which mainly consists of convolutional layers, pooling layers, and three types of blocks. Block1 includes a convolutional normalization activation function layer and multiple C3_1 modules. Block2 includes a convolutional normalization activation function layer and an upsampling layer. Block3 includes three C3_2 modules and a convolutional normalization activation function layer. The three modules are alternately and repeatedly used to form the backbone network.

[0131] In step 3.2, the multi-dimensional feature fusion method is to fuse the feature maps extracted from the shallow and deep blocks of the backbone network three times. The three sizes are 80×80×256, 40×40×512, and 40×40×256 respectively. The shallow feature maps F1, F2, and F4 extracted by the backbone network are connected with the deep feature maps F3, F5, and F6 through the connection function based on PyTorch. The connection function is as follows:

[0132] torch.cat(F1,F5)

[0133] torch.cat(F2,F3)

[0134] torch.cat(F4,F6);

[0135] In step 3.3, task decoupling is completed by Block 4, which includes multiple convolutional normalization activation function layers and 2D convolution layers;

[0136] In step 3.4, feature maps F6 and F7 are input into a convolutional normalized activation function layer and then branched into two paths, which are respectively input into two convolutional normalized activation function layers. One path is input into a 2D convolution layer and outputs the predicted category result. The other branch branches again and inputs into the 2D convolution layer and outputs the predicted position information and prediction confidence information respectively.

[0137] Step 4: Using a loss function reweighting method guided by the detection accuracy metric, the class weight matrix is ​​continuously adjusted during training to achieve rail fastener anomaly detection. This step can effectively improve sample imbalance and reasonably improve the detection accuracy of the small sample class.

[0138] The step 4 specifically includes the following steps:

[0139] Step 4.1: Select the detection accuracy index AP and the IoU threshold as [0.5:0.95];

[0140] Step 4.2: During the training process, before each round of training, perform a weighted calculation on the AP matrix of the validation set of all previous rounds, where m is the number of training rounds, c is the category code, and k = 1, 2, ..., m-1;

[0141]

[0142] Step 4.3, perform maximum normalization operation on the weighted AP matrix aveAP;

[0143] In step 4.4, considering the gradient explosion and disappearance caused by weight magnitude overflow and weight zeroing, additional constraints are added for correction to obtain the weight w of the category loss function based on the cross entropy function. c m , thereby achieving dynamic changes in network learning tendencies.

[0144]

[0145] In this example, a total of 2125 pairs of grayscale and depth images of fasteners were collected, covering seven categories: normal fasteners, missing spring clips, loose spring clips, nuts, missing nuts, missing baffles, and foreign object intrusion. The anomaly detection algorithm proposed in this example was then used to construct a fusion dataset of railway fasteners and perform detection. The detection results are shown in Table 1. As can be seen, all types of fastener anomalies were detected, with virtually no false positives or missed detections. In this dataset, this detection method achieved a mean average predictive power (mAP) of 95.6%.

[0146] Table 1 Experimental test results

[0147]

[0148] The innovation of the rail fastener anomaly detection method proposed in Example 3 mainly lies in the use of a pixel-level dimensional complementary image fusion method to efficiently fuse the depth map and its corresponding grayscale map, thereby constructing a railway fastener fusion dataset, and constructing a novel feature fusion-decoupling module to realize multi-dimensional feature fusion and task decoupling of the backbone network. The loss function reweighting method guided by the detection accuracy indicator is adopted, and the category weight matrix is ​​continuously adjusted during the training process to improve the detection accuracy of difficult sample categories, thereby realizing railway fastener anomaly detection and solving the problem of misjudgment caused by the interference of light, shadow and surface stains in two-dimensional anomaly detection due to the lack of third-dimensional depth information.

[0149] Example 4

[0150] This embodiment 4 provides a non-transitory computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the above-mentioned rail fastener anomaly detection method is implemented. The method includes:

[0151] Acquire rail fastener images;

[0152] The acquired rail fastener images are processed using a pre-trained anomaly detection model to obtain rail fastener anomaly detection results. Training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale image to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to achieve multi-dimensional feature fusion and task decoupling of the backbone network; and using a loss function reweighting method guided by a detection accuracy indicator to continuously adjust the category weight matrix during the training process to achieve rail fastener anomaly detection.

[0153] Example 5

[0154] This embodiment 5 provides a computer device, including a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute a rail fastener anomaly detection method, the method comprising:

[0155] Acquire rail fastener images;

[0156] The acquired rail fastener images are processed using a pre-trained anomaly detection model to obtain rail fastener anomaly detection results. Training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale image to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to achieve multi-dimensional feature fusion and task decoupling of the backbone network; and using a loss function reweighting method guided by a detection accuracy indicator to continuously adjust the category weight matrix during the training process to achieve rail fastener anomaly detection.

[0157] Example 6

[0158] This embodiment 6 provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to execute instructions for implementing the above-mentioned rail fastener abnormality detection method, which includes:

[0159] Acquire rail fastener images;

[0160] The acquired rail fastener images are processed using a pre-trained anomaly detection model to obtain rail fastener anomaly detection results. Training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale image to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to achieve multi-dimensional feature fusion and task decoupling of the backbone network; and using a loss function reweighting method guided by a detection accuracy indicator to continuously adjust the category weight matrix during the training process to achieve rail fastener anomaly detection.

[0161] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0165] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solutions disclosed in the present invention without the need for creative work should be included in the scope of protection of the present invention.

Claims

1. A method for detecting abnormalities in rail fasteners, characterized in that: include: Acquire rail fastener images; The acquired rail fastener images are processed using a pre-trained anomaly detection model to obtain rail fastener anomaly detection results; wherein, training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale map to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to realize multi-dimensional feature fusion and task decoupling of the backbone network; using a loss function reweighting method guided by the detection accuracy index to continuously adjust the category weight matrix during the training process to realize rail fastener anomaly detection; wherein, constructing A feature fusion-decoupling module is built to realize multi-dimensional feature fusion and task decoupling of the backbone network, including: the backbone network consists of convolutional layers, pooling layers and three types of Blocks. Block1 includes a convolutional normalization activation function layer and multiple C3 modules, Block2 includes a convolutional normalization activation function layer and an upsampling layer, and Block3 includes three C3 modules and a convolutional normalization activation function layer. The three modules are alternately and repeatedly constructed to form the backbone network; task decoupling is completed by Block4, which includes multiple convolutional normalization activation function layers and 2D convolution layers.

2. The rail fastener anomaly detection method according to claim 1, characterized in that: Using a pixel-level dimensional complementary image fusion method, we fuse the depth map with the corresponding grayscale image to construct a rail fastener fusion dataset, including: The grayscale scaling coefficient c is 1.0, and the adjustment constant γ is 0.

7. The GAMMA operator is used to enhance the brightness and contrast of the grayscale image. Gray is the grayscale pixel matrix, Img Gray ' is the enhanced grayscale pixel matrix; Img Gray ′=c·Img Gray γ ; Normalize the depth map to the same order of magnitude as the grayscale image, Img Depth is the depth pixel matrix, Img Depth ' is the normalized depth pixel matrix, min is the minimum value of the matrix, max is the maximum value of the matrix, (i, j) represents the i-th row and j-th column of the pixel matrix; Perform a dot multiplication operation on the two-dimensional pixel matrix at the pixel level and the third-dimensional depth matrix, Img Multiply is the product pixel matrix; Img Multiply =Img Gray ′(i,j)·Img Depth ′(i,j) Overflow processing is performed by normalizing with 0 as the lower limit to obtain the railway fastener fusion image, Img Fused is the fused pixel matrix; 3. The rail fastener anomaly detection method according to claim 1, characterized in that: The multi-dimensional feature fusion method is to fuse the feature maps extracted from the shallow and deep blocks of the backbone network three times. The three sizes are 80×80×256, 40×40×512, and 40×40×256 respectively. The shallow feature maps F1, F2, and F4 extracted by the backbone network are connected with the deep feature maps F3, F5, and F6 through the connection function based on PyTorch. The connection function is as follows: torch.cat(F1,F5) torch.cat(F2,F3) torch.cat(F4,F6).

4. The rail fastener anomaly detection method according to claim 3, characterized in that: The last two deep feature maps F6 and F7 are input into a convolutional normalized activation function layer and then branched into two paths, which are respectively input into two convolutional normalized activation function layers. One branch inputs the 2D convolution layer and outputs the predicted category result, and the other branch branches again and inputs the 2D convolution layer and outputs the predicted position information and prediction confidence information respectively.

5. The rail fastener anomaly detection method according to claim 4, characterized in that: Using a loss function reweighting method guided by the detection accuracy indicator, the category weight matrix is ​​continuously adjusted during the training process to achieve rail fastener anomaly detection, including: Select the detection accuracy index AP and the IoU threshold as [0.5:0.95]; Before each round of training, the AP matrix of the validation set of all previous rounds is weighted and calculated, where m is the number of training rounds, c is the category code, and k = 1, 2, ..., m-1; Perform maximum normalization operation on the weighted AP matrix aveAP; Add constraints to modify the above matrix to obtain the weight w of the category loss function based on the cross entropy function c m , thereby achieving dynamic changes in network learning tendencies; 6. A rail fastener anomaly detection system, characterized in that: include: An acquisition module, used for acquiring rail fastener images; A detection module is configured to process the acquired rail fastener images using a pre-trained anomaly detection model to obtain rail fastener anomaly detection results. Training the anomaly detection model includes: using a pixel-level dimensional complementary image fusion method to fuse the depth map of the rail fastener image with the corresponding grayscale image to construct a rail fastener fusion dataset; constructing a feature fusion-decoupling module to achieve multi-dimensional feature fusion and task decoupling of the backbone network; and using a loss function reweighting method guided by a detection accuracy indicator to continuously adjust the category weight matrix during the training process to achieve rail fastener anomaly detection. Among them, a feature fusion-decoupling module is constructed to realize multi-dimensional feature fusion and task decoupling of the backbone network, including: the backbone network is composed of convolutional layers, pooling layers and three types of Blocks. Block1 includes a convolutional normalization activation function layer and multiple C3 modules, Block2 includes a convolutional normalization activation function layer and an upsampling layer, and Block3 includes three C3 modules and a convolutional normalization activation function layer. The three modules alternate and repeat to form the backbone network; task decoupling is completed by Block4, which includes multiple convolutional normalization activation function layers and 2D convolution layers.

7. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the rail fastener anomaly detection method according to any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the processor and the memory communicate with each other, the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the rail fastener anomaly detection method according to any one of claims 1 to 5.

9. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute instructions for implementing the rail fastener anomaly detection method as described in any one of claims 1 to 5.

Images