A pantograph foreign matter detection method based on deep learning

Abstract

The application provides a pantograph foreign matter detection method based on deep learning, and a pantograph foreign matter detection model is first trained: a plurality of pantograph images in a normal state are acquired, and each image is preprocessed; a local region is subjected to convolution processing to obtain a feature vector; a full connection layer is used to cascade and perform dimension transformation on each feature vector, and a three-dimensional feature vector corresponding to the local region is output, which is recorded as a spatial coordinate; a loss function is calculated, and it is judged whether the loss function is less than a threshold A; if yes, network parameters are saved, and model training is completed; if no, an optimized deep learning model is iterated out by using an optimization method with the loss function as a target function, network parameters are saved, and model training is completed; and then the trained model is used to detect foreign matters in actually collected pantograph images to judge whether the current pantograph is shielded by foreign matters, and the method can accurately identify abnormalities in pantograph images, and effectively improve the safety and economy of train operation and maintenance.

Description

Technical Field

[0001] This invention relates to the field of train pantograph detection, specifically to a deep learning-based method for detecting foreign objects in a pantograph. Background Technology

[0002] In recent years, my country's railway development has been rapid, and railways are one of the main modes of transportation for people. Their operational safety has received widespread attention, placing higher demands on railway transportation safety management. Among these concerns, abnormal conditions of the pantograph have always been a key focus of safe transportation. When the pantograph is invaded by foreign objects (plastic bags, sky lanterns, tree branches, etc.), it can cause malfunctions such as pantograph jamming, wire pulling, and line fires. Therefore, once a foreign object intrusion occurs, the train must be immediately stopped for inspection and repair to ensure safe operation. Because the pantograph is located on the roof of the train, maintenance personnel need to climb onto the roof to check for abnormalities, making inspection difficult. Furthermore, foreign object intrusion into the pantograph is characterized by its randomness and complexity.

[0003] Traditional methods for detecting foreign objects on pantographs involve manual climbing onto the roof in conjunction with video recording from the vehicle. Manually climbing to inspect critical areas of the pantograph is labor-intensive and poses safety hazards; roof-mounted video recording relies on manual observation, which is prone to visual fatigue and can lead to missed detections. Other existing technologies include:

[0004] Reference CN202011266814.1 proposes a real-time detection method for foreign objects intruding into substations, which is based on an improved YOLO network model and performs deep learning detection. Reference CN112733771A proposes a method and system for identifying foreign object faults across railway trains, which uses a Faster-RCNN training model to identify and detect foreign objects. Both of these methods are supervised learning models, requiring a certain amount of images showing foreign object intrusion to ensure the accuracy of the training model. However, in actual train operation scenarios, the probability of foreign object intrusion into the pantograph is relatively low, the number of samples is limited, and it is difficult to complete model training in a short time, resulting in poor feasibility and difficulty in widespread application of these methods.

[0005] Literature CN201510996377.1 proposes a location detection method based on a foreign object intrusion security control device. It detects foreign objects by analyzing the waveform of laser detection signals. This detection method is highly dependent on lasers and has a high system complexity.

[0006] Literature CN202010245745.X proposes a method and device for identifying and locating foreign objects in tunnels. It detects foreign objects in tunnels based on acoustic wave characteristic signals. However, the application of this method is limited to tunnels and requires the acquisition of tunnel wall reflection signals. If acoustic wave reflection cannot be obtained in an open outdoor environment, the method will fail.

[0007] Literature CN111626207B proposes a method and system for detecting foreign object intrusion in front of a train based on image processing. This method involves installing a monitoring camera in front of the train and determining the presence of foreign objects by comparing the similarity between real-time images from the camera and images without foreign object intrusion. This method is suitable for detecting relatively simple track conditions in front of the train and can detect large stones, sky lanterns, and other foreign objects. However, it is less effective at detecting smaller objects, and is prone to missed detections and false positives. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a deep learning-based method for detecting foreign objects in a pantograph. This method requires only the acquisition of a small number of pantograph sample images under normal conditions, eliminating the need for extensive sample accumulation. A neural network model can be quickly trained based on these normal samples, thus mitigating spatial distance loss. Correlation loss By using the loss function for optimization iteration, a deep learning model with reasonable parameters is obtained. This model can accurately identify anomalies in pantograph images and can accurately detect small foreign objects. It features fast training speed, high recognition accuracy, and low false detection rate, effectively improving the safety and economy of train operation and maintenance.

[0009] The technical solution is as follows:

[0010] A deep learning-based method for foreign object detection in pantographs involves training a pantograph foreign object detection model using the following steps:

[0011] S1. Acquire multiple pantograph images under normal conditions and preprocess each image separately;

[0012] The preprocessing includes: extracting the region of interest (ROI) of the carbon skateboard from a single pantograph image, scaling the ROI to a preset size, and dividing the ROI into M local regions along the length of the carbon skateboard.

[0013] S2. Divide a single local region into N equal blocks, perform convolution processing in the convolutional layer of the deep learning model, and obtain the feature vector corresponding to each block.

[0014] The fully connected layer concatenates and transforms the dimensions of each feature vector, outputs the three-dimensional feature vector corresponding to the local region, and records it as spatial coordinates.

[0015] Traverse each local region and use the same processing to obtain each spatial coordinate;

[0016] S3. Calculate the loss function

[0017]

[0018] The spatial distance loss is denoted as: , representing the sum of the Euclidean distances between each spatial coordinate and the preset spatial coordinate origin;

[0019] Correlation loss, denoted as: , represents the sum of the correlation coefficients corresponding to all local regions; where the correlation coefficient of a single local region is: traversing each block region, taking the currently traversed block region as the center region, calculating the difference between the feature vectors of each neighboring block region in the P neighborhood and the center region, and summing all the differences to record as the correlation coefficient of the local region.

[0020] S4. Determine the loss function If the value is less than the threshold A, then save the network parameters in the current model and complete the model training.

[0021] If not, then use the loss function As the objective function, iteratively solve using optimization methods. The value is less than threshold A or Find the deep learning model corresponding to the minimum value, save the network parameters of the model, and complete the model training;

[0022] The following steps are used to perform foreign object detection on the actually acquired pantograph images using a trained deep learning model:

[0023] The actual pantograph images are preprocessed to obtain M local regions. Each local region is then input into a trained deep learning model. The model outputs M spatial coordinates corresponding to the pantograph image. The Euclidean distance between each spatial coordinate and the preset spatial coordinate origin is calculated. If each Euclidean distance value is less than the preset threshold R, the pantograph is not obstructed by foreign objects; otherwise, it is obstructed.

[0024] Furthermore, the preset spatial coordinate origin is any pre-set three-dimensional coordinate point;

[0025] The preset threshold R is set to the square root of the mean of the shorter and longer sides of a single segmented region.

[0026] Alternatively: The preset threshold value R is 30~100mm;

[0027] Threshold A = h × M × G × preset threshold R, where h represents the scaling factor, which takes a value of 1.5 to 3, M represents the total number of local regions in the region of interest of a single carbon skateboard, and G represents the total number of pantograph images obtained in the normal state in step S1.

[0028] Preferably, the camera perspective used to acquire multiple pantograph images under normal conditions in step S1 is consistent with the camera perspective used to acquire pantograph images during actual foreign object detection.

[0029] Preferably, the camera used to capture pantograph images is mounted on the roof of the train.

[0030] The region of interest for the carbon skateboard contains only a single carbon skateboard.

[0031] Preferably, in step S1, 15 to 100 pantograph images under normal conditions are acquired, including pantographs of different models; the pantograph images are rotated and scaled to increase the number of samples, and then each image is preprocessed.

[0032] Preferably, M, N, and P are preset values, with 2≤M≤12, 16≤N≤128, and 8≤P≤32.

[0033] Preferably, the longer side of the local region is divisible by the shorter side. In step S2, the single local region is divided into N equal sub-regions using the following method:

[0034] Draw C dividing lines perpendicular to the long side of the local region, dividing the local region into C square sub-image regions; C = long side size / short side size;

[0035] Each square sub-image region is then divided into multiple square blocks.

[0036] Furthermore, the deep learning model is the VGG convolutional neural network model, the MobileNet lightweight convolutional neural network, or the ResNets residual network;

[0037] In step S1, the method for extracting the region of interest of the carbon skateboard is either a grayscale template matching method based on NCC or a template matching method based on edge gradient;

[0038] In step S4, the optimization method is either gradient descent or Newton's method.

[0039] Furthermore, spatial distance loss The calculation method is as follows:

[0040]

[0041] in, This represents the spatial coordinates obtained from the i-th local region. The origin of the predefined spatial coordinates is represented by G, which represents the total number of pantograph images obtained in step S1 under normal conditions. The operators... This indicates the search for the L2 norm.

[0042] Furthermore, correlation loss The calculation method is as follows:

[0043] This indicates that the j-th block region in the i-th local region corresponds to the feature vector. Indicated by The feature vector corresponding to the k-th block region within the P-neighborhood of the region, G represents the total number of pantograph images obtained in step S1 under normal conditions, and the operator... This indicates the search for the L2 norm.

[0044] Compared with existing technologies, the present invention has the following characteristics:

[0045] 1) This method utilizes pantograph sample images under normal conditions, which have high imaging repeatability. Since there are no foreign objects in the local regions used for training, each local image should possess high similarity and consistency. This method abstracts this attribute into spatial distance loss. The model outputs three feature values, defined as spatial coordinates X, Y, and Z. The feature values ​​corresponding to normal samples should converge within a spatial sphere centered at the origin. The spatial distance loss is calculated by summing the Euclidean distances between each spatial coordinate and the preset origin. It participates in the optimization iteration to make the model parameters reasonable;

[0046] Furthermore, the features of each block region in a local image possess correlation with image features; this method abstracts this property into a correlation loss. The model's convolutional layers perform convolutions based on the P-neighborhood of the central region, traversing each region. Using the currently traversed region as the center, the differences between the feature vectors of each neighboring region in the P-neighborhood and the central region are calculated. All these differences are summed and recorded as the correlation coefficient for that local region. The sum of the correlation coefficients for all local regions is recorded as the correlation loss. It participates in the optimization iteration to further optimize the model parameters.

[0047] 2) This method only requires imaging data of the pantograph area, without the need for laser, acoustic or other sensor signals; it only requires a small number of pantograph sample images under normal conditions, without the need for a large number of sample accumulations. The deep learning models in existing methods usually require tens of thousands of training images to accurately identify defects and foreign objects. This method only requires dozens of image samples under normal conditions as training images, without the need for a large number of sample collections. It can still effectively identify foreign object images when there are few foreign object intrusion image samples. It has short training time, strong robustness, high feasibility and high detection accuracy. Attached Figure Description

[0048] Figure 1 This is the original image of the pantograph under normal conditions in the specific implementation embodiment;

[0049] Figure 2 The image shows the original pantograph with a foreign object (glove) present in the specific implementation embodiment;

[0050] Figure 3 The image shows the original pantograph with a foreign object (screwdriver) present in a specific embodiment.

[0051] Figure 4 This is a schematic diagram of the region of interest and a local area of ​​the carbon skateboard in a specific implementation embodiment;

[0052] Figure 5 This is a schematic diagram illustrating the structure of a local region processed in a deep learning model in a specific implementation method.

[0053] Figure 6 This is a schematic diagram of the central region and its 20-neighborhood when calculating the correlation loss in a specific implementation. Detailed Implementation

[0054] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] A deep learning-based method for foreign object detection in pantographs involves training a pantograph foreign object detection model using the following steps:

[0056] S1. Acquire multiple pantograph images under normal conditions and preprocess each image separately;

[0057] Preprocessing includes: extracting the region of interest (ROI) of the carbon slide plate from a single pantograph image, scaling the ROI to a preset size, and dividing the ROI into M local regions along the length of the carbon slide plate, where 2 ≤ M ≤ 12.

[0058] In this embodiment, when processing images, such as Figure 4 As shown, the region of interest for the carbon skateboard contains only a single carbon skateboard, M=5;

[0059] As a preferred implementation, 15 to 100 pantograph images under normal conditions are acquired, including pantographs of different models; the pantograph images are rotated and scaled to increase the number of samples, and then each image is preprocessed.

[0060] The methods for extracting the region of interest (ROI) of the carbon skateboard are: a grayscale template matching method based on NCC or a template matching method based on edge gradient;

[0061] S2, such as Figure 5As shown, a single local region is divided into N equal blocks, and convolution is performed in the convolutional layer of the deep learning model to obtain the feature vector corresponding to each block; where 16≤N≤128;

[0062] The fully connected layer concatenates and transforms the dimensions of each feature vector, outputs the three-dimensional feature vector corresponding to the local region (converted into a 3D Cartesian coordinate system spatial feature), and records it as spatial coordinates;

[0063] Traverse each local region and use the same processing to obtain each spatial coordinate;

[0064] S3. Calculate the loss function

[0065]

[0066] The spatial distance loss is denoted as: , representing the sum of the Euclidean distances between each spatial coordinate and the preset spatial coordinate origin;

[0067] Because pantograph sample images under normal conditions have high image repeatability, and there are no foreign objects in the local regions used for training, each local image should have high similarity and consistency. This method abstracts this attribute into spatial distance loss. The model outputs three feature values, which are defined as spatial coordinates X, Y, and Z, as follows: Figure 5 As shown, the feature values ​​corresponding to normal samples should converge inside a spatial sphere centered at the origin of the spatial coordinate system (e.g., setting a radius threshold R=50 for the origin (0, 0, 0)). The spatial distance loss is calculated by summing the Euclidean distances between each spatial coordinate and the preset origin. It participates in the optimization iteration to make the model parameters reasonable;

[0068] More specifically, the correlation loss is denoted as: , representing the sum of correlation coefficients corresponding to all local regions; where the correlation coefficient of a single local region is: traversing each block region, taking the currently traversed block region as the center region, calculating the difference between the feature vectors of each neighboring block region in the P-neighborhood and the center region, and summing all the differences to record as the correlation coefficient of the local region; 8≤P≤32; in this embodiment, as Figure 6 As shown, P=20.

[0069] Spatial distance loss The calculation method is as follows:

[0070]

[0071] in, This represents the spatial coordinates obtained from the i-th local region. The origin of the predefined spatial coordinates is represented by G, which represents the total number of pantograph images obtained in step S1 under normal conditions (in this embodiment, G=30 images, M=5). The operators... This indicates the search for the L2 norm.

[0072] Correlation loss The calculation method is as follows:

[0073] This indicates that the j-th block region in the i-th local region corresponds to the feature vector. Indicated by The feature vector corresponding to the k-th block region within the P-neighborhood of the region, G represents the total number of pantograph images obtained in step S1 under normal conditions, and the operator... This indicates the search for the L2 norm.

[0074] Since the features of different regions in a local image are correlated with the image features, this method abstracts this property into a correlation loss. The model's convolutional layers perform convolutions based on the P-neighborhood of the central region, traversing each region. Using the currently traversed region as the center, the differences between the feature vectors of each neighboring region in the P-neighborhood and the central region are calculated. All these differences are summed and recorded as the correlation coefficient for that local region. The sum of the correlation coefficients for all local regions is recorded as the correlation loss. Participate in the optimization iteration to further optimize the model parameters.

[0075] S4. Determine the loss function If the value is less than the threshold A, then save the network parameters in the current model and complete the model training.

[0076] If not, then use the loss function As the objective function, iteratively solve for it using optimization methods (gradient descent or Newton's method). The value is less than threshold A or Find the deep learning model corresponding to the minimum value, save the network parameters of the model, and complete the model training.

[0077] It should be noted that if at the end of the iteration, If the minimum value is greater than the threshold A, then the values ​​of M and N need to be increased, or the number of pantograph images under normal conditions needs to be increased in S1; then S1 needs to be repeated and the model needs to be trained again.

[0078] According to the conventional training process of deep learning models, the network parameter weights W of the model are pre-set before training begins.k and bias value b k The initial value of k is k=1~t, where t is determined by the kernel function, the number of layers in the convolutional neural network, and the size of the convolutional kernel. The preset initial value can be set to a random number or other pre-training empirical value.

[0079] During training, the deviation between the deep learning model's output and the expected result is calculated through forward propagation, and the parameter weights W in S1 are iteratively optimized through backpropagation. k and bias value b k Save the optimized model parameter weight values ​​W. k and bias value b k Complete model training.

[0080] The above steps describe the training process of the deep learning model for foreign object detection in pantographs. After the model training is complete, the trained deep learning model is used to detect foreign objects in the actual pantograph images according to the following steps:

[0081] The actual pantograph images are preprocessed to obtain M local regions. Each local region is then input into a trained deep learning model. The model outputs M spatial coordinates corresponding to the pantograph image. The Euclidean distance between each spatial coordinate and the preset spatial coordinate origin is calculated. If each Euclidean distance value is less than the preset threshold R, the pantograph is not obstructed by foreign objects; otherwise, it is obstructed.

[0082] The preset origin of the spatial coordinates is any three-dimensional coordinate point that has been set in advance. In order to facilitate the calculation, the origin of the spatial coordinates is set to (0,0,0) in this embodiment. In actual use, it can also be set to (1,1,1), (1,1,2), etc. Once the origin of the spatial coordinates is set, it remains unchanged during the training process and during actual detection.

[0083] In practice, the preset threshold R is set to the square root of the mean of the shorter and longer sides of a single segmented region. Alternatively: The preset threshold R is set to an empirical value of 30~100mm;

[0084] Threshold A = h × M × G × preset threshold R, where h represents the scaling factor, which takes a value of 1.5 to 3, M represents the total number of local regions in the region of interest of a single carbon skateboard, and G represents the total number of pantograph images obtained in the normal state in step S1.

[0085] In this embodiment, the pantograph visual inspection device is installed in an inspection shed at the track throat near the train depot. The device is positioned above the train roof and is used to capture images of the pantograph on the roof of the train before departure / stop. Figure 1 Images of the pantograph under normal conditions and without foreign objects. Figure 2The pantograph image shows an abnormal state where a glove left behind by a staff member indicates the presence of a foreign object. Figure 3 The pantograph image shows an abnormal state where a screwdriver left behind by a worker was present, indicating the presence of a foreign object.

[0086] As a preferred implementation, the camera perspective used to acquire multiple pantograph images under normal conditions in S1 is consistent with the camera perspective used to acquire pantograph images during actual foreign object detection.

[0087] If the longer side of a local region is divisible by its shorter side, in step S2, the single local region is divided into N equal sub-regions using the following method:

[0088] Draw C dividing lines perpendicular to the long side of the local region, dividing the local region into C square sub-image regions; C = long side size / short side size;

[0089] Each square sub-image region is then divided into multiple square blocks.

[0090] The deep learning model is either the VGG convolutional neural network model, the MobileNet lightweight convolutional neural network model, or the ResNets residual network model; such as VGG16, VGG19, and ResNet18. In this embodiment, the VGG16 convolutional network model is selected, which consists of 13 convolutional layers, 5 pooling layers, and 3 fully connected layers.

[0091] The foregoing description of specific exemplary embodiments of the present invention is for illustrative and descriptive purposes. It is not intended to be exhaustive, nor to limit the invention to the precise forms disclosed; obviously, many changes and variations are possible in accordance with the foregoing teachings. The exemplary embodiments were chosen and described to explain the specific principles of the invention and its practical application, thereby enabling others skilled in the art to implement and utilize various exemplary embodiments of the invention, as well as their different alternatives and modifications. The scope of the invention is intended to be defined by the appended claims and their equivalents.

Claims

1. A foreign object detection method for pantographs based on deep learning, characterized in that, The following steps are used to train the pantograph foreign object detection model: S1. Acquire multiple pantograph images under normal conditions and preprocess each image separately; The preprocessing includes: extracting the region of interest (ROI) of the carbon skateboard from a single pantograph image, scaling the ROI to a preset size, and dividing the ROI into M local regions along the length of the carbon skateboard. S2. Divide a single local region into N equal blocks, perform convolution processing in the convolutional layer of the deep learning model, and obtain the feature vector corresponding to each block. The fully connected layer concatenates and transforms the dimensions of each feature vector, outputs the three-dimensional feature vector corresponding to the local region, and records it as spatial coordinates. Traverse each local region and use the same processing to obtain each spatial coordinate; S3. Calculate the loss function. total loss total =loss center +loss relation The spatial distance loss is denoted as: loss center , representing the sum of the Euclidean distances between each spatial coordinate and the preset spatial coordinate origin; Correlation loss, denoted as: loss relation , represents the sum of the correlation coefficients corresponding to all local regions; where the correlation coefficient of a single local region is: traversing each block region, taking the currently traversed block region as the center region, calculating the difference between the feature vectors of each neighboring block region in the P neighborhood and the center region, and summing all the differences to record as the correlation coefficient of the local region. S4. Determine the loss function. total If the value is less than the threshold A, then save the network parameters in the current model and complete the model training. If not, then use the loss function. total As the objective function, the loss is obtained iteratively using optimization methods. total The value is less than the threshold A or the loss total Find the deep learning model corresponding to the minimum value, save the network parameters of the model, and complete the model training; The following steps are used to perform foreign object detection on the actually acquired pantograph images using a trained deep learning model: The actual pantograph images are preprocessed to obtain M local regions. Each local region is then input into a trained deep learning model. The model outputs M spatial coordinates corresponding to the pantograph image. The Euclidean distance between each spatial coordinate and the preset spatial coordinate origin is calculated. If each Euclidean distance value is less than the preset threshold R, the pantograph is not obstructed by foreign objects; otherwise, it is obstructed.

2. The deep learning-based foreign object detection method for pantographs as described in claim 1, characterized in that: The preset spatial coordinate origin is any pre-set three-dimensional coordinate point; The preset threshold R is set to the square root of the mean of the short and long sides of a single segmented region; Alternatively: The preset threshold value R is 30-100mm; Threshold A = h × M × G × preset threshold R, where h represents the scaling factor, which takes a value of 1.5 to 3, M represents the total number of local regions in the region of interest of a single carbon skateboard, and G represents the total number of pantograph images obtained in the normal state in step S1.

3. The deep learning-based foreign object detection method for pantographs as described in claim 1, characterized in that: In step S1, the camera perspective used to acquire multiple pantograph images under normal conditions is consistent with the camera perspective used when acquiring pantograph images during actual foreign object detection.

4. The deep learning-based foreign object detection method for pantographs as described in claim 1 or 3, characterized in that: The camera used to capture images of the pantograph is mounted on the roof of the train. The region of interest for the carbon skateboard contains only a single carbon skateboard.

5. The deep learning-based foreign object detection method for pantographs as described in claim 1, characterized in that: In step S1, 15 to 100 pantograph images under normal conditions are acquired, including pantographs of different models; the pantograph images are rotated and scaled to increase the number of samples, and then each image is preprocessed.

6. The deep learning-based foreign object detection method for pantographs as described in claim 1, characterized in that: M, N, and P are preset values, where 2≤M≤12, 16≤N≤128, and 8≤P≤32.

7. The deep learning-based foreign object detection method for pantographs as described in claim 1, characterized in that: The longer side of the local region is divisible by the shorter side. In step S2, the single local region is divided into N equal sub-regions using the following method: Draw C dividing lines perpendicular to the long side of the local region, dividing the local region into C square sub-image regions; C = long side size / short side size; Each square sub-image region is then divided into multiple square blocks.

8. The deep learning-based foreign object detection method for pantographs as described in claim 1, characterized in that: The deep learning model is the VGG convolutional neural network model, the MobileNet lightweight convolutional neural network, or the ResNets residual network; In step S1, the method for extracting the region of interest of the carbon skateboard is either a grayscale template matching method based on NCC or a template matching method based on edge gradient; In step S4, the optimization method is either gradient descent or Newton's method.

9. The foreign object detection method for pantographs based on deep learning as described in claim 1, characterized in that: Spatial distance loss center The calculation method is as follows: Among them, (x i ,y i ,z i () represents the spatial coordinates obtained from the i-th local region, (X0,Y0,Z0) represents the predefined origin of the spatial coordinates, G represents the total number of pantograph images obtained in the normal state in step S1, and the operator ‖‖2 represents the calculation of the L2 norm.

10. The deep learning-based foreign object detection method for pantographs as described in claim 1, characterized in that: correlation loss relation The calculation method is as follows: Among them, V ij V represents the feature vector of the j-th block region in the i-th local region. ijk Indicates V ij Let G be the feature vector corresponding to the kth block region within the P neighborhood of the central region, G represents the total number of pantograph images obtained in the normal state in step S1, and the operator ‖‖2 represents the calculation of the L2 norm.

Images