Method and device for detecting electric poles and pole numbers along railway lines based on deep learning

Abstract

Embodiments of the present invention disclose a method and device for detecting utility poles and pole numbers along railway lines based on deep learning. The method comprises: obtaining an image of a railway scene to be detected, wherein the image of the railway scene to be detected includes utility poles; performing target detection on the image of the railway scene to be detected using a first target prediction network to obtain detection results of utility poles and pole number areas, wherein the pole number area includes a utility pole number plate and a background area surrounding the utility pole number plate; based on the detection results of the pole number area, segmenting a pole number area image from the image of the railway scene to be detected; performing target detection on the pole number area image using a second target prediction network to obtain detection results of utility pole numbers. Based on this method and device, the detection efficiency and accuracy of utility poles and pole numbers in railway scene images can be improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, electronic device, and storage medium for detecting utility poles and pole numbers along railways based on deep learning. Background Art

[0002] In high-speed rail power systems, the utility poles on the tracks are used to transmit power to electric locomotives and form part of the high-speed rail catenary. The high-speed rail catenary is the transmission line running overhead along the railway line, supplying power to the locomotives. The current used by high-speed trains is transmitted through the catenary above the locomotives. A power outage in the catenary, or poor contact between the train's pantograph and the catenary, can impact the train's power supply. Therefore, the quality and working condition of the catenary directly impact the efficiency of high-speed trains. In my country's vast high-speed rail power system, a large number of utility poles require regular inspection and maintenance. Furthermore, the complex and remote locations of high-speed rail lines mean they may remain unmanned for extended periods.

[0003] To improve the management efficiency of utility poles along railway lines, some studies have employed traditional computer vision techniques to automatically detect and analyze utility poles along railway lines. In 2017, Xie Xingyang et al. used nondestructive testing (NDT) techniques to assess the health of utility poles and proposed a novel NDT system based on narrowband frequency excitation and advanced signal processing techniques. To improve the accuracy of damage detection, this system employed frequency response functions and principal component analysis to extract signal features from single-frequency stress waves, thereby assessing the health of utility poles. In 2017, Li Tao et al. addressed the issue of partially occluded areas of utility poles in surveillance videos. Using bilateral filtering and HSV gradients to improve LSD line detection, they proposed an improved line segmentation detection algorithm. This method focuses on detecting local areas of utility poles to address the issue of missing information from occluded areas in surveillance videos, rather than inspecting the entire pole. In 2019, Jiang Nan et al. studied methods for detecting damage to high-altitude utility poles and proposed a method based on insulators. The authors believe that the damage of high-altitude utility poles is primarily determined by the condition of their insulators, making insulator defects a key indicator for high-altitude utility pole damage detection. This method, based on traditional edge detection algorithms and ellipse fitting, determines insulator damage based on fitting errors, making it poorly suited for existing large-scale data detection tasks.

[0004] However, target detection algorithms based on traditional computer vision technology mainly rely on manual acquisition of feature information about the target, which requires huge labor costs and cannot meet the requirements of accuracy and speed. Summary of the Invention

[0005] An object of embodiments of the present invention is to solve at least the above problems and / or disadvantages and to provide at least the advantages to be described hereinafter.

[0006] The embodiments of the present invention provide a method and device for detecting utility poles and pole numbers along railways based on deep learning, which can improve the detection efficiency and detection accuracy of utility poles and pole numbers along railways.

[0007] In the first aspect, a method for detecting utility poles and pole numbers along railway lines based on deep learning is provided, comprising:

[0008] Acquire an image of a scene along a railway to be detected, wherein the image of the scene along the railway to be detected includes electric poles;

[0009] Performing target detection on the scene image along the railway to be detected by a first target prediction network to obtain detection results of utility poles and pole number areas, wherein the pole number areas include utility pole number plates and background areas surrounding the utility pole number plates;

[0010] Segmenting a pole number area image from the scene image along the railway to be detected according to the detection result of the pole number area;

[0011] Target detection is performed on the pole number area image through a second target prediction network to obtain a detection result of the pole number of the utility pole.

[0012] Optionally, the area of ​​the pole number region is 2 to 10 times the area of ​​the telephone pole number plate.

[0013] Optionally, the width of the pole number area is 2 to 4 times the width of the telephone pole number plate, and the height of the pole number area is consistent with the height of the telephone pole number plate.

[0014] Optionally, the anchor box in the first object detection network is determined by:

[0015] Label the electric poles and pole number regions in multiple railway scene images to obtain multiple true bounding boxes of electric poles and multiple true bounding boxes of pole number regions;

[0016] Clustering the plurality of utility pole true bounding boxes and the plurality of pole number area true bounding boxes to obtain a plurality of initial anchor boxes;

[0017] The sizes of the multiple initial anchor boxes are scaled based on a linear scale to obtain multiple anchor boxes.

[0018] Optionally, scaling the multiple initial anchor frames based on a linear scale to obtain multiple anchor frames includes:

[0019] sorting the multiple initial anchor boxes according to their widths or heights;

[0020] The multiple initial anchor frames are scaled based on the following formula to obtain multiple anchor frames:

[0021] w′1=αw1,

[0022] w′ N =βw N ,

[0023]

[0024]

[0025] Among them, w1, w N Represent the width of the first and Nth initial anchor boxes respectively, w′1 represents the width of the first anchor box corresponding to the first initial anchor box after scaling, and w′ N represents the width of the Nth anchor box corresponding to the Nth initial anchor box after scaling, w n , h n Represents the width and height of the nth initial anchor box, w′ n , h′ n They represent the width and height of the nth anchor box corresponding to the nth initial anchor box after scaling, n = 1, 2, ..., N, where N is the number of initial anchor boxes; α and β are scaling coefficients, and their values ​​range from [0, 1].

[0026] Optionally, the first target detection network is a YOLOv3-tiny network; and / or the second target detection network is a YOLOv3-tiny network.

[0027] Optionally, the detection result of the utility pole includes a confidence level of the utility pole detection frame; and obtaining the scene image along the railway to be detected includes:

[0028] Obtain a video of the scene along the railway to be detected;

[0029] Splitting the video of the scene along the railway to be detected into a plurality of continuous images of the scene along the railway to be detected;

[0030] After performing target detection on the pole number area image by the second target prediction network to obtain a detection result of the pole number of the utility pole, the method further includes:

[0031] Dividing the detected railway scene images corresponding to the same electric pole number among the multiple detected railway scene images into a group;

[0032] According to the confidence of the utility pole detection frames of several detected railway scene images in each group, at least one detected railway scene image is selected and stored as the target image of the utility pole corresponding to the current group of detected railway scene images, and the other detected railway scene images in the current group of detected railway scene images are deleted.

[0033] In the second aspect, a device for detecting utility poles and pole numbers along railway lines based on deep learning is provided, comprising:

[0034] An image acquisition module is used to acquire an image of a scene along the railway to be detected, wherein the image of the scene along the railway to be detected includes electric poles;

[0035] a first target detection module, configured to perform target detection on the scene image along the railway to be detected using a first target prediction network, and obtain detection results of utility poles and pole number areas, wherein the pole number areas include utility pole number plates and a background area surrounding the utility pole number plates;

[0036] An image segmentation module is used to segment the pole number area image from the scene image along the railway to be detected according to the detection result of the pole number area;

[0037] The second target detection module is used to perform target detection on the pole number area image through a second target prediction network to obtain a detection result of the pole number of the utility pole.

[0038] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the described method.

[0039] In a fourth aspect, a storage medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, the method described is implemented.

[0040] The embodiments of the present invention have at least the following beneficial effects:

[0041] The embodiments of the present invention provide a method, device, electronic device, and storage medium for detecting utility poles and pole numbers along railways based on deep learning. The method first obtains a scene image along the railway to be detected, wherein the scene image along the railway to be detected includes utility poles, performs target detection on the scene image along the railway to be detected through a first target prediction network, and obtains detection results of utility poles and pole number areas, wherein the pole number area includes the utility pole number plate and the background area around the utility pole number plate. Then, based on the detection results of the pole number area, a pole number area image is segmented from the scene image along the railway to be detected, and finally, target detection is performed on the pole number area image through a second target prediction network to obtain detection results of utility pole numbers. Based on this method and device, the detection efficiency and detection accuracy of utility poles and pole numbers in scene images along railways can be improved.

[0042] Other advantages, objectives, and features of the embodiments of the present invention will be reflected in part through the following description, and in part will be understood by those skilled in the art through study and practice of the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a method for detecting utility poles and pole numbers along a railway line based on deep learning, provided in one embodiment of the present invention;

[0044] Figure 2 A schematic diagram of the structure of a YOLOv3-tiny network provided in one embodiment of the present invention;

[0045] Figure 3a A portion of images in the first-stage data set provided by another embodiment of the present invention; Figure 3b A portion of images in a second-stage data set provided by another embodiment of the present invention;

[0046] Figure 4 It is a visualization image of the detection results of the experimental scheme (1);

[0047] Figure 5a This is a visualization image of the test results of the first stage in experimental scheme (2); Figure 5b This is a visualization image of the detection results of the second stage in experimental scheme (2);

[0048] Figure 6 This is a visualization image of the test results of the second stage in the experimental scheme (3);

[0049] Figure 7 This is a visualization image of the test results of the first stage in the experimental scheme (4);

[0050] Figure 8aA visual image of the detection results of the first stage in the test scheme (5) provided in another embodiment of the present invention; Figure 8b A visual image of the detection results of the second stage in the test scheme (5) provided in another embodiment of the present invention;

[0051] Figure 9 A schematic diagram of the structure of a device for detecting utility poles and pole numbers along railways based on deep learning, provided by one embodiment of the present invention;

[0052] Figure 10 A schematic structural diagram of an electronic device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0054] In the scene images along the railway, the sizes of the telephone poles and the pole numbers of the telephone poles are quite different, and the pixels of the scene images along the railway are relatively low. Therefore, the effect of detecting the telephone poles and the pole numbers of the telephone poles from the scene images along the railway at one time is poor, and it is difficult to meet the requirements for detection accuracy. Based on this, in the target detection process of the scene images along the railway to be detected, the target detection task is divided into two stages. In the first stage, the scene images along the railway to be detected are detected for the telephone poles and pole number areas. Then, based on the detection results for the pole number area obtained in the first stage, the scene images along the railway to be detected are segmented to obtain the pole number area image. Finally, the pole number area image is detected for the telephone poles. The two-stage target detection method provided in this embodiment can realize the detection of large target telephone poles and small target telephone pole numbers, and the detection accuracy of both targets can meet the requirements.

[0055] Figure 1 The flowchart of the method for detecting electric poles and pole numbers along the railway line based on deep learning provided by the embodiment of the present invention is executed by a system with processing capabilities, a server device or an electric pole and pole number detection device along the railway line based on deep learning. Figure 1 As shown, the method includes:

[0056] Step 110: Acquire a scene image along the railway to be detected, wherein the scene image along the railway to be detected includes electric poles.

[0057] Here, the railway scene images may be images captured using an image capture device such as a camera. In some embodiments, acquiring the railway scene images to be detected includes: acquiring a video of the railway scene to be detected; and splitting the video of the railway scene to be detected into a plurality of consecutive railway scene images to be detected. A video capture device such as a camera may be installed on a train. As the train travels along the railway, the video capture device captures the railway scene video, and then splits the railway scene video into a plurality of railway scene lines.

[0058] Step 120: Perform target detection on the scene image along the railway to be detected through a first target prediction network to obtain detection results of telephone poles and pole number areas, wherein the pole number area includes telephone pole number plates and a background area around the telephone pole number plates.

[0059] In practice, each utility pole has a corresponding pole number plate, typically fixed to the top of the pole. The pole number plate is painted with the pole number. The pole number consists of several digits, typically three or four, ranging from 0 to 9.

[0060] In railway scene images, there are certain size differences between utility poles and pole number plates. Furthermore, some railway scene images have low brightness, making the pole number plates less distinct. When the pole number area is configured to consist solely of the pole number plates, excluding any background area, some pole number areas may not be detected, resulting in the corresponding pole numbers also being undetectable. Therefore, the pole number area is designed to include both the pole number plates and the surrounding background area. This background area provides richer image features, facilitating object detection within the pole number area.

[0061] In some embodiments, the area of ​​the pole number region is 2 to 10 times the area of ​​the pole number plate. This is because increasing the area of ​​the pole number region is equivalent to expanding the background area surrounding the pole number plate, thereby utilizing the background area around the pole number plate to provide richer image features, thereby improving the detection accuracy of the pole number region. However, if the area of ​​the pole number region is too large, the size difference between the pole number region as a whole and the pole number contained therein will be too large, which will affect the detection accuracy of the pole number from the pole number region image.

[0062] In railway scene images, the background area at the height of the telephone pole number plate is typically a utility pole, and its image features are relatively simple. However, the background area at the width of the telephone pole number plate is more likely to provide rich image features, thereby helping to improve the detection accuracy of the pole number area. In some examples, the width of the pole number area is 2 to 4 times the width of the telephone pole number plate, and the height of the pole number area is consistent with the height of the telephone pole number plate.

[0063] In this step, the first target prediction network uses a neural network capable of target detection. Deep learning-based target detection algorithms are primarily divided into two categories: two-stage networks, represented by Faster RCNN, and one-stage networks, represented by the YOLO series of networks. Two-stage target detection algorithms, such as the R-CNN (Regions with Convolutional Neural Network Features) series, offer high detection accuracy but are time-consuming and cannot be used for real-time detection. One-stage target detection algorithms, such as the SSD (Single Shot Multibox Detector) algorithm and the YOLO (You Only Look Once) series, offer the advantage of being end-to-end: an image is input to the model, and the model directly outputs the location and category of the target within the image. Therefore, compared to two-stage target detection models, one-stage target detection models significantly reduce time loss while maintaining detection accuracy, meeting the requirements of real-time detection and making them more suitable for industrial detection applications. Preferably, in order to achieve real-time detection of utility poles and utility pole numbers, the embodiment of the present invention adopts a one-stage target detection model to implement the first target detection network. For example, the YOLO network can be used to implement the first target detection network.

[0064] In a typical object detector, features learned by the convolutional layer are passed to a classifier or regressor for prediction, resulting in bounding box coordinates, category labels, and other information. The YOLO series of networks uses features learned by deep convolutional neural networks to detect objects. Within the YOLO series of networks, YOLOv3 utilizes multi-scale feature detection, encompassing three scales: 13×13, 26×26, and 52×52. Feature maps with larger scales are more effective at detecting relatively small objects. Based on these three feature map sizes, object detection at various scales is possible.

[0065] YOLOv3 is a fully convolutional network that uses convolutional layers to make predictions. The size of the convolution kernel of the last convolutional layer is: 1×1×(k×(5+C)), where 1×1 is the size of the convolution kernel, k represents the number of bounding boxes that can be predicted for each grid cell (usually 3 by default), and each bounding box has 5+C attributes (coordinates (t x ,t y ,t w ,t h ), confidence, and category probabilities of C categories). The prediction process uses (5+C)×k 1×1 convolution kernels for convolution prediction to achieve the purpose of improving recognition accuracy and efficiency. Since 1×1 convolution is used, the size of the predicted feature map does not change. During detection, bounding boxes with scores below the threshold are first ignored, and then the remaining bounding boxes are filtered according to non-maximum suppression (NMS), that is, the bounding boxes are sorted according to the scores, and then the IOU value of the bounding box with the highest score and other bounding boxes is calculated. The IOU value is used to measure the repetition of the bounding box, and the bounding boxes with high repetition are filtered out. Finally, the remaining bounding boxes are used as the detection results. The loss function loss(object) of YOLOv3 is shown in formula (1), which mainly consists of three parts: the first part (lbox) is the coordinate error generated by predicting the position of the target bounding box, the second part (lobj) is the error generated by predicting the target confidence, and the third part (lcls) is the error generated by predicting the category.

[0066] loss(object)=lbox-lobj-lcls (1)

[0067]

[0068]

[0069]

[0070] Where K×K is the number of all grid cells in the input image, which can be set to 13×13, 26×26, or 52×52 in YOLOv3. M is the number of bounding boxes contained in each grid cell. Used to determine whether the j-th bounding box in the i-th grid cell has an object to be detected. If so, it is 1, otherwise it is 0; The meaning is the opposite of this; Represents the central abscissa, ordinate, width, height, confidence, and category probability of the true bounding box in the i-th grid cell; x i ,yi ,w i ,h i ,C i ,p i (c) The center horizontal coordinate, vertical coordinate, width, height, confidence and category probability of the bounding box predicted for the i-th grid cell. Note that x i ,y i ,w i ,h i It is not the absolute position and size of the bounding box, but only the relative value relative to the center position of the grid cell and the size of the prior box.

[0071] The YOLOv3-tiny network is an improved network based on the YOLOv3 network. The biggest difference between it and the YOLOv3 network lies in the feature extraction network. Some feature layers of YOLOv3 are deleted, and only two prediction branches are retained. Specifically, Figure 2 The schematic diagram of the YOLOv3-tiny feature extraction network is shown. The YOLOv3-tiny backbone feature extraction network consists of six stacked convolutional and max-pooling layers, each consisting of one convolutional layer and one max-pooling layer. Four convolutional layers are connected after the sixth max-pooling layer. The output of the seventh convolutional layer is then upsampled, and the upsampled features are fused with the features from the fifth convolutional layer to form a concatenated layer. Further convolution operations produce a 26×26 output layer, and the features from the tenth convolutional layer are directly output to form a 13×13 output layer. As can be seen, the YOLOv3-tiny network has only two feature detection scales: 13×13 and 26×26. The YOLOv3-tiny network uses the same loss function as YOLOv3. During training, the optimal model is saved based on the lowest validation set loss within a set epoch range. YOLOv3-tiny is a lightweight model with advantages such as a simple network structure, low computational complexity, high speed, and the ability to run on mobile devices or devices, enabling real-time detection and making it suitable for industrial target detection. Therefore, to achieve the required speed and time consumption, this embodiment of the present invention preferably uses the YOLOv3-tiny network for detecting utility poles and pole number areas.

[0072] The embodiment of the present invention preferably adopts a first target detection network that introduces an anchor box mechanism. Specifically, anchor boxes are used in FasterRCNN to prevent the model from blindly searching for targets during training and detection, thereby accelerating model convergence. The YOLO series introduced the "anchor box" mechanism starting from YOLOv2, constraining the predicted target range, introducing size prior experience, and realizing multi-scale learning. YOLOv3 uses the k-means algorithm to cluster on the groundtruth boxes of all samples in the dataset to obtain anchor boxes. In the YOLOv3 network, the model effect is best when the number of anchor boxes is 9, and at the same time, 9 anchor boxes can meet YOLOv3's output requirements for three scales. In the YOLOv3-tiny network, there are only two scale outputs, so the number of anchor boxes is 6. Therefore, when the embodiment of the present invention adopts the YOLO series network as the first target detection network, the number of anchor boxes in the first target detection network can be set according to the situation of the YOLO series network.

[0073] Generally speaking, the ground-truth bounding boxes of the objects are annotated in the sample images in the training set, and the ground-truth bounding boxes are clustered. The shape and size of the anchor boxes are set according to the most common shapes and sizes. In railway scene images, there are only two types of objects: telephone poles and telephone pole numbers. The types of objects are few and the types are single. At the same time, the sizes of these two types of objects vary greatly. However, the size differences of each type of object in different scene images are small, and the size distribution is relatively concentrated. As a result, the size distribution of the anchor boxes obtained by clustering the telephone poles and pole numbers in railway scene images is relatively concentrated, which fails to reflect the advantages of multi-scale detection. In turn, the detection accuracy of the first object detection network for railway scene images is insufficient. Based on this, a linear scaling mechanism is introduced in the anchor box determination process to scale the size of the anchor boxes to increase the diversity of the anchor boxes and improve the detection effect of the first object detection network.

[0074] Specifically, the anchor frames in the first target detection network are determined by: labeling the utility poles and pole number areas in multiple railway scene images to obtain multiple true bounding boxes of utility poles and multiple true bounding boxes of pole number areas; clustering the multiple true bounding boxes of utility poles and the multiple true bounding boxes of pole number areas to obtain multiple initial anchor frames; and scaling the sizes of the multiple initial anchor frames based on a linear scale to obtain multiple anchor frames.

[0075] Furthermore, in order to maintain the distribution law of the initial anchor frames and make the distribution law of the scaled anchor frames basically unchanged, while changing the size of the initial anchor frames and increasing the diversity of the anchor frame sizes, the step of scaling the multiple initial anchor frames based on the linear scale to obtain multiple anchor frames includes: sorting the multiple initial anchor frames according to their widths or heights; and scaling the multiple initial anchor frames based on the following formula (2) to obtain multiple anchor frames:

[0076] w′1=αw1,

[0077] w′ N =βw N ,

[0078]

[0079]

[0080] Among them, w1, w N Represent the width of the first and Nth initial anchor boxes respectively, w′1 represents the width of the first anchor box corresponding to the first initial anchor box after scaling, and w′ N represents the width of the Nth anchor box corresponding to the Nth initial anchor box after scaling, w n , h n Represents the width and height of the nth initial anchor box, w′ n , h′ n where α and β represent the width and height of the nth anchor box after scaling, respectively, corresponding to the nth initial anchor box, where n = 1, 2, …, N, where N is the number of initial anchor boxes. α and β are scaling coefficients, ranging from 0 to 1. In some examples, α and β are set to 0.5.

[0081] It should be noted that when the embodiment of the present invention adopts the YOLOv3-tiny network as the first target detection network, the number of initial anchor frames and the number of anchor frames after linear scaling are both 6.

[0082] It should be understood that in the process of labeling the utility poles and pole number regions in multiple railway scene images to obtain multiple true bounding boxes for the utility poles and multiple true bounding boxes for the pole number regions, the railway scene images processed are those in the training set, which are different from the railway scene images to be detected. Furthermore, the true bounding boxes for the pole number regions include the pole number plates and the surrounding background area. In some embodiments, the area of ​​the true bounding boxes for the pole number regions is approximately 2 to 10 times the area of ​​the pole number plates.

[0083] Here, the detection results for utility poles can include the location and size of the utility pole detection frame in the image of the railway scene to be detected, the confidence level of the utility pole detection frame, the category, and the category probability. The detection results for pole number regions can include the location of the pole number region detection frame in the image of the railway scene to be detected, the confidence level of the pole number region detection frame, the category, and the category probability. The confidence level of a detection frame indicates the degree of confidence that the detection frame actually contains the target and that all features of the target are included.

[0084] Step 130 : Segmenting a pole number area image from the scene image along the railway to be detected according to the detection result of the pole number area.

[0085] Specifically, an image cropping tool may be used to crop the scene image along the railway to be detected, and separate the pole number area image from the scene image along the railway to be detected.

[0086] Step 140: Perform target detection on the pole number area image through a second target prediction network to obtain a detection result of the pole number of the utility pole.

[0087] The second target prediction network utilizes a neural network capable of target detection. Preferably, to achieve real-time detection of utility poles and pole numbers, embodiments of the present invention utilize a one-stage target detection model to implement the first target detection network. For example, a YOLO network can be used to implement the second target detection network. To meet speed and time requirements, a YOLOv3-tiny network is preferably used for pole number detection.

[0088] When using the YOLOv3-tiny network to detect the pole numbers in the pole number area image, the default anchor frame of the YOLOv3-tiny network can be used, and the number of anchor frames is 6. Here, the default anchor frame is set based on the detection target in the VOC dataset.

[0089] In some embodiments, the detection results of the utility poles and the detection results of the utility pole numbers can be automatically combined and visualized in the scene image along the railway where target detection has been achieved.

[0090] When a video capture device is used to capture a railway scene video, a large number of railway scene images can be split from the railway scene video, and multiple railway scene images can be obtained for the same electric pole. In the prior art, the railway scene video or all the split railway scene images may be stored, which takes up a large amount of storage space and has low data utilization efficiency. In some embodiments, after the pole number area image is subjected to target detection by the second target prediction network to obtain the detection result of the electric pole number, the method further includes: dividing the plurality of detected railway scene images corresponding to the same electric pole number into a group; selecting at least one detected railway scene image according to the confidence of the electric pole detection frame of the plurality of detected railway scene images in each group, and storing it as the target image of the electric pole corresponding to the current group of detected railway scene images, and deleting the other detected railway scene images in the current group of detected railway scene images.

[0091] Specifically, after completing target detection of utility poles and pole numbers in multiple railway scene images to be inspected, all detected railway scene images are divided into several groups based on the pole numbers matched to each image. Images within each group share the same pole number. For each group, images that clearly and completely display the poles are selected for storage, while the remaining images are deleted. This selection is based on the confidence levels of the pole detection frames in the images within the group, with one or more images from each group selected for storage. For example, a confidence threshold can be set, and images corresponding to pole detection frames exceeding the threshold can be selected. Alternatively, images can be sorted from highest to lowest confidence, with images corresponding to pole detection frames with the highest confidence levels selected. Furthermore, if only one railway scene image is available for a particular pole, the detected railway scene image can be directly stored.

[0092] In summary, an embodiment of the present invention provides a method for detecting utility poles and pole numbers along railways based on deep learning. First, an image of a scene along the railway to be detected is obtained, wherein the image of the scene along the railway to be detected includes utility poles. Target detection is performed on the image of the scene along the railway to be detected through a first target prediction network to obtain detection results of utility poles and pole number areas, wherein the pole number area includes a utility pole number plate and a background area around the utility pole number plate. Then, based on the detection results of the pole number area, a pole number area image is segmented from the image of the scene along the railway to be detected. Finally, target detection is performed on the pole number area image through a second target prediction network to obtain detection results of utility pole numbers. Based on this method, the detection efficiency and accuracy of utility poles and pole numbers in the image of the scene along the railway can be improved.

[0093] The following provides a specific implementation scenario to further illustrate the method for detecting utility poles and pole numbers along railways based on deep learning provided by an embodiment of the present invention.

[0094] The training and testing environment used an NVIDIA GeForce RTX 2060 GPU, an Intel i7-10875H processor, Windows 10 as the operating system, and Python 3.6 as the compilation environment. The model architecture was deployed on TensorFlow, and the deep neural network parameters were configured using Keras. CUDA 9.0 and CUDNN 7.0.4 were also installed to support GPU use.

[0095] To ensure data diversity, we collected images of railway scenes from 14 locations along two high-speed rail lines to construct a dataset. The images of railway scenes include electric poles. The training dataset was screened from 67,346 railway scene images. The final dataset for the first stage consisted of 11,040 images, of which 9,439 were used as training sets, 497 as validation sets, and 1,104 as test sets. LabelImg was used to annotate the images with the label format required by YOLO: {category 1, (x min ,y min ), (x max ,y max ), category 2, …}. The first-stage dataset, referred to as dataset-1, is used for the task of detecting objects in utility poles and pole number areas. Based on the first-stage dataset, the second-stage dataset was cropped using the coordinates of the target pole number areas. After screening, a total of 7,833 images were obtained, of which 6,697 were used as the training set, 353 as the validation set, and 783 as the test set. The pole number areas were also labeled. The second-stage dataset, referred to as dataset-2, is used for the task of detecting objects in utility poles and pole numbers.

[0096] Figure 3a and Figure 3b Partial images from the first and second phase datasets are shown. Because images of railway scenes along two high-speed rail lines were collected, the first phase dataset contains two distinct pole shapes. The pole number areas in the corresponding railway scene images also differ, as do the pole numbers, with four or three digits arranged vertically and three digits arranged horizontally.

[0097] This embodiment of the present invention uses the YOLOv3-tiny network with 6 anchor boxes. The first-stage target detection network performs a binary classification of the poles and pole number areas, while the second-stage target detection network performs a ten-class classification of the numbers contained in the pole number areas. Both use LabelImg to save the ground truth labeled boxes as XML files. The ground truth labeled categories and detection box coordinates of the targets are then extracted and saved as txt files for training, validation, and testing.

[0098] A small dataset (320 images) was constructed by selecting some images from the first-stage dataset and the second-stage dataset, and the anchor frame design scheme was tested on the small dataset. Three schemes were designed for the anchor frame in the embodiment of the present invention to prove that the detection accuracy of the first-stage target detection task can be greatly improved based on the anchor frame obtained by re-clustering and linear scaling. Among them, the anchor frame design scheme (1) uses the default anchor frame of the YOLOv3-tiny network in both the first and second-stage target detection tasks. The anchor frame design scheme (2) clusters the real bounding box of the telephone pole and the real bounding box of the pole number area based on the training set in the small dataset to obtain 6 re-clustered anchor frames. The first-stage target detection task uses the anchor frame obtained by the re-clustering for target detection, and the second-stage target detection task uses the default anchor frame of the YOLOv3-tiny network. Anchor frame design scheme (3) clusters the ground-truth bounding boxes of the utility poles and the pole number area based on the training set in the small dataset to obtain six initial anchor frames. These six initial anchor frames are then linearly scaled according to formula (2) to obtain six anchor frames, where the initial anchor frames are sorted according to their width, and α and β are set to 0.5. The first-stage target detection task uses the anchor frames obtained by re-clustering and linear scaling for target detection. The second-stage target detection task uses the default anchor frames of the YOLOv3-tiny network.

[0099] The average precision (mAP) and recall (Recall) are used as the measurement criteria. Based on the three anchor design schemes, the test data of the first stage object detection task is shown in Table 1.

[0100] Table 1 The impact of different anchor box design schemes on the performance of the first stage target detection network

[0101]

[0102] As can be seen from Table 1, the anchor frames in anchor frame design scheme (1) are the anchor frames obtained by clustering the YOLOv3-tiny network on the VOC dataset, which is not applicable to the railway scene dataset involved in the embodiment of the present invention. The main reason is that the target types of the samples in the VOC dataset are many, the sizes are different, and the size difference is large. The clustered anchor frames have different shapes and cannot accurately reflect the shape and size characteristics of the two types of targets, namely, the telephone poles and the pole number areas involved in the embodiment of the present invention. The anchor frames designed by anchor frame design scheme (2) have poor performance improvement for the YOLOv3-tiny network. After analysis, it may be that there are too few training data on the small dataset, and the shapes of the telephone poles in the small dataset are quite different, resulting in poor learning effect of the YOLOv3-tiny network. Based on the anchor frames obtained by anchor frame design scheme (3), the average accuracy of the YOLOv3-tiny network for the telephone poles and the pole number areas is improved by 16.74% and 59.59% respectively compared with the first two anchor frame design schemes. In the anchor frame design scheme (3), the re-clustered anchor frames are further linearly scaled, which can discretize the anchor frames with too concentrated size distribution and increase the diversity of anchor frame sizes, thereby improving the detection rate of large target utility poles and small target pole number areas.

[0103] Another specific implementation scenario is provided below to further illustrate the method for detecting utility poles and pole numbers along railways based on deep learning provided by an embodiment of the present invention.

[0104] The training and testing environment and data set of this embodiment of the present invention are the same as those of the previous embodiment. This embodiment provides five test scenarios to demonstrate that the two-stage target detection method provided by this embodiment of the present invention can detect both large target utility poles and small target utility pole numbers, and that the detection accuracy for both targets can meet the requirements. The implementation process of the five test scenarios is as follows:

[0105] Experimental plan (1) directly detects telephone poles and telephone pole numbers from railway scene images. Each digit of the telephone pole number is a number between 0 and 9. Therefore, the detection targets include telephone poles and 10 numbers, totaling 11 categories.

[0106] Experimental scheme (2) performs two-stage object detection. In the first stage, the utility poles are detected from the railway scene image; then, based on the detection results of the utility poles, the utility pole images are automatically cropped from the railway scene image; in the second stage, the utility pole numbers are detected from the utility pole images.

[0107] Experimental scheme (3) performs two-stage object detection. In the first stage, the utility poles are detected from the railway scene image; then the pole number area image is manually cropped from the railway scene image, and the pole number area contains the utility pole number plate; in the second stage, the pole number of the utility pole is detected from the pole number area image.

[0108] Experimental scheme (4) performs two-stage object detection. In the first stage, the utility poles and utility pole number plates are detected from the railway scene image. Then, based on the detection results of the utility pole number plates, the utility pole number plate images are automatically cropped from the railway scene image. In the second stage, the utility pole number plates are detected from the utility pole number plate images.

[0109] Experimental plan (5) performs two-stage target detection. In the first stage, the poles and pole number areas are detected from the railway scene image. The pole number area includes the pole number plate and the background area around the pole number plate. The area of ​​the pole number area is roughly 2 to 10 times the size of the pole number plate. Then, based on the detection results of the pole number area, the pole number area image is automatically cropped from the railway scene image. In the second stage, the pole number of the pole is detected from the pole number area image.

[0110] Experimental scheme (1) uses an object detection network implemented by the YOLOv3-tiny network. In experimental schemes (2) to (5), two target detection networks are used in the two stages respectively, and both target detection networks are implemented by the YOLOv3-tiny network. Experimental schemes (1) to (3) all use the default anchor box of the YOLOv3-tiny network; Experimental schemes (4) and (5) use anchor boxes obtained by re-clustering and linear scaling. Specifically, re-clustering is performed based on the training set of the first stage dataset, sorting is performed according to the width of the initial anchor box, and the initial anchor box is linearly scaled based on formula (2), with α and β taking the values ​​of 0.5.

[0111] Figure 4 The visualization image of the detection results of the experimental scheme (1) is shown. Figure 4 It can be seen that in experimental scheme (1), when detecting large-target utility poles and small-target utility pole numbers from railway scene images at the same time, the YOLOv3-tiny network can well capture the features of the large-target utility poles, but cannot detect the small-target numbers. Figure 5a The results of the first phase of the test scheme (2) are shown; Figure 5b The results of the second phase of the test scheme (2) are shown. Figure 5a It can be seen that the first stage can detect the electric poles from the scene image along the railway; Figure 5b It can be seen that in the second stage, the pole numbers cannot be detected from some pole images. The reason is that the size difference between the poles and the pole numbers in these pole images is large, making it difficult to detect the pole numbers from the pole images. Figure 6 The results of the second phase of the test scheme (3) are shown. Figure 6It can be seen that when the pole number of the utility pole is detected for the pole number area image containing the utility pole number plate, the pole number can be detected, which significantly improves the detection effect of the number. Figure 7 The results of the first phase of the test scheme (4) are shown. Figure 7 Analysis shows that the telephone poles can be detected in the first stage. However, due to the low brightness of the scene images along the railway, the features of the telephone pole license plates are not obvious, making it difficult to detect the telephone pole license plates from the scene images along the railway. Figure 8a The test results of the first stage in the test scheme (5) provided in the embodiment of the present invention are shown; Figure 8b The results of the second phase of the test scheme (5) provided in the embodiment of the present invention are shown. Figure 8a Analysis shows that in the first phase, expanding the detection range of telephone pole number plates and setting the pole number area to include the telephone pole number plates and the background area around the telephone pole number plates will help improve the detection rate of the pole number area in the railway scene map. The first target detection network used in the first phase has good feature learning of different shapes of telephone poles, can distinguish different shapes of telephone poles, and achieves relatively accurate positioning of the pole number area. Figure 8b Analysis shows that the setting of pole number areas can also improve the detection effect of telephone pole numbers.

[0112] The test data of experimental scheme (5) are shown in Table 2.

[0113] Table 2 Test data of test scheme (5)

[0114]

[0115] As can be seen from Table 2, in the two-stage target detection method provided by the embodiment of the present invention, both the first target detection network and the second target detection network maintain relatively high average precision and recall rates.

[0116] Figure 9 The figure shows a schematic diagram of the structure of the electric pole and pole number detection device along the railway based on deep learning provided by an embodiment of the present invention. Figure 9As shown, the deep learning-based device 900 for detecting electric poles and pole numbers along the railway includes: an image acquisition module 910, used to acquire a scene image along the railway to be detected, wherein the scene image along the railway to be detected includes electric poles; a first target detection module 920, used to perform target detection on the scene image along the railway to be detected through a first target prediction network, and obtain detection results of electric poles and pole number areas, wherein the pole number area includes electric pole number plates and the background area around the electric pole number plates; an image segmentation module 930, used to segment a pole number area image from the scene image along the railway to be detected according to the detection results of the pole number area; a second target detection module 940, used to perform target detection on the pole number area image through a second target prediction network, and obtain detection results of electric pole pole numbers.

[0117] In some embodiments, the area of ​​the pole number region is 2 to 10 times the area of ​​the telephone pole number plate.

[0118] In some embodiments, the width of the pole number area is 2 to 4 times the width of the pole number plate, and the height of the pole number area is consistent with the height of the pole number plate.

[0119] In some embodiments, the apparatus comprises:

[0120] A true bounding box annotation module is used to annotate the electric poles and pole number regions in multiple railway scene images to obtain multiple true bounding boxes of electric poles and multiple true bounding boxes of pole number regions;

[0121] An initial anchor frame clustering module, configured to cluster the plurality of utility pole true bounding boxes and the plurality of pole number area true bounding boxes to obtain a plurality of initial anchor frames;

[0122] The initial anchor frame scaling module is configured to scale the sizes of the multiple initial anchor frames based on a linear scale to obtain multiple anchor frames.

[0123] Optionally, the initial anchor frame scaling module includes:

[0124] an initial anchor frame sorting unit, configured to sort the multiple initial anchor frames according to their widths or heights;

[0125] The initial anchor frame scaling unit is configured to scale the multiple initial anchor frames based on the following formula to obtain multiple anchor frames:

[0126] w′1=αw1,

[0127] w′ N =βw N ,

[0128]

[0129]

[0130] Among them, w1, w N Represent the width of the first and Nth initial anchor boxes respectively, w′1 represents the width of the first anchor box corresponding to the first initial anchor box after scaling, and w′ N represents the width of the Nth anchor box corresponding to the Nth initial anchor box after scaling, w n , h n Represents the width and height of the nth initial anchor box, w′ n , h′ n They represent the width and height of the nth anchor box corresponding to the nth initial anchor box after scaling, n = 1, 2, ..., N, where N is the number of initial anchor boxes; α and β are scaling coefficients, and their values ​​range from [0, 1].

[0131] In some embodiments, the first object detection network is a YOLOv3-tiny network; and / or the second object detection network is a YOLOv3-tiny network.

[0132] In some embodiments, the detection result of the utility pole includes the confidence level of the utility pole detection frame; and the image acquisition module includes:

[0133] A video acquisition unit, used to acquire a video of a scene along the railway to be detected;

[0134] A video splitting unit, configured to split the video of the scene along the railway to be detected into a plurality of continuous images of the scene along the railway to be detected;

[0135] The device further comprises:

[0136] An image segmentation module is used to group the detected railway scene images corresponding to the same electric pole number among the plurality of detected railway scene images into one group;

[0137] The image selection module is used to select at least one detected railway scene image according to the confidence of the telephone pole detection frame of several detected railway scene images in each group, and store it as the target image of the telephone pole corresponding to the current group of detected railway scene images, and delete the other detected railway scene images in the current group of detected railway scene images.

[0138] Figure 10 FIG. 1 shows an electronic device according to an embodiment of the present invention. Figure 10As shown, the electronic device 1000 includes: at least one processor 1010, and a memory 1020 communicatively connected to the at least one processor 1010, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute a method.

[0139] Specifically, the memory 1020 and the processor 1010 are connected together via the bus 1030, and can be a general memory and processor, which are not specifically limited here. When the processor 1010 runs the computer program stored in the memory 520, it can execute the embodiment of the present invention in combination with Figures 1 to 9 Described operations and functions.

[0140] In an embodiment of the present invention, the electronic device 1000 may include but is not limited to: a personal computer, a server computer, a workstation, a desktop computer, a laptop computer, a notebook computer, a mobile computing device, a smart phone, a tablet computer, a personal digital assistant (PDA), a handheld device, a messaging device, a wearable computing device, and the like.

[0141] An embodiment of the present invention further provides a storage medium having a computer program stored thereon, which implements a method when the program is executed by a processor. For specific implementation, please refer to the method embodiment and will not be described in detail here. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and the computer or processor of the system or device reads and executes the instructions stored in the storage medium. The program code read from the storage medium itself can implement the functions of any of the above embodiments, and therefore the machine-readable code and the storage medium storing the machine-readable code constitute part of the present invention.

[0142] Storage media include, but are not limited to, floppy disks, hard disks, magneto-optical disks, optical disks, magnetic tapes, non-volatile memory cards, and ROMs. Program code can also be downloaded from a server computer or a cloud via a communication network.

[0143] It should be noted that not all steps and modules in the above processes and system structures are required, and certain steps and units can be omitted according to actual needs. The execution order of each step is not fixed and can be determined as needed. The device structure described in the above embodiments can be a physical structure or a logical structure. A module or unit may be implemented by the same physical entity, a module or unit may be implemented by multiple physical entities respectively, or a module or unit may be implemented by multiple components in multiple independent devices.

[0144] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and embodiments. They can be applied to a variety of fields suitable for the embodiments of the present invention. Those skilled in the art will readily realize further modifications. Therefore, without departing from the general concept defined by the claims and their equivalents, the embodiments of the present invention are not limited to the specific details and illustrations shown and described herein.

Claims

1. A method for detecting electric poles and pole numbers along railway lines based on deep learning, characterized in that: include: Acquire an image of a scene along a railway to be detected, wherein the image of the scene along the railway to be detected includes electric poles; Performing target detection on the scene image along the railway to be detected by a first target prediction network to obtain detection results of utility poles and pole number areas, wherein the pole number areas include utility pole number plates and background areas surrounding the utility pole number plates; Segmenting a pole number area image from the scene image along the railway to be detected according to the detection result of the pole number area; Performing target detection on the pole number area image through a second target prediction network to obtain a detection result of the pole number of the utility pole; The area of ​​the pole number area is 2 to 10 times the area of ​​the pole number plate; The width of the pole number area is 2 to 4 times the width of the pole number plate, and the height of the pole number area is consistent with the height of the pole number plate; The detection result of the electric pole includes the confidence level of the electric pole detection frame; and the step of obtaining the scene image along the railway to be detected includes: Obtain a video of the scene along the railway to be detected; Splitting the video of the scene along the railway to be detected into a plurality of continuous images of the scene along the railway to be detected; After performing target detection on the pole number area image by the second target prediction network to obtain a detection result of the pole number of the utility pole, the method further includes: Dividing the detected railway scene images corresponding to the same electric pole number among the multiple detected railway scene images into a group; According to the confidence of the utility pole detection frames of several detected railway scene images in each group, at least one detected railway scene image is selected and stored as the target image of the utility pole corresponding to the current group of detected railway scene images, and the other detected railway scene images in the current group of detected railway scene images are deleted.

2. The method for detecting electric poles and pole numbers along railways based on deep learning according to claim 1, characterized in that: The anchor boxes in the first target prediction network are determined by: Label the electric poles and pole number regions in multiple railway scene images to obtain multiple true bounding boxes of electric poles and multiple true bounding boxes of pole number regions; Clustering the plurality of utility pole true bounding boxes and the plurality of pole number area true bounding boxes to obtain a plurality of initial anchor boxes; The sizes of the multiple initial anchor boxes are scaled based on a linear scale to obtain multiple anchor boxes.

3. The method for detecting utility poles and pole numbers along railway lines based on deep learning according to claim 2, characterized in that: Scaling the multiple initial anchor frames based on a linear scale to obtain multiple anchor frames includes: sorting the multiple initial anchor boxes according to their widths or heights; The multiple initial anchor frames are scaled based on the following formula to obtain multiple anchor frames: ， ， ， ， in, Represents the width of the first and Nth initial anchor boxes, respectively, represents the width of the first anchor box corresponding to the first initial anchor box after scaling, represents the width of the Nth anchor box corresponding to the Nth initial anchor box after scaling, Respectively represent n The width and height of the initial anchor box, Respectively represent the scaled and n The initial anchor box corresponds to the n The width and height of the anchor box, , N is the number of initial anchor boxes; is the scaling factor, and its value range is [0,1].

4. The method for detecting electric poles and pole numbers along railway lines based on deep learning according to claim 1, characterized in that: The first target prediction network is a YOLOv3-tiny network; and / or the second target prediction network is a YOLOv3-tiny network.

5. A device for detecting electric poles and pole numbers along railway lines based on deep learning, characterized in that: include: An image acquisition module is used to acquire an image of a scene along the railway to be detected, wherein the image of the scene along the railway to be detected includes electric poles; a first target detection module, configured to perform target detection on the scene image along the railway to be detected using a first target prediction network, and obtain detection results of utility poles and pole number areas, wherein the pole number areas include utility pole number plates and a background area surrounding the utility pole number plates; An image segmentation module is used to segment the pole number area image from the scene image along the railway to be detected according to the detection result of the pole number area; A second target detection module is used to perform target detection on the pole number area image through a second target prediction network to obtain a detection result of the pole number of the utility pole; The area of ​​the pole number area is 2 to 10 times the area of ​​the pole number plate; The width of the pole number area is 2 to 4 times the width of the pole number plate, and the height of the pole number area is consistent with the height of the pole number plate; The detection result of the utility pole includes the confidence level of the utility pole detection frame; the image acquisition module includes: A video acquisition unit, used to acquire a video of a scene along the railway to be detected; A video splitting unit, configured to split the video of the scene along the railway to be detected into a plurality of continuous images of the scene along the railway to be detected; The device further comprises: An image segmentation module is used to group a plurality of detected railway scene images corresponding to the same electric pole number among the plurality of detected railway scene images; The image selection module is used to select at least one detected railway scene image according to the confidence of the telephone pole detection frame of several detected railway scene images in each group, and store it as the target image of the telephone pole corresponding to the current group of detected railway scene images, and delete the other detected railway scene images in the current group of detected railway scene images.

6. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to cause the at least one processor to perform the method according to any one of claims 1 to 4.

7. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images