Pantograph and catenary arc visual detection method

Abstract

The application provides a pantograph and catenary arc visual detection method, and belongs to the technical field of computer vision pattern recognition and target detection. A four-stage feature down-sampling module built by using a self-attention mechanism models the global context of an image at each stage, captures the global features of a target and models long-range semantic dependency relationships, and aggregates features and position information from the entire input domain. Meanwhile, the encoder and the decoder are connected through a skip layer connection mode, the low-dimensional shallow semantic information at each stage in the down-sampling is fused with the high-dimensional deep semantic information at each stage in the up-sampling, the detection precision is improved, and the data requirement is significantly reduced. The multi-dimensional global feature fusion network is trained by using a training data set and a stochastic gradient descent method, and the serialized image features are first gradually extracted by four cascaded self-attention modules in the encoder to obtain high-dimensional deep features, and four features with different dimensions are generated. The method is used for pantograph and catenary arc visual automatic detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, pattern recognition and object detection. BACKGROUND

[0002] The pantograph and catenary system of electrified railway is the only source of power for electrified trains, and its safe and stable operation is crucial. The instantaneous high temperature during arcing can seriously endanger the safety of train insulation equipment, exacerbate the wear of carbon slide plates, and cause serious damage such as contact wire breakage during long-term operation, which has a serious impact on train operation safety. By analyzing the arcing image, the arcing occurrence rate, arcing intensity, size, and shape are calculated to provide data support for the current collection quality analysis of the pantograph and catenary. In recent years, the research on arcing detection of electrified railway pantograph and catenary has made great progress. From the classic threshold segmentation algorithm and edge detection algorithm to the detection algorithm based on convolutional neural network, they have been well applied in arcing image detection. The arcing detection method based on threshold segmentation algorithm distinguishes arcing from background by selecting appropriate threshold. The arcing detection method based on edge detection calculates the pixel value change between pixels and adjacent pixels in the image, obtains the arcing edge according to the first and second derivatives of the change degree, and extracts the arcing target through post-processing algorithm. The arcing detection method based on convolutional neural network uses the convolution module of neural network to extract the features of arcing image, generates accurate candidate regions, and then uses non-maximum suppression to obtain the detection result. The disadvantages of the arcing detection method based on threshold segmentation algorithm are as follows, first, the threshold needs to be reset when the environment changes; second, the threshold segmentation algorithm will be misdetected when there are disturbances similar to the pixel value of arcing. The disadvantages of the arcing detection method based on edge detection are as follows, first, the edge detection algorithm needs to calculate each pixel and its adjacent pixels, and the time required for post-processing is long, which cannot meet the real-time requirement; second, the edge detection algorithm will misposition the arcing edge when there are disturbances in the arcing image. SUMMARY

[0003] The purpose of the present application is to provide a pantograph and catenary arcing visual detection method which can effectively solve the technical problem of real-time and accurate robust detection of arcing under complex background.

[0004] The purpose of the present application is achieved by the following technical scheme: a pantograph and catenary arcing visual detection method, comprising the following steps:

[0005] (1) Construction of pantograph and catenary arcing dataset

[0006] The construction of the pantograph-catenary arc data set is divided into two steps: first, the pantograph-catenary arc video with diversity is collected by an automatic camera, and then the arc image data is extracted from the pantograph-catenary arc video frame by frame, and the pantograph-catenary arc image is labeled using the labelme software; then, the extracted arc image data is enhanced and denoised, the original data is set by setting the Gaussian white noise of different sizes, distribution and shape of the arc image, and the rain, snow and fog weather elements are added for data enhancement, the original image is denoised by using a Gaussian filter algorithm with a filter radius of 2, the standard deviation in the horizontal direction is set to 1, the standard deviation in the vertical direction is set to 2, and the filter kernel size is 5*5; the length and width of each picture are 224 pixel values, the rain, snow and fog weather elements are enhanced for each picture used for training, and the enhanced arc image is obtained;

[0007] (2) Construction of pantograph-catenary arc detection model

[0008] The pantograph-catenary arc detection model includes an encoder and a decoder, wherein the encoder includes an image block projection encoding module, four self-attention mechanism layers, and each self-attention mechanism layer includes a self-attention mechanism module; the decoder includes a feature shape reorganization module, four multi-dimensional global feature adaptive weight fusion modules, and three semantic modules;

[0009] The image block projection encoding module uses a linear embedding layer to convert the input image into a corresponding one-hot feature vector through linear projection, the self-attention mechanism module calculates the inter-class and intra-class information in the input feature vector Q through the multi-head attention module W-MSA and the sliding window multi-head attention module SW-MSA to extract the features; the feature shape reorganization module uses a 3*3 convolution layer to restore the serialized image features calculated by the self-attention mechanism module to the original image features, and restores the two-dimensional pixel arrangement order; the decoder is a decoder based on a convolutional neural network, and the up-sampling layer of the sampling module includes three identical convolution modules, in each module, first, the features after up-sampling of the previous layer are combined with the converted features in the down-sampling layer of the encoder through channel stacking to obtain high-dimensional fusion features, then a 1*1 convolution layer is used for dimension reduction, then two 3*3 convolution layers are used in series to extract features, and finally an up-convolution module is used to expand the two-dimensional size of the features by four times to obtain the final semantic features R semantic ; The multi-dimensional global feature adaptive weight fusion module takes the output features R of each down-sampling layer of the encoder as the original input, obtains three global features of the global mean R1, the global maximum R2 and the global threshold R3 from R, and then fuses R, R1, R2 and R3 in the form of channel stacking to obtain the global fusion features R0 through adaptive weights a, b and g, the value range of a, b and g is (0, 1), and the calculation formula is as follows:

[0010]

[0011] R2 = repeat(MAX(R [*，i，j] ))

[0012]

[0013]

[0014] R0 = R + aR1 + bR2 + gR3

[0015] wherein repeat represents repeat padding, H = 224, W = 224 represent that the length of the feature map is 224 pixels and the width is 224 pixels, M represents the average value of the average value of all channels in the image sent in the same batch, S represents the feature map reduction multiple, S e {2, 4, 6, 8}, R [*i，j] pixel value of each pixel in the feature map, wherein '*' represents a feature channel, 'i' represents a pixel in the i-th row of the feature map, i has a value range of [0, 224], 'j' represents a pixel in the j-th column, j has a value range of [0, 224], B represents the number of images input into the model at the beginning, C represents the number of channels, C e {96, 192, 384, 768}, the dimensions of R1, R2 and R3 are the same as those of the original recombined feature; finally, a 1x1 convolution layer is used to reduce the dimension of R0, a 3x3 convolution layer is used to extract R0 feature, and then an up-convolution is used to reduce the image length and width of R0 to The global fusion features obtained in the four different stages are fused in a channel stacking manner, and a 1x1 and a 3x3 convolution layer are used to reduce the dimension and calculate to obtain the global feature R global ; finally, the semantic feature R semantic and the global feature R global are fused in a channel stacking manner, and a 1x1 and a 3x3 convolution layer are used to reduce the dimension to obtain the global enhanced pixel feature R semantic-global ; finally, the global enhanced pixel feature is restored to the original input image size by using the up-convolution operation and the pixel attribute in the image is predicted, the pixel value of the arcing pixel is 255, and the pixel value of other background pixels is unchanged;

[0016] (3) Hierarchical training of the pantograph-catenary arcing detection model

[0017] The arc image data input into the pantograph arc detection model is divided into three categories. The first category is a large arc target, and the number of arc pixels is 300-1000 pixels. This type of data is directly sent to the pantograph arc detection model for training. The second category is a medium arc target, and the number of arc pixels is 100-300 pixels. This type of image is augmented by rotating it counterclockwise by 90 degrees, cutting each of the two arc images in half, and then combining the two different halves together. Then it is sent to the model for training. The third category is a small arc target, and the number of arc pixels is 10-100 pixels. This type of image is augmented by rotating, cutting and splicing, and the pixel values of the two images are weighted and superimposed according to the following formula,

[0018] f2=f0*(1-a)+a*f1

[0019] where f0 represents the value of each pixel in the first picture, f1 represents the value of each pixel in the second picture, and f2 represents the value of each pixel in the superimposed picture. The data set in the training stage is divided into training set and test set according to the ratio of 7:3. GPU is used to accelerate the calculation of pantograph arc detection model weight in the training stage. Cross-Entropy Loss and Dice Loss are used as the loss function of the pantograph arc detection model to update the weight parameters during back propagation. The combined loss function L0 is as follows:

[0020] L1=L ce (F [i，j] ,GT [i，j] )

[0021]

[0022] L o =a*L1+b*L2

[0023] L ce represents Cross-Entropy Loss, L1 represents Cross-Entropy Loss, L2 represents Dice Loss, L0 represents combined loss function, F [i，j] represents the value of the pixel in the pantograph arc detection model prediction result, GT [i，j] represents the true value of the pixel, and the empirical constants a=0.5 and b=0.5. At this point, the pantograph arc detection model has the ability to detect arc.

[0024] (4) Image input

[0025] The image input comes from the video collected by the camera installed beside the pantograph. The single frame pictures are converted in time sequence. If the input image is empty, it is terminated, otherwise it jumps to (5).

[0026] (5) Arc detection

[0027] The non-empty fuel arc image from the camera is input into the pantograph arc detection model, target features are extracted by the encoder of the model, the decoder restores the image to the original image resolution and predicts the attributes of each pixel in the image, and the arc detection result is obtained at this time;

[0028] (6) Online learning and model updating

[0029] After obtaining the detection result of the real-time data of the pantograph arc, the real-time data and the corresponding detection result are simultaneously input into the pantograph arc detection model for training, and the weight parameters of the pantograph arc detection model are updated, and when there is no real-time data input, the weight parameter updating is stopped, and the step (4) is jumped to;

[0030] In the initial network training stage, the multi-dimensional global feature fusion network is trained using the training data set and the stochastic gradient descent method. The serialized image features first pass through four cascaded self-attention modules in the encoder to gradually extract high-dimensional deep features, and four features of different dimensions are generated. After the four features of different dimensions pass through the feature conversion module, they enter the semantic upsampling module and the multi-dimensional global feature adaptive weight fusion module in turn. In the semantic upsampling module, high-dimensional information is constantly fused with low-dimensional information to obtain detailed pixel class description. In the fusion module, the global fusion features corresponding to different feature dimensions are calculated, and the four global fusion features of the same size are fused by channel superposition to obtain the final full-resolution fusion feature. Finally, the full-resolution fusion feature is fused with the semantic module output, so as to realize accurate target detection. In addition, after each detection is completed, the network is fine-tuned and updated according to the difference between the predicted value and the actual value.

[0031] The advantages and positive effects of the present application compared with the prior art are as follows:

[0032] The method provided by the application utilizes a four-stage feature down-sampling module built by a self-attention mechanism to model the global context of an image at each stage, capture the global features of a target and model the remote semantic dependency, and widely aggregates the features and position information from the entire input domain. Meanwhile, the encoder and the decoder are connected through a skip connection mode, the low-dimensional shallow semantic information at each stage in the down-sampling is fused with the high-dimensional deep semantic information at each stage in the up-sampling, the detection precision is improved, and the data requirement is significantly reduced. Moreover, the multi-dimensional global feature adaptive weight fusion module provided by the application can significantly improve the global information calculation capability of the model, enhance the target pixel representation capability, and be stronger in target edge delineation and more accurate in attribute judgment of edge pixels. The application can solve the problems of low accuracy, long reasoning time and low robustness in the arc burning detection task. Moreover, the application can still maintain high detection accuracy for the arc burning target in complex environments, such as rainy, snowy and foggy weather, strong light reflection and carbon slide reflection. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The model structure diagram of the application.

[0034] Figure 2 The sub-module structure diagram of the model of the application.

[0035] Figure 3 The method flowchart of the application. DETAILED DESCRIPTION

[0036] The application can be applied to various occasions of visual target detection and segmentation, including road crack detection, material crack segmentation, picture cutout, intelligent arc burning detection of catenary, foreign matter intrusion detection of traction substation, environmental perception in the field of automatic driving, medical image segmentation diagnosis, intelligent transportation system and the like.

[0037] Taking the intelligent arc burning detection of the catenary as an example: one of the measurement standards of the catenary current collection quality is the occurrence rate of the arc burning and the intensity of the visible light of the arc burning. The arc burning occurrence rate refers to the number of times of the arc burning in a certain time, and the arc burning needs to be accurately detected in each frame of the video; the intensity of the visible light of the arc burning refers to the proportion of the pixels of the visible light of the arc burning in the entire image, and the arc burning needs to be accurately distinguished from the background in each frame. In the first step, the catenary arc burning video is converted into an image set frame by frame, then all the images are divided into a training set and a test set, and the training set is labeled by using the Labelme software. In the second step, the Gaussian white noise is used to simulate the rainy, snowy and foggy weather to improve the robustness of the model, and the data set is expanded by using the flipping and rotating modes. In the third step, the image is sent into the image block projection coding layer of the segmentation network to divide the image with the dimension of (224, 224, 3) into Each image block is one-dimensional serialized feature with the size of 4*4 and the dimension of 4*4*3. Then the features are extracted by a four-stage self-attention mechanism encoder, and the output of each stage is converted into image features with different sizes and dimensions by a feature conversion module and input into a multi-dimensional global feature adaptive weight fusion module. The fourth step is divided into two modules. In the multi-dimensional global feature adaptive weight fusion module, the global maximum value can better enhance the representation ability of the arc burning pixel, the global threshold value can reduce the influence of sunlight, reflection and other interference on the global maximum value, the global mean value can improve the environmental perception of the network, and the adaptive weight fusion can combine the global features of the three dimensions with the original features. In the semantic upsampling module, the high-dimensional features of the decoder and the low-dimensional features of the encoder are continuously fused and upsampled to restore the image size to Finally, the semantic feature R semantic is fused with the global feature R global to obtain the global enhanced pixel feature R semantic-global . The global enhanced pixel feature R semantic-global is convolved to obtain a result image with the same size as the original input image, and the attribute value of each pixel in the image is predicted by a last 3*3 convolution layer. Through actual testing, the detection accuracy of the present application in a 900 arc picture data set can reach 97.78%, the segmentation accuracy can reach 89.13%, and the inference time of a single arc picture is 22.62ms.

[0038] The method of the present application can be realized by programming in any computer programming language (such as python language), and the detection system software based on the method can realize real-time target detection application in any PC or embedded system.

Claims

1. A pantograph and catenary arcing visual detection method, comprising the following steps: (1) Construction of pantograph and catenary arcing data set The construction of the pantograph and catenary arcing data set is divided into two steps: first, pantograph and catenary arcing videos with diversity are collected by an automatic camera, and then pantograph and catenary arcing image data are extracted frame by frame from the pantograph and catenary arcing videos, and the pantograph and catenary arcing images are labeled using labelme software; then, the extracted pantograph and catenary arcing image data are enhanced and denoised, the original data is set by setting different pantograph and catenary arcing image sizes, distributions and shapes of Gaussian white noise, and weather elements such as rain, snow and fog are added for data enhancement, and the original image is denoised using a Gaussian filter algorithm with a filter radius of 2, the standard deviation in the horizontal direction is set to 1, the standard deviation in the vertical direction is set to 2, and the filter kernel size is 5x5; each picture has a length and width of 224 pixels, and each picture for training is enhanced with weather elements such as rain, snow and fog to obtain enhanced pantograph and catenary arcing images; (2) Construction of pantograph and catenary arcing detection model The pantograph and catenary arcing detection model includes an encoder and a decoder, wherein the encoder includes an image block projection encoding module, four self-attention mechanism layers, and each self-attention mechanism layer includes a self-attention mechanism module; the decoder includes a feature shape reconstruction module, four multi-dimensional global feature adaptive weight fusion modules, and three semantic modules; The image block projection coding module adopts a linear embedding layer to convert the input image into a corresponding one-hot feature vector through linear projection, and the self-attention mechanism module calculates long-distance information to extract features through a multi-head attention module W-MSA and a sliding window multi-head attention module SW-MSA. The feature shape reconstruction module uses a 3x3 convolution layer to restore the serialized image features calculated by the self-attention mechanism module to the original image features and restore the two-dimensional pixel arrangement order; The decoder is a decoder based on a convolutional neural network. An up-sampling layer of a sampling module of the decoder comprises three identical convolution modules. In each module, first, high-dimensional fused features are obtained by channel superposition of the features converted in the down-sampling layer of the encoder and the features converted in the up-sampling layer of the previous layer, then, the high-dimensional fused features are reduced in dimension by using a 1*1 convolution layer, then, the features are extracted by using two 3*3 convolution layers connected in series, and finally, the two-dimensional size of the features is expanded four times by using an up-convolution module to obtain final semantic features ; Multi-dimensional The global feature adaptive weight fusion module fuses the features output by each down-sampling layer of the encoder R As the original input, by R Get the global mean R 1、Global maximum R 2 and the global threshold R 3 three global features, and then through the adaptive weight 、 、 Will R 、 R 1、 R 2、 R 3 in the channel superposition way to get global fusion features R 0, 、 、 The value range of 0, 1, the calculation formula is as follows: ； ； ； ； ； in repeat This represents repeated filling. =224、 =224 means the feature map is 224 pixels long and 224 pixels wide. M This represents the average of all channels in the same batch of images. S The feature map is reduced by a factor of 10. , This represents the pixel value of each pixel in the feature map, where Representative feature channel, ' 'Represents the pixel in the i-th row of the feature map, where i ranges from [0, 224],' 'Represents the pixel in column j, where j ranges from [0, 224], B This represents the number of images initially input into the model. C Represents the number of channels. C , R 1. R 2. R All three dimensions are the same as the original reconstructed features; finally, a 1×1 convolutional layer is used for processing. R Dimensionality reduction is performed using a 3×3 convolutional layer to extract... R 0 features, then use an up convolution to R The image dimensions of 0 are reduced to the original image size. The global fusion features obtained from the four different stages are fused together by channel stacking, and then the dimensionality is reduced and calculated by a 1×1 and a 3×3 convolutional layer to obtain the global features. Finally, semantic features With global features The channels are fused together, and the dimensionality is reduced by a 1×1 and a 3×3 convolutional layer to obtain globally enhanced pixel features. Finally, an up-convolution operation is applied to the global enhanced pixel features to restore the original input image size and predict the attribute of each pixel in the image. The pixel value of the arc pixel is 255, and the pixel values ​​of other background pixels remain unchanged. (3) Hierarchical training of pantograph and catenary arcing detection model The pantograph and catenary arcing image data input into the pantograph and catenary arcing detection model are divided into three categories: the first category is a large arcing target with 300-1000 pixels, and this category of data is directly input into the pantograph and catenary arcing detection model for training; the second category is a medium arcing target with 100-300 pixels, and this category of image is augmented by rotating it 90 degrees counterclockwise, cutting each half of the two arcing images, and then combining the two different halves together, and then inputting it into the model for training; the third category is a small arcing target with 10-100 pixels, and this category is augmented by rotating the image, cutting and splicing, and weighting and superimposing the pixel values of the two images according to the following formula, ； wherein f 0 represents the value of each pixel in the first picture, f 1 represents the value of each pixel in the second picture, f 2 represents the value of each pixel in the superimposed picture, which is augmented and then fed into the model for training; the data set in the training stage is divided into a training set and a test set according to a ratio of 7:3, and a GPU is required to accelerate the calculation of the weight of the pantograph-catenary arc detection model in the training stage; the joint loss function of the cross-entropy loss function Cross-Entropy Loss and the set similarity measurement loss function Dice Loss is used to iteratively update the weight parameters of the pantograph-catenary arc detection model during back propagation as follows: ； ； ； represents Cross-Entropy Loss, represents Cross-Entropy Loss function, represents Dice Loss, represents Joint Loss function, represents the value of a pixel in the prediction result of the catenary arc detection model, represents the true value of a pixel, an empirical constant a = 0.5, b = 0.5, up to this, the catenary arc detection model has the arc detection capability; (4) Image input The image input comes from the video collected by the camera installed beside the pantograph, which is converted into single-frame pictures in time sequence, and if the input image is empty, the process is terminated, otherwise it jumps to (5); (5) Arcing detection The non-empty arcing image from the camera is input into the pantograph and catenary arcing detection model, the target features are extracted by the encoder of the model, the image is restored to the original resolution by the decoder, and the attributes of each pixel in the image are predicted, and the arcing detection result is obtained; (6) Online learning and model updating After obtaining the detection result of the real-time data of the pantograph-catenary arc, the real-time data and the corresponding detection result are simultaneously sent into the pantograph-catenary arc detection model for training, and the weight parameters of the pantograph-catenary arc detection model are updated; when there is no real-time data input, the updating of the weight parameters is stopped, and the process jumps to (4).

Images