Pantograph arc monitoring method based on multi-mode arc detection network

Abstract

The invention discloses a pantograph arc monitoring method based on a multi-mode arc detection network, and belongs to the technical field of pantograph arc monitoring, and the method comprises the following steps: building a multi-mode arc detection network based on a de-noising diffusion probability model DDPM and a semantic decoder, and carrying out the self-supervision training through an unmarked pantograph image, a trained multi-mode arc detection network is obtained; and inputting a pantograph image in an actual line into the trained multi-mode arc detection network to obtain an arc prediction label, and completing pantograph arc monitoring. According to the method, the visible light image features and the infrared light image features are extracted by using the de-noising diffusion probability model DDPM, robust and discriminative features can be generated, and the dependence of a multi-mode arc detection network on the arc data labeled with labels is reduced; the semantic decoder is used for fusing the multi-modal data of the image, and the dependence of the multi-modal arc detection network on the labeled arc data is reduced through the audio-visual correlation of the arc characteristics.

Description

Technical Field

[0001] The present invention belongs to the technical field of pantograph arc monitoring, and in particular relates to a pantograph arc monitoring method based on a multi-modal arc detection network. Background Art

[0002] As the only energy supply for trains, the safety and stability of the pantograph-contact network system is of vital importance. In electrified railways, arcing is an important indicator for evaluating the contact status between the pantograph and the contact network, and is crucial for the quality control of current collection. The current collection performance of the pantograph and the contact network directly affects the speed increase and stable operation of the railway, while arcing accelerates the ablation and wear of the slide plate and the contact wire, and generates strong electromagnetic interference and radiation, affecting train communications and polluting the environment.

[0003] Arc detection technology in the pantograph-catenary system is mainly divided into contact and non-contact types. Common contact detection methods include oscillating current detection, current waveform detection and resistance detection. Oscillating current detection determines the state of the pantograph-catenary system by monitoring the changes in high-frequency current; current waveform detection detects arcs by identifying the distortion characteristics of voltage and current waveforms; resistance detection uses the principle of increased resistance during arcing to detect current fluctuations in the circuit; however, in actual train operation, factors such as unstable current and power fluctuations can easily interfere with current and voltage parameters, affecting the accuracy of contact detection results. Non-contact detection improves the reliability and accuracy of measurement by eliminating the influence of unstable current and power fluctuations on current and voltage parameters. However, it is sensitive to environmental requirements and is easily disturbed by factors such as noise, infrared, and temperature in complex environments, resulting in an increase in the rate of misjudgment.

[0004] With the development of deep learning, non-contact arc detection methods based on deep learning have shown great potential in processing visible light images. With their powerful nonlinear fitting capabilities, these methods reduce the sensitivity to environmental changes and noise and improve measurement accuracy. However, these deep learning-based detection methods rely heavily on supervised learning models and require the establishment of a large number of arc datasets with precise annotations. However, arc events occur infrequently, and it is expensive and time-consuming to obtain and annotate arc datasets, which limits the use of the above methods on a large number of unlabeled pantograph and contact network images. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a pantograph arc monitoring method based on a multi-modal arc detection network, which solves the problem that arc monitoring of the pantograph-contact network system is easily affected by environmental factors and difficult to use in practice.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a pantograph arc monitoring method based on a multi-modal arc detection network, comprising the following steps: S1: Establish a multimodal arc detection network based on denoising diffusion probability model DDPM and semantic decoder; S2: obtaining an unlabeled pantograph visible light image and an unlabeled pantograph infrared image, and performing self-supervised training on a multimodal arc detection network to obtain a trained multimodal arc detection network; S3: Obtain the pantograph and overhead line contact images in the actual line and input them into the trained multimodal arc detection network to obtain arc prediction labels and complete pantograph arc monitoring.

[0007] The beneficial effects of the present invention are as follows: the present invention utilizes a denoising diffusion probability model DDPM for self-supervised pre-training to extract visible light image features and infrared light image features, and can generate robust and discriminative features, thereby reducing the dependence of a multimodal arc detection network on large-scale labeled arc data, reducing training costs, and improving generalization capabilities. At the same time, a hierarchical variation semantic decoder is utilized to fuse multimodal data of visible light images and infrared images, and an audio-visual semantic decoder is utilized to effectively process the interaction of different modal features through the audio-visual correlation of arc features, thereby improving the accuracy and reliability of arc monitoring, further reducing the dependence of a multimodal arc detection network on labeled arc data, and enhancing robustness and generalization performance in complex scenarios.

[0008] Further: the multimodal arc detection network in S1 includes a denoising diffusion probability model DDPM, a hierarchical change semantic decoding module and an audio-visual semantic decoding module connected in sequence; The denoising diffusion probability model DDPM is used to capture the potential structures and patterns in visible light images and infrared light images, and obtain multi-scale feature information under different diffusion steps; The hierarchical change semantic decoding module includes a plurality of hierarchical change semantic decoders, which are used to process multi-scale feature information under different diffusion steps and output fusion features; The audio-visual semantic decoding module is used to process the fused features and output arc prediction labels.

[0009] The beneficial effects of the above further scheme are: by extracting feature information through the denoising diffusion probability model DDPM, the dependence of the multimodal arc detection network on large-scale labeled arc data can be reduced. Through the hierarchical change semantic decoding module and the audio-visual semantic decoding module, high-precision arc monitoring is achieved, and the dependence on large-scale labeled arc data is further reduced.

[0010] Furthermore: the multimodal arc detection network is self-supervised trained, and the specific training steps are as follows: S201: Acquire an unlabeled pantograph visible light image and an unlabeled pantograph infrared light image to obtain image data and a true arc label; S202: Based on the Markov chain, the denoising diffusion probability model DDPM is used to perform forward diffusion on the image data, and Gaussian noise is added to the image data to obtain image data close to the standard normal distribution; S203: Calculate the mean function of the denoising diffusion probability model DDPM; S204: according to the mean function, using the denoising diffusion probability model DDPM to perform reverse diffusion on the image data close to the standard normal distribution, remove Gaussian noise in the image data close to the standard normal distribution, and obtain clean image data; S205: After completing S204, the loss function of the denoising diffusion probability model DDPM is calculated, and it is determined whether the loss value of the denoising diffusion probability model DDPM meets the convergence condition. If so, the self-supervised training of the denoising diffusion probability model DDPM is completed to obtain the trained denoising diffusion probability model DDPM. Otherwise, return to S202; S206: sequentially connect the trained denoising diffusion probability model DDPM to the hierarchical variation semantic decoding module and the audio-visual semantic decoding module to obtain a multimodal arc detection network; S207: inputting the image data into a multimodal arc detection network for training, obtaining an arc prediction label, and calculating a cross entropy loss function between the arc prediction label and the arc true label; S208: Determine whether the loss value of the cross entropy loss function meets the convergence condition. If so, obtain a trained multimodal arc detection network. Otherwise, return to S207.

[0011] The beneficial effects of the above further scheme are: based on the denoising diffusion probability model DDPM, the hierarchical change semantic decoding module and the audio-visual semantic decoding module, it can reduce the dependence on labeled arc data and realize arc monitoring, and improve the accuracy and reliability of arc monitoring through training.

[0012] Further: the expression of the forward diffusion of S202 is as follows:

[0013]

[0014]

[0015] in, is the probability distribution of forward diffusion, For each image data in the forward diffusion process, is the initial image data, is the time step, is the number of diffusion steps, For the Image data for diffusion steps, is a Gaussian distribution, is a constant, is the initial visible light image, is the initial infrared image; The expression of the mean function of the denoising diffusion probability model DDPM in S203 is as follows:

[0016]

[0017] in, is the mean function of the denoising diffusion probability model DDPM, For the Image data for diffusion steps, is the number of diffusion steps, is Gaussian noise, is the Gaussian noise predicted by the neural network, are the parameters of the denoising diffusion probability model DDPM, is the noise attenuation based on the time step, is a constant, From time step 1 to time step All The cumulative product of is the index of the time step; The expression of the reverse diffusion in S204 is as follows:

[0018]

[0019]

[0020] in, is the probability distribution of reverse diffusion, is the variance, is the covariance matrix.

[0021] The beneficial effect of the above further scheme is that the pre-training process of the denoising diffusion probability model DDPM is expressed by mathematical expressions, which facilitates in-depth analysis of the generalization ability of the model and optimization of the model.

[0022] Furthermore, the loss function of the denoising diffusion probability model DDPM is expressed as follows:

[0023] in, is the loss function of the denoising diffusion probability model DDPM, is the expected value, is the number of diffusion steps, is the initial image data, is Gaussian noise, is the Gaussian noise predicted by the neural network, is from time step 1 to time step All The cumulative product of is the noise attenuation based on the time step.

[0024] The beneficial effect of the above further scheme is that through the calculation of the loss function, the performance of the denoising diffusion probability model DDPM can be judged to avoid overfitting, which causes a decrease in prediction performance and generalization ability.

[0025] Further: The expression of the cross entropy loss function is as follows:

[0026] in, is the calculation result of the cross entropy loss function, is the cross entropy loss function, Predict labels for arcs, is the real label of the arc.

[0027] The beneficial effect of the above further scheme is that the performance of the multimodal arc detection network can be reflected through the cross entropy loss function, and the technical personnel can be guided to improve and optimize the multimodal arc detection network.

[0028] Further: the specific steps of S3 are as follows: S301: Acquire the contact image of the pantograph and the overhead line in the actual line; S302: Input the pantograph and contact network images in the actual line into the denoising diffusion probability model DDPM in the trained multimodal arc detection network to obtain multi-scale feature information under different diffusion steps; S303: inputting the multi-scale feature information under different diffusion steps into the hierarchical change semantic decoding module to obtain fusion features; S304: Input the fused features into the audio-visual semantic decoding module to obtain arc prediction labels.

[0029] The beneficial effects of the above further scheme are: the denoising diffusion probability model DDPM, the hierarchical change semantic decoding module and the audio-visual semantic decoding module in the trained multimodal arc detection network can extract and fuse features of the pantograph image, obtain accurate and reliable arc prediction labels, and realize accurate arc monitoring.

[0030] Furthermore: the expression of the multi-scale feature information under different diffusion steps is as follows:

[0031]

[0032]

[0033]

[0034] in, is the multi-scale feature information of the visible light image, is the multi-scale feature information of the infrared image, For different scales, the value is an integer from 0 to 4. is the weight of the denoising diffusion probability model DDPM, is the weight of the denoising diffusion probability model DDPM, The initial pantograph and overhead line contact visible light image with Gaussian noise added, is the initial infrared image of the pantograph and overhead line contact with Gaussian noise added. is the initial visible light image of the pantograph and overhead line contact, is the initial infrared image of the pantograph and overhead contact. From time step 1 to time step All The cumulative product of is the noise attenuation based on the time step, is Gaussian noise.

[0035] The beneficial effect of the above further scheme is: by extracting multi-scale features of visible light images and infrared light images, it is convenient for the multimodal arc detection network to understand the global semantic information of the image, and under the condition of few samples, the segmentation effect of the multimodal arc detection network is improved.

[0036] Further: the expression of the fusion feature is as follows:

[0037]

[0038]

[0039]

[0040]

[0041] in, For the The fusion features output by the layer-wise semantic decoder, For addition operation, is the spatial attention mechanism, is the second feature after fusion, is the first feature after fusion, For the The fusion features output by the layer-wise semantic decoder, is bilinear upsampling, The sampling factor is set to 2, For the The scale characteristics of a visible light image, For the The scale characteristics of the infrared image, For splicing operation, is the multi-scale feature information of the visible light image, is the multi-scale feature information of the infrared image, is the number of diffusion steps, The splicing dimension.

[0042] The beneficial effect of the above further scheme is: by fusing the multi-scale feature information under different diffusion steps, it is possible to handle the scale changes of feature information of different scales, which is convenient for subsequent steps. At the same time, by fusing features, it is possible to capture details and features at different scales, thereby improving the robustness and accuracy of the multimodal arc detection network.

[0043] Further: the expression of the arc prediction label is as follows:

[0044]

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051] in, Predict labels for arcs, is the convolution function, To highlight the visual features of the audition correlation, For addition operation, is the fusion feature output by the last level change semantic decoder, is the cross-modal attention, is the key value, is the transition parameter, For the updated prompt feature, is the fully connected layer, To fine-tune the operation, for The projection of is the fused audio prompt feature, is the visual clue feature, is the audio feature, is the audio noise characteristic, is the global average pooling, The splicing dimension.

[0052] The beneficial effect of the above further scheme is: through the audio-visual semantic decoding module, the audio-visual correlation of arc features is utilized to effectively process the interaction between different modal features and reduce the model's dependence on labeled arc data. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A flow chart of a pantograph arc monitoring method based on a multi-modal arc detection network; Figure 2 It is the structural diagram of the multi-modal arc detection network; Figure 3 This is the self-supervised pre-training and feature extraction structure diagram of the denoising diffusion probability model DDPM; Figure 4 This is the structural diagram of the level change semantic decoding module; Figure 5 This is the structural diagram of the audio-visual semantic decoding module; Figure 6 The performance of different models on the arc dataset is plotted against the amount of training data; Figure 7 Test the generalization performance of different models on different datasets. DETAILED DESCRIPTION

[0054] The specific implementation modes of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0055] Example 1 like Figure 1As shown, a flow chart of a pantograph arc monitoring method based on a multi-modal arc detection network includes the following steps: S1: Establish a multimodal arc detection network based on denoising diffusion probability model DDPM and semantic decoder; S2: obtaining an unlabeled pantograph visible light image and an unlabeled pantograph infrared image, and performing self-supervised training on a multimodal arc detection network to obtain a trained multimodal arc detection network; S3: Obtain the pantograph and overhead contact images in the actual line, and input them into the trained multimodal arc detection network to obtain the arc prediction label and complete the pantograph arc monitoring.

[0056] like Figure 2 As shown, it is a structural diagram of a multimodal arc detection network, the input includes a visible light image and an infrared light image, wherein the multimodal arc detection network includes a denoising diffusion probability model DDPM, a hierarchical change semantic decoding module, and an audio-visual semantic decoding module connected in sequence; Denoising Diffusion Probabilistic Model (DDPM) is used to capture the potential structures and patterns in visible light images and infrared images, and obtain multi-scale feature information under different diffusion steps; The change semantic decoding module includes several hierarchical change semantic decoders, which are used to process multi-scale feature information under different diffusion steps and output fusion features; The audio-visual semantic decoding module is used to process the fused features and output arc prediction labels.

[0057] like Figure 3 As shown in FIG, it is a self-supervised pre-training and feature extraction structure diagram of the denoising diffusion probability model DDPM; in the multimodal arc detection network, the denoising diffusion probability model DDPM needs to be trained first, and then multi-scale feature extraction is performed; including (a) self-supervised pre-training based on DDPM and (b) feature extraction based on DDPM; first, a large number of unlabeled pantograph and overhead contact images, including visible light images and infrared light images, are used to perform self-supervised training on the denoising diffusion probability model DDPM, so that the denoising diffusion probability model DDPM can learn key semantic features from visible light and infrared images, and the pre-trained DDPM is used as a feature extractor, and the hierarchical change semantic decoding module and the audio-visual semantic decoding module are connected in sequence. The multimodal arc detection network in S2 is self-supervised trained, and its specific training steps are as follows: S201: Obtain an unlabeled pantograph visible light image and an unlabeled pantograph infrared light image to obtain image data ; S202: Based on The Markov chain with time steps is used to perform forward diffusion of image data using the denoising diffusion probability model DDPM. Add Gaussian noise to , until the image data Close to the standard normal distribution, we get image data close to the standard normal distribution. The expression of forward diffusion is as follows:

[0058]

[0059]

[0060] in, is the probability distribution of forward diffusion, For each image data in the forward diffusion process, is the initial image data, is the time step, is the number of diffusion steps, For the Image data for diffusion steps, is a Gaussian distribution, is a constant, is the initial visible light image, is the initial infrared image; S203: Calculate the mean function of the denoising diffusion probability model DDPM. The expression of the mean function is as follows:

[0061]

[0062] in, is the mean function of the denoising diffusion probability model DDPM, For the Image data for diffusion steps, is the number of diffusion steps, is Gaussian noise, is the Gaussian noise predicted by the neural network, are the parameters of the denoising diffusion probability model DDPM, is the noise attenuation based on the time step, is a constant, From time step 1 to time step All The cumulative product of is the index of the time step; S204: According to the mean function, the denoising diffusion probability model DDPM is used to perform reverse diffusion on the image data close to the standard normal distribution, remove the Gaussian noise in the image data close to the standard normal distribution, and reconstruct the image data that is similar to the initial image data. Similar images, get clean image data; the expression of back diffusion is as follows:

[0063]

[0064]

[0065] in, is the probability distribution of reverse diffusion, is the variance, is the covariance matrix.

[0066] S205: After completing S204, the loss function of the denoising diffusion probability model DDPM is calculated, and it is determined whether the loss value of the denoising diffusion probability model DDPM meets the convergence condition. If so, the self-supervised training of the denoising diffusion probability model DDPM is completed to obtain the trained denoising diffusion probability model DDPM. Otherwise, return to S202; the convergence condition of the denoising diffusion probability model DDPM can be set to when the loss value of the denoising diffusion probability model DDPM is less than a threshold value of 0.0001, the denoising diffusion probability model DDPM converges; wherein, the expression of the loss function of the denoising diffusion probability model DDPM is as follows:

[0067] in, is the loss function of the denoising diffusion probability model DDPM, is the expected value, is the number of diffusion steps, is the initial image data, is Gaussian noise, is the Gaussian noise predicted by the neural network, is from time step 1 to time step All The cumulative product of is the noise attenuation based on the time step.

[0068] S206: The trained denoising diffusion probability model DDPM is sequentially connected to the hierarchical variation semantic decoding module and the audio-visual semantic decoding module to obtain a trained multimodal arc detection network.

[0069] S207: inputting the image data into a multimodal arc detection network for training, obtaining an arc prediction label, and calculating a cross entropy loss function between the arc prediction label and the arc true label; S208: Determine whether the loss value of the cross entropy loss function meets the convergence condition. If so, obtain the trained multimodal arc detection network. Otherwise, return to S207. The convergence condition of the cross entropy loss function can be set to when the loss value of the cross entropy loss function is less than a threshold of 0.0001, it is judged to be converged. The expression of the cross entropy loss function is as follows:

[0070] in, is the calculation result of the cross entropy loss function, is the cross entropy loss function, Predict labels for arcs, is the real label of the arc.

[0071] The pantograph and contact network images of the actual line are obtained and input into the trained multimodal arc detection network. The denoising diffusion probability model DDPM, the hierarchical variation semantic decoding module and the audio-visual semantic decoding module in the multimodal arc detection network process them in sequence, and finally the arc prediction label is obtained. The specific steps are as follows: S301: Acquire the pantograph and overhead line contact images in the actual line, including the initial pantograph and overhead line contact visible light images Infrared image of initial pantograph and overhead contact .

[0072] S302: Input the pantograph and contact network images in the actual line into the denoising diffusion probability model DDPM in the trained multimodal arc detection network to obtain multi-scale feature information under different diffusion steps; The specific process is as follows: , is the visible light image of the initial contact between the pantograph and the overhead line Infrared image of initial pantograph and overhead contact Add Gaussian noise , generate a noisy image and , which is expressed as follows:

[0073]

[0074] in, The initial pantograph and overhead line contact visible light image with Gaussian noise added, is the initial infrared image of the pantograph and overhead line contact with Gaussian noise added. is the initial visible light image of the pantograph and overhead line contact, is the initial infrared image of the pantograph and overhead contact. From time step 1 to time step All The cumulative product of is the noise attenuation based on the time step, is Gaussian noise; The noisy image and Input it into the pre-trained denoising diffusion probability model DDPM to obtain the multi-scale feature information under the diffusion time step 𝑡, and finally obtain the multi-scale feature information under different diffusion steps. The expression is as follows:

[0075]

[0076] in, is the multi-scale feature information of the visible light image, is the multi-scale feature information of the infrared image, For different scales, the value is an integer from 0 to 4. is the weight of the denoising diffusion probability model DDPM, is the weight of the denoising diffusion probability model DDPM; when the scale When the scale is , it mainly captures the edge and gradient features of the image, which helps to identify the contour and structure of the object; when the scale is When , the main high-level abstract features are extracted; when the scale When , the size of the main feature map extracted is consistent with the input image, and contains rich detail information.

[0077] S303: inputting the multi-scale feature information under different diffusion steps into the hierarchical change semantic decoding module to obtain fusion features; like Figure 4 As shown in the figure, it is the structural diagram of the hierarchical semantic decoding module. By designing several hierarchical semantic decoders Establishing multi-scale feature information and the output of the layer-wise semantic decoder The mapping relationship between them is as follows: First, the multi-scale feature information under different diffusion steps is spliced, and its expression is as follows:

[0078]

[0079] in, For the The scale characteristics of a visible light image, For the The scale characteristics of the infrared image, For splicing operation, is the time step, is the splicing dimension; Then, the concatenated features are passed through two convolutional layers Conv. The first convolutional layer reduces the concatenated features, and the second convolutional layer outputs the same dimension as the first convolutional layer. The visible light image features and infrared light image features obtained after the two convolutional layers are used , and obtain the second feature after fusion , which is expressed as follows:

[0080] in, is the first feature after fusion, For addition operation, is the convolutional layer, For the The scale characteristics of a visible light image, For the The scale characteristics of infrared images; Next, the first feature after fusion The feature is combined with the value output by the semantic encoder of the previous layer to reduce the loss of features as the number of network layers increases. The expression is as follows:

[0081] in, is the second feature after fusion, For the The fusion features output by the layer-wise semantic decoder, For addition operation, is bilinear upsampling, Indicates that the sampling factor of bilinear upsampling is set to 2; Finally, the spatial attention mechanism and channel attention mechanism are used to correct the channel and spatial information of the fused features and output them. The expression is as follows:

[0082] in, For the The fusion features output by the layer-wise semantic decoder, For addition operation, is the spatial attention mechanism, It is the channel attention mechanism; In one embodiment of the present invention, five sequentially connected level-change semantic decoders are set, which can be expressed as:

[0083]

[0084]

[0085]

[0086]

[0087] in, is the fusion feature output by the first level change semantic decoder, is the fusion feature output by the second level change semantic decoder, is the fusion feature output by the third level change semantic decoder, is the fusion feature output by the 4th level change semantic decoder, is the fusion feature output by the fifth level change semantic decoder, is the first level change semantic decoder, is the second level change semantic decoder, is the third level change semantic decoder, is the 4th level change semantic decoder, It is the 5th level change semantic decoder.

[0088] S304: Input the fused features into the audio-visual semantic decoding module to obtain arc prediction labels. The specific process is as follows: like Figure 5 As shown in the figure, it is a structural diagram of the audio-visual semantic decoding module, which includes an audio cue encoding module and a sound source feature guidance module. In the cue encoding module, the VGGish model is used to extract audio features. This model is specially used to capture the time and spectrum information of audio. The audio is preprocessed and converted into a mono waveform with a sampling rate of 16kHz. The mel spectrogram is generated by Fourier transform and then input into the VGGish model to extract audio features. ,in ; Using audio features , visual clue features and audio noise characteristics Perform feature retrieval.

[0089] First, the visual features output by the hierarchical semantic decoding module Perform global average pooling to integrate them into a global vector to introduce visual context information and enhance audio-visual relevance. Then, the comprehensive visual features are converted into audio features through a fully connected layer. Visual cue features of the same dimension , which is expressed as follows:

[0090] in, is the visual clue feature, is the fully connected layer, is the global average pooling, Visual features output by the hierarchical semantic decoding module; Finally, through The audio features , visual clue features and audio noise characteristics Splice to generate fused audio prompt features , which is expressed as follows:

[0091] in, is the fused audio prompt feature, For splicing operation, is the splicing dimension; Through the fully connected layer, the fused features Projection , converting the fused features of high-dimensional multimodal into a more compact representation to facilitate subsequent steps. Its expression is as follows:

[0092] in, for projection; Using cross-modal attention Establish and The relationship between Fine-tune to strengthen the correlation between visual features and auditory features. The expression is as follows:

[0093]

[0094] in, For the updated prompt feature, To fine-tune the operation, is the transition parameter, For addition operation, It is a cross-modal attention mechanism; Will and As the key value of the cross-modal attention mechanism, and calculate the visual features that highlight the relevance of the audition , which is expressed as follows:

[0095]

[0096] in, To highlight the visual features of the audition correlation, is the key value of the cross-modal attention mechanism; Finally, the visual features that highlight the correlation between audition and listening are passed through two convolutional layers to obtain the arc prediction label, which is expressed as follows:

[0097] in, Predict labels for arcs, is the convolutional layer.

[0098] The beneficial effects of the present invention are as follows: the present invention utilizes a denoising diffusion probability model DDPM for self-supervised pre-training to extract visible light image features and infrared light image features, and can generate robust and discriminative features, thereby reducing the dependence of a multimodal arc detection network on large-scale labeled arc data, reducing training costs, and improving generalization capabilities. At the same time, a hierarchical variation semantic decoder is utilized to fuse multimodal data of visible light images and infrared images, and an audio-visual semantic decoder is utilized to effectively process the interaction of different modal features through the audio-visual correlation of arc features, thereby improving the accuracy and reliability of arc monitoring, further reducing the dependence of a multimodal arc detection network on labeled arc data, and enhancing robustness and generalization performance in complex scenarios.

[0099] Example 2 According to a pantograph arc monitoring method based on a multimodal arc detection network provided by the present invention, it is used in a certain subway line. For this subway line, a variety of non-contact detection equipment are deployed, including visible light cameras, infrared cameras, laser geometric parameter acquisition devices and ultraviolet arc detection devices. The installation of these devices ensures that any abnormality can be captured in time during the operation of the train and fed back to the data acquisition system in the car. A total of 3598 arc image data were selected from the line data obtained in different scenarios, which were used as data sets and divided into 3000 images for training sets and 598 images for test sets; at the same time, image enhancement technology can be used to process the data set, simulating the influence of complex environments such as rain, snow, fog and motion blur on arc detection, and divided into zero-shot and few-shot test subsets respectively.

[0100] In one embodiment of the present invention, the multimodal arc detection network of the present invention is carried out in the following hardware and software environment: Intel Gold 6240 CPU, A100 80GB GPU, Ubuntu22.04 operating system, Python3.11.4 and Py Torch 2.0.1; during the training process, the basic learning rate is set to 0.001, the batch size is 4, the AdamW optimizer is used, the weight decay coefficient is 1e-2, the momentum parameter is set to 0.9 and 0.999, and the cosine annealing learning rate strategy is adopted. The learning rate gradually decays from the maximum value of 0.001 to the minimum value of 0. The model is trained for a total of 280 epochs, and the hyperparameters are set to , .

[0101] In one embodiment of the present invention, four main indicators are used: intersection over union (IoU), average intersection over union (MIoU), precision and recall. The multimodal arc detection network in the pantograph arc monitoring method based on the multimodal arc detection network of the present invention is used to detect different arc shapes in complex scenarios for performance evaluation. The results are shown in Table 1:

[0102] As can be seen from Table 1, although the segmentation index IoU and recall rate of the multimodal arc detection network (DDPM-MILNet) in the present invention are 6.89% and 2.11% lower than those of large arcs, respectively, it still performs outstandingly among all models; DDPM-MILNet has significant advantages in detecting arcs of different sizes and shapes, especially in small arc detection.

[0103] In one embodiment of the present invention, DPM-MILNet, GAVS and AVSegFormer are tested for performance under different data amounts, such as Figure 6 As shown in the figure, the performance of different models on the arc data set is related to the amount of training data. The horizontal axis Proportion of training data is the amount of training data, and the vertical axes F-Score and MioU are both model evaluation indicators. The blue line is the relationship line of the multi-modal arc detection network (DDPM-MILNet), the gray line is the relationship line of GAVS, and the yellow line is the relationship line of AVSegFormer. Figure 6 It can be seen that DDPM-MILNet can achieve the best performance when other models use 100% data by using only 30% of the data; in addition, as the amount of data increases, the improvement of model performance tends to be slow, which shows that by using the denoising diffusion probability model DDPM for pre-training and fine-tuning the semantic decoder, the dependence of the multimodal arc detection network (DDPM-MILNet) on large amounts of data can be reduced.

[0104] In one embodiment of the present invention, the data set is processed using data enhancement technology to obtain zero-shot and few-shot test subsets; the performance of the model in complex environments, including rain, snow, fog, motion blur and Gaussian noise, which may interfere with the images captured by the camera. Figure 7 As shown in the figure, the generalization performance of different models is tested on different data sets. The horizontal axis is divided into three conditions: 1-shot, 3-shot and 5-shot, corresponding to 10 sample training, 30 sample training and 50 sample training respectively. The vertical axis F-Score and MioU are both model evaluation indicators. Figure 7 It can be seen that compared with other methods, the multimodal arc detection network (DDPM-MILNet) of the present invention can achieve excellent detection effects with a small amount of data under 1-shot, 3-shot and 5-shot conditions, showing excellent generalization ability. Even in a very limited training data set, the model can still achieve good generalization effects and has strong adaptability and robustness to sparse samples in complex environments.

[0105] The beneficial effects of the present invention are as follows: by using a denoising diffusion probability model DDPM as a pre-training model, feature extraction can be performed without any labeled data, which can solve the problem that a large amount of pantograph and contact network data in real scenarios are not labeled or difficult to accurately label. At the same time, the weights of the pre-trained denoising diffusion probability model DDPM are frozen in model training, and only lightweight classifier parameters are optimized, which not only reduces the dependence on large-scale labeled data and reduces training costs, but also improves the generalization ability of the model; the present invention utilizes the audio-visual correlation of arc features to effectively process the interaction of different modal features, further reduces the dependence on arc data through multimodal fusion, and enhances the robustness and generalization performance of the model in complex scenarios. The present invention improves the accuracy and reliability of arc detection, and provides a new solution for state monitoring and anomaly detection of rail transit systems.

Claims

1. A pantograph arc monitoring method based on a multi-modal arc detection network, characterized in that: The following steps are involved: S1: Establish a multimodal arc detection network based on denoising diffusion probability model DDPM and semantic decoder; S2: obtaining an unlabeled pantograph visible light image and an unlabeled pantograph infrared image, and performing self-supervised training on a multimodal arc detection network to obtain a trained multimodal arc detection network; S3: Obtain the pantograph and overhead line contact images in the actual line and input them into the trained multimodal arc detection network to obtain arc prediction labels and complete pantograph arc monitoring.

2. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 1, characterized in that: The multimodal arc detection network in S1 includes a denoising diffusion probability model DDPM, a hierarchical change semantic decoding module and an audio-visual semantic decoding module connected in sequence; The denoising diffusion probability model DDPM is used to capture the potential structures and patterns in visible light images and infrared light images, and obtain multi-scale feature information under different diffusion steps; The hierarchical change semantic decoding module includes a plurality of hierarchical change semantic decoders, which are used to process multi-scale feature information under different diffusion steps and output fusion features; The audio-visual semantic decoding module is used to process the fused features and output arc prediction labels.

3. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 1, characterized in that: The multimodal arc detection network is trained by self-supervision, and the specific training steps are as follows: S201: Acquire an unlabeled pantograph visible light image and an unlabeled pantograph infrared light image to obtain image data and a true arc label; S202: Based on the Markov chain, the denoising diffusion probability model DDPM is used to perform forward diffusion on the image data, and Gaussian noise is added to the image data to obtain image data close to the standard normal distribution; S203: Calculate the mean function of the denoising diffusion probability model DDPM; S204: according to the mean function, using the denoising diffusion probability model DDPM to perform reverse diffusion on the image data close to the standard normal distribution, remove Gaussian noise in the image data close to the standard normal distribution, and obtain clean image data; S205: After completing S204, the loss function of the denoising diffusion probability model DDPM is calculated, and it is determined whether the loss value of the denoising diffusion probability model DDPM meets the convergence condition. If so, the self-supervised training of the denoising diffusion probability model DDPM is completed to obtain the trained denoising diffusion probability model DDPM. Otherwise, return to S202; S206: sequentially connect the trained denoising diffusion probability model DDPM to the hierarchical variation semantic decoding module and the audio-visual semantic decoding module to obtain a multimodal arc detection network; S207: inputting the image data into a multimodal arc detection network for training, obtaining an arc prediction label, and calculating a cross entropy loss function between the arc prediction label and the arc true label; S208: Determine whether the loss value of the cross entropy loss function meets the convergence condition. If so, obtain a trained multimodal arc detection network. Otherwise, return to S207.

4. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 3, characterized in that: The expression of the forward diffusion of S202 is as follows: in, is the probability distribution of forward diffusion, For each image data in the forward diffusion process, is the initial image data, is the time step, is the number of diffusion steps, For the Image data for diffusion steps, is a Gaussian distribution, is a constant, is the initial visible light image, is the initial infrared image; The expression of the mean function of the denoising diffusion probability model DDPM in S203 is as follows: in, is the mean function of the denoising diffusion probability model DDPM, For the Image data for diffusion steps, is the number of diffusion steps, is Gaussian noise, is the Gaussian noise predicted by the neural network, are the parameters of the denoising diffusion probability model DDPM, is the noise attenuation based on the time step, is a constant, From time step 1 to time step All The cumulative product of is the index of the time step; The expression of the reverse diffusion in S204 is as follows: in, is the probability distribution of reverse diffusion, is the variance, is the covariance matrix.

5. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 3, characterized in that: The loss function of the denoising diffusion probability model DDPM is expressed as follows: in, is the loss function of the denoising diffusion probability model DDPM, is the expected value, is the number of diffusion steps, is the initial image data, is Gaussian noise, is the Gaussian noise predicted by the neural network, is from time step 1 to time step All The cumulative product of is the noise attenuation based on the time step.

6. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 3, characterized in that: The expression of the cross entropy loss function is as follows: in, is the calculation result of the cross entropy loss function, is the cross entropy loss function, Predict labels for arcs, is the real label of the arc.

7. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 1, characterized in that: The specific steps of S3 are as follows: S301: Acquire the contact image of the pantograph and the overhead line in the actual line; S302: input the pantograph and contact network images in the actual line into the denoising diffusion probability model DDPM in the trained multimodal arc detection network to obtain multi-scale feature information under different diffusion steps; S303: inputting the multi-scale feature information under different diffusion steps into the hierarchical change semantic decoding module to obtain fusion features; S304: Input the fused features into the audio-visual semantic decoding module to obtain arc prediction labels.

8. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 7, characterized in that: The expression of the multi-scale feature information under different diffusion steps is as follows: in, is the multi-scale feature information of the visible light image, is the multi-scale feature information of the infrared image, For different scales, the value is an integer from 0 to 4. is the weight of the denoising diffusion probability model DDPM, is the weight of the denoising diffusion probability model DDPM, The initial pantograph and overhead line contact visible light image with Gaussian noise added, is the initial infrared image of the pantograph and overhead line contact with Gaussian noise added. is the initial visible light image of the pantograph and overhead line contact, is the initial infrared image of the pantograph and overhead contact. From time step 1 to time step All The cumulative product of is the noise attenuation based on the time step, is Gaussian noise.

9. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 7, characterized in that: The expression of the fusion feature is as follows: in, For the The fusion features output by the level-by-level semantic decoder, For addition operation, is the spatial attention mechanism, is the second feature after fusion, is the first feature after fusion, For the The fusion features output by the level-by-level semantic decoder, is bilinear upsampling, The sampling factor is set to 2, For the The scale characteristics of a visible light image, For the The scale characteristics of the infrared image, For splicing operation, is the multi-scale feature information of the visible light image, is the multi-scale feature information of the infrared image, is the number of diffusion steps, The splicing dimension.

10. The pantograph arc monitoring method based on a multi-modal arc detection network according to claim 7, characterized in that: The expression of the arc prediction label is as follows: in, Predict labels for arcs, is the convolution function, To highlight the visual features of the audition correlation, For addition operation, is the fusion feature output by the last level change semantic decoder, is the cross-modal attention, is the key value, is the transition parameter, For the updated prompt feature, is the fully connected layer, To fine-tune the operation, for The projection of is the fused audio prompt feature, is the visual clue feature, is the audio feature, is the audio noise characteristic, is the global average pooling, The splicing dimension.

Images