Catenary dropper string defect detection method based on CNN and Transformer fusion

Abstract

The present invention discloses a method for detecting defects in contact network suspension strings based on the fusion of CNN and Transformer, which relates to the field of defect detection technology. The method collects images of contact network suspension strings and performs image enhancement processing to obtain a sample set of suspension string defects; a convolution module is constructed using a variable convolutional network based on constraints, and a self-attention module is constructed based on an improved high-efficiency multi-head self-attention mechanism. The convolution module and the self-attention module are deeply fused based on the optimal module allocation ratio to generate a multi-block cross-hybrid network to improve the FasterRCNN network. The improved network is trained and verified based on the suspension string defect sample set, and a trained model is obtained and deployed in a suspension string detection device. The images of the contact network suspension strings are captured in real time and input into the suspension string detection device for suspension string defect detection, thereby identifying the suspension string defects of the contact network. The present application can be applied to complex natural scene environments, and improves the accuracy and recall rate of suspension string defect recognition under real complex natural conditions.

Description

Technical Field

[0001] The present invention relates to the field of defect detection technology, and in particular to a high-speed railway contact network dropper string defect detection method based on deep cross-fusion of CNN and transformer. Background Art

[0002] Dropper strings are a critical component of high-speed rail catenary systems, ensuring smooth and continuous current flow through EMUs and mitigating vibrations between the contact wire and the catenary cables. However, dropper strings are susceptible to temperature, climate, and high-frequency vibrations, leading to frequent loosening, breakage, and other issues. This can impact pantograph current flow, or even damage the pantograph or the contact wire, leading to train failures. Therefore, real-time detection and early warning of dropper string defects are crucial for ensuring the safety and reliability of the catenary system and are crucial for the safe operation of high-speed railways.

[0003] Most existing methods for detecting catenary dropper defects rely on machine learning and neural network models, but they have the following disadvantages:

[0004] (1) Affected by the actual scene, the defect recognition effect of the suspension string is poor in natural scene environments such as rainy days, foggy days, strong sunlight, and night;

[0005] (2) When the suspension string is blocked, the defects of the suspension string cannot be accurately identified, and defects may be missed. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a contact network dropper string defect detection method based on the fusion of CNN and Transformer, which helps to solve the problems of poor defect recognition effect and low recognition accuracy of the current contact network dropper string defect detection method when the dropper string is blocked and in complex scenes.

[0007] The object of the present invention is achieved through the following technical solutions:

[0008] The present invention provides a method for detecting defects in catenary droppers based on the fusion of CNN and Transformer, comprising:

[0009] Collecting images of catenary droppers and performing image enhancement processing on the images to obtain a dropper string defect sample set;

[0010] A convolutional module is constructed using a constrained variable convolutional network, and a self-attention module is constructed based on an improved high-efficiency multi-head self-attention mechanism. The convolutional module and the self-attention module are deeply fused based on the optimal module allocation ratio to generate a multi-block cross-hybrid network.

[0011] Using the multi-block cross-hybrid network to improve the FasterRCNN network, and training and verifying the improved FasterRCNN network based on the dropper string defect sample set to obtain a trained FasterRCNN improved model;

[0012] The improved FasterRCNN model is deployed in the dropper string detection equipment to capture the images of the high-speed railway contact network dropper strings in real time and input them into the dropper string detection equipment for dropper string defect detection to identify dropper string defects in the contact network.

[0013] Furthermore, the collecting of the catenary dropper string images and performing image enhancement processing on the dropper string images to obtain a dropper string defect sample set specifically includes:

[0014] Collect images of overhead catenary strings during high-speed railway operation;

[0015] Based on the improved image enhancement algorithm, the catenary string images are enhanced. N mask regions are randomly generated in any string image. Based on the number N of mask regions, the mask regions that block the key features of the string are filtered out to obtain a sample set of string defects.

[0016] The sample set of suspension string defects is divided into training samples and validation samples.

[0017] Furthermore, filtering the masked areas that block the key features of the suspension string according to the number N of masked areas specifically includes:

[0018] If N = 1, the single mask area in the string image that completely blocks the string is filtered out;

[0019] If N=2, the two mask areas in the string image that block the upper and lower ends of the string are filtered out;

[0020] If N ≥ 3, filter out the single mask area that completely blocks the hanging string in the hanging string image, and filter out any two mask areas that block the upper and lower ends of the hanging string in the hanging string image.

[0021] Furthermore, the convolution module is constructed using a constraint-based variable convolutional network, and the self-attention module is constructed based on the improved high-efficiency multi-head self-attention mechanism. The convolution module and the self-attention module are deeply integrated based on the optimal module allocation ratio to generate a multi-block cross-hybrid network, specifically including:

[0022] Based on the height-to-width ratio of the aspect ratio drop string, a constraint relationship is used to constrain the height-to-width coordinate ratio of the sampling point position in the variable convolutional network. At the same time, the height-to-width coordinates of the sampling point position are restricted to not exceed the height-to-width of the input feature map. This results in a constraint-based variable convolutional network, and a convolution module is constructed using the constraint-based variable convolutional network.

[0023] Perform spatial dimensionality reduction operations on the key vector K and value vector V in the original Transformer self-attention mechanism to obtain an improved high-efficiency multi-head self-attention mechanism and construct a self-attention module based on the improved high-efficiency multi-head self-attention mechanism;

[0024] Based on the backbone network architecture in the FasterRCNN model, the number of convolutional modules and self-attention modules in the backbone network architecture is allocated according to the optimal module allocation ratio, and the convolutional modules and self-attention modules are fused in a new paradigm to generate a multi-block cross-hybrid network.

[0025] Furthermore, the optimal module allocation ratio is specifically: number of convolution modules: number of self-attention modules = 7:2.

[0026] Furthermore, the method of improving the FasterRCNN network by using the multi-block cross-hybrid network and training and verifying the improved FasterRCNN network based on the dropper string defect sample set to obtain a trained FasterRCNN improved model specifically includes:

[0027] The backbone network architecture of the FasterRCNN network is replaced with the network architecture of the multi-block cross-hybrid network to obtain an improved FasterRCNN model;

[0028] The improved FasterRCNN network is trained based on the training samples in the suspension string defect sample set. After the training is completed, the model is verified using the verification samples to obtain the trained and verified FasterRCNN improved model.

[0029] Beneficial effects of the present invention: The present invention provides a method for detecting contact network dropper string defects based on the fusion of CNN and Transformer, the method collects contact network dropper string images and performs image enhancement processing on the dropper string images to obtain a dropper string defect sample set; a convolution module is constructed using a variable convolution network based on constraints, and a self-attention module is constructed based on an improved high-efficiency multi-head self-attention mechanism, and the convolution module and the self-attention module are deeply fused based on the optimal module allocation ratio to generate a multi-block cross-hybrid network; the multi-block cross-hybrid network is used to improve the FasterRCNN network, and the improved FasterRCNN network is trained and verified based on the dropper string defect sample set to obtain a trained FasterRCNN improved model; the FasterRCNN improved model is deployed in a dropper string detection device, and the contact network dropper string images of the high-speed railway are captured in real time and input into the dropper string detection device for dropper string defect detection to identify dropper string defects in the contact network. The present application obtains a dropper string defect sample set by performing image enhancement processing on the dropper string images, increases the dropper string defect samples, and solves the problem of dropper strings being blocked and the small number of dropper string training samples. At the same time, this application meets the requirements of suspension string feature and improves the ability to identify suspension string defects by constructing a convolution module based on a constraint-based variable convolutional network. At the same time, a self-attention module is constructed based on the improved high-efficiency multi-head self-attention mechanism, and the convolution module and the self-attention module are deeply fused based on the optimal module allocation ratio to generate a multi-block cross-hybrid network. The CNN and transformer are cross-fused to solve the problem of poor defect recognition in natural scene environments such as rainy days, foggy days, strong sunlight, and night for suspension strings. It can also accurately identify defects of small targets that are obscured and far away, thereby improving the recall rate and precision of suspension string defect recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the method for detecting defects in a catenary dropper string based on the fusion of CNN and Transformer of the present invention;

[0031] Figure 2 It is a schematic diagram of the existing types of dropper string defects;

[0032] Figure 3 This is a schematic diagram of the problem of identifying the defects of the suspension string in a real application environment;

[0033] Figure 4 This is an example of an existing model misidentifying a hanging string;

[0034] Figure 5 This is the improved model architecture diagram of FasterRCNN;

[0035] Figure 6 is a sample image of the hanging string after image enhancement;

[0036] Figure 7 It is a schematic diagram of convolution sampling for defect recognition using different convolutional neural networks;

[0037] Figure 8 It is a schematic diagram of the principle of bilinear interpolation;

[0038] Figure 9 This is a comparison diagram of the backbone network architecture of the traditional hybrid network and the multi-block cross-fusion hybrid network;

[0039] Figure 10 This is a schematic diagram of the network structure formed by the cross-fusion of CB blocks and TB blocks in a single stage of the backbone network of this application;

[0040] Figure 11 It is a schematic diagram of the CB block structure of the present invention;

[0041] Figure 12 It is a schematic diagram of the TB block structure of the present invention. DETAILED DESCRIPTION

[0042] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described with reference to the accompanying drawings.

[0043] The catenary system is a vital component of electrified railways and the power source for high-speed train operation. Droppers transmit vibration, force, and current between the contact wire and the load-bearing cables. They are crucial components for improving the current-carrying and load-bearing performance of the catenary system and are a key component for ensuring the safe operation of the electrified railway catenary system. Droppers typically consist of dropper wires and dropper clamps at each end. In a flexible catenary system, droppers can increase the number of contact wire suspension points per span without adding additional support columns, thereby improving the overall elasticity of the catenary system and enhancing the sag of the contact wire.

[0044] During railway transportation, if a dropper string breaks and is not promptly detected and repaired, the contact wire at the location of the break will sink, causing the contact wire to become non-parallel to the track and affecting the pantograph's current flow. A completely broken dropper string, when drooping, can easily become entangled with the contact wire under the influence of wind, potentially damaging the pantograph or the contact wire, leading to train failures and endangering passenger life and property. Therefore, real-time detection and early warning of dropper string defects to ensure the safety and reliability of the catenary are of great significance to the safe operation of high-speed railways.

[0045] Therefore, this application cross-integrates CNN and transformer, which can identify suspension string defects in natural scene environments such as rainy days, foggy days, strong sunlight and night. It can also accurately identify suspension string defects that are obscured and small targets at a distance, thereby improving the recall rate and precision of suspension string defect identification.

[0046] Reference Figure 1 As shown, Figure 1 A method for detecting catenary dropper defects based on the fusion of CNN and Transformer is presented, including:

[0047] S1: collecting catenary dropper string images and performing image enhancement processing on the dropper string images to obtain a dropper string defect sample set;

[0048] S2: Construct a convolutional module using a constrained variable convolutional network, and construct a self-attention module based on an improved high-efficiency multi-head self-attention mechanism. The convolutional module and the self-attention module are deeply fused based on the optimal module allocation ratio to generate a multi-block cross-hybrid network.

[0049] S3: using the multi-block cross-hybrid network FasterRCNN network to improve, and based on the dropper string defect sample set, training and verifying the improved FasterRCNN network to obtain a trained FasterRCNN improved model;

[0050] S4: Deploy the improved FasterRCNN model on the dropper string detection equipment, capture the high-speed railway contact network dropper string images in real time and input them into the dropper string detection equipment to perform dropper string defect detection and identify dropper string defects in the contact network.

[0051] The existing FasterRCNN model cannot meet the requirements of string defect detection in complex real-world scenarios, and suffers from a large number of low recall and low-precision recognition issues. In addition, real-time image capture under high-speed motion requires low inference latency, which the existing FasterRCNN model cannot meet. Therefore, this application mainly uses the FasterRCNN network to improve the string defect detection.

[0052] Specifically, refer to Figure 2 As shown in the figure, due to the long-term exposure of the suspension strings to the external environment, they are significantly affected by climatic factors and the operating environment. Furthermore, the suspension string components are susceptible to high-frequency vibration, which can cause stress fatigue, string breakage, and string detachment. The number of defects increases exponentially. There are many types of suspension string defects. For ease of classification and identification, we have divided them into five major defects: string breakage, string detachment (string clamp detachment at the upper or lower end), string bending, string loosening, and improper string installation. Figure 2 The drawing in part a1 shows the defect of broken suspension string, the drawing in part d1 shows the defect of bent suspension string, the drawings in parts e1 and f1 show the defect of loose suspension string, and the drawings in parts b1 and c1 respectively show the defects of upper and lower detachment of suspension string.

[0053] China's high-speed rail network covers a vast area, boasts a long mileage, and features complex and diverse climate and topography. Influenced by natural weather, seasonal changes, lighting conditions, and obstructions to the string, string defect identification is extremely challenging in practical applications. Figure 3 The problems encountered in the identification of hanging strings in practical applications are listed. Figure 3 Part a2 in the figure shows the problem of image smearing and blurring caused by the high-speed movement of the train; the figure in part b2 shows the problem of dim tunnel lighting and the entire image being dim; the figure in part c2 shows the problem of a dirty lens and partial obstruction of the image; the figure in part d2 shows the problem of the image encountering a complex background, with the target object and the background intertwined, making it difficult to distinguish the target string; the figures in parts e2 and f2 show that the string in the image is blocked by the pantograph or interrupted by the arc. The figures in parts g2, h2, i2, and j2 show that the string image is blurred and difficult to identify due to adverse weather conditions such as heavy fog, night, heavy rain, and strong sunlight. It can be seen that in real-world application environments, the string recognition process will encounter various complex situations, which greatly increases the difficulty of identifying string defects.

[0054] Reference Figure 4 As shown in the figure, the existing technology uses FasterRCNN and its optimized improved model, or CNN and Transformer fusion model to identify the defects of hanging strings, but there are the following problems: 1. Positioning tubes, electrical connection wire joints, electrical connection wires and windproof wires are mistakenly identified as hanging strings, such as Figure 4 2. The hanging string is blocked and mistakenly identified as a broken hanging string defect, such as Figure 4 As shown in the figure c3; 3. Missed recognition of small targets at long distances, such as Figure 4 As shown in the figure f3 in the figure, in real-world scenarios, dropper strings and their defects suffer from low recall and precision. Existing models cannot meet the requirements for dropper string defect identification. Therefore, it is necessary to conduct extensive analysis of the internal characteristics of the dropper strings. Based on the analysis results, a corresponding model should be designed and constructed to meet the requirements for dropper string defect identification, thereby ensuring the safe operation of the high-speed rail contact network.

[0055] This application makes three improvements based on the FasterRCNN model to realize the detection of dropper string defects in obscured conditions, and to improve the recall rate and precision of dropper string defect detection, thereby providing strong support for the safe and intelligent operation of high-speed rail contact networks.

[0056] The existing FasterRCNN model architecture mainly consists of the input layer, backbone network (also known as the main network), neck, head detection head and output layer. The backbone network is divided into four stages, each containing a certain number of convolutional modules.

[0057] The improved model of this application is as follows Figure 5 As shown, three major improvements were made: 1) Based on the existing image enhancement methods and the inherent characteristics of the suspension string, a Limited cutout algorithm (i.e., L-cutout) was proposed to solve the problem of small number of suspension string training samples; 2) In view of the limitations of variable convolution on the recognition of suspension string defects and the huge differences in the aspect ratio of the suspension strings, a constrained variable convolution (C-DCV) was proposed to improve the recognition performance of the suspension strings; 3) In view of the dependence of image recognition on CNN local perception and ViT long range, a new backbone network was constructed to detect suspension string defects. Inspired by the residual network, this application replaces the 3x3 convolution with C-DCV to construct a new type of residual network, which we call CNN block (abbreviated as CB in this application). This application also optimizes the efficiency of HMSA, and combines it with FFN to construct an efficient self-attention module, called transformer block (abbreviated as TB in this application).

[0058] Furthermore, in one embodiment, collecting the catenary dropper string image and performing image enhancement processing on the dropper string image to obtain the dropper string defect sample set specifically includes:

[0059] Collect images of overhead catenary strings during high-speed railway operation;

[0060] Based on the improved image enhancement algorithm, the catenary string images are enhanced. N mask regions are randomly generated in any string image. Based on the number N of mask regions, the mask regions that block the key features of the string are filtered out to obtain a sample set of string defects.

[0061] The sample set of suspension string defects is divided into training samples and validation samples.

[0062] The filtering of the masked areas that block the key features of the suspension string according to the number N of masked areas specifically includes:

[0063] If N = 1, the single mask area in the string image that completely blocks the string is filtered out;

[0064] If N=2, the two mask areas in the string image that block the upper and lower ends of the string are filtered out;

[0065] If N ≥ 3, filter out the single mask area that completely blocks the hanging string in the hanging string image, and filter out any two mask areas that block the upper and lower ends of the hanging string in the hanging string image.

[0066] In the specific practice process, this application first performs image enhancement on the collected hanging string images. Image enhancement is beneficial to recognition scenarios such as image classification, target detection, and semantic segmentation. Usually, image enhancement uses traditional image enhancement algorithms, such as flip (Flip, Rotation, Scale, etc.). In recent years, new image enhancement algorithms such as cutout, mixup, and cutmix have emerged, which can further improve image recognition capabilities. However, if these new algorithms are directly applied to hanging string defect detection, the detection recall rate and precision rate will be seriously reduced. The reason for this is mainly because these algorithms use random masks, image overlays and other operations to cover up the key features of the hanging strings, that is, they cover up the dropper clamps at both ends, leaving the dropper wire, which causes the model to identify the lines as hanging strings in the subsequent recognition process.

[0067] Reference to the enhanced recognition results of existing image enhancement technology on hanging string samples Figure 6 As shown, Figure 6 The figure in part a4 represents the standard hanging string sample; the figures in parts b4 and c4 show the image enhancement results that retain the key hanging string information; the figures in parts d4, e4 and f4 show the image enhancement results that block the key hanging string information.

[0068] To address the problems of the above-mentioned image enhancement algorithms, this application proposes an improved algorithm based on the cutout method, called the Limited cutout (L-cutout) algorithm. The algorithm randomly generates a mask area, but the mask area (also called mask block) cannot block the significant features of the suspension string, that is, at least one suspension string clip is left unblocked (such as Figure 6 (See parts b4 and c4 in the accompanying figure). L-cutout has the following advantages in identifying dropper string defects: 1) It improves the ability to identify obscured dropper strings; 2) It increases the number of dropper string defect samples, which is conducive to sufficient model training.

[0069] The general idea of the L-cutout algorithm of this application is to randomly generate mask areas in the hanging string image, but it is necessary to filter out the mask areas that block the key features of the hanging string (such as Figure 6 The specific processing of the L-cutout algorithm is as follows:

[0070] First, let the position and size of the randomly generated mask area in the image be (X mask, Y mask ,W mask , H mask ) indicates that X mask ,Y mask Indicates the coordinates of the starting point, W mask, Hmask Indicates the width and height of the mask in the image. The ground truth of the hanging string sample is marked with (X GT ,Y GT ,W GT ,H GT ) indicates that X GT ,Y GT Indicates the starting coordinates, W GT ,H GT Indicates height and width. The physical hanging string is viewed in the image, L dw Indicates the length of the suspension string in the figure, L dc Indicates the length of the string clamp.

[0071] Secondly, the algorithm filters the mask area. The filtering logic of the algorithm is as follows:

[0072]

[0073]

[0074] Among them, the specific implementation process of the algorithm's filtering logic is as follows: according to the processing logic of the second line, N mask blocks are randomly generated in the input sample; when the number of mask blocks N is 1, the single mask block that completely obscures the suspension string in the suspension string image is filtered out, and the suspension string sample is obtained according to the processing logic of lines 3 to 17; when the number of mask blocks N is 2, the two mask blocks that obscure the upper and lower ends of the suspension string in the suspension string image are filtered out, and the suspension string sample is obtained according to the processing logic of lines 18 to 29; when the number of mask blocks N≥3, each mask block enters the processing logic branch of N=1 for filtering, and at the same time, any two mask blocks enter the processing logic branch of N=2 for filtering.

[0075] Furthermore, in one embodiment, the convolution module is constructed using a constraint-based variable convolutional network, and the self-attention module is constructed based on an improved high-efficiency multi-head self-attention mechanism. The convolution module and the self-attention module are deeply fused based on the optimal module allocation ratio to generate a multi-block cross-hybrid network, specifically including:

[0076] Based on the height-to-width ratio of the aspect ratio drop string, a constraint relationship is used to constrain the height-to-width coordinate ratio of the sampling point position in the variable convolutional network. At the same time, the height-to-width coordinates of the sampling point position are restricted to not exceed the height-to-width of the input feature map. This results in a constraint-based variable convolutional network, and a convolution module is constructed using the constraint-based variable convolutional network.

[0077] Perform spatial dimensionality reduction operations on the key vector K and value vector V in the original Transformer self-attention mechanism to obtain an improved high-efficiency multi-head self-attention mechanism and construct a self-attention module based on the improved high-efficiency multi-head self-attention mechanism;

[0078] Based on the backbone network architecture in the FasterRCNN model, the number of convolutional modules and self-attention modules in the backbone network architecture is allocated according to the optimal module allocation ratio, and the convolutional modules and self-attention modules are fused in a new paradigm to generate a multi-block cross-hybrid network.

[0079] Among them, the optimal module allocation ratio is specifically: the number of convolution modules (CB): the number of self-attention modules (TB) = 7:2, which is the optimal allocation ratio of this application. Furthermore, the allocation of the number of convolution modules and the number of self-attention modules in the backbone network architecture according to the optimal module allocation ratio is specifically: according to the number of modules in each stage in the backbone network, the modules in the stage are divided into multiple groups of modules in the form of 7 CB blocks and 2 TB blocks as a group, and the multiple groups of modules are connected in series to form a stage in the backbone network. This application mainly allocates and merges the number of modules of Stage2, Stage3 and Stage4 in the backbone network. In other embodiments of the present application, the module allocation ratio can also be adjusted accordingly according to the architecture of the network in the model, and this application will not go into details here.

[0080] In specific practice, deformable convolution can be closer to the shape and size of the object when sampling, while the sampling of convolutional neural network cannot change with the change of object shape. Limited by the fixed geometric structure of CNN module, convolutional neural network is more friendly to objects with equal height and width, while deformable convolutional neural network is more advantageous for identifying objects with diverse shapes. Since the width and height ratio of the suspension string are not equal, the height is usually much larger than the width. The use of convolutional neural network is not conducive to the identification of suspension string. Therefore, a variable convolutional neural network is selected to identify suspension string defects.

[0081] Reference Figure 7 This application introduces three convolution sampling graphs based on the hanging string image, where Dropper represents the hanging string sample, SC represents the standard convolution, DCV2 represents the deformable convolution, and C-DCV represents the constraint-based deformable convolution. Figure 7 Figure a5 shows the actual image of the hanging string, Figure b5 shows the sampling diagram of the standard convolution, Figure c5 shows the sampling diagram of the deformable convolution, and Figure d5 shows the sampling diagram of the constraint-based deformable convolution. It can be seen that C-DCV can change the sampling process accordingly according to the aspect ratio of the hanging string.

[0082] Directly applying DCV1 and DCV2 to dropper string defect detection slightly improved the overall model performance, but failed to fully utilize the capabilities of variable convolution. The main reason is that although variable convolution collects and recognizes objects based on shape, it lacks a specific feature specific to dropper strings, which are much taller than they are wide.

[0083] After an in-depth study of the applicable objects of deformable convolution and analysis based on the inherent characteristics of suspension strings, this application proposes a convolutional network with the inherent characteristics of the special case of suspension strings, called Constraint-based deformable ConvNets (C-DCV). It can not only be used for suspension string defect detection, but also for the recognition of other objects that are taller than they are wide.

[0084] Specifically, the constraint-based deformable convolution algorithm (hereinafter referred to as C-DCV algorithm) is shown in the following formula (1):

[0085]

[0086] Among them, H and W represent the height and width of the input feature map, p k represents the original sampling point in the input feature map, h pk and w pk Indicates p k The height and width coordinates of △w k and △h k Indicates p k Corresponding offset; x represents p k Position characteristics; α represents a scalable parameter, the purpose is to ensure △w k and △h k The proportional relationship between them is limited. k +offset k (△w k ,△h k ) is p k The sampling point coordinates are as follows: Since DCV2 generates △w without restriction through convolution operation k and △h k , resulting in exceeding the boundary. Therefore, the C-DCV of this application adopts a constraint relationship, constraining h pk and w pk The proportional relationship between them makes it close to the height-width ratio of the hanging string, while limiting h pk and w pk Cannot exceed the range of the input feature map.

[0087] At the same time, offset k (△w k ,△h k ) Generate △w through convolution k and △hk , which may be a decimal, so p+p k +offset k (△w k ,△h k ) coordinates are not integers, so it is necessary to obtain the integer coordinate positions in the image through bilinear interpolation, then obtain the corresponding eigenvalues based on the integer coordinates, and decompose the decimal coordinates into four adjacent integer coordinate points to calculate the result. The specific operation process is as follows:

[0088] The principle of bilinear interpolation is as follows Figure 8 As shown, the training obtains the coordinates P k (X,Y) is not an integer, P k The corresponding four adjacent coordinate points in the figure are Q11(x1,y1), Q12(x1,y2), Q21(x2,y1), Q22(x2,y2), and their corresponding eigenvalues are f(Q 11 ), f(Q 12 ), f(Q 21 ), f(Q 22 ). We calculate the P value through the bilinear interpolation method. The principle of bilinear interpolation can be implemented by referring to the existing technology (http: / / www.cnblogs.com / yssongest / p / 5303151.html).

[0089] First, perform linear interpolation in the x direction to obtain formula (2):

[0090]

[0091] Then perform linear interpolation in the y direction to obtain formula (3):

[0092]

[0093] All in all, the final result of bilinear interpolation is formula (4):

[0094]

[0095] Since image bilinear interpolation only uses 4 adjacent points, the denominator of the above formula 4 is 1. Here P k The (X,Y) calculation process is shown in the following formula (5):

[0096]

[0097] Furthermore, in one embodiment, the method of improving the FasterRCNN network by using the multi-block cross-hybrid network and training and verifying the improved FasterRCNN network based on the dropper string defect sample set to obtain a trained FasterRCNN improved model specifically includes:

[0098] The backbone network architecture of the FasterRCNN network is replaced with the network architecture of the multi-block cross-hybrid network to obtain an improved FasterRCNN model;

[0099] The improved FasterRCNN model is trained based on the training samples in the suspension string defect sample set. After the training is completed, the model is verified using the verification samples to obtain the trained and verified FasterRCNN improved model.

[0100] In practice, the identification of suspension string defects requires not only high recall and high precision, but also real-time image inference at no less than 12fps. Although many existing algorithms are capable of meeting the requirements for suspension string defect identification, their recall and precision are insufficient, seriously affecting the safe operation of high-speed railways. To address this issue, this application proposes a new model that combines CNN and transformer to solve the problem of suspension string defect identification. First, a high-performance and efficient multi-head attention is proposed to improve the computational efficiency and reduce weight parameters of the transformer, and then a new paradigm of fusion of CNN and transformer is constructed.

[0101] The specific processing process is as follows:

[0102] 1. High-performance and high-efficiency multi-head self-attention mechanism (HE-MHSA):

[0103] The self-attention mechanism is an attention mechanism based on scaled dot product. It projects the input vector of the original sequence into three different spaces as query, key and value, namely the query vector Q, key vector K and value vector V. Each input in the sequence will perform attention calculation on the entire sequence, including itself.

[0104] The computational cost and memory consumption of the original transformer's self-attention increase exponentially with increasing input image resolution. Many works have attempted to reduce the computational cost of self-attention by reducing the spatial resolution of the input K and V vectors. However, these operations can lose important feature information and even introduce new noise, weakening the representational power of subsequent MHSA.

[0105] To address the above issues, we propose HE-MHSA, which not only reduces computational and memory overhead but also maintains the model's representational capabilities. The specific implementation process is as follows: First, we construct a new spatial dimensionality reduction operation, as shown in Equation (6). Its purpose is to reduce the dimensions of vectors K and V. The spatial dimensionality reduction operation proposed in this application neither loses feature information nor introduces new noise.

[0106]

[0107] Among them, SR(.), AVG(.), and DW(.) represent the usual spatial dimensionality reduction operation, the average pooling operation, and the variable convolution operation respectively. These three types of operations perform dimensionality reduction operations on the key vector K, and obtain the vector K after the usual spatial dimensionality reduction operation, the average pooling operation, and the variable convolution operation respectively. sr , K avg , K dw . Perform the above three types of dimensionality reduction operations on vector V respectively, and obtain vector V after the usual spatial dimensionality reduction operation, average pooling operation and variable convolution operation respectively. sr , V avg , V dw Then the reduced dimensionality vector K sr , K avg , K dw Add to obtain the low-resolution vector K*, that is, K sr +K avg +K dw →K*, and at the same time, the reduced-dimensional vector V sr , V avg , V dw Add to obtain the low-resolution vector V*, that is, V sr +V avg +V dw →V*.

[0108] Secondly, z, V* and K* are used as input and applied to MHSA to obtain the feature Z, as shown in formula (7):

[0109]

[0110] Here, z represents the feature image, which serves as the query input of the transformer block. The low-resolution vectors V* and K* represent the key and value inputs respectively. The output feature Z is obtained by processing the transformer block.

[0111] 2. New paradigm of CNN and ViT integration

[0112] The backbone network of FasterRCNN consists of four stages, such as Figure 9As shown in the figure in part a6, each stage consists of multiple overlapping CNN blocks. The fusion network of CNN and ViT is as follows Figure 9 As shown in the accompanying drawings, the parts b6, c6, d6, e6 and f6 in FIG. Figure 9 The networks shown in parts b6, c6, and d6 of the accompanying figures replace the last one or several stages of the FasterRCNN backbone with multiple overlapping transformer blocks to construct a new detector. While these approaches significantly improve image recognition, they increase computational complexity and weight parameters, resulting in significant inference latency and failing to meet the real-time recognition requirements on high-speed trains. Figure 9 The networks shown in the figures e6 and f6 in the figure make a trade-off between latency and accuracy, but cannot meet the high recall and high precision requirements of hanging string recognition.

[0113] Reference Figure 9 As shown in the g6 part of the figure, based on the above problems, this application proposes a new high-precision and high-efficiency CNN and ViT fusion new paradigm, called multi-block cross-fusion hybrid network, that is, multi-block cross-fusion hybrid network (MCHN), the network adopts (CBxN C +TBxN T )xL method replaces the existing next-ViT (CB xN+TB x1)xL.

[0114] The main differences between the MHCN and next-ViT in this application are as follows:

[0115] 1) The number of transformer blocks in each stage of this application is variable, not fixed to 1, which can further capture global features and long-range dependencies;

[0116] 2) The constrained variable convolution used by the CNN block of MHCN in this application replaces the standard variable convolution;

[0117] 3) The transformer block of this application uses a high-performance and efficient multi-head self-attention mechanism to replace the MHSA of the original transformer.

[0118] Based on the above improvements, the MHCN network model constructed in this application has two advantages: 1) It can not only meet the requirements of high precision and high recall, but also meet the requirements of real-time reasoning at high speed. Furthermore, according to the requirements of string detection, this method makes a better compromise between precision and efficiency and meets the requirements of actual scenarios. 2) The MHCN network is compatible with all traditional methods. Traditional modes are Figure 9 The network structure shown in the figure in part g6 is a special case. For example, when NT=1, the MHCN network evolves into Figure 9 The network shown in the figure in part e6 of the figure. In addition, in the MHCN of this application, the allocation ratio of CNN blocks and transformer blocks in stages 2 to 4 is more flexible, and the allocation ratio of related modules can be configured according to the actual scenario requirements.

[0119] This application is fixed by Figure 9 The total number of modules in stage1, stage2 and stage4 is used to study the reasonable allocation ratio of CNN block and transformer block, so as to further optimize the performance of the model. For a fairer comparison, all models use approximately equal number of blocks in the same stage. Through a large number of experiments, it is verified that MHCN has more advantages than next-ViT in hanging string recognition and inference speed. In the MHCN model, the optimal configuration ratio of CNN block and transformer block is 7:2, and the model performance is better and the delay is smaller. Among them, the network structure formed by cross-fusion of CB block and TB block according to the optimal configuration ratio is as follows Figure 10 As shown, the modules are arranged in groups of 7 CB blocks and 2 TB blocks, and multiple groups of modules are connected in series to form a stage.

[0120] Specifically, refer to Figure 11 As shown, the specific structure of the CB block provided by this application includes a 3×3 C-DCV layer, a Batch Norm layer and a Dy-Relu layer connected in sequence. Figure 12 As shown, the TB block provided by the present application includes a HE-MHSA layer and a FFN layer connected in sequence.

[0121] In order to further verify the method proposed in this application, this application experimentally verifies the performance of the method by setting up a corresponding experimental environment. The specific process is as follows:

[0122] (1) Experimental environment

[0123] We have accumulated approximately 14,300 sample data, including samples of seven categories: normal string, broken string, detached string, bent string, loose string, and improper string installation. The number of samples in each category is shown in Table 1:

[0124] Table 1 Number of hanging string samples

[0125]

[0126] The trainset occupies 80% of the dataset, and the testset occupies 20%. The model loss algorithm uses focal loss to address difficult and easy samples and imbalanced samples.

[0127] Due to the imbalance of samples, the model is more inclined to types with a large number of samples. Therefore, different additional strategies are adopted for the different number of samples in each category: 20% of samples are added to types with a large number of samples, and the L-cutout algorithm is used as the enhancement method to improve occlusion recognition; for types with a small number of samples, a combination of traditional methods (e.g., rotation, filp, etc.) and L-cutout methods is adopted, where the traditional method uses a 1:1 ratio enhancement and the L-cutout method uses a 20% enhancement.

[0128] (2) image enhancement experiments (cutout, mixup, cutmix, and L-cutout) were compared;

[0129] Experimental Environment: We used hanging strings as our training and validation sets, and FasterRCNN as our experimental network. We replaced the L-cutout enhancement algorithm with cutout, mixup, and cutmix, respectively, and trained the cutout, mixup, and cutmix enhanced samples on the same model. The evaluation results are shown in Table 2:

[0130] Table 2. Comparison of the effects of enhanced algorithms

[0131]

[0132] Table 2 clearly shows that different data augmentation methods produce different results. L-cutout significantly outperforms cutout, mixup, and cutmix. This is primarily due to the random augmentation used by cutout, mixup, and cutmix, which obscures or covers key features of the drop string, resulting in reduced recognition accuracy. L-cutout, on the other hand, avoids the loss of critical information about the drop string during data augmentation.

[0133] (3) Comparison between DCV2 and C-DCV

[0134] We compared the effects of DCV2 and C-DCV on string defect detection. DCV2 and C-DCV replaced the 3x3 convolutional neural networks in the backbone network stages 2, 3, and 4, respectively. The two networks were trained to verify their respective recall and precision rates. Table 3 shows a comparison of the sampling effects of DCV2 and C-DCV.

[0135] Table 3 Comparison of sampling effects between DCV2 and C-DCV

[0136]

[0137] Table 3 shows that although C-DCV and DCV2 have equivalent inference efficiency, C-DCV achieves higher precision and recall for string-dropping recognition. This is primarily because C-DCV favors the inherent characteristics of string-dropping during sampling. Specifically, the learned height-to-width ratio constraint in the offset improves the model's string-dropping recognition capability.

[0138] (4) The impact of integrating new paradigms on the model:

[0139] For fair comparison, all models use approximately the same number of blocks in the same stage. The number of blocks in stages 1, 2, and 4 of the backbone network is fixed at 3, 4, and 3, respectively. We adjust the number of blocks in stage 3 and the ratio of CNN blocks to transformer blocks to compare MCHN with traditional hybrid networks. The experiments used ImageNet-22K as pre-training for 300 epochs. Fine-tuning and evaluation were performed using dropper samples.

[0140] Under the same test environment, MCHN is compared with the traditional hybrid network, where the blocks of stage 1, stage 2, and stage 4 in the backbone network are equal.

[0141] Table 4 Comparison of recognition effects between MCHN and traditional hybrid network

[0142]

[0143] The experimental results presented in Table 4, compared with traditional hybrid networks, show that simply replacing the CNN blocks in stage 3 with transformer blocks does not significantly improve recognition performance and increases inference latency. Cross-fusing CNN blocks and transformer blocks in this stage improves performance without significantly increasing inference latency. However, the distribution ratio of CNN blocks to transformer blocks in stage 3 is suboptimal, preventing the fusion effect from being realized. MCHN optimizes the distribution ratio of CNN blocks to transformer blocks, achieving a 7:2 ratio that maximizes the advantages of their fusion, significantly improving recognition performance and reducing inference latency.

[0144] This application performs a cross-depth fusion of CNN and transformer. The generated multi-block cross-hybrid network is also compatible with existing fusion modes. In view of the intrinsic characteristics of the suspension string, this application also proposes a constraint-based variable convolution and restriction-based L-cutout data enhancement algorithm. A large number of experiments have shown that the multi-block cross-hybrid network constructed in this application can not only significantly improve the defect recall rate and accuracy of suspension string recognition in complex application scenarios, but also has a small recognition delay.

[0145] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting catenary dropper defects based on the fusion of CNN and Transformer, characterized in that: include: Collecting images of catenary droppers and performing image enhancement processing on the images to obtain a dropper string defect sample set; A convolutional module is constructed using a constrained variable convolutional network, and a self-attention module is constructed based on an improved high-efficiency multi-head self-attention mechanism. The convolutional module and the self-attention module are deeply fused based on the optimal module allocation ratio to generate a multi-block cross-hybrid network. Using the multi-block cross-hybrid network to improve the FasterRCNN network, and training and verifying the improved FasterRCNN network based on the dropper string defect sample set to obtain a trained FasterRCNN improved model; The improved FasterRCNN model is deployed in the dropper string detection equipment to capture the images of the high-speed railway contact network dropper strings in real time and input them into the dropper string detection equipment for dropper string defect detection to identify dropper string defects in the contact network.

2. The method for detecting defects in catenary droppers based on CNN and Transformer fusion according to claim 1 is characterized in that: The collecting of the catenary dropper string images and performing image enhancement processing on the dropper string images to obtain the dropper string defect sample set specifically includes: Collect images of overhead catenary strings during high-speed railway operation; Based on the improved image enhancement algorithm, the catenary string images are enhanced. N mask regions are randomly generated in any string image. Based on the number N of mask regions, the mask regions that block the key features of the string are filtered out to obtain a sample set of string defects. The sample set of suspension string defects is divided into training samples and validation samples.

3. The method for detecting catenary dropper defects based on CNN and Transformer fusion according to claim 2 is characterized in that: The filtering of the masked areas that block the key features of the suspension string according to the number N of masked areas specifically includes: If N=1, the single mask area in the string image that completely blocks the string is filtered out; If N=2, the two mask areas in the string image that block the upper and lower ends of the string are filtered out; If N ≥ 3, filter out the single mask area that completely blocks the hanging string in the hanging string image, and filter out any two mask areas that block the upper and lower ends of the hanging string in the hanging string image.

4. The method for detecting catenary dropper defects based on CNN and Transformer fusion according to claim 1 is characterized in that: The convolution module is constructed using a constrained variable convolutional network, and the self-attention module is constructed based on an improved high-efficiency multi-head self-attention mechanism. The convolution module and the self-attention module are deeply integrated based on the optimal module allocation ratio to generate a multi-block cross-hybrid network, specifically including: Based on the height-to-width ratio of the aspect ratio drop string, a constraint relationship is used to constrain the height-to-width coordinate ratio of the sampling point position in the variable convolutional network. At the same time, the height-to-width coordinates of the sampling point position are restricted to not exceed the height-to-width of the input feature map. This results in a constraint-based variable convolutional network, and a convolution module is constructed using the constraint-based variable convolutional network. Perform spatial dimensionality reduction operations on the key vector K and value vector V in the original Transformer self-attention mechanism to obtain an improved high-efficiency multi-head self-attention mechanism and construct a self-attention module based on the improved high-efficiency multi-head self-attention mechanism; Based on the backbone network architecture in the FasterRCNN model, the number of convolutional modules and self-attention modules in the backbone network architecture is allocated according to the optimal module allocation ratio, and the convolutional modules and self-attention modules are fused in a new paradigm to generate a multi-block cross-hybrid network.

5. The method for detecting catenary dropper defects based on CNN and Transformer fusion according to claim 4 is characterized in that: The optimal module allocation ratio is specifically: number of convolution modules: number of self-attention modules = 7:

2.

6. The method for detecting catenary dropper defects based on CNN and Transformer fusion according to claim 1 is characterized in that: The method of improving the FasterRCNN network by using the multi-block cross-hybrid network and training and verifying the improved FasterRCNN network based on the dropper string defect sample set to obtain a trained FasterRCNN improved model specifically includes: The backbone network architecture of the FasterRCNN network is replaced with the network architecture of the multi-block cross-hybrid network to obtain an improved FasterRCNN model; The improved FasterRCNN model is trained based on the training samples in the suspension string defect sample set. After the training is completed, the model is verified using the verification samples to obtain the trained and verified FasterRCNN improved model.

Images