A cross-domain fault detection method based on multi-scale fusion and deformable convolution

Abstract

The present invention provides a cross-domain fault detection method based on multi-scale fusion and deformable convolution to solve the corresponding problems in cross-domain target detection of train fault images. It includes the following steps: preparing a train fault dataset under normal scenes as a source domain dataset and a train fault dataset under foggy scenes as a target domain dataset; designing a domain adaptive feature pyramid and adding it to the RetinaNet target detection network to achieve feature alignment between the source domain and the target domain; adding a deformable convolution module to the backbone of the target detection network to achieve adaptability to faults with large morphological differences; adding an ASFF module to the FPN in the original detection network to solve the problem of inconsistent learning objectives at different scales in the FPN network; changing the regression subnetwork loss function of the RetinaNet network to CIOU loss to optimize the loss of the model. This cross-domain fault detection method based on multi-scale fusion and deformable convolution solves the problem that the target detection model has poor train fault detection results in foggy scenes.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a cross-domain fault detection method based on multi-scale fusion and deformable convolution. Background Art

[0002] Object detection is one of the most challenging tasks in computer vision. It consists of two subtasks: locating the target object in an image or video, and identifying and labeling it. Since the advent of the deep learning era, CNN-based deep network models, with their powerful feature extraction and generalization capabilities, have significantly improved the performance of object detection algorithms compared to traditional methods.

[0003] Currently, mainstream object detection models, such as Faster-RCNN and YOLO, are supervised learning models that rely on a large number of labeled samples. However, in the actual detection of train fault images, due to camera shooting, weather conditions, and other factors, train fault images may appear blurry or too dark. Images may also contain fog, ice, and snow. These objective factors can lead to domain shift between the training and test sets, making the existing object detection model unable to effectively identify faults in these test set images. The most direct way to address this problem is to re-collect and annotate the images in the test set that have experienced domain shift. However, current image annotation methods still rely primarily on manual labeling, and the image collection and annotation process is time-consuming and labor-intensive. To address the significant cost of repeated dataset annotation, domain adaptation methods in transfer learning can be used to mitigate the degradation in model performance.

[0004] Currently, the application of domain adaptation methods in the field of target detection can be divided into three main categories: adversarial domain adaptation, reconstruction-based domain adaptation, and hybrid-based domain adaptation. Adversarial domain adaptation involves adding a domain discriminator to the network, which reduces domain shift by confusing the domain-invariant features of the source and target domains. Reconstruction-based domain adaptation involves reconstructing source or target domain data, generating data using techniques such as CycleGAN, to obtain intermediate domain data. Using this intermediate domain data for training can improve training results. Hybrid-based methods combine the two aforementioned approaches. The fundamental idea behind the attention mechanism in computer vision is to enable the model to ignore irrelevant information and focus on key information. Summary of the Invention

[0005] To address the low accuracy of train fault detection in foggy conditions, the present invention provides a cross-domain fault detection method based on multi-scale fusion and deformable convolution. This method addresses the issue of differing data distributions between the train fault training and test sets by utilizing a domain-adaptive feature pyramid to improve the cross-domain robustness of the original target detection model. Furthermore, deformable convolution is used to enhance the network's adaptability due to the large differences in fault morphology. Furthermore, ASFF is used to address the inconsistency of learning objectives at different scales in the FPN network in the original RetinaNet, and CIOU is used to help the detection network better optimize losses.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A cross-domain fault detection method based on multi-scale fusion and deformable convolution includes the following steps:

[0008] Construct source domain dataset and target domain dataset;

[0009] A cross-domain fault detection network based on RetinaNet is constructed. The cross-domain fault detection network includes a domain-adaptive feature pyramid, a backbone embedded with deformable convolutions, an FPN network, and a regression subnetwork. The domain-adaptive feature pyramid includes a local domain adaptation module and a global domain adaptation module, which are used to fuse the multi-scale information extracted from the backbone.

[0010] Training the cross-domain fault detection network using the source domain dataset and the target domain dataset;

[0011] The data to be tested is input into the trained cross-domain fault detection network, and the fault type is output.

[0012] Preferably, the backbone uses ResNet-50, which includes several convolution modules. Part of the feature vectors output by it are input into the local domain adaptation module for strong alignment, and the remaining features are first fused with multi-scale information through the feature pyramid structure, and then input into the global domain adaptation module for weak alignment.

[0013] Preferably, the loss function of the local domain adaptation module is:

[0014]

[0015]

[0016]

[0017] in, represents the local domain adaptation loss of the source domain, represents the local domain adaptation loss of the target domain, It represents the loss of the entire local domain adaptation module, taking the average of the source domain and target domain losses. 、 Represent the input source domain features, The feature extractor of the domain adaptation part network represented by represents the domain discriminator, Represents the height and width of the input feature map, and They represent the number of input source domain samples and the number of target domain samples respectively.

[0018] Preferably, the loss function of the global domain adaptation module is:

[0019]

[0020]

[0021]

[0022] in, represents the global domain adaptation loss of the source domain, represents the global domain adaptation loss of the target domain, represents the loss of the i-th global domain adaptation module, It represents the difficulty and ease sample balance factor, represents the domain discriminator of the global domain adaptation module, It represents the feature extractor of the global domain adaptation part of the network.

[0023] Preferably, the way to embed the deformable convolution is to add an offset to the convolution kernel of the backbone.

[0024] Preferably, a supplementary module ASFF is embedded in the FPN network.

[0025] Preferably, the regression subnetwork uses CIOU loss as the loss function.

[0026] The present invention also provides a cross-domain fault detection system based on multi-scale fusion and deformable convolution, the system comprising: a domain adaptive feature pyramid module for performing multi-scale domain adversarial training on the network to achieve feature alignment between the source domain and the target domain;

[0027] Deformable convolution module, used to enhance the network's fault detection capabilities for complex deformations;

[0028] The ASFF module is used to solve the problem of inconsistent cross-scale feature learning objectives in the original FPN network;

[0029] The loss function module is used to solve the problem that the loss function Smooth L1 Loss and IoU in the original regression sub-network are not equivalent.

[0030] The present invention also provides a method for detecting train faults in foggy weather based on the above cross-domain fault detection method, comprising the following steps:

[0031] Construct a train failure dataset under normal scenarios as the source domain dataset and a train failure dataset under foggy scenarios as the target domain dataset;

[0032] Build a cross-domain fault detection network based on RetinaNet;

[0033] Training the RetinaNet-based cross-domain fault detection network using the source domain dataset and the target domain dataset;

[0034] The test data is input into the trained RetinaNet-based cross-domain fault detection network to output the type of train fault in foggy weather.

[0035] Beneficial effects: The present invention proposes a cross-domain fault detection method based on multi-scale fusion and deformable convolution. By adding the constructed domain adaptive feature pyramid to the RetinaNet network, the network is assisted in multi-scale domain adaptive adversarial training, so as to complete the feature alignment between the source domain and the target. By embedding the deformable convolution into the backbone of RetinaNet, the adaptability of the network to complex deformed fault samples is improved. By adding ASFF to the FPN network of RetinaNet, the problem of inconsistent learning objectives of different scales of the FPN network is solved. CIOU is used to replace the Smooth L1 Loss in the RetinaNet regression subnetwork to solve the problem of inconsistency between Smooth L1 Loss and the actual evaluation indicator IOU of the model. The above design enhances the robustness of cross-domain detection and improves the recognition rate and accuracy of train faults in foggy scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a cross-domain fault detection method based on multi-scale fusion and deformable convolution proposed by the present invention;

[0037] Figure 2 This is a structural diagram of the cross-domain fault detection method based on multi-scale fusion and deformable convolution;

[0038] Figure 3 It is a domain adaptive feature pyramid structure diagram;

[0039] Figure 4 Schematic diagram of deformable convolution embedding;

[0040] Figure 5 It is the combined image of FPN and ASFF;

[0041] Figure 6 The experimental results of this method are shown in Figure 2. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0043] The present invention provides a cross-domain fault detection method based on multi-scale fusion and deformable convolution. A domain-adaptive feature pyramid is designed to achieve feature alignment between the source and target domains. A deformable convolution module is added to the backbone of the target detection network to achieve adaptability to faults with large morphological differences. An ASFF module is added to the FPN in the original detection network to solve the problem of inconsistent learning targets at different scales in the FPN network. The loss function of the regression subnetwork of the RetinaNet network is changed to CIOU loss to optimize the loss of the model.

[0044] Figure 1 This is a flow chart of the cross-domain fault detection method based on multi-scale fusion and deformable convolution proposed in the present invention. Figure 2 This is the overall structure diagram of this method. Figure 1 As shown, the method includes:

[0045] S1, prepare a train failure dataset under normal scenarios as the source domain dataset and a train failure dataset under foggy scenarios as the target domain dataset;

[0046] S2, builds a domain adaptive feature pyramid and adds it to the RetinaNet target detection network. The domain adaptive feature pyramid structure is as follows Figure 3 As shown;

[0047] S3, embeds the deformable convolution into the backbone to form Figure 2 Deform Block in. The specific embedding method is as follows Figure 4 As shown;

[0048] S4, add ASFF to the FPN network. The specific combination method is as follows Figure 5 As shown;

[0049] S5, uses CIOU instead of Smooth L1 Loss;

[0050] S6, input the labeled source domain dataset and the unlabeled target domain dataset to complete domain adaptive train fault detection.

[0051] Step S1 is as follows: prepare train fault data and normal data under normal scenarios. These data include image data under good conditions and their corresponding labels. That is, on the image, each part to be detected has a corresponding label and bounding box.

[0052] Prepare train failure data and normal data under foggy conditions. This type of data only contains image data and does not require annotation of target information.

[0053] Step S2 is as follows:

[0054] The RetinaNet domain adaptation network is constructed using the Pytorch framework. The model is a single-stage target detection model. The overall network structure is as follows Figure 2 As shown in the figure, the multi-scale information extracted from the RetinaNet backbone is fused using a feature pyramid structure. The backbone of RetinaNet uses ResNet-50, which has five convolutional modules. The last four convolutional modules are named res2, res3, res4, and res5, respectively. The features extracted from these modules are named C2, C3, C4, and C5, respectively. The corresponding feature channel dimensions are 256, 512, 1024, and 2048, respectively.

[0055] The first feature vector in res2 is input into the local domain adaptation module for strong alignment, and its loss function is:

[0056]

[0057]

[0058]

[0059] in, represents the local domain adaptation loss of the source domain, represents the local domain adaptation loss of the target domain. It represents the loss of the entire local domain adaptation module, taking the average of the source domain and target domain losses.

[0060] Before being input into the domain adaptation module, res3, res4, and res5 are first subjected to multi-scale feature fusion through a feature pyramid structure. The fused features retain the original number of channels while adding new feature information. The fused feature information is then input into the three global domain adaptation modules for domain adaptive feature alignment. Global domain adaptation uses a weak alignment mode. The loss function uses Focal loss, which emphasizes the alignment of samples that are similar overall, i.e., the alignment of samples that are difficult for the domain classifier to classify, rather than the alignment of samples that are dissimilar overall. The loss function is:

[0061]

[0062]

[0063]

[0064] in, represents the global domain adaptation loss of the source domain, represents the global domain adaptation loss of the target domain. It represents the loss of the ith global domain adaptation module, which is the average of the source domain and target domain losses. This method uses three global domain adaptation modules in total, so the total loss of global domain adaptation can be expressed as:

[0065]

[0066] in, 、 、 is a weight parameter. For different global domain adaptation modules, the present invention will use different weights.

[0067] For the entire domain adaptive adversarial network, the loss function can be expressed as:

[0068]

[0069] Step S3 includes the following specific steps:

[0070] The deformable convolution module is embedded in the last three convolution modules of ResNet-50. The specific embedding method is to add an offset to the 3×3 convolution kernel. Represents the size of the convolution kernel and the receptive field, and the original output feature after the convolution kernel is

[0071]

[0072] in express For each spatial position point in Represents the convolution kernel weight matrix. The position in the receptive field grid represented by . Add a deformable convolution to the convolution kernel and add an offset to each sampling point in the standard convolution. , then the output feature Becomes as follows:

[0073]

[0074] After transformation, the position is irregular and offset. On, the actual adopted position becomes ,in Generally speaking, it is a decimal, so the pixel value at the offset sampling position can be obtained by linear interpolation.

[0075] Step S4 includes the following specific steps:

[0076] ASFF is embedded into the original FPN network as a supplementary module for FPN. Assume that the FPN network outputs features level 1, level 2, and level 3 after upsampling. Three new features, ASFF1, ASFF2, and ASFF3, are formed through weighted addition. Taking the formation of ASFF1 as an example, the features output by level 2 and level 3 need to be downsampled to the same size as the level 1 feature, and then weighted fusion is performed using three learnable parameters. The weighted fusion formula is as follows:

[0077]

[0078] It represents the feature at (i, j) after the feature of level n is resized to level 1. The above formula means that the feature map after level 1, 2, and 3 resize is at each (i, j) and its respective spatial weight parameter 、 、 Multiply and add the two together to get the fused ASFF1. ASFF2 and ASFF3 are generated in the same way as ASFF1. The new features ASFF1, ASFF2, and ASFF3 replace the original level 1, level 2, and level 3.

[0079] Step S5 specifically includes the following steps:

[0080] Using CIOU loss as the loss function of the regression sub-network, the calculation formula of CIOU loss is as follows:

[0081]

[0082] is the weight coefficient, Used to measure the similarity of aspect ratios, Refers to calculating the Euclidean distance, is the diagonal length of the smallest enclosing box covering the predicted box and the true box, They refer to the center points of the predicted box and the real box respectively, 、 and 、 They represent the height and width of the predicted box and the real box respectively. CIOU loss is calculated using 1-CIOU, so the final CIOU loss is:

[0083]

[0084] The final training function of this method can be expressed as:

[0085]

[0086] The training model was built and trained on 13,658 source domain images and 13,658 target domain images for 70,000 iterations using the NVIDIA Tesla V100 graphics card in the server. Finally, the model was validated on a validation set of 1,235 images.

[0087] The present invention can explicitly display the classification results, meet the actual operation scenarios and application requirements in industry, and improve the accuracy of detection.

[0088] like Figure 6 The top half shows the detection results for foggy weather faults using the native RetinaNet network, while the bottom half shows the detection results using our proposed method. Clearly, the native RetinaNet network's detection results contain both false positives and missed detections, while our proposed method's detection results are all correct, demonstrating the effectiveness of our method.

[0089] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A cross-domain fault detection method based on multi-scale fusion and deformable convolution, characterized in that: The following steps are involved: Construct source domain dataset and target domain dataset; A cross-domain fault detection network based on RetinaNet is constructed. The cross-domain fault detection network includes a domain-adaptive feature pyramid, a backbone embedded with deformable convolutions, an FPN network, and a regression subnetwork. The domain-adaptive feature pyramid includes a local domain adaptation module and a global domain adaptation module, which are used to fuse the multi-scale information extracted from the backbone. Training the cross-domain fault detection network using the source domain dataset and the target domain dataset; Input the data to be tested into the trained cross-domain fault detection network and output the fault type; The backbone uses ResNet-50, which includes several convolutional modules. Some of the feature vectors it outputs are input to the local domain adaptation module for strong alignment. The remaining features are first fused with multi-scale information through the feature pyramid structure, and then input to the global domain adaptation module for weak alignment. The loss function of the local domain adaptation module is: ； ； ； represents the local domain adaptation loss of the source domain, represents the local domain adaptation loss of the target domain, It represents the loss of the entire local domain adaptation module, taking the average of the source domain and target domain losses. 、 Represent the input source domain features, The feature extractor of the domain adaptation part network represented by represents the domain discriminator of the local domain adaptation module, Represents the height and width of the input feature map, and Respectively represent the number of input source domain samples and the number of target domain samples; The FPN network is embedded with a supplementary module ASFF; The method for detecting train faults in foggy weather based on the cross-domain fault detection method includes the following steps: Construct a train failure dataset under normal scenarios as the source domain dataset and a train failure dataset under foggy scenarios as the target domain dataset; Build a cross-domain fault detection network based on RetinaNet; Training the RetinaNet-based cross-domain fault detection network using the source domain dataset and the target domain dataset; The test data is input into the trained cross-domain fault detection network based on RetinaNet, and the foggy train fault detection results are output.

2. The cross-domain fault detection method based on multi-scale fusion and deformable convolution according to claim 1 is characterized in that: The loss function of the global domain adaptation module is: ； ； ； represents the global domain adaptation loss of the source domain, represents the global domain adaptation loss of the target domain, represents the loss of the i-th global domain adaptation module, It represents the difficulty and ease sample balance factor, represents the domain discriminator of the global domain adaptation module, It represents the feature extractor of the global domain adaptation part of the network.

3. The cross-domain fault detection method based on multi-scale fusion and deformable convolution according to claim 1 is characterized in that: The way to embed deformable convolution is to add an offset to the backbone convolution kernel.

4. The cross-domain fault detection method based on multi-scale fusion and deformable convolution according to claim 1 is characterized in that: The regression sub-network uses CIOU loss as the loss function.

Images