An image interactive fusion method for wheelset tread defect recognition

Abstract

This invention discloses an image interactive fusion method for wheelset tread defect recognition. The method mainly comprises five stages: data acquisition and processing, multi-scale interactive attention feature extraction, constraint-coupled decoding, fusion inference, and result display. The adaptive hybrid interactive attention module used in the multi-scale interactive attention feature extraction stage facilitates the differentiation of target shapes in the model, while the multi-scale sparse feature extraction module facilitates scale differentiation during feature extraction. The constraint-coupled decoding stage incorporates consistency loss and reconstruction loss into the modal decoder, improving the accuracy of model recognition. In summary, this invention solves the problem of difficulty in distinguishing target shape and scale during feature extraction, resulting in high model recognition accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of wheel set tread recognition, in particular to an image interactive fusion method for wheel set tread defect recognition. BACKGROUND

[0002] With the train operation entering the era of high speed and heavy load, its intelligent operation and maintenance are more and more concerned. As an important running and supporting component of the train, the high-frequency vibration caused by the high-speed rotation of the wheel set, the wheel-rail temperature rise caused by the sliding / rolling phenomenon in the wheel-rail contact process and other factors will cause wheel-rail contact fatigue, resulting in multiple types of defects such as scuffing, spalling and block shedding on the wheel set tread. The accurate recognition of the tread defects can provide key support for the train operation and maintenance. However, due to the complex and variable wheel-rail contact conditions and other factors, the tread defect data samples are mostly characterized by small shape difference between different types of defects and large size difference between the same type of defects, so that the existing deep learning network often faces the problems of difficulty in distinguishing the target shape and size and low recognition accuracy of the model during feature extraction, which is difficult to meet the real-time diagnosis demand of the train intelligent operation and maintenance on the wheel set tread defects.

[0003] The patent with the publication number CN114663344A discloses a train wheel set tread defect recognition method based on image fusion. When the train wheel set tread defect is recognized, the visible light camera and the infrared camera are used to collect images and obtain the train wheel set tread area image. A fusion model of the visible light image and the infrared image is constructed based on the neural network, and the model is trained until the model converges. The corresponding visible light image and infrared image are input into the trained fusion model to obtain the fusion image. The region growing method is used to aggregate the pixel points in the fusion image according to the similarity of the image gray values to obtain the image of the train wheel set tread defect area. The conventional neural network is used to construct the fusion model of the visible light image and the infrared image, and when the wheel set tread defect is recognized, there are problems such as difficulty in distinguishing the target shape and size and low recognition accuracy of the model. SUMMARY

[0004] In view of the above technical problems, the present application provides an image interactive fusion method for wheel set tread defect recognition.

[0005] The present application adopts the following specific technical solutions:

[0006] An image interactive fusion method for wheel set tread defect recognition, the method main body contains five stages, which are data acquisition and processing stage, multi-scale interactive attention feature extraction stage, constraint coupling decoding stage, fusion inference stage and result display stage. The method comprises the following steps:

[0007] S1, data acquisition and processing stage: the RGB image samples of the wheel tread defects are collected on site, and the original RGB image samples are encoded by using the Poisson encoder to obtain the Poisson modal POS image for image fusion.

[0008] S2, multi-scale interactive attention feature extraction stage:

[0009] S2.1, the pre-trained lightweight network Mobilenetv2 fused with the RGB and Poisson images is used as the model backbone network, the bottom layer features of the modal of the RGB image and the Poisson modal image collected in step S1 are extracted, and the input feature mapping x m ∈R H×W×C is the m modal input image, and its final encoding feature is as follows:

[0010] h m =Mobile m (x m ),m∈{r,p}

[0011] The feature mapping x m ∈R H×W×C In the formula, H, W and C represent the length, width and channel number of the input image, and in the formula In the formula, H1, W1 and C1 represent the length, width and channel number of the feature map respectively, Mobile m respectively represent the encoding feature of the m modal, the Mobilenetv2 network, and r and p represent the features of the RGB image modal and the Poisson modal POS image modal respectively;

[0012] S2.2, the multi-scale sparse feature extraction module DGASPP is used for multi-scale feature extraction of the bottom layer features, so as to solve the problem of large defect scale difference in the image sample. The multi-scale sparse feature extraction module DGASPP performs multi-scale feature extraction on the encoding features h m of the two modal, and the extraction formula is:

[0013] s m =DGASPP m (h m ),m∈{r,p}

[0014] In the formula, DGASPP m respectively represent the multi-scale encoding feature of the m modal and the DGASPP module;

[0015] S2.3, using the adaptive hybrid interactive attention module AHMA module to extract spatial channel attention features with interactive information in the underlying features, to solve the problem of small shape difference of defects in the image sample, and using the AHMA module to extract spatial channel attention features with interactive information:

[0016] m r ,m p =AHMA(s r ,s p )

[0017] In the formula, respectively represent the attention weighted features of the two modalities; AHMA is an adaptive hybrid interactive attention module.

[0018] S3, constraint coupling decoding stage: enabling the constraint coupling decoding module, using the modal decoder to decode the features extracted in the multi-scale interactive attention feature extraction module in step S2.2, and using the improved encoding-decoding structure network Deeplabv3 in the modal decoder to improve the decoding part as follows:

[0019] S3.1, adding additional feature splicing convolution CConv and shortcut convolution SConv to obtain decoding features at more scales;

[0020] S3.2, adding consistency constraint loss between decoding features of different modalities to capture interactive features in the decoding features;

[0021] S3.3, increasing the reconstruction loss between the input image and the reconstructed image to strengthen the shape feature extraction;

[0022] S3.4, constructing a total target loss function composed of task loss, consistency loss and reconstruction loss to guide network learning related features, and the formula of the total target loss function is:

[0023] L total =μL task +(1-μ)(L consis +L recon )

[0024] In the formula: μ is a loss function adjustment factor, L task is a task loss function, L consis is a consistency loss function, L recon is a reconstruction loss function, and the total target loss function is used to adjust the contribution of the task loss and the decoding part loss to network learning, and the network is trained with the goal of minimizing the above loss function;

[0025] S4. Fusion Inference Stage: Global average pooling and multilayer perceptron are used to fuse and infer interactive attention features. The fusion inference formula is as follows:

[0026]

[0027] In the formula, P represents the fusion inference output; MLP represents the multilayer perceptron; and avg represents the average pooling operation. This indicates a channel splicing operation.

[0028] S5. Results Display Stage: Display the scores of each category and the category to which some of the test set image samples belong.

[0029] Furthermore, the constraint-coupled decoding module is used to assist in training the network during the multi-scale interactive attention feature extraction stage and does not participate in testing.

[0030] Furthermore, the multi-scale sparse feature extraction module DGASPP is formed by introducing the ideas of ghost modules and dense connections into the hollow space pyramid pooling module ASPP.

[0031] Furthermore, in step S2.3, the adaptive hybrid interactive attention module AHMA consists of a hybrid nonlocal module HNL, attention matrix fusion, and adaptive attention reliability weights.

[0032] Furthermore, in step S3.4, the task loss function L task The formula is:

[0033] L task =-α(1-P) γ log(P)

[0034] In the formula, α = 0.25 represents the weighting factor, and γ = 2 represents the adjustment factor.

[0035] Furthermore, in step S3.4, the consistency loss function L consis The formula is:

[0036]

[0037] The decoding features of the m-mode under 16x and 8x downsampling are respectively expressed as follows: Where m∈{r, p}, in the formula These represent the decoding features of the 16x downsampled RGB image mode and the Poisson mode POS image mode, respectively. The above-mentioned consistency loss function is derived using mean square error (MSE) loss at the 16x and 8x downsampled decoding features between modes, respectively, representing the decoding features of the 8x downsampled RGB image mode and the Poisson POS image mode.

[0038] Furthermore, the decoding features of the m-mode with 16x downsampling and 8x downsampling. and The formulas are as follows:

[0039]

[0040]

[0041] in up2 represents a bilinear interpolation operation with twice the online sampling rate. This indicates a channel splicing operation.

[0042] Furthermore, decoding features CConv8 and decoding features in the formula CConv in the formula 16 They have the same topology.

[0043] Furthermore, in step S3.4, the loss function L is reconstructed. recon for:

[0044]

[0045] Where x r x p Represent the inputs of the RGB image and the Poisson image, respectively, r r r p These represent the decoded and reconstructed outputs of the RGB image and the Poisson image, respectively.

[0046] Furthermore, the image interaction fusion method was validated and analyzed through experiments.

[0047] The beneficial effects of this invention are as follows:

[0048] The beneficial effects of the image interactive fusion method for wheelset tread defect identification designed in this invention are as follows:

[0049] (1) An adaptive hybrid interactive attention module is proposed in the multi-scale interactive attention feature extraction stage. This module introduces channel branching and adaptive interaction mechanism into the non-local attention module, enabling it to improve the information gain of attention by utilizing intermodal interaction information, optimize the target shape feature capture capability, facilitate the differentiation of target shapes when extracting features in this invention, and solve the problem of small difference in defect shapes in the model.

[0050] (2) In the multi-scale interactive attention feature extraction stage, a multi-scale sparse feature extraction module is proposed. This module introduces the ideas of "ghost module" and "dense connection" into the hollow space pyramid pooling module, so that it can optimize the multi-scale sparse representation by associating the branch information of each scale, which is beneficial to the scale differentiation when extracting features in this invention and solves the problem of large difference in defect scale.

[0051] (3) A constraint-coupled decoding stage was designed. This stage introduces consistency loss and reconstruction loss into the modal decoder, enabling it to better capture defect shape features and intermodal interaction features during training. The model of this invention has high recognition accuracy. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 A simplified flowchart of an image interaction fusion method for identifying wheel tread defects;

[0054] Figure 2 This is a schematic diagram of the data acquisition and processing stage;

[0055] Figure 3 This is a schematic diagram showing some sample examples;

[0056] Figure 4 This is a schematic diagram of the multi-scale interactive attention feature extraction stage;

[0057] Figure 5 This is a schematic diagram of the fusion inference stage;

[0058] Figure 6 This is a schematic diagram of the constraint-coupled decoding stage;

[0059] Figure 7 This is a schematic diagram of the results presentation stage;

[0060] Figure 8 This is a schematic diagram of the DGASPP module structure. Detailed Implementation

[0061] The following examples further explain and clarify the invention, but the specific examples do not limit the invention in any way.

[0062] Example 1

[0063] An image interactive fusion method for wheelset tread defect recognition, such as Figure 1 As shown, the main body of the method comprises five stages: data acquisition and processing, multi-scale interactive attention feature extraction, constraint coupling decoding, fusion inference, and result display. The method includes the following steps:

[0064] S1, Data Acquisition and Processing Stage: (e.g.)Figure 2 As shown, RGB image samples of wheel tread defects were collected on site, and Poisson encoders were used to encode the original RGB image samples to obtain Poisson modal POS images for image fusion.

[0065] The dataset used in the experiment of this embodiment 1 was collected by a CCD camera in the wheel and axle workshop of the vehicle depot according to a unified standard. The database established by this invention contains nine state images and one interference image of the wheelset tread under the action of wheel-rail contact force during the actual operation of the train.

[0066] Specifically, as shown in Table 1, the tread defect dataset contains a total of 343 images of 10 different wheelset conditions. Among them, there are 57 normal images, 52 scratches, 47 cracks, 45 abrasions, 42 peelings, 29 stains (interference images), 24 scratches, 21 edge wear images, 20 circumferential wear images, and 6 chipped images.

[0067] Tread defect class Number of training samples Number of testing samples Tread normal 34 23 Tread bruise 31 21 Tread crack 28 19 Tread chip 27 18 Tread cut 25 17 Tread dirt 17 12 Tread gouge 14 10 Edge wear 12 9 Circumferential wear 12 8 Tread chunk 3 3

[0068] Table 1

[0069] To facilitate network training and testing, defects in the original RGB image are reconstructed into a 224×224 pixel image, and Poisson coding is performed on the RGB image to obtain the POS image. Some sample examples are shown below. Figure 3 As shown. Finally, the data for each state was approximately divided in a 3:2 ratio, resulting in a training set of 203 images and a test set of 140 images.

[0070] S2, such as Figure 4 As shown, the multi-scale interactive attention feature extraction stage:

[0071] S2.1 Using the pre-trained lightweight network Mobilenetv2, which integrates RGB and Poisson images, as the backbone network, extract the low-level features of the modalities from the RGB and Poisson modal images collected in step S1. Let the input feature mapping be x. m ∈R 224×224×3 Given an m-modal input image, its final encoded features are as follows:

[0072] h m =Mobile m (x m ),m∈{r,p} (1)

[0073] h m ∈R 7×7×128 Mobile m Let r and p represent the coding features of the m-mode and the Mobilenetv2 network, respectively, and let r and p represent the features of the RGB image mode and the Poisson POS image mode, respectively.

[0074] S2.2 A multi-scale sparse feature extraction module (DGASPP) is used to extract multi-scale features from the low-level features to address the problem of large scale differences in defects in image samples. The DGASPP module encodes the features h of two modalities. m Multi-scale feature extraction is performed using the following formula:

[0075] s m =DGASPP m (h m ),m∈{r,p} (2)

[0076] In formula (2), s m ∈R 7×7×128 DGASPP m These represent the multi-scale coding features of the m-modal and the DGASPP module, respectively.

[0077] S2.3. The Adaptive Hybrid Interactive Attention Module (AHMA) is used to extract spatial channel attention features with interactive information from the low-level features to address the problem of small differences in defect shape in image samples. The formula for extracting spatial channel attention features with interactive information using the AHMA module is as follows:

[0078] m r ,m p =AHMA(s) r ,s p (3)

[0079] In formula (3), m r ,m p ∈R 7×7×128 These represent the attention weighted features of the two modalities, respectively; AHMA is an adaptive hybrid interactive attention module.

[0080] S3, Constraint-coupled decoding stage: such as Figure 5 As shown, the constraint-coupled decoding module is enabled, and the modal decoder is used to decode the features extracted by the multi-scale interactive attention feature extraction module in step S2.2. The modal decoder uses the Deeplabv3 network with an improved encoder-decoder structure. The improvement process for the decoding part is as follows:

[0081] S3.1, Add additional feature concatenation convolutions CConv and shortcut convolutions SConv to obtain decoding features at more scales. Let the 16x downsampling and 8x downsampling decoding features of the m-modal in the MobileNetv2 module be represented as follows: Where m∈{r, p}, and m is the decoding feature of the m-mode under 16x and 8x downsampling. and The formulas are as follows:

[0082]

[0083]

[0084] In formula (4) In formula (5) up2 represents a bilinear interpolation operation with twice the online sampling rate. Indicates a channel splicing operation;

[0085] S3.2, Add consistency constraint loss among decoding features of different modalities to capture interaction features in the decoding features, where the task loss function L task The formula is:

[0086] L task =-α(1-P) γ log(P) (6)

[0087] In the formula, α = 0.25 represents the weighting factor, and γ = 2 represents the adjustment factor;

[0088] Where the consistency loss function L consis The formula is:

[0089]

[0090] In formula (7) These represent the decoding features of the 16x downsampled RGB image mode and the Poisson mode POS image mode, respectively. The above-mentioned consistency loss function is obtained by using the mean square error (MSE) loss at the 16x and 8x downsampled decoding features between modes, respectively, representing the decoding features of the 8x downsampled RGB image mode and the Poisson POS image mode.

[0091] S3.3. Add a reconstruction loss between the input image and the reconstructed image to enhance shape feature extraction, wherein the reconstruction loss function L recon for:

[0092]

[0093] Where x r x p Represent the inputs of the RGB image and the Poisson image, respectively, r r r p These represent the decoded and reconstructed outputs of the RGB image and the Poisson image, respectively.

[0094] S3.4 Construct a total objective loss function consisting of task loss, consistency loss, and reconstruction loss to guide the network in learning relevant features. The formula for the total objective loss function is:

[0095] L total =μL task +(1-μ)(L consis +L recon (9)

[0096] In formula (9): μ is the loss function adjustment factor, μ is set to 0.8, L task Let L be the task loss function. consis Let L be the consistency loss function. recon To reconstruct the loss function, the overall objective loss function is used to adjust the contribution of task loss and decoding part loss to network learning, and the network is trained with the goal of minimizing the above loss function;

[0097] S4, Fusion Inference Stage: (e.g., ...) Figure 6 As shown, global average pooling and multilayer perceptron are used to fuse and infer interactive attention features. The fusion and inference formula is as follows:

[0098]

[0099] In formula (10), P represents the fusion inference output; MLP represents the multilayer perceptron; avg represents the average pooling operation; Indicates a channel splicing operation;

[0100] S5. Results Presentation Stage: (e.g., ...) Figure 7 As shown, this displays the scores for each category and the category to which a portion of the test set image samples belong.

[0101] Furthermore, the constrained coupling decoding stage is used to assist the training of the network in the multi-scale interactive attention feature extraction stage and does not participate in the testing.

[0102] Furthermore, the multi-scale sparse feature extraction module DGASPP is formed by introducing the concepts of ghost modules and dense connections into the hollow space pyramid pooling module ASPP. The structure of the DGASPP module is as follows: Figure 8 As shown, the DGASPP module is designed with three GASPP Block branches, each with 128 output channels and different dilation coefficients. These three branches are densely connected to obtain global features. Another pooling branch is added to average the input feature map. Finally, the input feature map and the features from the four branches are concatenated, and an 11-channel convolution is used to compress the features into 128 channels to restore the input channel dimension. The multi-scale sparse feature extraction module DGASPP proposed in this invention can optimize multi-scale sparse representation by associating information from each scale branch, which is beneficial for distinguishing scales when extracting features and solving the problem of large scale differences in defects.

[0103] Furthermore, in step S2.3, the adaptive hybrid interactive attention module AHMA consists of a hybrid nonlocal module HNL, attention matrix fusion, and adaptive attention reliability weights.

[0104] Furthermore, decoding features CConv8 and decoding features in the formula CConv in the formula 16 They have the same topology and are used to integrate the corresponding downsampled features with the previous layer's decoding features.

[0105] Furthermore, the image interaction fusion method was validated and analyzed through experiments.

[0106] Example 2

[0107] The difference between Example 2 and Example 1 is that Example 2 only uses the HNL module mentioned in Example 1 to extract attention features in the multi-scale interactive attention feature extraction stage.

[0108] With consistent experimental parameter settings, the results of extracting attentional features using the HNL module and the AHMA module are shown in Table 2 below.

[0109] Module Acc P R F1 HNL 0.85 0.859 0.8.4 0.836 AHMA 0.871 0.903 0.877 0.883

[0110] Table 2

[0111] The experiment primarily used the following metrics: accuracy (Acc), recall (R), precision (P), F1 score (F1), Pa, and T. Accuracy (Acc) represents the proportion of correctly predicted (positive and negative) samples out of the total number of samples; recall (R) represents the proportion of correctly predicted positive samples out of all positive samples; precision (P) represents the proportion of correctly predicted positive samples out of all positive samples; the F1 score is the weighted harmonic average of recall and precision; Pa represents the number of model parameters (in M); and T represents the time consumed by testing the model on a single sample on the CPU (in seconds).

[0112] As can be seen from Table 2, the classification indicators of the AHMA module used in Example 1 are better than those of the HNL module used in Example 2, which verifies the effectiveness and superiority of the AHMA module. The various indicators of the HNL module decreased by 0.021, 0.044, 0.043 and 0.047 respectively compared with AHMA. This shows that the AHMA module formed by the modal interaction strategy designed on the basis of the HNL module in Example 1 can effectively capture intermodal interaction information and improve the feature extraction capability of the HNL module.

[0113] Example 3

[0114] The difference between Example 3 and Example 1 is that Example 3 only uses the ASPP scale feature extraction module mentioned in Example 1 for feature extraction in the multi-scale interactive attention feature extraction stage.

[0115] With consistent experimental parameter settings, the results of extracting attentional features using the DGASPP module and the ASPP module are shown in Table 3 below.

[0116] Module Acc P R F1 ASPP 0.821 0.863 0.81 0.819 DGASPP 0.871 0.903 0.877 0.883

[0117] Table 3

[0118] As can be seen from Table 3, the DGASPP module proposed in Example 1 can maintain a relatively small number of parameters while significantly outperforming the ASPP module in terms of performance. This indicates that the DGASPP module in Example 1 can extract multi-scale features more effectively, thereby more effectively alleviating the problem of large differences in defect scales.

[0119] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An image interactive fusion method for wheelset tread defect identification, characterized in that, The method comprises five stages, namely, a data acquisition and processing stage, a multi-scale interactive attention feature extraction stage, a constraint coupling decoding stage, a fusion inference stage, and a result display stage. The method comprises the following steps: S1, a data acquisition and processing stage: collecting an RGB image sample of a wheel tread defect in the field, and using a Poisson encoder to encode the original RGB image sample to obtain a Poisson modal POS image for image fusion; S2, a multi-scale interactive attention feature extraction stage: S2.1, the pre-trained lightweight network Mobilenetv2 fused with RGB and Poisson images is used as the model backbone network to extract the bottom features of the modalities of the RGB images and Poisson modal images collected in step S1. Let the input feature mapping x m ∈R H ×W×C For m modal input images, the final encoding features are as follows: h m = Mobile m (x m ), m e {r, p} characteristic map x m ∈R H×W×C H, W, C in the formula represent the length, width, and channel number of the input image, where H1, W1, C1 in the formula represent the length, width, and channel number of the feature map, respectively, Mobile m respectively represent the encoding features of m modalities, and the MobileNetV2 network S2.2, a multi-scale sparse feature extraction module DGASPP is used for multi-scale feature extraction of the bottom layer features, the multi-scale sparse feature extraction module DGASPP extracts the encoded features h m Multi-scale feature extraction is performed, and the extraction formula is: In the formula DGASPP m respectively represent the multi-scale coding features of m modalities, and the DGASPP module. S2.3, an adaptive hybrid interactive attention module AHMA is used to extract spatial channel attention features with interactive information in the bottom layer features to solve the problem of small shape difference in the image sample, and the formula for extracting spatial channel attention features with interactive information using the AHMA module is as follows: m r ,m p = AHMA(s r ,s p ) In the formula, respectively represent the attention weighted features of two modalities; AHMA is an adaptive hybrid interactive attention module; S3, a constraint coupling decoding stage: enabling a constraint coupling decoding module, and using a modal decoder to decode the steps The features extracted in the multi-scale interactive attention feature extraction module in S2.2, the network Deeplabv3 with an improved encoding and decoding structure is used in the modal decoder, and the improvement process of the decoding part is as follows: S3.1, additional feature splicing convolution CConv and shortcut convolution SConv are added to obtain decoding features at more scales; S3.2, a consistency constraint loss is added between decoding features of different modalities to capture interactive features in the decoding features; S3.3, a reconstruction loss is added between the input image and the reconstructed image to extract shape features; S3.4, a total target loss function composed of a task loss, a consistency loss, and a reconstruction loss is constructed to guide the network to learn related features, and the formula of the total target loss function is as follows: L total = μL task + (1 - μ) (L consis + L recon ) where: μ is a loss function adjustment factor, L task is a task loss function, L consis is a consistency loss function, L recon is a reconstruction loss function; S4, a fusion inference stage: global average pooling and a multilayer perceptron are used to fuse and infer the interactive attention features, and the fusion inference formula is as follows: In the formula, P represents the fusion inference output, MLP represents the multilayer perceptron, avg represents the average pooling operation, and represents the channel splicing operation; S5, a result display stage: displaying the scores of each category and the belonging category of part of the test set image samples.

2. The image interactive fusion method for wheelset tread defect identification according to claim 1, characterized in that, The constraint coupling decoding module is used to assist the multi-scale interactive attention feature extraction stage in training the network and does not participate in the test.

3. The image interactive fusion method for wheelset tread defect identification according to claim 1, characterized in that, The multi-scale sparse feature extraction module DGASPP is formed by introducing a ghost module and a dense connection idea into the atrous spatial pyramid pooling module ASPP.

4. The image interactive fusion method for wheelset tread defect identification according to claim 1, characterized in that, The adaptive hybrid interactive attention module AHMA in step S2.3 is composed of a hybrid non-local module HNL, an attention matrix fusion, and an adaptive attention reliability weight.

5. The image interactive fusion method for wheelset tread defect identification according to claim 1, characterized in that, The task loss function L in the step S3.4 task The formula is: L task = -a(1 - P) γ log(P) In the formula, α=0.25 represents a weight factor, and γ=2 represents an adjustment factor.

6. The image interactive fusion method for wheelset tread defect identification according to claim 1, characterized in that, The consistency loss function L in the step S3.4 consis The formula is: The 16-fold down-sampling, 8-fold down-sampling decoding features of the m modalities are respectively represented as wherein m∈{r,p}, in the formula respectively represent the decoding features of the 16-fold down-sampled RGB image modality and the Poisson modality POS image modality, respectively represent the decoding features of the 8-fold down-sampled RGB image modality and the Poisson modality POS image modality, and the consistency loss function is obtained by using a mean square error (MSE) loss at the 16-fold and 8-fold down-sampled decoding features between the modalities.

7. The image interactive fusion method for wheelset tread defect identification according to claim 6, characterized in that, 16x down-sampling, 8x down-sampling decoding features of the m modalities and The formulas are respectively: wherein up2 denotes a bilinear interpolation 2x upsampling operation, and denotes a channel concatenation operation.

8. The image interactive fusion method for wheelset tread defect identification according to claim 7, characterized in that, The decoding feature CConv8 in the formula and the decoding feature CConv in the formula 16 have the same topology.

9. The image interactive fusion method for wheelset tread defect identification according to claim 1, characterized in that, The reconstruction loss function L in step S3.4 is: recon is: where x r , x p represent the RGB image and Poisson image input, respectively, r r , r p represent the RGB image and Poisson image decoded reconstruction output, respectively.

10. The image interactive fusion method for wheelset tread defect identification according to claim 1, characterized in that, The image interactive fusion method is verified and analyzed by experiments.

Images