Lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation

Abstract

The invention discloses a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation, and relates to the technical field of image processing. Comprising a training stage and a testing stage: in the training stage, through a knowledge distillation framework, hyperspherical feature mapping distillation, frequency domain attention distillation and corresponding prediction map distillation are compared, and knowledge in a double-flow multi-teacher network is transferred to a lightweight single-flow student network; the performance of the student model is improved as much as possible while it is ensured that the parameter quantity of the uniflow student model is kept small. Inputting the data-enhanced rail surface defect image into a convolutional neural network for training to obtain a significant rail surface defect prediction map; in a test stage, inputting a to-be-detected rail surface defect image into the uniflow student network training model to obtain a significant rail surface defect prediction map; the method has the advantage of improving the detection efficiency and accuracy of the rail surface defect image.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation. Background Art

[0002] As the backbone of the modern transportation network, railway transportation carries a large number of passenger and freight transportation tasks. Its efficient and economical transportation mode plays a vital role in economic development. As the basic structure of the railway system, the quality and status of the rails are directly related to the safety and reliability of train operation. During the railway operation, the rails may have defects such as scars, holes, and scratches due to various factors. If these defects are not detected in time, serious accidents may occur, causing huge economic losses and casualties. Therefore, timely detection and maintenance of rail surface defects is a key link to ensure the safety of railway transportation.

[0003] Traditional rail surface defect detection methods mainly rely on manual inspections or simple automated equipment, which are often inefficient and easily affected by human factors and environmental conditions. With the advancement of science and technology, salient object detection methods based on RGB-D (red, green, blue, and depth) technology have begun to be applied to rail surface defect detection. These methods can identify potential defect areas by analyzing the RGB information and depth information of the rail surface. However, traditional RGB-D salient object detection methods rely too much on artificially designed prior knowledge in feature extraction, which not only increases the workload, but also limits the generalization ability and robustness of the detection system.

[0004] In recent years, with the introduction of convolutional neural networks (CNN) and the development of deep learning technology, RGB-D salient object detection algorithms based on deep learning have shown great potential. These algorithms can automatically learn the feature representation of rail surface defects from a large amount of data without manual intervention, greatly improving the accuracy and efficiency of detection. However, with the improvement of detection accuracy, models based on deep learning often require more parameters and higher computing resources. Due to the limited resources and computing power of the terminal devices equipped with algorithms in actual engineering application scenarios, a large number of models with excellent performance cannot be actually applied in actual engineering scenarios due to the limitation of model parameters.

[0005] In order to solve this problem, model compression technology came into being. Model compression aims to reduce the number of model parameters and computational complexity while maintaining the performance of the model as much as possible. Common model compression technologies include pruning, quantization, and knowledge distillation, among which knowledge distillation is the most widely used model compression technology. Knowledge distillation designs a teacher network with good performance and a lightweight student network, and continuously transfers the more advanced features extracted from the teacher network with higher accuracy to the student network with smaller parameters during training iterations, so that the student network can continuously learn the knowledge transferred from the teacher network, thereby further improving the performance of the student network while maintaining a low computational complexity.

[0006] Most knowledge distillation architectures design a teacher network corresponding to a student network, and design a series of distillation losses for constraints, so as to ensure that the student model can well receive valuable knowledge from the teacher network. However, the ability of a teacher has certain limitations. When the number of student model parameters is very small, the single knowledge structure of a single teacher model may have a very limited effect on the improvement of the student model performance, resulting in some knowledge not being well utilized by the student model during the migration process. This requires the exploration of more effective knowledge distillation strategies.

[0007] Therefore, it is an urgent problem for those skilled in the art to propose a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation to solve the difficulties existing in the prior art. Summary of the invention

[0008] In view of this, the present invention provides a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation, which improves the detection efficiency and accuracy of rail surface defect images.

[0009] In order to achieve the above object, the present invention adopts the following technical solution:

[0010] A lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation includes two processes: training phase and testing phase:

[0011] The specific steps of the training phase are as follows:

[0012] S1-1, selecting a plurality of original rail surface defect images, a depth map corresponding to each original rail surface defect image, and a true significant rail surface defect image to form a training set;

[0013] S1-2. Based on the training set obtained in S1, a knowledge distillation convolutional neural network is constructed, including a two-stream teacher convolutional neural network and a student convolutional neural network, wherein the two-stream teacher convolutional neural network includes: a first teacher convolutional neural network and a second teacher convolutional neural network;

[0014] S1-3, training the first teacher convolutional neural network and the second teacher convolutional neural network to obtain weights of the first teacher convolutional neural network and the second teacher convolutional neural network respectively;

[0015] S1-4, performing data enhancement on each original rail surface defect image and the corresponding depth map in the training set, and inputting them as the original input image into the knowledge distillation convolutional neural network for training to obtain the corresponding predicted rail surface defect image;

[0016] S1-5, calculate the loss function value between the predicted image and the corresponding real image;

[0017] S1-6, repeat S1-4 and S1-5 until the convolutional neural network converges, use the mean absolute error as the criterion for preserving network weights during training, and obtain the multi-teacher knowledge distillation framework;

[0018] The testing phase includes the following steps:

[0019] S2-1: Select the Pth group of rail surface defect images to be detected in the test set and mark them as I p , where 1≤p≤P, P=362;

[0020] S2-2: Based on the training results in S1, a multi-teacher knowledge distillation framework is constructed, and various channels of the P-th group of rail surface defect images to be detected are input into the multi-teacher knowledge distillation framework, and the salient rail surface defect prediction image corresponding to the rail surface defect image is output. The obtained prediction image is recorded as S p pre .

[0021] In the above method, optionally, the knowledge distillation convolutional neural network in S1-2 includes: a two-stream teacher convolutional neural network and a single-stream student convolutional neural network;

[0022] Among them, the first teacher convolutional neural network takes RGB images and depth images as input, including a backbone network, an adaptive channel registration module, and a multi-level frequency domain interleaved decoder;

[0023] The first teacher convolutional neural network uses shunted-b as the backbone network;

[0024] The second teacher convolutional neural network takes the RGB image and the depth image as input, and includes a backbone network, an adaptive channel registration module, and a multi-level frequency domain interleaved decoder;

[0025] The second teacher convolutional neural network uses shunted-s as the backbone network;

[0026] The single-stream student convolutional neural network takes RGB images as the input of the model and uses shunted-t as the backbone network, which consists of two parts: an adaptive channel registration module and a multi-level frequency domain interleaved decoder.

[0027] The above method, optionally, first teacher convolutional neural network:

[0028] The RGB image passes through the first layer of the first teacher convolutional neural network backbone, the second layer of the first teacher convolutional neural network backbone, the third layer of the first teacher convolutional neural network backbone, and the fourth layer of the first teacher convolutional neural network backbone in sequence;

[0029] The depth image sequentially passes through the fifth layer of the first teacher convolutional neural network backbone, the sixth layer of the first teacher convolutional neural network backbone, the seventh layer of the first teacher convolutional neural network backbone, and the eighth layer of the first teacher convolutional neural network backbone;

[0030] The features output by the teacher backbone network corresponding to the vertical direction are input to the first teacher adaptive registration module 1, the first teacher adaptive channel registration module 2, the first teacher adaptive channel registration module 3 and the first teacher adaptive channel registration module 4, and are transmitted layer by layer between the first teacher adaptive channel registration modules;

[0031] Finally, the obtained adaptive channel alignment features are passed to the corresponding first teacher multi-level frequency domain interleaved decoder one, the first teacher multi-level frequency domain interleaved decoder two, the first teacher multi-level frequency domain interleaved decoder three and the first teacher frequency domain interleaved decoder four, and finally the first teacher prediction output one, the first teacher prediction output two, the first teacher prediction output three and the first teacher prediction output four are obtained.

[0032] The above method, optionally, a second teacher convolutional neural network:

[0033] The RGB image passes through the first layer of the second teacher convolutional neural network backbone, the second layer of the second teacher convolutional neural network backbone, the third layer of the second teacher convolutional neural network backbone, and the fourth layer of the second teacher convolutional neural network backbone in sequence;

[0034] The depth image passes through the fifth layer of the second teacher convolutional neural network backbone, the sixth layer of the second teacher convolutional neural network backbone, the seventh layer of the second teacher convolutional neural network backbone, and the eighth layer of the second teacher convolutional neural network backbone in sequence;

[0035] The features output by the teacher backbone network corresponding to the vertical direction are input to the second teacher adaptive registration module 1, the second teacher adaptive channel registration module 2, the second teacher adaptive channel registration module 3 and the second teacher adaptive channel registration module 4, and are transmitted layer by layer between the second teacher adaptive channel registration modules;

[0036] Finally, the obtained adaptive channel alignment features are passed to the corresponding second teacher multi-level frequency domain interleaved decoder one, the second teacher multi-level frequency domain interleaved decoder two, the second teacher multi-level frequency domain interleaved decoder three and the first teacher frequency domain interleaved decoder four, and finally the second teacher prediction output one, the second teacher prediction output two, the second teacher prediction output three and the second teacher prediction output four are obtained.

[0037] The above method, optional, single-stream student network:

[0038] The RGB image passes through the first layer of the student backbone network, the second layer of the student backbone network, the third layer of the student backbone network and the fourth layer of the student backbone network in sequence;

[0039] The features output by the student backbone network corresponding to the vertical direction are input into the student adaptive registration module 1, the student adaptive channel registration module 2, the student adaptive channel registration module 3 and the student adaptive channel registration module 4, and are transmitted layer by layer between the student adaptive channel registration modules;

[0040] Finally, the obtained adaptive channel alignment features are passed to the corresponding student multi-level frequency domain interleaved decoder 1, student multi-level frequency domain interleaved decoder 2, student multi-level frequency domain interleaved decoder 3 and student frequency domain interleaved decoder 4, and finally the student prediction output 1, student prediction output 2, student prediction output 3 and student prediction output 4 are obtained.

[0041] The above method, optionally, the multi-teacher knowledge distillation framework in S2-2 includes:

[0042] (1) Freeze the weights of the first teacher convolutional neural network and the second teacher convolutional neural network, design a hypersphere contrast distillation loss, and obtain hypersphere mapping features and frequency domain attention features, wherein the distillation loss includes: the distillation loss through the final prediction graph and the frequency domain attention distillation loss;

[0043] (2) Each original rail surface defect image and the corresponding depth map in the training set are subjected to data enhancement by brightness, contrast, color adjustment, random flipping, and cropping as the original input image, and then input into the convolutional neural network for training to obtain the significant predicted rail surface defect image, denoted as S pre ;

[0044] (3) Calculate the loss function value between the predicted image and the corresponding real image;

[0045] (4) Repeat (2) and (3) until the convolutional neural network converges. During the training process, the mean absolute error is used as the criterion for saving the network weights. The network weight with the minimum mean absolute error value is saved as the output of the student network model training, which is recorded as We obtain a multi-teacher knowledge distillation framework.

[0046] The above method, optional, hypersphere mapping feature:

[0047] First, the first teacher adaptive channel registration features 1 to 4 are respectively subjected to the first teacher hypersphere mapping to obtain four hypersphere mapping features denoted as Similarly, the second teacher adaptive channel registration features 1 to 4 are respectively subjected to the second teacher hypersphere mapping to obtain four hypersphere mapping features denoted as Finally, the student adaptive channel registration features 1 to 4 are respectively subjected to student hypersphere mapping to obtain four hypersphere mapping features denoted as

[0048] Frequency domain attention features:

[0049] First, the first teacher multi-level frequency domain interleaved features 1 to 4 are respectively subjected to the first teacher frequency domain attention to obtain four frequency domain attention features recorded as Similarly, the second teacher multi-level frequency domain interleaved features 1 to 4 are respectively subjected to the second teacher frequency domain attention to obtain four frequency domain attention features recorded as Finally, the student multi-level frequency domain interleaved features 1 to 4 are used as the student frequency domain attention to obtain four frequency domain attention features recorded as

[0050] The above method optionally calculates the loss function value between the predicted image and the corresponding real image:

[0051] The binary cross entropy loss is used to calculate the true semantic segmentation image {I True The loss function value BCELoss(I True (i,j),S pre ); the loss function (IoU) of the image target detection task is used to calculate the set of real semantic segmentation images {I True The loss function value IoU(I True (i,j),S pre ), using the sum of these two losses as the loss function value;

[0052] Secondly, freeze the gradient updates of the first teacher convolutional neural network and the second teacher convolutional neural network, and use contrast loss to constrain the hypersphere mapping features for knowledge transfer, which is recorded as:

[0053]

[0054] Where i = 1, 2, 3, 4;

[0055] Use KL divergence to transfer knowledge of frequency domain attention features;

[0056] Recorded as

[0057] Where i = 1, 2, 3, 4;

[0058] Finally, the binary cross entropy loss and intersection-over-union loss are used to constrain the prediction graphs of the first teacher network and the second teacher network to the prediction graph of the student network, thereby further improving the performance of the student model, which is recorded as:

[0059]

[0060] In the above method, optionally, the specific contents of S2-2 include:

[0061] The various channels of the P group of rail surface defect images to be detected are input into the multi-teacher knowledge distillation framework. The various channels include: R channel component, G channel component, B channel component and depth map component. The significant rail surface defect prediction map corresponding to the rail surface defect image is output. The obtained prediction image is recorded as

[0062] It can be seen from the above technical solution that compared with the prior art, the lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation of the present invention has the following beneficial effects:

[0063] By comparing the hypersphere feature map distillation, frequency domain attention distillation and corresponding prediction map distillation, the knowledge in the dual-stream multi-teacher network is transferred to the relatively lightweight single-stream student network, while ensuring that the number of single-stream student model parameters remains small, the student model performance is improved as much as possible. Data preprocessing uses data-enhanced rail surface defect images to be input into the convolutional neural network for training to obtain a significant rail surface defect prediction map; in the test phase, the rail surface defect image to be detected is input into the single-stream student network training model to obtain a significant rail surface defect prediction map; the advantage is that the detection efficiency and accuracy of rail surface defect images are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0065] Figure 1 A flow chart of a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation provided by the present invention;

[0066] Figure 2 The overall implementation block diagram of the first teacher network method provided by the present invention;

[0067] Figure 3 The adaptive channel registration module provided by the present invention;

[0068] Figure 4 A multi-stage frequency domain interleaving decoder provided by the present invention;

[0069] Figure 5 The overall implementation block diagram of the second teacher network method provided by the present invention;

[0070] Figure 6 The overall implementation block diagram of the student network method provided by the present invention;

[0071] Figure 7 A block diagram for implementing the multi-teacher knowledge distillation method provided by the present invention;

[0072] Figure 8a is an original rail surface image;

[0073] Figure 8b In order to utilize the method of the present invention Figure 8a The original rail surface image shown is predicted to obtain a predicted rail surface defect detection image, and the defect type is scar;

[0074] Figure 9a is another original rail surface image;

[0075] Figure 9b In order to utilize the method of the present invention Figure 9a The original rail surface image shown is predicted to obtain a predicted rail surface defect detection image, and the defect type is a hole;

[0076] Fig.10a is another original rail surface image;

[0077] Fig.10b In order to utilize the method of the present invention Fig.10a The original rail surface image shown is predicted to obtain a predicted rail surface defect detection image, and the defect type is scratches. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0080] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.

[0081] Reference Figure 1 As shown, the present invention discloses a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation, including two processes: a training phase and a testing phase:

[0082] The specific steps of the training phase are as follows:

[0083] S1-1, selecting a plurality of original rail surface defect images, a depth map corresponding to each original rail surface defect image, and a true significant rail surface defect image to form a training set;

[0084] S1-2. Based on the training set obtained in S1, a knowledge distillation convolutional neural network is constructed, including a two-stream teacher convolutional neural network and a student convolutional neural network, wherein the two-stream teacher convolutional neural network includes: a first teacher convolutional neural network and a second teacher convolutional neural network;

[0085] S1-3, training the first teacher convolutional neural network and the second teacher convolutional neural network to obtain weights of the first teacher convolutional neural network and the second teacher convolutional neural network respectively;

[0086] S1-4, performing data enhancement on each original rail surface defect image and the corresponding depth map in the training set, and inputting them as the original input image into the knowledge distillation convolutional neural network for training to obtain the corresponding predicted rail surface defect image;

[0087] S1-5, calculate the loss function value between the predicted image and the corresponding real image;

[0088] S1-6, repeat S1-4 and S1-5 until the convolutional neural network converges, use the mean absolute error as the criterion for preserving network weights during training, and obtain the multi-teacher knowledge distillation framework;

[0089] The testing phase includes the following steps:

[0090] S2-1: Select the Pth group of rail surface defect images to be detected in the test set and mark them as I p , where 1≤p≤P, P=362;

[0091] S2-2: Based on the training results in S1, a multi-teacher knowledge distillation framework is constructed, and various channels of the P-th group of rail surface defect images to be detected are input into the multi-teacher knowledge distillation framework, and the salient rail surface defect prediction image corresponding to the rail surface defect image is output. The obtained prediction image is recorded as S p pre .

[0092] Specifically, S1-3: Before performing knowledge distillation training, the first teacher convolutional neural network and the second teacher convolutional neural network need to be trained to obtain the weights of the corresponding teacher networks respectively. The steps of training the student network and the teacher network separately are introduced below.

[0093] S1-4: Each original rail surface defect image and the corresponding depth map in the training set are enhanced by brightness, contrast, color adjustment, random flipping, cropping, etc., and then used as the original input image, which is input into the convolutional neural network for training to obtain the corresponding predicted rail surface defect image. The predicted image is an image containing only significant rail surface defects. The output of the significant predicted rail surface defect image obtained by the first teacher convolutional neural network and the second teacher convolutional neural network is recorded as and

[0094] S1-5: Calculate the loss function value between the predicted image and the corresponding real image; use binary cross entropy loss (BCEloss) to calculate the real semantic segmentation image {I True The loss function value between the loss of the set consisting of (i, j)} and the set consisting of the significant rail surface defect prediction map is and The loss function (IoU) of the image target detection task is used to calculate the set of real semantic segmentation images {I True The loss function value between the loss of the set consisting of (i, j)} and the significant rail surface defect prediction map and The sum of these two losses is used as the loss function value.

[0095] S1-6: Repeat S1-4 and S1-5 for V times until the convolutional neural network converges. During the training process, the mean absolute error (MAE) is used as the standard for saving network weights. During the convergence process of network training, the network weights of each round will be calculated with the validation set with the parameters without updating the gradient. Then the network weight with the smallest MAE value will be saved as the output of our first teacher convolutional neural network model training, which is recorded as The output of the second teacher convolutional neural network model training is recorded as Wherein, V>1, and in this embodiment, V=200.

[0096] Furthermore, the knowledge distillation convolutional neural network in S1-2 includes: a two-stream teacher convolutional neural network and a single-stream student convolutional neural network;

[0097] Among them, the first teacher convolutional neural network takes RGB images and depth images as input, including a backbone network, an adaptive channel registration module, and a multi-level frequency domain interleaved decoder;

[0098] The first teacher convolutional neural network uses shunted-b as the backbone network;

[0099] The second teacher convolutional neural network takes the RGB image and the depth image as input, and includes a backbone network, an adaptive channel registration module, and a multi-level frequency domain interleaved decoder;

[0100] The second teacher convolutional neural network uses shunted-s as the backbone network;

[0101] The single-stream student convolutional neural network takes RGB images as the input of the model and uses shunted-t as the backbone network, which consists of two parts: an adaptive channel registration module and a multi-level frequency domain interleaved decoder.

[0102] Furthermore, the first teacher convolutional neural network is Figure 2 As shown:

[0103] The RGB image passes through the first layer of the first teacher convolutional neural network backbone, the second layer of the first teacher convolutional neural network backbone, the third layer of the first teacher convolutional neural network backbone, and the fourth layer of the first teacher convolutional neural network backbone in sequence;

[0104] The depth image sequentially passes through the fifth layer of the first teacher convolutional neural network backbone, the sixth layer of the first teacher convolutional neural network backbone, the seventh layer of the first teacher convolutional neural network backbone, and the eighth layer of the first teacher convolutional neural network backbone;

[0105] The features output by the teacher backbone network corresponding to the vertical direction are input to the first teacher adaptive registration module 1, the first teacher adaptive channel registration module 2, the first teacher adaptive channel registration module 3 and the first teacher adaptive channel registration module 4, and are transmitted layer by layer between the first teacher adaptive channel registration modules;

[0106] Finally, the obtained adaptive channel alignment features are passed to the corresponding first teacher multi-level frequency domain interleaved decoder one, the first teacher multi-level frequency domain interleaved decoder two, the first teacher multi-level frequency domain interleaved decoder three and the first teacher frequency domain interleaved decoder four, and finally the first teacher prediction output one, the first teacher prediction output two, the first teacher prediction output three and the first teacher prediction output four are obtained.

[0107] Specifically, the input of the first teacher neural network is the RGB image and depth image of each image. The width and height of the image are W=256 and H=256 respectively (RGB channels are R channel components, G channel components and B channel components, and the depth image is a single channel component, namely D channel). Both feature extractions are performed through the shunted-B backbone network. The structures of the two feature extraction flows are similar, specifically:

[0108] Each layer of the first teacher convolutional neural network backbone consists of a linear embedding layer and several shunt transformer blocks, each of which is composed of layer normalization, shunt multi-head self-attention, layer normalization and a specific detail feedback module. The number of input channels of the first layer is 3. After three shunt transformer blocks with two-head self-attention, the number of channels becomes 64, and the image size becomes The second layer input channel is the input of the first layer, and then passes through four shunt transformer blocks with four self-attention heads. The number of channels becomes 128 and the image size becomes The input of the third layer is the output of the second layer. After passing through twenty-four shunt transformer blocks with eight self-attention heads, the number of channels becomes 256 and the image size becomes The input of the fourth layer is the output of the third layer. After passing through two shunt transformer blocks with sixteen self-attention heads, the number of channels becomes 512 and the image size becomes

[0109] The output of the first teacher's color image after the backbone network is recorded as The output of the depth map after the backbone network is recorded as Then the features of the corresponding backbone network are input into the first teacher adaptive channel registration module for feature fusion. The specific fusion process is as follows: Figure 5 As shown:

[0110] The first teacher's adaptive channel registration network first receives the color map and depth map output of the first teacher's convolutional neural network backbone of the corresponding layer as input. Except for the first teacher's adaptive channel registration module one that additionally receives the features spliced ​​from the corresponding teacher backbone network, the remaining three layers all use the adaptive channel registration features of the previous layer as input, which are recorded as input one, input two, and input three respectively. First, input one, input two, and input three are first spliced ​​along the channel dimension and input into the channel attention module. Finally, they are normalized by the sigmoid activation function to generate a channel mask with screening ability in the channel dimension. Finally, input one and input three are multiplied by features, and the three are added together. Finally, the added features are spliced ​​with input three in the channel dimension to obtain the final output one. The first teacher's adaptive channel registration modules one to four all perform the same operation, and finally the obtained output is sent to the first teacher's multi-level frequency domain interleaved decoder. The specific process is as follows Figure 4 As shown:

[0111] Except for the fourth layer, the input of the first teacher multi-level frequency domain interleaved decoder is provided by the first teacher adaptive registration module four, and the remaining three are provided by the output of the first teacher multi-level frequency domain interleaved decoder of the previous layer, denoted as input four and input five. First, input four is sent to the decomposition high-pass filter one and the decomposition low-pass filter one to realize the conversion of the color image to the frequency domain features, and obtain the low-frequency feature one and the high-frequency feature one respectively. The low-frequency feature one is further sent to the discrete wavelet transform one for further processing to obtain the discrete wavelet feature one; input five is sent to the decomposition high-pass filter two and the decomposition low-pass filter two to realize the conversion of the depth map to the frequency domain features, and obtain the low-frequency feature two and the high-frequency feature two respectively. The high-frequency feature two is sent to discrete wavelet transform two for further processing to obtain discrete wavelet feature two; then, discrete wavelet transform one and discrete wavelet transform are spliced ​​in channel dimension and then sent to reconstruction low-pass filter one and reconstruction low-pass filter two for feature reconstruction; then, the high-pass filter feature one and discrete wavelet feature one are added, and the high-pass filter feature two and discrete wavelet feature two are added, and sent to reconstruction high-pass filter one and reconstruction high-pass filter two for feature reconstruction, respectively, and then an inverse Fourier transform is performed to remap them back to the original feature space, and finally multiplied with input four and input five respectively and added to obtain output two, which is the first teacher output prediction map of each layer.

[0112] Furthermore, the second teacher convolutional neural network is Figure 5 As shown:

[0113] The RGB image passes through the first layer of the second teacher convolutional neural network backbone, the second layer of the second teacher convolutional neural network backbone, the third layer of the second teacher convolutional neural network backbone, and the fourth layer of the second teacher convolutional neural network backbone in sequence;

[0114] The depth image passes through the fifth layer of the second teacher convolutional neural network backbone, the sixth layer of the second teacher convolutional neural network backbone, the seventh layer of the second teacher convolutional neural network backbone, and the eighth layer of the second teacher convolutional neural network backbone in sequence;

[0115] The features output by the teacher backbone network corresponding to the vertical direction are input to the second teacher adaptive registration module 1, the second teacher adaptive channel registration module 2, the second teacher adaptive channel registration module 3 and the second teacher adaptive channel registration module 4, and are transmitted layer by layer between the second teacher adaptive channel registration modules;

[0116] Finally, the obtained adaptive channel alignment features are passed to the corresponding second teacher multi-level frequency domain interleaved decoder one, the second teacher multi-level frequency domain interleaved decoder two, the second teacher multi-level frequency domain interleaved decoder three and the first teacher frequency domain interleaved decoder four, and finally the second teacher prediction output one, the second teacher prediction output two, the second teacher prediction output three and the second teacher prediction output four are obtained.

[0117] Specifically, the input of the second teacher neural network is also the RGB image and depth image of each image. The width and height of the image are W=256 and H=256 respectively. Both are extracted through the shunted-S backbone network. The structures of the two feature extraction flows are similar, specifically:

[0118] The second teacher convolutional neural network backbone, the number of input channels of the first layer is 3, after 2 split transformer blocks with dual-head self-attention, the number of channels becomes 64, and the image size becomes The second layer input channel is the input of the first layer, and then passes through four shunt transformer blocks with four self-attention heads. The number of channels becomes 128 and the image size becomes The input of the third layer is the output of the second layer. After twelve shunt transformer blocks with eight self-attention heads, the number of channels becomes 256 and the image size becomes The input of the fourth layer is the output of the third layer. After passing through a shunt transformer block with sixteen self-attention heads, the number of channels becomes 512 and the image size becomes

[0119] The output of the second teacher's color image after the backbone network is recorded as The output of the depth map after the backbone network is recorded as Then the features of the corresponding backbone network are input into the second teacher adaptive channel registration module for feature fusion. The specific fusion process is as follows: Figure 3 As shown:

[0120] The second teacher's adaptive channel registration network first receives the color map and depth map output of the second teacher's convolutional neural network backbone of the corresponding layer as input. Except for the second teacher's adaptive channel registration module one that additionally receives the features spliced ​​from the corresponding teacher backbone network, the remaining three layers all use the adaptive channel registration features of the previous layer as input, which are recorded as input one, input two, and input three respectively. First, input one, input two, and input three are first spliced ​​along the channel dimension and input into the channel attention module. Finally, they are normalized by the sigmoid activation function to generate a channel mask with screening ability in the channel dimension. Finally, input one and input three are multiplied by features, and the three are added together. Finally, the added features are spliced ​​with input three in the channel dimension to obtain the final output one. The second teacher's adaptive channel registration modules one to four perform the same operation, and finally the obtained output is sent to the second teacher's multi-level frequency domain interleaved decoder. The specific process is as follows Figure 4 As shown:

[0121] Except for the fourth layer, the input of the second teacher multi-level frequency domain interleaved decoder is provided by the second teacher adaptive registration module four, and the remaining three are provided by the output of the second teacher multi-level frequency domain interleaved decoder of the previous layer, denoted as input four and input five. First, input four is sent to the decomposition high-pass filter one and the decomposition low-pass filter one to realize the conversion of the color image to the frequency domain features, and obtain the low-frequency feature one and the high-frequency feature one respectively. The low-frequency feature one is further sent to the discrete wavelet transform one for further processing to obtain the discrete wavelet feature one; input five is sent to the decomposition high-pass filter two and the decomposition low-pass filter two to realize the conversion of the depth map to the frequency domain features, and obtain the low-frequency feature two and the high-frequency feature two respectively. The high-frequency feature two is sent to discrete wavelet transform two for further processing to obtain discrete wavelet feature two; then, discrete wavelet transform one and discrete wavelet transform are spliced ​​in channel dimension and then sent to reconstruction low-pass filter one and reconstruction low-pass filter two for feature reconstruction; then, the high-pass filter feature one and discrete wavelet feature one are added, and the high-pass filter feature two and discrete wavelet feature two are added, and sent to reconstruction high-pass filter one and reconstruction high-pass filter two for feature reconstruction, respectively, and then an inverse Fourier transform is performed to remap them back to the original feature space, and finally multiplied with input four and input five respectively and added to obtain output two, which is the second teacher output prediction map of each layer.

[0122] Further, single-stream student network:

[0123] The RGB image passes through the first layer of the student backbone network, the second layer of the student backbone network, the third layer of the student backbone network and the fourth layer of the student backbone network in sequence;

[0124] The features output by the student backbone network corresponding to the vertical direction are input into the student adaptive registration module 1, the student adaptive channel registration module 2, the student adaptive channel registration module 3 and the student adaptive channel registration module 4, and are transmitted layer by layer between the student adaptive channel registration modules;

[0125] Finally, the obtained adaptive channel alignment features are passed to the corresponding student multi-level frequency domain interleaved decoder 1, student multi-level frequency domain interleaved decoder 2, student multi-level frequency domain interleaved decoder 3 and student frequency domain interleaved decoder 4, and finally the student prediction output 1, student prediction output 2, student prediction output 3 and student prediction output 4 are obtained.

[0126] Specifically, the input of the student neural network is the RGB image of each picture, the width and height of the picture are W = 256 and H = 256 respectively, and the features are extracted through the shunted-t backbone network, specifically:

[0127] Student backbone network. The number of input channels in the first layer is 3. After a split transformer block with dual-head self-attention, the number of channels becomes 64 and the image size becomes The second layer input channel is the input of the first layer, and then passes through two shunt transformer blocks with four self-attention heads. The number of channels becomes 128 and the image size becomes The input of the third layer is the output of the second layer. After passing through four shunt transformer blocks with eight self-attention heads, the number of channels becomes 256 and the image size becomes The input of the fourth layer is the output of the third layer. After passing through a shunt transformer block with sixteen self-attention heads, the number of channels becomes 512 and the image size becomes

[0128] The output of the student's color image after the backbone network is recorded as Then the features of the corresponding backbone network are input into the student adaptive channel registration module for feature fusion. The specific fusion process is as follows: Figure 6 As shown:

[0129] The student adaptive channel registration network first receives the output of the color map of the first teacher convolutional neural network backbone of the corresponding layer as input. Except for the student adaptive channel registration module one which additionally receives the output features from the corresponding student backbone network, the remaining three layers all use the adaptive channel registration features of the previous layer as input, which are recorded as input one, input two and input three respectively. First, input one, input two and input three are first concatenated along the channel dimension and input into the channel attention module. Finally, they are normalized by the sigmoid activation function to generate a channel mask with screening capability in the channel dimension. Finally, input one and input three are feature multiplied and the three are added. Finally, the added features are concatenated with input three in the channel dimension to obtain the final output one. The student adaptive channel registration modules one to four perform the same operation, and finally the obtained output is sent to the student multi-level frequency domain interleaved decoder. The specific process is as follows: Figure 4 As shown:

[0130] Except for the fourth layer, the input of the student multi-level frequency domain interleaved decoder is provided by the student adaptive registration module four, and the remaining three layers are provided by the output of the student multi-level frequency domain interleaved decoder of the previous layer, denoted as input four and input five. First, input four is sent to the decomposition high-pass filter one and the decomposition low-pass filter one to realize the conversion of the color image to the frequency domain features, and obtain the low-frequency feature one and the high-frequency feature one respectively. The low-frequency feature one is further sent to the discrete wavelet transform one for further processing to obtain the discrete wavelet feature one; input five is sent to the decomposition high-pass filter two and the decomposition low-pass filter two to realize the conversion of the depth map to the frequency domain features, and obtain the low-frequency feature two and the high-frequency feature two respectively. The low-frequency feature one is further sent to the discrete wavelet transform one for further processing to obtain the discrete wavelet feature one. Feature 2 is sent to discrete wavelet transform 2 for further processing to obtain discrete wavelet feature 2; then, discrete wavelet transform 1 and discrete wavelet transform are spliced ​​in channel dimension and then sent to reconstruction low-pass filter 1 and reconstruction low-pass filter 2 for feature reconstruction; then, high-pass filter feature 1 and discrete wavelet feature 1 are added, and high-pass filter feature 2 and discrete wavelet feature 2 are added, and then sent to reconstruction high-pass filter 1 and reconstruction high-pass filter 2 for feature reconstruction, followed by inverse Fourier transform, remapped back to the original feature space, and finally multiplied with input 4 and input 5 respectively and added to obtain output 2, which is the student output prediction map of each layer.

[0131] Furthermore, the multi-teacher knowledge distillation framework in S2-2 is as follows Figure 7 As shown, including:

[0132] (1) Freeze the weights of the first teacher convolutional neural network and the second teacher convolutional neural network, design a hypersphere contrast distillation loss, and obtain hypersphere mapping features and frequency domain attention features, wherein the distillation loss includes: the distillation loss through the final prediction graph and the frequency domain attention distillation loss;

[0133] (2) Each original rail surface defect image and the corresponding depth map in the training set are subjected to data enhancement by brightness, contrast, color adjustment, random flipping, and cropping as the original input image, and then input into the convolutional neural network for training to obtain the significant predicted rail surface defect image, denoted as S pre ;

[0134] (3) Calculate the loss function value between the predicted image and the corresponding real image;

[0135] (4) Repeat (2) and (3) until the convolutional neural network converges. During the training process, the mean absolute error is used as the criterion for saving the network weights. The network weight with the minimum mean absolute error value is saved as the output of the student network model training, which is recorded as We obtain a multi-teacher knowledge distillation framework.

[0136] Specifically, repeat steps (2) and (3) for V times until the convolutional neural network converges. During the training process, the mean absolute error (MAE) is used as the standard for saving network weights. During the convergence process of network training, the network weights of each round will be calculated with the validation set with the parameters without updating the gradient. Then the network weight with the smallest MAE value will be saved as the output of our student network model training, which is recorded as Wherein, V>1, and in this embodiment, V=200.

[0137] Furthermore, the hypersphere mapping features:

[0138] First, the first teacher adaptive channel registration features 1 to 4 are respectively subjected to the first teacher hypersphere mapping to obtain four hypersphere mapping features denoted as Similarly, the second teacher adaptive channel registration features 1 to 4 are respectively subjected to the second teacher hypersphere mapping to obtain four hypersphere mapping features recorded as Finally, the student adaptive channel registration features 1 to 4 are respectively subjected to student hypersphere mapping to obtain four hypersphere mapping features denoted as

[0139] Frequency domain attention features:

[0140] First, the first teacher multi-level frequency domain interleaved features 1 to 4 are respectively subjected to the first teacher frequency domain attention to obtain four frequency domain attention features recorded as Similarly, the second teacher multi-level frequency domain interleaved features 1 to 4 are respectively subjected to the second teacher frequency domain attention to obtain four frequency domain attention features recorded as Finally, the student multi-level frequency domain interleaved features 1 to 4 are used as the student frequency domain attention to obtain four frequency domain attention features recorded as

[0141] Furthermore, the loss function value between the predicted image and the corresponding real image is calculated:

[0142] The binary cross entropy loss (BCEloss) is used to calculate the true semantic segmentation image {I True The loss function value BCELoss(I True (i,j),S pre ); the loss function (IoU) of the image target detection task is used to calculate the set of real semantic segmentation images {I True The loss function value IoU(I True (i,j),S pre ), using the sum of these two losses as the loss function value;

[0143] Secondly, freeze the gradient updates of the first teacher convolutional neural network and the second teacher convolutional neural network, and use contrast loss to constrain the hypersphere mapping features for knowledge transfer, which is recorded as:

[0144] Where i = 1, 2, 3, 4;

[0145] Use KL divergence to transfer knowledge of frequency domain attention features;

[0146] Recorded as

[0147] Where i = 1, 2, 3, 4;

[0148] Finally, the binary cross entropy loss and intersection-over-union loss are used to constrain the prediction graphs of the first teacher network and the second teacher network to the prediction graph of the student network, thereby further improving the performance of the student model, which is recorded as:

[0149]

[0150] Furthermore, the specific contents of S2-2 include:

[0151] The various channels of the P group of rail surface defect images to be detected are input into the multi-teacher knowledge distillation framework. The various channels include: R channel component, G channel component, B channel component and depth map component. The significant rail surface defect prediction map corresponding to the rail surface defect image is output. The obtained prediction image is recorded as

[0152] In a specific embodiment, the architecture of a lightweight RGB-D rail surface defect detection network with multi-teacher knowledge distillation is built using the Python-based deep learning library Pytorch3.6. We use five commonly used objective parameters for evaluating target detection methods as evaluation indicators: structural measurement (S m ), maximum enhanced alignment (maxE m ), the maximum F method (maxF m ), mean absolute error (MAE) is used to evaluate the detection performance of rail surface defect images.

[0153] The method of the present invention is used to predict each rail surface defect image in the rail surface defect detection test set, and the predicted significant rail surface defect image corresponding to each rail surface defect image is obtained, reflecting the structural measurement (S) of the defect detection effect of the method of the present invention. m ), maximum enhanced alignment (maxE m ), the maximum F method (maxF m ), and the mean absolute error (MAE) is listed in Table 1. From the data listed in Table 1, it can be seen that the rail surface defect detection result obtained by the method of the present invention is good, indicating that it is feasible and effective to use the method of the present invention to obtain the predicted significant rail surface defect image corresponding to the rail surface defect image.

[0154] Table 1 Evaluation results of the method of the present invention on the test set

[0155]

[0156] Figure 8b In order to utilize the method of the present invention Figure 8a The original rail surface image shown is predicted to obtain a predicted rail surface defect detection image, and the defect type is scar; Figure 9b In order to utilize the method of the present invention Figure 8a The original rail surface image shown is predicted to obtain a predicted rail surface defect detection image, and the defect type is a hole; Fig.10b In order to utilize the method of the present invention Fig.10a The original rail surface image shown in the figure is used for prediction, and the predicted rail surface defect detection image is obtained, and the defect type is scratches. Figure 8a and Figure 8b ,contrast Figure 9a and Figure 9b ,contrast Fig.10a and Fig.10b It can be seen that the detection accuracy of the predicted significant rail surface defect image obtained by the method of the present invention is high.

[0157] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0158] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0159] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0160] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation, characterized in that It includes two processes: training phase and testing phase: The specific steps of the training phase are as follows: S1-1, selecting a plurality of original rail surface defect images, a depth map corresponding to each original rail surface defect image, and a true significant rail surface defect image to form a training set; S1-2. Based on the training set obtained in S1, a knowledge distillation convolutional neural network is constructed, including a two-stream teacher convolutional neural network and a student convolutional neural network, wherein the two-stream teacher convolutional neural network includes: a first teacher convolutional neural network and a second teacher convolutional neural network; S1-3, training the first teacher convolutional neural network and the second teacher convolutional neural network to obtain weights of the first teacher convolutional neural network and the second teacher convolutional neural network respectively; S1-4, performing data enhancement on each original rail surface defect image and the corresponding depth map in the training set, and inputting them as the original input image into the knowledge distillation convolutional neural network for training to obtain the corresponding predicted rail surface defect image; S1-5, calculate the loss function value between the predicted image and the corresponding real image; S1-6, repeat S1-4 and S1-5 until the convolutional neural network converges, use the mean absolute error as the criterion for preserving network weights during training, and obtain the multi-teacher knowledge distillation framework; The testing phase includes the following steps: S2-1: Select the Pth group of rail surface defect images to be detected in the test set and mark them as I p , where 1≤p≤P, P=362; S2-2: Based on the training results in S1, a multi-teacher knowledge distillation framework is constructed, and various channels of the P-th group of rail surface defect images to be detected are input into the multi-teacher knowledge distillation framework, and the salient rail surface defect prediction image corresponding to the rail surface defect image is output. The obtained prediction image is recorded as S p pre .

2. According to claim 1, a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation is characterized in that: The knowledge distillation convolutional neural network in S1-2 includes: a two-stream teacher convolutional neural network and a single-stream student convolutional neural network; Among them, the first teacher convolutional neural network takes RGB images and depth images as input, including a backbone network, an adaptive channel registration module, and a multi-level frequency domain interleaved decoder; The first teacher convolutional neural network uses shunted-b as the backbone network; The second teacher convolutional neural network takes the RGB image and the depth image as input, and includes a backbone network, an adaptive channel registration module, and a multi-level frequency domain interleaved decoder; The second teacher convolutional neural network uses shunted-s as the backbone network; The single-stream student convolutional neural network takes RGB images as the input of the model and uses shunted-t as the backbone network, which consists of two parts: an adaptive channel registration module and a multi-level frequency domain interleaved decoder.

3. According to claim 2, a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation is characterized in that: First Teacher Convolutional Neural Network: The RGB image passes through the first teacher convolutional neural network backbone first layer, the first teacher convolutional neural network backbone second layer, the first teacher convolutional neural network backbone third layer and the first teacher convolutional neural network backbone fourth layer in sequence; The depth image sequentially passes through the fifth layer of the first teacher convolutional neural network backbone, the sixth layer of the first teacher convolutional neural network backbone, the seventh layer of the first teacher convolutional neural network backbone, and the eighth layer of the first teacher convolutional neural network backbone; The features output by the teacher backbone network corresponding to the vertical direction are input to the first teacher adaptive registration module 1, the first teacher adaptive channel registration module 2, the first teacher adaptive channel registration module 3 and the first teacher adaptive channel registration module 4, and are transmitted layer by layer between the first teacher adaptive channel registration modules; Finally, the obtained adaptive channel alignment features are passed to the corresponding first teacher multi-level frequency domain interleaved decoder one, the first teacher multi-level frequency domain interleaved decoder two, the first teacher multi-level frequency domain interleaved decoder three and the first teacher frequency domain interleaved decoder four, and finally the first teacher prediction output one, the first teacher prediction output two, the first teacher prediction output three and the first teacher prediction output four are obtained.

4. A lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation according to claim 2, characterized in that: Second Teacher Convolutional Neural Network: The RGB image passes through the first layer of the second teacher convolutional neural network backbone, the second layer of the second teacher convolutional neural network backbone, the third layer of the second teacher convolutional neural network backbone, and the fourth layer of the second teacher convolutional neural network backbone in sequence; The depth image passes through the fifth layer of the second teacher convolutional neural network backbone, the sixth layer of the second teacher convolutional neural network backbone, the seventh layer of the second teacher convolutional neural network backbone, and the eighth layer of the second teacher convolutional neural network backbone in sequence; The features output by the teacher backbone network corresponding to the vertical direction are input to the second teacher adaptive registration module 1, the second teacher adaptive channel registration module 2, the second teacher adaptive channel registration module 3 and the second teacher adaptive channel registration module 4, and are transmitted layer by layer between the second teacher adaptive channel registration modules; Finally, the obtained adaptive channel alignment features are passed to the corresponding second teacher multi-level frequency domain interleaved decoder one, the second teacher multi-level frequency domain interleaved decoder two, the second teacher multi-level frequency domain interleaved decoder three and the first teacher frequency domain interleaved decoder four, and finally the second teacher prediction output one, the second teacher prediction output two, the second teacher prediction output three and the second teacher prediction output four are obtained.

5. According to claim 2, a lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation is characterized in that: Single Stream Student Network: The RGB image passes through the first layer of the student backbone network, the second layer of the student backbone network, the third layer of the student backbone network and the fourth layer of the student backbone network in sequence; The features output by the student backbone network corresponding to the vertical direction are input into the student adaptive registration module 1, the student adaptive channel registration module 2, the student adaptive channel registration module 3 and the student adaptive channel registration module 4, and are transmitted layer by layer between the student adaptive channel registration modules; Finally, the obtained adaptive channel alignment features are passed to the corresponding student multi-level frequency domain interleaved decoder 1, student multi-level frequency domain interleaved decoder 2, student multi-level frequency domain interleaved decoder 3 and student frequency domain interleaved decoder 4, and finally the student prediction output 1, student prediction output 2, student prediction output 3 and student prediction output 4 are obtained.

6. A lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation according to claim 1, characterized in that: The multi-teacher knowledge distillation framework in S2-2 includes: (1) Freeze the weights of the first teacher convolutional neural network and the second teacher convolutional neural network, design a hypersphere contrast distillation loss, and obtain hypersphere mapping features and frequency domain attention features, wherein the distillation loss includes: the distillation loss through the final prediction graph and the frequency domain attention distillation loss; (2) Each original rail surface defect image and the corresponding depth map in the training set are subjected to data enhancement by brightness, contrast, color adjustment, random flipping, and cropping as the original input image, and then input into the convolutional neural network for training to obtain the significant predicted rail surface defect image, denoted as S pre ; (3) Calculate the loss function value between the predicted image and the corresponding real image; (4) Repeat (2) and (3) until the convolutional neural network converges. During the training process, the mean absolute error is used as the criterion for saving the network weights. The network weight with the minimum mean absolute error value is saved as the output of the student network model training, which is denoted by W. S Best ; Get the multi-teacher knowledge distillation framework.

7. A lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation according to claim 6, characterized in that: Hypersphere mapping features: First, the first teacher adaptive channel registration features 1 to 4 are respectively subjected to the first teacher hypersphere mapping to obtain four hypersphere mapping features denoted as Similarly, the second teacher adaptive channel registration features 1 to 4 are respectively subjected to the second teacher hypersphere mapping to obtain four hypersphere mapping features denoted as Finally, the student adaptive channel registration features 1 to 4 are respectively subjected to student hypersphere mapping to obtain four hypersphere mapping features denoted as Frequency domain attention features: First, the first teacher multi-level frequency domain interleaved features 1 to 4 are respectively subjected to the first teacher frequency domain attention to obtain four frequency domain attention features recorded as Similarly, the second teacher multi-level frequency domain interleaved features 1 to 4 are respectively subjected to the second teacher frequency domain attention to obtain four frequency domain attention features recorded as Finally, the student multi-level frequency domain interleaved features 1 to 4 are used as the student frequency domain attention to obtain four frequency domain attention features recorded as 8. A lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation according to claim 7, characterized in that: Calculate the loss function value between the predicted image and the corresponding real image: The binary cross entropy loss is used to calculate the true semantic segmentation image {I True The loss function value BCELoss(I True (i,j),S pre );Use the loss function of the image target detection task to calculate the set of real semantic segmentation images {I True The loss function value IoU(I True (i,j),S pre ), using the sum of these two losses as the loss function value; Secondly, freeze the gradient updates of the first teacher convolutional neural network and the second teacher convolutional neural network, and use contrast loss to constrain the hypersphere mapping features for knowledge transfer, which is recorded as: Where i = 1, 2, 3, 4; Use KL divergence to transfer knowledge of frequency domain attention features; Recorded as Where i = 1, 2, 3, 4; Finally, the combination of binary cross entropy loss and intersection-over-union ratio is used to constrain the prediction graph of the first teacher network and the prediction graph of the second teacher network to the prediction graph of the student network, which is denoted as:

9. The lightweight RGB-D rail surface defect detection method based on multi-teacher knowledge distillation according to claim 1 is characterized in that: The specific contents of S2-2 include: The various channels of the P group of rail surface defect images to be detected are input into the multi-teacher knowledge distillation framework. The various channels include: R channel component, G channel component, B channel component and depth map component. The significant rail surface defect prediction map corresponding to the rail surface defect image is output. The obtained prediction image is recorded as

Images