A mirror image segmentation method based on knowledge distillation and adversarial learning

Abstract

The present application relates to the field of mirror image segmentation network lightweight, aiming at the problem that the existing mirror image segmentation method has too large parameter quantity, is difficult to train and has high cost of transplantation in limited resource edge device, a mirror image segmentation method based on knowledge distillation and adversarial learning is proposed, including the following steps: preprocessing the mirror image; using multiple loss training student network to obtain mirror feature map reflecting mirror image features; calculating distillation loss; taking the mirror feature map output by the student network and the teacher network as input to calculate adversarial loss; training and testing using public data set, and outputting mirror image segmentation result. The present application introduces knowledge distillation theory and adversarial learning strategy to perform model migration learning, so that an optimized model parameter and efficiency can be trained, system resource occupation is reduced, and certain accuracy is retained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of mirror image segmentation network lightweight, and particularly relates to a mirror image segmentation method based on knowledge distillation and adversarial learning. BACKGROUND

[0002] The present application relates to the field of mirror image segmentation network lightweight, and particularly relates to a mirror image segmentation method based on knowledge distillation and adversarial learning.

[0003] Mirrors are ubiquitous in life, but existing computer vision systems do not take into account the existence of mirrors, which can cause the system to confuse objects in the mirror with objects in the outside world, resulting in a significant decline in recognition performance. Mirrors usually reflect similar content around them, which makes it difficult for the system to correctly distinguish between the two, resulting in unpredictable errors, so it is crucial to detect the presence of these mirrors.

[0004] However, existing models designed for mirror image segmentation tasks focus more on improving accuracy and ignore efficiency. The better the performance of the neural network, the deeper the network, the wider the network structure, and the more parameters the network has, which will result in more storage requirements and computational overhead. These networks are often unable to be successfully transplanted to mobile devices and embedded devices. Therefore, it is crucial to design a lightweight and efficient model for mirror segmentation tasks.

[0005] The present application takes the mirror image segmentation task as the research object, introduces the knowledge distillation theory and the adversarial learning strategy to the model through the research of mainstream lightweight neural network model, and carries out the transfer learning, so as to train the model algorithm which can optimize the model parameters and efficiency, reduce the system resource occupation, and retain a certain accuracy. SUMMARY

[0006] The present application relates to the field of mirror image segmentation network lightweight, and particularly relates to a mirror image segmentation method based on knowledge distillation and adversarial learning.

[0007] The technical scheme adopted by the present application to solve its technical problems comprises the following steps:

[0008] Step 1: Preprocess the mirror image;

[0009] Step 2: Train the student network using multiple loss functions, so that the student network learns from the mirror image obtained after step 1, and obtains a mirror feature map reflecting the features of the mirror image;

[0010] Step 3: Use the mirror feature map output by the teacher network and compare it with the mirror feature map obtained in step 2 to calculate the distillation loss;

[0011] Step 4: Construct a mirror segmentation adversarial learning framework, input the feature maps output by the student network and the teacher network to calculate the adversarial loss;

[0012] Step 5: Train and test using public datasets and output mirror image segmentation results.

[0013] Further, the step 1 mirror image preprocessing, the specific steps are as follows:

[0014] 1-1. Data preparation and preprocessing, that is, using the Canny algorithm to detect the edge of the mirror in each image, the Canny algorithm is realized by using the findContours function in the openCV library, after reading the label image, it is transmitted into findContours to obtain the list of mirror edge pixel point position information contour_list;

[0015] 1-2. Making a mirror edge mask image, the specific operation is: passing the contour_list of the last step into the drawContours method in the openCV library, and drawContours draws the edge of the label image according to the position information of the edge pixel points.

[0016] Further, the step 2 uses multiple loss to train the student network, and the specific steps are as follows:

[0017] 2-1. Constructing a multiple loss training student network. The student network is a MobileNetV3-small network, which inherits the depth separable convolution module of V1 and the linear bottleneck residual module of V2, and introduces the SE channel attention structure.

[0018] The first 14 feature layers of the original MobileNetV3-small model are retained, the last two layers of the original MobileNetV3-small model are deleted, and suitable image segmentation modules are added at the first layer, the fourth layer, the ninth layer and the twelfth layer; Introducing the CCFE module proposed by MirrorNet, then using inverse convolution ConvTranspose, BatchNorm layer and Relu activation function, finally connecting a Conv with an output channel number of 1, and using Sigmoid activation function to output the segmentation result;

[0019] The student network includes 14 feature layers, and the input feature map shape of each feature layer from the first to the 14th layer is represented by height x width x channel in sequence as follows: 224 x 224 x 3, 112 x 112 x 16, 56 x 56 x 16, 28 x 28 x 24, 28 x 28 x 24, 14 x 14 x 40, 14 x 14 x 40, 14 x 14 x 40, 14 x 14 x 48, 14 x 14 x 48, 7 x 7 x 96, 7 x 7 x 96, 7 x 7 x 96, 7 x 7 x 576;

[0020] The block structure of each layer and the convolution kernel size of the internal Conv from the first to the 14th layer are (conv2d, 3 x 3), (bneck, 3 x 3), (bneck, 3 x 3), (bneck, 3 x 3), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (conv2d, 1 x 1), (pool, 7 x 7), respectively.

[0021] The expansion parameters from the 2nd layer to the 12th layer are 16, 72, 88, 96, 240, 240, 120, 144, 288, 576, 576, respectively.

[0022] The channel number of the block structure output from the 1st layer to the 13th layer is 16, 16, 24, 24, 40, 40, 40, 48, 48, 96, 96, 96, 576, respectively.

[0023] Attention mechanism is introduced in other layers except the 1st, 3rd, 4th and 14th layers.

[0024] The step length used by the second Conv of each layer block structure from the first to the 14th layer is 2, 2, 2, 1, 2, 1, 1, 1, 1, 2, 1, 1, 1, 1, respectively.

[0025] The activation function of the 2nd, 3rd and 4th layers is RELU, and the 14th layer has no activation function, and the activation function of other layers is h-swish.

[0026] The last two layers of the original author's model are deleted for consideration of parameters and efficiency, and corresponding segmentation modules are added to align the output size of the two models to facilitate step 3 knowledge distillation. The student network is jointly optimized by multiple losses, which can be mathematically represented as:

[0027]

[0028] where S(x) represents the feature map output by the student network, and y represents the mask of the mirror image. Alpha is a self-defined weight, and as preferred, the value of alpha is 0.3.

[0029] 2-2. Calculate the edge loss function. The mirror image has a significant feature, that is, the range of scenes inside the mirror is determined by the size of the mirror, and the inside and outside of the mirror are discontinuous, so the key to solving the problem lies in detecting the edge of the mirror. Since the second derivative can be used to detect edges, we use the Laplace operator to obtain the significant object boundary, and then use the cross-entropy loss to supervise the generation of the mirror boundary. It can be expressed in mathematical form as follows:

[0030]

[0031] where y i is a binary label 0 or 1, p i is the probability of outputting the y label, and is calculated by the second derivative operator, and the specific calculation process is as follows:

[0032] p i = abs(tanh(conv(f, K laplace )))

[0033] where f is the feature map output by the student model, and the Laplace operator is called as the convolution kernel to calculate the significant object boundary, and then the arctangent function is used and the absolute value is taken to map the final output value to the interval [0, 1], which is convenient for the calculation of binary cross-entropy.

[0034] 2-3. Calculate the fusion loss function. The fusion loss function is the segmentation task loss of the original student model. After inputting the mirror image to be segmented into the student network S and obtaining the segmentation result, the loss with the label value is calculated. For the mirror segmentation task of the student model, a certain training strategy is adopted, that is, the binary cross-entropy loss is used for training within the first 84 epochs, and the Lovász-hinge loss is changed after 84 epochs. The hinge loss is an IoU-based loss, which can improve the overall quality of the result in the segmentation task, and fine-tune the parameters of the student network. Its training method can be expressed as the following formula:

[0035]

[0036] Further, step 3 constructs a knowledge distillation framework to calculate the distillation loss, which is as follows:

[0037] 3-1. Select MirrorNet network as the teacher network, input a mirror image I to be segmented into the student neural network S, and input the trained teacher neural network T at the same time. The segmentation effect of the teacher network T is better than that of the student network S, and the segmentation results of the two are obtained, feature maps SF and feature maps TF.

[0038] 3-2. Calculate the distillation loss. Calculate the binary relative entropy of the feature maps SF of the student network and the feature maps TF of the teacher network obtained in 3-1, and the distillation loss is derived from the KL divergence, which is used to measure the distance between two distributions.

[0039] Using relative entropy can make the generation of SF gradually approach the distribution of TF, so as to achieve the goal of knowledge transfer from the teacher network.

[0040] The binary relative entropy loss can be mathematically represented as:

[0041]

[0042] Where t i represents the predicted value of the teacher network, s i represents the predicted value of the student network, and N represents the number of predicted values.

[0043] Further, step 4 constructs a mirror segmentation adversarial learning framework to calculate the adversarial loss, and the specific steps are as follows:

[0044] 4-1. Construct a mirror segmentation adversarial learning framework. Using the principle of adversarial learning, the optimization process of the generated adversarial model module is a process of mutual competition and mutual confrontation. The performance of the generator and the discriminator is continuously improved in the iteration process until the generated data of the final model is consistent with the performance of the real data. At this time, the generator and the discriminator cannot be further optimized. This idea is transformed and applied in the field of mirror image segmentation. The student model acts as the role of the generator G, and the feature map f output by the teacher model can be regarded as real data. A discriminator is constructed to limit the attention area information of the student model. The discriminator is composed of three convolutional layers and two self-attention mechanism modules. Each convolutional layer is composed of a Conv and a LeakyRelu activation function, and is directly connected. Each self-attention mechanism module is composed of three independent Conv and a Softmax activation function. The key-value pair attention mechanism is used. The feature map result is calculated through different Conv, and the weighted average of the input feature map f is calculated through the Softmax activation. The key-value pair attention module represents the input information x as a set of key-value pairs, i.e. x = {(k1, v1), (k2, v2),..., (k N , v N}. Wherein k represents the key, used to calculate the attention distribution, v represents the value, used to calculate the attention distribution weighting information. The specific calculation process can be divided into three steps:

[0045] First, use two identical 1x1Conv, whose output channel is one-eighth of the input channel, to calculate the query and key of the input information x respectively, and then perform matrix multiplication operation to obtain the energy matrix E of the key;

[0046] Second, normalize the energy matrix E using the Softmax activation function;

[0047] Third, use a 1x1Conv to calculate the value v of the input information x, and perform matrix multiplication operation between the energy matrix E obtained in the second step and v;

[0048] 4-2. Calculate the adversarial loss. The segmentation results of the teacher model and the student model are sent into the discriminator constructed in 4-1, and after obtaining the attention distribution aggregation information of the two, the adversarial loss function is used for calculation, so that the student model can strengthen the attention to the mirror like the teacher model. Its mathematical expression is:

[0049]

[0050] Wherein, N represents the Tensor size of the discriminator output, represents the attention value of the student model, represents the attention value of the teacher model.

[0051] Further, the step 5 trains the model and tests the data, specifically as follows:

[0052] 5-1. Prepare the data set (such as public data set MSD) and pre-process according to the description in step 1.

[0053] 5-2. Randomly divide the pre-processed data in 5-1 into training set and test set, obtain 3063 images for training and 955 images for testing, input the training data into the student network for training and use the test data for model test;

[0054] 5-3. Use the model trained and tested in step 5-2 to input picture data into the model, and output the mirror image segmentation result.

[0055] Further, after the step 5-3, there are the following steps:

[0056] The mirror image segmentation result output by step 5-3 is compared with the following existing mirror image segmentation methods: MaskRCNN, PFA, respectively, and one or more of the following indicators of the mirror image segmentation result output by step 5-3 and the existing mirror image segmentation method are calculated: segmentation accuracy, time efficiency of the model, reliability and practicality of the analysis result.

[0057] The present application has the beneficial effects of:

[0058] A mirror image segmentation algorithm based on knowledge distillation and adversarial learning is proposed herein, a teacher-student network model is constructed using the knowledge distillation theory, and the segmentation accuracy of the compact network is improved based on the characteristics of adversarial learning without increasing the network parameters and complexity. The experimental results obviously confirm the efficiency and practicality of the proposed method. In summary, the proposed method can not only reduce the threshold of industrial use of artificial intelligence, but also improve the visual perception of intelligent devices to environmental changes. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is the step flowchart of the present application;

[0060] Figure 2 is the overall architecture flowchart of the present application;

[0061] Figure 3 is the MobileNetV3-small network structure diagram DETAILED DESCRIPTION

[0062] The present application will be further described below in conjunction with the drawings.

[0063] The mirror image segmentation algorithm based on knowledge distillation and adversarial learning, the specific steps are referred to Figure 1 The overall architecture flowchart is shown in Figure 2 The specific steps are as follows:

[0064] Step 1: mirror image preprocessing;

[0065] Step 2: train the student network with multiple loss functions;

[0066] Step 3: establish a knowledge distillation framework and calculate the distillation loss;

[0067] Step 4: establish an adversarial learning framework and calculate the adversarial loss;

[0068] Step 5: train and test using public datasets and output the final result.

[0069] Step 1: mirror image preprocessing, the specific steps are as follows:

[0070] 1-1. Data preparation and preprocessing, that is, using the Canny algorithm to detect the edge of the mirror in each image, the Canny algorithm is implemented using the findContours function in the openCV library, after reading the label image, it is passed into findContours to get the list of mirror edge pixel position information contour_list.

[0071] 1-2. Fill in the position of the obtained edge pixel points and thicken them to make a mask image of the mirror edge. The specific operation is: pass the contour_list of the last step into the drawContours method in the openCV library as a parameter, and drawContours method draws the edge of the label image according to the position information of the edge pixel points.

[0072] Step 2 uses multi-loss to train the student network, and the specific steps are as follows:

[0073] 2-1. Construct the student network. The student network is a MobileNetV3-small network, which directly performs the segmentation task, and it is optimized by two joint losses during training, which can be mathematically expressed as:

[0074]

[0075] Where S(x) represents the feature map output by the student network, y represents the mask of the mirror image. Alpha is a custom weight, and the value of alpha is set to 0.3. is the edge loss function, is the fusion loss function. During training, a certain training strategy is adopted, that is, within the first 84 epochs, is expressed as binary cross-entropy loss, and then changed to Lovász-hinge loss.

[0076] 2-2. Edge loss function.

[0077] Using the Laplace operator, the significant object boundary is obtained based on the second-order derivative, and then the cross-entropy loss is used to supervise the generation of the mirror boundary. It can be mathematically expressed as:

[0078]

[0079] Where, y i is a binary label 0 or 1, p i is the probability of output y label, which is calculated by the second-order differential operator, and its specific calculation process is as follows:

[0080] p i = abs(tanh(conv(f, K laplace )))

[0081] where f is the feature map output by the student model, the Laplace operator is called as the convolution kernel to calculate the significant object boundary, then the final output value is mapped to the interval [0, 1] by using the arctangent function and taking the absolute value, which is convenient for the calculation of binary cross entropy.

[0082] 2-3. Fusion loss function. This loss function is the segmentation task loss of the original student network. After inputting the mirror image to be segmented into the student model S and obtaining the segmentation result, the loss with the label value is calculated. For the mirror segmentation task of the student model, a certain training strategy is adopted, that is, the binary cross entropy loss is used for training within the first 84 epochs, and the Lovász-hinge loss is used for training after 84 epochs. The parameters of the student network are fine-tuned through the Lovász-hinge loss. Its training method can be expressed as the following formula:

[0083]

[0084] Step 3: Build a knowledge distillation framework to calculate the distillation loss, as follows:

[0085] 3-1. Select a pre-trained MirrorNet network as the teacher network, and the input data of the teacher network is consistent with that of the student network. Obtain the segmentation results of the teacher network and the student network and calculate the distillation error loss. The distillation error loss is binary relative entropy, and the teacher network does not participate in training and the parameters are fixed.

[0086] Build a multi-loss student network: the student network is an improved MobileNetV3-small network, and the specific network structure is as shown in Figure 3

[0087] The first column Input represents the input feature map shape of each feature layer of mobilenetV3, which is represented by height x width x channel.

[0088] ​The second column Operator represents the block structure and the kernel size k of the internal Conv each feature layer will go through. We can see that in MobileNetV3, the feature extraction goes through many bneck structures. A bneck is composed of multiple convolution layers and two optional SE structures and residual structures. The specific structure is as follows: first, a Conv with a kernel size of 1x1 is used to convert the input channels to the expansion channel exp_size, then a BatchNorm layer is connected to perform normalization operation, then an h-swish activation function is used, then a Conv with a kernel size of k and a step size of s is used, the input and output channels of which are both the expansion channel exp_size, then a BatchNorm layer, an SE module and an h-swish activation function are used. Finally, a Conv with a kernel size of 1x1 is connected to convert the expansion channel exp_size to the required output channel out, and a BatchNorm layer is connected for normalization.

[0089] The SE structure is based on attention mechanism, which aims to learn the feature weight of each channel. First, a global average pooling layer GAP is used to compress the features to obtain a relatively rough global feature value, then a fully connected layer Linear is used to calculate the feature weight of each channel, and finally the input and the weight are multiplied by matrix point multiplication operation.

[0090] The residual structure performs matrix addition operation on the input of the first column and the output of the bneck module.

[0091] The third column represents the expansion factor, i.e. the expansion parameter, which is multiplied by the number of channels after the input channel is increased in the bneck internal structure.

[0092] The fourth column represents the number of output channels of the block structure respectively.

[0093] The fifth column SE represents whether to introduce attention mechanism in this layer.

[0094] The sixth column NL represents the type of activation function, HS represents h-swish, and RE represents RELU.

[0095] The seventh column represents the step size used by the second Conv of each block structure.

[0096] The last two layers of the original author's model are deleted, i.e. Figure 3The last two layers in the network are deleted, and the first layer, the fourth layer, the ninth layer and the twelfth layer are added with modules suitable for image segmentation. The CCFE module proposed by MirrorNet is introduced, followed by an inverse convolution ConvTranspose, a BatchNorm layer and a Relu activation function, and finally a Conv with an output channel number of 1 is connected, and a Sigmoid activation function is used to output the segmentation result.

[0097] Step 4 builds a mirror segmentation adversarial learning framework to calculate the adversarial loss, and the specific steps are as follows:

[0098] 4-1. Build a mirror segmentation adversarial learning framework: the student model is a generator G, and the feature map f output by the teacher model is regarded as real data, and a discriminator is constructed to limit the attention area information of the student model;

[0099] The discriminator consists of three convolutional layers and two self-attention mechanism modules; wherein each convolutional layer consists of a Conv and a LeakyRelu activation function, and is directly connected; each self-attention mechanism module consists of three independent Convs and a Softmax activation function, uses a key-value pair attention mechanism, respectively calculates the attention distribution of the input information through different Convs, and after the Softmax activation, the input feature map f is weighted and averaged;

[0100] The key-value pair attention mechanism is to express the input information x as a set of key-value pairs, i.e. x={(k1,v1),(k1,v2),...,(k N ,v N )};

[0101] Wherein k represents the key, which is used to calculate the attention distribution, and v represents the value, which is used to calculate the attention distribution weighted information; the calculation process of the discriminator can be divided into three steps:

[0102] First, use two identical 1x1 Convs, whose output channels are one-eighth of the input channels, to calculate the query and key of the input information x respectively, and perform matrix multiplication operation on the two to obtain the energy matrix E of the key.

[0103] Second, normalize the energy matrix E using the Softmax activation function.

[0104] Third, use a 1x1 Conv to calculate the value v of the input information x, and perform matrix multiplication operation on the energy matrix E obtained in the second step and v.

[0105] 4-2. Calculate the adversarial loss: send the segmentation results of the teacher model and the student model into the discriminator constructed in 4-1 respectively, get the attention distribution aggregation information of the two, and then use the adversarial loss function to calculate, so that the student model can strengthen the attention to the mirror like the teacher model; the mathematical expression of the adversarial loss function is:

[0106]

[0107] wherein N represents the Tensor size of the discriminator output, represents the attention value of the student model, represents the attention value of the teacher model.

[0108] Step 5: Train the model and test the data, as follows:

[0109] 5-1. Prepare the data set and pre-process according to step 1.

[0110] 5-2. Randomly divide the pre-processed data in 5-1 into training set and test set, get 3063 images for training and 955 images for testing, input the training data into the student network for training and use the test data for model test.

[0111] 5-3. Use the model trained and tested in step 5-2 to input picture data into the model and output mirror image segmentation results. In order to verify the efficiency of the proposed method, it is compared with the current excellent method (such as Mask RCNN, PFA and other methods), the segmentation accuracy and model efficiency are calculated, and the reliability and practicability of the analysis results are analyzed. Five commonly used indicators in related fields are used as reference for segmentation result accuracy, which are intersection over union (IoU), pixel accuracy (PA), maximum F-measure (Fβ), mean absolute error (MAE) and balanced error (BER). The first two indicators are the focus, and the last three indicators are auxiliary reference. The direction of the arrow represents the direction of the good and bad indicators. The efficiency of the model is measured by the number of floating point operations (FLOPs) and the number of model parameters (Parameters).

[0112] The experimental results are as follows:

[0113] 1. The accuracy of the method and FCN-8s, Mask RCNN, PFA, GDDNet, MirrorNet method is detected respectively, and the comparison is made from different reference standards. See Table 1 for detailed data results.

[0114] Table 1 Comparison of segmentation accuracy of the method and other methods

[0115] PA↑ IoU↑ MAE↓ BER↓ Fβ↑ FCN-8s 0.850 44.94 0.150 25.09 0.603 Mask RCNN 0.820 63.10 0.095 14.38 0.756 PFA 0.869 58.14 0.133 17.04 0.670 GDDNet 0.928 75.39 0.072 8.373 0.825 Teacher 0.933 75.05 0.067 9.807 0.849 The method 0.902 68.53 0.098 10.01 0.752

[0116] 2. The performance of the method is tested respectively with resnext_101_32x4d, MirrorNet and MobileNetV3-small. The specific data results are shown in Table 2.

[0117] Table 2 Comparison of efficiency of the method and other methods

[0118] Parameters(M)↓ FLOPs(G)↓ resnext_101_32x4d 51.57 24.90 MirrorNet 121.77 77.66 mobilenetv3_small 1.67 2.99 The method 1.12 1.98

Claims

1. A mirror image segmentation method based on knowledge distillation and adversarial learning, characterized in that, The method comprises the following steps: Step 1: pre-processing the mirror image; Step 2: training the student network using multi-loss, so that the student network learns on the mirror image obtained after pre-processing in step 1 to obtain a mirror feature map reflecting the features of the mirror image; the structure of the student network is as follows: the first 14 feature layers of the original MobileNetV3-small model are retained, the last two feature layers of the original MobileNetV3-small model are deleted, and a module suitable for image segmentation is added to the first layer, the fourth layer, the ninth layer and the twelfth layer; the CCFE module proposed by MirrorNet is introduced, then an inverse convolution ConvTranspose, a BatchNorm layer and a Relu activation function are used, and finally a Conv with an output channel number of 1 is connected, and a Sigmoid activation function is used to output the segmentation result; Step 3: outputting the mirror feature map using the teacher network and comparing it with the mirror feature map obtained in step 2 to calculate the distillation loss; Step 4: constructing a mirror segmentation adversarial learning framework, taking the mirror feature maps output by the student network and the teacher network as inputs to calculate the adversarial loss; Step 5: training and testing using a public dataset to output the mirror image segmentation result; The specific steps of step 3 for calculating the distillation loss are as follows: 3-1. Selecting the MirrorNet network as the teacher network, inputting a mirror image I to be segmented into the student neural network S and the trained teacher neural network T respectively to obtain the segmentation results, feature maps SF and feature maps TF; 3-2. Calculate the distillation loss: calculate the binary relative entropy of the feature maps SF of the student network and the feature maps TF of the teacher network obtained in 3-1, and the distillation loss is derived from the KL divergence, which is used to measure the distance between two distributions; the binary relative entropy loss can be mathematically expressed as: where t i denotes the prediction value of the teacher network, s i denotes the prediction value of the student network, and N denotes the number of prediction values.

2. The mirror image segmentation method based on knowledge distillation and adversarial learning according to claim 1, wherein, The specific steps of step 1 for pre-processing the mirror image are as follows: 1-1. Detecting the edges of the mirror in each image using the Canny algorithm: the Canny algorithm is implemented using the findContours function in the openCV library, and the Canny algorithm is used to read the label image and input it into findContours to obtain a list of contour_list containing the position information of the mirror edge pixel points; 1-2. Passing the contour_list in the previous step as a parameter into the drawContours method in the openCV library, and the drawContours method draws the edges of the label image according to the position information of the edge pixel points.

3. The mirror image segmentation method based on knowledge distillation and adversarial learning according to claim 2, wherein, The specific steps of step 2 are as follows: 2-1. Constructing a multi-loss student network: The student network comprises 14 feature layers, and the input feature map shape of each feature layer from the first layer to the 14th layer is sequentially represented by height x width x channel as follows: 224 x 224 x 3, 112 x 112 x 16, 56 x 56 x 16, 28 x 28 x 24, 28 x 28 x 24, 14 x 14 x 40, 14 x 14 x 40, 14 x 14 x 40, 14 x 14 x 48, 14 x 14 x 48, 7 x 7 x 96, 7 x 7 x 96, 7 x 7 x 96, 7 x 7 x 576; The block structure of each layer and the convolution kernel size of the internal Conv are (conv2d, 3 x 3), (bneck, 3 x 3), (bneck, 3 x 3), (bneck, 3 x 3), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (bneck, 5 x 5), (conv2d, 1 x 1), (pool, 7 x 7) from the first layer to the 14th layer; The expansion parameters from the 2nd layer to the 12th layer are sequentially 16, 72, 88, 96, 240, 240, 120, 144, 288, 576, 576; The channel number of the block structure output from the first layer to the 13th layer is sequentially 16, 16, 24, 24, 40, 40, 40, 48, 48, 96, 96, 96, 576; Attention mechanism is introduced in other layers except the 1st, 3rd, 4th and 14th layers; The step size used by the second Conv of each layer block structure is 2, 2, 2, 1, 2, 1, 1, 1, 1, 2, 1, 1, 1, 1 from the first layer to the 14th layer The activation function of the 2nd, 3rd and 4th layers is RELU, and the 14th layer has no activation function, and the activation function of other layers is h-swish The output feature maps of the student network and the teacher network are knowledge distilled in the manner of step 3; the student network is optimized by multiple losses, which can be mathematically represented as: Where S(x) represents the feature map output by the student network, y represents the mask of the mirror image, and a is a self-defined weight; 2-2. Calculate the edge loss function: Use the Laplace operator to obtain the significant object boundary based on the second derivative, Then use the cross-entropy loss to supervise the generation of the mirror boundary; which can be mathematically represented as: where y i is a binary label 0 or 1, p i is the probability of outputting the label y, is calculated by a second-order differential operator, and the specific calculation process is as follows: p i = abs(tanh(conv(f, K laplace ))) Where f is the feature map output by the student model, the Laplace operator is used to calculate the significant object boundary, then the arctangent function is used and the absolute value is taken, and the final output value is mapped to the [0, 1] interval, which is convenient for the calculation of binary cross-entropy. 2-3. Calculate the fusion loss function; the fusion loss function is the segmentation task loss of the original student network, input the specular image to be segmented into the student network S, get the segmentation result, and calculate the loss with the label value; for the specular segmentation task of the student network, a certain training strategy is adopted, that is, binary cross entropy loss is used for training within the first 84 epochs, and Lovász-hinge loss is used for training after 84 epochs; The parameters of the student network are fine-tuned by Lovász-hinge loss; its training method can be expressed as the following formula:

4. The mirror image segmentation method based on knowledge distillation and adversarial learning according to claim 3, wherein, The specific steps of step 4 for calculating the adversarial loss are as follows: 4-1. Construct a specular segmentation adversarial learning framework: the student model is a generator G, and the feature map f output by the teacher model is regarded as real data, and a discriminator is constructed to limit the attention area information of the student model; The discriminator is composed of three convolutional layers and two self-attention mechanism modules; wherein each convolutional layer is composed of a Conv and a LeakyRelu activation function, and is directly connected; each self-attention mechanism module is composed of three independent Convs and a Softmax activation function, using key-value pair attention mechanism, respectively calculating the attention distribution of the input information through different Convs, and performing weighted average on the input feature map f after Softmax activation; The key-value pair attention mechanism is to represent the input information x as a set of key-value pairs, i.e., x = {(k1, v1), (k2, v2),..., (k N , N )}; Wherein k represents the key used to calculate the attention distribution, and v represents the value used to calculate the attention distribution weighted information; the calculation process of the discriminator can be divided into three steps: First, use two identical 1x1 Convs, whose output channels are one-eighth of the input channels, to calculate the query and key of the input information x respectively, and perform matrix multiplication operation to obtain the energy matrix E of the key; Second, normalize the energy matrix E using the Softmax activation function; Third, use a 1x1 Conv to calculate the value v of the input information x, and perform matrix multiplication operation on the energy matrix E obtained in the second step and v; 4-2. Calculate the adversarial loss: input the segmentation results of the teacher model and the student model into the discriminator constructed in 4-1, get the attention distribution aggregation information of the two, and then calculate the adversarial loss function, so that the student model can strengthen the attention to the mirror like the teacher model; the mathematical expression of the adversarial loss function is: Wherein, N represents the Tensor size of the discriminator output, represents the attention value of the student model, represents the attention value of the teacher model.

5. The mirror image segmentation method based on knowledge distillation and adversarial learning according to claim 4, wherein, The specific steps of step 5 are as follows: 5-1. Prepare the data set and pre-process according to step 1; 5-2. Randomly divide the pre-processed data in 5-1 into training set and test set to obtain 3063 images for training and 955 images for testing, input the training data into the student network for training and use the test data for model testing; 5-3. Use the model trained and tested in step 5-2 to input picture data into the model and output the specular image segmentation result.

6. The mirror image segmentation method based on knowledge distillation and adversarial learning according to claim 5, wherein, After step 5-3, the following steps are also included: The mirror surface image segmentation result output by step 5-3 is compared with the following existing mirror surface image segmentation methods: Mask RCNN, PFA respectively, and one or more of the following indicators of the mirror surface image segmentation result output by step 5-3 and the existing mirror surface image segmentation methods are calculated: segmentation accuracy, time efficiency of the model, reliability and practicability of the analysis result.

Images