A Semantic Segmentation Method for Continuous Images Based on Multi-level Knowledge Distillation

Abstract

The present invention provides a continuous image semantic segmentation method based on multi-level knowledge distillation, comprising the following steps: S1: selecting a teacher model and a student model, and segmenting the continuous image respectively; S2: extracting the intermediate feature values ​​of the teacher model and the student model as low-level knowledge, and designing an improved intermediate feature distillation; S3: taking the multi-scale fusion feature map as the middle-level knowledge, and designing the spatial structure knowledge distillation based on this; S4: for continuous images, taking the model prediction value as the high-level knowledge, and encoding the dependency between adjacent frames to design the inter-frame dependency distillation item. The continuous image semantic segmentation method based on multi-level knowledge distillation provided by the present invention, the semantic segmentation model trained based on this method has achieved a better balance in model accuracy and operation efficiency compared with the mainstream semantic segmentation model, and has a good application prospect in the field of embedded artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of image semantic segmentation, and in particular to a continuous image semantic segmentation method based on multi-level knowledge distillation. Background Art

[0002] As a core problem of computer vision, scene understanding is becoming increasingly important. In reality, more and more application scenarios require inferring relevant knowledge or semantics from images, including autonomous driving, human-computer interaction, augmented reality, etc. Semantic segmentation, as one of the key problems of computer vision, is a high-level task that paves the way for scene understanding in a macro sense.

[0003] The popularity of deep learning has brought about earth-shaking changes in the field of semantic segmentation, and convolutional neural networks have gradually replaced traditional machine learning methods. Since the introduction of the full convolutional network and encoder-decoder structure, the accuracy of semantic segmentation models based on deep neural networks has been continuously improved. At the same time, the number of parameters within the semantic segmentation model has exploded, and the model reasoning speed has become slower. Therefore, for practical applications, even if the accuracy of complex semantic segmentation models is high, the model occupies too much memory and takes a long time to reason, which cannot meet the real-time requirements of actual scenarios and is not conducive to actual deployment. For this reason, research on lightweight models has become a hot topic in the field of semantic segmentation.

[0004] Research on lightweight models can be roughly divided into the following directions: Generally, complex models have a modular design, and the depth and width of the models are large. Therefore, some research works use more efficient and concise model designs to reduce the size of the models. Model pruning is widely used in lightweight model design. For complex networks that have been trained, model pruning uses a specific evaluation criterion to filter out redundant parameters within the network. The kernel sparsification-based method is to induce the update of network weights during training to make the model parameters more sparse. However, due to the low computational efficiency of sparse matrices on current hardware platforms, the model acceleration effect is not obvious. Although the above methods can effectively reduce the size of the model, the accuracy loss is also serious.

[0005] Since the concept of knowledge distillation was proposed, knowledge distillation has been widely used in model compression research. The label prediction value output by the teacher model is used as the "soft target" for the student model to learn. By aligning the output distribution of the student model and the teacher model, the knowledge is transferred to the student model. The semantic segmentation model trained by the knowledge distillation method can greatly reduce the number of parameters of the semantic segmentation model and improve the model's operation speed without sacrificing too much model accuracy. However, only distilling the prediction results is not conducive to lightweight student models. Compared with the final label prediction value, the intermediate feature layer of the teacher model contains richer information. By imitating the process of extracting features by the teacher model, the student model can better learn the feature distribution of image data. Secondly, for dense prediction tasks such as semantic segmentation, it is crucial to consider the spatial structure knowledge of the image, which will be related to the accuracy of the segmented object boundary. In addition, the input of the model in the actual application scenario is a continuous image sequence, and in the method of using knowledge distillation to train the semantic segmentation model, the relationship between continuous frames is rarely paid attention to.

[0006] Therefore, it is necessary to provide a continuous image semantic segmentation method based on multi-level knowledge distillation to solve the above technical problems. Summary of the invention

[0007] The present invention provides a continuous image semantic segmentation method based on multi-level knowledge distillation, which solves the problem of slow running speed of the semantic segmentation model.

[0008] In order to solve the above technical problems, the present invention provides a continuous image semantic segmentation method based on multi-level knowledge distillation, comprising the following steps:

[0009] S1: Select the teacher model and the student model to segment the continuous images respectively;

[0010] S2: Extract the intermediate feature values ​​of the teacher model and the student model as low-level knowledge and design an improved intermediate feature distillation;

[0011] The intermediate feature distillation specifically includes the following steps:

[0012] S21: The features extracted by the teacher model and the student model in the i-th convolutional layer are marked as and

[0013] S22: retain all positive values ​​in the channel and pass them to the student network. For the negative values ​​in each channel, first calculate their average value a, and then assign a to all neurons in the channel whose values ​​are less than the average value.

[0014] S23: Select Then add a 1×1 convolution layer to ensure that the feature dimension after conversion is the same as that of the teacher model;

[0015] S24: Define T i With S i is the result of the conversion of the teacher model and the student model at the i-th feature layer, and the intermediate feature distillation loss is defined as As shown, K is the number of intermediate feature layers.

[0016] S3: Take the multi-scale fusion feature map as the middle-level knowledge and design the spatial structure knowledge distillation based on it;

[0017] S4: For continuous images, the model prediction value is used as high-level knowledge to encode the dependency between adjacent frames and design the inter-frame dependency distillation term;

[0018] S5: To avoid the change of semantic labels caused by the movement of pixels in the previous and next frames, a semantic consistency loss is designed to ensure the invariance of pixel semantic labels;

[0019] S6: Combine the traditional knowledge distillation method with the loss term proposed in the above steps to train the student model.

[0020] Preferably, the spatial structure knowledge distillation in S3 specifically includes the following steps:

[0021] S31: Encode the structural knowledge of the image into an affinity graph, and achieve distillation of structural knowledge by aligning the affinity graphs generated by the teacher model and the student model;

[0022] S32: The final feature maps of the teacher model and the student model are set to F T 、F S , with a dimension of W×H×C. After the average pooling operation with a kernel size of 16, the feature map is represented as F' T and F' S , at this time the dimension becomes W'×H'×C;

[0023] S33: Use cosine similarity to represent the relationship between two pixels, and calculate the similarity between all pixels to represent the spatial structure knowledge of the image, with F' T For example, for F' T The pixel point f at the i-th and j-th position in the feature map i 、f j , the cosine similarity between the two is expressed as:

[0024]

[0025] Where m ij represents a pixel in the affinity graph M, the dimension of M is N×N×1, N=W'×H', and the affinity graphs calculated by the teacher model and the student model are respectively represented as Mt 、M s , the corresponding pixel point in the affinity map is represented as The spatial structure knowledge distillation loss is defined as follows:

[0026]

[0027] Preferably, the preceding and following frame images I in the inter-frame dependency distillation item are designed in S4. t ,I t+1 , and the output results of the two in the teacher model and the student model are recorded as T t , T t+1 , S t , S t+1 , the inter-frame dependency distillation loss is expressed as follows:

[0028]

[0029] Preferably, in the formula, sim() is used to calculate the similarity between previous and next frames, and M represents the number of images in the sequence.

[0030] Preferably, the designing of semantic consistency loss in S5 to ensure the invariance of pixel semantic labels specifically includes the following steps:

[0031] S51: The segmentation result of the teacher model is used as a "pseudo-label". For an image sequence of length L, the prediction result of the student model at a pixel position (i, j) in the t-th frame and the t+1-th frame image is defined as The corresponding teacher model result is

[0032] S52: Divide the target pixel points into ranges. In the teacher model segmentation result, the number of pixel points that satisfy the consistency of semantic labels of the previous and next frames can be defined as follows:

[0033]

[0034] If the prediction results of the student model on the pixels of the previous and next frames are different from the labels corresponding to the teacher model, the traditional knowledge distillation method is used for correction. Otherwise, the semantic consistency loss is used for correction.

[0035] The number of pixels that satisfy inconsistent semantic labels can be expressed as:

[0036]

[0037] The final semantic consistency loss function is defined as follows:

[0038]

[0039] Preferably, the φ function in the formula is expressed as:

[0040]

[0041] Preferably, the delta function in formula (6) is expressed as:

[0042]

[0043] Preferably, the overall loss function is defined as a formula in S6. In the entire loss function, the cross entropy loss with the true value is the main loss, and the rest are auxiliary losses. A weight coefficient α is required for all auxiliary losses. Considering that the error value of the intermediate feature distillation loss calculation is large, in order to avoid over-correction of this loss during the gradient descent process, a weight coefficient β is required before the feature distillation loss. In the experiment, α and β are set to 0.1 and 10 respectively. -5 .

[0044] L all =L ce +α(L pro +L str +L dep +L sc +βL dis ).

[0045] Where L ce and L pro It is the cross entropy loss and label prediction probability value distillation loss in the traditional knowledge distillation method.

[0046] Preferably, in S6, the trained complex model is used as the teacher network, and the ImageNet pre-trained model is used as the initial weight of the student model. During the training process, the parameters of the student model and the parameters of the 1×1 convolutional layer in the intermediate feature distillation are continuously updated. During the model inference process, only the parameters of the student model are used.

[0047] Preferably, the teacher model and the student model in S1 are both based on the PSPNet network architecture, ResNet101 is selected as the backbone network of the teacher model, and ResNet18 is selected as the backbone network of the student model.

[0048] Compared with the related art, the continuous image semantic segmentation method based on multi-level knowledge distillation provided by the present invention has the following beneficial effects:

[0049] A multi-level knowledge distillation scheme is proposed to distill the knowledge of the teacher model from three levels: high, medium, and low. Feature distillation and spatial structure knowledge distillation are performed based on low-level and medium-level knowledge.

[0050] For continuous image sequences, the present invention additionally considers the dependencies between adjacent frames and uses the high-level knowledge of the model to encode the dependencies into a similarity graph and distill it to the student model;

[0051] In order to avoid the change of semantic labels caused by the movement of pixels in the previous and next frames, the present invention defines semantic consistency loss to effectively ensure the invariance of pixel semantic labels;

[0052] Compared with mainstream semantic segmentation models, the semantic segmentation model trained based on this method achieves a better balance between model accuracy and operating efficiency, and has good application prospects in the field of embedded artificial intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is the overall framework of the continuous image semantic segmentation method based on multi-level knowledge distillation;

[0054] Figure 2 Schematic diagram of image spatial structure knowledge distillation;

[0055] Figure 3 Schematic diagram of improved intermediate feature distillation;

[0056] Figure 4 It is the feature conversion function of the teacher model in the intermediate feature distillation process;

[0057] Figure 5 Segmented image segments on two consecutive sequences for a model trained based on semantic consistency loss and a model without considering semantic consistency loss;

[0058] Figure 6 The segmentation results of different models on the Cityscapes validation set;

[0059] Figure 7 A flowchart of the steps of the continuous image semantic segmentation method based on multi-level knowledge distillation. DETAILED DESCRIPTION

[0060] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0061] Please refer to Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 ,in, Figure 1 It is the overall framework of the continuous image semantic segmentation method based on multi-level knowledge distillation; Figure 2 Schematic diagram of image spatial structure knowledge distillation; Figure 3 Schematic diagram of improved intermediate feature distillation; Figure 4 It is the feature conversion function of the teacher model in the intermediate feature distillation process; Figure 5 Segmented image segments on two consecutive sequences for a model trained based on semantic consistency loss and a model without considering semantic consistency loss; Figure 6 The segmentation results of different models on the Cityscapes validation set; Figure 7 The flowchart of the continuous image semantic segmentation method based on multi-level knowledge distillation is as follows. The continuous image semantic segmentation method based on multi-level knowledge distillation includes the following steps:

[0062] S1: Select the teacher model and the student model to segment the continuous images respectively;

[0063] The teacher model and the student model in S1 are both based on the PSPNet network architecture, which includes a pyramid pooling module that can aggregate contextual information from different regions. ResNet101 with stronger extraction capabilities is selected as the backbone network of the teacher model, and ResNet18 with a simpler structure is selected as the backbone network of the student model. For subsequent knowledge distillation operations, the teacher model and the student model are required to segment the training images respectively.

[0064] S2: Extract the intermediate feature values ​​of the teacher model and the student model as low-level knowledge and design an improved intermediate feature distillation;

[0065] The intermediate feature distillation specifically includes the following steps:

[0066] S21: The features extracted by the teacher model and the student model in the i-th convolutional layer are marked as and

[0067] S22: All positive values ​​in the channel are retained and passed to the student network. For the negative values ​​in each channel, their average value a is first calculated, and then all neurons in the channel with values ​​less than the average value are assigned a. The conversion function is as follows: Figure 4 As shown;

[0068] S23: Select Then add a 1×1 convolution layer to ensure that the feature dimension after conversion is the same as that of the teacher model for distillation;

[0069] S24: Define T i With S i is the result of the conversion between the teacher model and the student model at the i-th feature layer. The intermediate feature distillation loss is defined as shown in formula (1), where K is the number of intermediate feature layers.

[0070]

[0071] The BatchNorm value before the activation function is selected as the research object, and the feature values ​​of the teacher model and the student model are converted respectively. The intermediate feature distillation process is as follows Figure 3 shown.

[0072] In the current research on knowledge distillation, some methods try to use the intermediate features of the model as the object of distillation to align the intermediate feature distribution of the teacher model and the student model. However, in these feature distillation methods, the output results of the intermediate feature layer of the teacher model are directly distilled. Considering that each convolution layer in the feature layer has an activation function, and the neuron will lose a lot of information after the activation function is calculated, this method selects the feature value before the activation function as the object of distillation. In addition, since the output size of the intermediate feature layer of the teacher model and the student model is inconsistent, this method adds an additional convolution layer when extracting the intermediate features of the student model to convert the size of the student model feature value so as to align with the features of the teacher model.

[0073] S3: Take the multi-scale fusion feature map as the middle-level knowledge and design the spatial structure knowledge distillation based on it;

[0074] S4: For continuous images, the model prediction value is used as high-level knowledge to encode the dependency between adjacent frames and design the inter-frame dependency distillation term;

[0075] S5: To avoid the change of semantic labels caused by the movement of pixels in the previous and next frames, a semantic consistency loss is designed to ensure the invariance of pixel semantic labels;

[0076] S6: Combine the traditional knowledge distillation method with the loss term proposed in the above steps to train the student model.

[0077] The spatial structure knowledge distillation in S3 specifically includes the following steps:

[0078] S31: Encode the structural knowledge of the image into an affinity graph, and achieve distillation of structural knowledge by aligning the affinity graphs generated by the teacher model and the student model;

[0079] S32: The final feature maps of the teacher model and the student model are set to F T 、F S , with a dimension of W×H×C. After the average pooling operation with a kernel size of 16, the feature map is represented as F' T and F' S , at this time the dimension becomes W'×H'×C;

[0080] S33: Use cosine similarity to represent the relationship between two pixels, and calculate the similarity between all pixels to represent the spatial structure knowledge of the image, with F' T For example, for F'T The pixel point f at the i-th and j-th position in the feature map i 、f j , the cosine similarity between the two is expressed as:

[0081]

[0082] In formula (2), m ij represents a pixel in the affinity graph M, the dimension of M is N×N×1, N=W'×H', and the affinity graphs calculated by the teacher model and the student model are respectively represented as M t 、M s , the corresponding pixel point in the affinity map is represented as The spatial structure knowledge distillation loss is defined as follows:

[0083]

[0084] The affinity graph representing the spatial structure knowledge in S3 can be considered as a measure of the similarity of each pixel in the image. This idea is extended to represent the dependency relationship between the previous and next frames, and the cosine similarity is continued to be used to measure the relationship between the previous and next frame images.

[0085] In the S4, the previous and next frame images I are designed in the inter-frame dependency distillation item t ,I t+1 , and the output results of the two in the teacher model and the student model are recorded as T t , T t+1 , S t , S t+1 , the inter-frame dependency distillation loss is expressed as follows:

[0086]

[0087] In formula (4), sim() is used to calculate the similarity between the previous and next frames, and Q represents the number of images in the sequence.

[0088] The design of semantic consistency loss in S5 to ensure the invariance of pixel semantic labels specifically includes the following steps:

[0089] S51: The segmentation result of the teacher model is used as a "pseudo-label". For an image sequence of length L, the prediction result of the student model at a pixel position (i, j) in the t-th frame and the t+1-th frame image is defined as The corresponding teacher model result is

[0090] S52: Divide the target pixel points into ranges. In the teacher model segmentation result, the number of pixel points that satisfy the consistency of semantic labels of the previous and next frames can be defined as follows:

[0091]

[0092] If the prediction results of the student model on the pixels of the previous and next frames are different from the labels corresponding to the teacher model, the traditional knowledge distillation method is used for correction. Otherwise, the semantic consistency loss is used for correction.

[0093] The number of pixels that satisfy inconsistent semantic labels can be expressed as:

[0094]

[0095] The final semantic consistency loss function is defined as follows:

[0096]

[0097] The φ function in formula (5) is expressed as:

[0098]

[0099] The delta function in formula (6) is expressed as:

[0100]

[0101] The overall loss function defined in S6 is formula (10). In the entire loss function, the cross entropy loss with the true value is the main loss, and the rest are auxiliary losses. All auxiliary losses need to be matched with a weight coefficient α. Considering that the error value of the intermediate feature distillation loss calculation is large, in order to avoid over-correction of this loss during the gradient descent process, a weight coefficient β needs to be added before the feature distillation loss. In the experiment, α and β are set to 0.1 and 10 respectively. -5 .

[0102] L all =L ce +α(L pro +L str +L dep +L sc +βL dis ) (10).

[0103] Where L ce and L pro It is the cross entropy loss and label prediction probability value distillation loss in the traditional knowledge distillation method.

[0104] In S6, the trained complex model is used as the teacher network, and the ImageNet pre-trained model is used as the initial weight of the student model. During the training process, the parameters of the student model and the parameters of the 1×1 convolutional layer in the intermediate feature distillation are continuously updated. During the model inference process, only the parameters of the student model are used.

[0105] The application examples of the present invention are as follows:

[0106] 1. Dataset Introduction

[0107] Cityscapes is a semantic segmentation dataset for urban road scenes. It is mainly collected from street scenes in 50 different cities. It has 5,000 high-quality pixel-level annotated images, including 2,975 training sets, 500 validation sets, and 1,525 test sets. Each image contains 1024×2048 pixels and a total of 19 categories. In addition, the dataset also has 20,000 roughly annotated images. The present invention uses finely annotated images for training and testing.

[0108] 2. Verification of the effectiveness of each distillation loss

[0109] To demonstrate the effectiveness of each component in the multi-level knowledge distillation method, the present invention conducts ablation experiments on the Cityscapes dataset. The mean intersection over union (mIoU) and mean class accuracy (mAcc) are used as standards for measuring the accuracy of the semantic segmentation model. Table 1 shows the accuracy results of the semantic segmentation model trained based on different distillation loss functions on the Cityscapes validation set. As can be seen from Table 1, each distillation scheme will bring a higher mIoU score, among which the intermediate feature distillation loss has the largest improvement on the model accuracy, reaching 3.33%. After adding the inter-frame dependency distillation loss and semantic consistency loss, the mIoU of the student model increased by 2.17% and 2.19% respectively. This shows the necessity of considering the relationship between consecutive frames.

[0110] Table 1 Accuracy results of models trained with different loss terms on the Cityscapes validation set

[0111]

[0112] Figure 5 The model trained with semantic consistency loss and the model without semantic consistency loss are shown to segment image fragments in two consecutive sequences. Figure 5 In the first column, due to the lack of consideration of semantic consistency loss, the segmentation results of the street lights in the black border are discontinuous, and the segmentation results of the model in the red border area in the first column have inconsistent labels. A similar situation also occurs in the brown border in the third column. With the help of semantic consistency loss, the problem of inconsistent labels between the previous and next frames has been effectively solved. Figure 5 The second and fourth columns of can intuitively illustrate the effectiveness of semantic consistency loss.

[0113] 3. Model lightweight verification

[0114] In order to more directly demonstrate the advantages of the proposed method in terms of model lightweighting, the present invention compares various indicators of the teacher model and the student model based on the Cityscapes validation set, and the results are shown in Table 2. It can be seen that without losing too much accuracy, the number of parameters and floating-point operations of the student model are reduced by 79.7% and 78.74% respectively compared with the teacher model, and the memory capacity occupied by the student model is only 10.7% of the teacher model. After testing, the average inference time of the student model on an image with a resolution of 1024×2048 is only 18.1% of that of the teacher model. From the above analysis, it can be seen that the student model trained based on the multi-level knowledge distillation method has achieved a better balance between accuracy and operating efficiency.

[0115] Table 2 Performance comparison results of teacher model and student model

[0116]

[0117]

[0118] 4. Model superiority verification

[0119] This section compares the proposed method with existing lightweight semantic segmentation methods in terms of model lightweight and model accuracy based on the Cityscapes validation set. The results are shown in Table 3. Although the model trained by the present invention is not as lightweight as some methods, its segmentation accuracy is significantly higher than them.

[0120] In addition, in order to reflect the advantages of the multi-level knowledge distillation method, the present invention is also compared with the existing distillation method, and the comparison results based on the Cityscapes validation set are shown in Table 4. The table not only lists the model parameter quantity and accuracy of these methods, but also gives the percentage of accuracy improvement brought by these distillation methods in the last column. Since the accuracy of the baseline model of the present invention is lower than that of the baseline models of other methods, the accuracy of the final student model is also slightly lower. However, it can be clearly seen that the multi-level knowledge distillation method has the greatest improvement in the accuracy of the student model. On the basis of the original baseline model, the segmentation accuracy is improved by 8.78%, which means that the use of the multi-level knowledge distillation method can allow the student model to learn more comprehensive and richer knowledge.

[0121] Table 3 Comparison results with existing lightweight semantic segmentation methods

[0122]

[0123] Table 4 Comparison results with existing knowledge distillation methods

[0124]

[0125] Figure 6 The segmentation results of different models on the Cityscapes validation set are shown. From the second column from left to right in the picture, it can be seen that ESPNet's segmentation of road scenes is poor, and even large areas of predicted labels are wrong, and the segmentation of the area is not complete. The middle column of the picture is the segmentation result of the Baseline model. It can be seen that there are still some pixel labels that are predicted incorrectly, and compared with the segmentation results of the model of the present invention, the Baseline model's segmentation of object boundaries is not smooth enough, such as the car in the first picture from top to bottom in the third column, and the street on the left in the fourth picture.

[0126] Compared with the related art, the continuous image semantic segmentation method based on multi-level knowledge distillation provided by the present invention has the following beneficial effects:

[0127] A multi-level knowledge distillation scheme is proposed to distill the knowledge of the teacher model from three levels: high, medium, and low. Feature distillation and spatial structure knowledge distillation are performed based on low-level and medium-level knowledge.

[0128] For continuous image sequences, the present invention additionally considers the dependencies between adjacent frames and uses the high-level knowledge of the model to encode the dependencies into a similarity graph and distill it to the student model;

[0129] In order to avoid the change of semantic labels caused by the movement of pixels in the previous and next frames, the present invention defines semantic consistency loss to effectively ensure the invariance of pixel semantic labels;

[0130] Compared with mainstream semantic segmentation models, the semantic segmentation model trained based on this method achieves a better balance between model accuracy and operating efficiency, and has good application prospects in the field of embedded artificial intelligence.

[0131] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A continuous image semantic segmentation method based on multi-level knowledge distillation, characterized in that: The following steps are involved: S1: Select the teacher model and the student model to segment the continuous images respectively; S2: Extract the intermediate feature values ​​of the teacher model and the student model as low-level knowledge and design an improved intermediate feature distillation; The intermediate feature distillation specifically includes the following steps: S21: The features extracted by the teacher model and the student model in the i-th convolutional layer are marked as and S22: retain all positive values ​​in the channel and pass them to the student network. For the negative values ​​in each channel, first calculate their average value a, and then assign a to all neurons in the channel whose values ​​are less than the average value. S23: Select Then add a 1×1 convolution layer to ensure that the feature dimension after conversion is the same as that of the teacher model; S24: Define T i With S i is the result of the conversion of the teacher model and the student model at the i-th feature layer, and the intermediate feature distillation loss is defined as As shown, K is the number of intermediate feature layers; S3: Take the multi-scale fusion feature map as the middle-level knowledge and design the spatial structure knowledge distillation based on it; S4: For continuous images, the model prediction value is used as high-level knowledge to encode the dependency between adjacent frames and design the inter-frame dependency distillation term; S5: To avoid the change of semantic labels caused by the movement of pixels in the previous and next frames, a semantic consistency loss is designed to ensure the invariance of pixel semantic labels; S6: Combine the traditional knowledge distillation method and the loss term proposed in the above steps to train the student model; The design of semantic consistency loss in S5 to ensure the invariance of pixel semantic labels specifically includes the following steps: S51: The segmentation result of the teacher model is used as a "pseudo-label". For an image sequence of length L, the prediction result of the student model at a pixel position (i, j) in the t-th frame and the t+1-th frame image is defined as The corresponding teacher model result is S52: Divide the target pixel points into ranges. In the teacher model segmentation result, the number of pixel points that satisfy the consistency of semantic labels of the previous and next frames can be defined as follows: If the prediction results of the student model on the pixels of the previous and next frames are different from the labels corresponding to the teacher model, the traditional knowledge distillation method is used for correction. Otherwise, the semantic consistency loss is used for correction. The number of pixels that satisfy inconsistent semantic labels can be expressed as: The final semantic consistency loss function is defined as follows:

2. The continuous image semantic segmentation method based on multi-level knowledge distillation according to claim 1, characterized in that: The spatial structure knowledge distillation in S3 specifically includes the following steps: S31: Encode the structural knowledge of the image into an affinity graph, and achieve distillation of structural knowledge by aligning the affinity graphs generated by the teacher model and the student model; S32: The final feature maps of the teacher model and the student model are set to F T 、F S , with a dimension of W×H×C. After the average pooling operation with a kernel size of 16, the feature map is represented as F' T and F' S , at this time the dimension becomes W'×H'×C; S33: Use cosine similarity to represent the relationship between two pixels, and calculate the similarity between all pixels to represent the spatial structure knowledge of the image, with F' T For example, for F' T The pixel point f at the i-th and j-th position in the feature map i 、f j , the cosine similarity between the two is expressed as: Where m ij represents a pixel in the affinity graph M, the dimension of M is N×N×1, N=W'×H', and the affinity graphs calculated by the teacher model and the student model are respectively represented as M t 、M s , the corresponding pixel point in the affinity map is represented as The spatial structure knowledge distillation loss is defined as follows:

3. The continuous image semantic segmentation method based on multi-level knowledge distillation according to claim 1, characterized in that: In the S4, the previous and next frame images I are designed in the inter-frame dependency distillation item t ,I t+1 , and the output results of the two in the teacher model and the student model are recorded as T t , T t+1 , S t , S t+1 , the inter-frame dependency distillation loss is expressed as follows:

4. The method for continuous image semantic segmentation based on multi-level knowledge distillation according to claim 3, characterized in that: In the formula, sim() is used to calculate the similarity between the previous and next frames, and M represents the number of images in the sequence.

5. The method for continuous image semantic segmentation based on multi-level knowledge distillation according to claim 1, characterized in that: formula The φ function is expressed as:

6. The method for continuous image semantic segmentation based on multi-level knowledge distillation according to claim 1, characterized in that: formula The delta function is expressed as:

7. The method for continuous image semantic segmentation based on multi-level knowledge distillation according to claim 1, characterized in that: The overall loss function formula defined in S6 is: L all =L ce +α(L pro +L str +L dep +L sc +βL dis ); In the entire loss function, the cross entropy loss with the true value is the main loss, and the rest are auxiliary losses. All auxiliary losses need to be matched with a weight coefficient α. Considering that the error value of the intermediate feature distillation loss calculation is large, in order to avoid over-correction of this loss during the gradient descent process, a weight coefficient β needs to be added before the feature distillation loss. In the experiment, α and β are set to 0.1 and 10 respectively. -5 ; Where L ce and L pro It is the cross entropy loss and label prediction probability value distillation loss in the traditional knowledge distillation method.

8. The method for continuous image semantic segmentation based on multi-level knowledge distillation according to claim 1, characterized in that: In S6, the trained complex model is used as the teacher network, and the ImageNet pre-trained model is used as the initial weight of the student model. During the training process, the parameters of the student model and the parameters of the 1×1 convolutional layer in the intermediate feature distillation are continuously updated. During the model inference process, only the parameters of the student model are used.

9. The method for continuous image semantic segmentation based on multi-level knowledge distillation according to claim 1, characterized in that: The teacher model and the student model in S1 are both based on the PSPNet network architecture, ResNet101 is selected as the backbone network of the teacher model, and ResNet18 is selected as the backbone network of the student model.

Images