A hierarchical boundary knowledge distillation human parsing method and system

Abstract

The application discloses a hierarchical boundary knowledge distillation human body analysis method and system. The method comprises the following steps: inputting a human body analysis data set and performing pretreatment; constructing a teacher model and a student model with the same framework and different main networks; calculating a student model analysis loss, a student model logit distillation loss, a student model hierarchical boundary distillation loss, and a student model hierarchical global distillation loss; weighting and adding the logit distillation loss, the hierarchical boundary distillation loss, and the hierarchical global distillation loss to obtain a knowledge distillation loss; weighting and adding the knowledge distillation loss and the student model analysis loss to obtain a student model overall training loss; training the student model by using the student model overall training loss; and inputting a human body image into the trained student model to output an analysis prediction image. The application can guarantee high human body analysis precision, optimize the size and complexity of the model, and effectively balance the model precision and efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and semantic segmentation, and particularly relates to a hierarchical boundary knowledge distillation human parsing method and system. BACKGROUND

[0002] Human parsing is a subtask of image semantic segmentation, aiming to make fine-grained annotations on human regions in images, so as to distinguish different body parts and clothing details, and has wide application prospects in virtual fitting, human pose estimation, pedestrian re-identification, etc. With the rapid development of convolutional neural networks (CNN), various advanced high-precision CNNs have been proposed for human parsing tasks, but these high-precision networks are designed to be deep and complex in order to extract rich semantic information, and the models are often large and have high computational cost, which makes it difficult for these high-performance large-scale models to be deployed on mobile terminals, edge devices and other resource-limited platforms and devices, and even more difficult to be widely used in real life.

[0003] In order to solve this problem, lightweight base models such as EfficientNet are used for real-time semantic segmentation. In addition, model compression is also a common way to solve this problem. Four common methods of model compression include network pruning, weight decomposition, weight quantization or knowledge distillation. Among them, knowledge distillation is a method that can transfer the knowledge extracted from a larger teacher model to a smaller student network to improve the performance of the student network, which does not change the original network structure and can be used with other model compression methods at the same time, and has better scalability. The original knowledge distillation method is in image classification tasks, but in recent years it has been extended to multiple tasks including semantic segmentation. In the task of semantic segmentation, the knowledge distillation method can be divided into two categories: spatial correlation distillation and channel correlation distillation. Although the semantic segmentation method based on knowledge distillation has advantages in computational efficiency, it ignores the important influence of human boundaries on the segmentation results, resulting in problems such as non-smooth segmentation boundaries in the generated human parsing results.

[0004] Boundary information as an important auxiliary clue has been widely used to develop high-precision semantic segmentation networks, making the human shape better described. However, most existing methods need to design additional modules or subnetworks to learn boundary clues, thereby improving the accuracy of the human parsing task, but this inevitably leads to a significant increase in model size and computational cost. Therefore, for lightweight human parsing, it is very important to encode human boundaries as additional auxiliary clues in an effective and efficient way.

[0005] One of the current prior arts is an edge-guided diffusion based accurate human parsing method EDGNet in the paper "Toward Accurate Human Parsing Through EdgeGuided Diffusion". This method first introduces a horizontal-vertical aggregation module to aggregate context information from horizontal and vertical directions, thereby promoting the classification of semantic parts. Subsequently, multi-stage edge-guided diffusion is performed to refine the parsing results. Each stage includes an edge-guided diffusion and a channel aggregation module to enhance the semantic consistency within the body parts guided by semantic boundaries. In addition, to introduce additional high-resolution features to supplement the previously predicted details, the backbone features and semantic boundary features are combined with the diffusion prediction, effectively fusing features with low-level details and high-level semantic information through the channel aggregation module. This method uses joint learning of semantic boundary detection and human parsing to facilitate the positioning of boundary areas, thereby improving the accuracy of human parsing. However, the disadvantage of this method is that due to the use of multi-stage boundary diffusion scheme, it requires a long time for training and the model is relatively complex, which cannot be deployed to devices with high real-time requirements.

[0006] The second current prior art is the patent "Human Parsing Method and System Based on Knowledge Distillation (CN117789244A)". This method first trains a large and complex teacher model, and then during the training of a smaller and simpler student model, the boundary features, pose features, and parsing features are transferred from the trained teacher model to the student model being trained as knowledge, allowing the student model to maximize parsing speed while ensuring accuracy. This method combines knowledge distillation with human parsing tasks, and although it can balance the accuracy and speed of the student model to obtain a lightweight human parsing model with good performance, the disadvantage of this method is that during the knowledge distillation process, the boundary features, pose features, and parsing features are all transferred as knowledge. These three features may be mutually restrictive during training, limiting the accuracy of parsing and preventing better parsing results from being obtained.

[0007] The third prior art is a boundary privileged knowledge distillation method for semantic segmentation tasks in the paper "BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation". This method extracts knowledge from the subject and boundary of the teacher model and transfers it to a compact student model. The entire process can be divided into two sections. First, the method designs a boundary knowledge distillation loss that aligns spatial probabilities and aggregates context information to refine boundaries, aiming to distinguish ambiguous classes at the pixel level in the boundary area. Then, the method designs a body knowledge distillation loss that uses shape constraints to selectively focus on internal semantic regions, thereby eliminating high-uncertainty boundary samples and smoothing the learning curve, further improving the segmentation results in the body area. The disadvantage of this method is that since it extracts boundary knowledge from the final feature representation, and high-level features are usually obtained through multiple layers of transformation and dimension reduction, some low-level detail information may be lost, which may be important for boundary detection. Therefore, this method may not accurately transfer boundary knowledge to the model, and the performance of the final student model may not be maximized. SUMMARY

[0008] The purpose of the present application is to overcome the shortcomings of existing methods and propose a hierarchical boundary knowledge distillation human parsing method and system. The main problem solved by the present application is how to better utilize human-related information, maximize the accuracy of human parsing results, improve real-time efficiency, and control the size of the model so that the model can be deployed to devices with high real-time requirements.

[0009] To solve the above problems, the present application proposes a hierarchical boundary knowledge distillation human parsing method, which comprises:

[0010] Input the human parsing dataset, perform enhancement operations on the human parsing dataset, and obtain a preprocessed human parsing dataset;

[0011] Input the preprocessed human parsing dataset, construct a teacher model and a student model with the same framework and different backbone networks, and output the teacher model hierarchical features Student model hierarchical features Teacher model prediction results Student model prediction results Student model boundary features Student model horizontal distribution features And student model vertical class distribution features Where i = 2, 3, 4, 5;

[0012] input a real human parsing result y, find a boundary of a mask between two different semantic parts in the real human parsing result y, and obtain an edge annotation y b ; utilize the real human parsing result y and the edge annotation y b , the student model horizontal distribution feature , and the student model vertical class distribution feature , and obtain a student model parsing loss L parsing ; utilize the teacher model prediction result and the student model prediction result , and calculate a student model logit distillation loss L kd-logit ; input the teacher model hierarchical feature , the student model hierarchical feature , and obtain a teacher model edge map corresponding to each level and a student model edge map corresponding to each level from the hierarchical feature and the student model edge map , and a teacher model class probability map corresponding to each level and a student model class probability map corresponding to each level and the student model class probability map , and calculate a student model hierarchical boundary distillation loss L kd-edge , a student model hierarchical global distillation loss L kd-global ;

[0013] weight and add the student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge , and the student model hierarchical global distillation loss L kd-global , to obtain a student model knowledge distillation loss L KD weight and add the student model parsing loss L parsing and the student model knowledge distillation loss L KD , to obtain a student model overall training loss L total ;

[0014] utilize the student model overall loss function L total , train the student model, and obtain a trained student model;

[0015] input a human image into the trained student model, and output a parsing prediction image.

[0016] Preferably, the enhancement operation is specifically:

[0017] perform mean subtraction, random color jittering, random scale, and random left-right flipping on all images in the human parsing dataset in sequence, wherein the random scale ranges from 0.5 to 1.25 times the original image.

[0018] Preferably, the teacher model and the student model have the same framework and different backbone networks, in particular:

[0019] The overall framework of the teacher network and the student network is a class distribution guided network, which includes a backbone network, a class distribution guided high-resolution module, an edge extraction module and a feature fusion module, wherein the backbone network includes Conv1, Res2, Res3, Res4 and Res5.

[0020] The backbone network of the teacher model is a pre-trained ResNet101, and the backbone network of the student model is a ResNet50.

[0021] Preferably, the student model parses the loss Lparsing, the student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global , in particular:

[0022] The student model parsing loss L parsing is calculated as follows:

[0023] L parsing = λ1L b + λ2L c + L p ,

[0024] wherein λ1, λ2 are hyperparameters for balancing the loss of each part, L b is the weighted cross-entropy loss between the edge map obtained from the student model boundary feature and the edge map generated from the real human parsing result y, L c is the cross-entropy loss between the feature horizontal class distribution map and the feature vertical class distribution map obtained from the student model horizontal distribution feature and the student model vertical class distribution feature and the result horizontal class distribution map Q h and the result vertical class distribution map Q v generated from the real human parsing result y, L p is the cross-entropy loss between the student model prediction result and the real human parsing result y.

[0025] The softmax function is used to obtain the student model horizontal distribution feature and the student model vertical class distribution feature Obtaining a feature horizontal class distribution map and a feature vertical class distribution map The specific calculation formula is as follows:

[0026]

[0027] Wherein, C is the total number of categories of the human parsing dataset, exp(·) is the natural exponential;

[0028] For the result horizontal class distribution map Q h and the result vertical class distribution map Q v The specific calculation formula is as follows:

[0029]

[0030] Wherein, the matrix M is generated by one-hot encoding using the real human parsing result y, H and W are the height and width of the matrix M respectively, h∈[1, H], w∈[1, W], c∈[1, C];

[0031] The student model logit distillation loss L kd-logit The calculation formula is as follows:

[0032]

[0033] Wherein, and respectively represent the probability that the pixel p predicted by the teacher model and the student model belongs to the category i;

[0034] The student model hierarchical boundary distillation loss L kd-edge The calculation formula is as follows:

[0035]

[0036] Wherein, MSE represents the MSE loss function, and the specific calculation formula of the teacher model edge map and the student model edge map corresponding to each level is as follows:

[0037]

[0038] Wherein, G x and G y are two Sobel convolution kernels used to calculate the gradient of the hierarchical feature in the horizontal and vertical directions;

[0039] The calculation formula of the student model hierarchical global distillation loss L kd-global is as follows:

[0040]

[0041] wherein H'(i) and W'(i) represent the height and width of the i-th layer class probability map respectively, KL(·) represents the KL divergence, and respectively represent the hierarchical features of the teacher model and the student model using the softmax function, and generate the corresponding hierarchical class probability map, i.e., the class probability of the x-th pixel of the i-th layer of the student model and the teacher model;

[0042] the specific calculation formula of the hierarchical class probability map of the teacher model is:

[0043]

[0044] the specific calculation formula of the hierarchical class probability map of the student model is:

[0045]

[0046] Preferably, the student model knowledge distillation loss L KD and the student model overall training loss L total , specifically:

[0047] the calculation formula of the student model knowledge distillation loss L KD is:

[0048] L KD = α1L kd-logit + α2L kd-edge + α3L kd-global ,

[0049] wherein α1, α2, and α3 are hyperparameters for balancing the loss of each part;

[0050] the calculation formula of the student model overall training loss L total is:

[0051] L total = β1L parsing + β2L KD ,

[0052] wherein β1 and β2 are hyperparameters for balancing the loss of each part.

[0053] Correspondingly, the application also provides a hierarchical boundary knowledge distillation human parsing system, comprising:

[0054] a preprocessing unit configured to input a human parsing dataset, perform an enhancement operation on the human parsing dataset, and obtain a preprocessed human parsing dataset;​​

[0055] a model construction unit, configured to input the preprocessed human parsing dataset, construct a teacher model and a student model with the same framework and different backbone networks, and output teacher model hierarchical features student model hierarchical features teacher model prediction results student model prediction results student model boundary features student model horizontal distribution features and student model vertical class distribution features wherein i = 2, 3, 4, 5;

[0056] a loss function construction unit, configured to input a real human parsing result y, find a boundary of a mask between two different semantic parts in the real human parsing result y, and obtain edge annotation y b ; calculate, by using the real human parsing result y and the edge annotation y b , the student model horizontal distribution features and the student model vertical class distribution features , to obtain a student model parsing loss Lparsing; calculate, by using the teacher model prediction results and the student model prediction results , a student model logit distillation loss L kd-logit ; input the teacher model hierarchical features the student model hierarchical features respectively obtain, from the hierarchical features, a teacher model edge map corresponding to each level and a student model edge map corresponding to each level and a teacher model class probability map corresponding to each level and a student model class probability map corresponding to each level to obtain a student model hierarchical boundary distillation loss L kd-edge , a student model hierarchical global distillation loss L kd-global ; add up, by weighting, the student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global to obtain a student model knowledge distillation loss L KD add up, by weighting, the student model parsing loss L parsing and the student model knowledge distillation loss L KD to obtain a student model overall training loss L total ;

[0057] a training unit configured to train the student model using the student model overall loss function L total training the student model to obtain a trained student model;

[0058] an application unit configured to input the human body image into the trained student model and output an analysis prediction image.

[0059] The present application has the following beneficial effects:

[0060] The present application makes full use of the characteristics that human body part boundaries can promote the differentiation of adjacent parts by the human body analysis model, maintain the consistency and coherence of segmentation in the global range, and further promote the analysis accuracy, adopts a hierarchical boundary knowledge distillation method, applies the knowledge distillation method to the human body analysis task, and helps to compress the size of the model while maintaining high accuracy. This enables our method to be applied to devices with limited computing power and memory.

[0061] Compared with most existing human body analysis methods that do not use lightweight operations, the method adopted by the present application has the characteristics of fast computing speed and better trade-off between accuracy and efficiency; compared with existing knowledge distillation methods designed based on the human body analysis task, the present application can better utilize multi-scale boundary features and further better segment human semantic boundaries; compared with general knowledge distillation methods designed for general semantic segmentation tasks, the method adopted by the present application can more fully utilize the characteristics of the human body analysis task and is more suitable for the human body analysis task, and can obtain more accurate segmentation results.

[0062] The method can optimize the size and complexity of the model while ensuring high human body analysis accuracy, thereby effectively balancing the accuracy and efficiency of the model. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a flowchart of a hierarchical boundary knowledge distillation human body analysis method according to an embodiment of the present application;

[0064] Figure 2 is a schematic diagram of the overall framework of the teacher model and the student model according to an embodiment of the present application;

[0065] Figure 3 is a structural diagram of a hierarchical boundary knowledge distillation human body analysis system according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] With reference to the accompanying drawings: the technical solutions in the embodiments of the present application will be described clearly and completely, obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor belong to the scope of protection of the present application.

[0067] Figure 1 is a flow chart of a hierarchical boundary knowledge distillation human parsing method according to an embodiment of the present application, as shown in the figure, the method comprises: Figure 1

[0068] S1, input the human parsing dataset, perform enhancement operation on the human parsing dataset, and obtain the preprocessed human parsing dataset;

[0069] S2, input the preprocessed human parsing dataset, construct a teacher model and a student model with the same framework and different backbone networks, output the teacher model hierarchical feature student model hierarchical feature teacher model prediction result student model prediction result student model boundary feature student model horizontal distribution feature and student model vertical class distribution feature wherein i=2, 3, 4, 5;

[0070] S3, input the real human parsing result y, find the boundary of the mask between the two different semantic parts in the real human parsing result y, and obtain the edge annotation y b ; using the real human parsing result y and the edge annotation y b , the student model horizontal distribution feature and the student model vertical class distribution feature , the student model parsing loss L parsing is obtained; using the teacher model prediction result and student model prediction result , the student model logit distillation loss L kd-logit is calculated; input the teacher model hierarchical feature the student model hierarchical feature respectively from the hierarchical feature, the teacher model edge graph corresponding to each level and the student model edge graph and the teacher model class probability graph corresponding to each level and the student model class probability graph the student model hierarchical boundary distillation loss L​kd-edge , the student model hierarchical global distillation loss L kd-global ;

[0071] S4, the student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global are weighted and added to obtain the student model knowledge distillation loss L KD , the student model analytical loss L parsing and the student model knowledge distillation loss L KD are weighted and added to obtain the student model overall training loss L total ;

[0072] S5, the student model overall loss function L total is used to train the student model to obtain a trained student model;

[0073] S6, inputting the human body image into the trained student model to output an analytical prediction image.

[0074] Step S1 is specifically as follows:

[0075] S1-1, all images in the human body analytical data set are sequentially subjected to mean subtraction, random color jittering, random scale scaling, and random left-right flipping enhancement operations, wherein the random scale scaling range is 0.5 to 1.25 times of the original image.

[0076] Step S2 is specifically as follows:

[0077] S2-1, the overall framework of the teacher network and the student network is a class distribution guided network, which includes a backbone network, a class distribution guided high resolution module, an edge extraction module, and a feature fusion module, wherein the backbone network includes Conv1, Res2, Res3, Res4, and Res5, as shown in Figure 2 ;

[0078] The backbone network of the teacher model is a pre-trained ResNet101, and the backbone network of the student model is a ResNet50.

[0079] Step S3 is specifically as follows:

[0080] S3-1, the student model analytical loss L parsing is calculated according to the following formula:

[0081] L parsing = λ1L b + λ2L c + L p ,

[0082] wherein λ1, λ2 are hyperparameters balancing the loss of each part, L b is the weighted cross-entropy loss between the edge map obtained from the student model boundary feature and the edge map generated from the ground truth human parsing result y, L c is the cross-entropy loss between the feature horizontal class distribution map and the feature vertical class distribution map obtained from the student model horizontal distribution feature and the student model vertical class distribution feature and the result horizontal class distribution map Q h and the result vertical class distribution map Q v obtained from the ground truth human parsing result y, L p is the cross-entropy loss between the student model prediction result and the ground truth human parsing result y, L b , L c and L p can be calculated by directly calling the functions of the PyTorch framework;

[0083] The feature horizontal class distribution map and the feature vertical class distribution map are obtained using the softmax function from the student model horizontal distribution feature and the student model vertical class distribution feature The specific calculation formula is as follows:

[0084]

[0085] wherein C is the total number of classes of the human parsing dataset, exp(·) is the natural exponential;

[0086] For the result horizontal class distribution map Q h and the result vertical class distribution map Q v , the specific calculation formula is as follows:

[0087]

[0088] wherein the matrix M is generated by one-hot encoding using the ground truth human parsing result y, H and W are the height and width of the matrix M respectively, h ∈ [1, H], w ∈ [1, W], c ∈ [1, C];

[0089] The student model logit distillation loss L kd-logit is calculated according to the following formula:

[0090]

[0091] wherein, and respectively represent the probability of the pixel p belonging to the class i predicted by the teacher model and the student model;

[0092] S3-3, the student model hierarchical boundary distillation loss L kd-edge is calculated as follows:

[0093]

[0094] wherein, MSE represents the MSE loss function, and the teacher model edge map and the student model edge map are calculated as follows:

[0095]

[0096] wherein, G x and G y are two Sobel convolution kernels used to calculate the gradient of the hierarchical feature in the horizontal and vertical directions;

[0097] S3-4, the student model hierarchical global distillation loss L kd-global is calculated as follows:

[0098]

[0099] wherein, H'(i) and W'(i) respectively represent the height and width of the i-th layer class probability map, and KL(·) represents the KL divergence, and respectively represent the corresponding hierarchical class probability map generated by using the softmax function from the teacher model hierarchical feature and the student model hierarchical feature , i.e., the class probability of the x-th pixel of the i-th layer of the student model and the teacher model;

[0100] the specific calculation formula of the teacher model hierarchical class probability map is as follows:

[0101]

[0102] the specific calculation formula of the student model hierarchical class probability map is as follows:

[0103]

[0104] Step S4 is specifically as follows:

[0105] S4-1, the student model knowledge distillation loss L Kd The calculation formula is:

[0106] L Kd = alpha1L kd-logit + alpha2L kd-edge + alpha3L kd-global ,

[0107] Wherein, alpha1, alpha2, alpha3 are hyperparameters for balancing each part loss;

[0108] S4-2, the student model overall training loss L total The calculation formula is:

[0109] L total = beta1L parsing + beta2L KD ,

[0110] Wherein, beta1, beta2 are hyperparameters for balancing each part loss.

[0111] Correspondingly, the application also provides a hierarchical boundary knowledge distillation human parsing system, as shown in Figure 3 Comprise:

[0112] Preprocessing unit 1, for inputting human parsing data set, carrying out enhancement operation to human parsing data set, obtaining preprocessed human parsing data set;

[0113] Specifically, the enhancement operation comprises: sequentially performing mean subtraction, random color jittering, random scale zooming, random left-right flipping enhancement operation on all images in the human parsing data set, wherein the random scale zooming range is 0.5 to 1.25 times of the original image.

[0114] Model construction unit 2, for inputting the preprocessed human parsing data set, constructing teacher model and student model with same framework and different backbone network, outputting teacher model hierarchical feature Student model hierarchical feature Teacher model prediction result Student model prediction result Student model boundary feature Student model horizontal distribution feature And student model vertical class distribution feature Wherein i=2,3,4,5;

[0115] Specifically, the overall framework of the teacher network and the student network is a class distribution guided network, including a backbone network, a class distribution guided high-resolution module, an edge extraction module and a feature fusion module, wherein the backbone network includes Conv1, Res2, Res3, Res4 and Res5.

[0116] The backbone network of the teacher model is a pre-trained ResNet101, and the backbone network of the student model is ResNet50.

[0117] The loss function construction unit 3 is configured to input the real human parsing result y, find the boundary of the mask between two different semantic parts in the real human parsing result y, and obtain the edge label y b ; the real human parsing result y and the edge label y b , horizontal distribution features of the student model and vertical class distribution features of the student model are used to calculate the student model parsing loss Lparsing; the teacher model prediction result and the student model prediction result are used to calculate the student model logit distillation loss L kd-logit ; the teacher model hierarchical features and the student model hierarchical features are input respectively to obtain the teacher model edge map corresponding to each level and the student model edge map and the teacher model class probability map corresponding to each level and the student model class probability map are calculated to obtain the student model hierarchical boundary distillation loss L kd-edge , the student model hierarchical global distillation loss L kd-global ; the student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global are weighted and added to obtain the student model knowledge distillation loss L KD , the student model parsing loss L parsing and the student model knowledge distillation loss L KD are weighted and added to obtain the student model overall training loss L total .

[0118] Specifically, the student model parsing loss L parsing is calculated according to the following formula:

[0119] L parsing =λ1Lb + λ2L c + L p ,

[0120] wherein λ1, λ2 are hyperparameters balancing the loss of each part, L b is a weighted cross-entropy loss between the edge map obtained from the student model boundary feature and the edge map generated from the real human parsing result y, L c is a cross-entropy loss between the feature horizontal class distribution map and the feature vertical class distribution map obtained from the student model horizontal distribution feature and the student model vertical class distribution feature and the result horizontal class distribution map Q h and the result vertical class distribution map Q v generated from the real human parsing result y, L p is a cross-entropy loss between the student model prediction result and the real human parsing result y;

[0121] The feature horizontal class distribution map and the feature vertical class distribution map are obtained using the softmax function from the student model horizontal distribution feature and the student model vertical class distribution feature The specific calculation formula is as follows:

[0122]

[0123] wherein C is the total number of classes of the human parsing dataset, exp(·) is the natural exponential;

[0124] For the result horizontal class distribution map Q h and the result vertical class distribution map Q v The specific calculation formula is as follows:

[0125]

[0126] wherein the matrix M is generated using one-hot encoding from the real human parsing result y, H and W are the height and width of the matrix M respectively, h ∈ [1, H], w ∈ [1, W], c ∈ [1, C];

[0127] The student model logit distillation loss L kd-logit The calculation formula is as follows:

[0128]

[0129] wherein, and respectively represent the probability of the pixel p belonging to the class i predicted by the teacher model and the student model;

[0130] The hierarchical boundary distillation loss L kd-edge of the student model is calculated as follows:

[0131]

[0132] wherein, MSE represents the MSE loss function, and the teacher model edge map and the student model edge map of each level are calculated as follows:

[0133]

[0134] wherein, G x and G y are two Sobel convolution kernels used to calculate the gradient of the hierarchical feature in the horizontal and vertical directions;

[0135] The calculation formula of the hierarchical global distillation loss L kd-global of the student model is as follows:

[0136]

[0137] wherein, H'(i) and W'(i) respectively represent the height and width of the i-th level class probability map, and KL(·) represents the KL divergence, and respectively represent the corresponding hierarchical class probability map generated by using the softmax function from the teacher model hierarchical feature and the student model hierarchical feature , i.e., the class probability of the x-th pixel of the i-th level of the student model and the teacher model;

[0138] The specific calculation formula of the teacher model hierarchical class probability map is as follows:

[0139]

[0140] The specific calculation formula of the student model hierarchical class probability map is as follows:

[0141]

[0142] The calculation formula of the student model knowledge distillation loss L KD is as follows:

[0143] L KD = α1Lkd-logit + a2L kd-edge + a3L kd-global ,

[0144] wherein a1, a2, a3 are hyperparameters for balancing the loss of each part;

[0145] the student model overall training loss L total The calculation formula is:

[0146] L total = b1L parsing + b2L KD ,

[0147] wherein b1, b2 are hyperparameters for balancing the loss of each part.

[0148] The training unit 4 is configured to train the student model by using the student model overall loss function L total to obtain a trained student model;

[0149] The application unit 5 is configured to input the human body image into the trained student model and output the analysis prediction image.

[0150] Therefore, the present application proposes a hierarchical boundary knowledge distillation human body analysis method. First, because the teacher and student models have different sizes, the feature maps from the teacher and student do not necessarily have the same spatial dimensions. All before formally operating, first make the feature map of the student model pass through the feature alignment operation to make it have the same feature map dimension size as the corresponding feature map of the student model. The feature dimension alignment operation here is to operate the corresponding features of the student and teacher models layer by layer. Secondly, in each layer, the two features are respectively subjected to boundary knowledge distillation and structured knowledge distillation, so as to transfer the relevant knowledge of the teacher model to the student model layer by layer, and then improve the analysis result of the student model. This method can ensure high human body analysis accuracy while optimizing the size and complexity of the model, thereby effectively balancing the model accuracy and efficiency.

[0151] The above describes in detail the hierarchical boundary knowledge distillation human body analysis method and system provided by the embodiment of the present application. The principle and implementation mode of the present application are described by applying specific examples. The above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A hierarchical boundary knowledge distillation method for human body analysis, characterized in that, The method includes: Input a human body parsing dataset, perform augmentation operations on the human body parsing dataset, and obtain a preprocessed human body parsing dataset; Input the preprocessed human body parsing dataset, construct teacher and student models with the same framework but different backbone networks, and output hierarchical features of the teacher model. Hierarchical features of student models Teacher model prediction results Student model prediction results Student model boundary features Student model level distribution characteristics Vertical class distribution characteristics of student models Where i = 2, 3, 4, 5; Given a real human body parsing result y, find the boundary of the mask between two different semantic parts in the real human body parsing result y, and obtain the edge annotation y. b ; Utilizing real human body analysis results y and edge annotation y b The student model's level distribution characteristics and the vertical class distribution characteristics of the student model The analytical loss L of the student model is calculated. parsing ; Predicting results using the teacher model Student model prediction results Calculate the logit distillation loss L in the student model kd-logit Input the hierarchical features of the teacher model. The hierarchical features of the student model Obtain the teacher model edge map corresponding to each level from the hierarchical features respectively. and student model edge graph And the hierarchical category probability graph of the teacher model corresponding to each level. Hierarchical Class Probability Graph of Student Model The hierarchical boundary distillation loss L of the student model was calculated. kd-edge Student model hierarchical global distillation loss L kd-global ; The student model's logit distillation loss L kd-logit The hierarchical boundary distillation loss L of the student model kd-edge And the hierarchical global distillation loss L of the student model kd-global The weighted summation yields the student model knowledge distillation loss L. KD The analytical loss L of the student model is then used. parsing Knowledge distillation loss L from student model KD The weighted sums yield the overall training loss L of the student model. total ; The overall training loss L of the student model is used total The student model is trained to obtain a trained student model; Input a human image into a trained student model and output a parsed predicted image.

2. The hierarchical boundary knowledge distillation method for human body analysis as described in claim 1, characterized in that, The enhancement operation is specifically as follows: The human body analysis dataset is subjected to enhancement operations in sequence, including mean subtraction, random color jitter, random scaling, and random left and right flipping.

3. The hierarchical boundary knowledge distillation method for human body analysis as described in claim 1, characterized in that, The teacher and student models, which share the same framework but have different backbone networks, are specifically as follows: The overall framework of the teacher model and the student model is a class distribution guided network, including a backbone network, a class distribution guided high-resolution module, an edge extraction module, and a feature fusion module. The backbone network includes Conv1, Res2, Res3, Res4, and Res5. The backbone network of the teacher model is a pre-trained ResNet101, and the backbone network of the student model is a ResNet50.

4. The hierarchical boundary knowledge distillation method for human body analysis as described in claim 1, characterized in that, The student model analytical loss L parsing The student model's logit distillation loss L kd-logit The hierarchical boundary distillation loss L of the student model kd-edge And the hierarchical global distillation loss L of the student model kd-global Specifically: The student model analytical loss L parsing The calculation formula is as follows: L parsing =λ1L b +λ2L c +L p , Where λ1 and λ2 are hyperparameters balancing the losses of each component, L b From the student model boundary features The weighted cross-entropy loss between the obtained edge map and the edge map generated from the real human body parsing result y, L c From the student model level distribution characteristics and the vertical class distribution characteristics of the student model The obtained feature level class distribution map and feature vertical class distribution map And the result level class distribution map Q generated from the real human body analysis result y. h Vertical distribution plot of results Q v Cross-entropy loss between, L p The student model prediction result The cross-entropy loss between the actual human body analysis result y and the actual human body analysis result y; The softmax function is used to extract the level distribution features from the student model. and the vertical class distribution characteristics of the student model Obtain feature level class distribution map and feature vertical class distribution map The specific calculation formula is as follows: Where C is the total number of categories in the human body parsing dataset, and exp(·) is the natural index; For the resulting level class distribution map Q h The vertical class distribution plot Q of the results v The specific calculation formula is as follows: Wherein, matrix M is generated by one-hot encoding of the real human body parsing result y, H and W are the height and width of matrix M respectively, h∈[1,H], w∈[1,W], c∈[1,C]; The student model's logit distillation loss L kd-logit The calculation formula is as follows: in, and Let represent the probabilities that pixel p belongs to category k predicted by the teacher model and the student model, respectively. The student model hierarchical boundary distillation loss L kd-edge The calculation formula is as follows: Where MSE represents the MSE loss function, and the edge graph of the teacher model corresponds to each level. and student model edge graph The specific calculation formula is as follows: Among them, G x and G y It consists of two Sobel convolution kernels used to calculate the gradients in the horizontal and vertical directions of the hierarchical features; The student model hierarchical global distillation loss L kd-global The calculation formula is as follows: Where H′(i) and W′(i) represent the height and width of the i-th layer class probability map, respectively, and KL(·) represents the KL divergence. and These represent the use of the softmax function to hierarchically extract features from the teacher model. Hierarchical features of student models Generate the corresponding hierarchical category probability map, that is, the category probability of the x-th pixel in the i-th layer of the student model and the teacher model; Hierarchical Class Probability Graph of Teacher Model The specific calculation formula is as follows: Hierarchical Class Probability Graph of Student Model The specific calculation formula is as follows:

5. The hierarchical boundary knowledge distillation method for human body analysis as described in claim 1, characterized in that, The student model knowledge distillation loss L KD and the overall training loss L of the student model total Specifically: The student model knowledge distillation loss L KD The calculation formula is: L KD =α1L kd-logit +α2L kd-edge +α3L kd-global , where α1, α2, and α3 are hyperparameters used to balance the losses of each part; The overall training loss of the student model is L. total The calculation formula is: L total =β1L parsing +β2L KD , Among them, β1 and β2 are hyperparameters used to balance the losses of each part.

6. A hierarchical boundary knowledge distillation system for human body analysis, characterized in that, The system includes: The preprocessing unit is used to input the human body parsing dataset, perform augmentation operations on the human body parsing dataset, and obtain a preprocessed human body parsing dataset. The model building unit is used to take the preprocessed human parsing dataset as input, construct teacher and student models with the same framework but different backbone networks, and output hierarchical features of the teacher model. Hierarchical features of student models Teacher model prediction results Student model prediction results Student model boundary features Student model level distribution characteristics Vertical class distribution characteristics of student models Where i = 2, 3, 4, 5; The loss function construction unit is used as input to the real human body parsing result y, and finds the boundary of the mask between two different semantic parts in the real human body parsing result y to obtain the edge label y. b ; Utilizing real human body analysis results y and edge annotation y b The student model's level distribution characteristics and the vertical class distribution characteristics of the student model The analytical loss L of the student model is calculated. parsing ; Predicting results using the teacher model Student model prediction results Calculate the logit distillation loss L in the student model kd-logit Input the hierarchical features of the teacher model. The hierarchical features of the student model Obtain the teacher model edge map corresponding to each level from the hierarchical features respectively. and student model edge graph And the hierarchical category probability graph of the teacher model corresponding to each level. Hierarchical Class Probability Graph of Student Model The hierarchical boundary distillation loss L of the student model was calculated. kd-edge Student model hierarchical global distillation loss L kd-global The student model's logit distillation loss L kd-logit The hierarchical boundary distillation loss L of the student model kd-edge And the hierarchical global distillation loss L of the student model kd-global The weighted summation yields the student model knowledge distillation loss L. KD The analytical loss L of the student model is then used. parsing Knowledge distillation loss L from student model KD The weighted sums yield the overall training loss L of the student model. total ; Training unit, used to utilize the overall training loss L of the student model total The student model is trained to obtain a trained student model; The application unit is used to input human images into a trained student model and output a parsed predicted image.

7. The hierarchical boundary knowledge distillation human body analysis system as described in claim 6, characterized in that, The enhancement operations in the preprocessing unit are specifically as follows: The human body analysis dataset is subjected to enhancement operations in sequence, including mean subtraction, random color jitter, random scaling, and random left and right flipping.

8. The hierarchical boundary knowledge distillation human body analysis system as described in claim 6, characterized in that, The teacher and student models in the model building unit, which have the same framework but different backbone networks, are specifically as follows: The overall framework of the teacher model and the student model is a class distribution guided network, including a backbone network, a class distribution guided high-resolution module, an edge extraction module, and a feature fusion module. The backbone network includes Conv1, Res2, Res3, Res4, and Res5. The backbone network of the teacher model is a pre-trained ResNet101, and the backbone network of the student model is a ResNet50.

9. The hierarchical boundary knowledge distillation human body analysis system as described in claim 6, characterized in that, The student model analytical loss L in the loss function construction unit parsing The student model's logit distillation loss L kd-logit The hierarchical boundary distillation loss L of the student model kd-edge The hierarchical global distillation loss L of the student model kd-global The student model knowledge distillation loss L KD and the overall training loss L of the student model total Specifically: The student model analytical loss L parsing The calculation formula is as follows: L parsing =λ1L b +λ2L c +L p , Where λ1 and λ2 are hyperparameters balancing the losses of each component, L b From the student model boundary features The weighted cross-entropy loss between the obtained edge map and the edge map generated from the real human body parsing result y, L c From the student model level distribution characteristics and the vertical class distribution characteristics of the student model The obtained feature level class distribution map and feature vertical class distribution map And the result level class distribution map Q generated from the real human body analysis result y. h Vertical distribution plot of results Q v Cross-entropy loss between, L p The student model prediction result The cross-entropy loss between the actual human body analysis result y and the actual human body analysis result y; The softmax function is used to extract the level distribution features from the student model. and the vertical class distribution characteristics of the student model Obtain feature level class distribution map and feature vertical class distribution map The specific calculation formula is as follows: Where C is the total number of categories in the human body parsing dataset, and exp(·) is the natural index; For the resulting level class distribution map Q h The vertical class distribution plot Q of the results v The specific calculation formula is as follows: Wherein, matrix M is generated by one-hot encoding of the real human body parsing result y, H and W are the height and width of matrix M respectively, h∈[1,H], w∈[1,W], c∈[1,C]; The student model's logit distillation loss L kd-logit The calculation formula is as follows: in, and Let represent the probabilities that pixel p belongs to category k predicted by the teacher model and the student model, respectively. The student model hierarchical boundary distillation loss L kd-edge The calculation formula is as follows: Where MSE represents the MSE loss function, and the edge graph of the teacher model corresponds to each level. and student model edge graph The specific calculation formula is as follows: Among them, G x and G y It consists of two Sobel convolution kernels used to calculate the gradients in the horizontal and vertical directions of the hierarchical features; The student model hierarchical global distillation loss L kd-global The calculation formula is as follows: Where H′(i) and W′(i) represent the height and width of the i-th layer class probability map, respectively, and KL(·) represents the KL divergence. and These represent the use of the softmax function to hierarchically extract features from the teacher model. Hierarchical features of student models Generate the corresponding hierarchical category probability map, that is, the category probability of the x-th pixel in the i-th layer of the student model and the teacher model; Hierarchical Class Probability Graph of Teacher Model The specific calculation formula is as follows: Hierarchical Class Probability Graph of Student Model The specific calculation formula is as follows: The student model knowledge distillation loss L KD The calculation formula is: L KD =α1L kd-logit +α2L kd-edge +α3L kd-global , where α1, α2, and α3 are hyperparameters used to balance the losses of each part; The overall training loss of the student model is L. total The calculation formula is: L total =β1L parsing +β2L KD , Among them, β1 and β2 are hyperparameters used to balance the losses of each part.

Images