Human body analysis method and system for hierarchical boundary knowledge distillation

Abstract

The invention discloses a human body analysis method and system for hierarchical boundary knowledge distillation. Comprising the steps that a human body analysis data set is input and preprocessed; constructing a teacher model and a student model with the same framework and different backbone networks; calculating the analysis loss of the student model, the logit distillation loss of the student model, the hierarchical boundary distillation loss of the student model and the hierarchical global distillation loss of the student model; and carrying out weighted addition on the logit distillation loss, the hierarchical boundary distillation loss and the hierarchical global distillation loss to obtain knowledge distillation loss, carrying out weighted addition on the knowledge distillation loss and the student model analysis loss to obtain student model overall training loss, training the student model by using the student model overall training loss, inputting the human body image into the trained student model, and outputting an analysis prediction image. According to the method, the size and the complexity of the model can be optimized while relatively high human body analysis precision is ensured, so that effective balance between the model precision and the efficiency is realized.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and semantic segmentation, and in particular to a human body parsing method and system based on hierarchical boundary knowledge distillation. Background Art

[0002] Human body parsing is a subtask of image semantic segmentation, which aims to finely annotate the human body area in the image, so as to distinguish different body parts and clothing details. It has broad application prospects in the fields of virtual fitting, human posture estimation, pedestrian re-identification, etc. With the rapid development of convolutional neural networks (CNN), various advanced high-precision CNNs have been proposed for human body parsing tasks, but these high-precision networks are designed to be deep and complex in order to extract rich semantic information. The models are often large and the computational cost is also high, which makes it difficult for these high-performance large-scale models to be deployed on resource-limited platforms and devices such as mobile terminals and edge devices, and even more difficult to be widely used in real life.

[0003] To solve this problem, lightweight basic models such as EfficientNet are used for real-time semantic segmentation. In addition, model compression is also a common approach to solve this problem. The four common methods of model compression include network pruning, weight decomposition, weight quantization, or knowledge distillation. Among them, knowledge distillation is a method that can transfer the knowledge extracted from a larger teacher model to a smaller student network to improve the performance of the student network. It does not change the original network structure and can be used simultaneously with other model compression methods, with better scalability. The original knowledge distillation method was used in image classification tasks, but in recent years it has been extended to a variety of tasks including semantic segmentation. In the semantic segmentation task, knowledge distillation methods can be divided into two categories: spatial correlation distillation and channel correlation distillation. Although the semantic segmentation method based on knowledge distillation has advantages in computational efficiency, it ignores the important influence of human boundaries on the segmentation results, resulting in problems such as uneven segmentation boundaries in the generated human body parsing results.

[0004] Boundary information, as an important auxiliary cue, has been widely used to develop high-precision semantic segmentation networks, allowing the shape of the human body to be better depicted. However, most existing methods require the design of additional modules or sub-networks to learn boundary cues to improve the accuracy of human parsing tasks, but this inevitably leads to a significant increase in model size and computational cost. Therefore, for lightweight human parsing, it is very important to encode human boundaries as additional auxiliary cues in an effective and efficient way.

[0005] One of the current existing technologies is EDGNet, an accurate human parsing method based on edge-guided diffusion, in the paper "Toward Accurate Human Parsing Through EdgeGuided Diffusion". The method first introduces a horizontal-vertical aggregation module to aggregate contextual information from the horizontal and vertical directions to promote the classification of semantic parts. Subsequently, multi-stage edge-guided diffusion is performed to refine the parsing results. Each stage includes a boundary-guided diffusion and a channel aggregation module to enhance the semantic consistency within the body part guided by the semantic boundary. In addition, in order to introduce additional high-resolution features to supplement the details of the previous predictions, the backbone features and semantic boundary features are combined with the diffusion prediction, and the features with low-level details and high-level semantic information are effectively fused together through the channel aggregation module. The method adopts the joint learning of semantic boundary detection and human parsing to promote the localization of boundary areas, thereby promoting the accuracy of human parsing, but the disadvantage of this method is that due to the use of a multi-stage boundary diffusion scheme, the method requires a long time to train, and the model is relatively complex and cannot be deployed to devices with high real-time requirements.

[0006] The second current prior art is the patent "A human body parsing method and system based on knowledge distillation (CN117789244A)". This method first trains a larger and more complex teacher model, and then in the process of training a smaller and simpler student model, the boundary features, posture features and parsing features are used as knowledge and transferred from the trained teacher model to the student model being trained, so that the student model can maximize the parsing speed while ensuring accuracy. This method combines the knowledge distillation method with the human body parsing task. Although it can balance the accuracy and speed of the student model to obtain a lightweight human body parsing model with good performance, the disadvantage of this method is that since it transfers boundary features, posture features and parsing features as knowledge in the process of knowledge distillation, these three features may restrict each other during the training process, resulting in limited parsing accuracy and failure to obtain better parsing results.

[0007] The third existing technology is a boundary privileged knowledge distillation method for semantic segmentation tasks in the paper "BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation". This method extracts knowledge from the subject and boundary of the teacher model and transfers it to the compact student model. The whole process can be divided into two sections. First, the method designs a boundary knowledge distillation loss, which uses spatial probability alignment and aggregation of contextual information to refine the boundary, aiming to distinguish fuzzy categories at the pixel level in the boundary area. Then, the method designs a body knowledge distillation loss, which uses shape constraints to selectively focus on internal semantic areas, thereby eliminating high-uncertainty boundary samples and smoothing the learning curve, further improving the segmentation results of the body area. The disadvantage of this method is that since it extracts boundary knowledge from the final feature representation, and high-level features are usually obtained through multiple layers of transformation and dimensionality reduction, this may cause some low-level detail information to be lost, which may be important for boundary detection. Therefore, this method may make the boundary knowledge transferred by the model inaccurate, so that the performance of the final student model cannot be maximized. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the existing methods and propose a human body analysis method and system with hierarchical boundary knowledge distillation. The main problem solved by the present invention is how to better utilize the information related to the human body, maximize the accuracy of the human body analysis results, improve the real-time efficiency, and control the scale of the model so that the model can be deployed to devices with high real-time requirements.

[0009] In order to solve the above problems, the present invention proposes a human body parsing method based on hierarchical boundary knowledge distillation, which includes:

[0010] Input a human body parsing data set, perform an enhancement operation on the human body parsing data set, and obtain a preprocessed human body parsing data set;

[0011] Input the preprocessed human body parsing dataset, build a teacher model and a student model with the same framework and different backbone networks, and output the hierarchical features of the teacher model Student model hierarchical features Teacher model prediction results Student model prediction results Student model boundary characteristics Student model level distribution characteristics And the vertical class distribution characteristics of the student model Where i = 2, 3, 4, 5;

[0012] Input the real human parsing result y, find the boundary of the mask between two different semantic parts in the real human parsing result y, and obtain the edge annotation y b ; Using the real human body analysis results y and edge annotations y b , the student model level distribution characteristics And the vertical class distribution characteristics of the student model Calculate and get the student model parsing loss L parsing ; Use the teacher model to predict the results And the student model prediction results Calculate the student model logit distillation loss L kd-logit ; Input the hierarchical features of the teacher model The student model hierarchical features Obtain the teacher model edge graph corresponding to each level from the hierarchical features and the student model marginal graph And the teacher model category probability map corresponding to each level and student model category probability plot Calculate the student model hierarchical boundary distillation loss L kd-edge , student model hierarchical global distillation loss L kd-global ;

[0013] The student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global Weighted addition gives the student model knowledge distillation loss L KD , the student model parse loss L parsing and the student model knowledge distillation loss L KD Weighted addition gives the overall training loss L of the student model total ;

[0014] Using the overall loss function L of the student model total Training the student model to obtain a trained student model;

[0015] Input the human body image into the trained student model and output the parsed prediction image.

[0016] Preferably, the enhancement operation is specifically:

[0017] All images in the human body analysis dataset are subjected to enhancement operations of mean subtraction, random color jittering, random scale scaling, and random left-right flipping in sequence, wherein the random scale scaling range is 0.5 to 1.25 times of the original image.

[0018] Preferably, the teacher model and the student model having the same framework and different backbone networks are specifically:

[0019] The overall framework of the teacher network and the student network is a class distribution guided network, including a backbone network, a class distribution guided high-resolution module, an edge extraction module and a feature fusion module, wherein the backbone network includes Conv1, Res2, Res3, Res4 and Res5;

[0020] The backbone network of the teacher model is the pre-trained ResNet101, and the backbone network of the student model is ResNet50.

[0021] Preferably, the student model parsing loss Lparsing, the student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global , specifically:

[0022] The student model parses the loss L parsing The calculation formula is as follows:

[0023] L parsing =λ 1 L b +λ 2 L c +L p ,

[0024] Among them, λ 1 , 2 is a hyperparameter that balances the losses of each part, L b is the boundary feature from the student model The weighted cross entropy loss between the obtained edge map and the edge map generated from the true human parsing result y, L c is the level distribution feature from the student model And the vertical class distribution characteristics of the student model The obtained feature level class distribution map And feature vertical class distribution map And the result level class distribution map Q generated from the real human body analysis result y h And the resulting vertical class distribution graph Q v The cross entropy loss between p is the prediction result of the student model and the cross entropy loss between the true human parsing result y;

[0025] Use the softmax function to horizontally distribute features from the student model And the vertical class distribution characteristics of the student model Get feature level class distribution map And feature vertical class distribution map The specific calculation formula is as follows:

[0026]

[0027] Wherein, C is the total number of categories in the human body parsing dataset, and exp(·) is the natural exponent;

[0028] For the result level class distribution diagram Q h And the resulting vertical class distribution graph Q v , the specific calculation formula is as follows:

[0029]

[0030] Wherein, the matrix M is generated by one-hot encoding of the real human body analysis result y, H and W are the height and width of the matrix M, h∈[1,H], w∈[1,W], c∈[1,C];

[0031] The student model logit distillation loss L kd-logit The calculation formula is as follows:

[0032]

[0033] in, and Respectively represent the probability that the teacher model and the student model predict that the pixel p belongs to category i;

[0034] The student model hierarchical boundary distillation loss L kd-edge The calculation formula is as follows:

[0035]

[0036] Among them, MSE represents the MSE loss function, and the teacher model edge graph corresponding to each level and the student model marginal graph The specific calculation formula is as follows:

[0037]

[0038] Among them, G x and G y There are two Sobel convolution kernels, which are used to calculate the horizontal and vertical gradients of the hierarchical features;

[0039] The student model hierarchical global distillation loss L kd-global The calculation formula is as follows:

[0040]

[0041] Where H′(i) and W′(i) represent the height and width of the i-th layer category probability map, respectively, and KL(·) represents the KL divergence. and Respectively represent the hierarchical features from the teacher model using the softmax function and student model hierarchical features Generate the corresponding hierarchical category probability map, that is, the category probability of the xth pixel in the i-th layer of the student model and the teacher model;

[0042] Teacher model hierarchical category probability map The specific calculation formula is:

[0043]

[0044] Student model hierarchical category probability map The specific calculation formula is:

[0045]

[0046] Preferably, the student model knowledge distillation loss L KD and the overall training loss L of the student model total , specifically:

[0047] The student model knowledge distillation loss L KD The calculation formula is:

[0048] L KD =α 1 L kd-logit +α 2 L kd-edge +α 3 L kd-global ,

[0049] Among them, α 1 , α 2 , α 3 It is a hyperparameter used to balance the losses of each part;

[0050] The overall training loss of the student model is L total The calculation formula is:

[0051] L total =β 1 L parsing +β 2 L KD ,

[0052] Among them, β 1 , β2 is a hyperparameter used to balance the losses of each part.

[0053] Accordingly, the present invention also provides a human body parsing system for hierarchical boundary knowledge distillation, comprising:

[0054] A preprocessing unit, used for inputting a human body parsing data set, performing an enhancement operation on the human body parsing data set, and obtaining a preprocessed human body parsing data set;

[0055] A model building unit is used to input the preprocessed human body analysis data set, build a teacher model and a student model with the same framework and different backbone networks, and output the hierarchical features of the teacher model. Student model hierarchical features Teacher model prediction results Student model prediction results Student model boundary characteristics Student model level distribution characteristics And the vertical class distribution characteristics of the student model Where i = 2, 3, 4, 5;

[0056] The loss function construction unit is used to input the real human parsing result y, find the boundary of the mask between two different semantic parts in the real human parsing result y, and obtain the edge annotation y b ; Using the real human body analysis results y and edge annotations y b , the student model level distribution characteristics And the vertical class distribution characteristics of the student model Calculate and obtain the student model parsing loss Lparsing; use the teacher model to predict the result And the student model prediction results Calculate the student model logit distillation loss L kd-logit ; Input the hierarchical features of the teacher model The student model hierarchical features Obtain the teacher model edge graph corresponding to each level from the hierarchical features and the student model marginal graph And the teacher model category probability map corresponding to each level and student model category probability plot Calculate the student model hierarchical boundary distillation loss L kd-edge , student model hierarchical global distillation loss L kd-global ; Distill the student model logit loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss Lkd-global Weighted addition gives the student model knowledge distillation loss L KD , the student model parse loss L parsing and the student model knowledge distillation loss L KD Weighted addition gives the overall training loss L of the student model total ;

[0057] A training unit is used to utilize the overall loss function L of the student model total Training the student model to obtain a trained student model;

[0058] The application unit is used to input the human body image into the trained student model and output the parsed prediction image.

[0059] The implementation of the present invention has the following beneficial effects:

[0060] This paper makes full use of the fact that the body part boundaries can promote the human body parsing model to distinguish adjacent parts, maintain the consistency and coherence of segmentation in a global range, and thus promote the parsing accuracy. It adopts a hierarchical boundary knowledge distillation method and applies the knowledge distillation method to the human body parsing task, which helps to compress the model size while maintaining high accuracy. This allows our method to be applied to devices with limited computing power and memory.

[0061] Compared with most existing human body parsing methods that do not use lightweight operations, the method adopted by the present invention has the characteristics of fast calculation speed and better balance between accuracy and efficiency; compared with the existing knowledge distillation methods designed based on human body parsing tasks, the present invention can better utilize multi-scale boundary features and thus better segment the semantic boundaries of the human body; compared with general knowledge distillation methods designed for general semantic segmentation tasks, the method adopted by the present invention can more fully utilize the characteristics of human body parsing tasks, is more suitable for human body parsing tasks, and can obtain more accurate segmentation results.

[0062] This method can optimize the size and complexity of the model while ensuring high human body analysis accuracy, thereby achieving an effective trade-off between model accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flowchart of a human body analysis method of hierarchical boundary knowledge distillation according to an embodiment of the present invention;

[0064] Figure 2 It is a schematic diagram of the overall framework of the teacher model and the student model of an embodiment of the present invention;

[0065] Figure 3It is a structural diagram of a human body analysis system with hierarchical boundary knowledge distillation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Figure 1 is a flowchart of a human body analysis method of hierarchical boundary knowledge distillation according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0068] S1, inputting a human body parsing dataset, performing an enhancement operation on the human body parsing dataset, and obtaining a preprocessed human body parsing dataset;

[0069] S2, input the preprocessed human body parsing dataset, build a teacher model and a student model with the same framework and different backbone networks, and output the hierarchical features of the teacher model Student model hierarchical features Teacher model prediction results Student model prediction results Student model boundary characteristics Student model level distribution characteristics And the vertical class distribution characteristics of the student model Where i = 2, 3, 4, 5;

[0070] S3: Input the real human parsing result y, find the boundary of the mask between two different semantic parts in the real human parsing result y, and obtain the edge annotation y b ; Using the real human body analysis results y and edge annotations y b , the student model level distribution characteristics And the vertical class distribution characteristics of the student model Calculate and get the student model parsing loss L parsing ; Use the teacher model to predict the results And the student model prediction results Calculate the student model logit distillation loss L kd-logit ; Input the hierarchical features of the teacher model The student model hierarchical features Obtain the teacher model edge graph corresponding to each level from the hierarchical features and the student model marginal graph And the teacher model category probability map corresponding to each level and student model category probability plot Calculate the student model hierarchical boundary distillation loss L kd-edge , student model hierarchical global distillation loss L kd-global ;

[0071] S4, the student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global Weighted addition gives the student model knowledge distillation loss L KD , the student model parse loss L parsing and the student model knowledge distillation loss L KD Weighted addition gives the overall training loss L of the student model total ;

[0072] S5, using the overall loss function L of the student model total Training the student model to obtain a trained student model;

[0073] S6, input the human body image into the trained student model and output the parsed prediction image.

[0074] Step S1 is as follows:

[0075] S1-1, performing mean subtraction, random color jittering, random scale scaling, and random left-right flipping enhancement operations on all images in the human body analysis dataset in sequence, wherein the random scale scaling range is 0.5 to 1.25 times of the original image.

[0076] Step S2 is as follows:

[0077] S2-1, the overall framework of the teacher network and the student network is a class distribution guided network, including a backbone network, a class distribution guided high-resolution module, an edge extraction module and a feature fusion module, wherein the backbone network includes Conv1, Res2, Res3, Res4 and Res5, such as Figure 2 As shown;

[0078] The backbone network of the teacher model is the pre-trained ResNet101, and the backbone network of the student model is ResNet50.

[0079] Step S3 is as follows:

[0080] S3-1, the student model parsing loss L parsing The calculation formula is as follows:

[0081] L parsing =λ 1 L b +λ 2 L c +L p ,

[0082] Among them, λ 1 , 2 is a hyperparameter that balances the losses of each part, L b is the boundary feature from the student model The weighted cross entropy loss between the obtained edge map and the edge map generated from the true human parsing result y, L c is the level distribution feature from the student model And the vertical class distribution characteristics of the student model The obtained feature level class distribution map And feature vertical class distribution map And the result level class distribution map Q generated from the real human body analysis result y h And the resulting vertical class distribution graph Q v The cross entropy loss between p is the prediction result of the student model and the cross entropy loss between the true human parsing result y; L b , L c and L p It can be calculated by directly calling the function of the PyTorch framework;

[0083] Use the softmax function to horizontally distribute features from the student model And the vertical class distribution characteristics of the student model Get feature level class distribution map And feature vertical class distribution map The specific calculation formula is as follows:

[0084]

[0085] Wherein, C is the total number of categories in the human body parsing dataset, and exp(·) is the natural exponent;

[0086] For the result level class distribution diagram Q h And the resulting vertical class distribution graph Q v , the specific calculation formula is as follows:

[0087]

[0088] Wherein, the matrix M is generated by one-hot encoding of the real human body analysis result y, H and W are the height and width of the matrix M, h∈[1,H], w∈[1,W], c∈[1,C];

[0089] S3-2, the student model logit distillation loss L kd-logit The calculation formula is as follows:

[0090]

[0091] in, and Respectively represent the probability that the teacher model and the student model predict that the pixel p belongs to category i;

[0092] S3-3, the student model hierarchical boundary distillation loss L kd-edge The calculation formula is as follows:

[0093]

[0094] Among them, MSE represents the MSE loss function, and the teacher model edge graph corresponding to each level and the student model marginal graph The specific calculation formula is as follows:

[0095]

[0096] Among them, G x and G y There are two Sobel convolution kernels, which are used to calculate the horizontal and vertical gradients of the hierarchical features;

[0097] S3-4, the student model hierarchical global distillation loss L kd-global The calculation formula is as follows:

[0098]

[0099] Where H′(i) and W′(i) represent the height and width of the i-th layer category probability map, respectively, and KL(·) represents the KL divergence. and Respectively represent the hierarchical features from the teacher model using the softmax function and student model hierarchical features Generate the corresponding hierarchical category probability map, that is, the category probability of the xth pixel in the i-th layer of the student model and the teacher model;

[0100] Teacher model hierarchical category probability map The specific calculation formula is:

[0101]

[0102] Student model hierarchical category probability map The specific calculation formula is:

[0103]

[0104] Step S4 is as follows:

[0105] S4-1, the student model knowledge distillation loss L Kd The calculation formula is:

[0106] L Kd =α 1 L kd-logit +α 2 L kd-edge +α 3 L kd-global ,

[0107] Among them, α 1 , α 2 , α 3 It is a hyperparameter used to balance the losses of each part;

[0108] S4-2, the overall training loss of the student model L total The calculation formula is:

[0109] L total =β 1 L parsing +β 2 L KD ,

[0110] Among them, β 1 , β 2 is a hyperparameter used to balance the losses of each part.

[0111] Accordingly, the present invention also provides a human body parsing system based on hierarchical boundary knowledge distillation, such as Figure 3 As shown, including:

[0112] A preprocessing unit 1 is used to input a human body parsing data set, perform an enhancement operation on the human body parsing data set, and obtain a preprocessed human body parsing data set;

[0113] Specifically, the enhancement operation includes: performing mean subtraction, random color jittering, random scale scaling, and random left-right flipping enhancement operations on all images in the human body analysis data set in sequence, wherein the random scale scaling range is 0.5 to 1.25 times of the original image.

[0114] Model building unit 2, used to input the preprocessed human body analysis data set, build a teacher model and a student model with the same framework and different backbone networks, and output the hierarchical features of the teacher model Student model hierarchical features Teacher model prediction results Student model prediction results Student model boundary characteristics Student model level distribution characteristics And the vertical class distribution characteristics of the student model Where i = 2, 3, 4, 5;

[0115] Specifically, the overall framework of the teacher network and the student network is a class distribution guided network, including a backbone network, a class distribution guided high-resolution module, an edge extraction module and a feature fusion module, wherein the backbone network includes Conv1, Res2, Res3, Res4 and Res5;

[0116] The backbone network of the teacher model is the pre-trained ResNet101, and the backbone network of the student model is ResNet50.

[0117] The loss function construction unit 3 is used to input the real human parsing result y, find the boundary of the mask between two different semantic parts in the real human parsing result y, and obtain the edge annotation y b ; Using the real human body analysis results y and edge annotations y b , the student model level distribution characteristics And the vertical class distribution characteristics of the student model Calculate and obtain the student model parsing loss Lparsing; use the teacher model to predict the result And the student model prediction results Calculate the student model logit distillation loss L kd-logit ; Input the hierarchical features of the teacher model The student model hierarchical features Obtain the teacher model edge graph corresponding to each level from the hierarchical features and the student model marginal graph And the teacher model category probability map corresponding to each level and student model category probability plot Calculate the student model hierarchical boundary distillation loss L kd-edge , student model hierarchical global distillation loss L kd-global ; Distill the student model logit loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edgeand the student model hierarchical global distillation loss L kd-global Weighted addition gives the student model knowledge distillation loss L KD , the student model parse loss L parsing and the student model knowledge distillation loss L KD Weighted addition gives the overall training loss L of the student model total ;

[0118] Specifically, the student model parsing loss L parsing The calculation formula is as follows:

[0119] L parsing =λ 1 L b +λ 2 L c +L p ,

[0120] Among them, λ 1 , 2 is a hyperparameter that balances the losses of each part, L b is the boundary feature from the student model The weighted cross entropy loss between the obtained edge map and the edge map generated from the true human parsing result y, L c is the level distribution feature from the student model And the vertical class distribution characteristics of the student model The obtained feature level class distribution map And feature vertical class distribution map And the result level class distribution map Q generated from the real human body analysis result y h And the resulting vertical class distribution graph Q v The cross entropy loss between p is the prediction result of the student model and the cross entropy loss between the true human parsing result y;

[0121] Use the softmax function to horizontally distribute features from the student model And the vertical class distribution characteristics of the student model Get feature level class distribution map And feature vertical class distribution map The specific calculation formula is as follows:

[0122]

[0123] Wherein, C is the total number of categories in the human body parsing dataset, and exp(·) is the natural exponent;

[0124] For the result level class distribution diagram Qh And the resulting vertical class distribution graph Q v , the specific calculation formula is as follows:

[0125]

[0126] Wherein, the matrix M is generated by one-hot encoding of the real human body analysis result y, H and W are the height and width of the matrix M, h∈[1,H], w∈[1,W], c∈[1,C];

[0127] The student model logit distillation loss L kd-logit The calculation formula is as follows:

[0128]

[0129] in, and Respectively represent the probability that the teacher model and the student model predict that the pixel p belongs to category i;

[0130] The student model hierarchical boundary distillation loss L kd-edge The calculation formula is as follows:

[0131]

[0132] Among them, MSE represents the MSE loss function, and the teacher model edge graph corresponding to each level and the student model marginal graph The specific calculation formula is as follows:

[0133]

[0134] Among them, G x and G y There are two Sobel convolution kernels, which are used to calculate the horizontal and vertical gradients of the hierarchical features;

[0135] The student model hierarchical global distillation loss L kd-global The calculation formula is as follows:

[0136]

[0137] Where H′(i) and W′(i) represent the height and width of the i-th layer category probability map, respectively, and KL(·) represents the KL divergence. and Respectively represent the hierarchical features from the teacher model using the softmax function and student model hierarchical features Generate the corresponding hierarchical category probability map, that is, the category probability of the xth pixel in the i-th layer of the student model and the teacher model;

[0138] Teacher model hierarchical category probability map The specific calculation formula is:

[0139]

[0140] Student model hierarchical category probability map The specific calculation formula is:

[0141]

[0142] The student model knowledge distillation loss L KD The calculation formula is:

[0143] L KD =α 1 L kd-logit +α 2 L kd-edge +α 3 L kd-global ,

[0144] Among them, α 1 , α 2 , α 3 It is a hyperparameter used to balance the losses of each part;

[0145] The overall training loss of the student model is L total The calculation formula is:

[0146] L total =β 1 L parsing +β 2 L KD ,

[0147] Among them, β 1 , β 2 is a hyperparameter used to balance the losses of each part.

[0148] Training unit 4, for using the overall loss function L of the student model total Training the student model to obtain a trained student model;

[0149] The application unit 5 is used to input the human body image into the trained student model and output the analytical prediction image.

[0150] Therefore, the present invention proposes a human body parsing method of hierarchical boundary knowledge distillation. First, due to the different sizes of the teacher and student models, the feature maps from the teacher and the student do not necessarily have the same spatial dimensions. Before the formal operation, the feature map of the student model is first subjected to a feature alignment operation to make it have the same dimension size as the feature map corresponding to the student model. The feature dimension alignment operation here is performed on the corresponding features of the student and teacher models layer by layer. Secondly, in each layer, boundary knowledge distillation and structured knowledge distillation are performed on the two features respectively, so that the relevant knowledge of the teacher model is passed to the student model layer by layer, thereby improving the parsing results of the student model. This method can optimize the size and complexity of the model while ensuring high human body parsing accuracy, thereby achieving an effective trade-off between model accuracy and efficiency.

[0151] The above is a detailed introduction to the human body analysis method and system with hierarchical boundary knowledge distillation provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A human body parsing method based on hierarchical boundary knowledge distillation, characterized in that: The method comprises: Input a human body parsing data set, perform an enhancement operation on the human body parsing data set, and obtain a preprocessed human body parsing data set; Input the preprocessed human body parsing dataset, build a teacher model and a student model with the same framework and different backbone networks, and output the hierarchical features of the teacher model Student model hierarchical features Teacher model prediction results Student model prediction results Student model boundary characteristics Student model level distribution characteristics And the vertical class distribution characteristics of the student model Where i = 2, 3, 4, 5; Input the real human parsing result y, find the boundary of the mask between two different semantic parts in the real human parsing result y, and obtain the edge annotation y b ; Using the real human body analysis results y and edge annotations y b , the student model level distribution characteristics And the vertical class distribution characteristics of the student model Calculate and get the student model parsing loss L parsing ; Use the teacher model to predict the results And the student model prediction results Calculate the logit distillation loss of the student model; input the hierarchical features of the teacher model The student model hierarchical features Obtain the teacher model edge graph corresponding to each level from the hierarchical features and the student model marginal graph And the teacher model category probability map corresponding to each level and student model category probability plot Calculate the student model hierarchical boundary distillation loss L kd-edge , student model hierarchical global distillation loss L kd-global ; The student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global Weighted addition gives the student model knowledge distillation loss L KD , the student model parse loss L parsing and the student model knowledge distillation loss L KD Weighted addition gives the overall training loss L of the student model total ; Using the overall loss function L of the student model total Training the student model to obtain a trained student model; Input the human body image into the trained student model and output the parsed prediction image.

2. The human body parsing method based on hierarchical boundary knowledge distillation as claimed in claim 1, characterized in that: The enhancement operation is specifically as follows: All images in the human body analysis dataset are enhanced by mean subtraction, random color jittering, random scale scaling, and random left-right flipping.

3. The human body parsing method based on hierarchical boundary knowledge distillation as claimed in claim 1, characterized in that: The teacher model and student model with the same framework and different backbone networks are specifically: The overall framework of the teacher network and the student network is a class distribution guided network, including a backbone network, a class distribution guided high-resolution module, an edge extraction module and a feature fusion module, wherein the backbone network includes Conv1, Res2, Res3, Res4 and Res5; The backbone network of the teacher model is the pre-trained ResNet101, and the backbone network of the student model is ResNet50.

4. The human body parsing method based on hierarchical boundary knowledge distillation as claimed in claim 1, characterized in that: The student model parses the loss L parsing 、The student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global , specifically: The student model parses the loss L parsing The calculation formula is as follows: L parsing =λ1L b +λ2L c +L p , Among them, λ1 and λ2 are hyperparameters for balancing the losses of each part, and L b is the boundary feature from the student model The weighted cross entropy loss between the obtained edge map and the edge map generated from the true human parsing result y, L c is the level distribution feature from the student model And the vertical class distribution characteristics of the student model The obtained feature level class distribution map And feature vertical class distribution map And the result level class distribution map Q generated from the real human body analysis result y h And the resulting vertical class distribution graph Q v The cross entropy loss between p is the prediction result of the student model and the cross entropy loss between the true human parsing result y; Use the softmax function to horizontally distribute features from the student model And the vertical class distribution characteristics of the student model Get feature level class distribution map And feature vertical class distribution map The specific calculation formula is as follows: Wherein, C is the total number of categories in the human body parsing dataset, and exp(·) is the natural exponent; For the result level class distribution diagram Q h And the resulting vertical class distribution graph Q v , the specific calculation formula is as follows: Wherein, the matrix M is generated by one-hot encoding of the real human body analysis result y, H and W are the height and width of the matrix M, h∈[1,H], w∈[1,W], c∈[1,C]; The student model logit distillation loss L kd-logit The calculation formula is as follows: in, and Respectively represent the probability that the teacher model and the student model predict that the pixel p belongs to category i; The student model hierarchical boundary distillation loss L kd-edge The calculation formula is as follows: Among them, MSE represents the MSE loss function, and the teacher model edge graph corresponding to each level and the student model marginal graph The specific calculation formula is as follows: Among them, G x and G y There are two Sobel convolution kernels, which are used to calculate the horizontal and vertical gradients of the hierarchical features; The student model hierarchical global distillation loss L kd-global The calculation formula is as follows: Where H′(i) and W′(i) represent the height and width of the i-th layer category probability map, respectively, and KL(·) represents the KL divergence. and Respectively represent the hierarchical features from the teacher model using the softmax function and student model hierarchical features Generate the corresponding hierarchical category probability map, that is, the category probability of the xth pixel in the i-th layer of the student model and the teacher model; Teacher model hierarchical category probability map The specific calculation formula is: Student model hierarchical category probability map The specific calculation formula is:

5. The human body parsing method based on hierarchical boundary knowledge distillation as claimed in claim 1, characterized in that: The student model knowledge distillation loss L KD and the overall training loss L of the student model total , specifically: The student model knowledge distillation loss L KD The calculation formula is: L KD =α1L kd-logit +α2L kd-edge +α3L kd-global , where α1, α2, and α3 are hyperparameters used to balance the losses of each part; The overall training loss of the student model is L total The calculation formula is: L total =β1L parsing +β2L KD , Among them, β1 and β2 are hyperparameters used to balance the losses of each part.

6. A human body parsing system based on hierarchical boundary knowledge distillation, characterized in that: The system comprises: A preprocessing unit, used for inputting a human body parsing data set, performing an enhancement operation on the human body parsing data set, and obtaining a preprocessed human body parsing data set; A model building unit is used to input the preprocessed human body analysis data set, build a teacher model and a student model with the same framework and different backbone networks, and output the hierarchical features of the teacher model. Student model hierarchical features Teacher model prediction results Student model prediction results Student model boundary characteristics Student model level distribution characteristics And the vertical class distribution characteristics of the student model Where i = 2, 3, 4, 5; The loss function construction unit is used to input the real human parsing result y, find the boundary of the mask between two different semantic parts in the real human parsing result y, and obtain the edge annotation y b ; Using the real human body analysis results y and edge annotations y b , the student model level distribution characteristics And the vertical class distribution characteristics of the student model Calculate and get the student model parsing loss L parsing ; Use the teacher model to predict the results And the student model prediction results Calculate the student model logit distillation loss L kd-logit ; Input the hierarchical features of the teacher model The student model hierarchical features Obtain the teacher model edge graph corresponding to each level from the hierarchical features and the student model marginal graph And the teacher model category probability map corresponding to each level and student model category probability plot Calculate the student model hierarchical boundary distillation loss L kd-edge , student model hierarchical global distillation loss L kd-global ; Distill the student model logit loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge and the student model hierarchical global distillation loss L kd-global Weighted addition gives the student model knowledge distillation loss L KD , the student model parse loss L parsing and the student model knowledge distillation loss L KD Weighted addition gives the overall training loss L of the student model total ; A training unit is used to utilize the overall loss function L of the student model total Training the student model to obtain a trained student model; The application unit is used to input the human body image into the trained student model and output the parsed prediction image.

7. A human body analysis system based on hierarchical boundary knowledge distillation as claimed in claim 6, characterized in that: The enhancement operations in the preprocessing unit are specifically: All images in the human body analysis dataset are enhanced by mean subtraction, random color jittering, random scale scaling, and random left-right flipping.

8. The human body analysis system of hierarchical boundary knowledge distillation according to claim 6, characterized in that: The teacher model and student model in the model building unit with the same framework and different backbone networks are specifically: The overall framework of the teacher network and the student network is a class distribution guided network, including a backbone network, a class distribution guided high-resolution module, an edge extraction module and a feature fusion module, wherein the backbone network includes Conv1, Res2, Res3, Res4 and Res5; The backbone network of the teacher model is the pre-trained ResNet101, and the backbone network of the student model is ResNet50.

9. The human body analysis system of hierarchical boundary knowledge distillation according to claim 6, characterized in that: The student model parsing loss L in the loss function building unit parsing 、The student model logit distillation loss L kd-logit , the student model hierarchical boundary distillation loss L kd-edge , the student model hierarchical global distillation loss L kd-global , the student model knowledge distillation loss L KD and the overall training loss L of the student model total , specifically: The student model parses the loss L parsing The calculation formula is as follows: L parsing =λ1L b +λ2L c +L p , Among them, λ1 and λ2 are hyperparameters for balancing the losses of each part, and L b is the boundary feature from the student model The weighted cross entropy loss between the obtained edge map and the edge map generated from the true human parsing result y, L c is the level distribution feature from the student model And the vertical class distribution characteristics of the student model The obtained feature level class distribution map And feature vertical class distribution map And the result level class distribution map Q generated from the real human body analysis result y h And the resulting vertical class distribution graph Q v The cross entropy loss between p is the prediction result of the student model and the cross entropy loss between the true human parsing result y; Use the softmax function to horizontally distribute features from the student model And the vertical class distribution characteristics of the student model Get feature level class distribution map And feature vertical class distribution map The specific calculation formula is as follows: Wherein, C is the total number of categories in the human body parsing dataset, and exp(·) is the natural exponent; For the result level class distribution diagram Q h And the resulting vertical class distribution graph Q v , the specific calculation formula is as follows: Wherein, the matrix M is generated by one-hot encoding of the real human body analysis result y, H and W are the height and width of the matrix M, h∈[1,H], w∈[1,W], c∈[1,C]; The student model logit distillation loss L kd-logit The calculation formula is as follows: in, and Respectively represent the probability that the teacher model and the student model predict that the pixel p belongs to category i; The student model hierarchical boundary distillation loss L kd-edge The calculation formula is as follows: Among them, MSE represents the MSE loss function, and the teacher model edge graph corresponding to each level and the student model marginal graph The specific calculation formula is as follows: Among them, G x and G y There are two Sobel convolution kernels, which are used to calculate the horizontal and vertical gradients of the hierarchical features; The student model hierarchical global distillation loss L kd-global The calculation formula is as follows: Where H′(i) and W′(i) represent the height and width of the i-th layer category probability map, respectively, and KL(·) represents the KL divergence. and Respectively represent the hierarchical features from the teacher model using the softmax function and student model hierarchical features Generate the corresponding hierarchical category probability map, that is, the category probability of the xth pixel in the i-th layer of the student model and the teacher model; Teacher model hierarchical category probability map The specific calculation formula is: Student model hierarchical category probability map The specific calculation formula is: The student model knowledge distillation loss L KD The calculation formula is: L KD =α1L kd-Logit +α2L kd-edge +α3L kd-global , where α1, α2, and α3 are hyperparameters used to balance the losses of each part; The overall training loss of the student model is L total The calculation formula is: L total =β1L parsing +β2L KD , Among them, β1 and β2 are hyperparameters used to balance the losses of each part.

Images