A human body parsing method and system based on knowledge distillation

Abstract

The application discloses a human body analysis method based on knowledge distillation, comprising the following steps: performing data enhancement and processing on a human body image and an analysis image, generating edge labels, posture features, rough analysis features and refined analysis features by using an edge encoder, a posture encoder, an analysis encoder and a heterogeneous non-local module respectively, calculating a total analysis loss and a distillation loss of a student model by using the analysis features, wherein the total analysis loss and the distillation loss jointly constitute an overall loss function of the student model, training and optimizing student model parameters by using the overall loss function, and outputting a trained student model and a final analysis prediction image. The application further discloses a human body analysis system based on knowledge distillation. The application combines knowledge distillation and human body analysis, fully utilizes prior information such as human body posture and edges of the human body analysis task, and maximally improves the analysis speed of the model under the premise of ensuring accuracy by using the method of knowledge distillation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, knowledge distillation and human parsing, and particularly relates to a human parsing method and system based on knowledge distillation. BACKGROUND

[0002] In recent years, the Internet technology has developed rapidly. As a bridge for computers to understand the real world, the performance of computer vision tasks has been greatly improved due to the successful application of deep learning related technologies. Computer vision involves multiple specific research fields and tasks. Semantic segmentation solves the classification and positioning of objects at the same time, can identify the specific type of the problem and accurately locate the position of the object, belongs to the pixel-level semantic understanding problem, and is the core technology of many practical applications.

[0003] In the field of computer vision, semantic segmentation is an important basis for computer image understanding, which is mainly pixel-level classification of images, that is, pixel-by-pixel labeling of images belonging to a class. The segmentation result directly affects the subsequent image processing tasks. In practical application scenarios, human-centered image analysis and processing has become a mainstream research direction. Human parsing is a fine-grained human semantic segmentation, which aims to classify the human body parts and clothing accessories in images or videos at the pixel level, and divide the human body into different semantic parts, which is one of the basic tasks of human-centered visual understanding. Since human parsing plays a crucial role in a wide range of application fields such as security monitoring, autonomous driving, fashion analysis, visual special effects, and artistic creation, people have conducted a large number of researches on it, which has given birth to various excellent human parsing solutions and applications.

[0004] Some human parsing models have good performance, but have problems such as large amount of calculation, long training time, etc., and cannot have fast calculation speed, making it difficult to apply to small devices such as smartphones. The knowledge distillation method based on semantic segmentation task can compress the parameter amount and improve the processing speed of the network without reducing the parsing performance of the entire model, so that it can be applied to semantic segmentation tasks in small devices such as smartphones. However, since the human body has prior information such as posture and edge, the knowledge distillation method based on general semantic segmentation task cannot fully utilize these prior information, and may not be able to fully improve the accuracy of the student model under the premise of ensuring real-time performance.

[0005] One of the current prior arts is a human parsing method combining human pose and edge information proposed by Z. Zhang et al. in "Correlating Edge, Pose With Parsing", which is composed of a parsing encoder, a pose encoder, an edge encoder and a heterogeneous non-local module. A large amount of prior knowledge is used to capture rich structural knowledge. The three encoders respectively calculate the feature representations of human semantic parsing, pose and edge. The heterogeneous non-local module mixes and fuses the three features to obtain a refined parsing feature representation, thereby obtaining the final human parsing result. The disadvantage of this method is that although the parsing result is more accurate by combining edge information and pose information, a large amount of additional prior data is needed to train the model, which introduces additional computational complexity, increases the complexity of the program and the complexity of the calculation, and cannot guarantee real-time performance.

[0006] The second current prior art is a self-correcting human parsing strategy proposed by P. Li et al. in "Self-Correction for Human Parsing" to gradually improve the reliability of labels and models during training. This method is divided into two sub-processes: model aggregation and label optimization. The whole process starts from a model trained with noisy labels as the initial model. A cyclic training strategy is designed to iteratively aggregate the current model and the previous optimal model to infer more reliable pseudo labels. The corrected labels can be used to improve the robustness and accuracy of the model. The disadvantage of this method is that although a more accurate model and optimized human parsing labels can be obtained, the method is a cyclic iterative training process that needs to simultaneously iterate and optimize the model and the labels, which requires a long training time and a large amount of iterative optimization calculation, and is not suitable for direct application to real-time human parsing scenarios.

[0007] The third prior art is a normalized distillation method for semantic segmentation proposed by T. Liu et al. in “Normalized Feature Distillation for Semantic Segmentation”. The method first obtains the segmentation probability maps generated by the teacher model and the student model. Secondly, the features of the teacher model and the student model are normalized, and the L2 distance between the normalized features of the teacher model and the student model is minimized. During the training process, the training function is composed of three parts: the cross-entropy loss between the prediction of the student model and the real human pose label; the KL loss of the segmentation probability maps generated by the teacher model and the student model; and the L2 distance loss of the normalized features of the teacher model and the student model. The disadvantage of this method is that since this method is a knowledge distillation method for general semantic segmentation tasks, and in the human parsing task, due to the semantic structure characteristics of the human body, directly using the knowledge distillation method for general semantic segmentation tasks cannot fully utilize the human prior knowledge such as human pose information and edge information, and thus the ideal accuracy cannot be achieved. SUMMARY

[0008] The purpose of the present application is to overcome the shortcomings of the prior art and propose a human parsing method and system based on knowledge distillation. The main problems solved by the present application are: 1) how to fully utilize the human pose and other structural information in the knowledge distillation method to improve the effect of knowledge distillation. 2) how to solve the problem that the existing knowledge distillation method has large computational complexity and low efficiency, and is not suitable for large-scale popularization in commercial applications with high real-time requirements.

[0009] To solve the above problems, the present application proposes a human parsing method based on knowledge distillation, which comprises:

[0010] The human parsing dataset is used to perform data enhancement on the human body images and parsing images therein and process them into a unified size to obtain a human parsing dataset after data preprocessing, wherein the human parsing dataset includes real human pose labels and real human parsing results;

[0011] The data preprocessing human parsing dataset is used to construct teacher models and student models with the same structure and different sizes, including generating edge features, pose features and rough parsing features by using an edge encoder, a pose encoder and a parsing encoder, then using a heterogeneous non-local module to construct the correlation between the human edge features, the pose features and the parsing features, and obtaining refined parsing features by using the key context clues provided by the edge and the pose, to obtain the rough parsing features, the pose features, the edge features, the refined parsing features, the predicted pixel class probability and the final parsing prediction result of the teacher model and the student model.

[0012] obtaining an edge label by finding a boundary of a mask between two different semantic parts in the real human parsing result, and then calculating a total parsing loss of the final parsing prediction result and the real human parsing result, the coarse parsing feature of the student model and the real human parsing result, the pose feature of the student model and the real human pose label, and the edge feature of the student model and the edge label, and recording the total parsing loss as a total parsing loss of the student model;

[0013] calculating a distillation loss of predicted pixel class probabilities of the teacher model and the student model, a coarse parsing feature normalization distillation loss of the teacher model and the student model, a pose feature normalization distillation loss of the teacher model and the student model, an edge feature normalization distillation loss of the teacher model and the student model, and an intra-class similarity distillation loss of refined parsing features of the teacher model and the student model, thereby composing a distillation loss of the student model guided by the teacher model;

[0014] combining the total parsing loss of the student model and the distillation loss of the student model guided by the teacher model to jointly constitute an overall loss function of the student model, training and optimizing the student model parameters by using the overall loss function of the student model, and finally outputting a trained student model and a final parsing prediction image.

[0015] Preferably, the human parsing data set after the data preprocessing is used to construct a teacher model and a student model with the same structure but different sizes, which includes generating edge features, pose features, and coarse parsing features by using an edge encoder, a pose encoder, and a parsing encoder, then constructing the correlation among the edge features, the pose features, and the parsing features by using a heterogeneous non-local module, obtaining refined parsing features by using key context clues provided by the edge and the pose, and obtaining the coarse parsing features, the pose features, the edge features, the refined parsing features, the predicted pixel class probabilities, and the final parsing prediction result of the teacher model and the student model, specifically as follows:

[0016] The teacher model and the student model are both composed of three feature encoders and a feature mixing module, wherein the teacher model is a high-precision large-scale human parsing model that has been trained, and the student model is a small-scale human parsing model that needs to be trained. The network parameters of the student model are iteratively optimized according to the training process without changing the network parameters of the teacher model.

[0017] The backbone network of the teacher model and the student model is Res50 and Res18 respectively, three feature encoders are analytic encoder, pose encoder and edge encoder, the analytic encoder is used to generate rough analytic features; the pose encoder is used to generate pose features; the edge encoder is used to generate edge features; the generated edge features, pose features and rough analytic features are input into a feature mixing module for mixing and fusion to obtain final fine analytic features, and finally the final fine analytic features are used to obtain the final analytic prediction result.

[0018] Preferably, the edge label is obtained by finding the boundary of the mask between two different semantic parts in the real human body analytic result, and then the total analytic loss of the final analytic prediction result, the rough analytic features of the student model and the real human body analytic result, the pose features of the student model and the real human body pose label, and the edge features of the student model and the edge label is calculated, and is recorded as the total analytic loss of the student model, and specifically is:

[0019] In the process of training the student model, the total analytic loss L of the student model loss function is composed of analytic supervision, human key point position and semantic edge information parsing , the total analytic loss L parsing is composed of four parts, and specifically is:

[0020] L parsing =L p2 +L p +λ1L b +λ2L k +λ3L consistent

[0021] Wherein, λ1, λ2, λ3 are adjustable hyperparameters for balancing the three loss terms, L p2 is the cross entropy loss between the final analytic prediction result and the real human body analytic result; L p is the cross entropy loss between the analytic prediction obtained by the rough analytic features and the real human body analytic result; L b is the weighted cross entropy loss between the edge graph obtained by the edge features and the edge label; L k is the mean square error loss between the body joint prediction obtained by the regression heat map of the pose features and the real human body pose label; L consistent is used to explicitly maintain the consistency of the analytic prediction and the edge prediction, and the expression is specifically:

[0022]

[0023] Wherein, e nis an edge map obtained by using edge features of the student model, is an edge map generated by using final analytic prediction results of the student model, N represents the number of all pixels, N + represents the number of edge pixels, and only positive edge pixels n∈N + are involved in the calculation of the loss function.

[0024] Preferably, the distillation loss of the predicted pixel class probability of the teacher model and the student model, the coarse analytic feature normalization distillation loss of the teacher model and the student model, the pose feature normalization distillation loss of the teacher model and the student model, the edge feature normalization distillation loss of the teacher model and the student model, and the refined analytic feature intra-class similarity distillation loss of the teacher model and the student model are calculated, thereby composing the distillation loss of the student model guided by the teacher model, specifically:

[0025] The distillation loss L Distillation of the student model guided by the teacher model is composed of four parts, and the calculation formula is as follows:

[0026] L Distillation = αL kd + βL nfpd + γL nfkd + μL ifvd + θL nfbd

[0027] Wherein, α, β, γ, μ, θ are adjustable hyperparameters, L kd is the predicted pixel class probability distillation loss, L nfpd is the coarse analytic feature normalization distillation loss, L nfkd is the pose feature normalization distillation loss, L ifvd is the refined analytic feature intra-class similarity distillation loss, L nfbd is the edge feature normalization distillation loss.

[0028] The calculation formula of the predicted pixel class probability distillation loss L kd is as follows:

[0029]

[0030] Wherein, H' and W' represent the height and width of the prediction map respectively, and represent the class probability of the i-th pixel generated by the student model and the teacher model respectively, KL(·) represents the KL divergence;

[0031] The calculation formula of the coarse analytic feature normalization distillation loss L nfpd is as follows:

[0032]

[0033] wherein, and denote the coarse analytical features of the teacher model and the student model, respectively, Norm(·) denotes a normalization operation, and D(·) denotes an L2 distance, wherein C is the number of channels, B is the batch size, H' and W' are the height and width of the feature map, respectively, the features are normalized along the (H', W') dimension, and the calculation formula is as follows:

[0034]

[0035] wherein, F denotes the original feature, and μ and σ denote the mean and standard deviation of the feature, respectively;

[0036] The calculation formula of the pose feature normalization distillation loss L nfkd is as follows:

[0037]

[0038] wherein, and denote the pose features obtained by the teacher model and the student model, respectively, Norm(·) denotes a normalization operation, and D(·) denotes an L2 distance;

[0039] The calculation formula of the edge feature normalization distillation loss L nfbd is as follows:

[0040]

[0041] wherein, and denote the edge features obtained by the teacher model and the student model, respectively, Norm(·) denotes a normalization operation, and D(·) denotes an L2 distance;

[0042] The calculation formula of the fine analytical feature intra-class similarity distillation loss L ifvd is as follows:

[0043]

[0044] wherein, N denotes the number of pixels, and denote the intra-class feature similarity of pixel i of the teacher model and the student model, respectively;

[0045] The method for calculating the intra-class feature vector associated with the intra-class feature similarity corresponding to the refined analytical features of the student model and the teacher model comprises: firstly, averaging the features of all pixels with the same class label to obtain a prototype of each class; and then, calculating the cosine similarity between the feature of each pixel and the prototype of the class corresponding to the pixel to obtain the intra-class feature similarity of the pixel, and the calculation formula of the class feature vector is specifically as follows:

[0046]

[0047] wherein, M i represents the intra-class feature similarity of pixel i, f p2i represents the refined analytical feature corresponding to pixel i, S i represents the set of all pixels with the same class label as pixel i, |S i represents the size of the set S i , and Cos(·) represents a cosine similarity function.

[0048] Correspondingly, the application also provides a human parsing system based on knowledge distillation, comprising:

[0049] a data processing unit, which is configured to perform data enhancement on human images and parsing images in a human parsing dataset and process the human images and the parsing images into a unified size to obtain a human parsing dataset after data preprocessing, wherein the human parsing dataset comprises real human pose labels and real human parsing results;

[0050] a model analysis unit, which is configured to construct a teacher model and a student model with the same structure and different sizes by using the human parsing dataset after data preprocessing, and comprises an edge encoder, a pose encoder and a parsing encoder configured to generate edge features, pose features and rough parsing features, a heterogeneous non-local module configured to construct the correlation among the edge features, the pose features and the parsing features, and an edge and pose configured to provide key context clues to obtain refined parsing features, so as to obtain the rough parsing features, the pose features, the edge features, the refined parsing features, the predicted pixel class probability and the final parsing prediction result of the teacher model and the student model;

[0051] an analysis loss calculation unit, which is configured to find the boundary of a mask between two different semantic parts in the real human parsing result to obtain an edge label, and then calculate the total analysis loss of the final parsing prediction result and the real human parsing result, the rough parsing features of the student model and the real human parsing result, the pose features of the student model and the real human pose label, and the edge features of the student model and the edge label, and record the total analysis loss as the total analysis loss of the student model;

[0052] a distillation loss calculation unit, which calculates a distillation loss of predicted pixel class probabilities of the teacher model and the student model, a coarse parsing feature normalization distillation loss of the teacher model and the student model, a pose feature normalization distillation loss of the teacher model and the student model, an edge feature normalization distillation loss of the teacher model and the student model, and a refined parsing feature intra-class similarity distillation loss of the teacher model and the student model, so as to constitute a distillation loss of the student model guided by the teacher model;

[0053] a parsing prediction unit, which combines a total parsing loss of the student model and the distillation loss of the student model guided by the teacher model to jointly constitute an overall loss function of the student model, trains and optimizes student model parameters by using the overall loss function of the student model, and finally outputs a trained student model and a final parsing prediction image.

[0054] The present application has the following beneficial effects:

[0055] The present application ingeniously combines knowledge distillation and human parsing, fully utilizes human pose, edge and other prior information specific to the human parsing task, and maximizes the parsing speed of the model under the premise of ensuring accuracy by using the knowledge distillation method, that is, the present application fully utilizes human pose, edge and other prior information, designs a human parsing method based on knowledge distillation, trains a student model under the guidance of a trained teacher model, and enables a smaller student model to obtain similar accuracy to a larger teacher model, thereby being applied to occasions with higher real-time requirements.

[0056] Compared with a general human parsing method without a knowledge distillation strategy, the method adopted by the present application has the characteristics of fast calculation speed and low model complexity; compared with a method of directly using a self-correction strategy on a human parsing model for model training and label correction, the method adopted by the present application has the advantage of relatively short training practice; compared with a general knowledge distillation method for general semantic analysis tasks, the method adopted by the present application has the characteristics of being more capable of fully utilizing human prior information and being conducive to improving model accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 is a human parsing method flowchart based on knowledge distillation according to an embodiment of the present application;

[0058] Figure 2 is an overall framework diagram of a teacher model and a student model according to an embodiment of the present application;

[0059] Figure 3 is a human parsing system structure diagram based on knowledge distillation according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0061] Figure 1 is a flow chart of a human parsing method based on knowledge distillation according to an embodiment of the present application, as shown in the figure, the method comprises: Figure 1

[0062] S1, using a human parsing dataset, performing data enhancement on human images and parsing images in the human parsing dataset and processing them into a unified size to obtain a human parsing dataset after data preprocessing, the human parsing dataset after data preprocessing comprising real human pose labels and real human parsing results;

[0063] S2, constructing teacher models and student models with the same structure and different sizes using the human parsing dataset after data preprocessing, comprising generating edge features, pose features and rough parsing features using an edge encoder, a pose encoder and a parsing encoder, then constructing the correlation between the human edge features, the pose features and the parsing features using a heterogeneous non-local module, obtaining refined parsing features using key context clues provided by the edge and the pose, to obtain rough parsing features, pose features, edge features, refined parsing features, predicted pixel class probabilities and final parsing prediction results of the teacher models and the student models;

[0064] S3, finding the boundary of a mask between two different semantic parts in the real human parsing results to obtain an edge label, then calculating the total parsing loss of the final parsing prediction results and the real human parsing results, the rough parsing features of the student models and the real human parsing results, the pose features of the student models and the real human pose labels, and the edge features of the student models and the edge label, and recording the total parsing loss as the total parsing loss of the student models;

[0065] S4, calculating the distillation loss of the predicted pixel class probabilities of the teacher models and the student models, the rough parsing feature normalization distillation loss of the teacher models and the student models, the pose feature normalization distillation loss of the teacher models and the student models, the edge feature normalization distillation loss of the teacher models and the student models, and the refined parsing feature intra-class similarity distillation loss of the teacher models and the student models, to form the distillation loss of the student models guided by the teacher models;

[0066] ​S5, combine the total parsing loss of the student model and the distillation loss of the student model guided by the teacher model to form a whole loss function of the student model, train and optimize the student model parameters by using the whole loss function of the student model, and finally output the trained student model and the final parsing prediction image.

[0067] Step S2 is specifically as follows:

[0068] S2-1: The teacher model and the student model are both composed of three feature encoders and a feature mixing module, wherein the teacher model is a high-precision large-scale human parsing model that has been trained, and the student model is a small-scale human parsing model that needs to be trained, so in the following training process, the network parameters of the teacher model do not need to be changed, and the network parameters of the student model need to be iteratively optimized according to the training process.

[0069] The backbone networks of the teacher model and the student model are Res50 and Res18 respectively, and the overall framework diagram of the model is as shown in Figure 2 The three feature encoders are a parsing encoder, a pose encoder and an edge encoder, the rough parsing features are generated through the parsing encoder; the pose features are generated through the pose encoder; the edge features are generated through the edge encoder; the generated edge features, pose features and rough parsing features are input into the feature mixing module for mixing and fusion to obtain the final fine parsing features, and finally the final parsing prediction result is obtained according to the fine parsing features.

[0070] Step S3 is specifically as follows:

[0071] S3-1, in the process of training the student model, the total parsing loss L parsing of the student model loss function is composed of parsing supervision, human key point position and semantic edge information. parsing The total parsing loss L parsing is composed of four parts, specifically:

[0072] L p2 = L p + λ1L b + λ2L k + λ3L consistent

[0073] Wherein, λ1, λ2, λ3 are adjustable hyperparameters for balancing the three losses, L p2 is the cross-entropy loss between the final parsing prediction result and the true human parsing result; L p is the cross-entropy loss between the parsing prediction obtained by the rough parsing features and the true human parsing result; L bis the weighted cross-entropy loss between the edge map obtained by using the edge feature and the edge label; L k is the mean square error loss between the body joint prediction obtained by using the pose feature to obtain the regression heat map and the real human pose label; L consistent is used to explicitly maintain the consistency of the analytical prediction and the edge prediction, and its expression is specifically as follows:

[0074]

[0075] wherein e n is the edge map obtained by using the edge feature of the student model, is the edge map generated by using the final analytical prediction result of the student model, N represents the number of all pixels, N + represents the number of edge pixels, and only positive edge pixels n∈N + participate in the calculation of the loss function.

[0076] Step S4 is specifically as follows:

[0077] S4-1: Obtain the distillation loss L Distillation of the student model guided by the teacher model, which is composed of four parts, and the calculation formula is as follows:

[0078] L Distillation = αL kd + βL nfpd + γL nfkd + μL ifvd + θL nfbd

[0079] wherein α, β, γ, μ, θ are adjustable hyperparameters, L kd is the predicted pixel class probability distillation loss, L nfpd is the coarse analytical feature normalization distillation loss, L nfkd is the pose feature normalization distillation loss, L ifvd is the fine analytical feature intra-class similarity distillation loss, L nfbd is the edge feature normalization distillation loss.

[0080] S4-1-1: The calculation formula of the predicted pixel class probability distillation loss L kd is specifically as follows:

[0081]

[0082] wherein H' and W' respectively represent the height and width of the predicted map, and respectively represent the class probability of the i-th pixel generated by the student model and the teacher model, and KL(·) represents the KL divergence.

[0083] S4-1-2: Coarse resolution feature normalization distillation loss L nfpd The calculation formula is as follows:

[0084]

[0085] wherein, and respectively represent the coarse resolution features of the teacher model and the student model, Norm(·) represents a normalization operation, and D(·) represents an L2 distance, wherein C is the number of channels, B is the batch size, H' and W' are the height and width of the feature map respectively, the feature is normalized along the (H', W') dimension, and the calculation formula is as follows:

[0086]

[0087] wherein, F represents the original feature, and μ and σ respectively represent the mean and standard deviation of the feature.

[0088] S4-1-3: Pose feature normalization distillation loss L nfkd The calculation formula is as follows:

[0089]

[0090] wherein, and respectively represent the pose features obtained by the teacher model and the student model, Norm(·) represents a normalization operation, and D(·) represents an L2 distance.

[0091] S4-1-4: Edge feature normalization distillation loss L nfbd The calculation formula is as follows:

[0092]

[0093] wherein, and respectively represent the edge features obtained by the teacher model and the student model, Norm(·) represents a normalization operation, and D(·) represents an L2 distance.

[0094] S4-1-5: Fine resolution feature intra-class similarity distillation loss L ifvd The calculation formula is as follows:

[0095]

[0096] wherein, N represents the number of pixels, and respectively represent the intra-class feature similarity of pixel i of the teacher model and the student model.

[0097] S4-2: calculate the intra-class feature vector associated with the intra-class feature similarity corresponding to the refined analytical features of the student model and the teacher model, the calculation method comprising: first, average the features of all pixels with the same class label to obtain the prototype of each class, and then calculate the cosine similarity between the features of each pixel and the corresponding class prototype to obtain the intra-class feature similarity of the pixel, and the calculation formula of the class feature vector is as follows:

[0098]

[0099] Wherein, M i represents the intra-class feature similarity of pixel i, f p2i represents the refined analytical feature corresponding to pixel i, S i represents the set of all pixels with the same class label as pixel i, |S i represents the size of the set S i , and Cos(·) represents the cosine similarity function.

[0100] Step S5, specifically as follows:

[0101] S5-1: the overall loss function of the student model consists of two parts, one part is the distillation loss of the student model guided by the teacher model, and the other part is the total analytical loss of the student model.

[0102] Correspondingly, the present application also provides a human body analysis system based on knowledge distillation, as shown in Figure 3 , comprising:

[0103] A data processing unit 1 uses a human body analysis data set to perform data enhancement on human body images and analysis images therein and processes them into a unified size to obtain a data pre-processed human body analysis data set, which includes real human body pose labels and real human body analysis results.

[0104] A model analysis unit 2 constructs teacher models and student models with the same structure and different sizes using the data pre-processed human body analysis data set, including generating edge features, pose features and rough analysis features using edge encoders, pose encoders and analysis encoders, then using a heterogeneous non-local module to construct the correlation between human body edge features, pose features and analysis features, obtaining refined analysis features using key context clues provided by edges and poses, obtaining rough analysis features, pose features, edge features, refined analysis features, predicted pixel class probability and final analysis prediction result of the teacher model and the student model.

[0105] The parsing loss calculation unit 3 obtains an edge label by finding the boundary of the mask between two different semantic parts in the real human parsing result, and then calculates the total parsing loss of the final parsing prediction result and the real human parsing result, the coarse parsing feature of the student model and the real human parsing result, the pose feature of the student model and the real human pose label, and the edge feature of the student model and the edge label, and is recorded as the total parsing loss of the student model;

[0106] The distillation loss calculation unit 4 calculates the distillation loss of the predicted pixel class probability of the teacher model and the student model, the coarse parsing feature normalization distillation loss of the teacher model and the student model, the pose feature normalization distillation loss of the teacher model and the student model, the edge feature normalization distillation loss of the teacher model and the student model, and the fine parsing feature intra-class similarity distillation loss of the teacher model and the student model, thereby forming the distillation loss of the student model guided by the teacher model;

[0107] The parsing prediction unit 5 combines the total parsing loss of the student model and the distillation loss of the student model guided by the teacher model to form the overall loss function of the student model, trains and optimizes the student model parameters by using the overall loss function of the student model, and finally outputs the trained student model and the final parsing prediction image.

[0108] Therefore, the present application ingeniously combines knowledge distillation and human parsing, fully utilizes the human pose, edge and other prior information specific to the human parsing task, and maximizes the parsing speed of the model under the premise of ensuring accuracy by using the knowledge distillation method, that is, the present application fully utilizes the human pose, edge and other prior information, designs a human parsing method based on knowledge distillation, trains the student model under the guidance of the trained teacher model, so that the smaller student model can obtain similar accuracy to the larger teacher model, and is applied to occasions with higher real-time requirements. Compared with the general human parsing method without knowledge distillation strategy, the method adopted by the present application has the characteristics of fast calculation speed and low model complexity; compared with the method of directly using the self-correction strategy on the human parsing model for model training and label correction, the method adopted by the present application has the advantage of relatively short training practice; compared with the general knowledge distillation method for general semantic analysis task, the method adopted by the present application has the characteristics of being more capable of fully utilizing human prior information and being conducive to improving model accuracy.

[0109] The above describes in detail the human body analysis method and system provided by the embodiment of the present application based on knowledge distillation. The principles and implementation manners of the present application are described by using specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A human parsing method based on knowledge distillation, characterized in that, The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises:

2. The human parsing method based on knowledge distillation according to claim 1, wherein, The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The method comprises: The teacher model and the student model are both composed of three feature encoders and a feature mixing module, wherein the teacher model is a high-precision large-scale human parsing model that has been trained, and the student model is a small-scale human parsing model that needs to be trained, and the network parameters of the student model are iteratively optimized according to the training process without changing the network parameters of the teacher model; The main network of the teacher model and the student model is Res50 and Res18 respectively, the three feature encoders are parsing encoders, pose encoders and edge encoders, the rough parsing features are generated through the parsing encoders; the pose features are generated through the pose encoders; the edge features are generated through the edge encoders; the generated edge features, pose features and rough parsing features are input into the feature mixing module for mixed fusion to obtain the final fine parsing features, and finally the final parsing prediction result is obtained according to the final fine parsing features. 3.The knowledge distillation based human parsing method of claim 1, wherein, The edge label is obtained by finding the boundary of the mask between the two different semantic parts in the real human parsing result, and then the total parsing loss of the final parsing prediction result and the real human parsing result, the rough parsing features of the student model and the real human parsing result, the pose features of the student model and the real human pose label, and the edge features of the student model and the edge label are calculated, and the total parsing loss of the student model is recorded, which is specifically: In the process of training the student model, the total analytical loss L of the student model loss function is composed of analytical supervision, human key point position and semantic edge information parsing The total analytical loss L parsing Composed of four parts, specifically: L parsing = L p2 + L p + λ1L b + λ2L k + λ3L consistent wherein λ1, λ2, λ3 are adjustable hyperparameters for balancing the three losses, L p2 is the cross-entropy loss between the final parsing prediction and the ground truth human parsing result; L p is the cross-entropy loss between the parsing prediction obtained using the coarse parsing feature and the ground truth human parsing result; L b is the weighted cross-entropy loss between the edge map obtained using the edge feature and the edge label; L k is the mean square error loss between the body joint prediction obtained using the regression heat map of the pose feature and the ground truth human pose label; L consistent is used to explicitly maintain the consistency of the parsing prediction and the edge prediction, and its expression is specifically as follows: where e n is an edge map obtained using edge features of the student model, is an edge map generated using the final analytical prediction result of the student model, N represents the number of all pixels, N + represents the number of edge pixels, and only positive edge pixels n ∈ N + participate in the calculation of the loss function.

4. The human parsing method based on knowledge distillation according to claim 1, wherein, The distillation loss of the predicted pixel class probability of the teacher model and the student model, the rough parsing feature normalization distillation loss of the teacher model and the student model, the pose feature normalization distillation loss of the teacher model and the student model, the edge feature normalization distillation loss of the teacher model and the student model, and the fine parsing feature intra-class similarity distillation loss of the teacher model and the student model are calculated, thereby forming the distillation loss of the student model guided by the teacher model, which is specifically: obtaining a distillation loss L of the student model guided by the teacher model Distillation The distillation loss consists of four parts, and the calculation formula is as follows: L Distillation = aL kd + bL nfpd + gL nfkd + mL ifvd + 0L nfbd wherein, a, b, g, m, q are adjustable hyperparameters, L kd is the predicted pixel class probability distillation loss, L nfpd is the coarse resolution feature normalization distillation loss, L nfkd is the pose feature normalization distillation loss, L ifvd is the fine resolution feature intra-class similarity distillation loss, L nfbd is the edge feature normalization distillation loss; Predicted pixel class probability distillation loss L kd The calculation formula is as follows: where H' and W' represent the height and width of the prediction map, respectively, and represent the class probabilities of the i-th pixel produced by the student and teacher models, respectively, and KL(·) represents the KL divergence. coarse parsing feature normalization distillation loss L nfpd The calculation formula is as follows: wherein, and respectively represent the coarse analytical features of the teacher model and the student model, Norm(·) represents a normalization operation, D(·) represents an L2 distance, wherein C is a channel number, B is a batch size, H' and W' are respectively a height and a width of a feature map, the features are normalized along the (H', W') dimension, and the calculation formula is specifically as follows: Wherein, F represents the original feature, μ and σ represent the mean and standard deviation of the feature respectively; pose feature normalization distillation loss L nfkd The calculation formula of the pose feature normalization distillation loss L is as follows: wherein, and respectively denote the pose features obtained by the teacher model and the student model, Norm(·) denotes a normalization operation, and D(·) denotes an L2 distance; Edge feature normalization distillation loss L nfbd The calculation formula is as follows: where, and denote the edge features obtained by the teacher model and the student model, respectively, Norm(·) denotes a normalization operation, and D(·) denotes an L2 distance. Fine-grained resolution feature intra-class similarity distillation loss L ifvd The calculation formula is as follows: wherein N represents the number of pixels, and respectively represent the intra-class feature similarity of pixel i of the teacher model and the student model. The intra-class feature vector associated with the intra-class feature similarity of the fine parsing features of the student model and the teacher model is calculated, and the calculation method includes: first, the features of all pixels with the same class label are averaged to obtain the prototype of each class, and then the cosine similarity between the feature of each pixel and its corresponding class prototype is calculated to obtain the intra-class feature similarity of the pixel, and the calculation formula of the class feature vector is as follows: where M i represents the intra-class feature similarity of pixel i, f p2i represents the refined analytical feature corresponding to pixel i, S i represents the set of all pixels with the same class label as pixel i, |S i | represents the size of the set S i , and Cos(·) represents the cosine similarity function. 5.A human parsing system based on knowledge distillation, characterized in that, The system comprises: A data processing unit uses a human parsing dataset to perform data enhancement on human images and parsing images in the human parsing dataset and processes them into a unified size to obtain a human parsing dataset after data preprocessing, wherein the human parsing dataset includes real human pose labels and real human parsing results; The data processing unit uses a human parsing dataset to perform data enhancement on human images and parsing images in the human parsing dataset and processes them into a unified size to obtain a human parsing dataset after data preprocessing, wherein the human parsing dataset includes real human pose labels and real human parsing results; The model analysis unit constructs teacher models and student models with the same structure and different sizes from the human body analysis data set after data preprocessing, including generating edge features, pose features and rough analysis features by using edge encoders, pose encoders and analysis encoders, then using a heterogeneous non-local module to construct the correlation between human body edge features, pose features and analysis features, obtaining refined analysis features by using key context clues provided by edges and poses, obtaining rough analysis features, pose features, edge features, refined analysis features, predicted pixel category probabilities and final analysis prediction results of the teacher models and student models. The analysis loss calculation unit finds the boundary of the mask between two different semantic parts in the real human body analysis result to obtain edge labels, then calculates the total analysis loss of the final analysis prediction result and the real human body analysis result, the rough analysis features of the student model and the real human body analysis result, the pose features of the student model and the real human body pose label, and the edge features of the student model and the edge label, and records it as the total analysis loss of the student model. The distillation loss calculation unit calculates the distillation loss of the predicted pixel category probabilities of the teacher model and the student model, the rough analysis feature normalization distillation loss of the teacher model and the student model, the pose feature normalization distillation loss of the teacher model and the student model, the edge feature normalization distillation loss of the teacher model and the student model, and the refined analysis feature intra-class similarity distillation loss of the teacher model and the student model, thereby forming the distillation loss of the student model guided by the teacher model. The analysis prediction unit combines the total analysis loss of the student model and the distillation loss of the student model guided by the teacher model to form the overall loss function of the student model, trains and optimizes the parameters of the student model by using the overall loss function of the student model, and finally outputs the trained student model and the final analysis prediction image.

6. The knowledge distillation-based human parsing system of claim 5, wherein, The model analysis unit constructs teacher models and student models with the same structure and different sizes from the human body analysis data set after data preprocessing, including generating edge features, pose features and rough analysis features by using edge encoders, pose encoders and analysis encoders, then using a heterogeneous non-local module to construct the correlation between human body edge features, pose features and analysis features, obtaining refined analysis features by using key context clues provided by edges and poses, obtaining rough analysis features, pose features, edge features, refined analysis features, predicted pixel category probabilities and final analysis prediction results of the teacher models and student models. The teacher model and the student model are both composed of three feature encoders and a feature mixing module, wherein the teacher model is a high-precision large-scale human body analysis model that has been trained, and the student model is a small-scale human body analysis model that needs to be trained. Under the premise of not changing the network parameters of the teacher model, the network parameters of the student model are iteratively optimized according to the training process. The backbone network of the teacher model and the student model is Res50 and Res18 respectively, three feature encoders are analytic encoder, pose encoder and edge encoder, the analytic encoder is used to generate rough analytic features; the pose encoder is used to generate pose features; the edge encoder is used to generate edge features; the generated edge features, pose features and rough analytic features are input into a feature mixing module for mixed fusion to obtain final fine analytic features, and finally the final fine analytic features are used to obtain the final analytic prediction result.

7. The human parsing system based on knowledge distillation of claim 5, wherein, The edge label is obtained by finding the boundary of the mask between two different semantic parts in the real human body analytic result, then the total analytic loss of the final analytic prediction result and the real human body analytic result, the rough analytic features of the student model and the real human body analytic result, the pose features of the student model and the real human body pose label, and the edge features of the student model and the edge label are calculated, and the total analytic loss of the student model is recorded, specifically: In the process of training the student model, the total analytical loss L of the student model loss function is composed of analytical supervision, human key point position and semantic edge information parsing The total analytical loss L parsing is composed of four parts, specifically: L parsing = L p2 + L p + λ1L b + λ2L k + λ3L consistent wherein λ1, λ2, λ3 are adjustable hyperparameters for balancing the three losses, L p2 is the cross-entropy loss between the final parsing prediction and the ground truth human parsing result; L p is the cross-entropy loss between the parsing prediction obtained using the coarse parsing feature and the ground truth human parsing result; L b is the weighted cross-entropy loss between the edge map obtained using the edge feature and the edge label; L k is the mean square error loss between the body joint prediction obtained using the regression heat map of the pose feature and the ground truth human pose label; L consistent is used to explicitly maintain the consistency of the parsing prediction and the edge prediction, and its expression is specifically: where e n is an edge map obtained using the edge features of the student model, is an edge map generated using the final analytical prediction results of the student model, N represents the number of all pixels, N + represents the number of edge pixels, and only positive edge pixels n ∈ N + participate in the calculation of the loss function.

8. The knowledge distillation-based human parsing system of claim 5, wherein, The distillation loss of the predicted pixel class probability of the teacher model and the student model, the rough analytic feature normalization distillation loss of the teacher model and the student model, the pose feature normalization distillation loss of the teacher model and the student model, the edge feature normalization distillation loss of the teacher model and the student model, and the fine analytic feature intra-class similarity distillation loss of the teacher model and the student model are calculated, thereby forming the distillation loss of the student model guided by the teacher model, specifically: obtaining a distillation loss L of the student model guided by the teacher model Distillation The distillation loss consists of four parts, and the calculation formula is as follows: L Distillation = aL kd + bL nfpd + gL nfkd + mL ifvd + 0L nfbd wherein, a, b, g, m, q are adjustable hyperparameters, L kd is the predicted pixel class probability distillation loss, L nfpd is the coarse resolution feature normalization distillation loss, L nfkd is the pose feature normalization distillation loss, L ifvd is the fine resolution feature intra-class similarity distillation loss, L nfbd is the edge feature normalization distillation loss; Predicted pixel class probability distillation loss L kd The calculation formula is as follows: where H' and W' represent the height and width of the prediction map, respectively, and respectively represent the class probability of the i-th pixel generated by the student model and the teacher model, and KL(·) represents the KL divergence. coarse parsing feature normalization distillation loss L nfpd The calculation formula is as follows: wherein, and respectively represent the coarse analytical features of the teacher model and the student model, Norm(·) represents a normalization operation, D(·) represents an L2 distance, wherein C is a channel number, B is a batch size, H’ and W’ are respectively a height and a width of a feature map, the features are normalized along the (H’, W’) dimension, and the calculation formula is specifically as follows: Wherein, F represents the original feature, and μ and σ represent the mean and standard deviation of the feature respectively; pose feature normalization distillation loss L nfkd The calculation formula of the pose feature normalization distillation loss L is as follows: wherein, and respectively denote the pose features obtained by the teacher model and the student model, Norm(·) denotes a normalization operation, and D(·) denotes an L2 distance; Edge feature normalization distillation loss L nfbd The calculation formula is as follows: wherein, and denote the edge features obtained by the teacher model and the student model, respectively, Norm(·) denotes a normalization operation, and D(·) denotes an L2 distance. Fine-grained resolution feature intra-class similarity distillation loss L ifvd The calculation formula is as follows: wherein N represents the number of pixels, and respectively represent the intra-class feature similarity of pixel i of the teacher model and the student model. The intra-class feature vector associated with the intra-class feature similarity corresponding to the fine analytic feature of the student model and the teacher model is calculated, and the calculation method comprises: firstly, the features of all pixels with the same class label are averaged to obtain the prototype of each class, and then the cosine similarity between the feature of each pixel and the corresponding class prototype is calculated to obtain the intra-class feature similarity of the pixel, and the calculation formula of the class feature vector is as follows: where M i represents the intra-class feature similarity of pixel i, f p2i represents the refined analytical feature corresponding to pixel i, S i represents the set of all pixels with the same class label as pixel i, |S i | represents the size of the set S i , and Cos(·) represents the cosine similarity function.

Images