Instance-category feature association learning method for multi-person analysis
Through the example-category feature association learning method, multi-scale features are extracted in combination with deep residual networks and feature pyramid networks, and the example-category feature association module is constructed, which solves the problem of inaccurate multi-person analysis results in the existing technology, and achieves more efficient and accurate multi-person analysis effects.
Patent Information
- Application Number
- CN202411666209.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-03-11
AI Technical Summary
The existing single-stage multi-person analysis method fails to fully combine multi-scale information of the human body and human instance-category representation, resulting in inaccurate multi-person analysis results.
The example-category feature association learning method is adopted to extract multi-scale features through deep residual networks and feature pyramid networks, feature coding is performed by combining the deformable attention mechanism, and an example-category feature association module is constructed for feature association and fusion.
It improves the accuracy and efficiency of multi-person analysis, effectively solves the problem of separation of examples and category features, and improves the analytical accuracy of small-scale clothing accessories and human body parts.
Smart Images
Figure CN119600651A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an instance-category feature association learning method for multi-person analysis, and belongs to the field of computer vision and image graphics. Background Art
[0002] Multi-person parsing is one of the human-centric computer vision tasks, which aims to segment the human instances involved into different body parts and clothing accessories. It is widely used in downstream computer vision tasks such as action recognition, pedestrian tracking, virtual try-on and human-computer interaction. However, due to the complex human instances and category features in multi-person images, the different clothing accessories of different human instances, and the complex interactions between multiple human instances, multi-person parsing still faces challenges such as instance and category feature separation, small-scale clothing and accessories loss, and mutual occlusion between instances. Existing known methods consider adopting a two-stage (top-down and bottom-up) strategy. The top-down method detects all human instances first and then parses a single human instance. The bottom-up method identifies all human parts first and then groups them into different human instances. However, since the two-stage method usually requires pre-detection or post-grouping processing, human parsing is divided into two stages, which leads to the separation of instance and category features and the lack of direct supervision of the parsing results. The single-stage method can avoid the above problems, can process multi-person images faster and improve the reasoning speed and efficiency of the model. For example, Chu (<ACMInternational Conference on Multimedia> , 2023, pp. 1863-1873) uses a single-stage approach for multi-person pose estimation to decompose human body parsing into two tasks: localizing the human body and classifying human body parts using point set offsets. Dai (<IEEE Transactions on Multimedia> ,2023,1384-1394) divides the parsing pipeline into instance-aware kernel generation and part-aware human parsing, which are responsible for instance separation and instance-specific part segmentation, respectively, and can eliminate pre-detection or post-grouping processing. Chu(<Transactions on Image Processing> ,2024) proposed a unified and fast method and designed different joint optimization processes to obtain the final parsing results. Although the above single-stage method can quickly parse all human instances in multi-person scenes, it lacks the processing of multi-scale features of the human body and ignores the complex interactions between instances, resulting in inaccurate multi-person parsing results.
[0003] In summary, the present invention designs an instance-category feature association learning method for multi-person analysis to improve the accuracy and efficiency of multi-person analysis, in order to address the problems that the existing known single-stage methods fail to fully combine the multi-scale information of the human body and the instance-category representation of the human body, have insufficient modeling of the relationship between human instance-categories, and ignore the influence of human posture representation on multi-person analysis, resulting in inaccurate multi-person analysis results. Summary of the invention
[0004] In order to solve the above problems, the present invention provides an instance-category feature association learning method for multi-person analysis to meet the current accuracy requirements for human body analysis in multi-person scenes.
[0005] The technical solution of the present invention is: an instance-category feature association learning method for multi-person analysis, comprising the following steps:
[0006] Step 1: Input a multi-person image set and use the deep residual network to perform category kernel K cate Initialize, and combine the feature pyramid network and deformable attention mechanism to perform multi-scale feature encoding to obtain the multi-scale feature F of the human body neck ;
[0007] Step 2: By constructing an instance-category feature representation module including a center head, an instance head, and a category head, we can obtain instance centroid heat maps. Posture feature F pose , instance feature F ins and the category feature F cate ;
[0008] Step 3: Construct an instance-category feature association module for joint instance feature association learning and category feature association learning, and perform instance centroid heatmap Posture feature F pose and instance feature F ins , and the category feature F cate and the category kernel K cate Perform instance-category feature association and learn the instance feature cosine similarity graph M ins And the category feature cosine similarity graph M cate ;
[0009] Step 4: Calculate the direct index to obtain the instance feature cosine similarity graph M ins And the category feature cosine similarity graph M cate Perform instance-category feature fusion to finally generate refined multi-person parsing results.
[0010] The specific process of Step 1 is as follows:
[0011] First, input a multi-person image set D, use a deep residual network to extract human feature vectors and group them by category to obtain a human body part category feature set Z cate ={Z1,Z2,Z3,...,Z n}, where n represents the number of feature sets. cate Initialize the feature vector in and get the category kernel K cate ;
[0012] Secondly, based on the deep residual network, the human body hierarchical features C = {C2, C3, C4, C5} are extracted, and the human body hierarchical features are fused layer by layer using the feature pyramid network to obtain the basic human body features F base ;
[0013] Then, the human body part encoding set is defined as It includes upper body parts such as head, upper arms, torso, etc., lower body parts such as thighs, feet, etc., a total of 11 human body parts; the human joint point coding set is defined as It includes symmetrical joints such as elbows, wrists, knees, etc., and asymmetrical joints such as chest, spine, hips, etc., a total of 17 human joints; the clothing accessories code set is defined as These include head accessories such as hats and glasses, hand accessories such as gloves and watches, upper body accessories such as ties and coats, and lower body accessories such as pants and shoes, a total of 47 types of clothing accessories.
[0014] Finally, the deformable attention mechanism is introduced to construct 6 stacked deformable encoders (the 1st and 2nd encoders process the features of the corresponding clothing accessories encoding set, the 3rd and 4th encoders process the features of the corresponding human body parts encoding set, and the 5th and 6th encoders process the features of the corresponding human joint point encoding set) to obtain the basic features of the human body F base Perform multi-scale encoding to obtain features of human body parts Human joint features Japanese clothing accessories Multi-scale features of human body F neck ;
[0015] The specific process of Step 3 is as follows:
[0016] First, the posture features and instance features are normalized and fused to obtain the human instance feature F ins , the process is defined as: where g IFB (.) represents a 5-layer convolution function, and C×H×W represents the channel dimension;
[0017] Then, the human instance feature F ins and human body center of gravity heat map The instance coordinates in Perform instance feature association learning, the process is defined as: in The WHERE function is used to identify values exceeding θ c , N ins Indicates the number of index coordinates, in Q ins Select a specific instance feature cosine similarity graph M ins , specifically expressed as Where S(·) represents the normalization operation of cosine similarity, θ ins is the set threshold, I(·) is used as the indicator function;
[0018] Finally, the shape of the initialized category kernel is defined as Where N cate represents the number of categories in the image set, and C is the number of channels output by the previous layer. cate With the category kernel K cate Perform category feature association learning, the process is defined as: In Q cate Select the cosine similarity graph M of a specific category feature cate , specifically expressed as Where S(·) represents the normalization operation of cosine similarity, θ cate is the set threshold, and I(·) is used as the indicator function.
[0019] The beneficial effects of the present invention are:
[0020] 1. The known methods usually adopt a two-stage strategy of first detection and then analysis or first analysis and then grouping for multi-person image analysis, ignoring the separation problem of instance and category features, that is, processing instance and category features in different forms respectively, resulting in limited analysis effect. The present invention is based on a single-stage strategy, first extracting multi-scale features of the human body, then defining them as shared semantic features, and finally obtaining instance centroid heat map, posture features, instance features and category features by constructing a unified representation module of instance and category features, effectively solving the problem of instance and category feature separation in the two-stage strategy, and improving the efficiency and accuracy of multi-person analysis.
[0021] 2. The known methods only extract the basic features of the human body through simple up-and-down sampling operations in multi-scale feature learning, ignoring the complex structure of the human body instance, resulting in insufficient representation of the multi-scale features of the human body. The present invention defines the human body instance as three types of coding sets: human body parts, joints and clothing accessories, and combines the feature pyramid network and deformable attention mechanism for multi-scale feature coding. Finally, the multi-scale features of the human body including human body part features, joint features and clothing accessories features are obtained, which improves the parsing accuracy of small-scale clothing accessories and human body parts.
[0022] 3. The known methods only focus on the intrinsic connection of the part categories in a single human instance, ignoring the association between different human instances, resulting in problems such as mutual occlusion of instances in multi-person scenes. At the same time, since the complex posture of the human instance is not considered, it is difficult to achieve high-precision multi-person analysis. The present invention combines single-stage multi-person posture estimation, infers the whole body posture information of the human instance, and constructs an instance-category feature association module that combines instance feature association learning and category feature association learning, which can effectively distinguish human instances and improve the accuracy of multi-person analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 is a flow chart of the present invention;
[0024] Figure 2 It is a multi-scale feature flow chart of the present invention;
[0025] Figure 3 A flow chart showing features of the present invention;
[0026] Figure 4 It is a feature fusion flow chart of the present invention;
[0027] Figure 5 is a posture estimation result diagram of the present invention;
[0028] Figure 6 This is a diagram of the multi-person analysis results of the present invention. DETAILED DESCRIPTION
[0029] Example 1: Figure 1-Figure 6 As shown, a method for learning instance-category feature association for multi-person parsing includes the following steps:
[0030] Step 1: Input a multi-person image set and use the deep residual network to perform category kernel K cate Initialize, and combine the feature pyramid network and deformable attention mechanism to perform multi-scale feature encoding to obtain the multi-scale feature F of the human body neck ;
[0031] Step 2: By constructing an instance-category feature representation module including a center head, an instance head, and a category head, we can obtain instance centroid heat maps. Posture feature F pose , instance feature F ins and the category feature F cate ;
[0032] Step 3: Construct an instance-category feature association module for joint instance feature association learning and category feature association learning, and perform instance centroid heatmap Posture feature F poseand instance feature F ins , and the category feature F cate and the category kernel K cate Perform instance-category feature association and learn the instance feature cosine similarity graph M ins And the category feature cosine similarity graph M cate ;
[0033] Step 4: Calculate the direct index to obtain the instance feature cosine similarity graph M ins And the category feature cosine similarity graph M cate Perform instance-category feature fusion to finally generate refined multi-person parsing results.
[0034] The specific process of Step 1 is as follows:
[0035] First, input a multi-person image set D, use a deep residual network to extract human feature vectors and group them by category to obtain a human body part category feature set Z cate ={Z1,Z2,Z3,...,Z n}, where n represents the number of feature sets. cate Initialize the feature vector in and get the category kernel K cate ;
[0036] Secondly, based on the deep residual network, the human body hierarchical features C = {C2, C3, C4, C5} are extracted, and the human body hierarchical features are fused layer by layer using the feature pyramid network to obtain the basic human body features F base ;
[0037] Then, the human body part encoding set is defined as It includes upper body parts such as head, upper arms, torso, etc., lower body parts such as thighs, feet, etc., a total of 11 human body parts; the human joint point coding set is defined as It includes symmetrical joints such as elbows, wrists, knees, etc., and asymmetrical joints such as chest, spine, hips, etc., a total of 17 human joints; the clothing accessories code set is defined as These include head accessories such as hats and glasses, hand accessories such as gloves and watches, upper body accessories such as ties and coats, and lower body accessories such as pants and shoes, a total of 47 types of clothing accessories.
[0038] Finally, the deformable attention mechanism is introduced to construct 6 stacked deformable encoders (the 1st and 2nd encoders process the features of the corresponding clothing accessories encoding set, the 3rd and 4th encoders process the features of the corresponding human body parts encoding set, and the 5th and 6th encoders process the features of the corresponding human joint point encoding set) to obtain the basic features of the human body F base Perform multi-scale encoding to obtain features of human body parts Human joint features Japanese clothing accessories Multi-scale features of human body F neck ;
[0039] The specific process of Step 3 is as follows:
[0040] First, the posture features and instance features are normalized and fused to obtain the human instance feature F ins , the process is defined as: where g IFB (.) represents a 5-layer convolution function, and C×H×W represents the channel dimension;
[0041] Then, the human instance feature F ins and human body center of gravity heat map The instance coordinates in Perform instance feature association learning, the process is defined as: in The WHERE function is used to identify values exceeding θ c , N ins Indicates the number of index coordinates, in Q ins Select a specific instance feature cosine similarity graph M ins , specifically expressed as Where S(·) represents the normalization operation of cosine similarity, θ ins is the set threshold, I(·) is used as the indicator function;
[0042] Finally, the shape of the initialized category kernel is defined as Where N cate represents the number of categories in the image set, and C is the number of channels output by the previous layer. cate With the category kernel K cate Perform category feature association learning, the process is defined as: In Q cate Select the cosine similarity graph M of a specific category feature cate , specifically expressed as Where S(·) represents the normalization operation of cosine similarity, θ cate is the set threshold, and I(·) is used as the indicator function.
[0043] Example 2: Figure 1 As shown in FIG. 1 , a method for learning instance-category feature association for multi-person parsing is provided. The specific steps of the method are as follows:
[0044] Step 1: First, input a multi-person image set D, use a deep residual network to extract human feature vectors and group them by category to obtain a human body part category feature set Z. cate ={Z1,Z2,Z3,...,Z n}, where n represents the number of feature sets. cate Initialize the feature vector in and get the category kernel K cate ;
[0045] Secondly, if Figure 2 As shown in the figure, based on the ResNet-101 deep residual network, the human body hierarchical features C = {C2, C3, C4, C5} are extracted, and the human body hierarchical features are fused layer by layer using the FPN feature pyramid network to obtain the basic human body features F base ;
[0046] Then, the human body part encoding set is defined as It includes upper body parts such as head, upper arms, torso, etc., lower body parts such as thighs, feet, etc., a total of 11 human body parts; the human joint point coding set is defined as It includes symmetrical joints such as elbows, wrists, knees, etc., and asymmetrical joints such as chest, spine, hips, etc., a total of 17 human joints; the clothing accessories code set is defined as These include head accessories such as hats and glasses, hand accessories such as gloves and watches, upper body accessories such as ties and coats, and lower body accessories such as pants and shoes, a total of 47 types of clothing accessories.
[0047] Finally, the deformable attention mechanism is introduced to construct 6 stacked deformable encoders (the 1st and 2nd encoders process the features of the corresponding clothing accessories encoding set, the 3rd and 4th encoders process the features of the corresponding human body parts encoding set, and the 5th and 6th encoders process the features of the corresponding human joint point encoding set) to obtain the basic features of the human body F base Perform multi-scale encoding to obtain features of human body parts Human joint features Japanese clothing accessories Multi-scale features of human body F neck ;
[0048] Step 2, such as Figure 3 As shown in the figure, firstly, we build an instance-category feature unified representation module including the center head, instance head and category head. Considering the sparsity of the center of gravity of human instances, we neck The size is adjusted, and then 5 convolutional layers with 3*3 kernels are used in series to obtain the instance centroid heat map. The process is represented as Where Sigmoid represents the activation function.
[0049] Then, the instance head and the category head are not changed in F neck In the case of size, after 5 convolutional layers with 3*3 kernels, the instance feature F is obtained. ins , and the posture feature F pose It is obtained through the posture feature generator and normalized;
[0050] Finally, in the category head, the multi-scale features of the human body F neck Directly input into 5 convolutional layers with 3*3 kernels, and normalize these features to maintain the feature channel dimension to obtain the category feature F cate ;
[0051] Step 3: First, normalize the posture features and instance features and fuse them to obtain the human instance feature F. ins , the process is defined as:
[0052] where g IFB (.) represents a 5-layer convolution function, and C×H×W represents the channel dimension;
[0053] Then, the human instance feature F ins and human body center of gravity heat map The instance coordinates in Perform instance feature association learning, the process is defined as: in The WHERE function is used to identify values exceeding θ c , N ins Indicates the number of index coordinates, in Q ins Select a specific instance feature cosine similarity graph M ins , specifically expressed as Where S(·) represents the normalization operation of cosine similarity, θ ins is the set threshold, I(·) is used as the indicator function;
[0054] Finally, the shape of the initialized category kernel is defined as Where N cate represents the number of categories in the image set, and C is the number of channels output by the previous layer. cate With the category kernel K cate Perform category feature association learning, the process is defined as: In Q cate Select the cosine similarity graph M of a specific category feature cate , specifically expressed as Where S(·) represents the normalization operation of cosine similarity, θ cateis the set threshold, and I(·) is used as the indicator function.
[0055] Step 4, such as Figure 4 As shown, first, from the instance feature cosine similarity graph M ins In each position (x, y), select the best matching instance feature and obtain the instance feature index by calculating the maximum value of the cosine similarity The process is defined as: Among them, M ins (i,x,y) is the cosine similarity between instance i and the feature at position (x,y), and arg max represents the index selection function.
[0056] Then, from the category feature cosine similarity graph M cate In each position (x, y), select the best matching category feature and obtain the category feature index by calculating the maximum value of the cosine similarity The process is defined as: Among them, M cate (j,x,y) is the cosine similarity between the features of category j and position (x,y), and arg max represents the index selection function.
[0057] Finally, the corresponding instance and category feature vectors are selected from the instance feature matrix and the category feature matrix, which are specifically expressed as and And perform instance-category feature fusion, which is defined as Where γ represents the weight parameter, and R(x,y) is the final refined multi-person parsing result.
[0058] Figure 5 and Figure 6 This is a diagram of multi-person pose estimation and multi-person parsing results in the MHP v2.0 dataset. It can be seen from the figure that the method of the present invention has a good parsing effect on multi-person images. It can not only effectively distinguish multiple human body instances, but also accurately parse the small-scale parts and clothing accessories of each human body instance.
[0059] The method of the present invention is compared with other known multi-person parsing methods such as M-CE2P, RP-RCNN, UniParser, etc. on the public dataset MHP v2.0, and the results are shown in Table 1. It refers to the average accuracy of the prediction results when the overlap between the predicted human body parts and the real parts is greater than or equal to 50%. B is evaluated at different IoU thresholds (e.g., from 50% to 95%), and then the average precision, PCP, is calculated. 50It means the percentage of correctly parsed parts when the IoU between the predicted human body parts and the real parts is greater than or equal to 50%. If the values of the three evaluation indicators are all high, the results are better, which reflects the accuracy of the final results of the present invention.
[0060] Table 1 Method performance comparison
[0061]
[0062] It can be seen from Table 1 that the method of the present invention and PCP 50 They are all higher than the five known methods, reaching 52.0%, 54.9% and 53.1% respectively, that is, the accuracy is relatively high.
[0063] In summary, compared with the known methods, the method of the present invention has certain robustness for small-scale parts and clothing accessories, and has higher analytical accuracy.
[0064] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A method for learning instance-category feature association for multi-person parsing, characterized by: The steps include: Step 1: Input a multi-person image set, use a deep residual network to initialize the category kernel, and combine the feature pyramid network and deformable attention mechanism to perform multi-scale feature encoding to obtain multi-scale features of the human body; Step 2: By constructing an instance-category feature representation module including a center head, an instance head, and a category head, the instance centroid heat map, posture feature, instance feature, and category feature are obtained respectively; Step 3, build an instance-category feature association module for joint instance feature association learning and category feature association learning, perform instance-category feature association on instance centroid heat map, posture feature and instance feature, as well as category feature and category kernel, and learn to obtain instance feature cosine similarity map and category feature cosine similarity map; Step 4: By calculating the direct index, the instance feature cosine similarity map and the category feature cosine similarity map are fused with instance-category features to finally generate a refined multi-person parsing result.
2. The instance-category feature association learning method for multi-person analysis according to claim 1, characterized in that: The specific process of Step 1 is as follows: First, input a multi-person image set D, use a deep residual network to extract human feature vectors and group them by category to obtain a human body part category feature set Z cate ={Z1,Z2,Z3,...,Z n }, where n represents the number of feature sets, and the feature set Z cate Initialize the feature vector in and get the category kernel K cate ; Secondly, based on the deep residual network, the human body hierarchical features C = {C2, C3, C4, C5} are extracted, and the human body hierarchical features are fused layer by layer using the feature pyramid network to obtain the basic human body features F base ; Then, the human body part encoding set is defined as The human joint point encoding set is defined as The clothing accessories code set is defined as Finally, the deformable attention mechanism is introduced. By constructing 6 deformable encoders stacked in sequence, the 1st and 2nd encoders process the features of the corresponding clothing accessories encoding set, the 3rd and 4th encoders process the features of the corresponding human body parts encoding set, and the 5th and 6th encoders process the features of the human joint point encoding set. base Perform multi-scale encoding to obtain features of human body parts Human joint features Japanese clothing accessories Multi-scale features of human body F neck。 3. The instance-category feature association learning method for multi-person analysis according to claim 1, characterized in that: The specific process of Step 3 is as follows: First, the posture features and instance features are normalized and fused to obtain the human instance feature F ins , the process is defined as: where g IFB (.) represents a 5-layer convolution function, and C×H×W represents the channel dimension; Then, the human instance feature F ins and human body center of gravity heat map The instance coordinates in i∈{1,...,N c }Perform instance feature association learning, the process is defined as: in The WHERE function is used to identify values exceeding θ c , N ins Indicates the number of index coordinates, in Q ins Select a specific instance feature cosine similarity graph M ins , specifically expressed as Where S(·) represents the normalization operation of cosine similarity, θ ins is the set threshold, I(·) is used as the indicator function; Finally, the shape of the initialized category kernel is defined as Where N cate represents the number of categories in the image set, C is the number of channels output by the previous layer, and the category feature F cate With the category kernel K cate Perform category feature association learning, the process is defined as: In Q cate Select the cosine similarity graph M of a specific category feature cate , specifically expressed as Where S(·) represents the normalization operation of cosine similarity, θ cate is the set threshold, and I(·) is used as the indicator function.