A human body parsing method based on human body posture estimation and related device

Abstract

The application discloses a human body analysis method based on human body posture estimation and related devices, the method comprises the following steps: obtaining a human key point map of a to-be-analyzed image; inputting the human key point map and the to-be-analyzed image into a human body analysis module in a trained human body analysis model, and extracting a human body analysis feature map through the human body analysis module; inputting the human body analysis feature map into a pixel decoder in the human body analysis model, and extracting a plurality of first feature maps through the pixel decoder; determining a human body part segmentation map corresponding to the to-be-analyzed image based on the plurality of first feature maps and a Transformer decoder in the human body analysis model. The application combines human body posture estimation and human body analysis, uses the obtained human key point map as prior knowledge of human body analysis, makes the Transformer know different types of human key points in the to-be-analyzed image, and improves the accuracy of human body analysis.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a human body parsing method and related apparatus based on human pose estimation. Background Technology

[0002] Human body segmentation, also known as human semantic segmentation, is a hot research area in computer vision. It uses semantic segmentation techniques to perform pixel-level classification of images or videos centered on the human body. Human body segmentation divides a captured person into different fine-grained semantic parts, such as the head, torso, arms, and legs. As a more fine-grained semantic segmentation task, it is more challenging than human body segmentation based on finding the human outline. It also has many applications in real life, such as pedestrian re-identification, human behavior analysis, image editing, and virtual reality.

[0003] Current methods for human body segmentation employ graph convolutional networks to model the topological relationships of human body parts for segmentation. However, this method cannot resolve the boundary confusion problem inherent in human body segmentation graphs, and the inference of graph features consumes significant computational time. Summary of the Invention

[0004] The technical problem to be solved by this application is to provide a human body analysis method and related device based on human posture estimation, in order to address the shortcomings of the existing technology.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a human body analysis method based on human pose estimation, the method comprising:

[0006] Obtain the human body key point map of the image to be parsed;

[0007] The human body key point map and the image to be analyzed are input into the human body analysis module in the trained human body analysis model, and the human body analysis feature map is extracted through the human body analysis module.

[0008] The human body analysis feature map is input into the pixel decoder in the human body analysis model, and several first feature maps are extracted by the pixel decoder.

[0009] Based on the aforementioned first feature maps and the Transformer decoder in the human body parsing model, the human body part segmentation map corresponding to the image to be parsed is determined.

[0010] The human body parsing method based on human pose estimation, wherein each human key point in the human key point map carries semantic information reflecting the type of human key point.

[0011] The human body parsing method based on human pose estimation, wherein obtaining the human keypoint map of the image to be parsed specifically includes:

[0012] The image to be analyzed is input into the human key point module of the trained human key point model, and the human key point feature map is determined by the human key point module.

[0013] The human body key point feature map is input into the pixel decoder in the human body key point model, and several second feature maps are determined by the pixel decoder.

[0014] Based on several second feature maps and the left confidence decoder, right confidence decoder and identity decoder in the human key point model, the left confidence map, right confidence map and associated feature map are determined;

[0015] The left confidence map, right confidence map, and associated feature map are input into the fusion module of the human key point model, and the human key point map of the image to be analyzed is determined by the fusion module.

[0016] The human body analysis method based on human pose estimation includes a left confidence map comprising a first head confidence map, a first neck confidence map, and a left body position confidence map; and a right confidence map comprising a second head confidence map, a second neck confidence map, and a right body position confidence map. The head confidence map is the mean of the first head confidence map and the second head confidence map, and the neck confidence map is the mean of the first neck confidence map and the second neck confidence map.

[0017] The human parsing method based on human pose estimation includes a Transformer decoder, which comprises a cross-attention module, a self-attention module, and a feedforward network.

[0018] The human body parsing method based on human pose estimation includes a human body parsing module comprising a human body keypoint branch and a human body parsing branch. The human body keypoint branch includes at least a first decoding unit and a second decoding unit. The output of the first decoding unit serves as the input of the feature extraction unit in the first decoding stage of the human body parsing branch and the input of the second decoding unit. The output of the second decoding unit serves as the input of the feature extraction unit in the second decoding stage of the human body parsing branch.

[0019] The human body parsing method based on human pose estimation, wherein determining the human body part segmentation map corresponding to the image to be parsed based on the plurality of first feature maps and the Transformer decoder in the human body parsing model specifically includes:

[0020] A portion of the first feature maps from several first feature maps are input into a Transformer decoder, and the feature embedding is output by the Transformer decoder. The portion of the first feature maps does not include the first feature map with the largest size among the several first feature maps.

[0021] Based on the feature embedding, a part category map is determined;

[0022] The mask image is determined based on feature embedding and the first feature map with the largest size;

[0023] Based on the body part category map and the mask image, a human body part segmentation map corresponding to the image to be parsed is determined.

[0024] A second aspect of this application provides a human body analysis device based on human pose estimation, the device comprising:

[0025] The acquisition module is used to acquire the human key point map of the image to be parsed;

[0026] The control module is used to input the human key point map and the image to be parsed into the human parsing module of the trained human parsing model, and extract human parsing feature maps through the human parsing module; input the human parsing feature maps into the pixel decoder in the human parsing model, and extract several first feature maps through the pixel decoder; and determine the human body part segmentation map corresponding to the image to be parsed based on the several first feature maps and the Transformer decoder in the human parsing model.

[0027] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the human body parsing method based on human pose estimation as described above.

[0028] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory;

[0029] The memory stores a computer-readable program that can be executed by the processor;

[0030] When the processor executes the computer-readable program, it implements the steps in any of the above-described human body parsing methods based on human pose estimation.

[0031] Beneficial Effects: Compared with existing technologies, this application provides a human body parsing method and related apparatus based on human pose estimation. The method includes acquiring a human keypoint map of an image to be parsed; inputting the human keypoint map and the image to be parsed into a human body parsing module in a trained human body parsing model, and extracting human body parsing feature maps through the human body parsing module; inputting the human body parsing feature maps into a pixel decoder in the human body parsing model, and extracting several first feature maps through the pixel decoder; and determining a human body part segmentation map corresponding to the image to be parsed based on the several first feature maps and a Transformer decoder in the human body parsing model. This application combines human pose estimation with human body parsing, using the acquired human keypoint map as prior knowledge for human body parsing, enabling the Transformer to recognize different types of human keypoints in the image to be parsed, thus improving the accuracy of human body parsing. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart of the human body parsing method based on human pose estimation provided in this application.

[0034] Figure 2 A flowchart illustrating one implementation of the human body parsing method based on human pose estimation provided in this application.

[0035] Figure 3 This is an example diagram of key points on the human body.

[0036] Figure 4 The structural principle diagram of the human body analysis device based on human pose estimation provided in this application.

[0037] Figure 5 A schematic diagram of the terminal device provided in this application. Detailed Implementation

[0038] This application provides a human body analysis method and related apparatus based on human pose estimation. To make the purpose, technical solution, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0039] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0040] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0041] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0042] The inventors discovered through research that human body analysis, also known as human semantic segmentation, is a hot research area in computer vision. It uses semantic segmentation techniques to perform pixel-level classification of images or videos centered on the human body. Human body analysis segments a captured person in an image into different fine-grained semantic parts, such as the head, torso, arms, and legs. As a more fine-grained semantic segmentation task, it is more challenging than human body segmentation that focuses on finding the human outline. It also has many applications in real life, such as pedestrian re-identification, human behavior analysis, image editing, and virtual reality.

[0043] Current methods for human body segmentation employ graph convolutional networks to model the topological relationships of human body parts for segmentation. However, this method cannot resolve the boundary confusion problem inherent in human body segmentation graphs, and the inference of graph features consumes significant computational time.

[0044] To address these issues, some researchers have applied the Transformer to human body analysis, leveraging its superior image segmentation capabilities to improve efficiency and generate human figures that better match human expectations. However, the inventors discovered that when using the Transformer for human body analysis, the inability to capture key point information leads to problems such as confusion between left and right body parts, incorrect segmentation of small human figures, or misclassifying the background as a human body, thus compromising the accuracy of human body analysis.

[0045] Based on this, in this embodiment, a human keypoint map of the image to be parsed is obtained; the human keypoint map and the image to be parsed are input into the human parsing module of a trained human parsing model, and the human parsing module extracts human parsing feature maps; the human parsing feature maps are input into the pixel decoder of the human parsing model, and the pixel decoder extracts several first feature maps; based on the several first feature maps and the Transformer decoder in the human parsing model, a human body segmentation map corresponding to the image to be parsed is determined. This embodiment combines human pose estimation with human parsing, using the obtained human keypoint map as prior knowledge for human parsing, enabling the Transformer to know different types of human keypoints in the image to be parsed, thus improving the accuracy of human parsing.

[0046] The application content will be further explained below with reference to the accompanying drawings and through the description of the embodiments.

[0047] This embodiment provides a human body parsing method based on human pose estimation, such as... Figure 1 As shown, the method includes:

[0048] S10. Obtain the human body key point map of the image to be parsed.

[0049] Specifically, the image to be parsed is a human body image, meaning the image contains at least one human body. A human keypoint map is used to reflect the types of human keypoints in the image to be parsed. Each human keypoint in the map represents a body part. The human keypoint map can be a heatmap of human keypoints or a categorized heatmap. In a categorized heatmap, each human keypoint carries semantic information, which reflects the keypoint type of the human keypoint.

[0050] In this embodiment, the human body key point map is a typified heat map. The typified heat map improves the prior knowledge of the Transformer, enabling the Transformer to learn the location information and type information of the human body key points (i.e., the body part type corresponding to the human body key points). This not only solves the problem of small human body and background prediction errors, but also avoids confusion between left and right parts, thus improving the accuracy of human body parsing.

[0051] In this embodiment, the semantic information of each human keypoint in the human keypoint map can be expressed as color values, and the color values ​​carried by each human keypoint are different, so as to learn the type information of the human keypoint by learning the color values. Furthermore, the color values ​​of the human keypoints in the human keypoint map can be independent of each other, or there can be a correspondence between the human keypoints on the left side of the body and the human keypoints on the right side of the body in the human keypoint map. For example, such as... Figure 3 As shown in the human body keypoint diagram at the bottom, the color values ​​of the keypoints are all different, and the differences in color values ​​between the keypoints are maximized. Figure 3 As shown in the human body keypoint diagram at the top, the color value of the human body keypoints on the right side of the body is half the color value of the human body keypoints on the left side of the body. The correlation between the left and right body parts is represented by the correspondence between the human body keypoints on the left and right sides of the body in the keypoint diagram. Of course, in practical applications, the relationship between the color values ​​of various human body keypoints can also be represented in other ways, and other methods can also be used to represent the type information between human body keypoints, such as depth values.

[0052] The human keypoint map can be obtained through human pose estimation, such as OpenPose. However, OpenPose may fail to estimate small or rare human poses. Therefore, this application provides a human keypoint model for bottom-up human pose estimation. The human keypoint model includes a human keypoint module, a pixel decoder, a left confidence decoder, a right confidence decoder, an identity decoder, and a fusion module. The human keypoint module is connected to the pixel decoder, which is connected to the left confidence decoder, right confidence decoder, and identity decoder. The left confidence decoder, right confidence decoder, and identity decoder are all connected to the fusion module. The human keypoint module is used to extract human keypoint feature maps. The pixel decoder is used to determine the pixel-level embedding corresponding to the human keypoint feature map. The left confidence decoder is used to determine the left confidence map, the right confidence decoder is used to determine the right confidence map, and the identity decoder is used to determine the associated feature map, which reflects the human body to which the human keypoint belongs.

[0053] The fusion module is used to fuse the left confidence map, the right confidence map, and the associated feature map to obtain the human body key point map.

[0054] Based on this, such as Figure 2 As shown, obtaining the human key point map of the image to be parsed specifically includes:

[0055] S11. Input the image to be parsed into the human key point module in the trained human key point model, and determine the human key point feature map through the human key point module.

[0056] S12. Input the human body key point feature map into the pixel decoder in the human body key point model, and determine a number of second feature maps through the pixel decoder;

[0057] S13. Based on several second feature maps and the left confidence decoder, right confidence decoder and identity decoder in the human key point model, determine the left confidence map, right confidence map and associated feature map;

[0058] S14. Input the left confidence map, right confidence map and associated feature map into the fusion module in the human key point model, and determine the human key point map of the image to be analyzed through the fusion module.

[0059] Specifically, in step S11, the human key point module is the backbone network of the human key point model. The human key point module extracts features from the image to be analyzed to obtain the human key point feature map. The human key point module can be a network module with feature extraction function, such as Swin Transformer.

[0060] In step S12, the pixel decoder maintains the spatial information of the human body and provides feature images to the left-confidence decoder, right-confidence decoder, and identity decoder. The pixel decoder can employ a multi-scale deformable attention transformer, etc. In this embodiment, the pixel decoder can provide multi-scale feature images to the left-confidence decoder, right-confidence decoder, and identity decoder. That is, the plurality of second feature maps are multiple second feature maps with different image scales. For example, the plurality of second feature maps are four second feature maps, and the image sizes of the four second feature maps are respectively... and They are denoted as F0, F1, F2 and F3 respectively, where H represents the image height of the image to be parsed and W represents the image width of the image to be parsed.

[0061] In step S13, the left-confidence decoder, right-confidence decoder, and identity decoder all include a Transformer decoder. The Transformer decoder includes a cross-attention module, a self-attention module, and a feedforward network. The Transformer decoder performs cross-attention on each of the several second feature maps except for the largest-sized second feature map, so as to obtain the attention relationship between different human key points calculated by the Transformer decoder from the second feature maps of different scales, and then fuses it with the largest-sized second feature map.

[0062] The number of queries for both the left-confidence decoder and the right-confidence decoder is The identity decoder queries N, where N is the number of human body keypoints. That is, the left confidence graph includes confidence graphs for J human body keypoints, and the right confidence graph includes confidence graphs for J human body keypoints. The left confidence graph includes a first head confidence graph, a first neck confidence graph, and confidence graphs for each left side of the body. The right confidence graph includes a second head confidence graph, a second neck confidence graph, and confidence graphs for each right side of the body. The head confidence graph is the average of the first and second head confidence graphs, and the neck confidence graph is the average of the first and second neck confidence graphs. For example, the left confidence graph is represented as... The right confidence plot is represented as If i∈(2,3,...J-1), then the head confidence graph S head and neck confidence plot S neck They can be represented as follows:

[0063]

[0064]

[0065] Furthermore, the working processes of the left confidence decoder and the right confidence decoder are the same, that is, the determination processes of the left confidence map and the right confidence map are the same. Here, we will use the left confidence map and several second feature maps, including four second feature maps F0, F1, F2, and F3, as an example for explanation. The determination process of the left confidence map specifically includes:

[0066] 1) Input F0, F1, F2 and J queries to the left confidence decoder. First, calculate the cross-attention in the left confidence decoder:

[0067]

[0068] Where l represents the subscript of the layer, X l Q represents the query feature of the l-th layer. l V represents the query input at level l. l and Kl This indicates that the image features input at layer l are transformed by two different linear transformation functions f. v and f k The transformed matrix, X l-1 This represents the query characteristics of the (l-1)th layer.

[0069] 2) The results of cross-attention are normalized and passed through a self-attention layer. The result calculated by the self-attention layer will be normalized and then passed through a feedforward layer to output the final query feature. The Transformer decoder performs decoding operations on F0, F1, and F2, repeating these three decoding operations L times, for a total of 3L decoding operations. The decoded image will then be passed through a multilayer perceptron to generate a C... ε *J-dimensional embedding features C l C ε Indicates the number of channels.

[0070] 3) Embed feature C l The candidate left confidence map is fused with the second feature map of the largest size to obtain a candidate left confidence map. Then, the candidate left confidence map is upsampled to obtain the final left confidence map. Here, the candidate left confidence map S′... l It can be represented as:

[0071] Furthermore, the identity decoder and the left-confidence decoder operate in essentially the same way, the difference being that the identity decoder performs N queries and generates C embedding vectors. ε *The N-dimensional embedding features I and the associated feature map can be represented as L=(L0,L1,...,L...). N-1 ).

[0072] In step S14, after obtaining the left confidence map, right confidence map, and associated feature map, the body confidence map S = (S0, S1, ..., S...) can be determined based on the left and right confidence maps. N-1 Then, based on the body confidence map and the associated feature map, a complete human pose can be formed, which is called a human key point map. The associated feature map is used to represent the human key point labels, and the body confidence map is used to represent the confidence level of the human key points.

[0073] Furthermore, after the human body key point map, when the human body key points in the human body key point map carry semantic information that reflects the key point type, the human body key points in the human body key point map will also draw semantic information in a preset manner, wherein the preset manner is the semantic information representation form described in the above embodiment.

[0074] S20. Input the human body key point map and the image to be parsed into the human body parsing module in the trained human body parsing model, and extract the human body parsing feature map through the human body parsing module.

[0075] Specifically, the human body parsing model is a trained neural network model used to determine the human body segmentation map corresponding to the image to be parsed. The human body parsing model includes a human body parsing module, a pixel decoder, and a Transformer decoder. The human body parsing module is connected to the pixel decoder, and the pixel decoder is connected to the Transformer decoder. The human body parsing module is used to extract human body parsing feature maps based on the human body keypoint map and the image to be parsed. The human body parsing module can use a DeepLabV3+ backbone network or the backbone network of other network models to directly extract features from the fused image obtained by fusing the human body keypoint map and the image to be parsed; alternatively, the human body parsing module includes a human body keypoint branch and a human body parsing branch. The human body keypoint branch decodes the human body keypoint map, and the human body parsing branch decodes the image to be parsed. Then, the feature maps obtained from both decoding are fused to extract the human body parsing feature map. Of course, in practical applications, the human body parsing module can also use other network structures to extract human body parsing feature maps.

[0076] In this embodiment of the application, the human body parsing module includes a human body key point branch and a human body parsing branch. The human body key point branch includes at least a first decoding unit and a second decoding unit. The output of the first decoding unit serves as the input of the feature extraction unit in the first decoding stage of the human body parsing branch and the input of the second decoding unit. The output of the second decoding unit serves as the input of the feature extraction unit in the second decoding stage of the human body parsing branch.

[0077] Furthermore, the human body parsing branch can employ image segmentation decoders such as Mask2Former and Swin-Transformer, forming an UpperNet framework with the human keypoint branch and the human body parsing branch. The human keypoint branch interacts with at least the first two decoding stages of the human body parsing branch. This embodiment illustrates this using an Swin-Transformer decoder for the human body parsing branch, with the human keypoint branch interacting with the human body parsing branch twice. Figure 2As shown, the human keypoint branch includes a first decoding unit and a second decoding unit. The first decoding unit includes a PartPartition layer and a Linear Embedding layer. The second decoding unit includes a Part Merging layer. The output of the Linear Embedding layer in the first decoding unit is input into the Part Merging layer and an adder, respectively. The adder then merges the output of the Linear Embedding layer in the first decoding stage of the human body parsing branch with the output of the Linear Embedding layer. The resulting fused feature is input into the Swin-T block in the first decoding stage. The output of the Part Merging layer in the second decoding unit is input into an adder. The adder then merges the output of the Linear Embedding layer in the second decoding stage of the human body parsing branch with the output of the Linear Embedding layer. The resulting fused feature is input into the Swin-T block in the second decoding stage.

[0078] This embodiment of the application divides the human body key point map into segments using a segmentation layer. Each image patch is then mapped to dimension C to generate a human keypoint map as feature P. t Similarly, the human body parsing branch maps the image to be parsed into feature P. i Then feature P t and feature P i Pixel-level adder fusion is performed to obtain fused feature P f =P i +P t The color values ​​of each human keypoint in the human keypoint map are linearly embedded to obtain feature P, which contains both positional and semantic features of the human keypoints. t Then feature P t and feature P i Fusion yields P f , making P f Prior knowledge of key human body points can effectively guide human body segmentation. Then, through fusion in the second decoding stage, more guiding information can be obtained, thereby further improving the accuracy of human body segmentation.

[0079] In practical applications, fusion can be performed only in the first decoding stage, or it can be performed in the same way as the second decoding stage in subsequent decoding stages such as the third and fourth decoding stages to further improve segmentation accuracy. This application has found that the guidance information obtained after fusion in the second decoding stage can effectively guide the segmentation of human body parts. Therefore, to balance segmentation accuracy and efficiency, this application's embodiments employ a two-stage interaction between the first and second decoding stages.

[0080] S30. Input the human body analysis feature map into the pixel decoder in the human body analysis model, and extract several first feature maps through the pixel decoder.

[0081] Specifically, the pixel decoder in the human body parsing model can be the same as the pixel decoder in the human body keypoint model, which will not be elaborated here. Please refer to the description of the pixel decoder in the human body keypoint model for details. Correspondingly, the image sizes of the various first feature maps in the several first feature maps are also different.

[0082] S40. Based on the aforementioned first feature maps and the Transformer decoder in the human body parsing model, determine the human body part segmentation map corresponding to the image to be parsed.

[0083] Specifically, the human body segmentation map is the segmentation result obtained by segmenting the human body region in the image to be parsed according to human body parts. The human body segmentation map can only carry the part regions of each human body part, or it can carry both the part regions and part categories. The input of the Transformer decoder is all the first feature maps except the largest first feature map and N query vectors. The Transformer decoder calculates the attention relationship between different human body parts from the first feature maps of different scales, and obtains a C*N dimensional embedding vector, where C represents the dimension. Then, the category of each body part is determined based on the embedding vector, and the body mask map is determined based on the embedding vector and the largest first feature map. Finally, the human body segmentation map is determined based on the category of each body part and the body mask map. The process of the Transformer decoder determining the embedding vector is the same as the Transformer decoder in the left confidence decoder, so it will not be described again here.

[0084] Based on this, in one implementation of this embodiment, the human body part segmentation map includes human body part regions and human body part categories. Accordingly, determining the human body part segmentation map corresponding to the image to be parsed based on the plurality of first feature maps and the Transformer decoder in the human body parsing model specifically includes:

[0085] S41. Input a portion of the first feature maps from a plurality of first feature maps into a Transformer decoder, and output the feature embedding through the Transformer decoder, wherein the portion of the first feature maps does not include the first feature map with the largest size among the plurality of first feature maps;

[0086] S42. Determine the part category map based on the feature embedding;

[0087] S43. Determine the mask image based on feature embedding and the first feature map with the largest size;

[0088] S44. Based on the part category map and the mask image, determine the human body part segmentation map corresponding to the image to be parsed.

[0089] Specifically, the body part category map reflects the body part category of each human body part. This category map can be obtained through a classifier, which embeds features into the classifier and determines the body part category to obtain the category map. The mask image reflects the region of each human body part. The mask image is determined by performing an inner product of the embedded features and the first feature map at the largest scale, resulting in N H*W dimensional binary semantic segmentation maps. H and W represent the height and width of the binary semantic segmentation map, respectively. Each map represents the parsing result of a human body part, with the corresponding pixel value represented by 1 and other pixel values ​​represented by 0. The N binary semantic segmentation maps are then fused, replacing the 1s in the binary images with the labels of the corresponding segmented body parts. The N replaced images are then summed to obtain the final human body parsing result.

[0090] Of course, it is worth noting that the human body key point model and human body analysis model in the embodiments of this application can be used separately or combined into a whole model.

[0091] In summary, this embodiment provides a human body parsing method and related apparatus based on human pose estimation. The method includes acquiring a human keypoint map of an image to be parsed; inputting the human keypoint map and the image to be parsed into a human parsing module in a trained human body parsing model, and extracting human body parsing feature maps through the human body parsing module; inputting the human body parsing feature maps into a pixel decoder in the human body parsing model, and extracting several first feature maps through the pixel decoder; and determining a human body part segmentation map corresponding to the image to be parsed based on the several first feature maps and the Transformer decoder in the human body parsing model. This application combines human pose estimation with human body parsing, using the acquired human keypoint map as prior knowledge for human body parsing, enabling the Transformer to know different types of human keypoints in the image to be parsed, thereby improving the accuracy of human body parsing.

[0092] To further illustrate the effectiveness of the human body parsing method based on human pose estimation provided in this application, the human keypoint model TRPose provided in this application is compared with OpenPose on CrowdPose. The performance of the pre-trained TRPose was tested on the Pascal-Person-Part and LIP datasets to verify the accuracy of keypoints in the human keypoint map.

[0093] 1) As shown in Table 1, the performance of TRPose in this embodiment is better than OpenPose, especially in medium AP scenarios. M and severe AP H In crowded scenes.

[0094] 2) As shown in Table 2, TRPose, pre-trained on CrowdPose, was tested directly on the Pascal-Person-Part dataset. TRPose achieved better performance with 46.2% mAP. Notably, TRPose outperforms existing models.

[0095] 3) As shown in Table 3, training TRPose on CrowdPose can achieve relatively good results without fine-tuning. It can provide good keypoint estimation for the second stage of human body parsing.

[0096] Table 1 shows the comparison results on the CrowdPose test set.

[0097]

[0098] Table 2 shows the comparison results on the Pascal-person-part validation set.

[0099]

[0100] Table 3 shows the comparison results on the LIP validation set.

[0101]

[0102] Furthermore, this application compares the human body analysis method with existing human body analysis methods, and the comparison results are as follows:

[0103] 1) As shown in Table 4, on the LIP dataset, the human body analysis method provided in this application embodiment improves the mIoU metric by 1.57% compared to Swin-UperNet, reaching 61.26%, and achieves good accuracy in most categories such as hats, gloves, sunglasses, shoe uppers, outerwear, and socks. Guided by human keypoint maps, the human body analysis method TKFormer provided in this application embodiment significantly improves the accuracy of left and right body parts, such as left arm (+2.36%), right arm (+1.96%), left leg (+5.70%), right leg (+5.81%), left foot (+5.93%), and right foot (+5.69%). Furthermore, TKFormer can locate different types of clothing, reducing erroneous predictions and improving performance, such as shoe uppers (+0.49%) and socks (+0.65%).

[0104] 2) As shown in Table 5, on the Pascal-Person-Part dataset, the method provided in this application embodiment improves the mIoU by 2.56% compared to the HHP method, reaching 77.22%. Based on this, TKFormer-B and TKFormer-L improve by approximately 1.0% and 0.5%, respectively, with TKFormer-B averaging 77.69%, including improvements in the upper arm (+0.92%), forearm (+1.27%), upper limb (+0.73%), and lower limb (+0.39%).

[0105] Table 4 shows the results on the LIP dataset.

[0106]

[0107] Table 5 shows the results on the Pascal-Person-Part dataset.

[0108]

[0109] Based on the above-described human body parsing method based on human pose estimation, this embodiment provides a human body parsing device based on human pose estimation, such as... Figure 4 As shown, the device includes:

[0110] The acquisition module 100 is used to acquire the human body key point map of the image to be parsed;

[0111] The control module 200 is used to input the human body key point map and the image to be parsed into the human body parsing module in the trained human body parsing model, and extract human body parsing feature maps through the human body parsing module; input the human body parsing feature maps into the pixel decoder in the human body parsing model, and extract several first feature maps through the pixel decoder; and determine the human body part segmentation map corresponding to the image to be parsed based on the several first feature maps and the Transformer decoder in the human body parsing model.

[0112] Based on the above-described human body parsing method based on human pose estimation, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the human body parsing method based on human pose estimation as described in the above embodiment.

[0113] Based on the aforementioned human body parsing method based on human pose estimation, this application also provides a terminal device, such as... Figure 5As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0114] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0115] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0116] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0117] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A human body analysis method based on human pose estimation, characterized in that, The method includes: The image to be analyzed is input into the human key point module of the trained human key point model, and the human key point feature map is determined by the human key point module. The human key point feature map is input into the pixel decoder in the human key point model. Several second feature maps are determined by the pixel decoder. Based on the several second feature maps and the left confidence decoder, right confidence decoder and identity decoder in the human key point model, the left confidence map, right confidence map and associated feature map are determined. The left confidence map, right confidence map, and associated feature map are input into the fusion module of the human key point model, and the human key point map of the image to be analyzed is determined by the fusion module. The human body key point map and the image to be analyzed are input into the human body analysis module in the trained human body analysis model, and the human body analysis feature map is extracted through the human body analysis module. The human body analysis feature map is input into the pixel decoder in the human body analysis model, and several first feature maps are extracted by the pixel decoder. A portion of the first feature maps from several first feature maps are input into a Transformer decoder, and the feature embedding is output by the Transformer decoder. The portion of the first feature maps does not include the first feature map with the largest size among the several first feature maps. The region category map is determined based on the feature embedding, and the mask image is determined based on the feature embedding and the first feature map with the largest size. Based on the body part category map and the mask image, a human body part segmentation map corresponding to the image to be parsed is determined.

2. The human body analysis method based on human pose estimation according to claim 1, characterized in that, Each human key point in the human key point diagram carries semantic information that reflects the type of human key point.

3. The human body analysis method based on human pose estimation according to claim 1, characterized in that, The left confidence graph includes a first head confidence graph, a first neck confidence graph, and a left side body position confidence graph; the right confidence graph includes a second head confidence graph, a second neck confidence graph, and a right side body position confidence graph. The head confidence graph is the average of the first head confidence graph and the second head confidence graph, and the neck confidence graph is the average of the first neck confidence graph and the second neck confidence graph.

4. The human body analysis method based on human pose estimation according to claim 1, characterized in that, The left-confidence decoder, right-confidence decoder, and identity decoder all include a Transformer decoder, which includes a cross-attention module, a self-attention module, and a feedforward network.

5. The human body analysis method based on human pose estimation according to claim 1, characterized in that, The human body analysis module includes a human body key point branch and a human body analysis branch. The human body key point branch includes at least a first decoding unit and a second decoding unit. The output of the first decoding unit serves as the input of the feature extraction unit in the first decoding stage of the human body analysis branch and the input of the second decoding unit. The output of the second decoding unit serves as the input of the feature extraction unit in the second decoding stage of the human body analysis branch.

6. A human body analysis device based on human pose estimation, characterized in that, The device includes: The acquisition module is used to input the image to be parsed into the human keypoint module of a trained human keypoint model, and determine the human keypoint feature map through the human keypoint module; input the human keypoint feature map into the pixel decoder in the human keypoint model, and determine several second feature maps through the pixel decoder; and determine a left confidence map, a right confidence map, and an associated feature map based on the several second feature maps and the left confidence decoder, right confidence decoder, and identity decoder in the human keypoint model; input the left confidence map, right confidence map, and associated feature map into the fusion module in the human keypoint model, and determine the human keypoint map of the image to be parsed through the fusion module. A control module is configured to input the human keypoint map and the image to be parsed into the human parsing module of a trained human parsing model, and extract human parsing feature maps through the human parsing module; input the human parsing feature maps into the pixel decoder in the human parsing model, and extract several first feature maps through the pixel decoder; input a portion of the first feature maps into a Transformer decoder, and output feature embeddings through the Transformer decoder; determine a body part category map based on the feature embeddings; determine a mask image based on the feature embeddings and the largest first feature map; and determine a human body part segmentation map corresponding to the image to be parsed based on the body part category map and the mask image; wherein, the portion of the first feature maps does not include the largest first feature map among the several first feature maps.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the human body parsing method based on human pose estimation as described in any one of claims 1-5.

8. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the human body parsing method based on human pose estimation as described in any one of claims 1-5.

Images