Human body posture estimation method and device, processing equipment and storage medium

Abstract

The invention discloses a human body posture estimation method and device, processing equipment and a storage medium. The method comprises the following steps: constructing a first spatial awareness network model SSpose based on an activation maximization AM strategy; wherein the first Spose comprises an attention layer, and the attention layer is used for acquiring spatial correlation between a key point and an image position; based on an auxiliary task, inputting the unlabeled data sample into an auxiliary network for self-supervised training until a predetermined condition is met, and obtaining a trained second SSpose, the auxiliary network being constructed based on an encoder in the first SSpose; the second SSpose is used for human body pose estimation. According to the method, on one hand, the interpretability of the model is enhanced; and on the other hand, large-scale annotation data does not need to be used, so that the workload of data annotation is reduced.

Description

Technical Field

[0001] The present invention relates to, but is not limited to, the field of information processing, and in particular to a method, apparatus, processing device, and storage medium for estimating human posture. Background Art

[0002] Human pose estimation, also known as keypoint detection, is a fundamental yet challenging computer vision task. It aims to accurately infer human pose information from images or videos. This typically involves the localization of keypoints and the description of posture. Accurately estimating human pose enables the understanding and analysis of human movements, postures, and behaviors, playing a vital role in numerous application areas, including medical rehabilitation, motion analysis, and human-computer interaction. However, existing human pose estimation methods suffer from high requirements for labeled data, complex deployment, and limited interpretability. Summary of the Invention

[0003] In view of this, the present invention discloses a human body posture estimation method, apparatus, processing equipment and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for estimating a human body posture is provided, the method comprising:

[0005] Based on the activation maximization AM strategy, a first self-supervised spatial perception network model SSpose is constructed; wherein the first SSpose is capable of obtaining the spatial correlation between key points and image positions;

[0006] Based on the auxiliary task, unlabeled data samples are input into the auxiliary network for self-supervised training until predetermined conditions are met to obtain a trained second SSpose, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation.

[0007] In some embodiments, the first SSpose constructed includes: a hybrid CNN-Transfomer encoder, a multi-scale fusion module and a head structure; wherein the hybrid CNN-Transfomer encoder is used to obtain low-level image features and high-level global dependencies, and the low-level image features include multi-granularity image features; the multi-scale fusion module is used to fuse the obtained multi-granularity image features; the head structure is used to predict the key point heat map using the contributions of all image positions collected by the final attention layer of the Transformer.

[0008] In some embodiments, before inputting the unlabeled data sample into the auxiliary network for self-supervised training based on the auxiliary task until a predetermined condition is met and the trained second SSpose is obtained, the method further includes:

[0009] The convolution block of the first SSpose is replaced by a masked convolution block.

[0010] In some embodiments, the step of inputting unlabeled data samples into the auxiliary network for self-supervised training based on the auxiliary task until predetermined conditions are met to obtain the trained second SSpose includes:

[0011] Based on the auxiliary task and the hierarchical mask strategy, the unlabeled data samples are input into the auxiliary network for self-supervised training until the predetermined conditions are met, and the second SSpose after training is obtained.

[0012] In some embodiments, the method further comprises:

[0013] The labeled data samples are input into the second SSpose after self-supervised training to perform model fine-tuning.

[0014] In some embodiments, the auxiliary task is an image reconstruction task.

[0015] According to a second aspect of an embodiment of the present disclosure, a method for estimating a human body posture is provided, the method comprising:

[0016] Acquire an image to be estimated, wherein the image to be estimated includes key point features for determining a human body posture;

[0017] The image to be estimated is input into the second SSpose for human posture estimation after training for detection to obtain key points; wherein, the second SSpose is based on an auxiliary task and is obtained by inputting unlabeled data samples into an auxiliary network for self-supervised training until predetermined conditions are met, the auxiliary network is constructed based on the encoder in the first SSpose, and the SSpose is constructed based on the AM strategy.

[0018] According to a third aspect of an embodiment of the present disclosure, a human body posture estimation device is provided, the device comprising:

[0019] A construction module is configured to: construct a first spatial perception network model SSpose based on an activation maximization AM strategy; wherein the first SSpose is capable of obtaining spatial correlations between key points and image positions;

[0020] The training module is configured to: based on the auxiliary task, input unlabeled data samples into the auxiliary network for self-supervised training until predetermined conditions are met, and obtain a trained second SSpose, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a processing device is provided, the processing device comprising:

[0022] a memory for storing executable programs;

[0023] The processor is configured to implement any method described in the embodiments of the present disclosure when executing the executable program stored in the memory.

[0024] According to a fifth aspect of the embodiments of the present disclosure, a computer storage medium is provided, wherein the computer storage medium stores an executable program, and when the executable program is executed by a processor, it implements the method described in any one of the embodiments of the present disclosure.

[0025] In the present invention, a first self-supervised spatial perception network model (SSpose) is constructed based on an activation maximization (AM) strategy. The first SSpose includes an attention layer for acquiring spatial correlations between key points and image locations. Based on an auxiliary task, unlabeled data samples are input into the auxiliary network for self-supervised training until predetermined conditions are met, thereby obtaining a trained second SSpose. The auxiliary network is constructed based on the encoder in the first SSpose. The second SSpose is used for human pose estimation. In the present invention, since the first SSpose constructed based on the AM strategy includes an attention layer for acquiring spatial correlations between key points and image locations, the attention layer can be used to acquire spatial correlations between specific key points and various image locations. Thus, the process of constructing the auxiliary network based on the encoder in the first SSpose and training based on the auxiliary task essentially integrates the factors of spatial correlation between key points and image locations. The output of the trained second SSpose is essentially closely related to the spatial correlation between key points and image locations, which enhances the interpretability of the model output compared to a method without an attention layer. In addition, since the second SSpose is a model obtained based on self-supervised training, it does not require the use of large-scale labeled data, thereby reducing the workload of data annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a flow chart of a method for estimating a human body posture according to the first embodiment;

[0027] Figure 2 is a schematic diagram of a model (SSpose) according to the second embodiment;

[0028] Figure 3 is a schematic diagram of a model according to a third embodiment;

[0029] Figure 4 4 is a flow chart of a method for estimating a human body posture according to a fourth embodiment;

[0030] Figure 5 4 is a flow chart of a method for estimating a human body posture according to a fifth embodiment;

[0031] Figure 6 is a schematic diagram of a model according to a sixth embodiment;

[0032] Figure 7 4 is a flow chart of a method for estimating a human body posture according to a seventh embodiment;

[0033] Figure 8 is a schematic diagram of a model according to an eighth embodiment;

[0034] Figure 9 4 is a flow chart of a model training method according to the ninth embodiment;

[0035] Figure 10 Schematic diagram of a masked convolution module and a Transformer module according to the tenth embodiment;

[0036] Figure 11 is a schematic diagram of a model according to the eleventh embodiment;

[0037] Figure 12 1 is a flow chart of a method for estimating a human body posture according to a twelfth embodiment;

[0038] Figure 13 Schematic diagram of a human body posture estimation device according to the thirteenth embodiment.

[0039] Figure 14 Schematic diagram of a human body posture estimation device according to the fourteenth embodiment. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms "first\second\third" are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understandable that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein.

[0043] In the following description, “greater than” and “less than” are involved. It should be noted that in this disclosure, “greater than” can be used to indicate “greater than” or “equal to”; “less than” can be used to indicate “less than” or “equal to”.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0045] In order to better understand the embodiments of the present disclosure, the application scenarios of the technical solutions of the present disclosure are first described:

[0046] In some embodiments, the model is designed based on the CNN-Transformer architecture, but there are the following disadvantages: 1) Limited model interpretability. In model design, CNN is usually regarded as a "black box" and combined with the multi-head self-attention mechanism of Transformer to extract features. No objective explanation can be provided for the decision-making process and prediction results of the model, resulting in limited model interpretability. However, model interpretability is very important for increasing the credibility, robustness and usability of the model, as well as meeting legal, ethical and business requirements; 2) Limited model generalization ability. In model training, supervised learning methods are mostly used, which requires a large amount of labeled data. Accurately labeling the key points of the human body in the image is a time-consuming and professional knowledge-required task, and the labeling of complex movements may be subjective and inconsistent. Models trained on specific data sets usually have limited generalization capabilities.

[0047] This paper proposes a self-supervised human pose estimation method based on a multi-scale CNN-Transformer architecture. By designing reasonable self-supervised auxiliary tasks and network models, the model performance and generalization ability are improved by pre-training on a large-scale dataset (unlabeled) and then fine-tuning on a small sample size labeled dataset. Based on the AM principle, the key points are regressed through heat maps, and the predicted positions and dependent areas of different types of key points are displayed, verifying the interpretability of the proposed network.

[0048] like Figure 1 As shown, an embodiment of the present disclosure provides a method for estimating a human body posture, the method comprising:

[0049] Step S101: construct a first self-supervised spatial-aware model (SSpose) based on an activation maximization (AM) strategy; wherein the first Sspose includes an attention layer, and the attention layer is used to obtain the spatial correlation between key points and image positions.

[0050] Step S102: Based on the auxiliary task, the unlabeled data samples are input into the auxiliary network for self-supervised training until the predetermined conditions are met, and a second SSpose after training is obtained, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation.

[0051] It should be noted that the above method corresponds to the training process of the second SSpose, and the training process corresponding to the second SSpose can be executed on a dedicated device. After training the initial SSpose, the trained second SSpose can be obtained, and the trained second SSpose can be transplanted to an electronic device for the second SSpose application, and the trained second SSpose is applied. Of course, the training process of the initial SSpose and the estimation process of the trained second SSpose can also be performed on the same electronic device, which is not limited here. When the training process of the initial SSpose is executed on the dedicated device, after the initial SSpose training is completed, the trained second SSpose can be transplanted from the dedicated device to the electronic device that applies the second SSpose for execution.

[0052] In some embodiments, an image to be estimated is obtained, the image to be estimated including key point features for determining human pose. The image to be estimated is input into a second SSpose trained for human pose estimation for detection to obtain key points; wherein the second SSpose is based on an auxiliary task and is obtained by inputting unlabeled data samples into an auxiliary network for self-supervised training until predetermined conditions are met, the auxiliary network is constructed based on an encoder in a first SSpose, the first SSpose is constructed based on an AM strategy, and the first SSpose includes an attention layer, the attention layer being used to obtain spatial correlation between key points and image positions.

[0053] In some embodiments, based on the activation maximization AM strategy, a first self-supervised spatial perception network model SSpose is constructed; wherein, the first SSpose includes an attention layer, and the attention layer is used to obtain the spatial correlation between specific key points and image positions. Based on the auxiliary task, unlabeled data samples are input into the auxiliary network for self-supervised training until predetermined conditions are met, and a trained second SSpose is obtained, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation. An image to be estimated is obtained, and the image to be estimated includes key point features for determining human posture. The image to be estimated is input into the trained second SSpose for human posture estimation for detection to obtain key points.

[0054] In some embodiments, the image in the present disclosure may be a captured image of a pedestrian. For example, the pedestrian image may be an image of a single pedestrian. For example, a pedestrian detector (such as YOLOv4) may be used to obtain a single pedestrian image, which is then input into a second SSpose (e.g., a human pose estimation network model) for key point detection. For example, K key point coordinates may be obtained.

[0055] In some embodiments, the unlabeled data used in this disclosure can be selected from the ImageNet-1K dataset. This dataset can be a subset of the ImageNet dataset, containing 1,000 common image categories, consisting of 1.3M images, and divided into a training set and a validation set. In other words, SSpose can be trained on the ImageNet-1K training set.

[0056] In some embodiments, the labeled data used in this disclosure can be selected from the COCO2017 dataset. This dataset is a large-scale dataset that can be used for image classification, object detection, semantic segmentation, and pose estimation tasks. The selected human key point detection dataset labels 17 key points for each person in the image, and contains a total of 200K images and 250K human instances. The train2017 dataset, which contains 57K images and 150K human instances, is used for training, and the val2017 dataset, which contains 5K images, is used for validation.

[0057] In some embodiments, the SSpose constructed based on the AM strategy includes: a hybrid CNN-Transfomer encoder, a multi-scale fusion module and a head structure; wherein the hybrid CNN-Transfomer encoder is used to obtain low-level image features and high-level global dependencies, and the low-level image features include multi-granularity image features; the multi-scale fusion module is used to fuse the obtained multi-granularity image features; the head structure is used to predict the key point heat map using the contributions of all image positions collected by the final attention layer of the Transformer.

[0058] In some embodiments, see Figure 2 , shows the construction of the first self-supervised spatially aware human pose estimation model SSpose. The model consists of three parts: a hybrid CNN-Transformer encoder to simultaneously capture low-level image features and high-level global dependencies; a head structure to predict keypoint heatmaps; and a multi-scale fusion module to fuse the captured fine-grained and coarse-grained image information.

[0059] In some embodiments, the hybrid CNN-Transformer encoder (corresponding to Figure 2 The Encoder module in is designed into four stages, where stage 1 (corresponding to Figure 2 Stage 1 in Stage 2 (corresponding to Figure 2 Stage 2) and Stage 3 (corresponding to Figure 2 Stage 3 in the convolution stage is used to capture multi-granularity image information on high-resolution feature maps, and stage 4 (corresponding to Figure 2 Stage 4 in the figure is the transformer stage, which is used to capture high-level global dependencies on low-resolution feature maps.

[0060] In some embodiments, given an input image I, stage 1 first generates a high-resolution token embedding E1 using non-overlapping 2×2 strided convolutions, which is then fed into N1 convolutional blocks to extract features. Similarly, stages 2 and 3 further downsample the output of the previous stage to obtain token embeddings E2 and E3, which are then processed by N2 and N3 convolutional blocks, respectively. The convolutional blocks follow the design principles of the Transformer blocks and only use the multi-head self-attention (MHSA, see MHSA for details. Figure 3 ) operation is replaced by a 5×5 convolution. The multi-scale features extracted in the convolution stage are fused through a linear layer to obtain the token embedding E mf. Then, the E with the embedded location information mf are fed into stacked transformer blocks for processing to capture high-level global dependencies.

[0061] In some embodiments, the multi-scale fusion module is used to obtain token embeddings E1, E2, and E3 from different stages of the encoder, which can capture fine-grained and coarse-grained image information. In order to better utilize this multi-granularity representation, E1 and E2 are downsampled to the same size as E3 using stride-4 and stride-2 convolutions respectively, and the resulting multi-scale fusion feature E mf . For example,

[0062] Emf=StrideConv(E1,4)+StrideConv(E2,2)+E3

[0063] In some embodiments, for the head structure, in order to establish a connection with the interpretability of activation maximization (AM), the first SSpose model uses a simple head structure instead of a complex decoder structure to be attached to the encoder. The head structure uses the contribution (i.e., attention score) of all image positions collected by the final attention layer of the Transformer to predict K types of key point heat maps, and the maximum position in the heat map is the key point position. The encoder output can be upsampled to 1 / 4 of the input image size using transposed convolution, and then its number of channels is reduced to K using 1×1 convolution.

[0064] In some embodiments, AM-based keypoint position prediction can be implemented based on the following process:

[0065] In some embodiments, the core mechanism of Ttransformer is MHSA, such as Figure 3 As shown. It first inputs the sequence

[0066] X∈R L×d

[0067] Projected to the query vector Q∈R through three matrices Wq, Wk, Wv L×d , key vector K∈R L×d Sum value vector Q∈R L×d , then the attention score matrix A∈R N×N The calculation formula is as follows:

[0068]

[0069] Tokens x i ∈R d Each query qi∈Rd Calculate the similarity with all keys to obtain a weight vector w i =Ai∈R L×d Then, by comparing all elements in the value matrix V with w i The linear summation of the corresponding weights in is used to obtain an increment, which is then added to x i In doing so, the attention map can be viewed as a dynamic weight determined by the specific image content, which reweights the information flow in the forward propagation.

[0070] It should be noted that self-attention captures and reveals the contribution of predictions from each image position, and this contribution from different image positions can be reflected by the gradient. Therefore, by calculating x at the position j of the last attention layer of the input sequence j Activation h i ∈R K The derivative of (K key points) can specifically analyze the x at image position j j How does it affect the activation h at position i? i , which is the predicted key point heat map. We calculate h i Partial derivative at output heatmap i:

[0071]

[0072] Among them, v j ∈R^d is given by The transformed value vector, A(i,j) is a scalar value calculated by the dot product between qi and kj. Further assume that the function Given an attention score A(i,j), we can obtain:

[0073]

[0074] Where K, B∈R^(K×d) is the static weight shared by all positions. It can be seen that the function G is approximately linear with A_(i,j), that is, for the prediction h i The contribution of a directly depends on its attention score at the image location.

[0075] In some embodiments, the last attention layer in the encoder acts as an aggregator whose attention scores are treated as image-specific weights, aggregating contributions from all image positions according to the attention scores and forming the maximum activation in the predicted keypoint heatmap. Although the layers in the FFN and head cannot be ignored, they are position-dependent, meaning that they approximately linearly transform the contributions from all positions by the same transformation without changing their relative scale.

[0076] In some embodiments, the interpretability of activation maximization is that the input region that maximizes the activation of a given neuron can explain what the activated neuron is looking for. The learning goal is to maximize the activation of the neuron h(i*) at the desired location i* in the heatmap, where i* represents the ground truth location of the keypoint:

[0077]

[0078] Assuming the model has been optimized with parameters θ* and predicts a particular keypoint at position i (the maximum position in the heatmap), why the model predicts such a prediction can be explained by the fact that those positions j whose element j has a high attention score (≥δ) on i are dependencies that contribute significantly to the prediction. The dependencies can be found as follows:

[0079] J={j|A(i,j)(θ^*,I)≥δ}

[0080] Where A∈R L×L is the attention map of the last attention layer, and is also a function of θ* and I. Given an input image I and a query position i, A_(i,:) can reveal what the predicted position i is highly dependent on, defined as the dependent region, and A_(:,j) can reveal the region that position j mainly affects, defined as the affected region.

[0081] In some embodiments, CNN-based prediction methods also use heatmap activations as keypoint locations, but due to the depth and high nonlinearity of deep CNNs, it is not possible to directly find interpretable prediction patterns. The present invention extends AM to heatmap-based localization via Transformer, without requiring additional optimization cost, as the optimization is already implicitly performed in our training, i.e., A = A(θ*, I). The defined dependency region is the pattern we seek, which can show image-specific and keypoint-specific dependencies.

[0082] In the human pose estimation network model proposed in this disclosure, the convolution stage is used to extract low-level features, and the Transformer stage is used to capture high-level global dependencies. The last attention layer of the Transformer acts as an aggregator, aggregating different contributions from all image positions through attention scores, and finally forming the maximum position in the heat map. This type of key point localization method establishes a connection with the interpretability of AM through the Transformer and extends it to the localization task. The resulting attention score can indicate which specific image clues have a significant contribution to the predicted position. With this evidence, we can further analyze the behavior of the model by examining the impact of different experimental variables.

[0083] In some embodiments, see Figure 4 Before step S102, that is, based on the auxiliary task, the unlabeled data samples are input into the auxiliary network for self-supervised training until the predetermined conditions are met and the trained second Sspose is obtained, the method also includes: step S401, the convolution block of the first SSpose is replaced by a masked convolution block.

[0084] In some embodiments, based on the activation maximization (AM) strategy, a first self-supervised spatial perception network model SSpose is constructed; wherein the first SSpose includes an attention layer, which can be used to obtain the spatial correlation between specific key points and image positions. The convolution block of the first SSpose is replaced with a masked convolution block. Based on the auxiliary task, unlabeled data samples are input into the auxiliary network for self-supervised training until predetermined conditions are met, thereby obtaining a trained second SSpose, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation.

[0085] In some embodiments, see Figure 5 , step S102 may include: step S501, based on the auxiliary task and the hierarchical mask strategy, inputting the unlabeled data sample into the auxiliary network for self-supervised training until the predetermined conditions are met, and obtaining the second Sspose after training.

[0086] In some embodiments, based on the activation maximization AM strategy, a first self-supervised spatial perception network model SSpose is constructed; wherein the first SSpose includes an attention layer, and the attention layer is used to obtain the spatial correlation between key points and image positions. The convolution block of the first SSpose is replaced with a mask convolution block. Based on the auxiliary task and the hierarchical mask strategy (corresponding to Figure 6 The unlabeled data samples are input into the auxiliary network for self-supervised training until the predetermined conditions are met to obtain the first SSpose after training, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation.

[0087] In some embodiments, see Figure 6 , design auxiliary tasks, and build an auxiliary network based on the encoder of the first Sspose for pre-training.

[0088] In some embodiments, image reconstruction is selected as an auxiliary task, and an auxiliary network is constructed based on the encoder of the first Sspose for self-supervised pre-training. Figure 6The encoder structure of the auxiliary network is basically the same as that of SSpose, except that the convolution block is replaced by the mask convolution block (as shown in Figure 6 The decoder consists of stacked Transformer modules for mask token reconstruction.

[0089] In some embodiments, self-supervised pre-training is implemented based on a hierarchical masking strategy. For example, a random mask M3 (masking rate is p%, such as 75%) is first generated from the input token of stage 3, and then the mask M3 is upsampled by 2 times and 4 times to obtain masks M2 and M1 corresponding to stage 2 and stage 1, respectively. We also introduced masked convolution in the convolution stage to avoid the mask area from participating in the encoding process. In this way, the post-processed tokens can be completely separated into masked tokens and visible tokens. During the encoding process, the masked tokens corresponding to the four stages will be discarded and reconstructed by the decoder. The objective function only uses MSE as the loss function, and calculates the MSE loss before the prediction vector and the masked pixel value, that is, only the reconstruction of the masked patches is considered.

[0090] In some embodiments, during the pre-training process, only a small number of visible tokens are input to the encoder for feature representation learning. The multi-granularity tokens E1, E2, and E3 obtained by the encoder capture both fine-grained and coarse-grained image information. The encoded fused visible tokens E mf Together with the masked tokens, they are fed into the decoder to reconstruct the tokens that were masked during the encoding process. Since the encoder only needs to process a small portion of visible tokens, it improves training speed while alleviating scalability issues when pre-training large visual models.

[0091] In some embodiments, a first self-supervised spatial perception network model (SSpose) is constructed based on an activation maximization (AM) strategy. The first SSpose includes an attention layer that captures spatial correlations between keypoints and image locations. Based on an auxiliary task, unlabeled data samples are fed into the auxiliary network for self-supervised training until predetermined conditions are met, resulting in a trained second SSpose. The auxiliary network is constructed based on the encoder in the first SSpose. The second SSpose is used for human pose estimation. Labeled data samples are fed into the self-supervised trained second SSpose for model fine-tuning.

[0092] In some embodiments, the masked autoencoders (MAE) are extended to the multi-scale CNN-Transformer architecture for self-supervised learning by introducing a hierarchical masking strategy and sparse convolution, and the key points are regressed based on the AM principle heat map, which enhances the model interpretability while improving the model performance and generalization ability.

[0093] The second SSpose model in this invention uses the final attention layer of the Transformer to aggregate the contributions (attention scores) from all image locations to determine the maximum location in the heatmap for keypoint localization. This heatmap-based approach enables the model to explicitly capture the spatial correlation between specific keypoints and various image locations, thereby providing enhanced interpretability.

[0094] In this paper, by introducing masked convolution and layered masking strategies, MAE is integrated into the first SSpose for self-supervised training. While ensuring training efficiency, the model can learn universal feature representations, avoiding the overfitting problem caused by scarce training data, and improving model performance and generalization ability.

[0095] In the present invention, all convolutions can be replaced by depthwise separable convolutions (depthwise convolution + pointwise convolution), which greatly reduces the number of model parameters and computational complexity, saving resource costs while making model deployment easier.

[0096] In the present invention, since the first Sspose constructed based on the AM strategy includes an attention layer for obtaining the spatial correlation between key points and image positions, the attention layer can be used to obtain the spatial correlation between specific key points and various image positions. In this way, the process of constructing an auxiliary network based on the encoder in the first SSpose and training based on the auxiliary task actually integrates the factors of the spatial correlation between key points and image positions. The output result of the second Sspose after training is actually closely related to the spatial correlation between key points and image positions, which enhances the interpretability of the model compared to the method without setting the attention layer. In addition, since the second SSpose is a model obtained based on self-supervised training, there is no need to use large-scale labeled data, thereby reducing the workload of data labeling.

[0097] In some embodiments, see Figure 7 , provides a self-supervised human pose estimation method based on the CNN-Transformer architecture. This method first pre-trains the model on a large dataset and then fine-tunes the model based on a smaller dataset for the human pose estimation task. The method includes:

[0098] Step S701: Obtain a data set for SSpose pre-training and fine-tuning;

[0099] Step S702: Based on the AM principle, construct a first SSpose of the human posture estimation network model;

[0100] Step S703: Design an auxiliary task, build an auxiliary network based on the encoder of the first SSpose, perform self-supervised pre-training, and obtain a second SSpose;

[0101] Step S704: Fine-tune the second SSpose model using the labeled data;

[0102] Step S705: Following the top-down human pose estimation method, a pedestrian detector is first used to infer a single pedestrian image, which is then fed into the proposed human pose estimation network model for key point detection.

[0103] In some embodiments, see Figure 8 , showing the results of the auxiliary network model and its training process.

[0104] In some embodiments, Figure 8 The auxiliary network model in includes an encoder and a decoder.

[0105] In some embodiments, the encoder (see Figure 8 The Encoder in the left dotted box in the figure can be designed as a multi-scale CNN-Transformer structure, which can be composed of three stage structures: Stage 1, Stage 2 and Stage 3.

[0106] In some embodiments, Stage 1 and Stage 2 are convolution stages for extracting low-level features. The functions involved in the convolution stage mainly include a Patch Embedding layer and multiple mask convolution modules.

[0107] In some embodiments, the Patch Embedding layer is used to segment the input image into non-overlapping patches of the same size, and convert each patch and its position information into a vector representation (hereinafter referred to as tokens).

[0108] In some embodiments, the masked convolution module follows the design principles of the Transformer module, only replacing the multi-head self-attention (MHSA) operation with a 5×5 depth-wise separable convolution operation.

[0109] In some embodiments, Stage 3 is a Transformer stage, which is used to capture high-level global dependencies and process the tokens output by Stage 2 using stacked Transformer modules.

[0110] In some embodiments, stride-2 convolution is used between each stage to downsample tokens to half the previous spatial resolution.

[0111] In some embodiments, the decoder (see Figure 8 The middle right dotted box (Decoder) can be composed of linear layers and stacked Transformer modules.

[0112] In some embodiments, the linear layer is used to fuse the multi-scale (or different granularity) features E1, E2, and E3 obtained in the encoding stage. Here, E1 and E2 may correspond to the first feature, and E3 may correspond to the second feature.

[0113] In some embodiments, stacked Transformer modules are used to reconstruct the original data.

[0114] In some embodiments, the auxiliary network model (e.g., SSpose) can be self-supervised pre-trained on the ImageNet1K training set by generating random masks to mask part of the content of the input image and then reconstructing the image.

[0115] In some embodiments, see again Figure 8 , showing the model pre-training process, the specific pre-training process is as follows:

[0116] Because the encoder is designed as a CNN-Transformer architecture, directly generating random masks from Stage 1 would cause almost all tokens in the downsampled Stage 3 to contain some visible information, rendering the encoder no longer sparse. Therefore, to avoid visible information leakage during convolutional reconstruction, a layered masking strategy is considered: first, a random mask is generated from Stage 3, then upsampled by a factor of 2 and a factor of 4 to obtain masks corresponding to Stage 2 and Stage 1, respectively.

[0117] In some embodiments, see Figure 9 , the model training methods include:

[0118] Step S901: reshape the input image I to a size of H×W;

[0119] Step S902: Obtain masks corresponding to the three stages:

[0120] For example, first, a mask M3 is randomly generated from the tokens obtained in Stage 3; then, a fixed position of the mask M3 is upsampled by a factor of 2 to obtain the mask M2 corresponding to Stage 2; and then a fixed position of the mask M2 is upsampled by a factor of 2 to obtain the mask M1 corresponding to Stage 1. In this way, the tokens processed later can be completely separated into masked tokens and visible tokens.

[0121] Step S903: Input image I into the self-supervised network. The Patch Embedding layer divides the input image into multiple non-overlapping, p×p patches and converts each patch and its position information into a vector representation, i.e., tokens. Subsequent network processing is based on tokens.

[0122] Step S904: In convolution Stage 1 and Stage 2, the mask convolution module is used to extract low-level features on tokens. The internal structure of the mask convolution module is as follows: Figure 10 As shown in (left), it mainly includes convolutional layers, FFN layers, linear layers, and residual connections. Among them, Mask is the corresponding mask M1 and M2 obtained in step 702, which is used to mask out p% (75% in this embodiment) of the input tokens in Stage 1 and Stage 2; only 25% of the visible tokens are fed into the subsequent network, and convolution operations are performed to extract features using depthwise separable convolution (reducing the number of model parameters) and sparse convolution (avoiding visible information leakage);

[0123] Step S905: In Transformer Stage 3, mask M3 is also used to mask out 75% of the input tokens. Only 25% of the visible tokens are fed into the subsequent stacked Transformer module to capture high-level global dependencies. The internal structure of the Transformer module is as follows: Figure 10 (right) shows the main components of the MHSA layer, FFN layer, residual connection, and layer normalization.

[0124] Step S906: In the decoding stage, the output tokens E1∈R of Stage 1 is transformed through stride-4 and stride-2 convolution. H / 4×W / 4×C1 and the output tokens E2∈R of Stage 2 H / 8×W / 8×C2 Downsample to the output tokens E3∈R of Stage 3 H / 16×W / 16×C3 Finally, the multi-granularity tokens are fused through a linear layer to obtain visible tokens E d , as shown in Formula 1:

[0125] Ed=Linear(StrideConv(E1,4)+StrideConv(E2,2)+E3)

[0126] Step S907, tokens E can be seen d Together with the mask tokens, they are fed into the stacked Transformer module for image reconstruction.

[0127] In some embodiments, the objective function only uses MSE as the loss function, and calculates the MSE loss between the prediction vector and the masked pixel value, that is, only the reconstruction of the masked patches is considered.

[0128] During pre-training, only a small number of visible tokens are fed into the encoder for feature representation learning. The multi-granularity tokens E1, E2, and E3 obtained by the encoder capture both fine-grained and coarse-grained image information. The encoded fused visible tokens Ed are fed into the decoder along with the mask tokens to reconstruct the tokens masked during the encoding process. Since the encoder only needs to process a small number of visible tokens, this improves training speed while mitigating scalability issues when pre-training large visual models.

[0129] In some embodiments, a network model suitable for human pose estimation is constructed based on the encoder part of the auxiliary network model and is fine-tuned using a small amount of labeled data.

[0130] In some embodiments, see Figure 11 The network model constructed for the human pose estimation task, namely the first SSpose, is shown. Compared with the auxiliary network model, the encoder structure remains basically unchanged, the decoder part is directly removed, and a head is attached after the encoder to directly predict K (K = 17 in this embodiment) types of key point heat maps based on the AM principle.

[0131] In some embodiments, the following will specifically describe the key point position prediction based on AM:

[0132] The core mechanism of Ttransformer is MHSA, such as Figure 10 (right). It first inputs the sequence X∈R L×d Projected to the query vector Q∈R through three matrices Wq, Wk, Wv L×d , key vector K∈R L×d Sum value vector R L×d , then the attention score matrix A∈R N×N The calculation formula is as follows:

[0133]

[0134] Tokens x i ∈R d Each query qi∈R d Calculate the similarity with all keys to obtain a weight vector wi = Ai∈R 1×L Then, an increment is obtained by linearly summing all elements in the value matrix V with the corresponding weights in wi, and then adding this increment to xi. In doing so, the attention map can be regarded as a dynamic weight determined by the specific image content, which reweights the information flow in the forward propagation.

[0135] In some embodiments, self-attention captures and reveals the contribution of predictions from each image position, and this contribution from different image positions can be reflected by gradients. Therefore, by computing the activation hi∈R at position xj of the last attention layer with respect to the input sequence K The derivative of (K key points) can specifically analyze how xj at image position j affects the activation hi at position i, that is, the predicted key point heat map. Calculate the partial derivative of hi at the output heat map i:

[0136]

[0137] where vj∈R d is The transformed value vector, A(i,j) is a scalar value calculated by the dot product between qi and kj. Further assume that the function Given an attention score A(i,j), we can obtain:

[0138]

[0139] where K,B∈R K×d Static weights shared by all positions. It can be seen that the function G is approximately linear with A(i,j), that is, the contribution to the prediction of h_i directly depends on its attention score at the image position.

[0140] In some embodiments, the last attention layer in the encoder acts as an aggregator whose attention scores are treated as image-specific weights, aggregating contributions from all image positions according to the attention scores and forming the maximum activation in the predicted keypoint heatmap. Although the layers in the FFN and head cannot be ignored, they are position-dependent, meaning that they approximately linearly transform the contributions from all positions by the same transformation without changing their relative scale.

[0141] In some embodiments, the interpretability of activation maximization lies in the fact that the input region that maximizes the activation of a given neuron can explain what the activated neuron is looking for. The learning goal is to maximize the activation of the neuron h_(i*) at the desired location i^* in the heat map, where i* represents the ground truth location of the keypoint:

[0142]

[0143] Assuming the model has been optimized with parameters θ* and predicts a particular keypoint at position i (the maximum position in the heatmap), why the model predicts such a prediction can be explained by the fact that those positions j whose element j has a high attention score (≥δ) on i are dependencies that contribute significantly to the prediction. The dependencies can be found as follows:

[0144] J={j|A(i,j)(θ*,I)≥δ}

[0145] where A∈R L×L is the attention map of the last attention layer, and is also a function of θ* and I. Given an input image I and a query position i, A(i,:) can reveal what the predicted position i is highly dependent on, defined as the dependent region, and A(:,j) can reveal the region that position j mainly affects, defined as the affected region.

[0146] Related CNN-based prediction methods also use heatmap activations as keypoint locations, but due to the depth and high nonlinearity of deep CNNs, it is impossible to directly find interpretable prediction patterns. This paper extends AM to heatmap-based localization via Transformer, without requiring additional optimization effort, as the optimization is implicitly performed during training: A = A(θ*, I). The defined dependency regions are the patterns we seek, revealing both image-specific and keypoint-specific dependencies.

[0147] Thus, in the human pose estimation network model proposed in the present disclosure, the convolution stage is used to extract low-level features, and the Transformer stage is used to capture high-level global dependencies. The last attention layer of the Transformer acts as an aggregator, aggregating different contributions from all image positions through attention scores, and finally forming the maximum position in the heat map. This type of key point localization method establishes a connection with the interpretability of AM through the Transformer and extends it to the localization task. The resulting attention score can indicate which specific image clues have a significant contribution to the predicted position.

[0148] like Figure 12As shown, an embodiment of the present disclosure provides a human body posture estimation training method, the method comprising:

[0149] Step S1201: obtaining an image to be estimated, wherein the image to be estimated includes key point features for determining a human body posture;

[0150] Step S1202, input the image to be estimated into the second SSpose for human posture estimation after training for detection to obtain key points; wherein, the SSpose is based on an auxiliary task and is obtained by inputting unlabeled data samples into an auxiliary network for self-supervised training until predetermined conditions are met, the auxiliary network is constructed based on the encoder in the first SSpose, the first SSpose is constructed based on the AM strategy, the first SSpose includes an attention layer, and the attention layer is used to obtain the spatial correlation between the key points and the image position.

[0151] In some embodiments, a first spatial perception network model, a self-supervised spatial perception model (SSpose, Self-supervised Spatial-aware Model), is constructed based on an activation maximization (AM) strategy; wherein the first SSpose includes an attention layer, which is used to obtain spatial correlations between key points and image positions. Based on an auxiliary task, unlabeled data samples are input into the auxiliary network for self-supervised training until predetermined conditions are met, thereby obtaining a trained second SSpose, wherein the auxiliary network is constructed based on the encoder in the first SSpose; and the second SSpose is used for human pose estimation.

[0152] It should be noted that the above method corresponds to the application process of the second SSpose, and the training process corresponding to the second SSpose can be performed on a dedicated device. After training the initial SSpose, the trained second SSpose can be obtained, and the trained second SSpose can be transplanted to the electronic device used for the second SSpose application, and the trained second SSpose is applied. Of course, the training process of the initial SSpose and the estimation process of the trained second SSpose can also be performed on the same electronic device, which is not limited here. When the training process of the initial SSpose is performed on the dedicated device, after the initial SSpose training is completed, the trained second SSpose can be transplanted from the dedicated device to the electronic device that applies the second SSpose for execution.

[0153] In some embodiments, an image to be estimated is obtained, the image to be estimated including key point features for determining human pose. The image to be estimated is input into a second SSpose trained for human pose estimation for detection to obtain key points; wherein the second SSpose is based on an auxiliary task and is obtained by inputting unlabeled data samples into an auxiliary network for self-supervised training until predetermined conditions are met, the auxiliary network is constructed based on an encoder in a first SSpose, the first SSpose is constructed based on an AM strategy, and the first SSpose includes an attention layer, the attention layer being used to obtain spatial correlation between key points and image positions.

[0154] In some embodiments, a first spatial perception network model SSpose is constructed based on an activation maximization (AM) strategy; wherein the first SSpose includes an attention layer, which is used to obtain spatial correlations between key points and image positions. Based on an auxiliary task, unlabeled data samples are input into an auxiliary network for self-supervised training until predetermined conditions are met, thereby obtaining a trained second SSpose, wherein the auxiliary network is constructed based on the encoder in the first SSpose; and the second SSpose is used for human pose estimation. An image to be estimated is obtained, wherein the image to be estimated includes key point features for determining human pose. The image to be estimated is input into the trained second SSpose for human pose estimation for detection to obtain key points.

[0155] It should be noted that the training steps involved in this step can be found in the above Figure 1 The description of this part is not limited here.

[0156] like Figure 13 As shown, an embodiment of the present disclosure provides a human body posture estimation device, the device comprising:

[0157] A construction module 131 is configured to: construct a first self-supervised spatial perception network model SSpose based on an activation maximization (AM) strategy; wherein the first SSpose includes an attention layer, and the attention layer is used to obtain spatial correlation between key points and image positions;

[0158] The training module 132 is configured to: based on the auxiliary task, input the unlabeled data samples into the auxiliary network for self-supervised training until the predetermined conditions are met, and obtain a trained second SSpose, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation.

[0159] In some embodiments, the construction module 131 is further configured to: construct the first SSpose including: a hybrid CNN-Transfomer encoder, a multi-scale fusion module and a head structure; wherein the hybrid CNN-Transfomer encoder is used to obtain low-level image features and high-level global dependencies, and the low-level image features include multi-granularity image features; the multi-scale fusion module is used to fuse the obtained multi-granularity image features; the head structure is used to predict the key point heat map using the contributions of all image positions collected by the final attention layer of the Transformer.

[0160] In some embodiments, the building module 131 is further configured to:

[0161] The convolution block of the first SSpose is replaced by a masked convolution block.

[0162] In some embodiments, the training module 132 is further configured to:

[0163] Based on the auxiliary task and the hierarchical mask strategy, the unlabeled data samples are input into the auxiliary network for self-supervised training until the predetermined conditions are met, and the second SSpose after training is obtained.

[0164] In some embodiments, the training module 132 is further configured to:

[0165] The labeled data samples are input into the second SSpose after self-supervised training to perform model fine-tuning.

[0166] In some embodiments, the training module 132 is further configured such that: the auxiliary task is an image reconstruction task.

[0167] In some embodiments, see Figure 14 , provides a human body posture estimation device, the device comprising:

[0168] An acquisition module 141 is configured to acquire an image to be estimated, wherein the image to be estimated includes key point features for determining a human body posture;

[0169] The detection module 142 is configured to: input the image to be estimated into the second SSpose for human posture estimation after training for detection to obtain key points; wherein, the SSpose is based on an auxiliary task and is obtained by inputting unlabeled data samples into an auxiliary network for self-supervised training until predetermined conditions are met, the auxiliary network is constructed based on the encoder in the first SSpose, the first SSpose is constructed based on the AM strategy, and the first SSpose includes an attention layer, and the attention layer is used to obtain the spatial correlation between the key points and the image positions.

[0170] An embodiment of the present disclosure provides a processing device, the processing device comprising:

[0171] a memory for storing executable programs;

[0172] The processor is configured to implement any method described in the embodiments of the present disclosure when executing the executable program stored in the memory.

[0173] It is understood that the memory can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); magnetic surface memory can be magnetic disk memory or tape memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0174] Among them, the method for determining the topology structure disclosed in the present invention can be applied to the processor or implemented by the processor. The processor can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the method for determining the topology structure can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the various methods, steps and logic block diagrams disclosed in the present invention. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the present invention can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory. The processor reads the information in the memory and, in combination with its hardware, completes the steps of the method for determining the topology structure provided in the embodiment of the present application.

[0175] The present invention also provides a computer storage medium storing an executable program. When the executable program is executed by a processor, the method for determining a topology structure as described in any of the embodiments of the present disclosure is implemented. Specifically, it can be a computer-readable storage medium, such as a memory that stores a computer program. The computer program can be executed by a processor of a processing device to complete the steps described in the method of the embodiment of the present application. The computer-readable storage medium can be a memory such as ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface storage, optical disk, or CD-ROM.

[0176] The present disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the human pose estimation methods described above in the present disclosure.

[0177] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for estimating a human body posture, characterized in that: The method comprises: Based on the activation maximization AM strategy, a first self-supervised spatial perception network model SSpose is constructed; wherein the first SSpose includes an attention layer, and the attention layer is used to obtain the spatial correlation between the key points and the image positions; Based on the auxiliary task, unlabeled data samples are input into the auxiliary network for self-supervised training until predetermined conditions are met to obtain a trained second SSpose, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation.

2. The method according to claim 1, characterized in that The first SSpose constructed includes: a hybrid CNN-Transfomer encoder, a multi-scale fusion module and a head structure; wherein the hybrid CNN-Transfomer encoder is used to obtain low-level image features and high-level global dependencies, and the low-level image features include multi-granularity image features; the multi-scale fusion module is used to fuse the obtained multi-granularity image features; the head structure is used to predict the key point heat map using the contributions of all image positions collected by the final attention layer of the Transformer.

3. The method according to claim 1, characterized in that Before inputting the unlabeled data sample into the auxiliary network for self-supervised training based on the auxiliary task until a predetermined condition is met and the trained second SSpose is obtained, the method further includes: The convolution block of the first SSpose is replaced by a masked convolution block.

4. The method according to claim 1, wherein Based on the auxiliary task, inputting the unlabeled data sample into the auxiliary network for self-supervised training until a predetermined condition is met to obtain the trained second SSpose, including: Based on the auxiliary task and the hierarchical mask strategy, the unlabeled data samples are input into the auxiliary network for self-supervised training until the predetermined conditions are met, and the second SSpose after training is obtained.

5. The method according to claim 1, wherein The method further comprises: The labeled data samples are input into the second SSpose after self-supervised training to perform model fine-tuning.

6. The method according to any one of claims 1 to 5, characterized in that The auxiliary task is an image reconstruction task.

7. A method for estimating a human body posture, characterized in that: The method comprises: Acquire an image to be estimated, wherein the image to be estimated includes key point features for determining a human body posture; The image to be estimated is input into the second SSpose for human posture estimation after training for detection to obtain key points; wherein, the second SSpose is based on an auxiliary task and is obtained by inputting unlabeled data samples into an auxiliary network for self-supervised training until predetermined conditions are met, the auxiliary network is constructed based on the encoder in the first SSpose, the first SSpose is constructed based on the AM strategy, and the first SSpose includes an attention layer, and the attention layer is used to obtain the spatial correlation between the key points and the image positions.

8. A human body posture estimation device, characterized in that: The device comprises: A construction module is configured to: construct a first self-supervised spatial perception network model SSpose based on an activation maximization AM strategy; wherein the first Sspose includes an attention layer, and the attention layer is used to obtain spatial correlation between key points and image positions; The training module is configured to: based on the auxiliary task, input unlabeled data samples into the auxiliary network for self-supervised training until predetermined conditions are met, and obtain a trained second SSpose, wherein the auxiliary network is constructed based on the encoder in the first SSpose; the second SSpose is used for human posture estimation.

9. A processing device, characterized in that: The processing equipment includes: a memory for storing executable programs; A processor, configured to implement the method according to any one of claims 1 to 6 or 7 when executing the executable program stored in the memory.

10. A computer storage medium, characterized in that The computer storage medium stores an executable program, and when the executable program is executed by a processor, the method according to any one of claims 1 to 6 or 7 is implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 6 or 7 is implemented.

Images