A global recursive hybrid human pose estimation method based on key point prior guidance

Abstract

This invention discloses a global recursive hybrid human pose estimation method based on keypoint prior guidance, belonging to the fields of computer vision and deep learning technology. The method includes: extracting feature maps from the input image using a convolutional backbone network, and converting the feature maps into image block word sequences through image block embedding; constructing a skeleton-based keypoint adjacency matrix, and using a graph convolutional network to initially encode the keypoint words, endowing them with structure awareness; concatenating keypoint words and image block words, sequentially passing them through a Transformer layer to model global dependencies, modulating pose-guided features through a keypoint gating module, and recursively learning long-range features through a bidirectional Mamba module; finally, fusing the keypoint word features output from the Transformer and Mamba, and outputting a human keypoint heatmap through a prediction head network. This invention effectively solves the problems of insufficient structural modeling and noise accumulation in existing Transformer-Mamba hybrid models for human pose estimation, significantly improving pose estimation accuracy while maintaining computational efficiency.

Description

Technical Field

[0001] This invention belongs to the field of human pose estimation in computer vision and deep learning technology, specifically involving a global recursive hybrid human pose estimation method based on keypoint prior guidance. Background Technology

[0002] In recent years, Transformer-based human pose estimation methods have made significant progress and have gradually become one of the mainstream research directions in this field. TransPose was the first to introduce the Transformer architecture into the human pose estimation task, using a self-attention mechanism to model long-range dependencies between keypoints, breaking through the bottleneck of limited receptive fields in traditional convolutional neural networks. Building on this, TokenPose proposed a lexical representation method, transforming image features into a sequence of tokens, and predicting human poses through learnable keypoint tokens, achieving effective modeling of structural constraints between keypoints. ViTPose further explored the application of the pure vision Transformer architecture in pose estimation, achieving excellent performance through a global self-attention mechanism, verifying the great potential of the Transformer architecture in human pose estimation tasks.

[0003] However, the self-attention mechanism, a core component of the Transformer architecture, has... The computational complexity and The space complexity, where The sequence length is a key characteristic that presents a significant computational efficiency challenge for Transformer when handling long sequences. In human pose estimation tasks, to obtain accurate keypoint localization, models typically need to process high-resolution feature maps. The resulting long sequences significantly increase the computational cost of the self-attention mechanism, severely limiting the deployment and application of the model in resource-constrained scenarios. More critically, Transformer's global receptive field modeling is overly generalized, lacking focus on specific structural features, and its output features contain a large amount of pose-independent redundant information.

[0004] State-space models (SSMs), as an emerging sequence modeling paradigm, have garnered significant attention in recent years, particularly for the Mamba model. Mamba, by introducing a selective state-space mechanism, maintains linear complexity. Simultaneously, it achieves efficient long sequence modeling capabilities. Unlike the Transformer's self-attention mechanism, Mamba transforms sequence modeling into recursive computation through a state-space model, avoiding quadratic complexity attention matrix operations, thus exhibiting a significant efficiency advantage when processing long sequences. However, Mamba's global modeling capability is relatively weaker than Transformer's, and its recursive nature causes noise information to continuously accumulate during state propagation. Furthermore, applying Mamba to vision tasks still faces fundamental challenges: Mamba was originally designed to process one-dimensional sequence data, and its core mechanism relies on causal scanning; however, image data has a two-dimensional spatial structure, and directly flattening an image into a one-dimensional sequence would destroy this spatial structure.

[0005] Given the complementary characteristics of Transformer and Mamba, fusing the two is a meaningful direction for exploration: Transformer provides a global perspective and can directly model the global relationships between lexical units; Mamba provides reinforcement propagation of learned features, achieving long-range dependency recursive modeling with linear complexity. However, existing Transformer-Mamba hybrid models perform poorly in human pose estimation tasks, mainly due to two problems: (1) directly generated keypoint lexical units lack prior perception of the human skeleton structure, introducing additional noise; (2) redundant information from the global modeling output of Transformer is directly passed to Mamba, which further diffuses during the recursive process, affecting model performance. Therefore, how to effectively reduce computational complexity while maintaining the powerful modeling capabilities of Transformer and solve the noise accumulation problem in the hybrid architecture has become a key issue that urgently needs to be addressed in current human pose estimation research. Summary of the Invention

[0006] Technical Problem: The purpose of this invention is to provide a global recursive hybrid human pose estimation method based on key point prior guidance, so as to solve the problems of insufficient structural modeling and noise accumulation in the existing Transformer-Mamba hybrid model in human pose estimation.

[0007] Technical solution:

[0008] A global recursive hybrid human pose estimation method based on key point prior guidance, characterized by the following steps:

[0009] Step S1: Use a convolutional backbone network to extract features from the input human image to obtain an image feature map. Then, divide the feature map into multiple image blocks and perform linear projection through image block embedding to generate an image block word sequence.

[0010] Step S2: Construct a keypoint adjacency matrix based on the topological structure of the human skeleton, and use a graph convolutional network to structurally encode keypoint words to obtain a keypoint word sequence with structure awareness.

[0011] Step S3: Concatenate the key point word sequence with the image block word sequence to form a complete word sequence, and input the word sequence into the Transformer encoder for global dependency modeling;

[0012] Step S4: The Transformer output features are subjected to pose condition modulation through the key point gating module. The key point words are used to generate a global pose description and gating weights are generated accordingly. Channel-level adaptive modulation is performed on the image block words.

[0013] Step S5: Input the gated and modulated word sequence into the bidirectional Mamba module, and perform long-range feature modeling on the sequence through the state space recursion mechanism of forward and reverse scanning;

[0014] Step S6: Extract key point lexical features from the outputs of the Transformer encoder and the bidirectional Mamba module respectively, perform feature fusion, and output a human body key point heatmap through the prediction head network to achieve human pose estimation.

[0015] Furthermore, the convolutional backbone network in step S1 is a high-resolution network HRNet, which maintains high-resolution feature representation by connecting multiple sub-networks of different resolutions in parallel and exchanging information across resolutions.

[0016] Furthermore, the image block embedding method in step S1 includes: dividing the feature map output by the convolutional backbone network into multiple image blocks according to a preset size, and mapping each image block to a vector representation of a fixed dimension through linear projection, thereby constructing an image block word sequence.

[0017] Furthermore, the key point adjacency matrix in step S2 is constructed based on the connection relationship of the human skeleton. If there is a skeletal connection relationship between two key points or they are the same key point, the corresponding element of the adjacency matrix takes the value of 1; otherwise, it takes the value of 0, thus forming a symmetric adjacency matrix.

[0018] Furthermore, in step S2, the graph convolutional network aggregates neighborhood features of keypoint lexical units through an adjacency matrix, and combines linear projection and residual connections to encode keypoint structural information in order to obtain structure-aware keypoint lexical features.

[0019] Furthermore, the Transformer encoder in step S3 employs a multi-layer Transformer network with a pre-normalized structure, including a multi-head self-attention module and a feedforward network module, to model global dependencies in the lexical sequence.

[0020] Furthermore, the key point gating module in step S4 includes: first, performing pooling operations on key point terms to obtain a global pose description vector; then generating channel-level gating weights through a gating network; and finally using the gating weights to perform channel-by-channel modulation on image block term features to achieve pose-guided feature enhancement and noise suppression.

[0021] Furthermore, in step S5, the bidirectional Mamba module recursively models the input sequence using a selective state-space model, which performs forward and backward scans respectively, and fuses the outputs of the two to capture long-distance dependencies in the sequence.

[0022] Furthermore, the feature fusion method in step S6 is as follows: extract key point lexical features from the output sequence of the Transformer encoder and the output sequence of the bidirectional Mamba module respectively, and concatenate and fuse them in the feature dimension to obtain a fused key point feature representation.

[0023] Furthermore, the prediction head network in step S6 is a multilayer perceptron structure, which converts the fused key point lexical features into a human body key point heatmap through linear mapping, and determines the coordinates of the human body key points through the peak position of the heatmap.

[0024] Beneficial effects

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] (1) The keypoint prior graph convolutional network lexicalization method proposed in this invention introduces the topological information of the human skeleton, enabling keypoint lexicalization to have structural awareness in the initialization stage. This avoids the noise interference problem caused by unguided random learning and strengthens the perception of long-range associations in the subsequent Transformer and Mamba architectures. Compared with random initialization, the keypoint prior graph convolutional network lexicalization incorporates the topological information of the human skeleton in the initialization stage, making adjacent keypoints have similar feature representations, which conforms to the anatomical constraints of the human pose task. Secondly, graph convolutional encoding enables the model to utilize the structural relationship between keypoints in the early stage of training, accelerating the convergence process of the model. Finally, this initialization method provides a more structurally aware initial representation for the subsequent Transformer-Mamba hybrid pose prediction, which helps the model to model complex poses more accurately.

[0027] (2) The keypoint prior gating interaction mechanism designed in this invention generates a global pose description through keypoint lexical units, and then generates gating weights to adaptively modulate image block lexical units. This enables the model to dynamically focus on key regions in the image according to the pose context, suppressing the interference of background noise and reducing the cumulative effect of irrelevant information in state propagation from the source. The keypoint prior gating interaction network essentially constructs a sample-level channel attention, enabling the model to conditionally recalibrate image block lexical units according to the current pose context before entering the subsequent state space modeling. Compared with directly sending the Transformer output into the subsequent Mamba module, introducing gating interaction between the two can reduce the cumulative effect of irrelevant information in state propagation from the source, and improve the purity and stability of state updates.

[0028] (3) The global recursive hybrid architecture constructed in this invention is a combination of the Transformer module and the bidirectional Mamba module. The Transformer is responsible for capturing global dependencies. After pose enhancement guided by keypoints, the bidirectional Mamba module performs recursive propagation with linear complexity. The output features of the two modules are fused to predict the final heatmap. This architecture significantly reduces computational overhead while maintaining performance. The KTM encoder network follows a three-stage processing paradigm of "global modeling - pose enhancement - sequence recursion". In the first stage, the Transformer layer performs global dependency modeling on the word sequence through a multi-head self-attention mechanism, capturing the global correlation between keypoint words and image block words, and establishing a preliminary pose feature representation. In the second stage, the keypoint gating module performs channel-level adaptive modulation on the image block words based on the global pose description generated by the keypoint words, suppressing background information unrelated to pose prediction and enhancing pose-related semantic channels. In the third stage, the bidirectional Mamba module performs recursive modeling on the selected word sequence with linear complexity, achieving efficient long-range feature learning.

[0029] (4) This invention has been thoroughly validated on the MSCOCO dataset. The results show that the method of this invention surpasses all existing Mamba-based human pose estimation methods and achieves results competitive with mainstream benchmark models, verifying the effectiveness and practicality of the proposed method. Specifically, the KTMPose-B proposed in this invention reduces the number of parameters by 48.5% and the computational cost by 64.7% compared to the existing MambaPose method, while achieving a performance improvement of 2.8 AP. The KTMPose-L model significantly outperforms similar comparative models in both AP and AR metrics, achieving a performance improvement of at least 1.1 AP. Compared with the mainstream benchmark models TokenPose-B and TokenPose-L / D6, the method of this invention achieves performance improvements of 0.3 AP and 0.7 AP, respectively, while reducing the computational load.

[0030] (5) The effectiveness of each module design was verified through systematic ablation experiments. Experimental results show that in the lexicalized human pose estimation framework, bidirectional scanning can fully model the long-range dependencies between keypoint lexical units. Although further increasing the scanning direction formally expands the state propagation path, the new effective dependencies introduced by multi-directional recursion are very limited because the lexical sequence already contains explicit human structural information, leading to a saturation of performance improvement. The additional scanning direction significantly increases the recursion computation overhead. In addition, the introduction of a structure-aware gating interaction mechanism can achieve significant performance improvement with a small parameter increment, verifying the high compatibility between structural prior and sequence recursion mechanism. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall structure of an embodiment of the present invention;

[0032] Figure 2 This is a diagram of the Transformer encoder architecture in an embodiment of the present invention;

[0033] Figure 3 This is a diagram of the selective state-space model architecture in an embodiment of the present invention; Detailed Implementation

[0034] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 1 As shown, this invention proposes a global recursive hybrid human pose estimation method, KTMPose, based on keypoint prior guidance, which includes the following steps:

[0036] Step S1: Use a convolutional backbone network to extract feature maps from the input image, and convert the feature maps into image block word sequences through image block embedding;

[0037] Step S2: Construct a key point adjacency matrix based on the topology of the human skeleton, and use a graph convolutional network to initialize and encode key point words, so that key point words have structure awareness.

[0038] Step S3: Concatenate the encoded keypoint words with the image block words to form a complete word sequence, and then pass it through the Transformer layer for global dependency modeling.

[0039] Step S4: Implement pose-condition guided feature modulation through the key point gating module, generate a global pose description using key point terms, and generate gating weights to adaptively modulate image block terms.

[0040] Step S5: The bidirectional Mamba module performs long-range feature learning on the modulated word sequence with linear complexity;

[0041] Step S6: Fuse the key point lexical features output from the Transformer layer and the bidirectional Mamba module, and output a human key point heatmap through the prediction head network.

[0042] In step S1, the first four stages of HRNet-W48 are used as the backbone network of the model. HRNet maintains high-resolution feature representation throughout the network by connecting multiple sub-networks of different resolutions in parallel and exchanging information between them. This is crucial for human pose estimation tasks that require precise localization. Given an input image... , of which 3 and Representing the number of channels, initial image height, and width, respectively, the features extracted and output by the backbone network are represented as follows: Specifically, the HRNet-W48 backbone network generates feature maps. ,Right now , , .

[0043] After the backbone network outputs feature maps, they are converted into image block term sequences through image block embedding. Let the image block size be... The feature map is then divided into Each image patch is transformed into a linear projection of dimension [1, 2, 3]. The lexical units. This process can be represented as:

[0044]

[0045] The Rearrange operation transfers the feature map from... Rearranged as In this form, Linear is a linear projection layer, ultimately yielding an image block word sequence. The lexicalized features are then regularized using a Dropout layer to prevent overfitting.

[0046] The method for constructing the keypoint adjacency matrix in step S2 is as follows: the human skeleton has a well-defined topological structure, and keypoints are connected anatomically. Taking the COCO dataset as an example, 17 human keypoints include the nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles. These keypoints are connected by bones to form a specific human skeleton structure. This invention constructs a keypoint adjacency matrix based on the COCO human skeleton definition. ,in This refers to the number of key points.

[0047] The adjacency matrix is ​​defined as follows: if the key point With key points Directly connected or ,but ,otherwise Directly connected elements indicate a skeletal connection between two keypoints; diagonal elements... This indicates a self-join, used to preserve the node's own information; if there is a skeletal connection between two keypoints, the corresponding element is set to 1. This adjacency matrix is ​​a symmetric matrix, i.e. This indicates that the human skeleton is an undirected graph structure.

[0048] Specifically, the key connection points of this invention include:

[0049] Lower limb region: left ankle-left knee, left knee-left hip, right ankle-right knee, right knee-right hip, and left hip-right hip;

[0050] Trunk area: left shoulder-left hip, right shoulder-right hip, and left shoulder-right shoulder;

[0051] Upper limb area: left shoulder-left elbow, left elbow-left wrist, right shoulder-right elbow, right elbow-right wrist.

[0052] The human skeleton graph constructed using this adjacency matrix can explicitly model the local structural dependencies between adjacent keypoints, allowing information to propagate between directly connected keypoints, while information between non-adjacent keypoints must be passed step by step through intermediate nodes. This topology-based modeling approach provides a stable prior knowledge of human structure for subsequent keypoint feature interactions, thereby enhancing the structural consistency and physical plausibility of pose estimation.

[0053] It should be noted that the connections between head keypoints (nose, eyes, ears) were not explicitly included in the adjacency matrix construction. This is because: firstly, head keypoints are spatially close, their visual features are highly correlated, and the convolutional features themselves are sufficient to model their local relationships; secondly, the relative motion amplitude between head keypoints is small, and their structural constraints contribute little to overall pose inference; and thirdly, reducing weak semantic connections helps avoid noise accumulation and over-smoothing issues during graph propagation.

[0054] The graph convolutional encoding process in step S2 is as follows: To explicitly model the topological relationships between key points in the human body, this invention constructs an adjacency matrix based on a predefined human skeleton. A lightweight graph convolutional coding module was designed to process keypoint words. Perform structural encoding, where This represents the number of keypoints in the COCO dataset. It is 17. This represents the terminology dimension. The graph convolutional encoding process can be represented as:

[0055]

[0056]

[0057] in It is a learnable linear projection matrix. Representation layer normalization, These are learnable residual scaling factors used to adaptively adjust the contribution strength of graph convolution information. Specifically, matrix multiplication... First, neighborhood feature aggregation between keypoints is implemented, enabling each keypoint lexical unit to incorporate feature information from its skeleton neighbors. Then, linear projection is used... Feature transformation is performed. Residual connections preserve the semantic information of the original keypoint words, while learningable coefficients are used to transform the semantics. A dynamic balance is achieved between original features and structurally enhanced features. Finally, layer normalization is used to stabilize the feature distribution and improve the numerical stability of the training process.

[0058] Compared to random initialization, graph convolutional network lexicalization based on keypoint priors incorporates the topological information of the human skeleton during the initialization phase, resulting in similar feature representations between adjacent keypoints. This aligns with the anatomical constraints specific to human pose tasks. Secondly, graph convolutional encoding enables the model to utilize the structural relationships between keypoints in the early stages of training, accelerating the model's convergence process. Finally, this initialization method provides a more structure-aware initial representation for subsequent Transformer-Mamba hybrid pose prediction, helping the model to more accurately model complex poses.

[0059] Furthermore, such as Figure 2 As shown, the Transformer layer in step S3 adopts a residual Transformer encoder with a pre-normalized structure, containing a total of L layers. For the l-th layer, its calculation process can be formally represented as:

[0060]

[0061]

[0062] in Representation layer normalization, This indicates a multi-head self-attention module. This represents a feedforward network. This pre-normalization structure performs a normalization operation before the residual connections, which helps alleviate the vanishing gradient problem in deep network training and improves optimization stability.

[0063] The feedforward network consists of two fully connected layers, a nonlinear activation function, and Dropout regularization. Its computational form is as follows:

[0064]

[0065] in , For learnable parameters, This represents the GELU activation function. For the hidden layer dimension of the feedforward network, ,in Scaling ratio Dropout layers randomly zero out some neuron outputs during training, effectively preventing overfitting.

[0066] Multi-head self-attention mechanisms map input features to Each subspace corresponds to a number of attention heads, and attention is calculated independently within each subspace. To improve computational efficiency, this invention employs a fusion projection method, simultaneously generating queries, keys, and values ​​through a single linear layer:

[0067]

[0068] in To fuse the projection matrices, the projection results are then reshaped into a multi-head form:

[0069]

[0070] The calculation process for multi-head self-attention is as follows:

[0071]

[0072]

[0073]

[0074] in For the number of attention heads, Indicates the single-head feature dimension. This is the output projection matrix after multi-head stitching. Scaling factor. This is used to mitigate the numerical instability of inner product values ​​as the dimension increases. Attention weights are normalized using the Softmax function and then weighted and summed with the value vector to obtain the output of each attention head. The outputs of all attention heads are concatenated along the feature dimension, fused using a linear mapping, and then regularized using a Dropout layer.

[0075] To enhance the expressive power of spatial location information, this invention, when employing a sine-full location encoding mode, in addition to explicit location embedding in the input stage, also includes... Before layer self-attention computation, positional encoding is injected again into image block words:

[0076] in This represents the positional encoding matrix. This design continuously reinforces spatial structure information in deep networks, preventing positional information from gradually attenuating during inter-layer propagation. It is worth noting that keypoint terms do not participate in positional encoding injection, as their positional information should be learned rather than explicitly encoded.

[0077] The operation of the key point gating module in step S4 is as follows: Let the word sequence output by the Transformer be... ,in Key word units, These are image block terms. After global dependency modeling by the Transformer layer, keypoint terms have already encoded rich pose information.

[0078] First, global pose descriptions are extracted from keypoint lexics using mean pooling:

[0079]

[0080] in This represents the k-th key word. This is the global pose description vector. This vector aggregates information from all keypoint terms, representing the current model's overall understanding of human pose. By using mean pooling instead of max pooling or attention pooling, the global pose description can evenly reflect the contribution of each keypoint, avoiding any single keypoint dominating the pose representation.

[0081] Then, the gating weights are generated through the gating network:

[0082]

[0083] in and For learnable parameters, This represents the Sigmoid activation function. This is the gate weight vector. The Sigmoid activation function constrains the gate weights to... Within the interval, soft modulation of image patch lexical features is achieved. Each dimension of the gating weight corresponds to a channel of the lexical feature, enabling the model to selectively enhance or suppress features from different semantic channels.

[0084] Finally, channel-level adaptive modulation of image block terms is performed using gated weights:

[0085]

[0086] in The sequence is a block of words in the image, where ⊙ represents element-wise multiplication (broadcasting mechanism). The modulated word sequence is... .

[0087] Keypoint-prior gated interaction networks essentially construct a sample-level channel attention mechanism, enabling the model to conditionally relabel image patch terms based on the current pose context before entering subsequent state space modeling. Unlike typical channel attention, this gate weights are generated from keypoint terms, thus constituting an explicit pose conditional modulation mechanism.

[0088] Compared to directly feeding the Transformer output into subsequent Mamba modules, introducing gating interaction between the two has more significant structural advantages:

[0089] (1) The core feature of Mamba, based on the state-space model, is that it performs sequence modeling in a recursive manner, with the current state being accumulated from the previous state. Therefore, once irrelevant or noisy information enters the state update process, this information will be continuously propagated and superimposed during the recursion process, which may interfere with the modeling of subsequent key points. The gating mechanism performs channel-level filtering of image block words before entering Mamba, which is equivalent to imposing a pose condition constraint on the input sequence, reducing the cumulative effect of irrelevant information in state propagation from the source, and improving the purity and stability of state updates.

[0090] (2) While Transformer excels at modeling global dependencies, its output features remain general global representations without structural constraints on subsequent state recursion. By recalibrating image patch features using gating weights generated from keypoint priors, the sequence input to Mamba can be more focused on semantic channels related to the current pose, thereby improving the state space model's ability to capture key structural patterns. This phased approach of "global modeling first, conditional filtering second, and recursive modeling last" makes the two modeling mechanisms complementary.

[0091] (3) Since the gating weights are sample-level channel vectors, their computational complexity is... Regardless of the number of image patches This allows for conditional control of the entire sequence with almost no additional computational overhead. This lightweight modulation is more efficient than redesigning the cross-module attention structure.

[0092] In step S5, the bidirectional Mamba module employs a bidirectional scanning mechanism, performing forward and reverse recursion respectively:

[0093]

[0094]

[0095] The final output is the sum of the two:

[0096]

[0097] The core idea of ​​Mamba is to transform sequence modeling into recursive computation through a selective state-space mechanism, avoiding quadratic complexity attention matrix operations. In its implementation, the word sequence output by the Transformer is first normalized by layer for feature standardization, and then adjusted to the model dimension of the Mamba module through a linear mapping. Within Mamba, the input sequence First, dynamic parameters required for selective state space are generated through linear mapping, where human pose estimation... : ,in For the gated modulation vector, For the expanded input features ( , As an expansion factor, it is usually set to 2). These correspond to the dynamic components of the state input matrix and the state output matrix, respectively. Set the model hyperparameters (to 128). The time step modulation factor ( (Number of heads).

[0098] Time parameters through Activation function: This ensures that the value is positive, thereby maintaining the stability of the discrete state-space system. To enhance the ability to model local dependencies, the system is recursively analyzed before state-space recursion. Apply channel-wise one-dimensional depthwise convolution: The kernel size is , used to introduce short-range context information.

[0099] Continuous state matrix Stabilization is achieved through exponential mapping: , These are learnable parameters. This parameterization method ensures... Always keep the value negative to ensure system stability. For example... Figure 3 As shown, the core of selective state space lies in the dynamic parameterization of the input. The discretized state update form is as follows:

[0100]

[0101]

[0102] in These are learnable residual scaling parameters. This structure allows the model to dynamically adjust state decay and information writing intensity based on the input content, thereby achieving a selective memory mechanism.

[0103] To improve computational efficiency, Mamba employs a block-based scanning strategy. The sequence of length L is divided into several blocks of length L. The sub-blocks within the block use a lower triangular mask to achieve prefix accumulation: Cross-block state transfer is achieved through a state decay factor between blocks. For the first The cumulative decay parameter sequence within each sub-block Block size; For piecewise summation, prefix accumulation is performed on the input sequence and a lower triangular decay matrix is ​​constructed. Its elements Indicates from within the block position Arrive at the location Cumulative decay factor Fill the upper triangular area To ensure causality, the matrix is ​​exponentialized and used for parallel computation of the output within the block. Cross-block state propagation is achieved through a state decay factor. This low-rank decomposition form breaks down the complete recursion into two parts: intra-block computation and inter-block state propagation, keeping the overall complexity linear. It also supports efficient parallel computing.

[0104] After obtaining the sequence output Y, gated modulation and RMS normalization are introduced: , where ⊙ denotes element-wise multiplication, and RMSNorm is the root mean square layer normalization. Finally, the target dimension is restored through linear mapping.

[0105] The output of the bidirectional Mamba module, after being adjusted for dimensionality using a linear mapping, yields the final Mamba feature representation. By employing bidirectional selective state-space modeling, the model can effectively capture long-range dependencies while maintaining linear time complexity. Compared to self-attention-based global modeling methods, bidirectional Mamba avoids explicit construction. The attention matrix exhibits superior computational efficiency and memory usage when processing long sequences containing keypoint terms and image block terms, while maintaining the ability to model complex spatial structural relationships.

[0106] It should be noted that, unlike some Mamba variants designed for visual tasks, this invention does not employ a multi-directional (e.g., horizontal, vertical, diagonal) scanning strategy, but instead chooses a standard bidirectional sequence modeling structure. This design is primarily based on the following considerations: First, the input is not the original two-dimensional image feature map, but a sequence representation after lexicalization. This representation maps keypoint terms and image block terms uniformly to a one-dimensional sequence space, and its organization is closer to the native one-dimensional sequence modeling application scenario of Mamba. Second, although multi-directional scanning can enhance the ability to model spatial anisotropy in pure two-dimensional feature modeling, its computational overhead and memory usage also increase linearly with the number of scanning directions. Considering that this method has already obtained sufficient spatial relationship expression capabilities through the global interaction of keypoint terms and image block terms and the pre-transformer modeling, the performance gain from further superimposing multi-directional Mamba scanning is relatively limited, while the computational cost increases significantly. Therefore, this invention adopts a standard bidirectional scanning structure, balancing overall performance and computational efficiency. This structure maintains linear complexity. At the same time, it can fully capture sequence-level long-range dependencies and is highly compatible with lexical modeling strategies, thus achieving a better trade-off between efficiency and accuracy.

[0107] The feature fusion strategy in step S6 is as follows: the Transformer layer and the bidirectional Mamba module learn features from different perspectives on the lexical sequence: the Transformer captures explicit associations between lexical units through global self-attention, while Mamba achieves efficient long-range modeling through state-space recursion. Since the lexical sequence is organized with keypoint lexical units first and image patch lexical units last, this invention outputs the first... Keypoint lexical features are extracted from each location, and then concatenated and fused along the feature dimension.

[0108]

[0109]

[0110]

[0111] in and The key-point lexical features output from Transformer and Mamba are respectively used. The concatenation operation is performed along the feature dimension to obtain the fused feature. This fusion strategy allows the final keypoint features to simultaneously incorporate the global correlation information of the Transformer and the long program sequence information of Mamba, achieving a complementary advantage of the two architectures. It is worth noting that image patch terms are discarded during the fusion stage because the human pose estimation task only needs to predict keypoint locations; image patch terms, as contextual information, have already fulfilled their auxiliary role in the feature learning stage.

[0112] The prediction head network in step S6 works as follows: the prediction head network is responsible for converting the fused keypoint lexical features into a human keypoint heatmap. This invention uses a simple and efficient multilayer perceptron structure as the prediction head, first utilizing layer normalization to process the fused keypoint lexical features. Feature standardization is performed, and then the model is transformed into heatmap dimensions through two layers of linear mapping:

[0113]

[0114] in and For learnable parameters, For the hidden layer dimension, and These represent the height and width of the heatmap, respectively. Layer normalization helps stabilize the training process and accelerate convergence. The output of the linear mapping is reshaped and converted to a standard heatmap format.

[0115]

[0116] in This is the final output keypoint heatmap. For the COCO dataset, , , Each spatial location in the heatmap represents the probability distribution of the corresponding key point at that location, and the peak location is the predicted key point coordinate.

[0117] Example

[0118] I. System Configuration and Model Description

[0119] The proposed keypoint-prior-guided global recursive hybrid human pose estimation system is a software program based on modern computers, requiring an NVIDIA graphics card and support for the CUDA computing library and the PyTorch deep learning framework. This invention provides a high-performance human pose estimation network method, training the model using image and labeled data in dataset form, and performing inference and evaluation on test data. The workflow of the system's training and inference phases is as follows: Figure 1 As shown. Once trained, the model can be used for inference and testing, a process consistent with that of conventional deep learning models.

[0120] II. System Implementation Process

[0121] The following describes an example of a complete training and inference process using the system of the present invention.

[0122] The training and testing data used in the examples are both from the Microsoft COCO dataset, which contains 200,000 natural scene images and 250,000 human instances. The Train2017 set contains 57,000 images and 150,000 human instances, while the Val2017 set contains 5,000 images. In addition, the test-dev2017 set contains 20,000 human images. Each human instance is annotated with 17 keypoints, including the nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles.

[0123] The example uses a server running Ubuntu 20.04 to train and test the model. The server is equipped with an Intel(R) Xeon(R) Gold 6226R processor and an NVIDIA GeForce RTX3090 graphics card (with 24GB of video memory). The deep learning framework used in the experiment is PyTorch 2.5.1, with CUDA 12.6 used for computational acceleration.

[0124] This invention designs two model configurations, using HRNet-W32 and HRNet-W48 as the backbone networks, respectively. KTMPose-B uses the HRNet-W32 backbone network, with 2 Transformer layers, 12 attention heads, 2 Mamba layers, 10.34M parameters, and a computational cost of 4.69 GFLOPs. KTMPose-L uses the HRNet-W48 backbone network, with 2 Transformer layers, 8 attention heads, 2 Mamba layers, 19.39M parameters, and a computational cost of 8.74 GFLOPs. The input image resolution is uniformly set to 256×192.

[0125] The network was trained using the AdamW optimizer with an initial learning rate of 1×10⁻³, which was gradually reduced to 1×10⁻³ using a multi-step learning rate scheduler. 5 The model reached optimal performance after 300 training epochs. Batch training was used during the training process to speed up the training process, with a batch size of 200.

[0126] This invention provides a global recursive hybrid human pose estimation method based on keypoint prior guidance, comprising three main components: a convolutional backbone network for feature extraction, a KTM encoder network for keypoint prior lexicalization and hybrid modeling, and a prediction head network responsible for generating a human heatmap. The input image is first processed by the convolutional backbone network to initially extract feature maps, which are then converted into image block lexical sequences through image block embedding. Simultaneously, keypoint lexicals are initialized and encoded by a graph convolutional network with keypoint priors, enabling them to perceive keypoint structures. After concatenation of keypoint lexicals and image block lexicals, they are sequentially passed through a Transformer layer and a keypoint gating module for global dependency modeling and adaptive feature modulation, followed by long-range feature learning with linear complexity by a bidirectional Mamba module. Finally, the keypoint lexical features output from the Transformer layer and the Mamba module are concatenated and fused, and the prediction head network outputs a keypoint heatmap.

[0127] While the specific embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention, and modifications or variations without creative effort are still within the protection scope of the present invention.

Claims

1. A global recursive hybrid human pose estimation method based on keypoint prior guidance, characterized in that, Includes the following steps: Step S1: Use a convolutional backbone network to extract features from the input human image to obtain an image feature map. Then, divide the feature map into multiple image blocks and perform linear projection through image block embedding to generate an image block word sequence. Step S2: Construct a keypoint adjacency matrix based on the topological structure of the human skeleton, and use a graph convolutional network to structurally encode keypoint words to obtain a keypoint word sequence with structure awareness. Step S3: Concatenate the key point word sequence with the image block word sequence to form a complete word sequence, and input the word sequence into the Transformer encoder for global dependency modeling; Step S4: The Transformer output features are subjected to pose condition modulation through the key point gating module. The key point words are used to generate a global pose description and gating weights are generated accordingly. Channel-level adaptive modulation is performed on the image block words. Step S5: Input the gated and modulated word sequence into the bidirectional Mamba module, and perform long-range feature modeling on the sequence through the state space recursion mechanism of forward and reverse scanning; Step S6: Extract key point lexical features from the outputs of the Transformer encoder and the bidirectional Mamba module respectively, perform feature fusion, and output a human body key point heatmap through the prediction head network to achieve human pose estimation.

2. The method of claim 1, wherein, The convolutional backbone network in step S1 is a high-resolution network HRNet, which maintains high-resolution feature representation by connecting multiple sub-networks of different resolutions in parallel and exchanging information across resolutions; specifically, given an input image , of which 3 and Representing the number of channels, initial image height, and width, respectively, the features extracted and output by the backbone network are represented as follows: .

3. The method of claim 1, wherein, The image patch embedding method in step S1 includes: dividing the feature map output by the convolutional backbone network into multiple image patches of a preset size, and mapping each image patch into a fixed-dimensional vector representation through linear projection, thereby constructing an image patch word sequence; specifically, each image patch is converted into a vector representation of a fixed dimension through linear projection. The word element, this process is represented as: where the Rearrange operation rearranges the feature maps from to the form Linear is a linear projection layer, and finally gets the image tile token sequence , is the image tile size.

4. The method according to claim 1, characterized in that, The keypoint adjacency matrix in step S2 is constructed based on the connection relationships of the human skeleton. If two keypoints have a skeletal connection or are the same keypoint, the corresponding element in the adjacency matrix is ​​set to 1; otherwise, it is set to 0, thus forming a symmetric adjacency matrix. The keypoint connections include: Lower limb region: left ankle-left knee, left knee-left hip, right ankle-right knee, right knee-right hip, and left hip-right hip; Trunk area: left shoulder-left hip, right shoulder-right hip, and left shoulder-right shoulder; Upper limb area: left shoulder-left elbow, left elbow-left wrist, right shoulder-right elbow, right elbow-right wrist.

5. The method according to claim 1, characterized in that, In step S2, the graph convolutional network aggregates neighborhood features of keypoint words through an adjacency matrix and combines linear projection and residual connections to encode keypoint structural information, thereby obtaining structure-aware keypoint word features. Specifically, the graph convolutional encoding process is as follows: in Construct an adjacency matrix for a predefined human skeleton. Key word units, Indicates the number of key points. Representing the lexical dimension, It is a learnable linear projection matrix. Representation layer normalization, These are learnable residual scaling factors used to adaptively adjust the contribution strength of graph convolution information.

6. The method of claim 1, wherein, The Transformer encoder in step S3 employs a multi-layer Transformer network with a pre-normalization structure, including a multi-head self-attention module and a feedforward network module, to model global dependencies in the lexical sequence; specifically, it contains L layers. For the l-th layer, the computation process is formally represented as follows: wherein denotes layer normalization, denotes a multi-head self-attention module, denotes a feed-forward network; The feedforward network consists of two fully connected layers, a nonlinear activation function, and Dropout regularization. Its computational form is as follows: wherein , are learnable parameters, denotes a GELU activation function, is the hidden layer dimension of the feedforward network; Multi-head self-attention mechanisms map input features to Each subspace is used, and attention is computed independently within each subspace; a fusion projection approach is employed to simultaneously generate queries, keys, and values ​​through a single linear layer. in To fuse the projection matrices, the projection results are then reshaped into a multi-head form: The calculation process for multi-head self-attention is as follows: in For the number of attention heads, Indicates the single-head feature dimension. This is the output projection matrix after multi-head splicing; scaling factor This is used to mitigate the numerical instability of inner product values ​​as the dimension increases; the attention weights are normalized by the Softmax function and then weighted and summed with the value vector to obtain the output of each attention head; the outputs of all attention heads are concatenated along the feature dimension, fused by linear mapping, and then regularized by the Dropout layer; To enhance the expressive power of spatial location information, when adopting the sine-full location encoding mode, in addition to the explicit location embedding in the input stage, in the second stage... Before layer self-attention computation, positional encoding is injected again into image block words: wherein represents a position encoding matrix.

7. The method of claim 1, wherein, The keypoint gating module in step S4 includes: first, pooling keypoint terms to obtain a global pose description vector; then, generating channel-level gating weights through a gating network; finally, using the gating weights to modulate image block term features channel-by-channel to achieve pose-guided feature enhancement and noise suppression; specifically, let the term sequence output by the Transformer be... ,in Key word units, For image block words; First, global pose descriptions are extracted from keypoint lexics using mean pooling: in This represents the k-th key word. This is the global attitude description vector; Then, the gating weights are generated through the gating network: wherein and are learnable parameters, denotes a Sigmoid activation function, is a gating weight vector; Finally, channel-level adaptive modulation of image block terms is performed using gated weights: wherein is the image block token sequence, denotes element-wise multiplication, and the modulated token sequence is .

8. The method of claim 1, wherein, In step S5, the bidirectional Mamba module recursively models the input sequence using a selective state-space model, performing both forward and backward scans and fusing their outputs to capture long-distance dependencies in the sequence. Specifically, the bidirectional Mamba module employs a bidirectional scanning mechanism, performing forward and backward recursion separately. The final output is the sum of the two: The final Mamba feature representation is obtained. Specifically, the word sequence output by the Transformer is first normalized by layer normalization to standardize its features, and then adjusted to the model dimension of the Mamba module through linear mapping. Inside Mamba, the input sequence First, dynamic parameters required for selective state space are generated through linear mapping, where human pose estimation... : ,in For the gated modulation vector, For the expanded input features, , For expansion factor, These correspond to the dynamic components of the state input matrix and the state output matrix, respectively. For model hyperparameters, The time step modulation factor, Number of heads; Time parameters through Activation function: Ensure it is positive; before recursion in the state space, check... Apply channel-wise one-dimensional depthwise convolution: The kernel size is , used to introduce short-range context information; Continuous state matrix Stabilization by exponential mapping: are learnable parameters; the core of the selective state space is the dynamic parameterization of the input-dependent, the discrete state update form is:​ wherein is a learnable residual scaling parameter.

9. The method of claim 1, wherein, The feature fusion method in step S6 is as follows: key point lexical features are extracted from the output sequence of the Transformer encoder and the output sequence of the bidirectional Mamba module, respectively, and then concatenated and fused along the feature dimension to obtain a fused key point feature representation; specifically, from the first... Keypoint lexical features are extracted from each location, and then concatenated and fused along the feature dimension. wherein and are the keypoint token features output by Transformer and Mamba respectively, the concatenation operation is performed on the feature dimension to obtain the fused feature .

10. The method of claim 1, wherein, The prediction head network in step S6 is a multilayer perceptron structure. It converts the fused keypoint lexical features into a human keypoint heatmap through linear mapping, and determines the coordinates of the human keypoints based on the peak positions of the heatmap. Specifically, a simple and effective multilayer perceptron structure is used as the prediction head. First, layer normalization is applied to the fused keypoint lexical features... Feature standardization is performed, and then the model is transformed into heatmap dimensions through two layers of linear mapping: in and For learnable parameters, For the hidden layer dimension, and These represent the height and width of the heatmap, respectively; the output of the linear mapping is then reshaped and converted into a standard heatmap format. in This is the final output heatmap of key points; For the COCO dataset, , , .

Images