A lightweight high-precision human pose estimation method based on Mamba architecture

Abstract

The application discloses a lightweight high-precision human pose estimation method based on a Mamba architecture, and belongs to the technical field of human pose estimation.The application reduces the calculation cost by designing a lightweight neural network architecture, and simultaneously realizes high-precision human pose estimation.The method constructs a human pose estimation network of the Mamba architecture, and enhances the key point detection capability while ensuring real-time by using a lightweight feature extraction module; in the training stage, the network is trained by using a diversified human pose dataset, so that the adaptability of the network to various poses and scenes is ensured; in the testing stage, real-time human pose estimation is realized by using the lightweight network model; compared with the traditional method, the network model of the application is simple in design, the parameter quantity and the floating point operation number of the network are greatly reduced, the real-time of the model is enhanced, the precision of the pose estimation is significantly improved, and the application is suitable for resource-limited devices or real-time application scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of human posture estimation, and specifically provides a lightweight and high-precision human posture estimation method based on the Mamba architecture. Background Art

[0002] Human pose estimation is the process of reconstructing a person's pose by detecting and locating key points on the human body. This task is one of the major challenges in the field of computer vision and plays a key role in many computer vision applications, such as action recognition, intelligent video surveillance, and human-computer interaction. Real-time and high-precision human pose estimation are key challenges in computer vision. Real-time performance requires the system to process images and generate pose estimates in a very short time, while high-precision requires the system to accurately locate key points on the human body to ensure accurate pose estimation. There is often a trade-off between these two requirements. Striking a balance between high accuracy and real-time processing is a key focus of current technological development.

[0003] Most existing methods for human pose estimation fail to maintain high accuracy while meeting the requirements of real-time processing. For example, VitPose, a simple baseline model based on the Visual Transformer (ViT), is used for human pose estimation. This method introduces the ViT model to replace the traditional convolutional neural network (CNN), demonstrating the potential of the Transformer for human pose estimation. However, while VitPose achieves impressive accuracy, especially on some large-scale datasets, it still faces significant challenges in real-time processing. VitPose's model structure is relatively large and computationally complex. The Transformer model itself requires significant computing resources and memory, especially when the input image resolution is high, where the computational overhead increases significantly. Therefore, while VitPose can achieve high accuracy on some datasets, it struggles to maintain high accuracy while meeting the requirements of real-time processing in real-world applications, especially on resource-constrained devices. Summary of the Invention

[0004] To address the problem that existing human pose estimation methods are difficult to meet the requirements of real-time processing while maintaining high accuracy, the present invention provides a lightweight and high-precision human pose estimation method based on the Mamba architecture, which can significantly improve the detection accuracy of human pose estimation while maintaining the real-time performance of the model.

[0005] The present invention provides a lightweight and high-precision human posture estimation method based on the Mamba architecture, which includes the following steps.

[0006] S1: Construct a lightweight and high-precision human pose estimation network model based on the Mamba architecture; the network model includes an image encoding module, a lightweight feature extraction module, and a regression layer;

[0007] The image encoding module is used to preprocess the input image, including segmenting the image into several image blocks and straightening the image blocks into vector form. Subsequently, the image blocks are projected into a high-dimensional feature space through linear projection to improve the model's ability to express the image. At the same time, a position encoding mechanism is introduced into the image block feature vector to preserve the spatial information of the image, enabling the model to better understand the spatial structure of the human body.

[0008] The lightweight feature extraction module is used to extract features related to human posture from the encoded image blocks. Through a series of efficient computational operations, the lightweight feature extraction module can accurately identify key human body parts and posture information while maintaining real-time performance.

[0009] The regression layer uses the posture features generated by the lightweight feature extraction module to make the final prediction of the human body posture.

[0010] S2: Training phase: using a diverse human pose dataset covering a variety of poses and scenes to train a network with good generalization and adaptability;

[0011] S3: In the testing phase, the detected human body image is input into the trained network model for human body posture estimation.

[0012] By feeding images into a lightweight network structure, the model can quickly process the input data and perform real-time human pose estimation. Whether on low-resource devices or in demanding application scenarios, the model can efficiently complete pose prediction tasks, ensuring accurate real-time detection.

[0013] Furthermore, the image encoding module in S1 performs preprocessing and position encoding on the input image as follows:

[0014] First, the input image data is preprocessed, including slicing the image into several small patches for subsequent feature extraction. These patches are then mapped into a high-dimensional feature space using linear projection to improve the model's representation of the image. Finally, to preserve the spatial position information of the input image, a positional encoding is added to each patch, enabling the network to better capture and understand the spatial structure of human poses.

[0015] Furthermore, in S1, the lightweight feature extraction module is based on the Mamba architecture and uses the lightweight ViM block as the core unit. The lightweight feature extraction module is constructed by stacking 22 ViM blocks, where the ViM block consists of a linear layer, a separable convolution, and four lightweight SSM blocks, with the SSM block as the core.

[0016] The state space model (SSM) block is a recursive model that models dependencies in a sequence by recursively updating the hidden state. The update of the state space model only requires the input at the current moment and the state at the previous moment, so its computational complexity is O(L·d state ), where d state is the dimension of the hidden state, and the global self-attention mechanism of Transformer requires each position to interact with all other positions in the sequence, and the computational complexity grows quadratically, that is, O(L 2 ·d state ). At the same time, GPU-related hardware acceleration algorithms are also designed in the Mamba architecture to improve the inference speed of the model. Therefore, the Mamba architecture improves the accuracy of the model while achieving faster inference speed than traditional models. The scanning process of the original SSM is only one-way modeling, which is obviously inconsistent with the characteristics of visual data, because visual data itself is non-sequential and contains spatial information (such as local texture and global structure). Therefore, in the lightweight feature extraction module, the image blocks are expanded into sequences along four different traversal paths, and four SSM blocks process each image block sequence in parallel, and then merge all the results. The four SSM blocks use complementary traversal paths, so that the information of each pixel can effectively integrate contextual information from different directions, thereby promoting the establishment of a global receptive field while maintaining lightweight computational overhead.

[0017] Furthermore, during the training phase, the image encoding module performs the following steps:

[0018] 1) Divide the input image data into multiple small image blocks to extract features. This operation can effectively reduce computational complexity while still capturing important visual information.

[0019] 2) The segmented image blocks are mapped into a high-dimensional feature space through linear projection. Linear projection helps convert image information into feature representations suitable for further processing, enhancing the expressive power of the model.

[0020] 3) Position encoding is added to each image block. By fusing the position encoding with the projection features, the spatial position information of each block in the image is preserved. Position encoding enables the model to understand the relative position and structure of each part in the image, thereby improving the perception of spatial relationships of human posture.

[0021] Furthermore, during the training phase, the lightweight feature extraction module performs the following steps:

[0022] 1) The encoded image block is input into a lightweight feature extraction module, which consists of 22 lightweight ViM blocks stacked together; these ViM blocks are responsible for deep extraction of image features; in each ViM block, the image block first passes through a linear layer and a separable convolutional layer to extract deep features.

[0023] 2) Then, these features are input into four SSM blocks, each of which extracts image features related to human posture in parallel along its respective traversal path.

[0024] 3) Finally, the features extracted by the four SSM blocks are merged and then fused with the image block features obtained by the image encoding module through residual connection. The fused features are passed to the regression layer to generate the final human pose estimation result.

[0025] Furthermore, step S3 includes the following steps:

[0026] Input the human body image to be detected; after the human body image to be detected is input into the image encoding module, it is divided into several image blocks. Then, the image blocks are straightened into vector form and projected into the high-dimensional feature space through the linear layer. Finally, position coding is added to each image block; the image blocks with position coding are input into the lightweight feature extraction module to extract features related to human posture; the feature representation obtained by the lightweight feature extraction module is input into the regression layer to obtain the predicted human posture estimation result.

[0027] The beneficial effects described in the present invention are as follows: the method of the present invention is concise and efficient in design, and adopts a lightweight design strategy in the feature extraction module compared with traditional large-scale models, such as deep learning models based on convolutional neural networks or complex structures such as Vision Transformer. This concise structure not only reduces the complexity of the model, but also greatly reduces the number of parameters and floating-point operations (FLOPs) of the model through a reasonable network architecture. This enables the method of the present invention to maintain a low computational overhead and achieve real-time performance when processing complex posture estimation tasks, and is suitable for resource-constrained devices or application scenarios such as real-time video monitoring, motion capture, AR / VR interaction, etc. that have extremely high requirements for response speed. In addition, although the method of the present invention has been simplified, it still maintains excellent performance in posture estimation accuracy. By introducing an effective feature extraction module and a reasonable training strategy, the method of the present invention exhibits high robustness in different backgrounds and posture changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1It is a schematic flow diagram of the method of the present invention;

[0029] Figure 2 This is a schematic diagram of the structure of a lightweight and high-precision human posture estimation network model based on the Mamba architecture constructed by the present invention;

[0030] Figure 3 This is a schematic diagram of the specific structure of a ViM block in the lightweight feature extraction module;

[0031] Figure 4 It is a specific traversal path diagram of four SSM blocks in a ViM block. DETAILED DESCRIPTION

[0032] The technical solutions of the present invention are described in detail below through examples, but the protection scope of the present invention is not limited to the examples.

[0033] like Figure 1 As shown, the lightweight and high-precision human pose estimation method based on the Mamba architecture described in the present invention improves the basic human pose estimation process, uses an image encoding module and a lightweight feature extraction module to maintain the real-time performance of the model, and improves the accuracy of human pose estimation. Specifically, it includes the following steps:

[0034] S1: Build a lightweight and high-precision human pose estimation network model based on the Mamba architecture;

[0035] S2: Training phase: using a diverse human pose dataset covering a variety of poses and scenes to train a network with good generalization and adaptability;

[0036] S3: During the testing phase, images are fed into a lightweight network structure. The model rapidly processes the input data and performs real-time human pose estimation. Whether on low-resource devices or in demanding scenarios, the model efficiently completes pose prediction tasks, ensuring accurate real-time detection.

[0037] The lightweight and high-precision human posture estimation network model based on the Mamba architecture constructed in this invention is as follows: Figure 2 As shown, it includes an image encoding module, a lightweight feature extraction module and a regression layer.

[0038] In the present invention, when training the network model, the loss function of the network model in S2 is the MSE loss function. When training the network model, the operation of each module is as follows:

[0039] (1) When training the network model, the image encoding module performs the following steps:

[0040] S2.1.1. First, pre-process the input image data. Slice the image into several small blocks for subsequent feature extraction, where H and W represent the height and width of the input image; C represents the number of channels of the input image, usually 3.

[0041] S2.1.2. Subsequently, linear projection is used to map these image blocks into a high-dimensional feature space to improve the model's ability to express the image. The expression is as follows:

[0042] I patches =Projecion(Flatten(Conv(I)))

[0043] Among them, Conv(I) represents the convolution operation on the input image to achieve the purpose of slicing. The convolution kernel size of the convolution layer is (8,8) and the stride is 8 to ensure that the image blocks do not overlap. Subsequently, the image block is straightened into a vector form through the Flatten(·) operation. Finally, the image block is projected into a high-dimensional feature space through linear projection(·) to obtain the image block feature vector It contains N image blocks, where the feature dimension of each image block is Dim.

[0044] S2.1.3. At the same time, in order to preserve the spatial position information of the input image, the image encoding module adds a position code to the feature vector of each image block, so that the network can better capture and understand the spatial structural relationship of the human body posture. The method uses sine-cosine position coding, which is expressed as follows:

[0045]

[0046]

[0047] Where PE(pos,2i) and PE(pos,2i+1) represent the encoding values ​​for the even and odd dimensions, respectively, corresponding to the posth image block. pos represents the posth image block. i represents the dimension index, referring to the i-th dimension in the position encoding vector. Dim represents the total dimensionality of the encoding vector, which is equal to the size of the image block's projected features.

[0048] S2.1.3. The position code can be obtained by this coding formula, and its dimension is The encoded image block feature vector can be obtained by adding the position code to the image block feature vector.

[0049] (2) When training the network model, the lightweight feature extraction module performs the following steps:

[0050] S2.2.1. The encoded image block is input into the lightweight feature extraction module. In each ViM block, the image block first passes through a linear layer and a separable convolutional layer to extract deep features.

[0051] Since the lightweight feature extraction module consists of multiple ViM blocks with the same configuration, the ViM blocks perform the following steps, such as Figure 3 As shown, the main branch is the route through SSM on the right, which first performs a linear projection on the image block feature vector, and the expression is as follows:

[0052] I proj =Linear(I patches )

[0053] Where Linear(·) represents a linear projection operation. proj Represents the new image block feature vector after projection, and its dimension is Indicates I proj Contains N image blocks, where the feature dimension of each image block is E.

[0054] Then, the new image block feature vector I proj Convert it to a two-dimensional feature matrix and use separable convolution to further extract features. The expression is as follows:

[0055] I conv =SiLU(DWConv(I proj ))

[0056] Among them, DWConv(·) represents separable convolution, SiLu(·) represents activation function. proj I obtained by extracting features conv Represents the deep features of the image block, and its dimension is Get I conv Then it is converted into a one-dimensional feature vector to be input into the SSM block.

[0057] S2.2.2. Subsequently, the extracted deep features are input into the SSM block, and each SSM block extracts image features related to human posture in parallel along its own traversal path; the specific traversal path diagram of the four SSM blocks in a ViM block is as follows: Figure 4 shown.

[0058] The specific SSM block performs the following steps:

[0059] The features extracted from the image block in S2.2.1 are passed through a linear layer to obtain the dynamic time step feature Δ and the features B and C of the state space model, which are expressed as follows:

[0060] Δ,B,C=Split(Linear(Iconv ))

[0061] Among them, the dimensions of each variable are: E is the dimension of the image block feature vector, N is the number of image blocks, and D state is the dimension size of the SSM internal state.

[0062] At the same time, the features A and D of the state space model (SSM) are initialized during model initialization, and the dimensions are:

[0063] Through the obtained Δ, A, B, C and D, the feature extraction can be performed through the selective scanning operation (Selective Scan) of the SSM block. The expression is as follows:

[0064] y i =SelectiveScan(I conv ,Δ,A,B,C,D)

[0065] Where i is the i-th SSM block, y i is the output of the i-th SSM block, with dimension The scanning direction information of SSM will be integrated into the matrix B to achieve different traversal directions.

[0066] S2.2.3. Finally, the features extracted by the four SSM blocks are merged and fused with the features of the original image block (the image block obtained by the image encoding module) through residual connection. The fused features are passed to the regression layer to generate the human posture estimation results.

[0067] In the first ViM block, the output of the SSM blocks in four directions is obtained. Finally, the four output results are fused to obtain the final output Y l The dimension is Subsequently, the image block features obtained by the image coding module are fused with the SSM output through the residual connection, completing the feature extraction process of a ViM block. The expression is as follows:

[0068] F l =Y l +Linear(I conv )

[0069] Among them, F l is the output of the lth ViM block, Y l Represents the final fusion output of the SSM block in the lth ViM block; the feature F extracted by the ViM blocks of a total of L layers out The final regression layer is input to obtain the human pose estimation result.

[0070] (3) When training the network model, the regression layer performs the following steps:

[0071] The features obtained by the lightweight feature extraction module are input into the regression layer to regress the final pose estimation result. The expression is as follows:

[0072] Heatmap j =Deconv(F out ), j=1,2,…,J

[0073] Among them, Deconv(·) represents the deconvolution operation, and the posture estimation result obtained is a heat map H' and W' are manually set heatmap size hyperparameters, J represents the number of human joints defined manually, Heatmap j Represents the heat map of the j-th joint point.

[0074] In the testing phase of S3, step S3 includes the following steps: inputting a human body image to be detected; inputting the human body image into an image encoding module, which divides the image into several image blocks, and straightens the image blocks into vector form, and projects them into a high-dimensional feature space through a linear layer, and then adds position coding to each image block to retain spatial position information; inputting the feature vector with position coding into a lightweight feature extraction module to extract features related to human body posture; the module further extracts features through multiple lightweight ViM blocks to obtain high-quality human body feature representation; the high-quality human body feature representation obtained by the lightweight feature extraction module is input into the regression layer to obtain a predicted human body posture estimation result.

[0075] Despite its simplification, the proposed method maintains excellent performance in pose estimation accuracy. By introducing an effective feature extraction module and a rational training strategy, the proposed method demonstrates high robustness across diverse backgrounds and pose variations. A comparison of the proposed method with existing methods in Table 1 shows that the proposed method requires fewer parameters and computational complexity than existing methods, while also outperforming them in performance.

[0076] Table 1 Comparative Examples of the Present Invention and the Existing Method

[0077] Model Parameters (M) Computational Amount (Gflops) ResNet-50 345.5 ResNet-152 6812.7 ViTPose-S 225.3 The present invention 6.45.2

[0078] In summary, the present invention reduces computational costs by designing a lightweight neural network architecture while achieving high-precision human pose estimation. The method of the present invention first constructs a human pose estimation network based on the Mamba architecture, utilizing a lightweight feature extraction module to enhance key point detection capabilities while ensuring real-time performance. During the training phase, the network is trained using a diverse human pose dataset to ensure its adaptability to various poses and scenarios. During the testing phase, real-time human pose estimation is achieved using a lightweight network model. Compared with traditional methods, the network model of the present invention is simple in design, significantly reducing the number of network parameters and floating-point operations. This significantly improves the accuracy of pose estimation while enhancing the real-time performance of the model, making it suitable for resource-constrained devices or real-time application scenarios.

[0079] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to the form and details without departing from the spirit and scope of the present invention.

Claims

1. A lightweight and high-precision human posture estimation method based on the Mamba architecture, characterized in that: The steps include: S1. Construct a lightweight and high-precision human posture estimation network model based on the Mamba architecture; the network model includes an image encoding module, a lightweight feature extraction module and a regression layer; The image encoding module is used to preprocess and position encode the input image; the lightweight feature extraction module is used to extract features related to human posture from the encoded image; The regression layer predicts human posture using posture features generated by the lightweight feature extraction module; The lightweight feature extraction module is based on the Mamba architecture and uses the ViM block as the core unit. It is constructed by stacking 22 ViM blocks. The ViM block consists of a linear layer, a separable convolution, and four lightweight SSM blocks. The four lightweight SSM blocks use complementary traversal paths to enable each pixel to integrate contextual information from different directions. S2, training phase, using a diverse human posture dataset covering various postures and scenes to train the network model; S3, testing phase, inputs the human body image to be detected into the trained network model for human body posture estimation.

2. The human body posture estimation method according to claim 1, wherein The image encoding module in S1 preprocesses and positions the input image as follows: First, the input image data is preprocessed, including slicing the image into several image blocks; Then, the image blocks are straightened into vector form and mapped into high-dimensional feature space using linear projection method; Finally, a position code is added to each image block.

3. A lightweight and high-precision human posture estimation method based on Mamba architecture according to claim 1, characterized in that: During the training phase in S2, the image encoding module performs the following steps: 1) Divide the input image data into multiple image blocks; 2) Straighten the image block into a vector form and map it to a high-dimensional feature space through linear projection to obtain the projection features of the image block; 3) Add position code to each image block; 4) Add the position code to the image block feature vector to obtain the encoded image block feature vector.

4. A lightweight and high-precision human posture estimation method based on Mamba architecture according to claim 3, characterized in that: During the training phase in S2, the lightweight feature extraction module performs the following steps: 1) Input the encoded image block into the lightweight feature extraction module. In each ViM block, the image block first passes through a linear layer and a separable convolutional layer to extract deep features; 2) Subsequently, the extracted deep features are input into the SSM block, and each SSM block extracts image features related to human posture in parallel along its respective traversal path; 3) Finally, the features extracted by the four SSM blocks are combined and fused with the image block features obtained by the image encoding module through residual connection. The fused features are passed to the regression layer to generate the human pose estimation results.

5. A lightweight and high-precision human posture estimation method based on Mamba architecture according to claim 1, characterized in that: Step S3 includes the following steps: Input the human body image to be detected; After the human image to be detected is input into the image encoding module, it is divided into several image blocks. Then, the image blocks are straightened into vector form and projected into a high-dimensional feature space through a linear layer. Finally, a position code is added to each image block. The image blocks with position encoding are input into the lightweight feature extraction module to extract features related to human posture; The feature representation obtained by the lightweight feature extraction module is input into the regression layer to obtain the predicted human posture estimation result.

Images