Lightweight high-precision human body posture estimation method based on Mmba architecture

Abstract

The invention discloses a lightweight high-precision human body posture estimation method based on a Mama framework, belongs to the technical field of human body posture estimation, and aims to reduce the calculation cost and realize high-precision human body posture estimation by designing a lightweight neural network framework. The method comprises the following steps: firstly, constructing a human body posture estimation network of a Mama framework, and enhancing the key point detection capability while ensuring the real-time performance by using a lightweight feature extraction module; in the training stage, a diversified human body posture data set is used for training the network, and the adaptability of the network to various postures and scenes is ensured; in the testing stage, real-time human body posture estimation is achieved through the lightweight network model, compared with a traditional method, the network model is simple in design, the parameter quantity and the floating point operand of the network are greatly reduced, the real-time performance of the model is enhanced, meanwhile, the posture estimation precision is remarkably improved, and the real-time performance of the model is improved. The method is suitable for resource-limited equipment or real-time application scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of human body posture estimation, and specifically is a lightweight and high-precision human body posture estimation method based on Mamba architecture. Background Art

[0002] Human pose estimation refers to the process of reconstructing human pose by detecting and locating key points on the human body. This task is one of the important challenges in the field of computer vision and plays a key role in many computer vision applications, such as action recognition, intelligent video surveillance, and human-computer interaction. The real-time and high-precision human pose estimation are important challenges in computer vision. Real-time requires that the system can process images and give pose estimation results in a very short time, while high precision requires that the system can accurately locate the key points of the human body to ensure the accuracy of pose estimation. There is often a trade-off between the two. How to meet the needs of real-time processing while ensuring high precision is the focus of current technological development.

[0003] Most of the existing methods for human pose estimation cannot meet the requirements of real-time processing while maintaining high accuracy. For example, VitPose, a simple baseline model based on visual Transformer (ViT), is used for human pose estimation. This method introduces the ViT model to replace the traditional convolutional neural network (CNN), demonstrating the potential of Transformer in human pose estimation tasks. However, although VitPose has achieved relatively outstanding performance in accuracy, especially in some large-scale datasets, it still faces significant challenges in real-time processing. The model structure of VitPose is relatively large and the computational complexity is high. The Transformer model itself has a large demand for computing resources and memory, especially when the resolution of the input image is high, the computational overhead will increase significantly. Therefore, although VitPose can achieve high accuracy on some datasets, it is difficult to meet the requirements of real-time processing while maintaining high accuracy in actual application scenarios, especially on resource-constrained devices. Summary of the invention

[0004] In order to solve the problem that existing human posture estimation methods are difficult to meet the requirements of real-time processing while maintaining high accuracy, the present invention provides a lightweight and high-precision human posture estimation method based on the Mamba architecture, which can significantly improve the detection accuracy of human posture estimation while maintaining the real-time performance of the model.

[0005] The present invention discloses a lightweight and high-precision human body posture estimation method based on Mamba architecture, which comprises the following steps.

[0006] S1: Construct a lightweight and high-precision human posture estimation network model based on the Mamba architecture; the network model includes an image encoding module, a lightweight feature extraction module and a regression layer;

[0007] The image encoding module is used to pre-process the input image, including dividing the image into a number of image blocks, straightening the image blocks into vector form, and then projecting the image blocks into a high-dimensional feature space through linear projection to improve the model's ability to express the image; at the same time, a position encoding mechanism is introduced into the image block feature vector to retain the spatial information of the image, so that the model can better understand the spatial structure of the human body;

[0008] The lightweight feature extraction module is used to extract features related to human posture from the encoded image blocks. Through a series of efficient computing operations, the lightweight feature extraction module can accurately identify key parts and posture information of the human body while maintaining real-time performance;

[0009] The regression layer uses the posture features generated by the lightweight feature extraction module to make a final prediction of the human body posture.

[0010] S2: In the training phase, a diverse human posture dataset covering a variety of postures and scenes is used for training, thereby obtaining a trained network with good generalization ability and adaptability;

[0011] S3: In the testing phase, the detected human image is input into the trained network model for human posture estimation.

[0012] By feeding the image into a lightweight network structure, the model can quickly process the input data and perform real-time human posture estimation. Whether on low-resource devices or in demanding application scenarios, the model can efficiently complete the posture prediction task and ensure accurate real-time detection.

[0013] Furthermore, the method in which the image encoding module in S1 performs preprocessing and position encoding on the input image is as follows:

[0014] First, the input image data is preprocessed, including slicing the image into several small image blocks for subsequent feature extraction. Then, these image blocks are mapped into a high-dimensional feature space using a linear projection method to improve the model's ability to express images. Finally, in order to retain the spatial position information of the input image, position encoding is added to each image block, so that the network can better capture and understand the spatial structural relationship of human posture.

[0015] Furthermore, in S1, the lightweight feature extraction module is based on the Mamba architecture and uses a lightweight ViM block as a core unit. The lightweight feature extraction module is constructed by stacking 22 ViM blocks, wherein the ViM block consists of a linear layer, a separable convolution, and four lightweight SSM blocks, with the core being the SSM block.

[0016] The state space model (SSM) block is a recursive model that models dependencies in a sequence by recursively updating the hidden state. The update of the state space model only requires the input at the current moment and the state at the previous moment, so its computational complexity is O(L·d state ), where d state is the dimension of the hidden state, and the global self-attention mechanism of Transformer requires each position to interact with all other positions in the sequence, and the computational complexity grows quadratically, that is, O(L 2 ·d state ). At the same time, GPU-related hardware acceleration algorithms are designed in the Mamba architecture to improve the inference speed of the model. Therefore, the Mamba architecture improves the model accuracy while having a faster inference speed than traditional models. The scanning process of the original SSM is only one-way modeling, which is obviously not in line with the characteristics of visual data, because visual data itself is non-sequential and contains spatial information (such as local texture and global structure). Therefore, in the lightweight feature extraction module, the image blocks are expanded into sequences along four different traversal paths, and four SSM blocks process each image block sequence in parallel, and then merge all the results. The four SSM blocks use complementary traversal paths, so that the information of each pixel can effectively integrate contextual information from different directions, thereby facilitating the establishment of a global receptive field while maintaining lightweight computational overhead.

[0017] Furthermore, during the training phase, the image encoding module performs the following steps:

[0018] 1) Divide the input image data into multiple small image blocks to extract features from the image. This operation can effectively reduce the computational complexity while still capturing important visual information.

[0019] 2) The segmented image blocks are mapped to a high-dimensional feature space through linear projection. Linear projection helps to convert image information into feature representations suitable for further processing and enhance the expressive power of the model.

[0020] 3) Add position coding to each image block, and fuse the position coding with the projection features to retain the spatial position information of each block in the image. Position coding enables the model to understand the relative position and structure of each part of the image, thereby improving the ability to perceive the spatial relationship of human posture.

[0021] Furthermore, during the training phase, the lightweight feature extraction module performs the following steps:

[0022] 1) The encoded image block is input into a lightweight feature extraction module, which is composed of 22 lightweight ViM blocks stacked together; these ViM blocks are responsible for deep extraction of image features; in each ViM block, the image block first passes through a linear layer and a separable convolutional layer to extract deep features.

[0023] 2) Then, these features are input into four SSM blocks, each of which extracts image features related to human posture in parallel along its respective traversal path.

[0024] 3) Finally, the features extracted by the four SSM blocks will be merged and then fused with the image block features obtained by the image encoding module through residual connection. The fused features will be passed to the regression layer to generate the final human posture estimation result.

[0025] Furthermore, step S3 comprises the following steps:

[0026] Input the human body image to be detected; after the human body image to be detected is input into the image encoding module, it is divided into several image blocks, then the image blocks are straightened into vector form, projected into the high-dimensional feature space through the linear layer, and finally, each image block is added with a position code; the image block with the position code is input into the lightweight feature extraction module to extract features related to human body posture; the feature representation obtained by the lightweight feature extraction module is input into the regression layer to obtain the predicted human body posture estimation result.

[0027] The beneficial effects described in the present invention are as follows: the method of the present invention is concise and efficient in design, and adopts a lightweight design strategy in the feature extraction module compared with traditional large-scale models, such as deep learning models based on convolutional neural networks or complex structures such as Vision Transformer. This concise structure not only reduces the complexity of the model, but also greatly reduces the number of parameters and floating-point operations (FLOPs) of the model through a reasonable network architecture. This enables the method of the present invention to maintain a low computational overhead and achieve real-time performance when processing complex posture estimation tasks, and is suitable for resource-constrained devices or application scenarios such as real-time video monitoring, motion capture, AR / VR interaction, etc. that require extremely high response speed. In addition, although the method of the present invention has been simplified, it still maintains excellent performance in posture estimation accuracy. By introducing an effective feature extraction module and a reasonable training strategy, the method of the present invention exhibits high robustness in different backgrounds and posture changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1It is a schematic flow diagram of the method of the present invention;

[0029] Figure 2 It is a structural schematic diagram of a lightweight and high-precision human posture estimation network model based on the Mamba architecture constructed by the present invention;

[0030] Figure 3 It is a detailed structural diagram of a ViM block in the lightweight feature extraction module;

[0031] Figure 4 It is a specific traversal path diagram of four SSM blocks in a ViM block. DETAILED DESCRIPTION

[0032] The technical solution of the present invention is described in detail below through embodiments, but the protection scope of the present invention is not limited to the embodiments.

[0033] like Figure 1 As shown, the lightweight and high-precision human posture estimation method based on the Mamba architecture described in the present invention improves the basic human posture estimation process, uses an image encoding module and a lightweight feature extraction module to maintain the real-time performance of the model, and improves the accuracy of human posture estimation. Specifically, the method includes the following steps:

[0034] S1: Build a lightweight and high-precision human posture estimation network model based on the Mamba architecture;

[0035] S2: In the training phase, a diverse human posture dataset covering a variety of postures and scenes is used for training, thereby obtaining a trained network with good generalization ability and adaptability;

[0036] S3: In the testing phase, the image is input into a lightweight network structure, and the model can quickly process the input data and perform real-time human posture estimation. Whether on low-resource devices or in demanding application scenarios, the model can efficiently complete the posture prediction task and ensure accurate real-time detection.

[0037] The lightweight and high-precision human posture estimation network model based on the Mamba architecture constructed by the present invention is as follows: Figure 2 As shown, it includes an image encoding module, a lightweight feature extraction module and a regression layer.

[0038] In the present invention, when training the network model, the loss function for training the network model in S2 uses the MSE loss function. When training the network model, the operation of each module is as follows:

[0039] (I) When training the network model, the image encoding module performs the following steps:

[0040] S2.1.1. First, preprocess the input image data. Slice the image into several small blocks for subsequent feature extraction, where H and W represent the height and width of the input image; C represents the number of channels of the input image, usually 3.

[0041] S2.1.2. Then, linear projection is used to map these image blocks into a high-dimensional feature space to improve the model's ability to express images. The expression is as follows:

[0042] I patches =Projecion(Flatten(Conv(I)))

[0043] Among them, Conv(I) represents the convolution operation on the input image to achieve the purpose of slicing. The convolution kernel size of the convolution layer is (8,8) and the step size is 8 to ensure that the image blocks do not overlap. Then, the image block is straightened into a vector form through the Flatten(·) operation. Finally, the image block is projected into a high-dimensional feature space through linear projection Projection(·) to obtain the image block feature vector It contains N image blocks, each of which has a feature dimension of Dim.

[0044] S2.1.3. At the same time, in order to retain the spatial position information of the input image, the image encoding module adds position encoding to the feature vector of each image block, so that the network can better capture and understand the spatial structural relationship of the human body posture. The method uses sine-cosine position encoding, and the expression is as follows:

[0045]

[0046]

[0047] Among them, PE(pos,2i) and PE(pos,2i+1) represent the encoding values ​​of the even and odd dimensions corresponding to the posth image block, respectively. pos represents the posth image block. i represents the dimension index, which refers to the i-th dimension in the position encoding vector. Dim represents the total dimension size of the encoding vector, and its dimension size is equal to the size of the image block projection feature.

[0048] S2.1.3. The position encoding can be obtained through this encoding formula, and its dimension is The encoded image block feature vector can be obtained by adding the position code to the image block feature vector.

[0049] (ii) When training the network model, the lightweight feature extraction module performs the following steps:

[0050] S2.2.1. The encoded image block is input into the lightweight feature extraction module. In each ViM block, the image block first passes through a linear layer and a separable convolutional layer to extract deep features.

[0051] Since the lightweight feature extraction module consists of multiple ViM blocks with the same configuration, the ViM block performs the following steps, such as Figure 3 As shown, the main branch is the route through SSM on the right, which first performs linear projection on the image block feature vector, and the expression is as follows:

[0052] I proj =Linear(I patches )

[0053] Where Linear(·) represents a linear projection operation. proj Represents the new image block feature vector after projection, and its dimension is Indicates I proj It contains N image blocks, each of which has a feature dimension of E.

[0054] Then, the new image block feature vector I proj Convert it to a two-dimensional feature matrix and use separable convolution to further extract features. The expression is as follows:

[0055] I conv =SiLU(DWConv(I proj ))

[0056] Among them, DWConv(·) represents separable convolution, SiLu(·) represents activation function. proj I obtained by extracting features conv Represents the deep features of the image block, and its dimension is Get I conv Then it is converted into a one-dimensional feature vector to be input into the SSM block.

[0057] S2.2.2. Then, the extracted deep features are input into the SSM block, and each SSM block extracts image features related to human posture in parallel along its respective traversal path; the specific traversal path diagram of the four SSM blocks in a ViM block is as follows: Figure 4 shown.

[0058] The specific SSM block performs the following steps:

[0059] The features extracted from the image block in S2.2.1 are passed through a linear layer to obtain the dynamic time step feature Δ and the features B and C of the state space model, as shown below:

[0060] Δ,B,C=Split(Linear(Iconv ))

[0061] Among them, the dimensions of each variable are: E is the dimension of the image block feature vector, N is the number of image blocks, and D state is the dimension size of the SSM internal state.

[0062] At the same time, when the model is initialized, the features A and D of the state space model (SSM) are initialized, and the dimensions are:

[0063] Through the obtained Δ, A, B, C and D, the feature extraction can be performed through the selective scanning operation (Selective Scan) of the SSM block, and the expression is as follows:

[0064] y i =SelectiveScan(I conv ,Δ,A,B,C,D)

[0065] Among them, i is the i-th SSM block, y i is the output of the i-th SSM block, with dimension The scanning direction information of SSM will be integrated into the matrix B to achieve different traversal directions.

[0066] S2.2.3. Finally, the features extracted by the four SSM blocks are combined and fused with the features of the original image block (the image block obtained by the image encoding module) through residual connection. The fused features are passed to the regression layer to generate the human posture estimation results.

[0067] In the lth ViM block, the output of the SSM blocks in four directions is obtained Finally, the four output results are combined to get the final output Y l The dimension is Subsequently, the image block features obtained by the image coding module are fused with the SSM output through the residual connection, completing the feature extraction process of a ViM block. The expression is as follows:

[0068] F l =Y l +Linear(I conv )

[0069] Among them, F l is the output of the lth ViM block, Y l represents the final fused output of the SSM block in the lth ViM block; the feature F extracted by the ViM blocks of a total of L layers out The final regression layer is input to obtain the human posture estimation result.

[0070] (III) When training the network model, the regression layer performs the following steps:

[0071] The features obtained by the lightweight feature extraction module are input into the regression layer to regress the final posture estimation result. The expression is as follows:

[0072] Heatmap j =Deconv(F out ), j = 1, 2, ..., J

[0073] Among them, Deconv(·) represents the deconvolution operation, and the posture estimation result obtained is a heat map H' and W' are manually set heatmap size hyperparameters, J represents the number of human joints defined manually, Heatmap j Represents the heat map of the j-th joint point.

[0074] In the test phase in S3, step S3 includes the following steps: inputting a human body image to be detected; inputting the human body image into an image encoding module, which divides the image into several image blocks, and straightens the image blocks into vector form, and projects them into a high-dimensional feature space through a linear layer, and then adds a position code to each image block to retain the spatial position information; inputting the feature vector with the position code into a lightweight feature extraction module to extract features related to human body posture; the module further extracts features through multiple lightweight ViM blocks to obtain high-quality human body feature representation; the high-quality human body feature representation obtained by the lightweight feature extraction module is input into the regression layer to obtain a predicted human body posture estimation result.

[0075] Although the method of the present invention has been simplified, it still maintains excellent performance in posture estimation accuracy. By introducing an effective feature extraction module and a reasonable training strategy, the method of the present invention shows high robustness in different backgrounds and posture changes. From the comparison between the present invention and the existing method in Table 1, it can be seen that the present invention is lower in parameter quantity and computational complexity than the existing method, and at the same time, the performance is better than the existing method.

[0076] Table 1 Comparative Examples of the Present Invention and the Existing Method

[0077] Model Parameters (M) Computation (Gflops) ResNet-50 345.5 ResNet-152 6812.7 ViTPose-S 225.3 The present invention 6.45.2

[0078] In summary, the present invention reduces the computational cost by designing a lightweight neural network architecture, while achieving high-precision human posture estimation. The method of the present invention first constructs a human posture estimation network of the Mamba architecture, and uses a lightweight feature extraction module to enhance the key point detection capability while ensuring real-time performance; in the training phase, a diversified human posture data set is used to train the network to ensure its adaptability to various postures and scenes; in the testing phase, real-time human posture estimation is achieved through a lightweight network model. Compared with the traditional method, the network model of the present invention is concise in design, greatly reducing the number of network parameters and floating-point operations, while enhancing the real-time performance of the model, significantly improving the accuracy of posture estimation, and is suitable for resource-constrained devices or real-time application scenarios.

[0079] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes in form and details may be made without departing from the spirit and scope of the present invention.

Claims

1. A lightweight and high-precision human posture estimation method based on Mamba architecture, characterized in that: The steps include: S1. Constructing a lightweight and high-precision human posture estimation network model based on the Mamba architecture; the network model includes an image encoding module, a lightweight feature extraction module and a regression layer; The image encoding module is used to preprocess and position encode the input image; the lightweight feature extraction module is used to extract features related to human posture from the encoded image; The regression layer predicts human body posture using posture features generated by the lightweight feature extraction module; S2, training phase, using a diverse human posture dataset covering various postures and scenes to train the network model; S3, testing phase, input the human image to be detected into the trained network model for human posture estimation.

2. The method for estimating human body posture according to claim 1, characterized in that: The method for the image encoding module in S1 to preprocess and position encode the input image is as follows: First, the input image data is preprocessed, including slicing the image into several image blocks; Then, the image blocks are straightened into vector form and mapped into high-dimensional feature space using linear projection method; Finally, a position code is added to each image block.

3. According to a lightweight and high-precision human body posture estimation method based on Mamba architecture according to claim 1, it is characterized in that: In S1: The lightweight feature extraction module is based on the Mamba architecture, uses the ViM block as the core unit, and is constructed by stacking 22 ViM blocks; the ViM block consists of a linear layer, a separable convolution, and four lightweight SSM blocks.

4. According to a lightweight and high-precision human posture estimation method based on Mamba architecture according to claim 1, it is characterized in that: In the training phase in S2, the image encoding module performs the following steps: 1) Divide the input image data into multiple image blocks; 2) Straighten the image block into a vector form and map it to a high-dimensional feature space through linear projection to obtain the projection features of the image block; 3) Add position code to each image block; 4) Adding the position code to the image block feature vector can obtain the encoded image block feature vector.

5. According to the lightweight and high-precision human body posture estimation method based on Mamba architecture as claimed in claim 3, it is characterized in that: In the training phase in S2, the lightweight feature extraction module performs the following steps: 1) Input the encoded image block into the lightweight feature extraction module. In each ViM block, the image block first passes through a linear layer and a separable convolutional layer to extract deep features; 2) Subsequently, the extracted deep features are input into the SSM block, and each SSM block extracts image features related to human posture in parallel along its respective traversal path; 3) Finally, after the features extracted by the four SSM blocks are combined, they are fused with the image block features obtained by the image encoding module through residual connection. The fused features are passed to the regression layer to generate the human posture estimation results.

6. A lightweight and high-precision human posture estimation method based on Mamba architecture according to claim 1, characterized in that: Step S3 includes the following steps: Input the human body image to be detected; After the human body image to be detected is input into the image encoding module, it is divided into several image blocks. After that, the image blocks are straightened into vector form and projected into the high-dimensional feature space through the linear layer. Finally, the position code is added to each image block. The image blocks with position encoding are input into the lightweight feature extraction module to extract features related to human posture; The feature representation obtained by the lightweight feature extraction module is input into the regression layer to obtain the predicted human posture estimation result.

Images