An efficient human pose estimation method for embodied intelligent interaction

Abstract

The application discloses a kind of high-efficiency human posture estimation methods for embodied intelligent interaction, including feature extraction module, including convolution, for carrying out down-sampling to input image, obtain image down-sampling feature;Multi-stage posture representation module, the multi-stage posture representation module is composed of three dynamic Token perception stages, respectively using first dynamic Token perception submodule, second dynamic Token perception submodule and third dynamic Token perception submodule are realized, three dynamic Token perception submodule network structure is same, and all includes Token aggregation unit and mixed scanning unit;Image down-sampling feature is serialized and input to multi-stage posture representation module, for from coarse to fine perception human key point position in the image down-sampling feature.The application enhances the modeling capacity of human key point local and global information, realizes accurate, efficient posture estimation under lower computational load, improves the execution effect of robot, embodied intelligent interaction and other related tasks.

Description

Technical Field

[0001] This invention relates to the field of human behavior recognition technology, specifically to an efficient human pose estimation method for embodied intelligent interaction. Background Technology

[0002] From gracefully dancing humanoid robots to robotic dogs delivering goods on the streets, embodied intelligence has gradually moved from the forefront of artificial intelligence into practical applications. Against the backdrop of rapid development in embodied intelligence, robots, virtual humans, and various intelligent agents are constantly evolving from simply "seeing" to "understanding and interacting." In applications such as human-machine collaboration, immersive interaction, intelligent security, sports rehabilitation training, and AR / VR, intelligent agents need accurate, stable, and efficient perception and understanding of human posture to achieve rational decision-making and natural interaction with the environment. Human pose estimation (HPE), as a key link connecting visual perception and high-level behavioral understanding, directly affects the interaction quality and task execution performance of embodied intelligence systems, and can be effectively applied to fields such as robot motion, human action recognition, and spatial intelligence.

[0003] Early human pose estimation methods employed Convolutional Neural Networks (CNNs), largely overcoming the accuracy limitations of manual methods. However, as research progressed, the limitations of CNNs in extracting local image features restricted their ability to capture long-range keypoint correlations. In recent years, Transformers, with their inherent advantage in global feature extraction, have emerged in computer vision tasks, filling the global dependency problem that CNNs struggled to overcome. Notably, most current methods, when applying Transformers, often combine them with CNNs to improve the final performance of human keypoint estimation. This is primarily due to two reasons: firstly, CNNs are helpful in extracting local image features, while Transformers excel at capturing global information; combining them achieves complementarity between local and global aspects. Secondly, the Transformer's self-attention mechanism has quadratic computational complexity, which limits its application when computational resources are limited; therefore, the assistance of CNNs is needed to balance performance and efficiency.

[0004] Currently, Mamba, as a novel state-space model, achieves linear time complexity when handling long-range correlations, showing promising efficiency compared to Transformers. However, since the data in computer vision tasks is inherently non-causal and contains spatial information such as local and texture information, the parallel selective scanning operation of ordinary Mamba cannot fully adapt to the spatial structure and local features of image data. Multi-scanning methods and the addition of causal convolutions are generally used to address non-causality. Unlike traditional classification, detection, and segmentation tasks, human pose estimation requires full consideration of the correlation between keypoints within an image. Traditional Mamba struggles to capture the local dependencies of keypoints, leading to insufficient local features and weakening the perception of target joints. Furthermore, while using convolutions to shorten the input sequence can alleviate this problem, it may also lose detailed information in the image, ultimately affecting the accuracy of pose estimation. Therefore, applying Mamba to human pose estimation to achieve efficient embodied intelligent interaction remains a challenge. Summary of the Invention

[0005] To address the above problems, this invention provides an efficient human pose estimation method for embodied intelligent interaction, which effectively enhances its ability to model local and global information of human body key points, and ensures accurate estimation with low computational cost. This significantly improves the efficiency of human pose estimation in embodied intelligent interaction tasks and can effectively solve the problems in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: An efficient human pose estimation method for embodied intelligent interaction, including The feature extraction module includes convolution, which is used to downsample the input image to obtain image downsampled features; A multi-stage attitude representation module is provided, which consists of three dynamic token perception stages, implemented by a first dynamic token perception submodule, a second dynamic token perception submodule, and a third dynamic token perception submodule, respectively. The three dynamic token perception submodules have the same network structure, each including a token aggregation unit and a hybrid scanning unit. The image downsampling features are serialized and input into a multi-stage pose representation module for coarse-to-fine perception of the human body key point positions in the image downsampling features.

[0007] As a preferred technical solution of the present invention, the Token aggregation unit includes a key point location acquisition subunit; The input sequence is obtained by serializing the image downsampling features. ; The key point location acquisition subunit inputs the sequence After passing through the linear layer via the channel Map N keypoints and obtain the N keypoints and input sequence using the LogSoftmax function. The association probability of each pixel is calculated, and then the GumbelSoftmax function is used to generate a continuous and differentiable association probability. The expression is: ; in, Indicates a linear layer; And, in Based on a set of learnable key point token sequences , used for encoding The spatial distribution of N key points, and the key point localization guidance obtained through a linear layer. The expression is: ; in, This represents matrix multiplication.

[0008] As a preferred technical solution of the present invention, the Token aggregation unit further includes a key point Token aggregation subunit; The key point token aggregation subunit processes the input sequence. After a The convolutional and sigmoid function layers implement normalization operations, and the feature dimension transformation is achieved through a reshape operation to obtain normalized feature representations. The expression is: ;

[0009] Simultaneously, the key point token aggregation subunit obtains the tokens sequentially through the Argmax function and the Sort function. The association and sorting of each pixel with N keypoints are used to obtain the pixel index sequence belonging to the same keypoint. The expression is: ;

[0010] Furthermore, in Based on this, the normalized feature representation is processed using the Gather function. Perform dynamic token aggregation to obtain guidance on key point correlations Used to guide the input sequence The correlation between the key points and the key points is expressed as: ;

[0011] The Gather operation is implemented using torch.gather. Expanded dimensions for matching Dimensions.

[0012] As a preferred embodiment of the present invention, the hybrid scanning unit includes a multi-strategy scanning subunit for processing the input sequence. Guidance on the relevance of key points Perform a correlation scan; The four scanning methods are as follows: Guidance on the relevance of key points Perform associative scans S1 and S2 from left to right and from right to left; For the input sequence Perform top-down and bottom-up correlation scans S3 and S4; Concatenate S1, S2, S3, and S4 along the second dimension (the first dimension refers to BitchSize, which is omitted here) to obtain the scan output matrix at the current time t. The expression is: ;

[0013] The Cat operation is implemented through torch.cat and is used to combine different scanning methods; The scan output matrix Through two consecutive linear layers and embedding vector The four parameter matrices A, B, C, and D used for Mamba structure calculations are obtained as follows: ;

[0014] The hybrid scanning unit of this application has four scanning modes and incorporates key point location information. Specifically, the hybrid scanning unit further includes a key point location mixing subunit for mixing key point positioning guidance within the Mamba structure. ; The specific steps are as follows: First, the hidden state at time t is obtained using parameter matrices A and B. ; Then, guided by the parameter matrix C and hybrid keypoint localization. The parameters are fused to obtain a parameter matrix C with positioning guidance; Finally, the parameter matrix C with positioning guidance is combined with the hidden state. Obtain the enhanced parameter matrix C; combine the parameter matrix D with the scan output matrix. Obtain the enhancement parameter matrix D; obtain the hybrid scan output at the current time t using the enhancement parameter matrix C and the enhancement parameter matrix D. It is used to obtain the location of key points of the human body in the input image.

[0015] As a preferred technical solution of the present invention, the input sequence Spatial dimensions in the three dynamic token perception stages The size decreases by a factor of 2 sequentially, channel dimension The size increases by 2 times sequentially to obtain the first input sequence, the second input sequence, and the third input sequence, which are input into the first dynamic token perception submodule, the second dynamic token perception submodule, and the third dynamic token perception submodule.

[0016] In this application, the first dynamic token perception submodule performs dimensionality transformation on the image downsampling features to obtain a first input sequence, and uses a token aggregation unit and a hybrid scanning unit to perform global token aggregation and global token scanning on the first input sequence to obtain global aggregated features. The second dynamic token perception submodule obtains the second input sequence through global aggregation features, and uses a token aggregation unit and a hybrid scanning unit to perform coarse token aggregation and coarse token scanning on the second input sequence to obtain coarse aggregation features; The third dynamic token perception submodule obtains the third input sequence by coarsely aggregating features, and uses a token aggregation unit and a hybrid scanning unit to perform fine token aggregation and fine token scanning on key points of the third input sequence to obtain fine aggregated features. The refined aggregated features are used to characterize the location of key points of the human body in the input image.

[0017] As a preferred technical solution of the present invention, the global token aggregation and global token scanning of the image include applying a token aggregation unit and a hybrid scanning unit to all pixels in the image; The key point coarse token aggregation and coarse token scanning include applying a token aggregation unit and a hybrid scanning unit to the head, upper body and lower body of the human body in the image; The key point fine token aggregation and fine token scanning include applying a token aggregation unit and a hybrid scanning unit to the head, left hand, right hand, left leg and right leg of the human body in the image.

[0018] As a preferred technical solution of the present invention, all pixels in the image include all key points of the human body; The image contains three parts: the head, upper body, and lower body. Each part includes the number of key points of the human body within that part. The image contains five parts: the head, left hand, right hand, left leg, and right leg. Each part includes the number of key points of the human body within that part. The number of human body key points corresponds to the number of human body key points in the training dataset.

[0019] This invention also includes a multi-stage deconvolution module; The multi-stage deconvolution module includes a first deconvolution layer and a second deconvolution layer; The global aggregated features are downsampled and then fused with the fine aggregated features before being input into the first deconvolution layer to obtain the first deconvolution features; The first deconvolution feature and the coarse aggregated feature are fused and input into the second deconvolution layer to obtain the second deconvolution feature; The first and second deconvolution layers progressively enlarge the key points of the output of the multi-stage pose representation module, restoring it to the scale of the input image. Convolution is used to obtain the number of key points in the corresponding training dataset, and finally the location of human key points at the scale of the input image.

[0020] Compared with the prior art, the main beneficial effects achieved by the present invention are as follows: 1. This invention introduces a dynamic token perception mechanism into the multi-stage pose representation module, and through the joint design of the token aggregation unit and the hybrid scanning unit, it realizes the gradual representation of human key points from global to local and from coarse to fine. This not only alleviates the remote forgetting problem of traditional Mamba, but also enhances the fine-grained expression ability of local key points, thereby significantly improving the accuracy and robustness of pose estimation.

[0021] 2. This invention introduces key point correlation guidance information into the hybrid scanning strategy and adopts a multi-directional and multi-level scanning method, which effectively breaks through the limitation of traditional Mamba that only scans the input sequence. While ensuring low computational cost, it improves the comprehensive extraction capability of global and local features and improves the accuracy of key point localization.

[0022] 3. This invention fuses global aggregated features, coarse aggregated features, and fine aggregated features through a multi-stage deconvolution module, and progressively upsamples to restore the image to the input scale, ensuring the spatial consistency and detail fidelity of the predicted human keypoint locations. This method achieves higher average accuracy and average recall while significantly reducing computational load. During robot-human interaction, images acquired through cameras can quickly and accurately capture human motion postures, meeting the real-time interaction needs of robots and other embodied intelligent entities with humans. Attached Figure Description

[0023] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is an exploded view of the dynamic token perception submodule structure of the present invention; Figure 3 This is a network framework diagram for input image processing in this invention; Figure 4 This is a network framework diagram of the feature extraction module and the dynamic token perception submodule of the present invention; Figure 5 This is a network framework diagram of the Token aggregation unit of the present invention; Figure 6 This is a network framework diagram of the hybrid scanning unit of the present invention; Figure 7 This is a graph showing the experimental results of the present invention on COCO data. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example

[0025] Please see Figures 1 to 7 The present invention provides a technical solution: An efficient human pose estimation method for embodied intelligent interaction is proposed. After inputting an image, it can obtain a heat map representing the location of key points of the human body, thereby obtaining the key points of the human body.

[0026] Specific plans include The feature extraction module includes convolution, which is used to downsample the input image to obtain image downsampled features; Specifically, the feature extraction module consists of three convolutional layers: a large-kernel convolutional layer with a kernel size of 7 and a stride of 2, and two deep convolutional layers with a kernel size of 3. After each convolutional operation, batch normalization is applied, followed by a non-linear transformation using the ReLU activation function. The main purpose of the feature extraction module is to perform preliminary screening of the input image, removing irrelevant information. Through the design of shallow convolutional layers, it can quickly capture local features of the image, such as edges and textures, filtering out details that are not helpful for subsequent tasks.

[0027] This application only provides one method for setting the convolution size. However, for those skilled in the art, adjusting the convolution kernel and stride size to obtain image downsampling features at the corresponding scale should be considered within the scope of protection of this application.

[0028] A multi-stage attitude representation module is provided, which consists of three dynamic token perception stages, implemented by a first dynamic token perception submodule, a second dynamic token perception submodule, and a third dynamic token perception submodule, respectively. The three dynamic token perception submodules have the same network structure, each including a token aggregation unit and a hybrid scanning unit. In fact, the dynamic token-aware submodule proposed in this application includes not only the token aggregation unit and the hybrid scanning unit, but also basic neural network structures such as layer normalization, linear layers, convolutions, and SiLU activation functions. For those skilled in the art, these neural network structures in the dynamic token-aware submodule are similar to the traditional Mamba structure ("Mamba: Linear-time sequence modeling with selective state spaces" proposed by Gu et al.). The main difference lies in the token aggregation unit and the hybrid scanning unit, which is what distinguishes this application from previous methods and can be considered a major innovation of the dynamic token-aware submodule. Here, "token" is a standardized term for the input sequence, which can be translated as "marker" in Chinese, referring to the smallest unit of feature information within the input sequence.

[0029] The image downsampling features are serialized and input into a multi-stage pose representation module for coarse-to-fine perception of the human body key point positions in the image downsampling features.

[0030] In this application, the multi-stage pose representation module is used to coarsely and finely perceive the positions of human key points in the input image. After obtaining the image downsampling features, the three dynamic token perception stages within the multi-stage pose representation module are processed sequentially. The image downsampling features are obtained in the first dynamic token perception submodule. Next, dimensionality reduction and feature merging are performed. (equivalent to) (half of) and width (equivalent to) Half of the data is used as one-dimensional data to obtain the input sequence. Among them, the channel It is twice the size of the original channel C; similarly, the second dynamic token sensing submodule will... Hekuan Reduce the dimension by half again, channel Increase, and the third dynamic token-aware submodule will... Hekuan Reduce the dimension to 1 / 4 size, channels Continue to increase to 4 times.

[0031] To better understand the solution of this application, the input dimension and other scales in the embodiments are mainly based on the first dynamic token perception submodule in the matching token aggregation unit and the hybrid scanning unit.

[0032] Meanwhile, the number of the first dynamic token perception submodule, the second dynamic token perception submodule, and the third dynamic token perception submodule in this application are set to 2, 4, and 6 respectively in the multi-stage attitude representation module, in order to balance the overall computational load.

[0033] Specifically, the Token aggregation unit includes a key point location acquisition subunit; The input sequence is obtained by serializing the image downsampling features. ; in, and These represent the spatial dimension and channel dimension of the input sequence, respectively.

[0034] The key point location acquisition subunit inputs the sequence After passing through the linear layer via the channel Map N key points of the human body, and use the LogSoftmax function (a function that normalizes a vector of arbitrary real numbers into a probability distribution, where each element represents the probability of that class, and the sum of all elements is 1) to obtain the N key points and the input sequence. The association probability of each pixel is calculated, and then the GumbelSoftmax function (a technique for sampling discrete distributions; this application performs a differentiable operation on the data to facilitate network training) is used to generate continuous and differentiable association probabilities. The expression is: ;

[0035] in, Indicates a linear layer; And, in Based on a set of learnable key point token sequences , used for encoding The spatial distribution of N key points, and the key point localization guidance obtained through a linear layer. The expression is: ;

[0036] in, This represents matrix multiplication.

[0037] Specifically, the token aggregation unit also includes a key point token aggregation subunit; The key point token aggregation subunit processes the input sequence. After a The convolutional and sigmoid function layers implement normalization operations, and the feature dimension transformation is achieved through a reshape operation to obtain normalized feature representations. The expression is: ;

[0038] Meanwhile, the key point token aggregation subunit is obtained sequentially through the Argmax function (used to find the index corresponding to the maximum value in the array) and the Sort function (sorting all elements in a given range). The association and sorting of each pixel with N keypoints are used to obtain the pixel index sequence belonging to the same keypoint. The expression is: ;

[0039] Furthermore, in Based on this, the normalized feature representation is processed using the Gather function. Perform dynamic token aggregation to obtain guidance on key point correlations Used to guide the input sequence The correlation between the key points and the key points is expressed as: ;

[0040] The Gather operation is implemented using torch.gather. Expanded dimensions for matching Dimensions.

[0041] Here, the dimensions of the input image and input sequence are mainly matched to the first dynamic token perception submodule. Subsequent operations such as dimension reduction and channel changes are readily understood by those skilled in the art, so the changes in dimension size will not be elaborated here.

[0042] Specifically, the hybrid scanning unit includes a multi-strategy scanning subunit for scanning the input sequence. Guidance on the relevance of key points Perform a correlation scan; The four scanning methods are as follows: Guidance on the relevance of key points Perform associative scans S1 and S2 from left to right and from right to left; For the input sequence Perform top-down and bottom-up correlation scans S3 and S4; It should be noted that traditional Mamba's four-way scan always involves the input sequence. Unlike other methods, this application employs key point correlation guidance for scanning. By acquiring S1 and S2, this multi-directional, multi-level hybrid scanning strategy can effectively balance the extraction capabilities of local and global image information, while obtaining additional key point correlation guidance information, and more accurately capturing the true location of key points.

[0043] Concatenate S1, S2, S3, and S4 along the second dimension (the first dimension refers to BatchSize, which is omitted here) to obtain the scan output matrix at the current time t. The expression is: ;

[0044] The Cat operation is implemented through torch.cat and is used to combine different scanning methods; The scan output matrix Through two consecutive linear layers and embedding vector The four parameter matrices A, B, C, and D used for Mamba structure calculations are obtained as follows: ;

[0045] The hybrid scanning unit of this application has four scanning modes and incorporates key point location information. Specifically, the hybrid scanning unit further includes a key point location mixing subunit for mixing key point positioning guidance within the Mamba structure. ; The specific steps are as follows: First, the hidden state at time t is obtained using parameter matrices A and B. ; Then, guided by the parameter matrix C and hybrid keypoint localization. The parameters are fused to obtain a parameter matrix C with positioning guidance; Finally, the parameter matrix C with positioning guidance is combined with the hidden state. Obtain the enhanced parameter matrix C; combine the parameter matrix D with the scan output matrix. Obtain the enhancement parameter matrix D; obtain the hybrid scan output at the current time t using the enhancement parameter matrix C and the enhancement parameter matrix D. It is used to obtain the location of key points of the human body in the input image.

[0046] This application uses the Mamba framework, where the parameter matrix C acts similarly to the query matrix Q in the attention mechanism. Therefore, by adding the output of the key point information acquisition part to C, a global query function similar to that in the attention mechanism is achieved, thereby improving the global capture effect of the mixed scan output on the input sequence.

[0047] More specifically, the input sequence Spatial dimensions in the three dynamic token perception stages The size decreases by a factor of 2 sequentially, channel dimension The size increases by 2 times sequentially to obtain the first input sequence, the second input sequence, and the third input sequence, which are input into the first dynamic token perception submodule, the second dynamic token perception submodule, and the third dynamic token perception submodule.

[0048] In this application, the first dynamic token perception submodule performs dimensionality transformation on the image downsampling features to obtain a first input sequence, and uses a token aggregation unit and a hybrid scanning unit to perform global token aggregation and global token scanning on the first input sequence to obtain global aggregated features. The second dynamic token perception submodule obtains the second input sequence through global aggregation features, and uses a token aggregation unit and a hybrid scanning unit to perform coarse token aggregation and coarse token scanning on the second input sequence to obtain coarse aggregation features; The third dynamic token perception submodule obtains the third input sequence by coarsely aggregating features, and uses a token aggregation unit and a hybrid scanning unit to perform fine token aggregation and fine token scanning on key points of the third input sequence to obtain fine aggregated features. The refined aggregated features are used to characterize the location of key points of the human body in the input image.

[0049] The image global token aggregation and global token scanning include applying a token aggregation unit and a hybrid scanning unit to all pixels in the image; The key point coarse token aggregation and coarse token scanning include applying a token aggregation unit and a hybrid scanning unit to the head, upper body and lower body of the human body in the image; The key point fine token aggregation and fine token scanning include applying a token aggregation unit and a hybrid scanning unit to the head, left hand, right hand, left leg and right leg of the human body in the image.

[0050] All pixels in the image include all key points of the human body; The image contains three parts: the head, upper body, and lower body. Each part includes the number of key points of the human body within that part. The image contains five parts: the head, left hand, right hand, left leg, and right leg. Each part includes the number of key points of the human body within that part. The number of human body key points corresponds to the number of human body key points in the training dataset.

[0051] Taking the COCO dataset as an example, in the human pose estimation task, each human body is tokenized into 17 keypoints, including the nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Different tokens pay different levels of attention to different keypoints. Therefore, when encoding the feature map into a token sequence, identifying and aggregating tokens related to the current keypoint in keypoint order can make related tokens closer together in the sequence. This strategy can be seen as a variant of the local scanning strategy, which divides the image into different windows and scans within each window. Unlike directly scanning the entire feature map, this can more effectively capture local dependencies. Moreover, by aggregating highly correlated tokens, the model can establish a closer connection between the mutual contributions of tokens, thereby effectively alleviating the long-distance forgetting problem of traditional Mamba.

[0052] In this application, when applied to embodied intelligent interaction tasks, the robot needs to acquire human postures to recognize, respond to, and predict actions. After obtaining efficient human posture estimation, the robot can achieve accurate and rapid interaction with humans.

[0053] Furthermore, the application of this application is not limited to human posture recognition, but can also accurately and quickly acquire the posture of humanoid robots.

[0054] More specifically, this application refines the aggregated and scanned key point locations through three dynamic token perception stages. The main purpose is to gradually group together semantically similar tokens, making the distance between related tokens in the one-dimensional input sequence closer. This effectively improves Mamba's ability to extract local features, which can alleviate the problem of insufficient local features in traditional Mamba. Moreover, the aggregation of more relevant tokens can also reduce information loss due to remote forgetting.

[0055] There are five key points in the human head: the nose, the left and right eyes, and the left and right ears. Considering the unique characteristics of the head region, the aggregation method requires grouping the relevant labels of these densely packed key points together. This effectively reduces label separation and enhances the feature representation of the head region. Therefore, the head, upper body, and lower body have more key points than the five parts (head, left hand, right hand, left leg, and right leg). In the three dynamic token perception stages of this application, it is argued that the fewer groups in the aggregation strategy, the weaker its ability to extract local features, but the stronger its ability to extract global features.

[0056] Therefore, the purpose of the three dynamic token perception stages in this application is to adopt an acquisition strategy that shifts from focusing on all information to focusing on local information, thereby fully acquiring the global and local information of the input image and effectively ensuring the validity of this application.

[0057] The comparison results of the proposed method with other advanced methods are shown in the table below (COCO dataset):

[0058] Where AP represents average precision, AR represents average recall, and GFLOPs represent the computational cost of the model. The paper titles associated with methods 1 through 4 in the table are as follows: 1. Simple baselines for human pose estimation and tracking, published in 2018; 2. Deep high-resolution representation learning for human pose estimation, published in 2019; 3. Mambavision: A hybrid mamba-transformervision backbone, published in 2025; 4. Efficient dual-scale cross-attention network for human pose estimation, published in 2025.

[0059] It can be seen that the method proposed in this invention achieves an AP value of 75.0, and the computational cost is smaller than that of other similar methods currently available. Not only is it superior to traditional methods, but compared with the algorithms of 2025[3] and[4] of the same period, the method in this paper achieves better accuracy at a lower cost, which fully demonstrates the effectiveness of this method.

[0060] This invention also includes a multi-stage deconvolution module; The multi-stage deconvolution module includes a first deconvolution layer and a second deconvolution layer; The global aggregated features are downsampled and then fused with the fine aggregated features before being input into the first deconvolution layer to obtain the first deconvolution features; The first deconvolution feature and the coarse aggregated feature are fused and input into the second deconvolution layer to obtain the second deconvolution feature; The first and second deconvolution layers progressively enlarge the key points of the output of the multi-stage pose representation module, restoring it to the scale of the input image. Convolution is used to obtain the number of key points in the corresponding training dataset, and finally the location of human key points at the scale of the input image.

[0061] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A highly efficient human pose estimation method for embodied intelligent interaction, characterized in that: include The feature extraction module includes convolution, which is used to downsample the input image to obtain image downsampled features; A multi-stage attitude representation module is provided, which consists of three dynamic token perception stages, implemented by a first dynamic token perception submodule, a second dynamic token perception submodule, and a third dynamic token perception submodule, respectively. The three dynamic token perception submodules have the same network structure, each including a token aggregation unit and a hybrid scanning unit. The image downsampling features are serialized and input into a multi-stage pose representation module for coarse-to-fine perception of the human body key point positions in the image downsampling features; The token aggregation unit includes a key point location acquisition subunit; The input sequence is obtained by serializing the image downsampling features. ; in, These are the spatial dimension and channel dimension of the input sequence, respectively; The key point location acquisition subunit inputs the sequence After passing through the linear layer via the channel Map the N key points to the human body, and obtain the N key points and the input sequence through the LogSoftmax function. The association probability of each pixel is calculated, and then the GumbelSoftmax function is used to generate a continuous and differentiable association probability. The expression is: ； in, Indicates a linear layer; And, in Based on a set of learnable key point token sequences , used for encoding The spatial distribution of N key points, and the key point localization guidance obtained through a linear layer. The expression is: ； in, Indicates matrix multiplication; The token aggregation unit also includes a key point token aggregation subunit; The key point token aggregation subunit processes the input sequence. After a The convolutional and sigmoid function layers implement normalization operations, and the feature dimension transformation is achieved through a reshape operation to obtain normalized feature representations. The expression is: ； Meanwhile, the key point token aggregation subunit obtains the key points through the Argmax and Sort functions. The association and sorting of each pixel with N keypoints are used to obtain the pixel index sequence belonging to the same keypoint. The expression is: ； exist Based on this, the normalized feature representation is processed using the Gather function. Perform dynamic token aggregation to obtain guidance on key point correlations Used to guide the input sequence The correlation between the key points and the key points is expressed as: ； The Gather operation is implemented using torch.gather, where the pixel index sequence... Expanding dimensions and Maintain consistency in dimensions; The hybrid scanning unit includes a multi-strategy scanning subunit for processing the input sequence. Guidance on the relevance of key points Perform a correlation scan; The hybrid scanning unit further includes a key point location hybridization subunit, used to hybridize key point localization guidance within the Mamba structure. .

2. The efficient human pose estimation method for embodied intelligent interaction according to claim 1, characterized in that: The hybrid scanning unit includes a multi-strategy scanning subunit for processing the input sequence. Guidance on the relevance of key points Perform a correlation scan; The four scanning methods are as follows: Guidance on the relevance of key points Perform associative scans S1 and S2 from left to right and from right to left; For the input sequence Perform top-down and bottom-up correlation scans S3 and S4; By concatenating S1, S2, S3, and S4, the scan output matrix at the current time t is obtained. The expression is: ； The Cat operation is implemented using torch.cat and is used to combine the four scanning methods. The scan output matrix Through two consecutive linear layers and embedding vector The four parameter matrices A, B, C, and D used for Mamba structure calculations are obtained as follows: 。 3. The efficient human pose estimation method for embodied intelligent interaction according to claim 2, characterized in that: The hybrid scanning unit further includes a key point location hybridization subunit, used to hybridize key point localization guidance within the Mamba structure. ; The specific steps are as follows: First, the hidden state at time t is obtained using parameter matrices A and B. ; Then, guided by the parameter matrix C and hybrid keypoint localization. The parameters are fused to obtain a parameter matrix C with positioning guidance; Finally, the parameter matrix C with positioning guidance is combined with the hidden state. Obtain the enhanced parameter matrix C; combine the parameter matrix D with the scan output matrix. Obtain the enhancement parameter matrix D; obtain the hybrid scan output at the current time t using the enhancement parameter matrix C and the enhancement parameter matrix D. It is used to obtain the location of key points of the human body in the input image.

4. The efficient human pose estimation method for embodied intelligent interaction according to claim 3, characterized in that: The input sequence Spatial dimensions in the three dynamic token perception stages The size decreases by a factor of 2 sequentially, channel dimension The size increases by 2 times sequentially to obtain the first input sequence, the second input sequence, and the third input sequence, which are input into the first dynamic token perception submodule, the second dynamic token perception submodule, and the third dynamic token perception submodule.

5. The efficient human pose estimation method for embodied intelligent interaction according to claim 4, characterized in that: The first dynamic token perception submodule performs dimensionality transformation on the image downsampling features to obtain the first input sequence, and uses the token aggregation unit and the hybrid scanning unit to perform global token aggregation and global token scanning on the first input sequence to obtain global aggregated features; The second dynamic token perception submodule obtains the second input sequence through global aggregation features, and uses a token aggregation unit and a hybrid scanning unit to perform coarse token aggregation and coarse token scanning on the second input sequence to obtain coarse aggregation features; The third dynamic token perception submodule obtains the third input sequence by coarsely aggregating features, and uses a token aggregation unit and a hybrid scanning unit to perform fine token aggregation and fine token scanning on key points of the third input sequence to obtain fine aggregated features. The refined aggregated features are used to characterize the location of key points of the human body in the input image.

6. The efficient human pose estimation method for embodied intelligent interaction according to claim 5, characterized in that: The image global token aggregation and global token scanning include applying a token aggregation unit and a hybrid scanning unit to all pixels in the image; The key point coarse token aggregation and coarse token scanning include applying a token aggregation unit and a hybrid scanning unit to the head, upper body and lower body of the human body in the image; The key point fine token aggregation and fine token scanning include applying a token aggregation unit and a hybrid scanning unit to the head, left hand, right hand, left leg and right leg of the human body in the image.

7. The efficient human pose estimation method for embodied intelligent interaction according to claim 6, characterized in that: All pixels in the image include all key points of the human body; The image contains three parts: the head, upper body, and lower body. Each part includes the number of key points of the human body within that part. The image contains five parts: the head, left hand, right hand, left leg, and right leg. Each part includes the number of key points of the human body within that part. The number of human body key points corresponds to the number of human body key points in the training dataset.

8. A highly efficient human pose estimation method for embodied intelligent interaction according to any one of claims 1-7, characterized in that: It also includes a multi-stage deconvolution module; The multi-stage deconvolution module includes a first deconvolution layer and a second deconvolution layer; The global aggregated features are downsampled and then fused with the fine aggregated features before being input into the first deconvolution layer to obtain the first deconvolution features; The first deconvolution feature and the coarse aggregated feature are fused and input into the second deconvolution layer to obtain the second deconvolution feature; The first and second deconvolution layers progressively enlarge the key points of the output of the multi-stage pose representation module, restoring it to the scale of the input image. The number of keypoints in the corresponding training dataset is obtained by product, and finally the location of human keypoints at the scale of the input image is obtained.

Images