Human body posture estimation method based on graph attention and Transform

Abstract

The invention discloses a human body posture estimation method based on graph attention and Transform, and designs a set of multi-stage modeling framework which is guided by a heat graph and is characterized by structure initialization, graph attention network (GAT) modeling local space dependence, global context information extraction by a Transform encoder and posture decoder fusion optimization, and local precision and global consistency are considered. According to the method, the structural modeling capability and the context expression capability of human body posture estimation are effectively improved, and the problems of structural information loss, low reasoning efficiency, lack of global correlation modeling and the like of a traditional method are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of computer graphics and human pose, and particularly relates to a human pose estimation method based on graph attention and Transformer. BACKGROUND

[0002] Human pose estimation aims to automatically identify and locate specific joint points or landmarks in human images. These joint points are usually important features of the human body, such as shoulders, elbows, wrists, hips, knees, and ankles. Through joint point detection, structural information of the human body can be obtained, enabling a series of applications such as pose estimation, motion recognition, human modeling, and behavior analysis. Human joint point detection is a widely studied field in computer vision tasks, but still faces challenges in certain complex scenarios. For example, in the case of occlusion, changes in viewing angle, and multi-human interaction, the accuracy of detection is affected. Therefore, it is particularly important to propose a method that can accurately detect human joint points in complex scenarios.

[0003] Existing human pose estimation tasks have gradually shifted to deep learning-based methods, which can be roughly divided into direct regression and heat map regression methods. Direct regression methods learn the mapping relationship between input images and target joint positions and directly regress the coordinate values of joint points. For example, the SimpleBaseline method uses a deep convolutional neural network to extract image features and predicts the position of joint points through linear regression. Heat map regression methods predict the probability distribution of each joint point on the feature map to achieve more accurate localization. Typical methods such as HRNet generate high-resolution joint heat maps through multi-scale feature fusion to achieve accurate pose estimation. Another example is PoseNet, which improves the accuracy of pose estimation through step-by-step refinement. Existing human pose estimation methods still have deficiencies in structure modeling capability, inference efficiency, and robustness: heat map regression methods have high positioning accuracy, but they are highly dependent on high-resolution feature maps, have high computational overhead, and are difficult to meet real-time requirements; while direct coordinate regression methods lack explicit modeling of the spatial structure between key points, are prone to failure in ambiguous poses, occlusions, or complex backgrounds, and have difficulty in ensuring accuracy and stability. SUMMARY

[0004] The present application proposes a human pose estimation method based on graph attention and Transformer, aiming to solve the problems of insufficient structure modeling capability, inaccurate key point positioning, and poor global consistency of existing methods.

[0005] The method constructs a multi-stage modeling framework, which integrates heat map guidance, spatial structure modeling and context perception mechanism to improve the accuracy and expression ability of pose estimation. Specifically, first, the multi-scale features of the image are extracted by the backbone network, and the preliminary key point heat map is generated by using the heat map regression to realize the guiding initialization of the human skeleton structure; then, the human graph structure is constructed by taking the key points as nodes, the graph attention network (GAT) is introduced to model the spatial topological relationship between joints, and the local structure dependence is captured; on this basis, the Transformer encoder is further introduced to model the global semantic relationship between key points, and the context expression ability is enhanced; finally, the integrated features are analyzed and regressed by the pose decoder to output accurate and structure-consistent key point coordinates.

[0006] To achieve the above object, the technical scheme adopted by the present application comprises the following steps:

[0007] In the first aspect, the embodiment of the present application provides a human pose estimation method based on graph attention and Transformer, comprising the following steps

[0008] Obtain a public human pose dataset containing images and corresponding two-dimensional human key point annotations.

[0009] After the input image is sent into the backbone network to extract the deep feature map as the initial feature, the input is sent to the heat map decoder, the key point preliminary prediction coordinates are obtained according to the heat map, and the point guidance sampling is performed on the feature map to construct the key point feature matrix.

[0010] The skeleton graph is constructed based on the key point feature matrix, the local topological structure and global semantic dependence between key points are jointly modeled by the graph attention network GAT and the Transformer encoder, and the structure-aware key point feature is generated.

[0011] The structure-aware key point feature is input into the joint decoder for feature analysis and coordinate regression, and the two-dimensional coordinate set of the human key points in the image is output to represent the structure-consistent human pose estimation result.

[0012] In a possible implementation, the key point feature matrix construction specifically comprises the following operations:

[0013] The input image is first subjected to a convolutional neural network with multi-scale feature extraction capability to extract a deep feature map as an initial feature. Subsequently, the initial feature is input into a heat map decoder to output a multi-channel heat map tensor. Each channel of the heat map tensor corresponds to a key point category, and a maximum value position extraction operation is performed on each channel of the heat map tensor to obtain the position coordinates with the highest confidence from each channel as the predicted two-dimensional coordinates of the corresponding key point. After obtaining the predicted coordinates of all key points, a point-guided feature sampling operation is performed on the initial feature map guided by the set of coordinates. Specifically, for each key point, its predicted position is mapped to the spatial dimension of the initial feature map, and the channel feature at that position is extracted by bilinear interpolation to form a semantic representation of the key point. After sampling all key points, a key point feature matrix is constructed, where each row represents a feature vector of a key point that integrates its spatial position information and local semantic features.

[0014] In a possible implementation, the skeleton graph is constructed based on the key point feature matrix as follows:

[0015] Each key point is regarded as a node in the graph, and its input feature is the corresponding row vector in the key point feature matrix, representing the high-dimensional semantic feature of the key point. The entire feature matrix constitutes the input set of all nodes in the graph, integrating the local semantic information of the key points and the image spatial position. The connection relationship between the nodes in the graph is constructed according to the skeleton structure, representing the anatomical connection relationship between the key points.

[0016] In a possible implementation, the constructed key point graph is input into a graph attention network GAT to model the local topological dependence between the key points. The node feature sequence output by the GAT is input into a Transformer encoder to realize feature interaction between all key points by using the multi-head self-attention mechanism of the Transformer encoder, and finally output structure-aware key point features.

[0017] In a possible implementation, in the joint decoder, a nonlinear transformation is first performed on each structure-aware key point feature. Subsequently, a feedforward neural network is used for point-by-point feature mapping, and an Add&Norm residual connection mechanism is introduced to enhance the stability of network training and the efficiency of feature transmission. Finally, a multi-layer perceptron is used to map to two-dimensional coordinates in the image space. The prediction results of all key points constitute a two-dimensional key point coordinate set, representing the pose estimation result of the human body structure consistent and spatially coherent in the image.

[0018] In a possible implementation, a training phase is further included, and a mean square error loss function (MSE) is used to supervise the difference between the predicted heat map and the real label to guide the collaborative optimization of the backbone network, the structure modeling module and the joint decoder, and to improve the key point positioning accuracy and structural consistency.

[0019] In a second aspect, the embodiments of the present application provide a human pose estimation device based on graph attention and Transformer, comprising the following modules:

[0020] The data acquisition module is configured to acquire a public human pose dataset containing images and corresponding two-dimensional human key point annotations.

[0021] The matrix construction module is configured to extract a deep feature map as an initial feature by using a convolutional neural network with multi-scale feature extraction capability, input the initial feature into a heat map decoder, obtain a key point preliminary prediction coordinate according to the heat map, perform point-guided sampling on the feature map, and construct a key point feature matrix.

[0022] The structure perception module is configured to construct a skeleton graph based on the key point feature matrix, model local topological structures and global semantic dependencies between the key points by using a graph attention network (GAT) and a Transformer encoder in combination, and generate structure-perceived key point features.

[0023] The pose estimation module is configured to perform feature analysis and coordinate regression by using a joint decoder based on the structure-perceived key point features, and output a two-dimensional coordinate set of human key points in an image, which is used to represent a structure-consistent human pose estimation result.

[0024] In a third aspect, the embodiments of the present application provide an electronic device, comprising a processor and a memory.

[0025] The memory is configured to store a computer program.

[0026] The processor is configured to execute the program stored on the memory, and implement any of the human pose estimation methods provided in the present application.

[0027] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement any of the human pose estimation methods provided in the present application.

[0028] In a fifth aspect, the embodiments of the present application provide a computer program product containing instructions, which, when executed on a computer, cause the computer to perform any of the human pose estimation methods provided in the present application.

[0029] The present application has the following advantages:

[0030] The application realizes deep fusion of key point spatial relationship and context information by constructing a multi-stage structure modeling framework, combining heat map guided key point initialization, graph attention network modeling of spatial structure, and modeling of global semantic relationship by a Transformer encoder. Compared with traditional heat map regression or direct coordinate regression methods, the application not only improves the accuracy of key point positioning in pose estimation, but also significantly enhances the structure expression ability and the consistency of the whole body pose. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0032] Figure 1 is a general flowchart of the embodiments of the present application.

[0033] Figure 2 is a human pose estimation flowchart of the embodiments of the present application.

[0034] Figure 3 is a result graph of the present application tested on real world human images. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0036] The present application proposes a human pose estimation method based on graph attention and Transformer, designs a multi-stage modeling framework consisting of heat map guided structure initialization, graph attention network (GAT) modeling of local spatial dependence, Transformer encoder extracting global context information, and pose decoder fusing and optimizing, which takes into account both local precision and global consistency. The method effectively improves the structure modeling ability and context expression ability of human pose estimation, and solves the problems of structure information loss, low reasoning efficiency and lack of global correlation modeling in traditional methods.

[0037] As shown in Figure 1 The present application provides a human pose estimation method based on graph attention and Transformer, comprising the following steps:

[0038] Step 1: Introduce a public human pose dataset, which contains images and corresponding two-dimensional human keypoint annotations.

[0039] The task of the present application relates to two-dimensional estimation of human poses, and therefore requires a human keypoint annotated image dataset containing diversified poses and perspectives to support sufficient training of the model and improvement of its generalization ability. The present application first screens samples containing rich human motions and high-resolution images from public human pose datasets (such as COCO, Human3.6M, etc.), and obtains the corresponding two-dimensional keypoint annotations. In order to enhance the adaptability of the model to different perspectives, scales and image conditions, data augmentation operations including rotation, scaling, flipping and illumination changes are performed on the original image data to construct a training sample set with diversity.

[0040] Step 2: Input the input image into a convolutional neural network with multi-scale feature extraction capability to extract a deep feature map as an initial feature, input it into a heat map decoder, obtain the keypoint preliminary prediction coordinates according to the heat map, and perform point guided sampling on the feature map to construct a keypoint feature matrix.

[0041] The input image is first input into a convolutional neural network with multi-scale feature extraction capability (such as ResNet or HRNet), i.e. the backbone network, to extract shallow (C3), middle (C4) and deep (C5) feature maps in turn, which correspond to local texture, structural semantics and global pose information, respectively. When the backbone network adopts ResNet, C3, C4 and C5 are the output feature maps of the 3rd, 4th and 5th stages of the backbone network, respectively, with different receptive fields and semantic granularities. The deep feature map C5 in the backbone network has integrated spatial and semantic information of the shallow and middle layers in the forward propagation process, and has strong context modeling capability. Therefore, C5 is selected as the final output feature, which has fused the semantic information of the previous layers in the layer-by-layer transmission process as the initial feature (Initial Feature). Subsequently, the initial feature (Initial Feature) is input into the heat map decoder (Heatmap Decoder), which is composed of multiple stacked convolutional layers, for gradually enhancing the spatial positioning capability and generating a keypoint heat map. The output of the heat map decoder is a heat map with a size of HxWxC, where H and W are the height and width of the input image, respectively, and C is the number of keypoint categories. where K denotes the number of human keypoint categories (e.g. head, shoulder, elbow, knee, etc.), and H and W denote the height and width of the heat map, respectively. Each channel of the heat map tensor corresponds to a keypoint category, and can be regarded as a two-dimensional probability heat map, which is used to represent the confidence distribution of the keypoint at each position in the image space as its predicted position. The higher the pixel value in the channel, the more likely the corresponding spatial position is the real position of the keypoint. The entire heat map tensor is composed of multiple channels, which jointly depict the spatial distribution patterns of all key points in the image space. Further, maximum value position extraction operations are performed on each channel of the heat map tensor respectively, to obtain the position coordinates with the highest confidence from each channel as the predicted two-dimensional coordinates of the corresponding keypoint, which are used to represent the preliminary positioning results of each keypoint in the image. After obtaining the predicted coordinates of all key points, a point-guided feature sampling operation is performed on the initial feature map guided by the set of coordinates. Specifically, for each keypoint, its predicted position is mapped to the spatial dimensions of the initial feature map, and the channel features at that position are extracted by bilinear interpolation to form the semantic representation of the keypoint. The interpolation operation has spatial continuity and differentiability, which is suitable for the end-to-end training process of the network. After completing the sampling of all key points, a keypoint feature matrix is constructed, where each row represents a keypoint feature vector that integrates its spatial position information and local semantic features.

[0042] Step 3: Construct a skeleton graph based on the keypoint feature matrix, and jointly model the local topological structure and global semantic dependency between key points by a graph attention network (GAT) and a Transformer encoder to generate structure-aware keypoint features.

[0043] After obtaining the keypoint feature matrix, a keypoint graph is constructed based on the standard human skeleton topology, which is used to model the spatial structure relationship between keypoints. Specifically, each keypoint is regarded as a node in the graph, and its input feature is the corresponding row vector in the keypoint feature matrix, representing the high-dimensional semantic feature of the keypoint. The entire feature matrix constitutes the input set of all nodes in the graph, integrating the local semantic information of the keypoint and the image spatial position. The connection relationship between nodes in the graph (i.e., the edges of the graph) is constructed based on the skeleton structure defined in public human pose datasets such as COCO or Human3.6M, representing the anatomical connection relationship between keypoints, such as the shoulder and elbow, knee and ankle, etc. The constructed keypoint graph is input into the Graph Attention Network (GAT) to model the local topological dependence between keypoints. GAT dynamically allocates attention weights between adjacent keypoints by introducing an adaptive attention mechanism, thereby enhancing the feature interaction capability between structure-related nodes and obtaining graph node features with spatial structure perception capability. To further enhance the modeling capability of the global pose structure, the node feature sequence output by GAT is input into the Transformer encoder, which utilizes its multi-head self-attention mechanism to realize feature interaction between all keypoints. This module can capture the global semantic dependence between distant keypoints, making up for the deficiency of local graph modeling in long-distance dependence modeling. The final output is structure-aware keypoint features, which contain both the spatial connection structure between keypoints and the global pose semantic information.

[0044] Step 4: The structure-aware keypoint features are input into the joint decoder to perform feature analysis and coordinate regression, outputting a set of two-dimensional coordinates of human keypoints in the image, representing the structure-consistent human pose estimation result.

[0045] The structure-aware keypoint features are input into the joint decoder (Joint Decoder) to realize two-dimensional coordinate regression of each keypoint in the image. This structure-aware keypoint feature integrates local semantic information sampled from the image, spatial topology modeled by the Graph Attention Network (GAT), and global pose semantic information extracted by the Transformer encoder, possessing clear position directionality and structure expression capability. In the joint decoder, first, a nonlinear transformation is performed on each structure-aware keypoint feature to enhance the expressiveness and separability of the feature. Then, a Feed-Forward Network (FFN) is used for point-by-point feature mapping, and an Add&Norm residual connection mechanism is introduced to enhance the stability of network training and the efficiency of feature transmission. Finally, a Multi-Layer Perceptron (MLP) is used to map it to a two-dimensional coordinate in the image space, i.e., The prediction results of all key points constitute a two-dimensional key point coordinate set , representing the pose estimation results of the human structure consistent and spatially coherent in the image.

[0046] Step 5: The training stage adopts a mean square error loss function (MSE) to supervise the difference between the predicted heat map and the real label, guide the collaborative optimization of the backbone network, the structure modeling module and the joint decoder, and improve the key point positioning accuracy and structural consistency.

[0047] In the model training stage, the present application adopts a mean square error (MSE) loss function to supervise the difference between the predicted key point coordinates and the real label to optimize the pose estimation accuracy. The loss function is calculated point by point for the two-dimensional coordinates of each key point, and the formula is as follows:

[0048]

[0049] Wherein, K: represents the number of human body joints, and the number of human body joints is 17 in the Human3.6M dataset and the COCO dataset. : the kth key point coordinate predicted by the model, represents the corresponding real key point coordinate.

[0050] Table 1

[0051]

[0052] The embodiments of the present application are compared with various mainstream methods on the COCO test-dev2017 and val2017 datasets, as shown in Table 1 above. The method of the present application adopts a ResNet backbone network, the input resolution is 256x256, the calculation amount is 34.1 GFLOPs, and the inference speed reaches 76.4 fps, showing good efficiency. Although a lower input resolution is adopted, the method of the present application achieves an AP of 77.4 on the COCO test-dev2017 and an AP of 78.3 on the val2017, which are better than the methods of TokenPose, SimCC and ViTPose (lightweight version), and close to or even exceed some high-resolution models such as HRNet and TransPose in accuracy. The advantage of the method of the present application lies in the introduction of structure perception capability: the spatial dependence between key points is modeled through a graph attention network, and the global semantic relationship is captured through a Transformer encoder, which improves the completeness and accuracy of feature modeling. At the same time, the point-guided sampling mechanism effectively preserves the local semantics and spatial position of the image, enhancing the pose expression ability under low resolution conditions. Overall, we achieve a balance between speed and accuracy under the lightweight design.Figure 3 This is the result of testing the present invention on real-world human body images.

[0053] The present application also provides a structure-aware human posture estimation device, which includes the following modules:

[0054] Data acquisition module: used to obtain public human posture datasets containing images and corresponding two-dimensional human key point annotations.

[0055] Matrix construction module: The deep feature map is extracted through the backbone network as the initial feature, which is input into the heat map decoder. The preliminary predicted coordinates of the key points are obtained according to the heat map, and point-guided sampling is performed on the feature map to construct the key point feature matrix.

[0056] Structure-aware module: Builds a skeleton graph based on the key point feature matrix, and jointly models the local topological structure and global semantic dependency between key points through the graph attention network GAT and the Transformer encoder to generate structure-aware key point features.

[0057] Posture Estimation Module: Based on the key point features of structure perception, feature parsing and coordinate regression are performed through the joint decoder, and a two-dimensional coordinate set of the key points of the human body in the image is output to represent the structurally consistent human posture estimation results.

[0058] In a possible implementation, the matrix construction module specifically includes the following operations:

[0059] The input image is first passed through a convolutional neural network with multi-scale feature extraction capabilities to extract a deep feature map as the initial feature. Subsequently, the initial features are input into the heat map decoder, which outputs a multi-channel heat map tensor. Each channel of the heat map tensor corresponds to a key point category. The maximum position extraction operation is performed on each channel of the heat map tensor, and the position coordinates with the highest confidence are obtained from each channel as the predicted two-dimensional coordinates of the corresponding key point. After obtaining the predicted coordinates of all key points, a point-guided feature sampling operation is performed on the initial feature map with this set of coordinates as a guide. Specifically, for each key point, its predicted position is mapped to the spatial dimension of the initial feature map, and the channel features at this position are extracted by bilinear interpolation to form a semantic representation of the key point. After completing the sampling of all key points, a key point feature matrix is ​​constructed, in which each row represents the feature vector of a key point, which combines its spatial position information with local semantic features.

[0060] In a possible implementation, the structure perception module is specifically as follows:

[0061] Each key point is regarded as a node in the graph, and the input feature of the node is the corresponding row vector in the key point feature matrix, representing the high-dimensional semantic feature of the key point. The entire feature matrix constitutes the input set of all nodes in the graph, which integrates the local semantic information of the key point and the spatial position of the image. The connection relationship between the nodes in the graph is constructed according to the skeleton structure, which is used to represent the anatomical connection relationship between the key points.

[0062] The constructed key point graph is input into the graph attention network GAT to model the local topological dependence between the key points. The node feature sequence output by the GAT is input into the Transformer encoder, and the multi-head self-attention mechanism of the Transformer encoder is used to realize the feature interaction between all the key points, and finally the structure-aware key point features are output.

[0063] In a possible implementation, in the joint decoder of the pose estimation module, first, a nonlinear transformation is performed on each structure-aware key point feature. Then, a feedforward neural network is used for point-by-point feature mapping, and an Add&Norm residual connection mechanism is introduced to enhance the stability of network training and the efficiency of feature transmission. Finally, a multi-layer perceptron is used to map to a two-dimensional coordinate in the image space. The prediction results of all the key points constitute a two-dimensional key point coordinate set, representing the pose estimation result of the human body structure consistent and spatially coherent in the image.

[0064] In a possible implementation, the device further includes a training module: a mean square error loss function (MSE) is used to supervise the difference between the predicted heat map and the real label, guide the collaborative optimization of the backbone network, the structure modeling module and the joint decoder, and improve the key point positioning accuracy and the structure consistency.

[0065] Embodiments of the present application also provide an electronic device, and embodiments of the present application provide an electronic device, which includes a processor and a memory.

[0066] The memory is configured to store a computer program.

[0067] The processor is configured to execute the program stored in the memory, and implement any of the methods described in the present application.

[0068] In a possible implementation, the electronic device of the embodiments of the present application further includes a communication interface and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0069] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0070] The communication interface is used for communication between the above electronic device and other devices.

[0071] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0072] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0073] In another embodiment provided in the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the method described in any of the embodiments of the present application.

[0074] In another embodiment provided in the present application, a computer program product containing instructions is also provided, and when the computer program product is run on a computer, the computer is caused to execute the method described in any of the embodiments of the present application.

[0075] In the embodiments described above, all or some of the steps can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or some of the steps can be implemented in the form of one or more computer programs. The computer program can be stored in any computer readable medium, and when loaded into a computer system, the computer program can instruct the computer system to carry out a series of procedures or functions. The computer readable medium includes magnetic disk, optical disk, read-only memory (ROM), flash memory and the like, or any other medium suitable for storing computer programs. The computer system includes a general purpose computer, a special purpose computer, an embedded computer, a computer network, or other programmable apparatuses. The computer program can be transmitted from one computer readable medium to another computer readable medium in the form of a computer program instruction, for example, being transmitted from a website, a computer, a server or a data center to another website, computer, server or data center through a wired (such as a coaxial cable, an optical fiber, a digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable medium can be any available medium that can be accessed by a computer, or a data storage device such as a server, data center, etc. that includes one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0076] It should be noted that the relative terms such as first and second, etc., are used only to differentiate one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0077] Each of the embodiments in the specification is described in a related manner, and each embodiment focuses on the difference from other embodiments. The same and similar parts between the embodiments can be referred to each other.

[0078] The above only describes the preferred embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A human pose estimation method based on graph attention and Transformer, characterized in that: The steps are as follows Obtain a public human pose dataset containing images and corresponding 2D human keypoint annotations; The input image is fed into the backbone network, and after extracting the deep feature map as the initial feature, it is input into the heat map decoder. The preliminary predicted coordinates of the key points are obtained according to the heat map, and point-guided sampling is performed on the feature map to construct the key point feature matrix. A skeleton graph is constructed based on the key point feature matrix. The local topological structure and global semantic dependency between key points are jointly modeled through the graph attention network (GAT) and the Transformer encoder to generate structure-aware key point features. The structure-aware key point features are input into the joint decoder for feature parsing and coordinate regression, and a two-dimensional coordinate set of the human body key points in the image is output to represent the structurally consistent human posture estimation results.

2. A human pose estimation method based on graph attention and Transformer according to claim 1, characterized in that The key point feature matrix construction specifically includes the following operations: The input image is first passed through a convolutional neural network with multi-scale feature extraction capabilities to extract a deep feature map as the initial feature; then, the initial feature is input into a heat map decoder, and a multi-channel heat map tensor is output; each channel of the heat map tensor corresponds to a key point category, and a maximum position extraction operation is performed on each channel of the heat map tensor, and the position coordinates with the highest confidence are obtained from each channel as the predicted two-dimensional coordinates of the corresponding key point; after obtaining the predicted coordinates of all key points, a point-guided feature sampling operation is performed on the initial feature map guided by this set of coordinates; specifically, for each key point, its predicted position is mapped to the spatial dimension of the initial feature map, and the channel features at that position are extracted by bilinear interpolation to form a semantic representation of the key point; after completing the sampling of all key points, a key point feature matrix is ​​constructed, in which each row represents the feature vector of a key point, which combines its spatial position information with local semantic features.

3. The human pose estimation method based on graph attention and Transformer according to claim 1, characterized in that The skeleton graph is constructed based on the key point feature matrix as follows: Each key point is regarded as a node in the graph, and its input feature is the corresponding row vector in the key point feature matrix, which represents the high-dimensional semantic feature of the key point; the entire feature matrix constitutes the input set of all nodes in the graph, integrating the local semantic information of the key point with the image spatial position; the connection relationship between the nodes in the graph is constructed based on the skeleton structure, which is used to represent the anatomical connection relationship between the key points.

4. The method for human pose estimation based on graph attention and Transformer according to claim 3, characterized in that: The constructed key point graph is input into the graph attention network (GAT) to model the local topological dependencies between key points. The node feature sequence output by GAT is input into the Transformer encoder, and its multi-head self-attention mechanism is used to realize feature interaction between all key points, and finally the structure-aware key point features are output.

5. The method for human pose estimation based on graph attention and Transformer according to claim 1, characterized in that: In the joint decoder, each structure-aware keypoint feature is first subjected to a nonlinear transformation; Subsequently, a feedforward neural network is used for point-by-point feature mapping, and the Add&Norm residual connection mechanism is introduced to enhance the stability of network training and the efficiency of feature transfer. Finally, a multi-layer perceptron is used to map the data into two-dimensional coordinates in the image space. The prediction results of all key points constitute a two-dimensional key point coordinate set, which represents the posture estimation results of the human body in the image with consistent structure and spatial coherence.

6. The method for human pose estimation based on graph attention and Transformer according to claim 1, characterized in that: It also includes a training phase, which uses the mean square error loss function to supervise the difference between the predicted heat map and the true annotation, guiding the coordinated optimization of the backbone network, structural modeling module and joint decoder to improve the key point positioning accuracy and structural consistency.

7. A human pose estimation device based on graph attention and Transformer, characterized in that: Includes the following modules: Data acquisition module: used to obtain public human posture datasets containing images and corresponding two-dimensional human key point annotations; Matrix construction module: A convolutional neural network with multi-scale feature extraction capability extracts deep feature maps as initial features, which are then fed into a heat map decoder. The decoder then obtains the initial predicted coordinates of key points based on the heat map. Point-guided sampling is then performed on the feature map to construct a key point feature matrix. Structure-aware module: Builds a skeleton graph based on the key point feature matrix, and jointly models the local topological structure and global semantic dependency between key points through the graph attention network (GAT) and the Transformer encoder to generate structure-aware key point features. Posture Estimation Module: Based on the key point features of structure perception, feature parsing and coordinate regression are performed through the joint decoder, and a two-dimensional coordinate set of the key points of the human body in the image is output to represent the structurally consistent human posture estimation results.

8. An electronic device, characterized in that: including processor and memory; The memory is used to store computer programs; The processor is used to implement the human body posture estimation method described in any one of claims 1 to 6 when executing the program stored in the memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the human body posture estimation method according to any one of claims 1 to 6 is implemented.

Images