Human pose estimation method and device in body-aware occlusion scene, and medium

Abstract

The application discloses a human pose estimation method and device in a body-attached intelligent shielding scene, and relates to the technical field of body-attached intelligence, and comprises the following steps: performing convolution feature extraction on an input image to obtain image features; performing initial pose estimation on the image features to obtain initial positions of all joint nodes; constructing a human skeleton graph with the initial positions of all joint nodes as vertices and human skeleton bone connections as edges, and obtaining features of each joint node at the initial positions from the image features as vertex initial features; on the human skeleton graph, determining edge weights according to adjacent joint node joint types and relative positions, performing multi-layer graph convolution message passing based on the edge weights, complementing the features of the shielded joint nodes by the features of adjacent joint nodes, and obtaining a completed joint node feature map; and performing coordinate regression on the completed joint node feature map to obtain position coordinates of all joint nodes. The application uses a human skeleton topological structure to complement features, thereby improving the pose estimation accuracy in a shielding scene.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method, device and medium for estimating human posture in embodied intelligence occlusion scenarios. Background Technology

[0002] In embodied intelligence systems, machine vision technology needs to accurately estimate human posture from images or videos to support the intelligent agent in understanding human actions and achieving natural interaction. In the physical world, human movement is constrained by physical models such as bone length and joint connections. However, frequent occlusions in real-world scenes make some joints invisible, making it difficult for machine vision algorithms to obtain complete observations. This leads to increased posture estimation errors and consequently affects the reliability of embodied intelligence systems in real-world environments.

[0003] Currently, existing technologies typically employ convolutional neural networks to extract image features, followed by direct regression of all key node coordinates based on visible region features. However, when key points are occluded, the corresponding regions lack effective visual information, rendering the regression process unreliable and resulting in estimations that often deviate significantly from the actual positions. Furthermore, another approach attempts to infer features of occluded key points using the local image context surrounding them. However, this local context is susceptible to interference from the appearance of occluders, cluttered backgrounds, and similar body parts, leading to poor stability of the inferred features. These shortcomings significantly reduce pose estimation accuracy in real-world environments such as dense crowds and object occlusion, severely limiting the action understanding and interaction capabilities of embodied intelligent systems in complex physical scenarios. Summary of the Invention

[0004] This invention provides a method, device, and medium for human pose estimation in occlusion scenarios, and aims to solve the technical problem of how to provide a solution that can effectively improve the accuracy of human pose estimation in occlusion scenarios.

[0005] In a first aspect, the present invention provides a method for estimating human pose in an embodied intelligent occlusion scenario, comprising: In response to receiving an input image, convolutional feature extraction is performed on the input image to obtain image features; Initial pose estimation is performed on the image features to obtain the initial positions of all joints; Using the initial positions of all the joints as vertices and the bone connections between the joints in the human skeleton as edges, a human skeleton graph is constructed, and the features of each joint at the initial position are obtained from the image features as the initial features of each vertex in the human skeleton graph. On the human skeleton map, edge weights are determined based on the joint type and relative position between adjacent joints, and multi-layer graph convolution message passing is performed based on the edge weights so that the features of the occluded joints are completed by the features of the adjacent joints, resulting in a completed joint feature map. Coordinate regression is performed on the completed joint feature map to obtain the position coordinates of all joint nodes.

[0006] Secondly, the present invention also provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0007] Thirdly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0008] This invention provides a method, device, and medium for human pose estimation in embodied intelligent occlusion scenarios. Image features are extracted through convolutional feature extraction, providing a feature foundation for subsequent joint localization. Further, initial pose estimation is performed on the image features to obtain the initial positions of all joints, ensuring that the initial position estimates of each joint can still be obtained even under occlusion. Further, a human skeleton graph is constructed using the initial positions of all joints as vertices and skeletal connections as edges, and the initial features of each vertex are obtained from the image features, introducing the human skeleton topology into the feature representation. Further, on the human skeleton graph, edge weights are determined based on the joint type and relative position between adjacent joints, making the edge weights reflect the structural correlation between adjacent joints. Multi-layer graph convolutional message passing is performed based on the edge weights, allowing the features of occluded joints to be completed by the features of adjacent joints, enhancing the robustness and accuracy of the occluded joint features. Further, coordinate regression is performed on the completed joint feature graph to obtain the position coordinates of all joints. Therefore, this method uses the topological prior of the human skeleton for feature completion, and can still obtain effective feature representations when some joints are not visible, thereby improving the accuracy of human pose estimation in occluded scenes. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating a human pose estimation method in an embodied intelligent occlusion scenario provided by an embodiment of the present invention; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0013] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0016] Please see Figure 1This invention provides a method for estimating human posture in embodied intelligent occlusion scenarios. Accurately understanding human posture and movements is a fundamental capability for embodied intelligent robots performing visual perception and human-computer interaction tasks. Furthermore, embodied intelligent robots typically use cameras or depth sensors to acquire images and identify human posture by locating human joints in the images. However, in real physical environments, the human body is often occluded by objects, other people, or parts of the body itself, making some joints invisible or only partially visible in the image. For example, in crowded places, a person in the camera's field of view may be occluded by others; in indoor scenes, furniture such as tables, chairs, and cabinets may obscure the lower limbs or torso; in cases of self-occlusion, turning or crossing arms can also cause some joints to become invisible. Furthermore, occlusion makes it difficult to directly infer the location of occluded joints from images because the corresponding image area lacks effective visual evidence. Traditional methods often produce large localization errors, severely impacting subsequent tasks such as action recognition and intent understanding. To address this issue, this invention proposes a joint-aware feature completion network based on prior knowledge of the human skeleton structure. By modeling the human skeleton as a graph, information is propagated at the structural level using a graph convolutional network, allowing the features of occluded joints to be completed by adjacent, visible joint features, thereby improving the accuracy of human pose estimation in occluded scenarios. Furthermore, this method can be deployed on embodied intelligent robots, such as service robots, inspection robots, and interactive robots, or it can be deployed on a cloud server for use by terminal devices. This embodiment utilizes the inherent topological structure of the human skeleton as a strong prior, and performs feature reasoning and completion at the structural level through a graph convolutional network, solving the problem of large localization errors when joints are invisible due to severe occlusion. Specifically, the method includes the following steps: S1, in response to receiving an input image, performs convolutional feature extraction on the input image to obtain image features.

[0017] In specific implementation, the input image is an image containing a human body captured by an image sensor mounted on an embodied intelligent robot. The image sensor can be a color camera, an infrared camera, a depth camera, or a combination thereof. Furthermore, the input image can be a single-frame static image or a frame from a continuous video stream. The resolution, color format, lens distortion, etc., of the input image can be adjusted according to the actual hardware configuration, and this invention does not impose any limitations on this. Further, after receiving the input image, the system performs convolutional feature extraction on the input image. Convolutional feature extraction is implemented through a convolutional neural network. A convolutional neural network is a deep neural network containing convolutional layers, pooling layers, normalization layers, and nonlinear activation layers, capable of extracting feature representations with spatial resolution and semantic information from the original image. Further, the convolutional layers slide multiple learnable convolutional kernels across the input image, calculating a weighted sum of local regions to generate a feature map. The weights and biases of the convolutional kernels are learnable parameters of the convolutional neural network. The pooling layer reduces the spatial size of the feature map through downsampling, expanding the receptive field while reducing computational cost. Nonlinear activation layers introduce nonlinear transformations, and commonly used activation functions include Rectified Linear Unit (ReLU) and Leaky ReLU. Furthermore, the network structure for convolutional feature extraction can employ common backbone networks such as ResNet, MobileNet, and Hourglass. This invention does not restrict the specific network structure, as long as it can extract multi-channel feature maps from the input image. Furthermore, the obtained image features are a set of multi-channel two-dimensional feature maps, whose spatial resolution is usually lower than that of the input image, but with more channels, each corresponding to a type of feature response. Furthermore, the image features contain rich appearance information, edge information, texture information, and high-level semantic information, providing a unified feature foundation for subsequent initial pose estimation and keypoint feature extraction. Furthermore, the spatial resolution of the image features may be reduced by a factor of two compared to the input image, for example, reduced to 1 / 4 or 1 / 8 of the original size; the specific scaling ratio depends on the design of the backbone network.

[0018] S2, perform initial pose estimation on the image features to obtain the initial positions of all joints.

[0019] In specific implementation, the purpose of initial pose estimation is to perform a rough localization of all key points before introducing an occlusion completion mechanism. All key points refer to the predefined set of human key points based on the human pose estimation task. Further, common key point sets include the head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle, etc. The number of key points can be increased or decreased according to specific application scenarios, and this invention does not specifically limit this. Further, the initial position is the two-dimensional coordinate of each key point in the image coordinate system, or the three-dimensional coordinate in three-dimensional space. Further, for occluded key points, due to the lack of effective visual evidence in the image, the estimated value of the initial position may have a large deviation, but it can still serve as a reference for subsequent construction of the human skeleton map and extraction of initial vertex features. Further, initial pose estimation can be implemented in various ways, such as heatmap regression, direct coordinate regression, heatmap plus offset regression, etc., and this invention does not specifically limit this. This embodiment does not limit the specific implementation method of initial pose estimation, as long as it can output the initial positions of all key points. Image features are first processed by an initial pose estimation network, which outputs the initial positions of all joints. This initial pose estimation network can be a lightweight regression head, for example, by applying several convolutional and fully connected layers to the image features, directly outputting the coordinates of each joint. Alternatively, a heatmap regression method can be used, generating a heatmap for each joint and then obtaining the joint coordinates by taking the peak position of the heatmap. Furthermore, heatmap regression preserves spatial information and has relatively high localization accuracy, making it a commonly used initial pose estimation method.

[0020] In some preferred embodiments, the initial pose estimation of the image features to obtain the initial positions of all joints includes: performing heatmap regression on the image features to obtain a heatmap of each joint in all the joints; normalizing the heatmap of each joint to obtain a probability distribution map of each joint; multiplying each probability distribution map element-wise with a preset coordinate grid and summing the results to obtain the initial position coordinates of each joint; and using the initial position coordinates of each joint as the initial positions of all joints.

[0021] In practical implementation, in the complete human pose estimation process, initial pose estimation is used to obtain the initial positions of all joints, which is a prerequisite for constructing the human skeleton map and extracting initial vertex features. The accuracy of the initial positions directly affects the quality of subsequent convolutional feature completion. Therefore, this embodiment estimates the initial positions of all joints from image features by combining heatmap regression and integral regression, avoiding the positioning error caused by insufficient spatial resolution of the network output when directly regressing coordinates. The specific implementation method is as follows: First, the system receives the input image and performs convolutional feature extraction to obtain image features. The input image is acquired by an image sensor and contains a human body, either fully or partially occluded. Convolutional feature extraction is performed through a convolutional neural network, generating multi-channel image features. These image features have a lower spatial resolution than the input image but contain rich semantic information.

[0022] Furthermore, the system performs heatmap regression on the image features to obtain a heatmap for each joint in all relevant nodes. Heatmap regression is a commonly used joint localization method. The system predicts a heatmap for each joint, and the spatial size of the heatmap can be consistent with the spatial size of the image features, or it can be upsampled to the same resolution as the input image. Further, the value at each position in the heatmap represents the probability or response intensity that the position belongs to the center of the corresponding joint. Heatmap regression can be implemented by applying one or more convolutional layers to the image features. Each convolutional layer can include batch normalization and non-linear activation. Batch normalization is used to normalize the input and stabilize the training process. Further, the heatmaps of all relevant nodes constitute a multi-channel heatmap tensor, with the number of channels matching the number of joints. For visible joints, the heatmap typically shows a peak near the actual joint location; the peak position is the predicted joint location. For occluded joints, the peak in the heatmap may not be prominent or may deviate from the actual location, but it still provides some prior knowledge of the location.

[0023] Furthermore, the system normalizes the heatmap of each joint point to obtain a probability distribution map for each joint point. Normalization can be achieved using the Softmax function, which maps the response values ​​at each location in the heatmap to probability values, ensuring that the sum of the probabilities at all locations in the probability distribution map for each joint point is 1. The Softmax function is an exponential normalization function, capable of converting any real-valued vector into a probability distribution. The normalized probability distribution map reflects the probability that each location is the center of the joint point. For heatmaps with obvious peaks, the normalized probability distribution is concentrated near the peaks; for heatmaps with less obvious peaks, the probability distribution is more dispersed. Normalization provides a probabilistic interpretation for subsequent integral regression and also makes the probability distributions between different joint points comparable.

[0024] Furthermore, the system performs element-wise multiplication and summation of each probability distribution map with a preset coordinate grid to obtain the initial position coordinates of each joint. The preset coordinate grid is a two-dimensional grid with the same size as the probability distribution map, and each grid position corresponds to a fixed two-dimensional coordinate, such as normalized coordinates or pixel coordinates. Element-wise multiplication means multiplying the probability value of each position in the probability distribution map with the coordinate value of the corresponding position in the preset coordinate grid, and then summing the products of all positions to obtain the weighted average coordinates of the joint. This process is also called integral regression or soft Argmax operation. Since the sum of all probabilities in the probability distribution map of each joint is 1, the weighted average coordinates can be understood as the expected value of the joint position. Compared with the hard Argmax operation that directly takes the peak position of the heatmap, integral regression can obtain continuous coordinate values, and the entire calculation process is differentiable, which is beneficial for end-to-end training. Integral regression can also output a compromise position estimate when there are multiple peaks in the heatmap, and has a certain robustness to occlusion.

[0025] Furthermore, the system uses the initial position coordinates of each joint as the initial position of all joints. The set of initial positions of all joints will be used to subsequently construct the human skeleton map and to sample the features of each joint at its initial position from the image features.

[0026] This embodiment obtains the probability distribution map of each joint point through heatmap regression and obtains continuously differentiable initial position coordinates through integral regression, improving the accuracy of initial localization and training stability. Compared with direct coordinate regression, heatmap regression preserves spatial information, and integral regression avoids the non-differentiability problem caused by peak value taking, enabling the network to more effectively optimize joint point localization capabilities during training. Furthermore, the accuracy of the initial position provides a more reliable basis for vertex positions and feature sampling for subsequent graph convolution feature completion. Moreover, in occluded scenarios, the heatmap of occluded joint points may be unclear; integral regression can provide a reasonable initial position estimate based on the expected value of the probability distribution, providing an effective starting point for subsequent graph convolution feature completion.

[0027] S3. Using the initial positions of all the joints as vertices and the bone connections between the joints in the human skeleton as edges, construct a human skeleton graph, and obtain the features of each joint at the initial position from the image features as the initial features of each vertex in the human skeleton graph.

[0028] In specific implementation, the human skeleton graph is a graph data structure used to represent the spatial topological relationships of human joints. Vertices in the human skeleton graph correspond to all joints, and edges correspond to the skeletal connections between joints in the human skeleton. Furthermore, the topological structure of the human skeleton is predefined, reflecting the connection relationships of the human skeleton. For example, there is a skeletal connection between the left shoulder and the left elbow, therefore there is an edge between the left shoulder vertex and the left elbow vertex in the human skeleton graph; there is a skeletal connection between the left elbow and the left wrist, therefore there is an edge between the left elbow vertex and the left wrist vertex. Furthermore, the human skeleton graph can be an undirected graph or a directed graph; this embodiment uses an undirected graph for description, but a directed graph can also be used according to actual needs. Furthermore, when constructing the human skeleton graph, the system uses the initial positions of all joints obtained in step S2 as the spatial coordinates of each vertex, and determines the edges based on the preset skeletal connection relationships. Furthermore, the set of edges in the human skeleton graph is fixed and does not change with the input image, but the position coordinates of the vertices change with the input image. After constructing the human skeleton graph, initial features need to be assigned to each vertex. The initial features are derived from image features. Specifically, the system samples the features of each joint point at its initial position from the image features. Further, bilinear interpolation can be used for sampling, which involves obtaining the feature vector at the corresponding position based on continuous coordinates on a discrete feature map. Bilinear interpolation is a commonly used feature sampling method that can obtain a smooth sampling result by weighted averaging the feature values ​​of the surrounding four integer positions when the coordinates are not integers. Since the spatial resolution of image features is usually lower than that of the input image, the initial position coordinates need to be transformed according to the spatial scaling ratio of the image features to obtain the sampling coordinates in the image feature coordinate system. The initial feature vector of each vertex sampled from the image features contains local appearance information and contextual information near that joint point. Further, for visible joint points, the initial feature vector contains reliable features related to human body parts; for occluded joint points, the initial feature vector may contain the appearance features of the occluding object, which has lower reliability.

[0029] In some preferred embodiments, the step of obtaining the features of each joint at the initial position from the image features as the initial features of each vertex in the human skeleton diagram includes: obtaining the features of each joint in all nodes at the initial position from the image features; performing visibility prediction on the features of each joint at the initial position to obtain a visibility prediction score for each joint in all nodes; comparing the visibility prediction score of each joint with a preset threshold; if the visibility prediction score is greater than the preset threshold, determining that the joint is visible; if the visibility prediction score is less than or equal to the preset threshold, determining that the joint is invisible; generating a visibility mask based on the determination result, wherein the weight corresponding to a joint determined to be visible is 1, and the weight corresponding to a joint determined to be invisible is 0; multiplying the features of each joint at the initial position with the corresponding element of the visibility mask to obtain a weighted initial feature; and using the weighted initial feature as the initial feature of each vertex in the human skeleton diagram.

[0030] In practical implementation, within the complete human pose estimation process, initial features are used as the initial representations of each vertex in the human skeleton graph. Subsequent graph convolutional message passing will then complete these features. In real-world occlusion scenarios, the initial features of occluded joints may contain information about the appearance of the occluder. Directly using such features as initial vertex features would introduce noise into the graph convolutional message passing. Therefore, this embodiment generates a visibility mask through visibility prediction and weights the initial features to suppress interference from the initial features of invisible joints. The specific implementation is as follows: First, the system receives the input image and performs convolutional feature extraction to obtain image features. Simultaneously, the system performs initial pose estimation on the image features to obtain the initial positions of all joints. The image features are multi-channel feature maps output by the convolutional neural network, and the initial positions are the two-dimensional or three-dimensional coordinates of each joint in the image coordinate system.

[0031] Furthermore, the system extracts the features of each joint at its initial position from the image features. For each joint, the system uses bilinear interpolation to sample the feature vector at that position, centered on the initial position of the joint. The image features are multi-channel feature maps output by a convolutional neural network, and bilinear interpolation can obtain smooth feature values ​​from the discrete feature map based on continuous coordinates. The feature vector at the initial position of each joint is thus the initial candidate feature for that joint. The initial candidate feature contains local appearance and contextual information near the joint; however, due to occlusion, the initial candidate feature of occluded joints may contain appearance information of the occluding object, resulting in lower reliability.

[0032] Furthermore, the system performs visibility prediction on the features of each joint at its initial position across all relevant nodes, obtaining a visibility prediction score for each joint. Visibility prediction determines whether each joint is visible in the input image, i.e., whether it is occluded. The visibility prediction score is a scalar value representing the probability or confidence that the joint is visible. Visibility prediction can be implemented using a lightweight fully connected or convolutional layer, taking the feature vector of each joint at its initial position as input and outputting its visibility prediction score. This lightweight fully connected or convolutional layer constitutes the visibility prediction network. Since occluded joints typically contain information about the occlusion, their visibility prediction scores are lower; visible joints contain clear information about human body parts, resulting in higher visibility prediction scores. The visibility prediction network can learn the relationship between occlusion features and visibility during end-to-end training, thereby accurately predicting the visibility state of each joint.

[0033] Furthermore, the system compares the visibility prediction score of each joint with a preset threshold. The preset threshold is a pre-defined value used to distinguish between visible and invisible states, for example, set to 0.5. The system sequentially determines the relationship between the visibility prediction score of each joint and the preset threshold. If the visibility prediction score is greater than the preset threshold, the joint is determined to be visible; if the visibility prediction score is less than or equal to the preset threshold, the joint is determined to be invisible. Further, for joints determined to be visible, the system considers them to have reliable visual information in the input image; for joints determined to be invisible, the system considers them to lack reliable visual information in the input image.

[0034] Furthermore, the system generates a visibility mask based on the judgment result. Keys judged as visible have a weight of 1, while keys judged as invisible have a weight of 0. The visibility mask is a weight vector corresponding one-to-one with all key nodes, with each element taking a value of 1 or 0. Positions with a weight of 1 in the visibility mask indicate that the corresponding key is visible, while positions with a weight of 0 indicate that the corresponding key is invisible. The purpose of the visibility mask is to retain the features of visible key nodes and mask the features of invisible key nodes in subsequent feature weighting.

[0035] Furthermore, the system multiplies the features of each joint at its initial position with the corresponding element of the visibility mask to obtain weighted initial features. Element-wise multiplication means multiplying the initial candidate feature vector of each joint with the weight value corresponding to that joint in the visibility mask. For visible joints, the weight is 1, and the weighted initial features remain unchanged; for invisible joints, the weight is 0, and the weighted initial features become a zero vector. Based on this scheme, the initial features of invisible joints are set to zero. During subsequent graph convolution message passing, the features of these vertices will mainly be obtained from the aggregation of adjacent visible joints, rather than relying on their own unreliable initial features. Further, the system uses the weighted initial features as the initial features of each vertex in the human skeleton graph. These initial features will serve as the input features for the first layer of graph convolution and participate in subsequent graph convolution message passing.

[0036] This embodiment generates a visibility mask through visibility prediction and weights the initial features based on the visibility mask, setting the initial features of invisible joints to zero. This design avoids unreliable features of occluded joints interfering with neighboring joints during graph convolutional message passing, and also ensures that the completed features of occluded joints mainly originate from anatomically related visible joints, improving the reliability and interpretability of feature completion. Furthermore, the introduction of the visibility mask allows the network to explicitly model occlusion states, which is beneficial for improving the pose estimation accuracy in occluded scenarios. Moreover, during training, the visibility prediction network is jointly optimized with other network parameters, gradually learning accurate visibility judgment capabilities.

[0037] S4, on the human skeleton map, edge weights are determined according to the joint type and relative position between adjacent joints, and multi-layer graph convolution message passing is performed based on the edge weights, so that the features of the occluded joints are completed by the features of the adjacent joints, and the completed joint feature map is obtained.

[0038] In practice, edge weights are used to control the strength of feature aggregation between adjacent vertices during graph convolution message passing. Edge weights are not uniform but determined based on the joint type and relative position of adjacent joints. Joint type refers to the semantic category of each joint, such as left shoulder, left elbow, left wrist, right knee, etc. Furthermore, relative position refers to the spatial relationship between adjacent joint pairs in the image coordinate system, such as a relative offset vector. Joint type and relative position together determine the degree of anatomical association and spatial reliability between adjacent joint pairs. For example, there is a direct skeletal connection and anatomical association between the left elbow and left shoulder, and between the left elbow and left wrist, so the edge weights should be higher; while there is no direct skeletal connection between the left elbow and right knee, resulting in a low degree of anatomical association. If a high edge weight is incorrectly assigned between them, it will lead to the propagation of irrelevant information. Therefore, this embodiment determines edge weights jointly by joint type and relative position, so that anatomically related joint pairs have higher edge weights, while irrelevant joint pairs have lower edge weights. Furthermore, the edge weights can be computed through a learnable edge weight determination network that takes joint type identifiers and relative positions as inputs and outputs edge weight scalars.

[0039] Furthermore, after determining the edge weights, the system performs multi-layer graph convolutional message passing on the human skeleton graph. Graph convolutional message passing is a method for feature propagation on a graph. In each layer of graph convolution, each vertex weights and aggregates the features of its adjacent vertices according to the edge weights. The aggregated result is then fused with its own features, followed by feature transformation and activation processing to obtain updated vertex features. Multi-layer graph convolutional message passing involves repeating the above process multiple times, allowing features to propagate multiple layers along the human skeleton graph. After multi-layer message passing, the initial features of the occluded joint may be unreliable, but the features of its adjacent visible joints can be effectively propagated, completing the feature representation of the occluded joint. For example, when the left elbow is occluded, the left shoulder and left wrist are visible. Through graph convolutional message passing, the features of the left shoulder and left wrist will propagate to the left elbow vertex, completing the feature representation of the left elbow vertex and thus containing more effective information. Multi-layer graph convolution can increase the propagation range of features, enabling occluded joints to obtain information from a wider range of related joints. The number of graph convolutional message passing layers can be preset according to actual needs, such as 2, 3, or 4 layers. Each graph convolutional layer contains learnable feature transformation parameters. After multiple layers of graph convolutional message passing, each vertex obtains an updated feature vector, and the set of updated features for all vertices is the completed joint feature map. The completed joint feature map contains the completed features of the occluded joints and is more robust than the initial features.

[0040] In some preferred embodiments, determining edge weights on the human skeleton diagram based on the joint type and relative position between adjacent joints includes: obtaining two joints corresponding to each edge in the human skeleton diagram as adjacent joint pairs; obtaining the joint type identifier of each joint in all the relevant nodes and mapping the joint type identifier to a joint type embedding vector; calculating the relative position of the adjacent joint pair based on the initial positions of the two joints in the adjacent joint pair; encoding the relative position to obtain the relative position encoding of the adjacent joint pair; concatenating the joint type embedding vector with the relative position encoding to obtain the edge features of the adjacent joint pair; performing a nonlinear transformation on the edge features to obtain the initial edge weights of the adjacent joint pair; calculating the bone length corresponding to the adjacent joint pair based on the initial positions of the two joints in the adjacent joint pair; calculating the deviation between the bone length and the statistical prior of the proportion of lengths of various parts of the human body, and determining the prior consistency coefficient based on the deviation; and multiplying the initial edge weights by the prior consistency coefficient to obtain the edge weights of the adjacent joint pair.

[0041] In specific implementation, in the complete human pose estimation process, the system receives an input image, performs convolutional feature extraction on the input image to obtain image features, and performs initial pose estimation on the image features to obtain the initial positions of all joints. Then, it constructs a human skeleton map and obtains the initial features of each vertex. On the human skeleton map, edge weights are used to control the strength of feature aggregation between adjacent joints. To this end, this embodiment obtains anatomically meaningful edge weights through joint type embedding, relative position encoding, nonlinear transformation of edge features, and prior consistency constraints on bone length, realizing joint-aware message transmission. The specific implementation method is as follows: First, the system receives the input image, performs convolutional feature extraction on the input image to obtain image features, and then performs initial pose estimation on the image features to obtain the initial positions of all joints. Further, using the initial positions of all joints as vertices and the skeletal connections between joints in the human skeleton as edges, a human skeleton graph is constructed. In the human skeleton graph, vertices correspond to all joints, and edges correspond to skeletal connections; the set of edges is predefined.

[0042] Furthermore, the system obtains the two joints corresponding to each edge in the human skeleton diagram as adjacent joint pairs. The edges of the human skeleton diagram are predefined by the connection relationships of the human bones, and each edge connects two adjacent joints, such as the edge between the left shoulder and the left elbow, and the edge between the left elbow and the left wrist. The adjacent joint pair is the two joints connected by each edge, which is the basic unit for edge weight calculation.

[0043] Furthermore, the system obtains the joint type identifier for each joint in all relevant nodes and maps the joint type identifier to a joint type embedding vector. The joint type identifier is the semantic category number of each joint, such as left shoulder, left elbow, left wrist, right knee, etc. The joint type identifier can be represented by one-hot encoding or by indexing. The joint type embedding vector maps the joint type identifier to a low-dimensional continuous vector through an embedding layer, which is a learnable lookup table. The embedding vector can capture the semantic similarity and correlation between different joint types. For example, anatomically related joints such as the left elbow, left shoulder, and left wrist may be relatively close in the embedding space, while the left elbow and right knee may be relatively far apart. The embedding layer learns these semantic relationships during training.

[0044] Furthermore, the system calculates the relative position of adjacent joint pairs based on their initial positions. The relative position can be a two-dimensional offset vector between the two joints, or it can be the Euclidean distance and direction between them. The system subtracts the initial position coordinates of the two joints to obtain the relative position vector, which reflects the spatial relationship between adjacent joints. The relative position vector can be further normalized to eliminate the influence of human scale variations. For example, it can be divided by the length of the human torso or the height of the human bounding box in the image.

[0045] Furthermore, the system encodes relative positions to obtain relative position codes for adjacent joint pairs. Relative position encoding maps continuous position vectors into high-dimensional feature representations, enhancing the network's ability to express spatial relationships. Relative position encoding can be implemented using fully connected layers or position encoding functions, such as sinusoidal position encoding. The encoded relative position codes have the same or compatible dimensionality as the joint type embedding vectors.

[0046] Furthermore, the system concatenates the joint type embedding vector with the relative position encoding to obtain the edge features of adjacent joint pairs. The concatenation operation joins the two vectors along the feature dimension to form a new, longer vector. The edge features simultaneously contain type information and spatial relationship information of adjacent joint pairs, forming the basis for edge weight calculation.

[0047] Furthermore, the system performs a nonlinear transformation on the edge features to obtain the initial edge weights for adjacent keypoint pairs. This nonlinear transformation can be achieved using one or more fully connected layers with a nonlinear activation function, such as a modified linear unit activation function. These one or more fully connected layers constitute the edge weight determination network. The nonlinear transformation gives the calculation of edge weights a nonlinear expressive capability, enabling the learning of complex edge weight patterns. The initial edge weight is a scalar value representing the strength of feature aggregation between adjacent keypoint pairs.

[0048] Furthermore, the system calculates the bone lengths corresponding to adjacent joint pairs based on the initial positions of the two joints within each pair. Bone lengths can be obtained by calculating the Euclidean distance between the initial positions of the two joints. For example, the bone length between the left shoulder and left elbow is an estimated length of the upper arm, and the bone length between the left elbow and left wrist is an estimated length of the forearm. Bone length is an important geometric parameter of human posture, reflecting the scale of the human body in the current image.

[0049] Furthermore, the system calculates the deviation between bone length and the statistical prior of the proportions of various human body parts, and determines the prior consistency coefficient based on the deviation. The statistical prior of the proportions of various human body parts refers to the mean and standard deviation of the lengths of each bone obtained from a large amount of labeled human posture data, reflecting the reasonable range of human bone length. The system calculates the deviation between bone length and the statistical prior; the smaller the deviation, the more consistent the bone length is with the prior human structure; the larger the deviation, the more unreasonable the bone length is, possibly caused by initial position errors. The prior consistency coefficient is a value between 0 and 1, used to measure the degree of consistency between bone length and the statistical prior; the smaller the deviation, the closer the prior consistency coefficient is to 1.

[0050] Furthermore, the system multiplies the initial edge weights by the prior consistency coefficient to obtain the edge weights for adjacent joint pairs. In this way, the edge weights not only include joint type and relative position information but are also constrained by the prior statistical data on human bone length. When the initial position error causes a significant deviation of the bone length from the prior, the prior consistency coefficient is smaller, the edge weights are reduced, and thus the propagation of erroneous information is suppressed.

[0051] This embodiment constructs edge features through joint type embedding and relative position encoding, and obtains initial edge weights through nonlinear transformation, enabling the edge weights to perceive joint type and anatomical association. Based on this, a consistency constraint based on the statistical prior of the proportion of bone length to the length of various parts of the human body is introduced to correct the edge weights, further improving their reliability and interpretability. Furthermore, this edge weight determination method allows graph convolution message passing to occur between anatomically relevant joints, avoiding the propagation of erroneous information between irrelevant joints and significantly improving the rationality of feature completion. For example, the left elbow is more likely to obtain information from the left shoulder and left wrist than from the right knee, thus making the completion process more interpretable.

[0052] In some preferred embodiments, the step of calculating the deviation between the bone length and the statistical prior of the length ratio of various parts of the human body, and determining the prior consistency coefficient based on the deviation, includes: obtaining the bone type corresponding to the adjacent joint pair, and obtaining the mean length and standard deviation of the bone type on the labeled human pose dataset; subtracting the mean length from the bone length to obtain the length difference; dividing the length difference by the standard deviation to obtain the standardized deviation; and performing radial basis function mapping on the absolute value of the standardized deviation to obtain the prior consistency coefficient.

[0053] In specific implementation, during the edge weight determination process, the prior consistency coefficient is used to correct the edge weight. This embodiment further refines its calculation process by introducing the statistical mean and standard deviation corresponding to the bone type, calculating the standardized deviation of the bone length, and using radial basis function mapping to obtain the prior consistency coefficient. The specific implementation method is as follows: First, the system receives the input image, performs convolutional feature extraction to obtain image features, and then performs initial pose estimation on these features to obtain the initial positions of all joints. Next, using the initial positions of all joints as vertices and the bone connections between joints in the human skeleton as edges, a human skeleton graph is constructed. Then, the system obtains the two joints corresponding to each edge in the human skeleton graph as adjacent joint pairs. For each adjacent joint pair, the system calculates the bone length based on the initial positions of the two joints.

[0054] Furthermore, the system obtains the bone type corresponding to adjacent joint pairs. The bone type refers to the human skeletal segment corresponding to an adjacent joint pair, such as the upper arm, forearm, thigh, and lower leg. The bone type can be determined by the joint type of the adjacent joint pairs; for example, the left shoulder and left elbow correspond to the upper arm, the left elbow and left wrist correspond to the forearm, the left hip and left knee correspond to the thigh, and the left knee and left ankle correspond to the lower leg.

[0055] Furthermore, the system obtains the mean and standard deviation of bone length for each bone type on an annotated human pose dataset. The annotated human pose dataset contains a large number of human images and corresponding joint coordinate annotations, such as those constructed from motion capture data or manually annotated data. The system calculates the mean and standard deviation of length for each bone type from the annotated dataset. The mean length reflects the average length of that bone type in the population, while the standard deviation reflects the degree of individual variation in the length of that bone type. These statistical values ​​can be pre-calculated and stored for direct use during the training and inference phases. The statistical method could be to first calculate the corresponding bone length for each annotated sample, and then calculate the mean and standard deviation of the lengths for the same bone type.

[0056] It should be noted that, in one specific implementation, the labeled human pose dataset can use the COCO training set. This training set contains a large number of images labeled with the coordinates of human joints, typically exceeding 100,000 images. During statistical analysis, for each labeled human body, the bone length for each bone type is first calculated based on the joint coordinates, and then normalized to the torso height of the human body. Then, for each bone type, the mean and standard deviation of the normalized bone length are calculated, serving as the mean and standard deviation of the length for that bone type. Using torso height normalization eliminates overall scale differences in the human body within the images, ensuring that the statistically obtained mean and standard deviation of length reflect the relative proportions between different parts of the human body.

[0057] Furthermore, the system subtracts the mean length from the bone length to obtain the length difference. The bone length is the Euclidean distance between adjacent joint pairs in the current image, calculated from the initial position. The length difference indicates the degree of deviation between the current bone length and the average length of the population. A positive length value indicates that the current bone length is greater than the average, while a negative length value indicates that the current bone length is less than the average. The length difference directly reflects the magnitude of the deviation in bone length.

[0058] Furthermore, the system divides the length difference by the length standard deviation to obtain the standardized deviation. The standardized deviation eliminates the influence of differences in length standard deviations between different bone types, allowing for comparison of deviations across different bone types. The absolute value of the standardized deviation represents how many standard deviations the current bone length deviates from the mean. The larger the absolute value of the standardized deviation, the less the current bone length conforms to prior knowledge of human anatomy, and the more likely there is a significant error in the initial position.

[0059] Furthermore, the system performs radial basis function mapping on the absolute value of the standardized deviation to obtain the prior consistency coefficient. The radial basis function is a nonlinear function based on the distance between the input and the center point; a Gaussian function is commonly used. The system performs Gaussian radial basis function mapping on the absolute value of the standardized deviation, outputting a value between 0 and 1. When the absolute value of the standardized deviation is small, the radial basis function output is close to 1; when the absolute value of the standardized deviation is large, the radial basis function output is close to 0. The prior consistency coefficient, i.e., the output value of this radial basis function, reflects the degree of consistency between the current bone length and the statistical prior of the human body.

[0060] This embodiment transforms the deviation between bone length and statistical prior into a prior consistency coefficient between 0 and 1 through standardized deviation and radial basis function mapping, achieving continuous and smooth correction of edge weights. Furthermore, standardized deviation eliminates scale differences between different bone types, ensuring that the prior consistency coefficient maintains a consistent meaning across different bone types. Moreover, radial basis function mapping makes the prior consistency coefficient change slowly when the deviation is small and decreases rapidly when the deviation is large, effectively suppressing edge weights that deviate significantly from human priors and improving the stability and robustness of graph convolutional message passing. This calculation method does not require additional complex models, has low computational overhead, and is suitable for real-time operation.

[0061] In some preferred embodiments, the step of performing multi-layer graph convolution message passing based on the edge weights to complete the features of the occluded joints with the features of adjacent joints to obtain a completed joint feature map includes: S41, using the initial features of each vertex in the human skeleton graph as the input features of the first layer of graph convolution; S42, for each layer of graph convolution, weighting and aggregating the input features of adjacent vertices according to the edge weights to obtain aggregated features; S43, fusing the aggregated features with the input features of the current vertex itself to obtain fused features; S44, performing feature transformation and activation processing on the fused features to obtain updated vertex features; S45, using the updated vertex features as the input features of the next layer of graph convolution, and repeating S42 to S44 until the last layer of graph convolution is completed; S46, using the vertex features output by the last layer of graph convolution as the completed joint feature map.

[0062] In specific implementation, in the complete human pose estimation process, the system receives the input image, performs convolutional feature extraction on the input image to obtain image features, performs initial pose estimation on the image features to obtain the initial positions of all joints, constructs a human skeleton map and obtains the initial features of each vertex, and then determines the edge weights. This embodiment uses layer-by-layer feature aggregation, fusion, feature transformation, and activation processing to gradually complete the features of occluded joints with features from adjacent visible joints during multi-layer propagation. The specific implementation method is as follows: First, the system receives the input image, performs convolutional feature extraction to obtain image features, and then performs initial pose estimation on these features to obtain the initial positions of all joints. Next, using the initial positions of all joints as vertices and the skeletal connections between joints in the human skeleton as edges, a human skeleton graph is constructed. The features of each joint at its initial position are extracted from the image features and used as the initial features of each vertex in the human skeleton graph. Then, edge weights are determined based on the joint type and relative position between adjacent joints. After these steps are completed, the human skeleton graph, the initial features of each vertex, and the edge weights of each edge are obtained.

[0063] Furthermore, the system performs multi-layer graph convolutional message passing, specifically including the following steps: Step S41: Use the initial features of each vertex in the human skeleton image as the input features of the first layer graph convolution. The initial features of each vertex in the human skeleton image are the weighted initial features, with each vertex corresponding to a feature vector. These initial features will be used as the input features of the first layer graph convolution to calculate the updated vertex features after the first layer graph convolution.

[0064] Step S42: For each graph convolution layer, the input features of adjacent vertices are weighted and aggregated according to the edge weights to obtain aggregated features. For each vertex in the human skeleton graph, the system traverses all its adjacent vertices, multiplies the input features of each adjacent vertex by the edge weight of the corresponding edge, and then sums all the weighted features of adjacent vertices to obtain the aggregated features of that vertex. The aggregated features represent the set of information from neighboring keypoints. For occluded keypoints, their aggregated features mainly come from adjacent visible keypoints, because the input features of visible keypoints are not set to zero, while the input features of invisible keypoints are zero vectors, and their contribution after weighting is zero.

[0065] Step S43: Fuse the aggregated features with the current vertex's own input features to obtain fused features. The fusion method can be concatenation or element-wise addition. The current vertex's own input features contain its own local information, while the aggregated features contain information from neighboring vertex features. The fused features retain both its own and neighboring information, providing complete input for subsequent updates. For occluded vertex features, the vertex's own input features are zero vectors, and the fused features mainly contain information from the aggregated features.

[0066] Step S44: Perform feature transformation and activation processing on the fused features to obtain updated vertex features. Feature transformation can be implemented through a fully connected layer or a convolutional layer, mapping the fused features to a new feature space. This fully connected layer or convolutional layer forms part of a graph convolutional message passing network. Activation processing can employ rectified linear units, leaky rectified linear units, or other nonlinear activation functions to enhance the network's nonlinear expressive power. The updated vertex features contain the new features obtained after the keypoint has undergone a graph convolutional layer, where the features of occluded keypoints have already been preliminarily fused with information from neighboring keypoints.

[0067] Step S45: Use the updated vertex features as input features for the next layer of graph convolution, and repeat steps S42 to S44 until the last layer of graph convolution is completed. The number of layers in the multi-layer graph convolution can be preset according to actual needs, for example, 2 to 4 layers. As the number of layers increases, features can propagate further along the human skeleton map, and occluded joints can aggregate information from a wider range of related joints. At the same time, multi-layer propagation may also introduce excessive smoothing, so the choice of the number of layers needs to strike a balance between completion effect and feature discriminability.

[0068] Step S46: Use the vertex features output from the last layer of graph convolution as the completed joint feature map. The completed joint feature map contains the updated features of all joints, where the features of occluded joints have been completed by adjacent visible joints through multi-layer message passing. The completed joint feature map will be used for subsequent coordinate regression.

[0069] This embodiment utilizes multi-layer graph convolutional message passing to enable each vertex to aggregate neighbor information layer by layer. Specifically, occluded joints can obtain feature completion from anatomically relevant visible joints. Each graph convolutional layer includes aggregation, fusion, feature transformation, and activation processing, possessing non-linear expressive capabilities. Furthermore, multi-layer stacking allows features to propagate along the human skeleton graph, enhancing the depth and robustness of the completion. This message passing process fully utilizes the topological structure of the human skeleton graph and joint-aware edge weights, avoiding interference from irrelevant information and improving the quality of the completed features. Furthermore, the choice of the number of layers in the multi-layer graph convolution affects the completion effect; too few layers result in insufficient completion, while too many layers may lead to overly smoothed completion, requiring adjustment based on the specific task.

[0070] S5, perform coordinate regression on the completed joint feature map to obtain the position coordinates of all joint nodes.

[0071] In practice, coordinate regression is the process of mapping the completed joint feature map to the final position coordinates of all relevant nodes. The completed joint feature map contains the completed feature vector of each joint. The system performs regression calculations on these feature vectors, outputting the two-dimensional position coordinates of all relevant nodes in the image coordinate system, or the three-dimensional position coordinates in three-dimensional space. Furthermore, coordinate regression can be implemented using a coordinate regression network, which can contain fully connected layers. For example, the feature vectors of all joints can be flattened into a long vector, and then passed through one or more fully connected layers to output the coordinate values ​​of all joints. Other regression methods can also be used, such as regressing the coordinates of each joint separately, or regressing the offset based on the initial position. Further, this embodiment does not limit the specific implementation of coordinate regression, as long as it can output the position coordinates of all relevant nodes from the completed joint feature map. The learnable parameters of the coordinate regression network include the weights and biases of the fully connected layers. The position coordinates are the final output of human pose estimation, used by the embodied intelligent system to perform subsequent action understanding, intent reasoning, and human-computer interaction decisions.

[0072] This embodiment constructs a complete human pose estimation method in occluded scenarios through the aforementioned steps S1 to S5. The core of this method lies in modeling the human skeleton as a graph and performing feature completion at the structural level using a graph convolutional network, so that the features of occluded joints are completed by the features of adjacent joints. Furthermore, since the completion process utilizes the topological structure and joint type information of the human skeleton, the completion result better conforms to the constraints of human structure, avoiding unreasonable estimations caused by simply relying on image context completion. Furthermore, graph convolutional message passing is performed on the human skeleton graph, and the computational complexity is related to the number of joints, making it more efficient than completion based on global image features, and suitable for real-time operation on embody intelligent terminals.

[0073] In some preferred embodiments, the step of performing coordinate regression on the completed joint feature map to obtain the position coordinates of all joint nodes includes: flattening the features of each vertex in the completed joint feature map according to the order of the vertices in the human skeleton map to obtain a joint feature sequence; performing a first fully connected regression on the joint feature sequence to obtain the regression offset of all joint nodes; performing a second fully connected regression on the joint feature sequence to obtain the position confidence of all joint nodes; adding the regression offset to the initial position of all joint nodes to obtain the initial regression position coordinates of all joint nodes; and weighting the initial regression position coordinates of all joint nodes according to the position confidence to obtain the position coordinates of all joint nodes.

[0074] In specific implementation, in the complete human pose estimation process, the system receives the input image, performs convolutional feature extraction on the input image to obtain image features, performs initial pose estimation on the image features to obtain the initial positions of all joints, constructs a human skeleton map and obtains the initial features of each vertex, determines the edge weights, and performs multi-layer graph convolutional message passing to obtain the completed joint feature map. This embodiment flattens the completed joint feature map into a sequence, regresses the offset and confidence in parallel, and uses the confidence to weight the initial regressed position coordinates to improve the accuracy and reliability of the final joint position coordinates. The specific implementation method is as follows: First, the system receives the input image, performs convolutional feature extraction to obtain image features, and then performs initial pose estimation on these features to obtain the initial positions of all joints. Next, using the initial positions of all joints as vertices and the skeletal connections between joints in the human skeleton as edges, a human skeleton graph is constructed. The features of each joint at its initial position are extracted from the image features and used as the initial features of each vertex in the human skeleton graph. Then, edge weights are determined based on the joint type and relative position between adjacent joints, and multi-layer graph convolutional message passing is performed based on these edge weights to obtain the completed joint feature map.

[0075] Furthermore, the system flattens the features of each vertex in the completed joint feature map according to the order of the vertices in the human skeleton map, obtaining a joint feature sequence. The order of the vertices in the human skeleton map is a predefined fixed order, consistent with the order of all joints. The completed joint feature map contains the updated feature vector of each joint. The system concatenates or stacks the feature vectors of each vertex according to this fixed order to form the joint feature sequence. The joint feature sequence retains the feature information of all joints and can be used as input for coordinate regression.

[0076] Furthermore, the system performs a first fully connected regression on the joint feature sequence to obtain the regression offsets of all joints. The first fully connected regression involves transforming the joint feature sequence through one or more fully connected layers, outputting the regression offsets of all joints. These one or more fully connected layers constitute part of a coordinate regression network. The regression offset is the offset value relative to the initial position of all joints. Since the initial position already provides an approximate joint location, the regression offset is used to refine and correct the initial position. The regression offset can be represented as a two-dimensional offset vector of each joint in the image coordinate system, or a three-dimensional offset vector in three-dimensional space.

[0077] Furthermore, the system performs a second fully connected regression on the keypoint feature sequence to obtain the position confidence of all keypoints. The second fully connected regression is structurally similar to the first fully connected regression, but its output is the position confidence of all keypoints. The second fully connected regression is also implemented using fully connected layers in a coordinate regression network. Position confidence is a scalar value representing the reliability of the final position coordinates of the keypoint. Position confidence can be estimated based on the quality of the completed features. For keypoints with reliable completed features and sufficient neighbor information, the position confidence is high; for keypoints where the features still have significant uncertainty, the position confidence is low.

[0078] Furthermore, the system adds the regression offset to the initial positions of all joints to obtain the initial regression position coordinates of all joints. For each joint, the system adds the initial position coordinates of that joint to the corresponding regression offset element-wise to obtain the initial regression position coordinates of that joint. The initial regression position coordinates are a correction result of the initial position and are theoretically closer to the true position than the initial position.

[0079] Furthermore, the system weights the initial regressed position coordinates of all joints based on the position confidence score to obtain the position coordinates of all joints. The weighting method can be either multiplying the initial regressed position coordinates by the position confidence score or weighted fusion of multiple candidate coordinates. In a single regression scenario, the position confidence score can be used as the final position coordinate confidence output, or it can be used for subsequent filtering or smoothing. In scenarios requiring post-processing, joints with low position confidence scores can be marked as low-confidence joints for further processing by upper-layer applications. The final output of the position coordinates of all joints represents the result of human pose estimation.

[0080] This embodiment flattens the completed joint feature map into a sequence and utilizes fully connected regression to simultaneously output regression offsets and position confidence scores, achieving efficient mapping from completed features to final coordinates. The regression offsets refine the initial positions, while the position confidence scores assess the reliability of the refinement results; the combination of these two approaches achieves a balance between accuracy and reliability in the final position coordinates. Furthermore, compared to directly regressing absolute coordinates, the method based on the initial position plus offset reduces the regression range, improving training convergence speed and accuracy. The output of position confidence scores enables the system to identify low-confidence joint estimates, providing crucial uncertainty information for subsequent applications and contributing to improved decision-making security in embodied intelligence systems.

[0081] In some preferred embodiments, prior to receiving an input image, the method further includes a step of end-to-end training of network parameters, comprising: acquiring training samples, the training samples containing human images and labeled joint coordinates; performing random occlusion simulation on the human images in the training samples to obtain training images with artificially synthesized occlusion; using the training images with artificially synthesized occlusion as input images, performing steps S1-S5 to obtain the predicted position coordinates of all joints; calculating the mean square error between the predicted position coordinates of all joints and the labeled joint coordinates; and acquiring the annotations of the human images in the training samples. Visibility mask for joints; from the completed joint feature map obtained in steps S1-S5, obtain the features corresponding to each joint in all relevant nodes, perform visibility regression on the features corresponding to each joint in all relevant nodes, and obtain the visibility regression score of each joint in all relevant nodes; calculate the binary cross-entropy loss between the visibility regression score of each joint in all relevant nodes and the visibility mask of the labeled joints; perform a weighted summation of the mean square error and the binary cross-entropy loss to obtain the total loss; based on the total loss, update all network parameters involved in steps S1-S5 end-to-end.

[0082] In practice, the network parameters need to be trained and optimized before the model is put into use. This embodiment generates training data through random occlusion simulation and uses a multi-task loss function to optimize the network parameters end-to-end, enabling the trained model to have better generalization ability in real-world occlusion scenarios. The specific implementation method is as follows: First, the system acquires training samples, which include human images and labeled joint coordinates. Training samples can be obtained from publicly available human pose estimation datasets, or they can be collected by an embodied intelligent robot in a real-world scene and manually labeled. The human image is an image containing the complete human body, and the labeled joint coordinates are the actual position coordinates of each joint in the image, typically provided by manual annotation or a motion capture system. Training samples may also include occlusion annotations; however, in this embodiment, the occlusion state is generated through subsequent random occlusion simulation.

[0083] Furthermore, the system performs random occlusion simulation on the human images in the training samples to obtain training images with artificially synthesized occlusion. Random occlusion simulation refers to superimposing occlusion objects on the original human images to simulate the situation where human parts are occluded in a real scene. The occlusion objects can be randomly generated rectangular blocks, circular blocks, or object images sampled from the occlusion object dataset. The position, size, and number of occlusions can be randomly selected to increase the diversity of occlusion patterns. Random occlusion simulation can significantly increase the richness of training data, allowing the model to be exposed to various occlusion situations during the training phase, thus improving its robustness to occlusion. At the same time, the system can automatically determine which key points are occluded based on the position and coverage area of ​​the occlusion objects, thereby generating a visible key point mask. In the visible key point mask, the weight corresponding to visible key points is 1, and the weight corresponding to invisible key points is 0.

[0084] It should be noted that, in some preferred embodiments, when overlaying occlusions onto the original human image, the following random occlusion simulation method can be adopted: 1 to 5 rectangular occlusion blocks are randomly generated on each training image. The side length of each occlusion block is uniformly sampled within the range of 10% to 50% of the shorter side of the image, and the center position of the occlusion block is randomly sampled within the image region. The color of the occlusion block can be randomly selected from a preset color set, or it can be sampled from the object image in the occlusion dataset. The position, size, and number of occlusions can be randomly selected to increase the diversity of occlusion patterns. Random occlusion simulation can significantly increase the richness of training data, allowing the model to encounter various occlusion situations during the training phase, thus improving its robustness to occlusion. Simultaneously, the system can automatically determine which key points are occluded based on the position and coverage area of ​​the occlusion object, thereby generating a visible mask for the labeled key points.

[0085] Furthermore, the system uses training images with artificially synthesized occlusions as input images and executes steps S1 to S5 to obtain the predicted position coordinates of all key points. Steps S1 to S5 include convolutional feature extraction, initial pose estimation, human skeleton graph construction and initial feature acquisition, edge weight determination and graph convolutional message passing, and coordinate regression. During the forward propagation process, each step is executed sequentially, and data from the input image is processed through each network module to finally generate the predicted position coordinates.

[0086] Furthermore, the system calculates the mean squared error (MSE) between the predicted and labeled coordinates of all key points. MSE is a commonly used loss function in regression tasks, calculated by averaging the squared differences between the predicted and labeled coordinates of each key point. MSE reflects the overall deviation between the predicted and actual positions; a smaller value indicates a more accurate prediction.

[0087] Furthermore, the system obtains the visibility mask of labeled keypoints in the human images from the training samples. This mask is consistent with the visibility mask of labeled keypoints generated during the random occlusion simulation and is used to supervise the visibility regression task.

[0088] Furthermore, the system extracts the features corresponding to each joint in all relevant nodes from the completed joint feature map obtained from steps S1 to S5, and performs visibility regression on the features corresponding to each joint in all relevant nodes to obtain the visibility regression score for each joint in all relevant nodes. The visibility regression score is a scalar value representing the probability that the joint is visible. Visibility regression can be implemented by applying a lightweight fully connected layer to the completed joint feature map, which constitutes the visibility regression network. The visibility regression network takes the features corresponding to each joint in the completed joint feature map as input and outputs the visibility probability of each joint.

[0089] Furthermore, the system calculates the binary cross-entropy loss between the visibility regression score and the labeled visibility mask for each key in all relevant nodes. Binary cross-entropy loss is a commonly used loss function in binary classification tasks, used to measure the difference between the predicted probability and the true binary label. For each key, the system calculates the binary cross-entropy loss between the visibility regression score and the labeled visibility mask, and then averages it across all relevant nodes to obtain the total binary cross-entropy loss. This loss supervises the network to correctly predict the occlusion state of each key, enabling the network to perceive occlusion information during feature completion.

[0090] Furthermore, the system performs a weighted summation of the mean squared error and the binary cross-entropy loss to obtain the total loss. In the weighted summation, the mean squared error and the binary cross-entropy loss are multiplied by their respective loss weights and then added together. The loss weights are used to balance the importance of the coordinate regression task and the visibility regression task and can be adjusted according to the actual training effect. The total loss integrates the two optimization objectives of coordinate accuracy and occlusion state prediction, enabling the network to effectively model the visibility of key points while improving localization accuracy.

[0091] Furthermore, the system updates all network parameters involved in steps S1 to S5 end-to-end based on the total loss. During the execution of steps S1 to S5, the convolutional feature extraction network extracts convolutional features from the input image to generate image features; the initial pose estimation network estimates the initial pose of the image features to generate the initial positions of all joints; the edge weight determination network determines edge weights based on the joint type and relative position between adjacent joints; the graph convolutional message passing network performs multi-layer graph convolutional message passing based on the edge weights to generate a completed joint feature map; and the coordinate regression network performs coordinate regression on the completed joint feature map to generate the position coordinates of all joints. Additionally, the visibility regression network performs visibility regression on the features corresponding to each joint in the completed joint feature map to generate a visibility regression score. These networks all contain learnable parameters, such as the kernel weights and biases of convolutional layers, the weights and biases of fully connected layers, and the embedding vectors of embedding layers. End-to-end updating refers to using the backpropagation algorithm to differentiate the total loss with respect to the learnable parameters of each of the above networks and updating all learnable parameters based on the gradient. Furthermore, end-to-end training enables all network modules to be jointly optimized under a unified loss function, avoiding the error accumulation caused by staged training. After training is completed, the network parameters are saved for deployment and inference.

[0092] This embodiment enables the model to encounter diverse occlusion patterns by performing random occlusion simulations on the training data. Simultaneously, a weighted summation of the mean squared error of coordinate regression and the binary cross-entropy loss of visibility regression is formed, creating a multi-task loss function. This allows the network to accurately predict keypoint visibility while optimizing coordinate accuracy. Furthermore, end-to-end training jointly optimizes all network parameters under a unified objective, improving the model's robustness to occlusion scenarios and pose estimation accuracy. Moreover, this training method is independent of specific occlusion patterns, enabling the model to generalize better in real-world occlusion scenarios. Through multi-task learning, the network focuses not only on coordinate regression accuracy but also on visibility prediction during feature completion, enabling completed features to more effectively distinguish between visible and occluded keypoints, further improving the overall performance of pose estimation. Finally, the trained network can achieve high pose estimation accuracy in complex scenarios such as dense crowds and object occlusion.

[0093] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0094] Please see Figure 2 , Figure 2This is a schematic block diagram of a computer device provided in an embodiment of the present invention. The computer device 500 can be a terminal or a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0095] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0096] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it enables the processor 502 to perform a human pose estimation method in an embodied intelligent occlusion scenario.

[0097] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0098] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can perform a human pose estimation method in an embodied intelligent occlusion scenario.

[0099] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device 500 to which the present invention is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0100] The processor 502 is used to run a computer program 5032 stored in a memory to implement the steps of a human pose estimation method in an embodied intelligent occlusion scenario provided in any of the above method embodiments.

[0101] It should be understood that, in this embodiment of the invention, the processor 502 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0102] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0103] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the steps of the human pose estimation method in an embodied intelligent occlusion scenario provided in any of the above method embodiments.

[0104] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0105] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0106] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0107] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0108] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0109] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0110] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, this invention is also intended to include these modifications and variations as long as they fall within the scope of the claims of this invention and their equivalents.

[0111] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for estimating human pose in an embodied intelligent occlusion scenario, characterized in that, include: In response to receiving an input image, convolutional feature extraction is performed on the input image to obtain image features; Initial pose estimation is performed on the image features to obtain the initial positions of all joints; Using the initial positions of all the joints as vertices and the bone connections between the joints in the human skeleton as edges, a human skeleton graph is constructed, and the features of each joint at the initial position are obtained from the image features as the initial features of each vertex in the human skeleton graph. On the human skeleton map, edge weights are determined based on the joint type and relative position between adjacent joints, and multi-layer graph convolution message passing is performed based on the edge weights so that the features of the occluded joints are completed by the features of the adjacent joints, resulting in a completed joint feature map. Coordinate regression is performed on the completed joint feature map to obtain the position coordinates of all joint nodes.

2. The human pose estimation method in an embodied intelligent occlusion scenario according to claim 1, characterized in that, The initial pose estimation of the image features to obtain the initial positions of all key points includes: Perform heatmap regression on the image features to obtain a heatmap of each joint point in all relevant nodes; The heatmap of each joint is normalized to obtain the probability distribution map of each joint. The initial position coordinates of each joint point are obtained by multiplying each probability distribution map element by element with a preset coordinate grid and summing the results. The initial position coordinates of each joint are used as the initial positions of all joints.

3. The human pose estimation method in an embodied intelligent occlusion scenario according to claim 1, characterized in that, The step of obtaining the features of each joint at the initial position from the image features as the initial features of each vertex in the human skeleton image includes: The features of each joint in all relevant nodes at the initial position are obtained from the image features; Visibility prediction is performed on the features of each joint in all relevant nodes at the initial position to obtain the visibility prediction score of each joint in all relevant nodes. Compare the visibility prediction score of each key point in all relevant nodes with a preset threshold. If the visibility prediction score is greater than the preset threshold, then the key point is determined to be visible; If the visibility prediction score is less than or equal to the preset threshold, then the joint is determined to be invisible. A visibility mask is generated based on the determination result, where the weight of a key point that is determined to be visible is 1, and the weight of a key point that is determined to be invisible is 0. Multiply the features of each key point in all the relevant nodes at the initial position with the corresponding element of the visibility mask to obtain the weighted initial features; The weighted initial features are used as the initial features of each vertex in the human skeleton diagram.

4. The human pose estimation method in an embodied intelligent occlusion scenario according to claim 1, characterized in that, The process of determining edge weights on the human skeleton diagram based on the joint type and relative position between adjacent joints includes: Obtain two joint points corresponding to each edge in the human skeleton diagram, as adjacent joint point pairs; Obtain the joint type identifier of each joint point in all relevant nodes, and map the joint type identifier to a joint type embedding vector; Calculate the relative position of the adjacent joint pair based on the initial positions of the two joints in the adjacent joint pair; The relative positions are encoded to obtain the relative position codes of the adjacent joint point pairs; The joint type embedding vector is concatenated with the relative position encoding to obtain the edge features of the adjacent joint point pairs; The edge features are subjected to a nonlinear transformation to obtain the initial edge weights of the adjacent keypoint pairs; Calculate the bone length corresponding to the adjacent joint pair based on the initial positions of the two joints in the adjacent joint pair; Calculate the deviation between the bone length and the statistical prior of the proportion of lengths of various parts of the human body, and determine the prior consistency coefficient based on the deviation; The edge weights of the adjacent keypoint pairs are obtained by multiplying the initial edge weights by the prior consistency coefficients.

5. The human pose estimation method in an embodied intelligent occlusion scenario according to claim 4, characterized in that, The calculation of the deviation between the bone length and the statistical prior of the proportion of lengths of various parts of the human body, and the determination of the prior consistency coefficient based on the deviation, includes: Obtain the bone type corresponding to the adjacent joint pair, and obtain the mean length and standard deviation of the length of the bone type on the labeled human pose dataset; Subtract the average length from the bone length to obtain the length difference; Divide the length difference by the length standard deviation to obtain the standardized deviation; The prior consistency coefficient is obtained by taking the absolute value of the standardized deviation and then performing radial basis function mapping.

6. The human pose estimation method in an embodied intelligent occlusion scenario according to claim 1, characterized in that, The step of performing multi-layer graph convolutional message passing based on the edge weights, so that the features of the occluded joints are completed by the features of the adjacent joints, to obtain the completed joint feature map, includes: S41, the initial features of each vertex in the human skeleton diagram are used as the input features of the first layer graph convolution; S42, for each graph convolution layer, the input features of adjacent vertices are weighted and aggregated according to the edge weights to obtain aggregated features; S43, the aggregated features are fused with the input features of the current vertex itself to obtain fused features; S44, Perform feature transformation and activation processing on the fused features to obtain updated vertex features; S45, the updated vertex features are used as the input features for the next layer of graph convolution, and S42 to S44 are repeated until the last layer of graph convolution is completed; S46, the vertex features output by the convolution of the last layer are used as the completed joint feature map.

7. The human pose estimation method in an embodied intelligent occlusion scenario according to claim 1, characterized in that, The step of performing coordinate regression on the completed joint feature map to obtain the position coordinates of all joint nodes includes: The features of each vertex in the completed joint feature map are flattened according to the order of the vertices in the human skeleton map to obtain the joint feature sequence. Perform a first fully connected regression on the joint feature sequence to obtain the regression offset of all joints; A second fully connected regression is performed on the feature sequence of the key points to obtain the position confidence of all key nodes; Add the regression offset to the initial position of all the joints to obtain the initial regression position coordinates of all the joints; Based on the position confidence level, the initial regression position coordinates of all the key points are weighted to obtain the position coordinates of all the key points.

8. The human pose estimation method in an embodied intelligent occlusion scenario according to claim 1, characterized in that, Prior to receiving the input image, the method further includes a step of end-to-end training of network parameters, including: Obtain training samples, which include human images and labeled joint coordinates; Random occlusion simulation is performed on the human images in the training samples to obtain training images with artificially synthesized occlusion; Using the training image with artificially synthesized occlusion as the input image, the method described in claim 1 is executed to obtain the predicted position coordinates of all the joints; Calculate the mean square error between the predicted position coordinates of all the joint points and the labeled joint point coordinates; Obtain the visibility mask of the labeled key points in the human body image of the training sample; From the completed joint feature map obtained by performing the method as described in claim 1, the features corresponding to each joint in all the relevant nodes are obtained, and visibility regression is performed on the features corresponding to each joint in all the relevant nodes to obtain the visibility regression score of each joint in all the relevant nodes. Calculate the binary cross-entropy loss between the visibility regression score of each joint in all relevant nodes and the visibility mask of the labeled joint; The total loss is obtained by weighted summing of the mean squared error and the binary cross-entropy loss. Based on the total loss, end-to-end updates are performed on all network parameters involved in the method described in claim 1.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-8.

Images