Pose estimation method, apparatus, device, and medium

Abstract

The present disclosure provides a pose estimation method, which relates to the field of artificial intelligence. The method comprises: collecting N body images of a user at the same time from N perspectives, the N perspectives comprising an upper perspective, the collection position of the upper perspective being higher than the height of the user, and N being greater than or equal to 3; inputting the N body images into N two-dimensional pose estimation models respectively to obtain N two-dimensional pose data, wherein the N two-dimensional pose estimation models have a one-to-one mapping relationship with the N perspectives; and inputting the N two-dimensional pose data into a three-dimensional pose estimation model to obtain three-dimensional pose data of the user. The present disclosure also provides a pose estimation device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and more specifically, to a pose estimation method, apparatus, device, medium, and program product. Background Technology

[0002] Pose estimation is a computer vision technique that detects the pose of a person in an image. It can determine the location of a certain body part of a person in an image, which is the problem of locating human joints in images and videos. It can also be understood as searching for a specific pose in the space of all joint poses.

[0003] In related technologies, single-view cameras are generally used to capture images of the user's body. For example, a camera is installed directly facing the user, and its position is lower than the user's height. However, the information obtained from a single viewpoint is limited, resulting in defects in the modeling of the person and inaccurate pose estimation results. Summary of the Invention

[0004] In view of the above problems, this disclosure provides an attitude estimation method, apparatus, device, medium and program product.

[0005] One aspect of this disclosure provides a pose estimation method, comprising: acquiring N body images of a user simultaneously from N viewpoints, wherein the N viewpoints include an upper viewpoint, the acquisition position of the upper viewpoint being higher than the user's height, and N being greater than or equal to 3; inputting the N body images into N two-dimensional pose estimation models respectively to obtain N two-dimensional pose data, wherein the N two-dimensional pose estimation models have a one-to-one mapping relationship with the N viewpoints; and inputting the N two-dimensional pose data into a three-dimensional pose estimation model to obtain the user's three-dimensional pose data.

[0006] According to an embodiment of this disclosure, the three-dimensional pose estimation model includes N encoders. The step of inputting the N two-dimensional pose data into the three-dimensional pose estimation model to obtain the user's three-dimensional pose data includes: inputting the N two-dimensional pose data one-to-one into the N encoders to obtain N geometric representation data; determining the geometric representation data under a first viewpoint among the N geometric representation data, wherein the first viewpoint is a specific viewpoint among the N viewpoints; generating sub-geometric representation data under each second viewpoint based on the geometric representation data under the first viewpoint, wherein the second viewpoint is any viewpoint among the N viewpoints other than the first viewpoint; and obtaining the user's three-dimensional pose data based on the sub-geometric representation data under each second viewpoint.

[0007] According to an embodiment of this disclosure, generating sub-geometric representation data for each second perspective based on the geometric representation data from the first perspective includes: fusing the geometric representation data from the first perspective and the geometric representation data from each second perspective to generate the sub-geometric representation data from each second perspective.

[0008] According to an embodiment of this disclosure, the three-dimensional pose estimation model includes N-1 decoders. Obtaining the user's three-dimensional pose data based on the sub-geometric representation data under each second viewpoint includes: inputting the sub-geometric representation data under each second viewpoint into the N-1 decoders one-to-one to obtain N-1 two-dimensional skeleton data; and inputting the N-1 two-dimensional skeleton data into a linear regression layer to obtain the user's three-dimensional pose data.

[0009] According to an embodiment of this disclosure, the three-dimensional pose estimation model further includes an Nth decoder, and the method further includes: inputting geometric representation data from the first viewpoint into the Nth decoder to obtain Nth two-dimensional skeleton data; wherein, obtaining the user's three-dimensional pose data involves inputting the N-1 two-dimensional skeleton data and the Nth two-dimensional skeleton data into the linear regression layer to obtain the user's three-dimensional pose data.

[0010] According to an embodiment of this disclosure, before determining the geometric representation data under the first viewpoint among the N geometric representation data, determining the first viewpoint among the N geometric representation data includes: determining the occlusion area of ​​each body image among the N body images; determining the body image with the smallest occlusion area among the N body images, and the viewpoint corresponding to this image is taken as the first viewpoint.

[0011] According to embodiments of this disclosure, the three-dimensional pose estimation model is configured to be pre-trained via the following operations: inputting N two-dimensional pose sample data into the N encoders to be trained to obtain N geometric representation data samples, wherein the N two-dimensional pose data samples are obtained from N sample images, and the N training images are from N viewpoints of the same user at the same time; determining the geometric representation data samples under the first viewpoint among the N geometric representation data samples; generating sub-geometric representation data samples under each second viewpoint based on the geometric representation data samples under the first viewpoint; calculating the loss function value between the sub-geometric representation data samples and the geometric representation data samples under each second viewpoint to obtain N-1 loss function values; obtaining a comprehensive loss function value based on the N-1 loss function values ​​and their respective weights; and updating the N encoders to be trained based on the comprehensive loss function value.

[0012] Another aspect of this disclosure provides a posture estimation device, comprising: an image acquisition module for acquiring N body images of a user simultaneously from N viewpoints, wherein the N viewpoints include an upper viewpoint, the acquisition position of the upper viewpoint being higher than the user's height, and N being greater than or equal to 3; a two-dimensional prediction module for inputting the N body images into N two-dimensional posture estimation models respectively to obtain N two-dimensional posture data, wherein the N two-dimensional posture estimation models have a one-to-one mapping relationship with the N viewpoints; and a three-dimensional prediction module for inputting the N two-dimensional posture data into a three-dimensional posture estimation model to obtain the user's three-dimensional posture data.

[0013] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0014] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0015] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the method described above.

[0016] The above one or more embodiments have the following beneficial effects: They leverage the advantages of multiple viewpoints to estimate the target's pose, especially by introducing an upper viewpoint. Since the acquisition position in the upper viewpoint is higher than the user's height, the image acquired in the upper viewpoint can provide more information and avoid potential occlusion. Then, the images acquired from each viewpoint are used for two-dimensional prediction to obtain N two-dimensional pose data, which are then used as input to the three-dimensional pose estimation model. This fully utilizes the advantages of multiple viewpoints, linking and correlating user pose information between different viewpoints to obtain accurate three-dimensional pose data. Attached Figure Description

[0017] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The diagram illustrates an application scenario of the attitude estimation method according to embodiments of the present disclosure.

[0019] Figure 2 A flowchart illustrating an attitude estimation method according to an embodiment of the present disclosure is shown schematically;

[0020] Figure 3This schematically illustrates an architecture diagram for obtaining sub-geometric representation data according to embodiments of the present disclosure;

[0021] Figure 4 A flowchart illustrating the process of obtaining three-dimensional pose data according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 5 This schematically illustrates an architecture diagram for obtaining three-dimensional pose data according to an embodiment of the present disclosure;

[0023] Figure 6 A flowchart illustrating the process of obtaining three-dimensional pose data according to another embodiment of this disclosure is shown schematically;

[0024] Figure 7 This schematically illustrates an architecture diagram for obtaining three-dimensional pose data according to another embodiment of the present disclosure;

[0025] Figure 8 A flowchart illustrating the process of obtaining three-dimensional pose data according to another embodiment of this disclosure is shown schematically;

[0026] Figure 9 A flowchart illustrating the determination of a first perspective according to an embodiment of the present disclosure is shown schematically;

[0027] Figure 10 A flowchart illustrating the training of a 3D pose estimation model according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 11 A schematic block diagram of an attitude estimation apparatus according to an embodiment of the present disclosure is shown; and

[0029] Figure 12 A block diagram of an electronic device suitable for implementing an attitude estimation method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0030] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0031] To facilitate understanding of the technical solutions of the embodiments of this disclosure, some technical terms involved in this disclosure will be introduced first.

[0032] Pose estimation: Its task is to reconstruct human joints and limbs. The main challenge lies in reducing the complexity of the model analysis algorithm and being able to adapt to various changing situations and environments (lighting, occlusion, etc.).

[0033] Pose tracking: Extending pose estimation to video results in the task of human pose tracking. This mainly involves tracking the human body and key points of each person in a video scene.

[0034] 3D human pose estimation: Extending human pose to 3D, for example, inputting an RGB image and outputting 3D human key points, i.e., three-dimensional pose data.

[0035] BlazePose: A lightweight convolutional neural network architecture for human pose estimation. In BlazePose, pose prediction utilizes a validated two-step detector-tracker ML pipeline. A region of interest (ROI) is located within a frame, and the tracker then predicts all 33 pose keypoints based on this ROI. The detector operates only on the first frame; subsequent frames derive the ROI from the pose keypoints of the previous frame. Inputting an image into the BlazePose network structure yields the location information of each keypoint in the image coordinate system. Each keypoint location can be represented as (x, y, z, v). Here, x and y represent the normalized coordinates of the keypoint in the image coordinate system. z represents the keypoint depth with the midpoint of the hip as the origin; a smaller value indicates the keypoint is closer to the camera. v represents the probability that the keypoint is visible (not occluded).

[0036] Two-dimensional pose data: including the coordinates of key points of the human body in two dimensions from the corresponding viewpoint.

[0037] 3D pose data: includes the coordinates of 3D human body key points, each key point is represented by 3D coordinates, which can provide spatial and depth information.

[0038] Geometric representation data: The input two-dimensional attitude data is encoded using an encoder to generate a vector or matrix representation, which is used for subsequent attitude estimation.

[0039] Sub-geometric representation data: Geometric feature representations that are directly inferred from the geometric representation data of the first viewpoint, or generated by fusing information from multiple viewpoints, and used for subsequent attitude estimation.

[0040] Two-dimensional skeleton data: Numerical data used to represent human posture, consisting of a set of continuous skeletal structures. The skeletal structure comprises key points and the lines connecting them. While the representational form of two-dimensional skeleton data and two-dimensional pose data can be the same, the processes for obtaining them are different.

[0041] Taking a fitness assistant scenario based on 3D pose estimation as an example, such as a smart fitness mirror, the product works by installing a depth camera module, an AI human motion sensor, and an automatic brightness sensor inside the mirror. Through camera and sensor tracking of human movement, it achieves visual recognition of human joints, enabling fitness guidance services. The core technologies are pose estimation and pose tracking, with the core function of fitness correction heavily reliant on the accuracy of pose estimation and tracking. From a user experience perspective, products on the market often identify incorrect fitness postures as correct, or vice versa. Even with the addition of a 3D camera, the limitations of modeling a person from a single perspective cannot be completely avoided.

[0042] In some embodiments of this disclosure, a pose estimation method is provided that leverages the advantages of multiple viewpoints to estimate the pose of a target. In particular, an upper viewpoint is introduced. Since the acquisition position of the upper viewpoint is higher than the user's height, the image acquired from the upper viewpoint can provide more information and avoid potential occlusion. Then, the images acquired from each viewpoint are used for two-dimensional prediction to obtain N two-dimensional pose data, which are then used as input to a three-dimensional pose estimation model. This fully utilizes the advantages of multiple viewpoints, linking and correlating user pose information between different viewpoints to obtain accurate three-dimensional pose data.

[0043] In particular, this disclosure is not intended to be limited to fitness assistant scenarios; it can also be used for monitoring, where extracted postures are used to automatically monitor and understand human behavior. Another type of application is control, which refers to providing control functions using acquired motion. This can be used for game interfaces, virtual reality, or animation production, etc. Furthermore, human posture estimation can also be used for motion analysis, such as diagnosing orthopedic patients or conducting sports analysis to improve athletes' training levels.

[0044] In the technical solutions disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information. The collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with relevant laws and regulations, necessary confidentiality measures are taken, and they do not violate public order and good morals.

[0045] Figure 1 The diagram illustrates an application scenario of the attitude estimation method according to embodiments of the present disclosure.

[0046] like Figure 1As shown, application scenario 100 according to this embodiment may include a first camera 111, a second camera 112, and a third camera 113, a network 140, and a server 120. The first user 131 and the second user 132 are users whose images are to be captured. The network 140 serves as a medium for providing a communication link between the first camera 111, the second camera 112, the third camera 113, and the server 120. The network 140 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. The first camera 111, the second camera 112, and the third camera 113 may be respectively located at different locations within the venue.

[0047] The first camera 111, the second camera 112, and the third camera 113 can each capture images or videos of the same user from different perspectives. Then, the first camera 111, the second camera 112, and the third camera 113 can interact with the server 120 via network 140 to send videos or images, etc. The first camera 111, the second camera 112, and the third camera 113 can be surveillance cameras or various electronic devices with camera functions. These various electronic devices include, but are not limited to, smartphones, tablets, laptops, and desktop computers.

[0048] Server 120 can be a server that provides various services, such as processing videos or images and performing operations such as pose tracking, object detection, and pose estimation. In some embodiments, an electronic device can interact with server 120 to send user requests. The background management server can analyze and process the received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the electronic device.

[0049] like Figure 1 As shown, server 120 can process body images from different perspectives collected at the same time based on the received images or videos to obtain pose estimation results.

[0050] It should be noted that the attitude estimation method provided in this embodiment can generally be executed by server 120. Correspondingly, the attitude estimation device provided in this embodiment can generally be located in server 120. The attitude estimation method provided in this embodiment can also be executed by a server or server cluster that is different from server 120 but capable of communicating with the aforementioned camera and / or server 120. Correspondingly, the attitude estimation device provided in this embodiment can also be located in a server or server cluster that is different from the aforementioned camera and / or server 120 but capable of communicating with server 120.

[0051] It should be understood that Figure 1The number of cameras, users, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of cameras, users, networks, and servers can be included. Public places are not limited to... Figure 1 The content shown can be an outdoor or indoor location.

[0052] The following will be based on Figure 1 The described scene, through Figures 2 to 10 The attitude estimation method of the present disclosure embodiments will be described in detail.

[0053] Figure 2 A flowchart illustrating an attitude estimation method according to an embodiment of the present disclosure is shown schematically. Figure 2 As shown, the attitude estimation method in this embodiment includes:

[0054] When operating S210, N body images of the user are acquired simultaneously from N perspectives. The N perspectives include the top perspective, and the acquisition position of the top perspective is higher than the user's height. N is greater than or equal to 3.

[0055] For example, multiple cameras can be set up, with each camera capturing images of the user's body in the target area at any given time. The target area refers to a specific area in a public place, which can be determined based on the area that the cameras can capture. For example, it could be the equipment area of ​​a gym, the green screen area of ​​a filming location, or the service lobby area of ​​a bank branch.

[0056] The acquisition position from an upward perspective can be located at at least one of the following: directly above the target user's head, diagonally above in front of the user, or diagonally above behind the user. For example, when the user is standing upright, "directly above the head" is the position of a straight line pointing upwards from the top of the user's head. "Diagonally above in front of the user" is the position of a straight line pointing forward from the top of the user's head, until the line is parallel to the ground. "Diagonally above behind the user" is the position of a straight line pointing backwards from the top of the user's head, until the line is parallel to the ground.

[0057] N body images corresponding to N viewpoints represent single-view images taken at the same time. "Same time" means exactly the same moment, or moments with a time difference less than a tolerance threshold. For example, at a frame rate of 60 frames per second, two moments with a time difference less than 1 / 60 or 1 / 120 are considered to be the same moment.

[0058] When acquiring N body images from a video, multiple sets of single-view body images are arranged in chronological order, with each set containing N body images of the same user. When multiple people are present, an object detection model can be used to identify the target user from the video frames.

[0059] In operation S220, N body images are input into N two-dimensional pose estimation models to obtain N two-dimensional pose data. The N two-dimensional pose estimation models have a one-to-one mapping relationship with the N viewpoints.

[0060] Before being input into N two-dimensional pose estimation models, the N body images are located in the same coordinate system, and the coordinates of the same body parts are the same in each body image.

[0061] Two-dimensional pose estimation models can be pre-trained using the BlasePose model, and the two-dimensional pose estimation model corresponding to each viewpoint uses the training samples from that viewpoint for training. It can be understood that two-dimensional pose estimation models can be obtained using high-resolution networks (High-Resoultion Net, HRNet), the AlphaPose method of Regional Multi-Person Pose Estimation (RMPE), etc.

[0062] For example, 2D pose data includes information from 2D human keypoints and the image itself. 2D human keypoints are key points on a 2D image of the human body. Illustratively, human keypoints are represented using the human keypoints found in skeletal anatomy. In one example, human keypoints include, but are not limited to: head, neck, shoulders, elbows, hands, hips, knees, and feet.

[0063] In some embodiments, the keypoint information and motion transformations of the human body during certain activities such as fitness and dance are highly correlated. To convert keypoint location information into feature vectors, the center points of the left and right hips are considered the body's center point, and the distances from the body's center point to the center points of the left and right shoulders are considered the body's torso dimensions. Then, the relative position of each keypoint is initialized based on the body's center point, and the relative distance between each keypoint and the body's center point is used as part of the feature vector. Simultaneously, as... Figure 1 The human keypoint association map shown extracts paired distances between joints as another part of the feature vector according to the human joint association rules, such as the distance between the wrist and shoulder, the ankle and hip, and the distance between the two wrists. The final result is two-dimensional pose data including the aforementioned feature vectors.

[0064] In operation S230, N two-dimensional attitude data are input into the three-dimensional attitude estimation model to obtain the user's three-dimensional attitude data.

[0065] A 3D pose estimation model can be obtained by pre-training an encoder and decoder structure, which can be, for example, built based on a convolutional neural network. In other embodiments, the 3D pose estimation model can be obtained using pose estimation algorithms such as OpenPose or HMR (human mesh recovery). The 3D pose estimation model is used to estimate 3D pose data from 2D pose data from multiple perspectives, predicting the 3D human pose and shape.

[0066] According to embodiments of this disclosure, the advantages of multiple perspectives are utilized to estimate the target's pose, particularly by introducing an upper perspective. Since the acquisition position of the upper perspective is higher than the user's height, the image acquired from the upper perspective can provide more information and avoid potential occlusion. Then, the images acquired from each perspective are used for two-dimensional prediction to obtain N two-dimensional pose data, which are then used as input to the three-dimensional pose estimation model. This fully leverages the advantages of multiple perspectives, linking and correlating the user pose information between different perspectives to obtain accurate three-dimensional pose data.

[0067] Figure 3 An architectural diagram illustrating the acquisition of sub-geometric representation data according to an embodiment of the present disclosure is shown. Figure 4 A flowchart illustrating the process of obtaining three-dimensional pose data according to an embodiment of the present disclosure is shown.

[0068] The 3D pose estimation model includes N encoders, such as Figure 4 As shown, this embodiment is one example of operating S230, including:

[0069] When operating S410, N two-dimensional attitude data are input into N encoders in a one-to-one correspondence to obtain N geometric representation data.

[0070] For example, the encoder may include a ResNet-50 network and a PoseNet network. The ResNet-50 network is used to extract feature representations of the input image itself, and the PoseNet network is used to further extract feature representations from the two-dimensional keypoints. The feature representations of the input image itself and the feature representations of the two-dimensional keypoints constitute the geometric representation data.

[0071] In operation S420, geometric representation data under a first perspective is determined from N geometric representation data, where the first perspective is a specific perspective among the N perspectives.

[0072] In some embodiments, the first perspective may be predetermined by human intervention, such as one of an upward perspective, a frontal perspective, a left frontal perspective, or a right frontal perspective.

[0073] In other embodiments, the first viewpoint can be dynamically determined based on individual body images, such as... Figure 9As shown.

[0074] Figure 9 A flowchart illustrating the determination of a first viewpoint according to an embodiment of the present disclosure is shown schematically. Before determining the geometric representation data under the first viewpoint from N geometric representation data, as... Figure 9 As shown, this embodiment defines the first viewpoint as including:

[0075] In operation S910, the occlusion area of ​​each body image in N body images is determined.

[0076] For example, the body region of each body image can be identified, and this body region can be divided into multiple sub-regions to segment the image and determine the boundaries of the occluded region. After the occluded region is identified, the occlusion area can be determined by the number of pixels within the occluded region.

[0077] In operation S920, the body image with the smallest occlusion area among N body images is determined, and the viewpoint corresponding to this image is taken as the first viewpoint.

[0078] According to embodiments of this disclosure, the purpose of determining the first viewpoint is to provide more pose information. Specifically, since the body image in the first viewpoint has the smallest occlusion area, it can provide more complete body pose information, which facilitates improving the accuracy of subsequent pose estimation.

[0079] In operation S430, based on the geometric representation data under the first perspective, sub-geometric representation data under each second perspective is generated, where the second perspective is any perspective other than the first perspective among N perspectives.

[0080] In some embodiments, based solely on the geometric representation data from the first perspective, and according to prior knowledge of the human body, the human posture data from each second perspective can be inferred as sub-geometric representation data. Prior knowledge of the human body refers to the fact that the human body is composed of the head, torso, and limbs. Given the geometric representation data from the first perspective, geometric representation data from other perspectives, such as information including body contours and local shapes, can be derived based on body attributes (such as height and body type).

[0081] In other embodiments, the geometric representation data from the first viewpoint and the geometric representation data from each second viewpoint can be fused to generate sub-geometric representation data for each second viewpoint. (Refer to...) Figure 3The body images from the left, right, and top views are processed by a two-dimensional pose estimation model and an encoder to obtain geometric representations Gi, Gj, and Gm, respectively. These views are considered the first view, resulting in sub-geometric representations Gjm and Gim. It is understood that this disclosure is not intended to be limited to left, right, and top views; for example, it can also include user-centered views from the top left, bottom left, top right, bottom right, or any other arbitrary angle.

[0082] For example, the geometric representation data from the first perspective can be directly concatenated with the geometric representation data from each second perspective, for example, by executing the concat command to concatenate them and obtain each sub-geometric representation data.

[0083] For example, convolutional neural networks or recurrent neural networks can be used. For instance, geometric representation data from a first perspective and geometric representation data from any second perspective can be simultaneously input into a convolutional neural network, which can then be used to further extract features to obtain each sub-geometric representation data.

[0084] According to embodiments of this disclosure, each sub-geometric representation data integrates pose information from multiple perspectives. Since the pose information provided by the image from the first perspective is relatively complete, it further forms constraints on each second perspective and contains rich pose information, thus avoiding the problem of inaccurate pose estimation caused by deviations during data extraction.

[0085] When operating the S440, the user's 3D pose data is obtained based on the sub-geometric representation data under each second viewpoint.

[0086] For example, the sub-geometric representation data from each second-viewpoint can be simultaneously used as input to a classification network to obtain the 3D pose prediction category for the network's input. Furthermore, the classification network might input five categories, each with a corresponding 3D pose template. The category with the highest probability is determined as the target category, the template corresponding to that category is called, and specific parameters are adjusted based on extracted body information to obtain the final 3D pose data. This avoids adding further template constraints, ensuring conformity to human physiology. The 3D pose template can be a high-dimensional vector of 3D pose and can be represented as... Figure 1 The image shows key points of the human body.

[0087] For example, it can be like Figure 5 and Figure 6 As shown, it continues to be processed through the decoder and linear regression layer.

[0088] Figure 5 A schematic diagram illustrating the architecture for obtaining three-dimensional pose data according to an embodiment of the present disclosure is shown. Figure 6 A flowchart illustrating the process of obtaining three-dimensional pose data according to another embodiment of the present disclosure is shown.

[0089] The 3D pose estimation model includes N-1 decoders, such as Figure 6 As shown, this embodiment is one example of operating S430, including:

[0090] In operation of S610, the sub-geometric representation data of each second viewpoint is input into N-1 decoders in a one-to-one correspondence to obtain N-1 two-dimensional skeleton data.

[0091] For example, the decoder may include a 3D Branch neural network, a Depth Branch neural network, and a Projection Branch neural network. The 3D Branch neural network is used to obtain 3D prediction information based on the sub-geometric representation data of the corresponding second viewpoint, and the Depth Branch neural network is used to obtain depth information at each keypoint based on the sub-geometric representation data of the corresponding second viewpoint. The Projection Branch neural network is used to process the outputs of the 3D Branch neural network and the Depth Branch neural network to obtain 2D skeleton information.

[0092] For example, the decoder includes a long short-term memory neural network, which further extracts sequence information between key points through sub-geometric representation data corresponding to the second perspective to obtain two-dimensional skeleton information.

[0093] Reference Figure 5 The sub-geometric representations Gjm and Gim are input one-to-one into the corresponding decoders to obtain their respective two-dimensional skeleton data.

[0094] When operating the S620, N-1 two-dimensional skeleton data are input into the linear regression layer to obtain the user's three-dimensional pose data.

[0095] For example, a linear regression layer is a fully connected layer that multiplies the latent space representation (a low-dimensional vector) by a weight matrix and then adds a bias vector to obtain the 3D pose output (a high-dimensional vector). The process can be represented as y = W*x + b, where x is the latent space representation, y is the 3D pose output, W is the weight matrix, and b is the bias vector.

[0096] According to embodiments of this disclosure, if the 3D prediction information and the depth information of key points are obtained directly, errors may occur. Further processing using a decoder improves the accuracy and robustness of pose estimation.

[0097] Figure 7 A schematic diagram illustrating an architecture for obtaining three-dimensional pose data according to another embodiment of this disclosure is shown. Figure 8 A flowchart illustrating the process of obtaining three-dimensional pose data according to another embodiment of the present disclosure is shown.

[0098] The 3D pose estimation model also includes the Nth decoder, such as Figure 8 As shown, operation S820 in this embodiment is one embodiment of operation S430, including:

[0099] When operating the S810, the geometric representation data from the first-view perspective is input into the Nth decoder to obtain the Nth two-dimensional skeleton data.

[0100] When operating the S820, input N-1 two-dimensional skeleton data and the Nth two-dimensional skeleton data into the linear regression layer to obtain the user's three-dimensional pose data.

[0101] Reference Figure 7 The geometric representation Gm, sub-geometric representation Gjm, and sub-geometric representation Gim are input one-to-one into the corresponding decoder to obtain their respective two-dimensional skeleton data.

[0102] According to embodiments of this disclosure, incorporating geometric representation data from a first-view perspective along with other sub-geometric representation data into the process of obtaining 3D pose data can provide more pose information and improve the accuracy and robustness of pose estimation.

[0103] Figure 5 and Figure 7 The difference between the two architectures shown lies in whether or not first-person geometric representation data is involved. This can lead to changes in the number of decoders and the amount of data. For example, to save computational resources, within the range of accuracy requirements, one could adopt... Figure 5 The architecture shown is illustrated. To achieve optimal information content and improve accuracy, the following approach can be adopted: Figure 7 The architecture shown.

[0104] For example, in a gym setting, the layout of gym equipment and the spatial advantages of the gym can be utilized to estimate the pose of the target by taking advantage of multiple perspectives.

[0105] The process of obtaining a two-dimensional attitude estimation model in some embodiments of this disclosure is described below.

[0106] The tracker in BlasePose is trained under supervised training using human pose keypoint data observed from multiple perspectives (left, right, and top), which serves as a fine-tuning method to optimize model parameters. The steps are as follows:

[0107] Prepare a multi-view human keypoint dataset, using data pairs labeled at the same time t from three different viewpoints.

[0108] Using the tracker in BlasePose, 33 pose keypoints were predicted for the original image at each viewpoint at the same time t.

[0109] The predicted keypoints and the labeled keypoints in the dataset are aligned and normalized to obtain keypoints in a unified coordinate system.

[0110] We perform some data augmentation on the normalized keypoints to obtain the data-augmented keypoints.

[0111] Using the enhanced keypoints and the labeled keypoints in the dataset as supervision signals, a loss function is calculated, and the tracker's parameters are backpropagated until the tracker converges or achieves the expected result. The loss function is calculated by determining the Euclidean distance between the predicted and labeled keypoint coordinates at time t.

[0112] After optimizing the parameters of a single-view tracker, there is no linkage between different views. In order to make full use of the advantages of multiple views, a weakly supervised encoder-decoder is used to learn the geometric representation in the latent space. That is, the information of 2D key points is used as supervision to learn the shared 3D representation between different viewpoints.

[0113] To address the specific needs and equipment requirements of gym settings, a dataset specifically labeled with movements was collected, including but not limited to lat pulldowns and seated chest presses. A 3D pose estimation model was trained using a weakly supervised approach to evaluate the target fitness movements. The system was optimized in two aspects: multi-view optimization and optimization using a gym-specific dataset, thereby providing better guidance to users and improving user satisfaction.

[0114] Figure 10 A flowchart illustrating the training of a 3D pose estimation model according to an embodiment of the present disclosure is shown schematically. Figure 10 As shown, training the 3D pose estimation model in this embodiment includes:

[0115] In operation S1010, N two-dimensional pose sample data are input to N encoders to be trained to obtain N geometric representation data samples. The N two-dimensional pose data samples are obtained from N sample images, and the N training images come from N viewpoints of the same user at the same time.

[0116] N two-dimensional pose estimation models can be used to process N sample images to obtain N two-dimensional pose sample data. Alternatively, N two-dimensional pose sample data can be obtained through manual annotation.

[0117] For example, firstly, N sample images come from a gym-specific dataset. Professional coaches (athletes) use gym equipment to exercise, capturing images from three perspectives (left (i), right (j), and top (m)). The dataset includes 33 key points representing the human body in images taken from different perspectives at the same time t. Secondly, the BlazePose algorithm model adapts to the pose region of interest (ROI) from the "top" perspective and uses BlazePose to predict the human pose key points in the images from the left, right, and top perspectives at the same time t, forming a skeleton data pair (I) corresponding to time t (e.g., the t-th second). i t I j t I m t This yields N two-dimensional pose sample data, which are mapping pairs between the original image and the two-dimensional skeleton.

[0118] In operation S1020, the geometric representation data samples from the first-view perspective are determined among N geometric representation data samples. (Refer to...) Figure 9 This will not be elaborated upon here.

[0119] In operation S1030, based on the geometric representation data samples from the first perspective, sub-geometric representation data samples from the second perspective are generated.

[0120] In operation S1040, the loss function value between the sub-geometric representation data sample and the geometric representation data sample under each second view is calculated, resulting in N-1 loss function values.

[0121] The loss function value can characterize the difference between the sub-geometric representation data sample and the geometric representation data sample under the same viewpoint. Based on the fusion of pose information under the first viewpoint, it reduces the bias of the sub-geometric representation data sample, so as to retain sufficient information under both viewpoints.

[0122] In operation S1050, the comprehensive loss function value is obtained based on N-1 loss function values ​​and their respective weights.

[0123] In operation S1060, the N encoders to be trained are updated based on the comprehensive loss function value.

[0124] Reference Figure 5 We construct the loss Lj between Gj and Gjm, and the loss Li between Gim and Gi. Considering that the keypoints observed from different viewpoints are different, we assign different weight coefficients to Lj and Li, and calculate the total loss Lt = W1*Lj + W2*Lm. We then use the total loss Lt to constrain the training. The weight coefficients can be determined based on the occlusion area under each second viewpoint; for example, the smaller the area, the larger the weight.

[0125] In some embodiments, after training N-1 encoders, the decoder and linear regression layer can be trained again to obtain a trained 3D pose estimation model through multi-stage training.

[0126] In other embodiments, the encoder, decoder, and linear regression layer all participate in each training iteration. For example, after obtaining N-1 loss function values, the predicted 3D pose data and the labeled 3D pose data are used to calculate the loss function value, which, together with the N-1 loss function values, constitutes the comprehensive loss function. (Refer to...) Figure 7 In this architecture, the Nth decoder also participates in the training.

[0127] During the update of the linear regression layer, the W weight matrix and bias vector parameters are used as input and output through 2D keypoint data and corresponding 3D pose data. The mean squared error function is defined as the loss function to measure the difference between the output of the linear regression layer and the real 3D pose data. The loss function is minimized using optimization algorithms such as stochastic gradient descent to update the values ​​of the weight matrix W parameters and the bias vector b in the linear regression layer.

[0128] Based on the above attitude estimation method, this disclosure also provides an attitude estimation device. The following will be combined with... Figure 11 The device is described in detail.

[0129] Figure 11 A schematic block diagram of an attitude estimation apparatus according to an embodiment of the present disclosure is shown.

[0130] like Figure 11 As shown, the attitude estimation device 1100 of this embodiment includes an image acquisition module 1110, a two-dimensional prediction module 1120 and a three-dimensional prediction module 1130.

[0131] The image acquisition module 1110 can perform operation S210 to acquire N body images of the user from N perspectives at the same time. The N perspectives include the upper perspective, and the acquisition position of the upper perspective is higher than the user's height. N is greater than or equal to 3.

[0132] The two-dimensional prediction module 1120 can perform operation S220, which is used to input N body images into N two-dimensional pose estimation models respectively to obtain N two-dimensional pose data, wherein the N two-dimensional pose estimation models have a one-to-one mapping relationship with N viewpoints.

[0133] The 3D prediction module 1130 can perform operation S230 to input N two-dimensional attitude data into the 3D attitude estimation model to obtain the user's 3D attitude data.

[0134] In some embodiments, the three-dimensional prediction module 1130 may perform operations S410 to S440, operations S610 to S620, and operations S810 to S820, which will not be described in detail here.

[0135] In some embodiments, the 3D prediction module 1130 may include a first viewpoint determination unit, which can perform operations S910 to S920, which will not be described in detail here.

[0136] In some embodiments, the attitude estimation device 1100 may further include a model training module, which can perform operations S1010 to S1060, which will not be described in detail here.

[0137] It should be noted that the attitude estimation device 1100 includes components for performing the above-mentioned functions. Figures 2 to 10 The modules of each step in any of the described embodiments. The implementation methods, technical problems solved, functions achieved, and technical effects of each module / unit / subunit in the apparatus embodiments are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of the corresponding steps in the method embodiments, and will not be repeated here.

[0138] According to embodiments of this disclosure, any plurality of modules in the attitude estimation device 1100 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules may be combined with at least some of the functionality of other modules and implemented in one module.

[0139] According to embodiments of this disclosure, at least one of the attitude estimation devices 1100 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the attitude estimation devices 1100 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0140] Figure 12 A block diagram of an electronic device suitable for implementing an attitude estimation method according to an embodiment of the present disclosure is shown schematically.

[0141] like Figure 12As shown, an electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage portion 1208 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0142] RAM 1203 stores various programs and data required for the operation of electronic device 1200. Processor 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Processor 1201 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1202 and / or RAM 1203. It should be noted that programs may also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0143] According to embodiments of this disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to a bus 1204. The electronic device 1200 may also include one or more of the following components connected to the I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0144] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0145] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 1202 and / or RAM 1203 and / or one or more memories other than ROM 1202 and RAM 1203 described above.

[0146] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0147] When the computer program is executed by the processor 1201, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0148] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1209, and / or installed from the removable medium 1211. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0149] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209, and / or installed from the removable medium 1211. When the computer program is executed by the processor 1201, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0150] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0151] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A pose estimation method, comprising: N body images of the user are captured from N perspectives at the same time. The N perspectives include an upper perspective, and the acquisition position of the upper perspective is higher than the user's height. N is greater than or equal to 3. The N body images are respectively input into N two-dimensional pose estimation models to obtain N two-dimensional pose data, wherein the N two-dimensional pose estimation models have a one-to-one mapping relationship with the N viewpoints; The process of inputting the N two-dimensional attitude data into a three-dimensional attitude estimation model to obtain the user's three-dimensional attitude data includes: inputting the N two-dimensional attitude data into N encoders in the three-dimensional attitude estimation model in a one-to-one correspondence to obtain N geometric representation data; determining the geometric representation data under a first viewpoint among the N geometric representation data, wherein the first viewpoint is a specific viewpoint among the N viewpoints; generating sub-geometric representation data under each second viewpoint based on the geometric representation data under the first viewpoint, wherein the second viewpoint is any viewpoint among the N viewpoints other than the first viewpoint; and obtaining the user's three-dimensional attitude data based on the sub-geometric representation data under each second viewpoint. The step of obtaining the user's three-dimensional pose data based on the sub-geometric representation data under each second viewpoint includes: simultaneously inputting the sub-geometric representation data under each second viewpoint into a classification network to obtain a three-dimensional pose prediction category; determining a target category based on the three-dimensional pose prediction category; calling the template corresponding to the target category to extract the user's body information to correct parameters and obtain the three-dimensional pose data.

2. The method according to claim 1, wherein, The step of generating sub-geometric representation data for each second perspective based on the geometric representation data from the first perspective includes: The geometric representation data from the first perspective and the geometric representation data from each of the second perspectives are fused to generate sub-geometric representation data from each of the second perspectives.

3. The method according to claim 2, wherein, The 3D pose estimation model includes N-1 decoders, and obtaining the user's 3D pose data based on the sub-geometric representation data under each second viewpoint includes: Each sub-geometric representation data under the second perspective is input into the N-1 decoders in a one-to-one correspondence to obtain N-1 two-dimensional skeleton data; The N-1 two-dimensional skeleton data are input into a linear regression layer to obtain the user's three-dimensional pose data.

4. The method according to claim 3, wherein, The three-dimensional pose estimation model further includes an Nth decoder, and the method further includes: The geometric representation data from the first perspective is input into the Nth decoder to obtain the Nth two-dimensional skeleton data; Wherein, the three-dimensional pose data of the user is obtained: The N-1 two-dimensional skeleton data and the Nth two-dimensional skeleton data are input into the linear regression layer to obtain the user's three-dimensional pose data.

5. The method according to claim 1, wherein, Before determining the geometric representation data from the first perspective among the N geometric representation data, determining the first perspective among the N geometric representation data includes: Determine the occlusion area of ​​each body image in the N body images; The body image with the smallest occlusion area among the N body images is determined, and the viewpoint corresponding to this image is taken as the first viewpoint.

6. The method according to claim 3 or 4, wherein, The 3D pose estimation model is configured to be pre-trained via the following operations: N two-dimensional pose sample data are input into the N encoders to be trained to obtain N geometric representation data samples, wherein the N two-dimensional pose sample data are obtained from N sample images, and the N training images are from N viewpoints of the same user at the same time. Determine the geometric representation data sample under the first viewpoint from among the N geometric representation data samples; Based on the geometric representation data sample from the first perspective, generate the sub-geometric representation data sample from each second perspective; Calculate the loss function value between the sub-geometric representation data sample and the geometric representation data sample under each second perspective to obtain N-1 loss function values; Based on the N-1 loss function values ​​and their respective weights, the comprehensive loss function value is obtained; The N encoders to be trained are updated based on the comprehensive loss function value.

7. An attitude estimation device, comprising: The image acquisition module is used to acquire N body images of the user from N perspectives at the same time. The N perspectives include an upper perspective, and the acquisition position of the upper perspective is higher than the user's height. N is greater than or equal to 3. The two-dimensional prediction module is used to input the N body images into N two-dimensional pose estimation models respectively to obtain N two-dimensional pose data, wherein the N two-dimensional pose estimation models have a one-to-one mapping relationship with the N viewpoints; A 3D prediction module is used to input the N two-dimensional pose data into a 3D pose estimation model to obtain the user's 3D pose data. This includes: inputting the N two-dimensional pose data one-to-one into N encoders in the 3D pose estimation model to obtain N geometric representation data; determining the geometric representation data from a first viewpoint among the N geometric representation data, where the first viewpoint is a specific viewpoint among the N viewpoints; generating sub-geometric representation data from each second viewpoint based on the geometric representation data from the first viewpoint, where the second viewpoint is any viewpoint among the N viewpoints other than the first viewpoint; and obtaining the user's 3D pose data based on the sub-geometric representation data from each second viewpoint. The step of obtaining the user's 3D pose data from the sub-geometric representation data from each second viewpoint includes: simultaneously inputting the sub-geometric representation data from each second viewpoint into a classification network to obtain a 3D pose prediction category; determining a target category based on the 3D pose prediction category; and calling the template corresponding to the target category to extract the user's body information to correct parameters and obtain the 3D pose data.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images