Methods and equipment for determining 3D human posture

Abstract

Human 3D pose can be estimated through triangulation of 2D representations of key points (e.g., joint positions). Triangulation can utilize various metrics, such as confidence scores associated with the 2D representations of key points and / or temporal consistency among multiple 3D representations of key points. 2D representations can be grouped, and candidate 3D representations for each group can be determined by considering the confidence scores of individual 2D representations within the group. The candidate 3D representation with the minimum error can then be used to characterize the key points. Further 3D representations of key points determined from images taken at different times can be used to refine the 3D representation of the key points.

Description

Technical Field

[0001] This application relates to the field of human image estimation. Background Technology

[0002] Three-dimensional (3D) human pose estimation is one of the fundamental problems in computer vision, with applications in motion, action recognition, computer-aided living, human-computer interfaces, virtual reality (e.g., telepresence), robotics, and healthcare. Due to this wide range of applications, 3D human pose estimation (e.g., from two-dimensional (2D) images) has received considerable attention. However, reconstructing 3D human pose from 2D images remains a challenging problem, as it is an ill-conditioned problem that may involve many assumptions for a given input (e.g., due to depth blur, occlusion, etc.). Summary of the Invention

[0003] This paper describes systems, methods, and apparatuses for inferring 3D human poses from human images (e.g., 2D images). Using the techniques described herein, an example device is able to use triangulation methods and leverage various metrics (such as confidence scores or temporal consistency) to weight the contribution of an input image to 3D pose estimation, from images (e.g., noisy multi-view images) to infer human poses. Figure 2 A 3D human pose (e.g., including 3D human position) is reconstructed from a set of images (e.g., 3D images). Such an example device may include one or more processors configured to acquire a first set of images of a human body (e.g., a patient on an operating table or in front of a medical scanner), wherein each image in the first set of images may be captured by a corresponding image sensor (e.g., a camera). The one or more processors may also be configured to determine corresponding two-dimensional (2D) representations of human keypoints (e.g., joint positions) based on the first set of images. Each of these 2D representations may be associated with a confidence score that indicates the accuracy of the 2D representation, which may be determined by a detection device, a human annotator, or an independent entity (e.g., another device or a human body). Based on all or a subset of the 2D representations, the one or more processors may also be configured to determine a first 3D representation of the keypoints based on triangulation of the 2D representations, and in doing so, the confidence scores associated with each 2D representation may be taken into account. The one or more processors may then estimate the 3D pose of the human body at least based on the 3D representations of the keypoints (e.g., based on multiple keypoints of the human body, such as those corresponding to joint positions).

[0004] In the example, one or more processors may be configured to arrange 2D representations of keypoints into multiple groups, each group including two or more 2D representations. For each of these groups, the one or more processors may also be configured to predict a corresponding candidate 3D representation of the keypoint by triangulating the two or more 2D representations of the keypoint included in that group, and to determine a corresponding error associated with the prediction of the candidate 3D representations. The one or more processors may also be configured to select the candidate 3D representation with the minimum error as the first 3D representation of the aforementioned keypoint.

[0005] In the example above, the corresponding error associated with the prediction of the candidate 3D representation for each of the multiple groups can be determined by obtaining the 2D re-projection of the candidate 3D representation in the corresponding coordinate systems of two or more 2D representations in that group, and calculating the sum of the differences between the 2D re-projection and the two or more 2D representations. The sum of differences can be calculated by determining the corresponding difference between each 2D re-projection and its corresponding 2D representation, weighting that difference by a confidence score associated with the corresponding 2D representation, and calculating the sum of the weighted differences. The 2D re-projection can be determined based on the corresponding projection matrix of the camera (or other suitable image sensor) associated with the 2D re-projection, and the corresponding difference between each 2D re-projection and its corresponding 2D representation can be determined as the average Euclidean distance between the 2D re-projection and its corresponding 2D representation.

[0006] In the example, one or more processors of the above-described example device may also be configured to obtain a second 3D representation of key points of the human body based on a second set of images of the human body captured at an earlier time (e.g., earlier than the first image set), and to refine the first 3D representation based on the second 3D representation (e.g., to ensure consistency across time). In the example, one or more processors may be configured to determine the time span between the time the second image set was captured and the time the first image set was captured, and to refine the first 3D representation using the second 3D representation based on this time span. For example, one or more processors may be configured to apply weights to the second 3D representation during the refinement of the first 3D representation, wherein the weights may be inversely proportional to the time span (e.g., the larger the time span, the smaller the weights). For example, if the time span exceeds a threshold or if the difference between the first and second 3D representations exceeds a threshold, the weights may be set to zero (e.g., in these cases, the second representation may be discarded). In the example, if the human body has moved between the time of capturing the second set of images and the time of capturing the first set of images, one or more processors can be configured to adjust the second 3D representation of the key points based on the human body's movement before refining the first 3D representation using the second 3D representation. Attached Figure Description

[0007] The examples disclosed herein can be understood in more detail from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0008] Figure 1 This is a simplified block diagram illustrating an example of 3D human pose reconstruction as described in this article.

[0009] Figure 2 This is a simplified block diagram illustrating an example of determining a 3D representation of key points based on reprojection error.

[0010] Figure 3 This is a simplified block diagram illustrating an example of calculating the weighted projection error of a given 3D representation.

[0011] Figure 4 This is a flowchart illustrating an example of ensuring temporal consistency among multiple 3D representations.

[0012] Figure 5 This is a simplified block diagram illustrating example components of a device that can be configured to perform the tasks described herein. Detailed Implementation

[0013] The present disclosure is illustrated by way of example rather than limitation in the figures.

[0014] Figure 1 Examples of 3D human pose estimation according to one or more embodiments of the disclosure provided herein are illustrated. As shown, example system 100 can be configured to capture multi-view images (or any other type of information) of a human body (e.g., a patient in surgery or another medical setting) and infer the 3D pose of the human body based on the multi-view images (e.g., even if the images include noise or occlusion that may interfere with the estimation of the 3D pose). The 3D pose of the human body can be defined by the corresponding 3D poses of multiple human body keypoints, thus demonstrating that the 3D pose of the human body can be estimated by determining the 3D representations of keypoints that correspond to key human pose markers (e.g., joint positions).

[0015] In the example, system 100 may include multiple (e.g., two or more) image sensors 102, which in Figure 1The sensor 104 is denoted as S, for example, in a setup with three sensors, S = {1, 2, 3}. Image sensor 102 may include a camera (e.g., a digital color camera, etc.), red, green, and blue (RGB) sensors, a depth sensor, an RGB-D sensor, a thermal sensor such as an infrared (FIR) or near-infrared (NIR) sensor, and / or other types of image capture circuitry configured to generate images (e.g., 2D images or photographs) of a human body, object, or scene. Image sensor 102 may be placed at different locations in a medical environment (e.g., at different locations in an operating room), and each image sensor may be configured to acquire a corresponding image sequence 106 of human body 104 from a corresponding viewpoint. Depending on the type of image sensor or image capture circuitry used, the images 106 generated by image sensor 102 may include, for example, one or more photographs of human body 104 taken by a camera, one or more thermal images of human body 104 generated by a thermal sensor, etc.

[0016] In the example, system 100 may include components configured to be based on image sensors. i The captured images 106 are used to determine the 2D pose representation 110 of the human body 104 (referred to as x in this document). i The device 150 (e.g., a computing device) where i ∈ S). This 2D pose representation may include corresponding 2D representations of multiple human keypoints, which in some examples can be estimated using 2D keypoint detection technology 108, or in other examples derived based on manual annotation. 2D pose representation 110 (x i The accuracy compared to an ideal 2D representation can vary across the set of image sensors 102, for example, because different factors associated with the sensors 102 (e.g., inherent inaccuracies of the selected technique, lack of visual information, sensor noise, etc.) can introduce different errors into the estimation. To consider the 2D pose representation 110 (x) when estimating the 3D pose of the human body... i The accuracy of the changes can be applied to each 2D representation 110(x) i Determine (e.g., assign to it) a confidence score 112 (referred to as ρ in this paper). i , where i∈S), and is used to indicate each 2D pose estimate 110(x i The confidence score 112 (e.g., which may include a value between 0 and 1) can be determined by a separate component of device 150 or by a human annotator as part of 2D characterization determination 108, and this determination can be based on, for example, a corresponding image sensor. i Position and / or viewpoint, determined by image sensors i The quality of the generated image, the image sensor iObjects between and to the human body 104 (which could cause the human body to be occluded and thus invisible to the image sensor) and / or objects between and to the image sensor s i The environment in which the image sensor is installed, the equipment 150, or other factors or conditions associated with human annotators.

[0017] In the example, the confidence score 112 described herein can be determined using a neural network trained to take intermediate features extracted by a 2D keypoint detection network as input and use these features to predict the confidence score for the 2D keypoint detection (e.g., the neural network can be trained to associate “fuzzy” intermediate features with lower confidence scores). In the example, the confidence score 112 can be determined based on the type of algorithm employed by the 2D keypoint detection network (e.g., a more powerful detection algorithm may be associated with a higher confidence score) and / or based on the training scheme of the 2D keypoint detection network (e.g., a higher confidence score may be given to a detection network trained on a larger or more relevant dataset). In the example, the confidence score 112 can be determined by an annotator or evaluator (e.g., some evaluators may be more stringent than others, thus returning different confidence scores for the same prediction). The annotator can be, for example, a device or a person.

[0018] In the example, device 150 of system 100 may be configured to employ triangulation technique 114 to determine the corresponding 3D representations of human body keypoints, and to further estimate the 3D pose 116 of the human body (also referred to herein as X) based on the 3D representations of the body keypoints. For example, the 3D representation of each keypoint may be determined by triangulating all or a subset of the 2D representations of the keypoints (e.g., based on singular value decomposition or SVD) and by considering confidence scores associated with each 2D representation during triangulation. In the example, the confidence scores associated with each 2D representation may be taken as a factor into the calculation of the error associated with the triangulation. For example, when determining the 3D representation of a keypoint, the 2D representation of the keypoint (e.g., estimated based on image 106 captured by image sensor 102) may be arranged into multiple groups, wherein each group may include two or more 2D representations (e.g., each group may be associated with two or more image sensors 102). For multiple groups, candidate 3D representations of keypoints for each group can be predicted by: triangulating two or more 2D representations of the keypoints included in the group, determining the error associated with each candidate 3D representation, and selecting the candidate 3D representation with the minimum error as the 3D representation of the keypoint. The prediction error associated with the candidate 3D representations of the group can be determined by: reprojecting the candidate 3D representations onto the corresponding coordinate system of the group's 2D representations, and calculating the sum of the differences between the 2D reprojections and the corresponding 2D representations. The sum of differences can be calculated by: determining the difference between each 2D reprojection and the corresponding 2D representation, weighting the difference by the confidence score associated with the corresponding 2D representation, and then calculating the sum of the weighted differences. In this way, 2D representations with higher confidence scores can be given more weight in the estimation of 3D representations, thereby increasing the accuracy of 3D estimation (e.g., the 3D representation selected using the technique described herein can be the 3D representation whose reprojection is closest to the 2D representation with a high confidence score, in order to minimize the sum of losses).

[0019] In the example, device 150 of system 100 may be configured to consider temporal consistency (e.g., consistency over time) when estimating the 3D representation of keypoints. For example, in addition to determining the first 3D representation of keypoints based on a first set of images captured at a first time, device 150 may also determine the second 3D representation of keypoints based on a second set of images that may be captured at a second time (e.g., earlier than the first set of images). Device 150 may then refine the first 3D representation based on the second 3D representation (e.g., based on a weighted average of the first and second 3D representations). The appropriate weights assigned to the first and second representations may be determined based on the estimation accuracy (e.g., as indicated by reprojection error) or confidence level of the two representations. For example, device 150 may be configured to determine the time span between the first and second times and refine the first 3D representation using the second 3D representation based on that time span. Device 150 may, for example, apply weights to the second 3D representation during the refinement of the first 3D representation, wherein the weights may be inversely proportional to the time span (e.g., the larger the time span, the less weight can be given to the second 3D representation). For example, if the time span exceeds a threshold or if the difference between the first and second 3D representations exceeds a threshold, the weights can be set to zero (e.g., zero weights can have the effect of discarding the second representation). In the example, device 150 can determine (e.g., based on its own estimation or information obtained from another source) that the human body may have moved between the first and second time points, for example, because the operating table on which the human body lies has moved from one predefined location to another. In this case, device 150 can adjust (e.g., transform) the second 3D representation based on the human body's movement before using the second 3D representation to refine the first 3D representation.

[0020] Figure 2 An example process 200 associated with determining the 3D representation of keypoints is illustrated. Process 200 may begin by arranging existing 2D representations of the keypoints into multiple groups 202 (referred to herein as C). Each individual group within group 202 (referred herein as c, c ∈ C) may include two or more 2D representations (e.g., corresponding to two or more image sensors), and group 202 (C) may include an enumeration of different (e.g., all) combinations of existing 2D representations (e.g., group 202 may be constructed based on different combinations of image sensors). Figure 1 The example setup in [example] serves as an example. Based on a set of three image sensors (e.g., S = {1, 2, 3}), four groups (e.g., C = {{1, 2}, {1, 3}, {2, 3}, {1, 2, 3}}) can be established for the image sensors or for 2D representations generated based on the image sensors. For each individual group in group 202, triangulation 204 can be performed based on the 2D representations included in that group (e.g., such as...). Figure 1The triangulation 114 is used to derive candidate 3D representations 206 for the group. For example, for the group {x1,x2}, the 2D representations x1 and x2 can be triangulated into 3D representations X. {1,2} Using a similar technique, the 2D representation triangulation of other groups 202 can be converted into 3D representation X for groups {x1,x3}, {x2,x3}, and {x1,x2,x3}, respectively. {1,3} X {2,3} X {1,2,3} .

[0021] In the example, for each 3D representation 206, the weighted reprojection error 210 (referred to as l in this paper) can be calculated at 208. ρ (X c ), where c∈C), the details of which can be further described below. After calculating the weighted projection error 210, these errors can be compared to determine the minimum weighted projection error and its corresponding 3D representation. This 3D representation (e.g., with the minimum weighted projection error) can be represented as X * ,in, c∈C, and the 3D representation can be used to represent key points and undergo other refinement operations (e.g., time-consistency-based refinement as described in this paper).

[0022] Figure 3 An example is shown for computing a given 3D representation 304 (referred to as X in this paper). c An example technique for weighted projection error. 3D representation 304 can be based on a set of 2D representations 302 (referred to herein as {x}). i} i∈c , where c∈C) to determine. As described herein, this set of 2D representations (e.g., such as Figure 2 The set of 2D representations (202) in the image can be based on 2D images captured by two or more image sensors (e.g., such as...). Figure 1 The set of 2D images (106) is used for estimation, and the 3D representation is 304 (X). c ) can be represented by this set of 2D characters 302({x i} i∈c Triangulation is used to calculate. For example, if c = {1, 2}, then the 2D representation 302 can be represented as {x1, x2}, and the 3D representation 304 can be represented as X. {1,2} .

[0023] The accuracy of the 3D representation 304 can be determined or indicated by the reprojection error. In this example, such reprojection error can be calculated based on the projection matrix 306 of the image sensor (referred to herein as P), whose coordinate system can be used for reprojection. The projection matrix 306 can be defined by internal and / or external parameters of the image sensor, which can be the system described herein (e.g., Figure 1 System 100) or device (e.g., Figure 1 The device 150 in the image sensor is known. Therefore, the 2D reprojection of the 3D representation X in the coordinate system of the image sensor can be calculated using the projection matrix 306 of the image sensor, for example, based on the following equation: Among them, X c It can represent the 3D representation associated with group c. It can characterize X c 2D reprojection in the coordinate system of image sensor i (i∈c) in group c, and P i The projection matrix 306 can characterize the image sensor i.

[0024] Once the reprojection of the 3D representation to the 2D coordinate system of image sensor i is determined, it can be performed in 308, for example, based on the 2D reprojection (as described above). Where i∈c) and the original 2D representation (x) i The 3D representation X is calculated using the average Euclidean distance between i ∈ c). c The reprojection error (referred to as l(X) in this paper) c This operation can be illustrated by the following equation 1):

[0025]

[0026] Then, characterize the 2D characterization 302(x) i The confidence score for the accuracy of ) is 310 (ρ i Such as Figure 1 The confidence score 112) can be used to weight the reprojection error determined using equation 1). For example, using the confidence score 310 as input, it can be calculated based on the following equation 2), expressed as l. ρ (X c (For example, such as) Figure 2 The weighted projection error 210) and the weighted projection error 312 are:

[0027]

[0028] And it is possible to select from the group with the minimum sum of weighted projection errors (e.g., Figure 2Candidate 3D representations of the groups {x1,x2}, {x1,x3}, {x2,x3}, and one of {x1,x2,x3} are used to characterize key points in 3D space, as described in this paper.

[0029] Figure 4 This is a flowchart illustrating example operations that can be associated with refining the 3D representation of keypoints based on other 3D representations of keypoints determined using images captured at different time steps. As shown, the first refined 3D representation can be generated based on the image captured at the first time (referred to as t in this document) and can be represented as (X t The second 3D representation used to refine the first 3D representation can be generated based on an image captured at a second time (referred to as t-1 in this paper, which may be earlier than t), and can be represented as (X t-1 ).

[0030] At step 402, it can be determined whether information associated with the previous time step (t-1) has been provided. This information may include, for example, information from the previous time step X. t-1 3D representation of the object and its corresponding reprojection error (e.g., such as Figure 2 The weighted projection error 210 in this paper can be referred to as l. t-1 This includes weights (also known as decay weights or decay coefficients) that can reflect the time span between t and t-1 or the difference between two 3D representations associated with t and t-1 respectively. If it is determined at 402 that no information associated with t-1 is provided, the refinement operation can end, and the 3D representation associated with time t can be returned. If it is determined at 402 that information associated with t-1 is provided, the weights (d) to be applied can be further determined at 404. For example, if it is determined at 402 that information associated with the previous time step (X) is provided... t-1 , l t-1 If the information associated with d) is available, then determining a 404 error can involve deciding whether to apply a hard reset to the weights. If a hard reset of the weights exists, for example, d < 0, then the refinement operation can end and can return to the state with respect to the current time step (X). t The associated 3D representation. For example, if it is determined that the human body whose 3D pose is reconstructed has moved since a previous time step, a hard reset of the weights can be applied. In these cases, the weights can be set to zero or negative values, allowing the refinement operation to discard X. t-1 and return X t .

[0031] If it is determined at 404 that no hard reset needs to be applied to the weights, then at 406 a decision can be made regarding the 3D representation X. t-1 With 3D representation X t The difference or distance between them (e.g., the time difference or X between t-1 and t).t-1 With X t Another determination (in terms of the difference between them). If X t-1 With X t The distance between them is greater than a threshold, for example, |X t –X t-1 If |t–(t-1)| is greater than the threshold, the refinement operation can end and return X. t If X is determined in 406 t-1 With X t If the distance between them is not greater than the threshold, then the reprojection error l associated with time t can be calculated at 408. t and the reprojection error l associated with time t-1 t-1 Another determination. If the reprojection error l t Less than the reprojection error l weighted by d t-1 For example, l t <d*l t-1 The refinement operation can then end and return X. t Otherwise, the operation can proceed to 410. The reprojection error is calculated using weights. t-1 Weighting can compensate for the possibility that the human body has moved during the time span between t and t-1. For example, the longer the time span between t and t-1, the more likely the human body is to have moved, thus X t-1 The less likely it is to be used to refine X t Or it can represent key points in 3D space.

[0032] If the reprojection error is determined at 408... t Not less than the reprojection error l weighted by d t-1 Then, at 410, another determination can be made regarding whether the movement of the human body between t and t-1 can be determined (e.g., due to the adjustment of the operating table). If the person has already moved between t and t-1, but the movement cannot be determined (e.g., cannot be provided), then the refinement operation can end and return to X. t-1 Otherwise, the refinement operation can end and return the updated X. t-1 Previously, the movement of the human body could be determined (e.g., referred to as b in this article). t-1→t And the 3D representation X can be updated based on this movement. t-1 (For example, updated to X) t-1 +b t-1→t For example, if a human body lying on an operating table is moved from one predetermined position to another, then 3D prediction X... t-1 The transformation can be based on the movement before returning, taking the movement into account.

[0033] For the sake of simplicity, the operations associated with refining the 3D representation can be... Figure 4 These operations are depicted and / or described in a specific order. However, it should be understood that these operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that... Figure 4 Not all operations that are illustrated and described herein are possible with the example device. It should also be noted that not all illustrated operations require the example device to perform.

[0034] The systems, methods, and / or apparatuses described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 5 This is a block diagram illustrating an example device 500 that can be configured to perform the 3D pose estimation task described herein. As shown, device 500 may include a processor (e.g., one or more processors) 502, which may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller, reduced instruction set computer (RISC) processor, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), or any other circuitry or processor capable of performing the functions described herein. Device 500 may also include communication circuitry 504, memory 506, mass storage device 508, input device 510, and / or communication link 512 (e.g., communication bus) through which one or more components shown in the figure exchange information.

[0035] Communication circuitry 504 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including local area networks (LANs), wide area networks (WANs), the Internet, and wireless data networks (e.g., Wi-Fi, 3G, 4G / LTE, or 5G networks). Memory 606 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions that, when executed, cause processor 502 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc.). Mass storage device 508 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROMs or DVD-ROMs, etc., on which instructions and / or data may be stored for operation of processor 502. Input device 510 may include a keyboard, mouse, voice-controlled input device, touch-sensitive input device (e.g., touch screen), etc., for receiving user input from device 500.

[0036] It should be noted that device 500 can operate as a standalone device or can be connected to other computing devices (e.g., networked or grouped) to perform the functions described herein. And even in Figure 5 Only one example of each component is shown in the figure, and those skilled in the art will understand that the device 500 may include multiple instances of one or more components shown in the figure.

[0037] Although this disclosure has been described according to certain embodiments and generally associated methods, changes and variations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit this disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure. Furthermore, unless specifically stated otherwise, discussions using terms such as “analyze,” “determine,” “enable,” “identify,” and “modify” refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data representing physical (e.g., electronic) quantities within the registers and memories of the computer system into other data representing physical quantities within the computer system's memory or other such information storage, transmission, or display devices.

[0038] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art upon reading and understanding the above description. Therefore, the scope of this disclosure should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A method for estimating three-dimensional (3D) human pose, the method comprising: A first set of images of the human body is obtained, wherein each image in the first set of images is captured by a corresponding image sensor; Based on the first image set, corresponding two-dimensional (2D) representations of key points of the human body are determined, wherein each 2D representation is associated with a confidence score; A first 3D representation of the keypoint is determined based on the 2D representation of the keypoint and the confidence score associated with each 2D representation, wherein the 2D representations of the keypoint are arranged into multiple groups, each group comprising two or more of the 2D representations, each group comprising two or more of the 2D representations associated with two or more image sensors; for each of the multiple groups, a corresponding candidate 3D representation of the keypoint is predicted by triangulation of the two or more 2D representations of the keypoint included in the group; a corresponding error associated with the prediction of the candidate 3D representation of each of the multiple groups is determined; and the candidate 3D representation with the minimum error is selected as the first 3D representation of the keypoint, the first 3D representation being determined based on triangulation of all or a subset of the 2D representations and by considering the confidence score associated with each 2D representation during the triangulation; and The 3D pose of the human body is estimated based at least on the first 3D representation of the key points.

2. The method according to claim 1, wherein, The corresponding error associated with the prediction of the candidate 3D representation of each of the plurality of groups is determined by obtaining the 2D reprojection of the candidate 3D representation in the corresponding coordinate system of the two or more 2D representations of the group, and calculating the sum of the differences between the 2D reprojection and the two or more 2D representations.

3. The method according to claim 2, wherein, The sum of the differences is calculated by: determining the corresponding difference between each reprojection in the 2D reprojection and the corresponding 2D representation among the two or more 2D representations, weighting the differences by the confidence score associated with the corresponding 2D representation, and calculating the sum of the weighted differences.

4. The method according to claim 1, wherein, The confidence score associated with each of the 2D representations indicates the accuracy of the 2D representation.

5. The method according to claim 1, further comprising: A second 3D representation of the key points of the human body is obtained based on a second set of images of the human body, and the first 3D representation of the key points is refined using the second 3D representation of the key points, wherein the second set of images was captured earlier than the first set of images.

6. The method according to claim 5, wherein, Refining the first 3D representation of the key point using the second 3D representation of the key point includes: Determine the time span between the time of capturing the second image set and the time of capturing the first image set; and During the refinement of the first 3D representation, weights are applied to the second 3D representation, wherein the weights are inversely proportional to the time span.

7. The method according to claim 5, wherein, Refining the first 3D representation of the key point with the second 3D representation of the key point includes adjusting the second 3D representation of the key point based on the movement of the human body between the time of capturing the second set of images and the time of capturing the first set of images.

8. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-7.

9. An apparatus comprising: One or more processors configured to perform the method as described in any one of claims 1-7.

Images