A human three-dimensional posture estimation method and device based on a catadioptric system

Abstract

The present application relates to a kind of human three-dimensional posture estimation method and device based on catadioptric system.The steps of the method include: video acquisition including human catadioptric system is carried out;The video collected is preprocessed;The geometric model of single mirror stereo system is established, using the geometric model, according to the position of two-dimensional joint node in mirror inside and outside human in catadioptric system, the estimation of human three-dimensional posture is carried out using triangular positioning method;Using epipolar geometry and the symmetry constraint of human joint length, the position of the occluded node of human in image is inferred, and higher quality human three-dimensional joint node estimation result is generated.The present application greatly reduces the uncertainty of camera intrinsic parameter calibration using the prior information of human itself, and the occluded joint node position of human in image is inferred using the epipolar geometry property of catadioptric system and the prior information of human, so as to complete high-quality human three-dimensional posture estimation in catadioptric video.

Description

Technical Field

[0001] This invention relates to a method and apparatus for estimating human three-dimensional pose based on a catadioptric system, and more particularly to an algorithm for inferring human occlusion joints based on epipolar geometry and human symmetry constraints, belonging to the field of computer vision. Technical Background

[0002] Accurately reconstructing human 3D pose (3D coordinates of human joints) from real images in various indoor and outdoor scenarios has wide applications in many emerging fields, such as virtual reality, augmented reality, human-computer interaction, and humanoid robots. Over the past decade, numerous datasets related to 3D human pose have emerged, greatly advancing the field.

[0003] Existing methods for acquiring human 3D pose datasets can be broadly categorized into three types. The first type uses commercially available sensor-based motion capture systems synchronized with video to collect human 3D pose data, such as HumanEva, Human3.6M, and GPA. This type of data is primarily collected indoors and requires actors to wear tight-fitting clothing for data recording. To overcome the limitations of clothing in sensor-based data collection, some sensorless methods have been proposed. The MPI-INF-3DHP dataset presents a sensorless motion capture system that uses multiple cameras in a dome-shaped studio with a green screen to collect data from multiple perspectives. This sensorless system allows for greater diversity in the costumes of actors in the dataset, but it requires the construction of an expensive studio. The third type uses synthetic data, generating 3D data of animated characters using game engines. Data acquired in this way often lacks realism. Models trained using synthetic data often exhibit poor generalization performance on real test datasets.

[0004] When acquiring realistic 3D human pose data using the methods described above, high-precision sensors or multi-camera acquisition systems are either prohibitively expensive or limited in the scenarios where data can be acquired. This invention proposes a method for acquiring 3D human pose data using catadioptric video. In traditional catadioptric systems (optical systems that combine refractive lenses and reflective mirrors), each mirror can be used as a virtual camera. For example, Nayar et al. designed and implemented a real-time catadioptric stereoscopic vision system that uses only one camera and two plane mirrors. Utilizing catadioptric video for depth estimation of human joints presents two main challenges: First, as Nayar et al. pointed out, catadioptric systems require at least two mirrors for camera calibration and 3D estimation, which is impractical for catadioptric videos on social networks (e.g., some dance videos on YouTube or Bilibili that only have one mirror). Second, online dance videos with catadioptric effects are usually captured in real-world scenarios, which may involve occlusion, significantly increasing the difficulty of acquiring reliable 2D human joint data. Unlike sensor-based systems, the human pose data obtained through the above methods often exhibits significant noise, requiring further refinement of the data. Summary of the Invention

[0005] To address the aforementioned challenges, and drawing inspiration from the inherent properties of the human body, this invention is inspired by the following: Unlike general target objects, the human body possesses a wealth of powerful prior information, such as the symmetry of the lengths of the left and right skeletons and the limitations on the degrees of freedom of joints. In this invention, for a single-mirror system in catadioptric video, the invention significantly reduces the uncertainty of camera intrinsic parameter calibration by utilizing the prior information of the human body itself. Furthermore, it infers the positions of occluded joints in the image by leveraging the epipolar geometry of the catadioptric system and the prior information of the human body, thereby achieving high-quality 3D pose estimation of the human body in catadioptric video.

[0006] The technical solution adopted in this invention is as follows:

[0007] A method for estimating human three-dimensional pose based on a catadioptric system includes the following steps:

[0008] Perform video capture, including of the human body's catadioptric system;

[0009] The acquired video is preprocessed, including human bounding box detection, human 2D joint detection, and video segmentation.

[0010] A geometric model of a single-mirror stereo system is established. Using this geometric model, the three-dimensional posture of the human body is estimated by triangulation based on the positions of the two-dimensional joints of the human body inside and outside the mirror in the catadioptric system.

[0011] By using epipolar geometry and human joint length symmetry constraints, the positions of occluded nodes in the human body in the image are inferred, generating higher quality human 3D joint point estimation results.

[0012] Furthermore, the video acquisition process aims to ensure diversity in human posture, clothing, and background, and excludes videos with large camera movements to the greatest extent possible, thereby simplifying the composition of camera intrinsic parameter estimation.

[0013] Furthermore, the human body two-dimensional joint detection uses a confidence score to distinguish the occlusion status of each joint. When the confidence score si is less than a set threshold α, the joint is marked as occluded. When the number of occluded points in a video segment is greater than a set threshold N, the video segment is judged as severely occluded and such videos are filtered out.

[0014] Furthermore, the geometric model of the single-mirror solid system includes:

[0015] Assuming the world coordinate system and the camera coordinate system are aligned, the plane mirror π is represented by a normal vector n and a non-negative scalar d≥0, where d represents the orthogonal distance between the mirror π and the optical center O; P represents any point in space not on the mirror, and P' is its reflection point in space. The relationship between P and P' is as follows:

[0016] P′=P+2(dn T P)n

[0017] =(E-2nn) T )P+2dn

[0018] =RP+t,

[0019] R and t are calculated as follows, where E is the identity matrix:

[0020] R = E - 2nn T , t=2dn,

[0021] Using p and p' as separate tables to represent the pixel coordinates corresponding to P and P', the relationship between them can be expressed as follows:

[0022]

[0023] Where Z and Z' represent the z-axis coordinates of P and P' respectively; K is the camera intrinsic parameter matrix:

[0024]

[0025] Where f is the camera focal length, which is estimated based on the symmetry of the lengths of the left and right bones of the human body.

[0026] For any point P in space, we have:

[0027]

[0028] in,

[0029] Furthermore, the inference of the occlusion node positions of the human body in the image includes:

[0030] Human joints are divided into two categories: trunk joints and limb joints;

[0031] For trunk joints, temporal filtering smoothing is used to optimize occluded joints;

[0032] For limb joints, occluded joints are inferred by using epipolar geometry constraints and human skeleton prior constraints.

[0033] Furthermore, the location of the poles in the image is estimated using the following method:

[0034] Select some matching points with high confidence scores, obtain the lines connecting each set of matching points, and then calculate the intersection of any two lines to obtain the location of the candidate poles. Then calculate the median of the coordinates of all candidate poles to obtain a reliable estimate of the pole location.

[0035] A human body three-dimensional pose estimation device based on a catadioptric system employing the above method, comprising:

[0036] The video acquisition module is used to acquire video, including data from the human body's catadioptric system.

[0037] The preprocessing module is used to preprocess the acquired video, including human bounding box detection, human 2D joint point detection, and video segmentation.

[0038] The human body 3D joint estimation module is used to establish a geometric model of a single-mirror stereo system. Using this geometric model, the 3D pose of the human body is estimated by triangulation based on the positions of the 2D joints of the human body inside and outside the mirror in the catadioptric system. Furthermore, by using epipolar geometry and constraints of the symmetry of human joint lengths, the positions of occluded nodes of the human body in the image are inferred, generating higher quality human body 3D joint estimation results.

[0039] The key technical contributions and beneficial effects of this invention can be summarized in two aspects:

[0040] 1. This invention presents a novel method for intelligently estimating high-quality 3D human joint points from catadioptric videos on the internet. The algorithm proposed in this invention combines epipolar geometry constraints and prior human information constraints to infer the positions of occluded joint points in images, thereby achieving high-quality 3D pose estimation of human joint points.

[0041] 2. This invention proposes a method for estimating camera intrinsic parameters of video including a catadioptric system using human body symmetry information. Specifically, given a human motion video including a catadioptric system, this invention finds the camera intrinsic parameters that best satisfy human body symmetry, thereby completing the estimation of camera intrinsic parameters. Attached Figure Description

[0042] Figure 1 This is an example diagram of a video frame that includes a catadioptric system.

[0043] Figure 2 This is the geometric model corresponding to the catadioptric and reflective system.

[0044] Figure 3 This is an example of the process of optimizing human joint points in a video frame of a catadioptric system.

[0045] Figure 4 Six scenarios optimized for shoulder-elbow occlusion joint points.

[0046] Figure 5 Estimate the mean and variance of the focal length for each video segment in different videos. Detailed Implementation

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to embodiments and accompanying drawings.

[0048] This invention proposes a method for estimating human 3D pose based on a catadioptric system, which mainly includes the following: 1) Acquiring video from the Internet, including footage of a human catadioptric system. 2) Preprocessing the video of the catadioptric system, including human bounding box detection, human 2D joint detection, and video segmentation. 3) Estimating the human 3D pose using triangulation based on the positions of the human 2D joints inside and outside the mirror in the catadioptric system. 4) Estimating the camera intrinsic parameters (focal length) in the video of the catadioptric system. The camera focal length is mainly estimated based on the symmetry of the lengths of the left and right bones of the human body. 5) Inferring the positions of occlusion nodes of the human body in the image using constraints of epipolar geometry and the symmetry of human joint lengths, generating higher quality 3D joint estimation results for the human body in the video of the catadioptric system.

[0049] 1. Video capture including human body catadioptric systems

[0050] In recent years, numerous dance tutorial videos have been shared on social media. This invention selected Bilibili and YouTube as the two primary sources for data collection. On these two websites, videos tagged with "dance mirror tutorials" were searched, and the most relevant videos from the search results were selected and recorded. Videos without mirrors and those with excessive camera movement were manually filtered out. (See attached image.) Figure 1 As shown, some randomly selected video frames are displayed. As can be seen, the videos collected by this invention exhibit a high degree of diversity in terms of human posture, ethnicity, clothing, and gender.

[0051] Existing sensor-based motion capture systems often require subjects to wear tight-fitting clothing to acquire 3D human pose data, resulting in a lack of data diversity. This invention, when collecting video data from a catadioptric system, strives to ensure diversity in human pose, clothing, and background. Furthermore, it excludes videos with large camera movements to simplify the construction of camera intrinsic parameter estimation.

[0052] 2. Preprocessing of video from a catadioptric system

[0053] In the data preprocessing section, this invention extracts and saves 5 frames per second from all videos. Then, a human detector trained on MS-COCO is used to detect human bounding boxes. The human detector uses an FPN model based on ResNet-101. This invention only retains video frames showing two people (the dancer and their mirror image) within the field of view, and then takes 20 consecutive seconds to form a video segment. To ensure the diversity of the entire data, this invention retains a maximum of 100 video segments for each video.

[0054] Next, this invention uses an HR-Net model pre-trained on the MS-COCO human 2D joint detection data to detect 17 2D joints p for each person in the image. i =(x i y i s i ), where (x i y i ) represents pixel coordinates, s i This is the confidence score. Self-occlusion often occurs in the obtained data; this invention simply uses the confidence score to distinguish the occlusion status of each keypoint. When s iWhen the number of occluded points is less than α, the keypoints are marked as occluded. When the number of occluded points in a video segment is greater than N, the video segment is judged as severely occluded, and such videos will be filtered out and no longer used in subsequent algorithms. In the actual algorithm operation of this invention, α and N are set thresholds. In this embodiment, the values ​​of α and N are set to 0.7 and 1500, respectively.

[0055] 3. Geometric model of a single-mirror solid system

[0056] The geometric model used in this invention is as follows: Figure 2 As shown. Assume O is the optical center of the camera, and π is the reflector in the video. As is customary, a virtual camera representation is added to the reflector (e.g., ...). Figure 2 The optical center O' of the mirror camera in the middle.

[0057] With very few exceptions, a dancing mirror can be modeled as a plane mirror. Assume the world coordinate system is aligned with the camera coordinate system. The plane mirror π can be represented by a normal vector n and a non-negative scalar d ≥ 0. d represents the orthogonal distance between the mirror π and the optical center O. Let P represent any point in space not on the mirror. P' is its reflection point in space. The relationship between P and P' is as follows:

[0058] P′=P+2(dn T P)n

[0059] =(E-2nn) T )P+2dn

[0060] =RP+t, (1)

[0061] R and t are calculated as follows, where E is the identity matrix.

[0062] R = E - 2nn T , t=2dn, (2)

[0063] To estimate the depth of human joints, the camera's intrinsic parameter matrix is ​​first required. Since sufficient information about the camera is lacking in video data collected from the network, camera self-calibration is very difficult. This invention simplifies the camera intrinsic parameters, assuming the camera's optical center is above the image center, and primarily calibrates the camera's focal length. Specifically, this invention approximately obtains a camera intrinsic parameter matrix of the following form:

[0064]

[0065] Let p and p' represent the pixel coordinates corresponding to P and P'. For the pinhole camera model, the relationship between them can be expressed as:

[0066]

[0067] Where Z and Z' represent the z-axis coordinates of P and P', respectively.

[0068] 4. Estimation of three-dimensional joints in the human body

[0069] This invention estimates the three-dimensional joints of the human body based on the positions of the two-dimensional joints of the human body inside and outside the mirror in the catadioptric system using a triangulation method.

[0070] Based on the above introduction, we can see that the geometric model of this single-mirror solid system is mainly related to the normal vector n of the mirror and the focal length f. Now, assuming the camera focal length f is known, the estimation process for f will be introduced in the next section. To estimate n, we have the following observations:

[0071] For any point P in space, we have:

[0072]

[0073] in,

[0074] The proof is as follows: Let × denote the cross product of two vectors, and <,> denote the inner product. For example... Figure 2 As shown, points O, O', and P' are coplanar. According to the definition of the cross product, t×P' represents the normal vector of the plane, which is orthogonal to any point in the plane. Therefore:

[0075]

[0076] According to formula (3), the following results are obtained:

[0077]

[0078] That is how you get proof.

[0079] Formula (4) gives the calculation method for the mirror normal vector n. For all matching points (p, p') in the video segment, this invention uses the RANSAC algorithm to estimate the optimal n. R and t can be obtained according to formula (2). Then, in order to estimate the depth of the human joints, this invention applies the existing triangulation technology to formula (3) and returns two depth values ​​Z and Z'. After calculating R and t, and knowing a pair of two-dimensional joints, the position of the three-dimensional joints can be obtained according to formula (3).

[0080] 5. Estimation of camera intrinsic focal length

[0081] All the above calculations are performed under the condition that the camera focal length f is known. The estimation process for the focal length f is described below. For a camera, the focal length f is often within a range; therefore, this invention employs a search method to find the optimal focal length that satisfies the human body symmetry constraint. Specifically, for each candidate focal length f, all three-dimensional joints of the human body are calculated as described above. Next, for each f, the symmetry score of the left and right limb pairs of the human body under that f condition is calculated, for example:

[0082] min(||W l -A l ||2,||W r -A r ||2) / max(||W l -A l ||2,||W r -A r ||2)

[0083] Among them, W l Indicates the position of the left wrist in three-dimensional space, A l W represents the position of the left elbow in three-dimensional space. r Indicates the position of the right wrist in three-dimensional space, A r This represents the position of the right elbow in three-dimensional space. The focal length f, which best represents the symmetry of the human skeleton, will be preserved. Figure 5 This displays the average and standard deviation of the estimated focal lengths for all video segments within the same video. The average of the estimated focal lengths for all segments within the same video is used as an estimate of the focal length f of the camera that captured the video.

[0084] 6. Inference of joint points obscured by the human body

[0085] Since the algorithm of this invention estimates the three-dimensional human joints from matched two-dimensional human joints, the accuracy of the two-dimensional joint position estimation becomes crucial. Unfortunately, in most catadioptric video systems, self-occlusion inevitably occurs, leading to unreliable estimates of the two-dimensional human joints.

[0086] To recover occluded two-dimensional human body joints, this invention first categorizes human body joints into two types: trunk joints and limb joints. Trunk joints mainly include the head, neck, shoulders, and hips, while limb joints include the elbows, wrists, knees, and ankles. When the human body moves, the trunk typically moves more slowly, while the limbs tend to swing more rapidly. Based on this observation, this invention uses different strategies to fine-tune the two types of joints respectively.

[0087] For slow-moving trunk joints, this invention uses temporal filtering smoothing to optimize occluded joints. As mentioned earlier, a joint is considered occluded when its estimated confidence is below α. For trunk joints, their temporal joint sequence is represented as {..., h...} i-1 h i h i+1 , ...}. When h i When occluded (low confidence level), and h i-1 ,h i+1 Not obscured, use 0.5*(h) i-1 +h i+1 Replace the original h with ) i For all video segment sequences, this invention uses an occlusion strategy to optimize the torso joints.

[0088] For limb joints, this invention infers occluded human joints by using combined epipolar geometry constraints and prior constraints on the human skeleton. In a single-mirror reflection system, such as... Figure 3 Point e in the diagram is a pole. According to the properties of the catadioptric system and the epipole, all straight lines connecting two matching points (e.g., any pair p, p') will intersect at the pole e. If the camera capturing the video is visible in the field of view, then the pole is located precisely at the camera's position. Figure 3 An example is shown.

[0089] To optimize occluded limb joints, the location of the poles in the image must first be estimated. In a video with a fixed camera position, the position of the poles remains unchanged across all video frames. In practice, this invention selects matching points with high confidence scores, obtains the lines connecting each set of matching points, and then calculates the intersection of any two lines to obtain the positions of candidate poles. The median of the coordinates of all candidate poles is then calculated as a reliable estimate of the pole positions. Figure 3 As shown, let e = (x e ,y e (a) represents the pixel coordinates of the pole in the image. mr ,a ml ,a r ,a l (w) is used as a representation of the elbow. mr ,w ml ,w r ,w l ) represents the wrist. Where, a mr a represents the pixel coordinates of the right elbow in the mirror image. ml a represents the pixel coordinates of the left elbow in the mirror image. r a represents the pixel coordinates of the right elbow outside the mirror image.l Indicates the position of the left elbow outside the mirror, w mr w represents the pixel coordinates of the right wrist in the mirror image. ml w represents the pixel coordinates of the left wrist in the mirror image. r w represents the pixel coordinates of the right wrist outside the mirror. l Let w represent the pixel coordinates of the left wrist outside the mirror. Without loss of generality, assume w r The joints are occluded, and the model's estimated 2D joint positions are inaccurate due to w mr Not obscured, and w r It should be located at the polar line Therefore, the following polar constraints apply:

[0090]

[0091] Where (x) wr ,y wr ) is w r The candidate pixel coordinates. For unoccluded point pairs (w l ,w ml ),(a l ,a ml ) and (a r ,a mr They can be located using triangulation to calculate their corresponding three-dimensional coordinates W. l A l and A r A basic prior knowledge about the human skeleton: the left and right arms should be of equal length. Based on this prior knowledge, the following equation can be written:

[0092] ||W l -A l ||2=||W r -A r ||2, (8)

[0093] Where ||·||2 refers to the Euclidean distance, W r It is w r The corresponding three-dimensional coordinates, when (x) wr ,y wr This can then be obtained through triangulation. In practice, the present invention is located on the polar line as (x wr ,y wr Collect some candidate samples and retain the best solution that satisfies both formula (7) and formula (8).

[0094] like Figure 4 As shown, this illustrates the six possible occlusion scenarios for the "shoulder-elbow" relationship. Regarding the joint optimization solution of equations (7) and (8) introduced above, the solution is... Figure 4 The second case (Case 2) is described above. In fact, the third and fourth occlusion cases can also be simplified to conditional versions of the second case. To solve the problem of inferring joint points in other types of occlusion, this invention categorizes human limb joint points into four types: "shoulder-elbow," "elbow-wrist," "hip-knee," and "knee-ankle." Except for the "shoulder-elbow" case, which uses the above method to infer occlusion points, the other three cases can also infer occluded joint points in the image using the same method.

[0095] The human body three-dimensional pose representation used in this invention is based on joints. Other human body pose representations based on contours and volumes can also be accurately estimated using the methods mentioned in this invention.

[0096] Based on the same inventive concept, another embodiment of the present invention provides a human three-dimensional pose estimation device based on a catadioptric system employing the above-described method, comprising:

[0097] The video acquisition module is used to acquire video, including data from the human body's catadioptric system.

[0098] The preprocessing module is used to preprocess the acquired video, including human bounding box detection, human 2D joint point detection, and video segmentation.

[0099] The human body 3D joint estimation module is used to establish a geometric model of a single-mirror stereo system. Using this geometric model, the 3D pose of the human body is estimated by triangulation based on the positions of the 2D joints of the human body inside and outside the mirror in the catadioptric system. Furthermore, by using epipolar geometry and constraints of the symmetry of human joint lengths, the positions of occluded nodes of the human body in the image are inferred, generating higher quality human body 3D joint estimation results.

[0100] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0101] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0102] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A method for estimating human three-dimensional pose based on a catadioptric system, characterized in that, Includes the following steps: Perform video capture, including of the human body's catadioptric system; The acquired video is preprocessed, including human bounding box detection, human 2D joint detection, and video segmentation. A geometric model of a single-mirror stereo system is established. Using this geometric model, the three-dimensional posture of the human body is estimated by triangulation based on the positions of the two-dimensional joints of the human body inside and outside the mirror in the catadioptric system. By using epipolar geometry and human joint length symmetry constraints, the positions of occluded nodes in the human body in the image are inferred, generating higher quality human 3D joint point estimation results. The geometric model of the single-mirror solid system includes: Assuming the world coordinate system and the camera coordinate system are aligned, the plane mirror π has a normal vector. n and a nonnegative scalar d ≥ 0 means, d Represents the mirror π and the optical center O The orthogonal distance between them; P Represents any point in space that is not reflected in the mirror. P’ It is its reflection point in space. P The relationship between P and P' is as follows: R and t are calculated as follows, where E is the identity matrix: use p, p' Tables are displayed P, P' The corresponding pixel coordinates, and the relationship between them can be expressed as: in, Z , Z’ Tables are displayed P , P’ The z-axis coordinate; K is the camera intrinsic parameter matrix: ； in f The camera focal length is estimated based on the symmetry of the lengths of the left and right bones in the human body. For any point in space P ,have: in, ; The inference of the occlusion node position of the human body in the image includes: Human joints are divided into two categories: trunk joints and limb joints; For trunk joints, temporal filtering smoothing is used to optimize occluded joints; For limb joints, occluded joints are inferred by using epipolar geometry constraints and human skeleton prior constraints. The method of inferring occluded joints using epipolar geometry constraints and prior constraints on human skeletons includes: Let e ​​= (x e , y e (a) represents the pixel coordinates of the pole in the image. mr , a ml , a r , a l (w) is used as a representation of the elbow. mr ,w ml , w r , w l ) represents the wrist, where a mr a represents the pixel coordinates of the right elbow in the mirror image. ml a represents the pixel coordinates of the left elbow in the mirror image. r a represents the pixel coordinates of the right elbow outside the mirror image. l Indicates the position of the left elbow outside the mirror, w mr w represents the pixel coordinates of the right wrist in the mirror image. ml w represents the pixel coordinates of the left wrist in the mirror image. r w represents the pixel coordinates of the right wrist outside the mirror. l This represents the pixel coordinates of the left wrist outside the mirror; assuming w r Obscured, w mr Not obscured, w r Located at the polar line Above, there are the following polar constraints: Where (x) wr , y wr ) is w r The candidate pixel coordinates; for unoccluded point pairs (w l , w ml ), (a l , a ml ) and (a r , a mr They can be located using triangulation to calculate their corresponding three-dimensional coordinates W. l A l and A r ; The prior information about the human skeleton used is that the left and right arms should be of equal length; based on this prior information, the following equation can be written: in This refers to Euclidean distance, W r It is w r The corresponding three-dimensional coordinates, when (x) wr , y wr Then, triangulation can be used to obtain the value; by combining the two formulas above, the value of the occluded joint w can be obtained. r A reliable inference.

2. The method according to claim 1, characterized in that, The video acquisition process aims to ensure diversity in human posture, clothing, and background, while minimizing video footage with large camera movements to simplify the camera intrinsic parameter estimation.

3. The method according to claim 1, characterized in that, The human body 2D joint point detection uses a confidence score to distinguish the occlusion status of each joint point. When the confidence score is s... i When the value is less than a set threshold α, the key point is marked as occluded; When the number of occlusion points in a video segment exceeds the set threshold N, the video segment is judged as severely occluded and such videos are filtered out.

4. The method according to claim 1, characterized in that, The location of the poles in the image is estimated using the following method: Select some matching points with high confidence scores, obtain the lines connecting each set of matching points, and then calculate the intersection of any two lines to obtain the location of the candidate poles. Then calculate the median of the coordinates of all candidate poles to obtain a reliable estimate of the pole location.

5. A human body three-dimensional pose estimation device based on a catadioptric system employing the method described in any one of claims 1 to 4, characterized in that, include: The video acquisition module is used to acquire video, including data from the human body's catadioptric system. The preprocessing module is used to preprocess the acquired video, including human bounding box detection, human 2D joint point detection, and video segmentation. The human body 3D joint estimation module is used to establish a geometric model of a single-mirror stereo system. Using this geometric model, the 3D pose of the human body is estimated by triangulation based on the positions of the 2D joints of the human body inside and outside the mirror in the catadioptric system. Furthermore, by using epipolar geometry and constraints of the symmetry of human joint lengths, the positions of occluded nodes of the human body in the image are inferred, generating higher quality human body 3D joint estimation results.

6. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method described in any one of claims 1 to 4.

Images