A virtual-real mapping method and system based on monocular human pose reconstruction

Abstract

The application discloses a virtual-real mapping method and system based on monocular human posture reconstruction, wherein the virtual-real mapping method based on monocular human posture reconstruction comprises the following steps: S1, acquiring a user image sequence received by a monocular camera, and acquiring three-dimensional human key points of the user in a camera coordinate system based on a human posture estimation model; S2, receiving a real height value input by the user; S3, guiding and identifying a preset calibration action sequence performed by the user; S4, constructing a real physical world coordinate system based on the three-dimensional human key points, the real height value and the calibration action sequence, and calculating a transformation matrix from the camera coordinate system to the real physical world coordinate system; S5, calculating a transformation matrix from the real physical world coordinate system to a virtual space coordinate system based on the preset virtual space coordinate system; and S6, calculating and storing a final transformation matrix from the camera coordinate system to the virtual space coordinate system, so as to realize mapping of a human posture in the camera coordinate system to the virtual space.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a virtual-real mapping method and system based on monocular human pose reconstruction. Background Technology

[0002] With the development of applications such as motion-sensing games, virtual fitness, and immersive education, there is a need to accurately and in real-time map users' movements in real physical space to virtual space. The core of achieving this mapping lies in establishing a world coordinate system that is aligned with the direction of gravity, has a real physical scale, and is stable at its origin.

[0003] The existing technical solutions have the following limitations:

[0004] 1. Reliance on external equipment or complex calibration: For example, using multi-camera systems, inertial measurement units (IMUs), or pre-deployed markers (such as the calibration board used in CN119339337A). These methods increase system cost and deployment complexity, making them unsuitable for consumer applications.

[0005] 2. Reliance on temporal optimization and motion recovery structures (SLAM): such as WHAM and GVHMR, which require processing continuous video frames and optimizing the calculation of camera and human motion, have problems such as large computational overhead, poor real-time performance, and difficulty in initialization in fixed camera or static scenes, and cannot meet the requirements of interactive applications for real-time response of single frames.

[0006] 3. Lack of absolute physical scale and stable world orientation: Human pose estimation models based on monocular images (such as ROMP and VIBE) can directly output three-dimensional human body meshes, but their results are located in the camera coordinate system, the scale is blurred, and the coordinate system axis is independent of the physical world gravity direction and user orientation, so they cannot be directly used to drive virtual characters to move in a virtual environment with a consistent sense of direction.

[0007] Therefore, in view of the above-mentioned prior art, the present invention provides a virtual-real mapping method and system based on monocular human posture reconstruction. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by providing a virtual-real mapping method and system based on monocular human posture reconstruction.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A virtual-real mapping method based on monocular human pose reconstruction includes:

[0011] S1. Obtain the user image sequence received by the monocular camera, and obtain the three-dimensional human key points of the user in the camera coordinate system based on the human pose estimation model;

[0012] S2. Receive the user's actual height value;

[0013] S3. Guide and identify the preset calibration action sequence performed by the user;

[0014] S4. Construct a real physical world coordinate system based on 3D human key points, real height values, and calibrated action sequences, and calculate the transformation matrix from the camera coordinate system to the real physical world coordinate system;

[0015] S5. Based on the preset virtual space coordinate system, calculate the transformation matrix from the real physical world coordinate system to the virtual space coordinate system;

[0016] S6. Calculate and store the final transformation matrix from the camera coordinate system to the virtual space coordinate system to map the human pose in the camera coordinate system to the virtual space.

[0017] Furthermore, step S4 specifically includes:

[0018] S41. Receive the three-dimensional human height output by the human pose estimation model under the calibrated initial pose, calculate the global scale factor based on the received three-dimensional human height and the actual height value, and scale the coordinates of all three-dimensional human key points in the camera coordinate system according to the scale factor to obtain coordinates with physical scale.

[0019] S42. Obtain the coordinates of the left and right foot key points under the calibration starting pose, calculate the midpoint of the scaled left and right foot key point coordinates, and determine the calculated midpoint as the origin of the real physical world coordinate system;

[0020] S43. Extract the scaled coordinates of key points on the feet when the user stands in multiple different positions to form a three-dimensional point set of the feet; perform plane fitting on the three-dimensional point set of the feet to obtain the first direction estimate; use a neural network model to obtain the second direction estimate of the scene image; calculate the third direction estimate from the torso direction of the user's standing posture.

[0021] S44. Perform confidence-weighted fusion of the first direction estimate, the second direction estimate, and the third direction estimate to obtain the final vertical axis unit vector;

[0022] S45. Based on the key points when the user performs the indicative action in the calibration action sequence, calculate at least one forward candidate vector, project the forward candidate vector onto a plane orthogonal to the vertical axis unit vector, and normalize and fuse it to obtain the forward axis unit vector.

[0023] S46. Calculate the unit vector of the lateral axis according to the right-hand coordinate system rule;

[0024] S47. Based on the lateral axis unit vector, vertical axis unit vector, and forward axis unit vector, construct a rotation matrix. Based on the origin and camera position obtained in step S42, determine the translation vector to obtain the transformation matrix from the camera coordinate system to the real physical world coordinate system.

[0025] Furthermore, step S44 specifically includes:

[0026] A first confidence level based on the proportion of in-plane fitting points is assigned to the first direction estimation, a second confidence level based on the output probability of the neural network is assigned to the second direction estimation, and a third confidence level based on the confidence level of shoulder and hip keypoint detection is assigned to the third direction estimation. The confidence levels are weighted, fused, and normalized to obtain the final vertical axis unit vector.

[0027] Furthermore, the vertical axis unit vector is represented as:

[0028]

[0029] n g =w f n f +w net n net +w t n t

[0030]

[0031]

[0032]

[0033] Among them, Y R n represents the unit vector along the vertical axis; g w represents the direction vector after weighted fusion. f Indicated by c f Normalized weights; w net Indicated by c net Normalized weights; w t Indicated by c t Normalized weights; c f Indicates the first confidence level; c net Indicates the second confidence level; c t Indicates the third confidence level; n f Indicates the first direction estimate; n net Indicates the second direction estimate; n t This indicates a third-party estimate.

[0034] Furthermore, the indicative actions in step S45 include one or more of the following: a T-shaped posture, an arm pointing forward, and a walking motion between multiple stations.

[0035] Furthermore, when the indicative action is a T-shaped gesture, the forward candidate vector is represented as:

[0036] v T =Normalize(v shoulder × )

[0037] Among them, v T This represents the forward candidate vector for the T-pose; v shoulder The line vector representing the connection points of the user's left and right shoulders;

[0038] Or, when the indicative action is an arm pointing forward, the forward candidate vector is represented as:

[0039]

[0040] in, This represents the forward candidate vector when the arm is pointing forward; v arm This indicates that the key point of the hand points to the key point of the shoulder on the same side;

[0041] Alternatively, when the indicative action is a walking action between multiple stations, the forward candidate vector is represented as:

[0042]

[0043] Among them, v step Δp represents the forward candidate vector during walking; Δp represents the displacement vector of the pelvic key points between different stations.

[0044] Furthermore, the forward axis unit vector in step S45 is represented as:

[0045]

[0046] Among them, Z R Represents the forward axis unit vector; u i w represents the i-th forward candidate vector; i This indicates the quality score of the indicative action.

[0047] Furthermore, the lateral axis unit vector in step S46 is represented as:

[0048] X R =Z R ×Y R

[0049] Among them, X RThis represents the unit vector along the lateral axis.

[0050] Furthermore, the virtual spatial coordinate system preset in step S5 is a left-handed coordinate system.

[0051] Correspondingly, a virtual-real mapping system based on monocular human pose reconstruction is also provided, including:

[0052] The acquisition module is used to acquire the user image sequence received by the monocular camera and to acquire the three-dimensional human key points of the user in the camera coordinate system based on the human pose estimation model.

[0053] The receiving module is used to receive the user's actual height value;

[0054] The recognition module is used to guide and recognize the preset calibration action sequence performed by the user;

[0055] The first calculation module is used to construct a real physical world coordinate system based on 3D human key points, real height values, and calibrated action sequences, and to calculate the transformation matrix from the camera coordinate system to the real physical world coordinate system.

[0056] The second calculation module is used to calculate the transformation matrix from the real physical world coordinate system to the virtual space coordinate system based on the preset virtual space coordinate system.

[0057] The mapping module is used to calculate and store the final transformation matrix from the camera coordinate system to the virtual space coordinate system, thereby mapping the human pose in the camera coordinate system to the virtual space.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. Only a regular monocular RGB camera is required, without the need for additional hardware or complex algorithms such as SLAM, IMU, multiple cameras, calibration boards, etc., which greatly reduces the deployment threshold and cost;

[0060] 2. Users only need to input their height and follow simple instructions to complete a few natural movements (standing, walking to a few points, posing). The calibration can usually be completed in one minute. The calibration parameters are permanently valid and can be reused by multiple people.

[0061] 3. A multi-source information (foot plane, scene network, torso direction) fusion strategy is adopted to determine the vertical axis, and a multi-action (T-pose, pointing, walking) fusion strategy is adopted to determine the horizontal axis, which effectively overcomes the problem of noise or failure of single information source and improves the stability of the system in complex environments.

[0062] 4. Physical scale was restored by the user's height. The vertical axis and the user's forward axis, which are aligned with gravity, were determined by the fusion of the user's own movements and scene perception. This ensures that the user's forward movement, jumping and other actions in the real world are strictly consistent with the direction of movement in the virtual world, resulting in a strong sense of immersion. Attached Figure Description

[0063] Figure 1 This is a flowchart of a virtual-real mapping method based on monocular human pose reconstruction provided in Embodiment 1;

[0064] Figure 2 This is a schematic diagram illustrating the specific implementation of the real space coordinate system normal vector provided in Embodiment 1;

[0065] Figure 3 This is a schematic diagram illustrating the specific implementation of actions in the positive direction of the real spatial coordinate system provided in Embodiment 1. Detailed Implementation

[0066] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0067] The purpose of this invention is to address the shortcomings of existing technologies by providing a virtual-real mapping method and system based on monocular human posture reconstruction.

[0068] Example 1

[0069] This embodiment provides a virtual-real mapping method based on monocular human pose reconstruction, such as... Figure 1 As shown, it includes:

[0070] S1. Obtain the user image sequence received by the monocular camera, and obtain the three-dimensional human key points of the user in the camera coordinate system based on the human pose estimation model;

[0071] S2. Receive the user's actual height value;

[0072] S3. Guide and identify the preset calibration action sequence performed by the user;

[0073] S4. Construct a real physical world coordinate system based on 3D human key points, real height values, and calibrated action sequences, and calculate the transformation matrix from the camera coordinate system to the real physical world coordinate system;

[0074] S5. Based on the preset virtual space coordinate system, calculate the transformation matrix from the real physical world coordinate system to the virtual space coordinate system;

[0075] S6. Calculate and store the final transformation matrix from the camera coordinate system to the virtual space coordinate system to map the human pose in the camera coordinate system to the virtual space.

[0076] This embodiment can be applied to a home motion-sensing fitness system, wherein the system hardware includes a regular RGB monocular camera (1080p, 30 FPS) fixedly installed on the top of the TV, and a computing device (such as a game console or PC) running a virtual fitness application, wherein the computing device embeds a human posture estimation model, the camera's position remains fixed after installation, and the user's living room floor is the physical game space.

[0077] In step S1, the user image sequence received by the monocular camera is acquired, and the three-dimensional human key points of the user in the camera coordinate system are obtained based on the human pose estimation model.

[0078] The monocular camera continuously captures user images at a rate of 30 frames per second, generating a user image sequence I1, I2, ..., I t Each frame of image I t The input is fed into a pre-trained human pose estimation model, in which the ROMP model is used in this embodiment, but it is not limited to this.

[0079] The ROMP model outputs the parametric human model (SMPL) parameters corresponding to this frame, including pose parameters θ, shape parameters β, and root joint translation vectors γ. Based on these parameters, the 3D human keypoints J containing 24 standard joints can be obtained. 3d 3D human body key points J 3d The coordinates are defined in the camera coordinate system (C-frame). The camera coordinate system has its origin O at the optical center of the camera. C Z c The axis points in the shooting direction along the optical axis, X c Axis to the right, Y c The axis is downward. Coordinates in this coordinate system have no real physical scale; their units are arbitrary units defined by the model.

[0080] In step S2, the user's actual height value is received.

[0081] Before the initial calibration, the system prompts the user to enter the camera's field of view via screen, and then input their actual height value H using a controller or voice input. real For example, 175mm serves as an absolute reference for global physical scale reconstruction.

[0082] In step S3, the preset calibration action sequence performed by the user is guided and identified.

[0083] The system guides the user through a set of preset calibration action sequences using audiovisual instructions, and simultaneously records the image frames and 3D keypoint data corresponding to each action stage. The calibration action sequence includes:

[0084] The system guides the user to stand in a designated initial area within the camera's field of view, i.e., the initial position, and then maintains a natural upright posture for T1 seconds (e.g., 2 seconds). During this period, the system continuously captures multiple frames and calculates the average foot center position as the physical world origin O. R Accurate estimation is performed, and image frames and corresponding key points of the human body within this time period are recorded.

[0085] The system guides the user to move sequentially to K (K≥3, 3 in this embodiment) different preset positions (such as left front, right front, and directly front relative to the initial position), and to stand still at each position for T2 seconds (e.g., 1 second). The system acquires multiple frames of images at each position and records the multiple frames of images and key point data at each position, especially recording the key point coordinates of the left and right ankles or heels.

[0086] Guide the user back to the initial position and perform at least one of the following actions:

[0087] The user extends both arms horizontally to the sides of the body, forming a "T" shape, and holds the position for T3 seconds, thus creating the T-pose.

[0088] The user extends one arm (usually the right arm) horizontally forward and holds it.

[0089] Walk forward naturally 3-4 steps within the interactive area.

[0090] In step S4, a real physical world coordinate system is constructed based on 3D human key points, actual height values, and calibrated action sequences, and the transformation matrix from the camera coordinate system to the real physical world coordinate system is calculated.

[0091] like Figure 2-3 As shown, specifically:

[0092] S41. Receive the three-dimensional human height output by the human pose estimation model under the calibrated initial pose, calculate the global scale factor based on the received three-dimensional human height and the actual height value, and scale the coordinates of all three-dimensional human key points in the camera coordinate system according to the scale factor to obtain coordinates with physical scale.

[0093] Starting from the initial standing posture stage, select a frame with standard posture and obtain its human body key points. Calculate the coordinate difference vector between the highest point representing the top of the head (e.g., head key point) and the lowest point representing the bottom of the feet (e.g., the midpoint between the two ankle key points) in the camera coordinate system. Then, calculate the projection length of this difference vector in the vertical direction of the camera coordinate system, which is taken as the model's height H. smpl Finally, the global scale factor s is calculated and expressed as:

[0094]

[0095] The global scale factor s establishes the conversion relationship from the model's internal units to the real units.

[0096] For all subsequent calculations, the coordinates P of all 3D keypoints in the camera coordinate system involved will be used. C Multiplying each coordinate by the scale factor s yields intermediate coordinates with physical scale. The intermediate coordinates are still in the camera coordinate system, but they already have the correct physical scale.

[0097] S42. Obtain the coordinates of the left and right foot key points under the calibration starting pose, calculate the midpoint of the scaled left and right foot key point coordinates, and determine the calculated midpoint as the origin of the real physical world coordinate system;

[0098] Calculate the average center point of the midpoint coordinates of the left and right ankles in all frames recorded during the initial standing posture phase, and set this point as the origin O of the real physical world coordinate system (R-frame). R , is represented as:

[0099]

[0100] Where, p foot,L Indicates the midpoint coordinates of the left ankle; p foot,R This indicates the coordinates of the middle of the right ankle.

[0101] S43. Extract the scaled coordinates of key points on the feet when the user stands in multiple different positions to form a three-dimensional point set of the feet; perform plane fitting on the three-dimensional point set of the feet to obtain the first direction estimate; use a neural network model to obtain the second direction estimate of the scene image; calculate the third direction estimate from the torso direction of the user's standing posture.

[0102] A multi-source information fusion strategy is used to robustly estimate the direction perpendicular to the Y-axis, i.e. the opposite direction of gravity.

[0103] First direction estimation source: foot point cluster and plane fitting (multi-point sampling).

[0104] Assuming the user stops at K stations, and M frames are recorded at each station, calculate the foot midpoint p at the kth stop. k(Taking the left and right ankles or toes / heels), it is represented as:

[0105]

[0106] Where M represents the integer of the record; t represents the t-th frame; This represents the spatial coordinates of the left ankle at the k-th stop point in frame t; This represents the spatial coordinates of the right ankle at the k-th stop point in frame t.

[0107] The intermediate coordinates of all ankle (or heel) key points collected from K multi-point locations are used to form a three-dimensional point set P = p0, p1, ..., p K p0 is the initial origin.

[0108] The RANSAC algorithm for plane fitting is used to fit the point set P to obtain the optimal ground plane equation, and the unit normal vector n of the plane is extracted. p , is represented as:

[0109]

[0110] Since the fit is to the ground, its normal vector n p It usually points below the ground plane (i.e., in the direction of gravity), so the first estimated vector n is vertically upward. f for:

[0111]

[0112] Calculate this first estimated vector n f First confidence level c f The value is the proportion of interior points in the RANSAC fitting process (if c f If the value is less than 0.4, it is considered that the foot sampling is insufficient.

[0113] In this embodiment, if SVD is used to perform planar fitting on the point set P, then the minimum singular vector of V for the centered point matrix Q is n. f .

[0114] Second-direction estimation source: network plane estimation.

[0115] At the start of calibration or when the user moves away, acquire several frames of empty scene images without the user.

[0116] An empty scene image is input into a pre-trained scene plane segmentation neural network (such as Plane R-CNN), which outputs a pixel-level segmentation mask of the ground in the image. and its surface unit normal vector n in the camera coordinate system sSimilarly, taking the opposite direction yields the second estimated vector upwards: n net =-n s .

[0117] Calculate the second confidence level c of this second estimated vector. net The value is the average semantic confidence of the ground area predicted by the network.

[0118] In this embodiment, if SVD is used, the mesh mask is back-projected (if the network provides a depth or is combined with the root depth) to obtain several 3D planar points P. net And perform SVD to obtain n net (Related to confidence level).

[0119] Third direction estimation source: Human torso direction.

[0120] Calculate the torso vector in the user's initial standing posture. Select the pelvic center (midpoint of the left and right hip joints) and the chest center (midpoint of the left and right shoulder joints), and calculate the vector v connecting these two points. t , is represented as:

[0121]

[0122] in, This represents the center point of the three-dimensional joints of the left and right shoulders; This represents the center point of the three-dimensional joints of the left and right hip joints.

[0123] Based on the prior knowledge that the torso is roughly vertical when the human body is standing naturally, the normalized torso vector is used as the third estimated vector n of direction. t , is represented as:

[0124]

[0125] Calculate the third confidence level c of this third estimated vector. t The value is taken as the average of the two-dimensional detection confidence scores of the key shoulder and hip points involved (if c t If the value is less than 0.5, set it to 0.

[0126] S44. Perform confidence-weighted fusion of the first direction estimate, the second direction estimate, and the third direction estimate to obtain the final vertical axis unit vector;

[0127] Normalizing the three confidence levels yields the fusion weights for each source, expressed as:

[0128]

[0129]

[0130] Among them, w f Indicated by c f Normalized weights; w net Indicated by c net Normalized weights; w t Indicated by c t Normalized weights;

[0131] Calculate the direction vector n after weighted average g , is represented as:

[0132] n g =w f ·n f +w net ·n net +w t ·n t

[0133] For n g After normalization, the standard vertical axis (Y-axis) unit vector Y of the physical world coordinate system is finally obtained. R , is represented as:

[0134]

[0135] S45. Based on the key points when the user performs the indicative action in the calibration action sequence, calculate at least one forward candidate vector, project the forward candidate vector onto a plane orthogonal to the vertical axis unit vector, and normalize and fuse it to obtain the forward axis unit vector.

[0136] Based on the indicative gestures, the direction the user is facing is determined as the forward Z-axis; the indicative gestures include one or more of the following: T-shaped posture, forward arm pointing motion, and walking motion between multiple stations.

[0137] If the indicative movement is a T-shaped posture, calculate the difference vector of the midpoint coordinates of the left and right shoulder key points. , is represented as:

[0138]

[0139] Among them, S R Indicates the key point of the left shoulder; S L This indicates the key point on the right shoulder.

[0140] At this time, the forward candidate vector v T It is obtained by the cross product of the difference vector and the vertical axis, and is expressed as:

[0141] v T =Normalize(v shoulder × )

[0142] For the forward candidate vector v T Normalization is performed, and the expression is as follows:

[0143]

[0144] in, This represents the normalized forward candidate vector.

[0145] If the body posture is symmetrical, the direction may be reversed. This can be corrected by walking or using your fingers.

[0146] If the indicative action is a forward pointing motion of the arm, calculate the keypoint vector v from the shoulder on the pointing side to the wrist. arm , is represented as:

[0147]

[0148] in, Indicates the wrist joint; This indicates the shoulder joint.

[0149] At this time, the vector v arm Projecting this onto a horizontal plane orthogonal to the vertical axis yields the forward candidate vector, represented as:

[0150]

[0151] in, This represents the forward candidate vector when the arm is pointing forward; v arm This indicates that the key point of the hand points to the key point of the shoulder on the same side.

[0152] Forward candidate vectors Normalization is performed, and the expression is as follows:

[0153]

[0154] in, This represents the normalized forward candidate vector.

[0155] If the indicated action is walking between multiple stations (the most reliable method), take the key pelvic points (or root nodes) at position i before and after the start of walking, and calculate the displacement vector Δp, expressed as:

[0156]

[0157] in, Indicates the position after the walking session ends; This indicates the position before the start of walking.

[0158] At this time, the vector Projecting this onto a horizontal plane orthogonal to the vertical axis yields the forward candidate vector, represented as:

[0159]

[0160] Among them, v step Δp represents the forward candidate vector during walking; Δp represents the displacement vector of the pelvic key points between different stations.

[0161] Forward candidate vectors Normalization is performed, and the expression is as follows:

[0162]

[0163] in, This represents the normalized forward candidate vector.

[0164] Note: If the step size is too small (|Δp|<ε), it is considered invalid.

[0165] If there is only one reliable candidate direction (e.g., only a T-shaped pose was performed), then u is directly chosen. T Z, the unit vector along the forward axis R The initial direction.

[0166] If there are two reliable candidates (such as those who performed the T-pose or the arm pointing forward), the candidate can be selected as the initial direction by weighting the confidence level of each candidate or by choosing the one with higher quality.

[0167] In this embodiment, For any forward candidate vector, including , , and for each Calculate the mass fraction w i w i Including w T For shoulder-to-side confidence (shoulder keypoint confidence), w arm For the arm keypoint confidence + arm length > thresh, w step Given the step size plus smoothness (velocity stability), the forward axis unit vector is represented as:

[0168]

[0169] Among them, Z R Represents the forward axis unit vector; u i w represents the i-th forward candidate vector; i This indicates the quality score of the indicative action.

[0170] S46. Calculate the unit vector of the lateral axis according to the right-hand coordinate system rule;

[0171] Orthogonalization (ensuring Y is orthogonal to the subsequent X axis): When constructing Z, the Y component is removed from the Z projection and normalized to ensure orthogonality. For a strictly orthogonal basis, Gram-Schmidt can be used.

[0172] Given Z raw (From motion fusion), first orthogonalize it to Y. R Above, represented as:

[0173]

[0174]

[0175] According to the definition of a right-handed coordinate system, the lateral axis (X-axis) is determined by the cross product of the forward axis (Z-axis) and the vertical axis (Y-axis), expressed as:

[0176] X R =Z R ×Y R

[0177] Among them, X R This represents the unit vector along the lateral axis.

[0178] Thus, the orthogonal coordinate basis [X] of the physical world coordinate system R is established. R ,Y R Z R All confirmed.

[0179] S47. Based on the lateral axis unit vector, vertical axis unit vector, and forward axis unit vector, construct a rotation matrix. Based on the origin and camera position obtained in step S42, determine the translation vector to obtain the transformation matrix from the camera coordinate system to the real physical world coordinate system.

[0180] Construct a 3x3 rotation matrix R CR Their column vectors are respectively the X and Y coordinates of the physical world coordinate system. R ,Y R Z R The coordinates of the axis in the camera coordinate system are represented as follows:

[0181]

[0182] This matrix realizes the rotation from the R-frame to the C-frame.

[0183] Next, construct a 3x1 translation vector t. CR This vector is the origin O of the physical world coordinate system. R The coordinates in the camera coordinate system C are represented as follows:

[0184]

[0185] Combined into a complete 4x4 rigid body transformation matrix T CR (Using homogeneous coordinates), used to transform a point from the camera coordinate system to the physical world coordinate system, represented as:

[0186]

[0187] In step S5, based on the preset virtual space coordinate system, the transformation matrix from the real physical world coordinate system to the virtual space coordinate system is calculated.

[0188] Virtual spaces (such as the Unity game engine) typically use a left-handed coordinate system, with the Y-axis pointing upwards and the Z-axis pointing forwards.

[0189] Set the origin O of the virtual space coordinate system (V system) V With respect to the origin O of the physical world coordinate system R coincide.

[0190] Set the vertical axis (Y) of the virtual space V ) and the vertical axis of the physical world (Y R (The directions are consistent.)

[0191] To convert from a right-handed (R-type) to a left-handed (V-type) system while maintaining consistent forward definitions (Z... V With Z R (Both pointing forward), the X-axis (right side) of the R-frame needs to be reversed. This corresponds to a rotation around the Y-axis. R A 180-degree rotation of the axis is represented as:

[0192]

[0193] Assuming the origins of the physical coordinate system and the virtual coordinate system coincide, then the transformation matrix T is fixed. RV Represented as:

[0194]

[0195] The final mapping is represented as:

[0196]

[0197] By concatenating the above transformations, we obtain the complete transformation T, which directly maps the camera coordinate system to the virtual space coordinate system. C2V , is represented as:

[0198]

[0199] In step S6, the final transformation matrix from the camera coordinate system to the virtual space coordinate system is calculated and stored, thereby mapping the human pose in the camera coordinate system to the virtual space.

[0200] The calculated global scale factor s and the final transformation matrix T are used to... C2V Calibration data, including camera intrinsic parameters, is persistently stored in the device's configuration file or non-volatile memory. This calibration data is bound to a specific camera installation location. Once stored, it can be repeatedly loaded and used by different users or by the same user in different sessions, achieving "calibrate once, and it will be valid forever."

[0201] In subsequent motion-sensing interaction applications, the system executes the following in real time:

[0202] Note: Secure the camera in place, and have the user stand in the calibration area;

[0203] The system prompts users to remove their shoes (as suggested) and guides them to complete the action using voice / screen instructions.

[0204] Action 1: Stand still at the origin (position P0).

[0205] The user stands in the center of the screen and remains still for 2 seconds (to determine the origin O). R Preliminary position: midpoint of the foot).

[0206] The system records frame0.

[0207] Action 2: Dispersed sampling (stations P1, P2, P3, 3-4 points recommended).

[0208] The system guides the user to walk to three or four marker points in sequence (front left, front right, back left, etc.) and pause at each point for 1–2 seconds to complete a short stop.

[0209] Record several frames (e.g., 30 frames) at each point, calculate the foot center point (or ankle / toe average) at that point, and save it.

[0210] It is recommended that sampling points occupy a sufficient baseline in the interaction area (with a recommended spacing of ≥0.6 m between each point) to ensure geometric stability.

[0211] Action 3 (optional): T-pose / arm pointing / walking direction.

[0212] Used to determine the positive direction X R It is only executed when needed.

[0213] Network plane detection (parallel / optional):

[0214] During the calibration process, the system automatically runs a scene plane detection network (e.g., plane segmentation or monocular depth + plane segmentation model) to perform ground mask and normal prediction on the current image, obtaining a global plane estimate n. net With confidence level c net .

[0215] Integration and Confirmation:

[0216] The system obtains n by performing RANSAC plane fitting based on the foot point cluster. f and the network output n net Trunk direction n t The final Y is obtained by performing confidence-weighted fusion. R .

[0217] Display a virtual coordinate axis diagram on the screen and prompt the user "Does it look correct?" If the user confirms, save the calibration; otherwise, allow retrying single / multi-point calibration.

[0218] Parameter recommendations:

[0219] Number of stops K: 3–4;

[0220] Duration per point: 1–2 seconds (30–60 frames @ 30FPS);

[0221] Minimum distance between points: ≥0.6 m (recommended);

[0222] RANSAC inlier threshold: 0.02–0.05 m (2–5 cm);

[0223] Effective step size threshold ε: 0.05–0.1 m;

[0224] Fusion confidence threshold: If c f + c net If < 0.5, then decrease c. f Weights, preferably n t Prior knowledge of the scenario.

[0225] The method described in this embodiment, through an innovative process using the human body itself as a calibration object, utilizes only a regular camera and user height information, combined with simple movements, to achieve high-precision and robust calibration and alignment of the virtual and real spatial coordinate systems. This method eliminates the reliance on dedicated sensors, complex environmental setups, or time-consuming optimization processes, possessing outstanding advantages such as high real-time performance, ease of operation, low cost, and permanent use after a single calibration, greatly promoting the popularization and application of highly immersive motion-sensing interaction technology.

[0226] Example 2

[0227] This embodiment provides a virtual-real mapping system based on monocular human pose reconstruction, including:

[0228] The acquisition module is used to acquire the user image sequence received by the monocular camera and to acquire the three-dimensional human key points of the user in the camera coordinate system based on the human pose estimation model.

[0229] The receiving module is used to receive the user's actual height value;

[0230] The recognition module is used to guide and recognize the preset calibration action sequence performed by the user;

[0231] The first calculation module is used to construct a real physical world coordinate system based on 3D human key points, real height values, and calibrated action sequences, and to calculate the transformation matrix from the camera coordinate system to the real physical world coordinate system.

[0232] The second calculation module is used to calculate the transformation matrix from the real physical world coordinate system to the virtual space coordinate system based on the preset virtual space coordinate system.

[0233] The mapping module is used to calculate and store the final transformation matrix from the camera coordinate system to the virtual space coordinate system, thereby mapping the human pose in the camera coordinate system to the virtual space.

[0234] It should be noted that the virtual-real mapping system based on monocular human posture reconstruction provided in this embodiment is similar to that in Embodiment 1, and will not be described in detail here.

[0235] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A virtual-real mapping method based on monocular human pose reconstruction, characterized in that, include: S1. Obtain the user image sequence received by the monocular camera, and obtain the three-dimensional human key points of the user in the camera coordinate system based on the human pose estimation model; S2. Receive the user's actual height value; S3. Guide and identify the preset calibration action sequence performed by the user; S4. Construct a real physical world coordinate system based on 3D human key points, real height values, and calibrated action sequences, and calculate the transformation matrix from the camera coordinate system to the real physical world coordinate system; S5. Based on the preset virtual space coordinate system, calculate the transformation matrix from the real physical world coordinate system to the virtual space coordinate system; S6. Calculate and store the final transformation matrix from the camera coordinate system to the virtual space coordinate system to map the human pose in the camera coordinate system to the virtual space; Step S4 specifically involves: S41. Receive the three-dimensional human height output by the human pose estimation model under the calibrated initial pose, calculate the global scale factor based on the received three-dimensional human height and the actual height value, and scale the coordinates of all three-dimensional human key points in the camera coordinate system according to the scale factor to obtain coordinates with physical scale. S42. Obtain the coordinates of the left and right foot key points under the calibration starting pose, calculate the midpoint of the scaled left and right foot key point coordinates, and determine the calculated midpoint as the origin of the real physical world coordinate system; S43. Extract the scaled coordinates of key points on the feet when the user stands in multiple different positions to form a three-dimensional point set of the feet; perform plane fitting on the three-dimensional point set of the feet to obtain the first direction estimate; use a neural network model to obtain the second direction estimate of the scene image; calculate the third direction estimate from the torso direction of the user's standing posture. S44. Perform confidence-weighted fusion of the first direction estimate, the second direction estimate, and the third direction estimate to obtain the final vertical axis unit vector; S45. Based on the key points when the user performs the indicative action in the calibration action sequence, calculate at least one forward candidate vector, project the forward candidate vector onto a plane orthogonal to the vertical axis unit vector, and normalize and fuse it to obtain the forward axis unit vector. S46. Calculate the unit vector of the lateral axis according to the right-hand coordinate system rule; S47. Based on the lateral axis unit vector, vertical axis unit vector, and forward axis unit vector, construct a rotation matrix. Based on the origin and camera position obtained in step S42, determine the translation vector to obtain the transformation matrix from the camera coordinate system to the real physical world coordinate system.

2. The virtual-real mapping method based on monocular human pose reconstruction according to claim 1, characterized in that, Step S44 specifically involves: A first confidence level based on the proportion of in-plane fitting points is assigned to the first direction estimation, a second confidence level based on the output probability of the neural network is assigned to the second direction estimation, and a third confidence level based on the confidence level of shoulder and hip keypoint detection is assigned to the third direction estimation. The confidence levels are weighted, fused, and normalized to obtain the final vertical axis unit vector.

3. The virtual-real mapping method based on monocular human pose reconstruction according to claim 2, characterized in that, The vertical axis unit vector is represented as: ； n g =w f n f +w net n net +w t n t ； ； ； Among them, Y R n represents the unit vector along the vertical axis; g w represents the direction vector after weighted fusion. f Indicated by c f Normalized weights; w net Indicated by c net Normalized weights; w t Indicated by c t Normalized weights; c f Indicates the first confidence level; c net Indicates the second confidence level; c t Indicates the third confidence level; n f Indicates the first direction estimate; n net Indicates the second direction estimate; n t This indicates a third-party estimate.

4. The virtual-real mapping method based on monocular human pose reconstruction according to claim 3, characterized in that, The indicative actions in step S45 include one or more of the following: a T-shaped posture, an arm pointing forward, and walking between multiple stations.

5. The virtual-real mapping method based on monocular human pose reconstruction according to claim 4, characterized in that, When the indicated action is a T-shaped gesture, the forward candidate vector is represented as: v T =Normalize(v shoulder × ) Among them, v T This represents the forward candidate vector for the T-pose; v shoulder The line vector representing the connection points of the user's left and right shoulders; Or, when the indicative action is an arm pointing forward, the forward candidate vector is represented as: ； in, This represents the forward candidate vector when the arm is pointing forward; v arm This indicates that the key point of the hand points to the key point of the shoulder on the same side; Alternatively, when the indicative action is a walking action between multiple stations, the forward candidate vector is represented as: ； Among them, v step Δp represents the forward candidate vector during walking; Δp represents the displacement vector of the pelvic key points between different stations.

6. The virtual-real mapping method based on monocular human pose reconstruction according to claim 5, characterized in that, The forward axis unit vector in step S45 is represented as follows: ； Among them, Z R Represents the forward axis unit vector; u i w represents the i-th forward candidate vector; i This indicates the quality score of the indicative action.

7. A virtual-real mapping method based on monocular human pose reconstruction according to claim 6, characterized in that, The lateral axis unit vector in step S46 is represented as: X R =Z R ×Y R Among them, X R This represents the unit vector along the lateral axis.

8. The virtual-real mapping method based on monocular human pose reconstruction according to claim 1, characterized in that, The virtual space coordinate system preset in step S5 is a left-handed coordinate system.

9. A mapping system based on the virtual-real mapping method of monocular human pose reconstruction as described in any one of claims 1-8, characterized in that, include: The acquisition module is used to acquire the user image sequence received by the monocular camera and to acquire the three-dimensional human key points of the user in the camera coordinate system based on the human pose estimation model. The receiving module is used to receive the user's actual height value; The recognition module is used to guide and recognize the preset calibration action sequence performed by the user; The first calculation module is used to construct a real physical world coordinate system based on 3D human key points, real height values, and calibrated action sequences, and to calculate the transformation matrix from the camera coordinate system to the real physical world coordinate system. The second calculation module is used to calculate the transformation matrix from the real physical world coordinate system to the virtual space coordinate system based on the preset virtual space coordinate system. The mapping module is used to calculate and store the final transformation matrix from the camera coordinate system to the virtual space coordinate system, thereby mapping the human pose in the camera coordinate system to the virtual space.

Images