Method and system for real-time reconstruction of human whole-body motion based on three-point sensing signals

Abstract

A human whole body action real-time reconstruction method and system based on three-point sensing signals, according to the collected head and double-hand three-point sensor signals, a pre-estimated waist joint and root joint are generated through a waist joint and root joint pre-estimation algorithm; the pre-estimated waist joint and root joint are combined with a human skeleton model, and an upper body reconstruction result is generated through a reverse motion algorithm; in addition, a lower body reconstruction network based on BiGRU is used to generate a lower body reconstruction result for an input action feature sequence, and a double-threshold-based action post-processing technology is used to further stabilize the lower body action, and finally the whole body action is obtained by merging and reconstructing. The application uses only the sensing signals of three body parts of the head and the hands to pre-estimate the waist joint and the root joint, increases the number of known joints, more effectively reconstructs the upper body action and the lower body action at the same time, and significantly improves the reconstruction effect of the lower body action.

Description

Technical Field

[0001] This invention relates to a technology in the field of computer graphics, specifically to a method and system for real-time reconstruction of full-body motion based on sensor signals from three body parts: the head, left hand, and right hand. Background Technology

[0002] In the field of computer graphics, full-body motion reconstruction refers to accurately reconstructing the full-body posture of a person in a real environment within a 3D virtual space, and using signals collected from the real environment (video, sensor signals, etc.) to drive the virtual human's movement in 3D space. Sensor-based methods, such as those using inertial sensors, are generally unaffected by occlusion or lighting conditions. However, due to the lack of direct position measurement, a considerable number of IMUs are typically required to complete full-body motion capture. For example, the commercial motion capture device Xsens requires 17 IMUs. Furthermore, due to measurement noise and wander errors, inertial sensors cannot accurately track human movements for extended periods. Optical positioning sensor-based methods are commonly used in the field of Virtual Reality (VR). For instance, HTC Vive's Lighthouse indoor positioning technology uses infrared lasers for positioning, simultaneously measuring the controller's rotation angle and position. However, generally speaking, to capture full-body movements relatively completely in VR, at least six sensors are needed for the head, hands, waist, and feet. With the promotion of consumer-grade VR devices, the use of fewer sensors has become a trend, such as using only sensors for the head and hands. However, the problem with this is the complete lack of sensor signals from the lower body, which makes the reconstruction of lower body movements worse and fails to meet actual needs. Summary of the Invention

[0003] This invention addresses the problems of incorrect mapping between upper and lower body movements and unstable, frequently shaky lower body movements when using only three-point sensor signals of the upper body in existing technologies. It proposes a real-time human body movement reconstruction method and system based on three-point sensor signals. This method uses only sensor signals from the head and hands to estimate the lumbar and root joints, increasing the number of known joints and more effectively reconstructing upper and lower body movements. In addition, it invented a lower body reconstruction network based on BiGRU and a motion post-processing technique based on dual thresholds, which significantly improves the lower body movement reconstruction effect.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a real-time human body motion reconstruction method based on three-point sensor signals. Based on the collected head and hand three-point sensor signals, the method generates estimated lumbar and root joints using a pre-estimation algorithm. These are then combined with a human skeletal model, and an upper body reconstruction result is generated using a reverse motion algorithm. Additionally, a lower body reconstruction result is generated from the input motion feature sequence using a BiGRU-based lower body reconstruction network. A dual-threshold-based motion post-processing technique is used to further stabilize the lower body motion, and finally, the reconstructed whole-body motion is obtained.

[0006] This invention relates to a real-time human body motion reconstruction system based on three-point sensor signals in a VR environment, implementing the above-mentioned method. The system includes: a three-point sensor signal acquisition module, a joint prediction module, an upper body motion reconstruction module, an input motion feature construction module, a BiGRU-based lower body motion reconstruction module, a dual-threshold-based motion post-processing module, and a data communication module, all connected to the three-point sensor signals from the head and hands. The joint prediction module uses the acquired head and hand three-point sensor signals to predict the poses of the lumbar joints and root joints in advance, and uses these predictions for the upper body motion reconstruction module and the input motion feature construction module. The upper body motion reconstruction module reconstructs the trunk and arm movements based on the sensor signals acquired by the sensor signal acquisition module and the lumbar joint pose obtained by the joint prediction module. The BiGRU-based lower body motion reconstruction module uses the motion feature sequence obtained by the input motion feature construction module to obtain the lower body reconstruction result through the BiGRU-based lower body reconstruction network. The motion post-processing module based on dual thresholds processes the lower body reconstruction result to further improve motion stability. Finally, the data communication module is used for communication between the upper body and lower body motion data to merge them into the whole body motion.

[0007] Technical effect

[0008] Compared with existing technologies, this invention uses a joint prediction algorithm to predict the poses of the lumbar and root joints in advance using the collected three-point sensor signals, increasing the number of known joints and serving as an important basis for the reconstruction of upper and lower body movements. It uses a BiGRU-based lower body reconstruction network, introducing future information as a constraint, enabling more accurate and stable reconstruction of lower body movements even when there are errors in the prediction of the lumbar and root joints. Furthermore, it employs a dual-threshold motion post-processing technique, using the distance between the reconstructed toe joints and the ground contact point when the previous contact state was true, in addition to the existing probability threshold for foot-to-ground contact predicted by neural networks, to determine the foot-to-ground contact situation. Then, the IK post-processing method is used to further stabilize the lower body movements, thus reconstructing more stable and natural lower body movements even when the neural network prediction is incorrect. Attached Figure Description

[0009] Figure 1 This is a flowchart of the method of the present invention;

[0010] Figure 2 This is a schematic diagram of the system structure of the present invention;

[0011] Figure 3 Reconstructing the network architecture diagram for lower body motion;

[0012] Figure 4 Comparison of the effects of different methods for reconstructing networks for lower body motion;

[0013] Figure 5 This is a diagram showing the effect of the invention in a VR environment. Detailed Implementation

[0014] like Figure 1 As shown in this embodiment, a real-time human body motion reconstruction method based on three-point sensor signals is used. According to the collected sensor signals, the estimated lumbar joints and root joints are generated by the lumbar joint and root joint estimation algorithm. These are combined with the human skeleton model, and the upper body reconstruction result is generated by the inverse motion algorithm. In addition, the lower body reconstruction result is generated by the input motion feature sequence through the BiGRU-based lower body reconstruction network. The lower body motion is further stabilized by the motion post-processing technology based on dual thresholds. Finally, the reconstructed whole body motion is obtained.

[0015] The sensor signals are obtained as follows: The position and rotation of three sensors (head and hands) in a three-dimensional world coordinate system are acquired using a VR headset and left and right hand controllers: p ht p lt p rt q ht q lt and q rt .

[0016] The aforementioned method for estimating the lumbar and root joints estimates the position and rotation of the lumbar and root joints in the world coordinate system based on the offset of the current frame of the head sensor relative to the previous frame in the collected sensor signals: p hip q hip p root and q root Specifically, it includes:

[0017] 1) Calculate the human body's frontal orientation, i.e., the human body's frontal orientation vector v f =q root *forward, where: q root Let be the rotation of the root joint, represented by a quaternion, and forward be the forward vector of the world coordinate system.

[0018] Generally, forward = (0, 0, 1).

[0019] 2) Based on the position offset v of the head sensor hoffset Turn your waist joint towards -v f The direction is adjusted to accommodate bending or squatting due to vertical head movement, and then the position of the lumbar joints is updated.

[0020] The aforementioned movement specifically refers to: the amount of lumbar joint movement Δback = |v hoffset . y |*c back *(p ht .yp root .y), where: |v hoffset .y| represents the vertical offset of the head sensor, p ht .y represents the height of the head sensor, p root .y represents the height of the root joint, c back This is a manually adjustable parameter, representing the direction towards -v f The proportion of directional movement.

[0021] In this embodiment, c back =0.4.

[0022] The update specifically refers to the position p′ of the lumbar joint. hip =p hip +(Δback*(-vf))*k ps , where: k ps This represents the body displacement coefficient.

[0023] In this embodiment, k ps =0.8.

[0024] 3) Based on the rotation quaternion q of the head sensor ht Calculate the rotational offset Δq of the lumbar joint and use a spherical interpolation function to find the relationship between the unit quaternion I and Δq based on the body rotation coefficient k. rs Perform interpolation; finally, update the lumbar joint rotation.

[0025] The rotational offset is specifically as follows: Where: q ht The rotation quaternion for the head sensor. q is the inverse of the head joint rotation quaternion. hip and Let represent the rotational quaternion of the lumbar joint and its inverse, respectively, and Δq be the rotational offset of the lumbar joint.

[0026] The interpolation is specifically: q = Slerp(I, Δq, k) rs), where: q is the final rotational offset of the lumbar joint, I is the unit quaternion, Δq is the rotational offset of the lumbar joint, k rs This is the body rotation coefficient.

[0027] In this embodiment, k rs =0.3.

[0028] The aforementioned update of lumbar joint rotation specifically refers to: the rotation q′ of the lumbar joint. hip =q*q hip .

[0029] 4) Project the lumbar joints onto the ground to update the position and orientation of the root joints: p root =(p hip .x, O, p hip .z), at the same time, project the orientation of the lumbar joint onto the ground. in Indicates the vector pointing towards the lumbar joint. The root joint orientation vector is represented, and the root joint rotation is finally obtained based on the up vector in the world coordinate system.

[0030] The aforementioned inverse kinematics algorithm includes: reconstructing the torso using an iterative heuristic FABRIK algorithm (Andreas Aristidou, Joan Lasenby. FABRIK: Afast, iteractive solver for the Inverse Kinematics problem[J]. Graphical Models. 2011, 73: 243-260.), that is, using the lumbar joint and head sensors as end effectors, solving the torso posture in one iteration using two processes, forward and backward, to minimize errors; and reconstructing the arms using an analytical Two-bone IK algorithm (A. Aristidou, J. Lasenby, Y. Chrysanthou, et al. Inverse Kinematics Techniques in Computer Graphics: A Survey[J]. Computer Graphics Forum. 2017, 00: 1-24.), that is, using the left and right hand sensors as end effectors respectively, and determining the angle between the arms using the law of cosines.

[0031] The input motion feature sequence specifically includes: 20 frames of past motion features, 1 frame of current motion features, and 5 frames of future motion features. Each motion feature includes: the position and rotation angle of the root joint, the head and hand joints relative to the root joint, the position difference between two frames, the rotation angle difference between two frames, and the height of the waist joint. The position and position difference are represented by 3D vectors, the rotation angle and rotation angle difference are represented by 6D vectors, and the height is represented by 1D vectors. Therefore, the length of each sequence is 20+1+5=26, and the dimension is 4×9+4×9+1=73.

[0032] like Figure 3 As shown, the BiGRU-based lower body motion reconstruction network includes: a first linear layer, two BiGRU layers, a second linear layer, and a pair of third linear layers for output. The input feature vector first passes through a first linear layer with a dimension of 512 and is activated using the ReLU activation function. Then it passes through two BiGRU layers, each with 512 hidden units. Next, it passes through a second linear layer with a dimension of 1024 and is also activated using the ReLU activation function. Finally, it passes through two third linear layers with a dimension of 256, which output the rotation angles of the eight joints of the lower body and the probability of the feet contacting the ground. The probability of the feet contacting the ground is activated using the sigmoid function. The output sequence length is 26, where the dimension of each sequence is 8×6+2×1=50. Finally, the 21st frame of the sequence is taken as the final output.

[0033] During training, the lower body motion reconstruction network uses the L2 loss function to minimize the difference between the predicted and actual values ​​of the lower body joint rotation angle, and the cross-entropy loss function is used to train the probability of both feet touching the ground.

[0034] The training set for the lower body motion reconstruction network uses the CMU and PFNN motion capture datasets, and its construction process is as follows:

[0035] 1) Manually select motion sequences, where the motion types are divided into movement, upper body movements, and standing / sitting, and downsample all motion data files to 45FPS;

[0036] 2) Extract the position, rotation, position difference between two frames, rotation difference between two frames, and height of the waist joint relative to the root joint in each frame, as well as the head and hand joints relative to the root joint;

[0037] 3) Extract the rotation of the eight joints of the lower body in each frame, and label each frame with whether the feet are in contact with the ground according to the speed of the foot joints. The speed is calculated by the distance between the foot joints in two adjacent frames. If it is less than the threshold u, it is considered that the foot joints are in contact with the ground. In this embodiment, u = 0.0107.

[0038] 4) Construct the input action feature sequence and the corresponding output sequence using a sliding window of 45 frames;

[0039] 5) Save the input dataset and the output dataset.

[0040] The lower body reconstruction results are preferably further evaluated using a dual-threshold method to determine whether the feet are in contact with the ground. Specifically, this means that the probability of both feet being in contact with the ground, as output by the neural network, is greater than a threshold s, and the toe joint position FK calculated from forward kinematics is also considered. toebase (R Joutput (t) is the ground contact point p calculated when the previous contact state is true. ik When the distance between them is less than d, the foot is considered to be in contact with the ground, specifically:

[0041] Criterion for determining left foot contact with ground in frame t Criterion for determining right foot contact with ground in frame t Where: s lfoot and s rfoot FK represents the probability of the left and right feet contacting the ground, as output by the lower body motion reconstruction network. ltoebase (R Joutput (t)) and FK rtoebase (R Joutput (t) represents the reconstructed joint positions of the left and right toes, p lik and p rik s and d represent the contact points of the left and right feet when they last made contact with the ground, respectively, and are manually set dual thresholds.

[0042] In this embodiment, s = 0.5 and d = 0.2.

[0043] If at time t, the state of contact between the foot and the ground changes from 0 to 1, then the contact point p... ik Set to the position of the toe joint at this moment, p ik The position of the foot remains unchanged and is always the target that the toe joints need to touch until the foot's contact with the ground changes from 1 to 0. During this period, the positions of the hip, knee, and ankle joints are solved using the Two-bone IK algorithm to maintain contact with the ground, thereby preventing the foot from slipping.

[0044] If at time t, the contact state between the foot and the ground changes from 1 to 0, indicating that the foot loses contact with the ground at that moment, to make the movement smoother and more natural, 15 frames are used as the interpolation coefficient t to interpolate the toe joint position FK calculated from the forward kinematics at that time. toebase (R Joutput (t) and contact point pik Interpolation is performed to determine the new toe joint position p toebase .

[0045] like Figure 2 As shown in this embodiment, a real-time human body motion reconstruction system based on three-point sensor signals in a VR environment, implementing the above-mentioned method, includes: a three-point sensor signal acquisition module, a joint prediction module, an upper body motion reconstruction module, an input motion feature construction module, a BiGRU-based lower body motion reconstruction module, a dual-threshold-based motion post-processing module, and a data communication module. The joint prediction module, through the acquired head and hand three-point sensor signals, pre-estimates the pose of the lumbar joint and root joint, and uses this information for the upper body motion reconstruction module and the input motion feature construction module. The upper body motion reconstruction module reconstructs the trunk and arm movements based on sensor signals acquired by the sensor signal acquisition module and the lumbar joint pose obtained by the joint prediction module. The lower body motion reconstruction module based on BiGRU uses the motion feature sequence obtained by the input motion feature construction module to obtain the lower body reconstruction result through the BiGRU-based lower body reconstruction network. The motion post-processing module based on dual thresholds processes the lower body reconstruction result to further improve motion stability. Finally, the data communication module is used for communication between upper body and lower body motion data to merge them into a whole-body motion.

[0046] In this embodiment, the three-point sensor signal acquisition module uses the HTC Vive Pro VR headset and left and right hand controllers, and utilizes SteamVR to acquire the position and rotation angle of the three sensors on the head and hands.

[0047] The joint prediction module uses a joint prediction algorithm to predict the lumbar joints and root joints, and is used in the upper body motion reconstruction module and the input motion feature construction module.

[0048] The upper body motion reconstruction module includes a trunk reconstruction subunit and a two-arm reconstruction subunit. The trunk reconstruction subunit and the two-arm reconstruction subunit reconstruct the movements of the trunk and the two arms through an inverse kinematics algorithm.

[0049] In this embodiment, the Unity3D engine is used for visualization, and the Final IK library is used to implement the inverse kinematics algorithm.

[0050] The input action feature construction module extracts the position and rotation of the root joint, the head and hand joints relative to the root joint, the position difference between two frames, the rotation difference between two frames, and the height of the waist joint to construct the input action feature sequence.

[0051] The BiGRU-based lower body motion reconstruction module uses the input motion feature sequence constructed by the input motion feature construction module to calculate the rotation parameters of the eight joints of the lower body and the probability of the feet contacting the ground through the built-in BiGRU-based lower body motion reconstruction network.

[0052] The dual-threshold-based motion post-processing module performs threshold judgment verification on the calculation results of the lower body motion reconstruction network unit to further improve the stability of the reconstructed lower body motion.

[0053] In this embodiment, a lower body motion reconstruction network based on BiGRU is implemented using PyTorch, and CUDA is used for GPU acceleration.

[0054] The data communication module enables the lower body motion reconstruction module to obtain the results of the head, hands, and root joints in the upper body motion reconstruction as input, and to merge the upper and lower body reconstruction results to obtain the whole body motion.

[0055] In this embodiment, a Socket is used to implement the data communication module.

[0056] Through specific experiments, on an i5-9400F CPU and an NVIDIA GeForce RTX 2060 graphics card, HTC Vive Pro was used to collect sensor signals and SteamVR was used as the driver. The front end was implemented using Unity3D, and the neural network was built using PyTorch. The experimental data obtained are as follows:

[0057] Based on the joint estimation method, the position and rotation error of the lumbar joint are obtained as follows:

[0058] Table 1. Estimated position and rotation error of the lumbar joint

[0059]

[0060] It can be seen that, on average, the estimated positional error of the lumbar joint in this embodiment is 7.83 cm, and the rotational error is 16.36°, which is within an acceptable range.

[0061] Secondly, a comparison of lower body motion reconstruction networks with other methods is presented:

[0062] Table 2. Average position and rotation errors of the eight joints in the lower body.

[0063]

[0064] As can be seen, the lower body motion reconstruction network in this embodiment has the smallest average position error and achieves good results in most movements, especially in sitting / standing movements, where the position and rotation errors are the smallest.

[0065] like Figure 4 As shown, the lower body motion reconstruction network in this embodiment, during actual operation, makes the lower body motion visually closer to real motion. First, for the full-body IK algorithm, the lower body motion is obviously stiff, presenting a straight posture. Second, for LoBSTr and unidirectional GRU, the angle of the legs opening is too large, and there is often a large swinging motion from side to side. Therefore, although LoBSTr has the smallest average rotation error in terms of lower body motion as shown in Table 2, the reconstructed lower body motion is extremely unstable visually during dynamic continuous processes. In contrast, the results obtained by the lower body motion reconstruction network in this embodiment are closer to real motion in both posture and stability.

[0066] like Figure 5 As shown, this embodiment can accurately reconstruct the full-body movements of the human body in a VR environment in real time, including various types of movements such as walking, squatting, waving, sitting / standing, and carrying objects. Its frame rate is 45 FPS and the latency per frame is 115 ms, which has practical applicability.

[0067] In summary, this invention more effectively reconstructs upper and lower body movements through a joint prediction method and its corresponding joint prediction module, a BiGRU-based lower body reconstruction network and its corresponding BiGRU-based lower body motion reconstruction module, as well as a dual-threshold-based motion post-processing technique and its corresponding dual-threshold-based motion post-processing module.

[0068] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A method for real-time reconstruction of whole-body motion based on three-point sensor signals, characterized in that, Based on the collected head and hand three-point sensor signals, the estimated lumbar and root joints are generated through the lumbar and root joint estimation algorithm; combined with the human skeleton model, the upper body reconstruction result is generated through the inverse motion algorithm; the lower body reconstruction result is generated through the input motion feature sequence by the BiGRU-based lower body reconstruction network, and the lower body motion is further stabilized by the motion post-processing technology based on dual thresholds. Finally, the reconstructed full-body motion is obtained by merging the reconstructed motion. The aforementioned method for estimating the lumbar and root joints estimates the position and rotation of the lumbar and root joints in the world coordinate system based on the offset of the current frame of the head sensor relative to the previous frame in the collected sensor signals. , , and Specifically, it includes: 1) Calculate the human body's frontal orientation, i.e., the human body's frontal orientation vector. ,in: Rotation of the root node is represented using quaternions. The front vector of the world coordinate system; 2) Based on the position offset of the head sensor Turn the waist joint towards The direction is adjusted to accommodate bending or squatting due to vertical head movement, and then the position of the lumbar joints is updated. 3) Based on the rotation quaternion of the head sensor Calculate the rotational offset of the lumbar joint And using spherical interpolation functions in unit quaternions and Between based on body rotation coefficient Perform interpolation; finally, update the lumbar joint rotation. 4) Project the lumbar joints onto the ground to update the position and orientation of the root joints: At the same time, project the orientation of the lumbar joint onto the ground. ,in Indicates the vector pointing towards the lumbar joint. This represents the root joint orientation vector, finally determined according to the world coordinate system. The vector yields the root joint rotation: ; The BiGRU-based lower body motion reconstruction network comprises: a first linear layer, two BiGRU layers, a second linear layer, and a pair of third linear layers for output, arranged sequentially. The input feature vector first passes through a first linear layer of dimension 512, activated using the ReLU activation function. Then it passes through two BiGRU layers, each with 512 hidden units. Next, it passes through a second linear layer of dimension 1024, also activated using the ReLU activation function. Finally, it passes through two third linear layers of dimension 256, with a dual-branch output showing the rotation angles of the eight joints of the lower body and the probability of the feet contacting the ground. The probability of the feet contacting the ground is activated using the sigmoid function. The output sequence length is 26, with each sequence having a dimension of 8×6+2×1=50. Finally, the 21st frame of the sequence is taken as the final output. The lower body reconstruction results use a dual-threshold method to determine whether the feet are in contact with the ground. Specifically, if the probability of both feet being in contact with the ground, as output by the neural network, is greater than a certain threshold, then... and the position of the toe joints calculated from forward kinematics Ground contact point calculated when the previous contact state was true The distance between them is less than At that time, it is considered that the foot is in contact with the ground, specifically: No. Frame left foot in contact with the ground criterion ;No. Criterion for Right Foot Contact with Ground ,in: and This represents the probability that the left and right feet will contact the ground, as output by the lower body motion reconstruction network. and This indicates the reconstructed joint positions of the left and right toes. and These represent the contact points of the left and right feet during their last contact with the ground, respectively. and Two thresholds set manually; If in At that moment, the state of contact between the foot and the ground changes from 0 to 1, and the contact point... Set the position of the toe joints at this moment. It will remain unchanged and continue to be the target that the toe joints need to touch until the state of the foot contacting the ground changes from 1 to 0. During this period, the Two-bone IK algorithm is used to solve the positions of the hip joint, knee joint and ankle joint to maintain the state of contact with the ground, thereby avoiding the foot from slipping. If in At a certain moment, the foot's contact state with the ground changes from 1 to 0, indicating that the foot loses contact with the ground at that instant. To make the movement smoother and more natural, 15 frames are used as the interpolation coefficient. The position of the toe joints at this moment, calculated using forward kinematics. With contact point Interpolation is performed to determine the new toe joint position. .

2. The method for real-time reconstruction of whole-body motion based on three-point sensor signals according to claim 1, characterized in that, The sensor signals are obtained by using a VR headset and left and right hand controllers to acquire the position and rotation of three sensors (head and hands) in a three-dimensional world coordinate system. , , , , and .

3. The method for real-time reconstruction of whole-body motion based on three-point sensor signals according to claim 1, characterized in that, The aforementioned movement specifically refers to the amount of movement of the lumbar joint. The vertical offset of the head sensor. The height of the head sensor, Height of the root joint A manually adjustable parameter represents the direction. The proportion of directional movement; The update mentioned above specifically refers to: ,in: Represents the body displacement coefficient; The rotational offset is specifically as follows: ,in: The rotation quaternion for the head sensor. The inverse of the quaternion for head joint rotation. and Let these represent the rotational quaternions of the lumbar joints and their inverses, respectively. This represents the rotational offset of the lumbar joint. The interpolation mentioned above specifically refers to: ,in: For unit quaternions, This represents the rotational offset of the lumbar joint. The coefficient of body rotation; The aforementioned update of lumbar joint rotation specifically refers to: .

4. The method for real-time reconstruction of whole-body motion based on three-point sensor signals according to claim 1, characterized in that, The input motion feature sequence specifically includes: 20 frames of past motion features, 1 frame of current motion features, and 5 frames of future motion features. Each motion feature includes: the position and rotation angle of the root joint and the head and hand joints relative to the root joint in that frame, the position difference between two frames, the rotation angle difference between two frames, and the height of the waist joint. The position and position difference are represented by 3D vectors, the rotation angle and rotation angle difference are represented by 6D vectors, and the height is represented by 1D vectors. Therefore, the length of each sequence is 20+1+5=26, and the dimension is 4×9+4×9+1=73.

5. The method for real-time reconstruction of whole-body motion based on three-point sensor signals according to claim 1, characterized in that, During training, the lower body motion reconstruction network uses the L2 loss function to minimize the difference between the predicted and actual values ​​of the lower body joint rotation angle, and the cross-entropy loss function is used to train the probability of both feet touching the ground. The training set for the lower body motion reconstruction network uses the CMU and PFNN motion capture datasets, and its construction process is as follows: 1) Manually select motion sequences, where the motion types are divided into movement, upper body movements, and standing / sitting, and downsample all motion data files to 45FPS; 2) Extract the position, rotation, position difference between two frames, rotation difference between two frames, and height of the waist joint relative to the root joint in each frame, as well as the head and hand joints relative to the root joint; 3) Extract the rotation of the 8 joints of the lower body in each frame, and label each frame with whether the feet are touching the ground based on the speed of the foot joints. The speed is calculated based on the distance between the foot joint positions of two adjacent frames. If the distance is less than a threshold... If this occurs, it is considered as the foot joint making contact with the ground; 4) Construct the input action feature sequence and the corresponding output sequence using a sliding window of 45 frames; 5) Save the input dataset and the output dataset.

6. A real-time human body motion reconstruction system based on three-point sensor signals in a VR environment, implementing the method of any one of claims 1 to 5, characterized in that, include: The system comprises a three-point sensor signal acquisition module, a joint prediction module, an upper body motion reconstruction module, an input motion feature construction module, a BiGRU-based lower body motion reconstruction module, a dual-threshold-based motion post-processing module, and a data communication module. Specifically: the joint prediction module uses the acquired head and hand three-point sensor signals to predict the poses of the lumbar and root joints in advance, which are then used by the upper body motion reconstruction module and the input motion feature construction module. The upper body motion reconstruction module reconstructs the trunk and arm movements based on the sensor signals acquired by the sensor signal acquisition module and the lumbar joint pose obtained by the joint prediction module. The BiGRU-based lower body motion reconstruction module uses the motion feature sequence obtained by the input motion feature construction module and passes it through a BiGRU-based lower body reconstruction network to obtain the lower body reconstruction result. The dual-threshold-based motion post-processing module processes the lower body reconstruction result to further improve motion stability. Finally, the data communication module is used for communication between the upper and lower body motion data to merge them into a full-body motion.

7. The real-time human body motion reconstruction system according to claim 6, characterized in that, The upper body motion reconstruction module includes a trunk reconstruction subunit and a double arm reconstruction subunit. The trunk reconstruction subunit and the double arm reconstruction subunit reconstruct the movements of the trunk and the double arms through an inverse kinematics algorithm. The input motion feature construction module extracts the position and rotation of the root joint, the head and hand joints relative to the root joint, the position difference between two frames, the rotation difference between two frames, and the height of the waist joint to construct the input motion feature sequence. The BiGRU-based lower body motion reconstruction module uses the input motion feature sequence constructed by the input motion feature construction module to calculate the rotation parameters of the eight joints of the lower body and the probability of the feet contacting the ground through the built-in BiGRU-based lower body motion reconstruction network. The dual-threshold-based motion post-processing module performs threshold judgment verification on the calculation results of the lower body motion reconstruction network unit to further improve the stability of the reconstructed lower body motion.

Images