Pose information determination method and device, electronic equipment and computer medium
The problem of reduced accuracy of position estimation inertial motion capture technology is solved through pre-trained neural networks and optimization algorithms, and the accuracy and reliability of the inertial motion capture technology is achieved.
Patent Information
- Application Number
- CN202410984087.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In inertial motion capture technology, IMU sensors may generate errors when measuring motion, resulting in a decrease in accuracy of posture estimation.
The pre-trained pose information estimation neural network and physical information estimation network are used, combined with inertial sensor data and camera data, and the accuracy of pose information is improved by optimizing algorithms to process head speed, joint attitude, speed and ground contact state information.
By learning the nonlinear relationships in inertial data and utilizing physical laws, the cumulative errors in the prediction process are reduced, and the estimation accuracy and reliability of pose information are improved.
Smart Images

Figure CN118965256A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of human motion capture, and in particular to a method, device, electronic device and computer medium for determining posture information. Background Art
[0002] Inertial motion capture technology relies on IMU sensors to measure the acceleration and angular velocity of motion and calculate the position and attitude through integration operations. However, the sensor may produce errors during the measurement process. Over time, the errors gradually accumulate during the integration operation, which may reduce the accuracy of the pose estimation. Summary of the invention
[0003] The embodiments of the present application provide a method, device, electronic device and computer storage medium for determining posture information, which can improve the accuracy and reliability of posture information estimation. The above technical solution is as follows:
[0004] In a first aspect, an embodiment of the present application provides a method for determining posture information, the method comprising:
[0005] Obtaining relative inertia data of a plurality of preset joints of the target object relative to the root joint, and obtaining head velocity information of the target object;
[0006] The relative inertial data is processed by a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object;
[0007] Processing the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain joint velocity information and joint ground contact state information corresponding to the target object;
[0008] Based on a preset optimization algorithm, the head velocity information, the joint posture information, the joint velocity information and the joint ground contact state information are processed to obtain target posture information corresponding to the target object.
[0009] In a possible implementation, the joint position information corresponding to the target object includes first position information of each joint of all non-root joints of the target object relative to the root joint;
[0010] The joint posture information includes first posture information of each joint relative to the parent joint of the joint;
[0011] The joint speed information includes first speed information of each joint relative to the root joint;
[0012] The joint contact state information includes the contact probability corresponding to each joint, and the contact probability is used to indicate the probability of the corresponding joint contacting the ground.
[0013] In a possible implementation, the plurality of preset joints include the root joint, and the step of obtaining relative inertia data of the plurality of preset joints of the target object relative to the root joint includes:
[0014] Collecting global inertial data using inertial sensors worn at the plurality of preset joints of the target object, wherein the global inertial data includes global rotation information and global acceleration information corresponding to each of the plurality of preset joints;
[0015] Determining root rotation information and root acceleration information corresponding to the root joint from the global inertial data;
[0016] For each of the preset joints, determine the relative rotation information corresponding to the preset joint based on the root rotation information and the global rotation information corresponding to the preset joint;
[0017] Determine the relative acceleration information corresponding to the preset joint based on the root acceleration information and the global acceleration information corresponding to the preset joint, and then obtain a plurality of relative rotation information and a plurality of relative acceleration information corresponding to the plurality of preset joints;
[0018] The plurality of relative rotation information and the plurality of relative acceleration information are determined as relative inertia data of the plurality of preset joints of the target object relative to the root joint.
[0019] In a possible implementation manner, the step of obtaining the head velocity information of the target object includes:
[0020] Using a camera disposed on the head of the target object to collect a current image;
[0021] Acquire head relative rotation information and head relative acceleration information corresponding to the head of the target object at the moment of capturing the current image;
[0022] Determining a current position of the camera when acquiring the current image based on the current image, the head relative rotation information, and the head relative acceleration information;
[0023] Obtaining the historical position of the camera when capturing the previous frame of the current image;
[0024] The head speed information of the target object is determined according to the current position, the historical position and a preset acquisition period.
[0025] In a possible implementation, the posture information estimation neural network includes a fully convolutional neural network and a recurrent neural network. The pre-trained posture information estimation neural network processes the relative inertial data to obtain joint position information and joint posture information corresponding to the target object, including:
[0026] Inputting the relative inertia data into a pre-trained posture information estimation neural network, using the fully convolutional neural network to predict the positions of all non-root joints of the target object based on the relative inertia data, and obtaining corresponding joint position information of the target object output by the fully convolutional neural network;
[0027] The relative inertia data and the joint position information are input into the recurrent neural network, and the recurrent neural network is used to predict the postures of all the non-root joints to obtain the joint posture information corresponding to the target object.
[0028] In a possible implementation, the method further includes:
[0029] Get training data;
[0030] Inputting the initial posture information in the training data into the fully convolutional neural network to obtain a hidden state vector output by the fully convolutional neural network;
[0031] The hidden state vector is used as the initial hidden state of the initial recurrent neural network training to train the initial recurrent neural network.
[0032] In a possible implementation, the physical information estimation network includes a joint contact reasoning network and a joint linear velocity reasoning network;
[0033] The pre-trained physical information estimation network processes the joint position information and the relative inertia data to obtain the joint velocity information and joint contact state information corresponding to the target object, including:
[0034] Inputting the joint position information and the relative inertia data into a joint contact inference network in a pre-trained physical information estimation network to obtain joint contact state information corresponding to the target object;
[0035] The joint position information and the relative inertia data are input into the joint linear velocity inference network in the physical information estimation network to obtain the joint velocity information corresponding to the target object.
[0036] In a possible implementation, the head velocity information, the joint posture information, the joint velocity information and the joint contact state information are processed based on a preset optimization algorithm to obtain target posture information corresponding to the target object, including:
[0037] For each of the above joints, determining a constraint condition based on the ground contact probability corresponding to the above joint;
[0038] Using a preset optimization equation to process based on the constraint conditions, the head velocity information, the joint posture information and the joint velocity information, a target angular acceleration corresponding to the joint is obtained;
[0039] Determine the joint posture information corresponding to the joint according to the target angular acceleration, and then obtain multiple joint posture information corresponding to all the non-root joints;
[0040] The multiple joint posture information corresponding to all the non-root joints are determined as the target posture information corresponding to the target object.
[0041] In a possible implementation, determining the joint posture information corresponding to the joint according to the target angular acceleration includes:
[0042] Determine the target linear velocity corresponding to the joint according to the preset matrix and the target angular acceleration;
[0043] Accumulating the target linear velocity to obtain second position information corresponding to the joint;
[0044] Accumulating the target angular acceleration to obtain the target angular velocity corresponding to the joint;
[0045] Accumulating the target angular velocity to obtain second posture information corresponding to the joint;
[0046] The second position information and the second posture information are determined as joint posture information corresponding to the joint.
[0047] In a second aspect, an embodiment of the present application provides a device for determining posture information, the device comprising:
[0048] A first acquisition module is used to acquire relative inertia data of a plurality of preset joints of a target object relative to a root joint, and to acquire head velocity information of the target object;
[0049] A first processing module is used to process the relative inertial data through a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object;
[0050] A second processing module is used to process the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain joint velocity information and joint ground contact state information corresponding to the target object;
[0051] The first determination module is used to process the head velocity information, the joint posture information, the joint velocity information and the joint ground contact state information based on a preset optimization algorithm to obtain the target posture information corresponding to the target object.
[0052] In a possible implementation, the joint position information corresponding to the target object includes first position information of each joint of all non-root joints of the target object relative to the root joint;
[0053] The joint posture information includes the first posture information of each joint relative to the parent joint of the joint;
[0054] The joint speed information includes first speed information of each joint relative to the root joint;
[0055] The joint contact state information includes the contact probability corresponding to each joint, and the contact probability is used to indicate the probability of the corresponding joint contacting the ground.
[0056] In a possible implementation, the plurality of preset joints include the root joint, and the acquisition module includes:
[0057] A first acquisition unit, used for acquiring relative inertia data of a plurality of preset joints of the target object relative to the root joint;
[0058] A first acquisition unit is used to acquire global inertial data using inertial sensors worn at the plurality of preset joints of the target object, wherein the global inertial data includes global rotation information and global acceleration information corresponding to each of the plurality of preset joints;
[0059] A first determining unit, configured to determine root rotation information and root acceleration information corresponding to the root joint from the global inertial data;
[0060] A second determining unit is used to determine, for each of the preset joints, relative rotation information corresponding to the preset joint based on the root rotation information and the global rotation information corresponding to the preset joint;
[0061] A third determining unit is used to determine the relative acceleration information corresponding to the preset joint based on the root acceleration information and the global acceleration information corresponding to the preset joint, and then obtain a plurality of relative rotation information and a plurality of relative acceleration information corresponding to the plurality of preset joints;
[0062] The fourth determining unit is used to determine the plurality of relative rotation information and the plurality of relative acceleration information as relative inertia data of the plurality of preset joints of the target object relative to the root joint.
[0063] In a possible implementation, the acquisition module further includes:
[0064] A second acquisition unit, used to acquire a current image using a camera disposed on the head of the target object;
[0065] A second acquisition unit is used to acquire head relative rotation information and head relative acceleration information corresponding to the head of the target object at the moment of acquiring the current image;
[0066] a fifth determining unit, configured to determine a current position of the camera when acquiring the current image based on the current image, the head relative rotation information, and the head relative acceleration information;
[0067] A third acquisition unit is used to acquire the historical position of the camera when capturing the previous frame of the current image;
[0068] The sixth determining unit is used to determine the head speed information of the target object according to the current position, the historical position and a preset collection period.
[0069] In a possible implementation, the posture information estimation neural network includes a fully convolutional neural network and a recurrent neural network, and the first processing module includes:
[0070] A first prediction unit is used to input the relative inertia data into a pre-trained posture information estimation neural network, and use the fully convolutional neural network to predict the positions of all non-root joints of the target object according to the relative inertia data, so as to obtain corresponding joint position information of the target object output by the fully convolutional neural network;
[0071] The second prediction unit is used to input the above relative inertia data and the above joint position information into the above recurrent neural network, and use the above recurrent neural network to predict the postures of all the above non-root joints to obtain the joint posture information corresponding to the above target object.
[0072] In a possible implementation, the above-mentioned posture determination device further includes:
[0073] A second acquisition module is used to acquire training data;
[0074] A second determination module is used to input the initial posture information in the training data into the fully convolutional neural network to obtain a hidden state vector output by the fully convolutional neural network;
[0075] The training module is used to use the hidden state vector as the initial hidden state of the initial recurrent neural network training to train the initial recurrent neural network.
[0076] In a possible implementation, the physical information estimation network includes a joint contact reasoning network and a joint linear velocity reasoning network;
[0077] In a possible implementation, the second processing module includes:
[0078] a seventh determination unit, configured to input the joint position information and the relative inertia data into a joint contact inference network in a pre-trained physical information estimation network to obtain joint contact state information corresponding to the target object;
[0079] The eighth determination unit is used to input the above-mentioned joint position information and the above-mentioned relative inertia data into the joint linear velocity inference network in the above-mentioned physical information estimation network to obtain the joint velocity information corresponding to the above-mentioned target object.
[0080] In a possible implementation, the first determining module includes:
[0081] a ninth determining unit, configured to determine, for each of the joints, a constraint condition based on a ground contact probability corresponding to the joint;
[0082] A processing unit, configured to use a preset optimization equation to perform processing based on the constraint conditions, the head velocity information, the joint posture information, and the joint velocity information to obtain a target angular acceleration corresponding to the joint;
[0083] a tenth determining unit, configured to determine the joint posture information corresponding to the joint according to the target angular acceleration, and further obtain multiple joint posture information corresponding to all the non-root joints;
[0084] The eleventh determination unit is used to determine the multiple joint posture information corresponding to all the non-root joints as the target posture information corresponding to the target object.
[0085] In a possible implementation manner, the tenth determining unit includes:
[0086] The first determination subunit is used to determine the joint posture information corresponding to the joint according to the target angular acceleration, including:
[0087] A second determination subunit is used to determine a target linear velocity corresponding to the joint according to a preset matrix and the target angular acceleration;
[0088] A first accumulation subunit is used to accumulate the target linear velocity to obtain second position information corresponding to the joint;
[0089] A second accumulation subunit is used to accumulate the target angular acceleration to obtain a target angular velocity corresponding to the joint;
[0090] A third accumulation subunit is used to accumulate the target angular velocity to obtain second posture information corresponding to the joint;
[0091] The third determining subunit is used to determine the second position information and the second posture information as the joint posture information corresponding to the joint.
[0092] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory;
[0093] The above-mentioned memory stores a computer program, and the above-mentioned computer program is suitable for being loaded by the above-mentioned processor and executing the steps of the method provided by the first aspect of the embodiment of the present application or any possible implementation method of the first aspect.
[0094] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the method provided by the first aspect of the embodiment of the present application or any possible implementation method of the first aspect.
[0095] The embodiment of the present application obtains the relative inertia data of multiple preset joints of the target object relative to the root joint, and obtains the head velocity information of the target object; processes the relative inertia data through a pre-trained posture information estimation neural network to obtain the joint position information and joint posture information corresponding to the target object; processes the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain the joint velocity information and joint contact state information corresponding to the target object; based on a preset optimization algorithm, the head velocity information, the joint posture information, the joint velocity information and the joint contact state information are processed to obtain the target posture information corresponding to the target object. In this scheme, the posture information estimation neural network is used to learn complex nonlinear relationships from the relative inertia data, and then the physical information estimation network is used to optimize the information inferred from the posture information estimation neural network, so as to supplement and correct the prediction of the posture information estimation neural network using physical laws, and finally the preset optimization algorithm is used to reduce the cumulative error in the above prediction process, thereby improving the accuracy and reliability of posture information estimation. BRIEF DESCRIPTION OF THE DRAWINGS
[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0097] Figure 1 A schematic diagram of wearing a suitable hardware system provided by an exemplary embodiment of the present application;
[0098] Figure 2 A flowchart of a method for determining posture information provided by an exemplary embodiment of the present application;
[0099] Figure 3 A structural schematic diagram of a posture determination system provided for an exemplary embodiment of the present application;
[0100] Figure 4 A schematic diagram of the structure of a posture information estimation neural network provided by an exemplary embodiment of the present application;
[0101] Figure 5 A schematic diagram of a training process of a posture information estimation neural network provided for an exemplary embodiment of the present application;
[0102] Figure 6 A schematic diagram of the structure of a physical information estimation network provided by an exemplary embodiment of the present application;
[0103] Figure 7 A schematic diagram of the structure of a device for determining posture information provided by an exemplary embodiment of the present application;
[0104] Figure 8 A schematic structural diagram of an electronic device provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0105] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0106] The terms "first", "second", "third", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.
[0107] Please refer to the following Figure 1 , which exemplarily shows a schematic diagram of wearing a suitable hardware system provided by an embodiment of the present application. Figure 1 As shown, the system includes: a plurality of inertial sensors (such as Figure 1 Inertial sensor 100, inertial sensor 102, inertial sensor 103, inertial sensor 104 and inertial sensor 105), camera 110 arranged on the head of the target object, control device and display device, etc. The multiple inertial sensors, camera 110, control device and display device are connected through a network, for example, through a wired or wireless network connection (such as Bluetooth, WIFI, etc.).
[0108] Among them, there can be 6 inertial sensors, which can be specifically set in devices such as handles, motion capture gloves, and controllers. For example, 6 inertial sensors are set in 2 handles, 2 lower limb controllers, 1 head controller and 1 waist controller, where 2 handles (respectively worn on the left wrist and right wrist of the target object, 2 lower limb controllers (both equipped with inertial sensors) are respectively worn on the right calf and left calf, 1 head controller is worn on the head of the target object, and the waist controller is worn on the waist (or pelvis) of the target user. The above 6 inertial sensors are used to collect inertial data of the corresponding parts.
[0109] In some embodiments, the camera 110 (which may also be a mobile phone with a camera) disposed on the head of the target object can be used to determine the position of the head and thus obtain the speed of the head.
[0110] The control device is used to execute the following posture information determination method: obtaining relative inertia data of multiple preset joints of the target object relative to the root joint, and obtaining head velocity information of the target object; processing the relative inertia data through a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object; processing the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain joint velocity information and joint contact state information corresponding to the target object; based on a preset optimization algorithm, processing the head velocity information, the joint posture information, the joint velocity information and the joint contact state information to obtain target posture information corresponding to the target object.
[0111] In some embodiments, the above-mentioned display device can be used to display related pages (such as virtual reality (VR) game pages), and receive target posture information corresponding to the target object determined by the control device, and render the scene corresponding to the target object to achieve synchronization between the target object and the corresponding virtual character or perspective in the related page, providing a more immersive experience for the target object.
[0112] In some embodiments, the above-mentioned display device, control device, camera and head sensor can be set in the same device, for example, a head-mounted display device.
[0113] An exemplary embodiment of the present application provides a method for determining posture information. The method for determining posture information can be applied to the above terminal device. Figure 2 , which exemplarily shows a flow chart of a method for determining posture information provided by an embodiment of the present application. Figure 2 As shown, the posture information determination method includes the following S21-S24:
[0114] S21, obtaining relative inertia data of multiple preset joints of the target object relative to the root joint, and obtaining head velocity information of the target object.
[0115] The target object may be a person or a dummy model used for motion capture.
[0116] In some embodiments, the above-mentioned multiple preset joints can be at least part of all the joints of the human body. For example, the number of multiple preset joints is 6, and the 6 preset joints are: the joint corresponding to the head node, the joint corresponding to the left wrist node, the joint corresponding to the right wrist node, the joint corresponding to the pelvic node, the joint corresponding to the left ankle node, and the joint corresponding to the right ankle node.
[0117] Optionally, taking the Skinned Multi-Person Linear (SMPL) model as an example, the SMPL model includes 24 joint points: a head node, a neck node, a left collar node, a right collar node, a left shoulder node, a right shoulder node, three cervical vertebrae nodes, a pelvic node, a left elbow node, a right elbow node, a left wrist node, a right wrist node, a left hand node, a right hand node, a left hip node, a right hip node, a left knee node, a right knee node, a left ankle node, a right ankle node, a left foot node, and a right foot node. Among them, the root node (root) is a hypothetical node located in the pelvis, which is used to represent the position of the human body in the world coordinate system. The joints corresponding to the nodes in the above model can be rotated. In this model, the direction of the joint can be represented by the six-degree-of-freedom (6DOF) rotation vector of the coordinate system of each joint.
[0118] In some embodiments, the plurality of preset joints include the root joint.
[0119] S22. Processing the relative inertial data through a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object.
[0120] In some embodiments, the above-mentioned posture information estimation neural network includes a fully convolutional neural network and a recurrent neural network.
[0121] In the embodiment of the present application, the above two-stage neural network can be used to obtain the joint posture information corresponding to the target object. Specifically, first, in the first stage (full convolutional neural network), the prediction of the joint position is used as an intermediate task to obtain the joint position information corresponding to the target object, and then in the second stage (recurrent neural network), the joint angle is solved to obtain the joint posture information corresponding to the target object.
[0122] In some embodiments, the joint position information corresponding to the target object includes first position information of each joint of all non-root joints of the target object relative to the root joint. All non-root joints may be 23 joints corresponding to 23 nodes other than the root node among the 24 joint points.
[0123] In some embodiments, the joint posture information includes first posture information of each joint relative to a parent joint of the joint.
[0124] Among them, the parent joint refers to the joint at a higher position in the kinematic chain, and its movement will affect the child joints below it and the entire kinematic chain. The child joint refers to the joint affected by the movement of the parent joint, which is usually located below or around the parent joint.
[0125] For example, between the human shoulder joint (such as the joint corresponding to the left shoulder node above) and the elbow joint (such as the joint corresponding to the left elbow node above), the shoulder joint is the parent joint because it allows the upper arm to move in multiple axes, thereby affecting the movement of the elbow joint. The elbow joint is the child joint of the shoulder joint because its movement is directly affected by the movement of the shoulder joint.
[0126] S23, processing the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain joint velocity information and joint ground contact state information corresponding to the target object.
[0127] In some embodiments, the physical information estimation network includes a joint contact inference network and a joint linear velocity inference network.
[0128] In some embodiments, the joint velocity information includes first velocity information of each joint relative to the root joint.
[0129] In some embodiments, the joint contact state information includes the contact probability corresponding to each of the joints, and the contact probability is used to indicate the probability of the corresponding joint contacting the ground.
[0130] S24. Based on a preset optimization algorithm, the head velocity information, the joint posture information, the joint velocity information and the joint ground contact state information are processed to obtain target posture information corresponding to the target object.
[0131] In some embodiments, the above-mentioned preset optimization algorithm may include a proportional-derivative (PD) controller algorithm, specifically including an angle PD controller algorithm and a velocity PD controller algorithm, which are used to establish an association between the desired angular acceleration and the joint posture information, and to establish an association between the desired angular acceleration and the joint velocity information output by the posture information estimation neural network, and to perform real-time smoothing filtering in combination with the head velocity information and the joint contact status information to obtain the target posture information corresponding to the above-mentioned target object.
[0132] Figure 3 A structural diagram of a posture determination system provided by an exemplary embodiment of the present application is shown in FIG. Figure 3 As shown, the posture determination system includes a posture information estimation neural network, a physical information estimation neural network, a physical optimizer and a camera. The system is used to implement the posture information determination method in the present application, and regresses the necessary information required for human posture and physical optimization in a multi-stage manner. In the system input stage, the relative inertia data is respectively input into the posture information estimation neural network and the physical information estimation neural network, and the posture information estimation neural network outputs the joint position information and joint posture information corresponding to the target object; and the joint position information is input into the physical information estimation network, and the physical information estimation network is used to determine the joint speed information and joint contact state information based on the joint position information and relative inertia information; and then the joint position information, joint posture information, joint speed information, joint contact information are combined with the head speed information provided by the camera. Input into the physical optimizer based on the preset optimization algorithm for optimization, and the target posture information corresponding to the target object is obtained.
[0133] The embodiment of the present application obtains the relative inertia data of multiple preset joints of the target object relative to the root joint, and obtains the head velocity information of the target object; processes the relative inertia data through a pre-trained posture information estimation neural network to obtain the joint position information and joint posture information corresponding to the target object; processes the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain the joint velocity information and joint contact state information corresponding to the target object; based on a preset optimization algorithm, the head velocity information, the joint posture information, the joint velocity information and the joint contact state information are processed to obtain the target posture information corresponding to the target object. In this scheme, the posture information estimation neural network is used to learn complex nonlinear relationships from the relative inertia data, and then the physical information estimation network is used to optimize the information inferred from the posture information estimation neural network, so as to supplement and correct the prediction of the posture information estimation neural network using physical laws, and finally the preset optimization algorithm is used to reduce the cumulative error in the above prediction process, thereby improving the accuracy and reliability of posture information estimation.
[0134] In addition, in the embodiment of the present application, only data from 6 inertial sensors are needed to infer and output the whole body movements of the target object, and the output values are ergonomic.
[0135] In some embodiments, in S21, relative inertia data of a plurality of preset joints of the target object relative to the root joint is obtained, including S210-S214:
[0136] S210. Collect global inertial data using inertial sensors worn at the plurality of preset joints of the target object, wherein the global inertial data includes global rotation information and global acceleration information corresponding to each of the plurality of preset joints.
[0137] In some embodiments, the global inertial data is inertial data relative to the world coordinate system. The global inertial data can be expressed as: raw =[a0,a1,a2,a3,a4,a5,r0,r1,r2,r3,r4,r5], wherein a0,a1,a2,a3,a4,a5 represent the global acceleration information corresponding to multiple preset joints, and r0,r1,r2,r3,r4,r5 represent the global rotation information corresponding to multiple preset joints.
[0138] S211. Determine the root rotation information and root acceleration information corresponding to the root joint from the global inertial data.
[0139] The root rotation information is included in the plurality of global rotation information, and the root acceleration information is included in the plurality of global acceleration information. The global rotation information measured by the inertial sensor worn at the root joint in the global inertial data is determined as the root rotation information, and the global acceleration information measured by the inertial sensor worn at the root joint in the global inertial data is determined as the root acceleration information.
[0140] S212: For each of the preset joints, determine relative rotation information corresponding to the preset joint based on the root rotation information and the global rotation information corresponding to the preset joint.
[0141] In some embodiments, in S212, the relative rotation information corresponding to the preset joint is determined based on the root rotation information and the global rotation information corresponding to the preset joint, including: multiplying the transposed matrix of the global rotation information corresponding to the preset joint and the root rotation information to obtain the relative rotation information corresponding to the preset joint.
[0142] Optionally, the relative rotation information corresponding to the i-th preset joint can be expressed as:
[0143] r i_normalized =(r root ) T *r i
[0144] Among them, r root Represents the root rotation information corresponding to the root joint, (r root ) T The transposed matrix representing the root rotation information, r i Represents the global rotation information corresponding to the i-th preset joint.
[0145] S213, determining relative acceleration information corresponding to the preset joint based on the root acceleration information and the global acceleration information corresponding to the preset joint, and then obtaining multiple relative rotation information and multiple relative acceleration information corresponding to the multiple preset joints.
[0146] In some embodiments, in S213, the relative acceleration information corresponding to the preset joint is determined based on the root acceleration information and the global acceleration information corresponding to the preset joint, including: multiplying the transposed matrices of the global acceleration information corresponding to the preset joint and the root acceleration information, and subtracting the root acceleration information from the multiplication result to obtain the relative acceleration information corresponding to the preset joint.
[0147] Optionally, the relative acceleration information corresponding to the i-th preset joint can be expressed as:
[0148] a i_normalized=(a root ) T *a i -a root
[0149] Among them, a root Represents the root acceleration information corresponding to the root joint, (a root ) T The transposed matrix representing the root acceleration information, r i Represents the global rotation information corresponding to the i-th preset joint.
[0150] S214: Determine the plurality of relative rotation information and the plurality of relative acceleration information as relative inertia data of the plurality of preset joints of the target object relative to the root joint.
[0151] Optionally, the relative inertial data is sensor data that has been normalized to a reference frame of the root joint, and the relative inertial data can be expressed as:
[0152] x nocmalized =[a 0_normalized , a 1_normalized , a 2_normalized , a 3_normalized , a 4_normalized , a 5_normalized ,
[0153] r 0_normalized , r 1_normalized , r 2_normalized , r 3_normalized , r 4_normalized , r 5_normalized ]
[0154] Among them, a 0_normalized ~a 5_normalized is multiple relative acceleration information, r 0_normalized ~r 5_normalized It is multiple relative rotation information.
[0155] In the embodiment of the present application, by calculating the movement of each joint relative to the root joint, the dynamic behavior of the joint can be analyzed more accurately, and normalizing the sensor data to the same reference frame for processing can simplify the calculation process and effectively improve data processing efficiency.
[0156] In some embodiments, in S21, the step of obtaining the head speed information of the target object includes S215-S219:
[0157] S215: using a camera disposed on the head of the target object to capture a current image.
[0158] The camera on the head of the target object may be any one of a monocular camera, a binocular camera, and a red green blue-depth (RGB-D) camera.
[0159] S216: Acquire head relative rotation information and head relative acceleration information corresponding to the head of the target object at the moment of capturing the current image.
[0160] The head relative rotation information and head relative acceleration information corresponding to the head of the target object are included in the above relative inertia data.
[0161] S217: Determine the current position of the camera when acquiring the current image based on the current image, the head relative rotation information, and the head relative acceleration information.
[0162] Optionally, the position tracking of a monocular camera can be achieved using a hybrid tracking mode of an advanced visual odometry and Oriented FAST and Rotated BRIEF Simultaneous Localization and Mapping (ORB-SLAM3) system. ORB-SLAM3 can support monocular, binocular, and RGB-D cameras.
[0163] Specifically, the current image, head relative rotation information, and head relative acceleration information are input into the ORB-SLAM3 system, and the current position P of the camera output by the ORB-SLAM3 system when the current image is collected can be obtained. slam (t):
[0164] P slam (t) = Track (r 4_normalized (t),a 4_normalized (t),image)
[0165] Wherein, t is the time when the current image is collected, image represents the current image, and Track represents the process of tracking the camera position.
[0166] S218: Obtain the historical position of the camera when capturing the previous frame of the current image.
[0167] Among them, the method for determining the historical position of the camera when capturing the previous frame image of the above-mentioned current image is similar to the method for determining the current position of the above-mentioned camera when capturing the above-mentioned current image, and the specific steps are not repeated here.
[0168] S219: Determine the head speed information of the target object according to the current position, the historical position and a preset acquisition cycle, wherein the preset acquisition cycle may be an acquisition cycle of image or relative inertial data.
[0169] In some embodiments, in S219, the head speed information of the target object is determined according to the current position, the historical position and the preset acquisition period, including: calculating the difference between the current position and the historical position, and determining the ratio of the difference to the preset period as the head speed information of the target object. slam (t) can be expressed as:
[0170] v slam (t)=(P slam (t)-P slam (t-1)) / Δt
[0171] Among them, P slam (t) represents the current position, P slam (t-1) represents the above historical position, and Δt represents the preset collection period.
[0172] In the embodiment of the application itself, visual data and inertial data are combined, and the visual data is used to provide rich scene information and depth information, which can improve the accuracy of head position and speed estimation.
[0173] In some embodiments, in S22, the relative inertial data is processed by a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object, including S221-S222:
[0174] S221. Input the above relative inertia data into a pre-trained posture information estimation neural network, and use the above fully convolutional neural network to predict the positions of all non-root joints of the above target object based on the above relative inertia data, to obtain the corresponding joint position information of the above target object output by the above fully convolutional neural network.
[0175] S222. Input the relative inertia data and the joint position information into the recurrent neural network, and use the recurrent neural network to predict the postures of all the non-root joints to obtain the joint posture information corresponding to the target object.
[0176] Figure 4 A schematic diagram of a neural network for estimating posture information provided by an exemplary embodiment of the present application is shown in FIG. Figure 4 As shown, the pose information estimation network is divided into two stages to consider the hierarchical structure of the human kinematics tree.
[0177] In the first stage, the global inertial data x measured by the inertial sensor raw After normalization, the relative inertial data x is obtained normalized . The relative inertia data x normalized In the input posture information estimation neural network, a fully convolutional neural network is used to estimate the relative inertial data x normalized Perform full joint position inference to obtain the corresponding joint position information J of the target object output by the full convolutional neural network full (J full ∈R 69 ), that is, the first position information of each joint of all non-root joints of the target object relative to the above-mentioned root joint. The first position information is three-dimensional position information. 69 Refers to a vector space consisting of 69 real numbers. full It represents the three-dimensional positions of 23 non-root joints. Therefore, for the 23 non-root joints, 23×3=69 real numbers are needed to fully describe their positions.
[0178] Among them, at time t, the joint position information includes the first position information p of each joint of the target object relative to the above-mentioned root joint all (t) can be expressed as:
[0179] p all (t) = [p j (t)|j=1,2,...,J-1]∈R 69
[0180] Among them, j is the serial number of each joint.
[0181] In the second stage, the relative inertial data x normalized and joint position information J full Input into the recurrent neural network, use the recurrent neural network to predict the posture of all the above non-root joints, and obtain the global joint posture information R corresponding to the target object output by the recurrent neural network global , global joint posture information R global Includes the global posture information of each joint relative to the root joint.
[0182] Optionally, the global joint pose information R at time t can be expressed using 6DOF representation global (t) is expressed as:
[0183]
[0184] Among them, j is the serial number of each joint, and the use of 6DOF representation provides better continuity in the output.
[0185] Furthermore, the global joint posture information R can be calculated by presetting the forward motion formula global Perform joint rotation completion to obtain the joint posture information R corresponding to the target object local , that is, the first posture information of each joint relative to the parent joint of the above joint.
[0186] Optionally, for each joint, the global joint posture information R is calculated by presetting the forward kinematics formula. global Perform joint rotation completion to obtain the joint posture information R corresponding to the target object local (i,t) can be expressed as:
[0187]
[0188] Among them, parent(j) is the parent joint number corresponding to the j-th joint.
[0189] In the embodiment of the present application, by combining relative inertial data and joint position information, the position and posture of the joints are predicted more accurately, thereby improving the prediction accuracy of the overall movement; in addition, by dividing the processing into two stages, the first stage processes the position and the second stage processes the posture. This hierarchical processing method better reflects the hierarchical structure of the human kinematics tree and helps to improve the accuracy of the prediction.
[0190] In some embodiments, for the training of the above-mentioned posture information estimation neural network, the above-mentioned method further includes S01-S03:
[0191] S01. Obtain training data.
[0192] S02. Input the initial posture information in the above training data into the above fully convolutional neural network to obtain the hidden state vector output by the above fully convolutional neural network.
[0193] S03. Using the hidden state vector as the initial hidden state of the initial recurrent neural network training to train the initial recurrent neural network.
[0194] Figure 5 A schematic diagram of the training process of a posture information estimation neural network provided by an exemplary embodiment of the present application, wherein x_firstframe represents the above-mentioned initial posture information. Figure 5 In each Simple Recurrent Unit (SRU) network, multiple layers of 9 (e.g., 4) unidirectional SRUs can be stacked, and the hidden layer size of each SRU layer is 256. Each SRU network uses a front linear layer to adapt to input vectors x of different lengths and outputs an output vector of the desired output length; in addition, it also includes a normalization and discarding layer (i.e., Dropout layer) to enhance the generalization performance.
[0195] In addition, this application uses a recursive neural network (RNN) to capture key state change signals by retaining complete historical information, and adds another independent fully connected neural network (FCN) to regress the initial state of the RNN based on the posture information. FCN and RNN are trained together. At the beginning of each batch of training, the ground truth posture is input into the FCN to initialize the hidden state of the RNN, and then regular training is performed. This method does not require the implementation of RNN to be modified, which effectively improves the compatibility and efficiency of model training.
[0196] The SRU used in the embodiments of the present application is a variant of RNN, which optimizes computational efficiency in design and is particularly suitable for application scenarios that require real-time processing, such as real-time posture estimation. SRU reduces complex matrix operations by simplifying the structure of the cyclic unit, which enables the network to quickly perform forward propagation, thereby reducing latency. Specifically, SRU uses parallelization technology to process sequence data, effectively improving processing speed while maintaining performance comparable to that of traditional time recurrent neural networks (Long Short-Term Memory, LSTM).
[0197] In some embodiments, the above-mentioned fully convolutional neural network can be trained based on the following L2 norm loss (Ridge Loss) function:
[0198]
[0199] in, is the real position information of each joint relative to the root joint, p all (t) is the first position information predicted for the joint relative to the root joint.
[0200] In some embodiments, the above recurrent neural network can be trained based on the following L2 norm loss function:
[0201]
[0202] in, is the real rotation vector of the joint relative to the root joint (i.e. the real posture information), R all (t) is the first posture information predicted for the joint relative to the parent joint of the joint.
[0203] In some embodiments, in S23, the joint position information and the relative inertia data are processed by a pre-trained physical information estimation network to obtain the joint velocity information and joint contact state information corresponding to the target object, including S231-S232:
[0204] S231, inputting the joint position information and the relative inertia data into a joint contact inference network in a pre-trained physical information estimation network to obtain joint contact state information corresponding to the target object.
[0205] S232, inputting the joint position information and the relative inertia data into the joint linear velocity inference network in the physical information estimation network to obtain the joint velocity information corresponding to the target object.
[0206] Figure 6 The following is a schematic diagram of a physical information estimation network provided by an exemplary embodiment of the present application. Figure 6 As shown in FIG. 1 , the physical information estimation network includes a joint contact inference network and a joint linear velocity inference network. The joint contact inference network receives the joint position information J output in the previous stage. full , the relative inertial data x of the combined sensor input normalized , outputs the degree of contact between each joint and the ground. After passing through the sigmoid function, it is normalized to the probability θ of whether each joint is in contact with the sole of the foot. joint (t):
[0207] θ joint (t) = [θ j (t)∈(0,1)|j=1,2,...,J]∈R J
[0208] Among them, the sigmoid function is a nonlinear activation function widely used in machine learning and neural networks. The sigmoid function can map any real input to a real output between 0 and 1.
[0209] Optionally, the following binary cross entropy loss function (BCE) can be used to train the joint touchdown inference network:
[0210]
[0211] in, is the true label indicating whether each joint touches the ground, θ joint (t) is the predicted touchdown probability for this joint.
[0212] In some embodiments, the joint linear velocity inference network part receives the joint position information J output in the previous stage. full , the relative inertial data x of the combined sensor input normalized The output is the estimated position difference of each joint relative to the root node, and the position difference is directly divided by the preset acquisition period (such as 1 / 60 second) to obtain the estimated speed, that is, the first speed information v of the joint relative to the above-mentioned root joint joint (t):
[0213] v joint (t)=[(p j (t)-(p j-1 (t))*60|j=1,2,...,J-1]∈R J-1
[0214] Among them, (p j (t)-(p j-1 (t)) represents the position difference of the joint relative to the root node.
[0215] Optionally, the following L1 norm loss function can be used to train the above joint linear velocity inference network:
[0216]
[0217] in, Represents the true velocity of the joint relative to the above root joint, v joint (t) is the first velocity information predicted for the joint relative to the root joint.
[0218] In the embodiment of the present application, by combining the joint position information and the relative inertia data, it is possible to more accurately determine whether the joint has touched the ground, estimate the joint speed information, and use the quickly calculated joint speed information to provide instant feedback.
[0219] In some embodiments, in S24, based on a preset optimization algorithm, the head velocity information, the joint posture information, the joint velocity information and the joint contact state information are processed to obtain the target posture information corresponding to the target object, including S241-S244:
[0220] S241. For each of the above joints, determine a constraint condition based on the ground contact probability corresponding to the above joint.
[0221] In some embodiments, the constraint conditions in the embodiments of the present application are used to limit the speed of the contact point between the target object and the ground, and to perform collision constraints on the joint points that may pass through the ground.
[0222] Optionally, the constraints corresponding to each joint can be described using the following expressions:
[0223] |v exp_x (t),v exp_z (t)|<=0.1cm / s,v exp_y (t)>=0cm / s
[0224] when p rbdl_j (t)<0.02m orθ j (t)>0.8
[0225] Among them, v exp_x (t) is the velocity of the current joint on the x-axis, v exp_z (t) is the velocity of the current joint on the z-axis, v exp_y (t) is the velocity of the current joint on the y-axis, p rbdl_j is the position of the current joint in the Robot Body Dynamics Library (RBDL), which can be given by RBDL. The above constraints mean that when the joint position information corresponding to the joint is less than 2 cm, or the touchdown probability θ corresponding to the joint j When (t) is greater than 80%, it can be considered that the joint has collided with the ground. At this time, the velocity of the joint on the Y axis (v exp_y (t)) cannot be a negative number, otherwise the position of the joint will continue to accumulate to a negative number, indicating that it has passed through the ground, which is obviously unreasonable. At the same time, the absolute value of the velocity of other joints is constrained to be less than 10cm / s.
[0226] In particular, in order to solve the sliding problem, the speed constraints on the foot joints (the joints corresponding to the left foot node and the right foot node) are more stringent. In addition to the above constraint information, the constraints corresponding to the foot joints also include:
[0227] |v exp_x (t),v exp_y (t),v exp_z (t)|<=(1-θ j (t))*0.4
[0228] where j=FOOT,when p j (t)<0.02m orθ j (t)>0.5
[0229] That is to say, for the foot joint, the corresponding limit speed is related to the ground contact probability θ corresponding to the joint j(t), where 0.4 is a limiting factor, which is used to control the maximum allowable value of joint rotation to simulate the motion characteristics of human joints in reality and help ensure that the foot is more stable when touching the ground, thereby reducing the probability of sliding or instability. In addition, controlling the first position information of the foot joint relative to the root joint to be less than 0.02 meters or the ground contact probability corresponding to the foot joint to be greater than 50% can prevent the virtual character in the VR game from abnormally jumping or falling when walking or running, enhance the stability of motion capture, and provide a more immersive experience for the target object.
[0230] The above constraints establish a connection between the desired angular acceleration and the linear velocity, and further constrain the interval of the angular acceleration by constraining the linear velocity.
[0231] In the embodiment of the present application, by determining the constraint conditions for the probability of the joint touching the ground, a more stable and natural movement can be simulated. For example, when the foot touches the ground, limiting the speed can reduce abrupt movements or unstable sliding steps, making the movement look smoother. In addition, by setting the speed constraint, the unreasonable situation of the joint crossing the ground during the simulation can be prevented, which is conducive to improving the consistency and credibility of the posture information.
[0232] In the embodiment of the present application, RBDL is used to physically optimize the data output by the posture information estimation neural network and the physical information estimation neural network each time, ensuring that each reasoning is carried out under the premise of conforming to human kinematics, introducing a physical solver, using a dual PD controller algorithm to smoothly control the irregular neural network output, using RBDL kinematics to calculate the collision between joints, obtaining a more accurate absolute displacement by constraining the direction of the ground force, measuring the possible residual force of the body and optimizing it using a penalty term. Therefore, RBDL can be used to combine the prior information of human motion, effectively simulate and calculate complex multi-degree-of-freedom motion, and simplify the problem by pre-set constraints, thereby optimizing the computational efficiency.
[0233] S242. Utilize a preset optimization equation to process based on the constraint conditions, the head velocity information, the joint posture information, and the joint velocity information to obtain a target angular acceleration corresponding to the joint.
[0234] In some embodiments, the above preset optimization equation can be expressed as:
[0235] min||σ exp (t)J(t)-K p *(v ref (t)-v eef (t-1))-K d *σ exp(t-1)J(t)|| 2
[0236] +||K p *(R local (t)-R local (t-1))+K d *σ exp (t-1)|| 2
[0237] Among them, ||σ exp (t) is the desired angular acceleration (target angular acceleration), J(t) is the transformation matrix from the current angular velocity to the linear velocity (given by RBDL), K p is the amplification filter coefficient, (v ref (t)-v ref (t-1)) describes the difference between the joint velocity information at the current moment and the previous moment, K d is the differential filter coefficient, which is related to the angular acceleration at the last moment, σ exp (t-1) is the angular acceleration at the previous moment, (R local (t)-R local (t-1)) describes the difference between the joint posture information at the current moment and the previous moment.
[0238] In some embodiments, a preset quadratic programming (QP) framework can be used to optimize the target angular acceleration corresponding to the joint based on rigid body mechanics and real-world physical laws, combined with constraints, the head velocity information, the joint posture information, and the joint velocity information. Specifically, the QP framework solves the preset optimization equation based on the constraints to obtain the target angular acceleration σ exp , so that it satisfies the constraints and the penalty term is minimized. The above QP framework refers to the problem of minimizing a quadratic objective function under certain constraints.
[0239] In the embodiment of the present application, the QP framework is used to accurately control and predict the movement of the joints, ensuring that the movement conforms to the laws of rigid body mechanics and the physical laws of the real world; and combined with various constraints, such as joint restrictions, dynamic restrictions, and kinematic restrictions, it is ensured that the optimized target angular acceleration is both reasonable and meets the actual movement requirements. In addition, the QP framework has good solution efficiency and can quickly calculate the optimal solution in a real-time system to meet the needs of real-time control and prediction.
[0240] S243. Determine the joint posture information corresponding to the joint according to the target angular acceleration, and then obtain multiple joint posture information corresponding to all the non-root joints.
[0241] Among them, the joint posture information corresponding to each joint includes the second position information and second posture information corresponding to the joint, the second position information is the result of optimizing the first position information of the joint, and the second posture information is the result of optimizing the first posture information of the joint.
[0242] S244, determining the multiple joint posture information corresponding to all the above-mentioned non-root joints as the target posture information corresponding to the above-mentioned target object.
[0243] Among them, the non-root joints and joint pose information correspond one to one.
[0244] In some embodiments, in S243, determining the joint posture information corresponding to the joint according to the target angular acceleration includes S2431-S2435:
[0245] S2431. Determine the target linear velocity corresponding to the joint according to the preset matrix and the target angular acceleration.
[0246] Among them, the preset matrix is the transformation matrix between linear velocity and angular velocity, that is, the above-mentioned J(t).
[0247] In some embodiments, the product of the preset matrix and the target angular acceleration can be determined as the target linear velocity corresponding to the joint.
[0248]
[0249] in, is the target angular acceleration corresponding to the joint at time t.
[0250] S2432. Accumulate the target linear velocity to obtain second position information corresponding to the joint.
[0251] Optionally, the accumulation operation of the above target linear velocity can be expressed as:
[0252]
[0253] Among them, P opt (t-1) represents the second position information of the joint at time t-1, P oot (t) represents the second position information of the joint at time t, and Δt is the acquisition period.
[0254] S2433. Accumulate the target angular acceleration to obtain the target angular velocity corresponding to the joint.
[0255] Optionally, the accumulation operation of the target angular acceleration can be expressed as:
[0256]
[0257] Among them, θ(t) represents the target angular velocity of the joint at time t, and θ(t-1) represents the target angular velocity of the joint at time t-1.
[0258] S2434. Accumulate the target angular velocity to obtain second posture information corresponding to the joint.
[0259] Optionally, the accumulation operation of the above target angular velocity can be expressed as:
[0260] r(t)=r(t-1)+θ(t)*Δt
[0261] Among them, r(t) represents the second posture information corresponding to the joint at time t, and r(t-1) represents the second posture information corresponding to the joint at time t-1.
[0262] S2435. Determine the second position information and the second posture information as joint posture information corresponding to the joint.
[0263] In some embodiments, a Kalman filter algorithm may also be used to fuse data from the magnetometer and the gyroscope to reduce the drift of the inertial sensor.
[0264] In the embodiment of the present application, a PD controller algorithm is introduced to make the posture response faster and use the trend of the differential quantity to predict the posture change; in addition, a joint collision controller is introduced to ensure that each joint will not break through the ground. In addition, the residual force of the joint after the action is completed is calculated, and the residual force is optimized so that no additional displacement occurs. The speed of the foot joint is emphasized to generate a credible ground contact displacement. The ground force feedback controller ensures that the ground support force and friction force are reasonable during the calculation of the displacement. Finally, a smoothing controller is used to smooth the result again.
[0265] The posture information determination method provided in the embodiment of the present application can be specifically implemented by a micro-electro-mechanical system (MEMS) and used in a variety of scenarios, such as VR games, sports rehabilitation, film and television animation production, robotics, etc. Taking sports rehabilitation as an example, the posture information determination method provided in the embodiment of the present application can automatically detect movements, correct scoliosis, determine the cycle and relative strength of muscle function, record user information, and perform corrections based on motion data.
[0266] Please refer to the following Figure 7 , which is a schematic diagram of the structure of a posture information determination device provided by an exemplary embodiment of the present application. Figure 7 As shown, the above-mentioned posture information determination device 700 includes:
[0267] A first acquisition module 701 is used to acquire relative inertia data of a plurality of preset joints of a target object relative to a root joint, and to acquire head velocity information of the target object;
[0268] A first processing module 702 is used to process the relative inertial data through a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object;
[0269] The second processing module 703 is used to process the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain the joint velocity information and joint contact state information corresponding to the target object;
[0270] The first determination module 704 is used to process the head velocity information, the joint posture information, the joint velocity information and the joint ground contact state information based on a preset optimization algorithm to obtain the target posture information corresponding to the target object.
[0271] In a possible implementation, the joint position information corresponding to the target object includes the first position information of each joint of all non-root joints of the target object relative to the root joint; the joint posture information includes the first posture information of each joint relative to the parent joint of the joint; the joint speed information includes the first speed information of each joint relative to the root joint; the joint contact state information includes the contact probability corresponding to each joint, and the contact probability is used to indicate the probability of the corresponding joint contacting the ground.
[0272] In a possible implementation, the plurality of preset joints include the root joint, and the acquisition module includes:
[0273] The first acquisition unit is used to acquire the relative inertia data of multiple preset joints of the target object relative to the root joint; the first acquisition unit is used to collect global inertia data using inertial sensors worn at the multiple preset joints of the target object, wherein the global inertia data includes global rotation information and global acceleration information corresponding to each preset joint in the multiple preset joints; the first determination unit is used to determine the root rotation information and root acceleration information corresponding to the root joint from the global inertia data; the second determination unit is used to determine the relative rotation information corresponding to the preset joint based on the root rotation information and the global rotation information corresponding to the preset joint for each preset joint; the third determination unit is used to determine the relative acceleration information corresponding to the preset joint based on the root acceleration information and the global acceleration information corresponding to the preset joint, and then obtain multiple relative rotation information and multiple relative acceleration information corresponding to the multiple preset joints; the fourth determination unit is used to determine the multiple relative rotation information and multiple relative acceleration information as the relative inertia data of the multiple preset joints of the target object relative to the root joint.
[0274] In a possible implementation, the acquisition module further includes:
[0275] The second acquisition unit is used to acquire the current image by using the camera arranged on the head of the above-mentioned target object; the second acquisition unit is used to acquire the head relative rotation information and the head relative acceleration information corresponding to the head of the above-mentioned target object at the moment of acquiring the above-mentioned current image; the fifth determination unit is used to determine the current position of the above-mentioned camera when acquiring the above-mentioned current image based on the above-mentioned current image, the above-mentioned head relative rotation information and the above-mentioned head relative acceleration information; the third acquisition unit is used to acquire the historical position of the above-mentioned camera when acquiring the previous frame image of the above-mentioned current image; the sixth determination unit is used to determine the head speed information of the above-mentioned target object according to the above-mentioned current position, the above-mentioned historical position and the preset acquisition period.
[0276] In a possible implementation, the posture information estimation neural network includes a fully convolutional neural network and a recurrent neural network, and the first processing module 702 includes:
[0277] The first prediction unit is used to input the above-mentioned relative inertia data into the pre-trained posture information estimation neural network, and use the above-mentioned full convolutional neural network to predict the positions of all non-root joints of the above-mentioned target object according to the above-mentioned relative inertia data, so as to obtain the corresponding joint position information of the above-mentioned target object output by the above-mentioned full convolutional neural network; the second prediction unit is used to input the above-mentioned relative inertia data and the above-mentioned joint position information into the above-mentioned recurrent neural network, and use the above-mentioned recurrent neural network to predict the postures of all the above-mentioned non-root joints to obtain the corresponding joint posture information of the above-mentioned target object.
[0278] In a possible implementation, the above-mentioned posture determination device further includes:
[0279] The second acquisition module is used to acquire training data; the second determination module is used to input the initial posture information in the above training data into the above fully convolutional neural network to obtain the hidden state vector output by the above fully convolutional neural network; the training module is used to use the above hidden state vector as the initial hidden state of the initial recurrent neural network training to train the above initial recurrent neural network.
[0280] In a possible implementation, the physical information estimation network includes a joint contact reasoning network and a joint linear velocity reasoning network;
[0281] In a possible implementation, the second processing module 703 includes:
[0282] The seventh determination unit is used to input the above-mentioned joint position information and the above-mentioned relative inertia data into the joint contact reasoning network in the pre-trained physical information estimation network to obtain the joint contact state information corresponding to the above-mentioned target object; the eighth determination unit is used to input the above-mentioned joint position information and the above-mentioned relative inertia data into the joint linear velocity reasoning network in the above-mentioned physical information estimation network to obtain the joint velocity information corresponding to the above-mentioned target object.
[0283] In a possible implementation, the first determining module 704 includes:
[0284] The ninth determination unit is used to determine the constraint conditions for each of the above-mentioned joints based on the ground contact probability corresponding to the above-mentioned joints; the processing unit is used to use a preset optimization equation to process based on the above-mentioned constraint conditions, the above-mentioned head velocity information, the above-mentioned joint posture information and the above-mentioned joint velocity information to obtain the target angular acceleration corresponding to the above-mentioned joints; the tenth determination unit is used to determine the joint posture information corresponding to the above-mentioned joints according to the above-mentioned target angular acceleration, and then obtain the multiple joint posture information corresponding to all the above-mentioned non-root joints; the eleventh determination unit is used to determine the multiple joint posture information corresponding to all the above-mentioned non-root joints as the target posture information corresponding to the above-mentioned target object.
[0285] In a possible implementation manner, the tenth determining unit includes:
[0286] The first determining subunit is used to determine the joint posture information corresponding to the above-mentioned joint according to the above-mentioned target angular acceleration, including: a second determining subunit is used to determine the target linear velocity corresponding to the above-mentioned joint according to a preset matrix and the above-mentioned target angular acceleration; a first accumulating subunit is used to accumulate the above-mentioned target linear velocity to obtain the second position information corresponding to the above-mentioned joint; the second accumulating subunit is used to accumulate the above-mentioned target angular acceleration to obtain the target angular velocity corresponding to the above-mentioned joint; the third accumulating subunit is used to accumulate the above-mentioned target angular velocity to obtain the second posture information corresponding to the above-mentioned joint; the third determining subunit is used to determine the above-mentioned second position information and the above-mentioned second posture information as the joint posture information corresponding to the above-mentioned joint.
[0287] The division of the modules in the above-mentioned posture information determination device 700 is only for illustration. In other embodiments, the posture information determination device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned posture information determination device. The implementation of each module in the posture information determination device provided in the embodiments of this specification can be in the form of a computer program. The computer program can be run on a terminal or a server. The program modules constituted by the computer program can be stored in the memory of a terminal or a server. When the computer program is executed by the processor, all or part of the steps of the posture information determination method described in the embodiments of this specification are implemented.
[0288] See next Figure 8 , which is a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. Figure 8 As shown, the electronic device 800 may include: a processor 810 and a memory 820 , and may also include a user interface 830 , a network interface 840 and a communication bus 850 .
[0289] Among them, the processor 810 may include one or more processing cores. The processor 810 uses various interfaces and lines to connect various parts of the entire electronic device 800, and executes various functions and processes data of the electronic device 800 by running or executing instructions, programs, code sets or instruction sets stored in the memory 820, and calling data stored in the memory 820. Optionally, the processor 810 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 810 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 810, and it can be implemented separately through a chip.
[0290] The memory 820 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 820 includes a non-transitory computer-readable storage medium. The memory 820 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a receiving function, a control function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 820 may optionally be at least one storage device located away from the aforementioned processor 810. As Figure 8 As shown, the memory 820 as a computer storage medium may include an operating system, a network communication module, a user interface module, and program instructions.
[0291] Optionally, the communication bus 850 is used to realize the connection and communication between these components. The user interface 830 may include a display screen (Display), a camera (Camera), and may also include a standard wired interface and a wireless interface; the network interface 840 may optionally include a standard wired interface and a wireless interface (such as a WIFI interface).
[0292] exist Figure 8 In the electronic device 800 shown, the processor 810 may be used to call the program instructions stored in the memory 820 and specifically perform the following operations:
[0293] Obtaining relative inertia data of a plurality of preset joints of the target object relative to the root joint, and obtaining head velocity information of the target object;
[0294] The relative inertial data is processed by a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object;
[0295] Processing the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain joint velocity information and joint ground contact state information corresponding to the target object;
[0296] Based on a preset optimization algorithm, the head velocity information, the joint posture information, the joint velocity information and the joint ground contact state information are processed to obtain target posture information corresponding to the target object.
[0297] In a possible implementation, the joint position information corresponding to the target object includes the first position information of each joint of all non-root joints of the target object relative to the root joint; the joint posture information includes the first posture information of each joint relative to the parent joint of the joint; the joint speed information includes the first speed information of each joint relative to the root joint; the joint contact state information includes the contact probability corresponding to each joint, and the contact probability is used to indicate the probability of the corresponding joint contacting the ground.
[0298] In a possible implementation, the plurality of preset joints include the root joint, and the step of obtaining relative inertia data of the plurality of preset joints of the target object relative to the root joint includes:
[0299] Collecting global inertial data using inertial sensors worn at the plurality of preset joints of the target object, wherein the global inertial data includes global rotation information and global acceleration information corresponding to each of the plurality of preset joints;
[0300] Determining root rotation information and root acceleration information corresponding to the root joint from the global inertial data;
[0301] For each of the preset joints, determine the relative rotation information corresponding to the preset joint based on the root rotation information and the global rotation information corresponding to the preset joint;
[0302] Determine the relative acceleration information corresponding to the preset joint based on the root acceleration information and the global acceleration information corresponding to the preset joint, and then obtain a plurality of relative rotation information and a plurality of relative acceleration information corresponding to the plurality of preset joints;
[0303] The plurality of relative rotation information and the plurality of relative acceleration information are determined as relative inertia data of the plurality of preset joints of the target object relative to the root joint.
[0304] In a possible implementation manner, the step of obtaining the head velocity information of the target object includes:
[0305] A current image is captured by a camera disposed on the head of the target object; head relative rotation information and head relative acceleration information corresponding to the head of the target object at the moment of capturing the current image are obtained; a current position of the camera when capturing the current image is determined based on the current image, the head relative rotation information and the head relative acceleration information; a historical position of the camera when capturing a previous frame of the current image is obtained; and head velocity information of the target object is determined based on the current position, the historical position and a preset capture period.
[0306] In a possible implementation, the posture information estimation neural network includes a fully convolutional neural network and a recurrent neural network, and the relative inertia data is processed by the pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object, including: inputting the relative inertia data into the pre-trained posture information estimation neural network, using the fully convolutional neural network to predict the positions of all non-root joints of the target object according to the relative inertia data, and obtaining the corresponding joint position information of the target object output by the fully convolutional neural network; inputting the relative inertia data and the joint position information into the recurrent neural network, using the recurrent neural network to predict the postures of all non-root joints, and obtaining the joint posture information corresponding to the target object.
[0307] In a possible implementation, the method further includes:
[0308] Acquire training data; input the initial posture information in the above training data into the above fully convolutional neural network to obtain the hidden state vector output by the above fully convolutional neural network; use the above hidden state vector as the initial hidden state of the initial recurrent neural network training to train the above initial recurrent neural network.
[0309] In a possible implementation, the physical information estimation network includes a joint contact reasoning network and a joint linear velocity reasoning network;
[0310] The pre-trained physical information estimation network processes the joint position information and the relative inertia data to obtain the joint velocity information and joint contact state information corresponding to the target object, including:
[0311] Inputting the joint position information and the relative inertia data into a joint contact inference network in a pre-trained physical information estimation network to obtain joint contact state information corresponding to the target object;
[0312] The joint position information and the relative inertia data are input into the joint linear velocity inference network in the physical information estimation network to obtain the joint velocity information corresponding to the target object.
[0313] In a possible implementation, the head velocity information, the joint posture information, the joint velocity information and the joint contact state information are processed based on a preset optimization algorithm to obtain target posture information corresponding to the target object, including:
[0314] For each of the above joints, determining a constraint condition based on the ground contact probability corresponding to the above joint;
[0315] Using a preset optimization equation to process based on the constraint conditions, the head velocity information, the joint posture information and the joint velocity information, a target angular acceleration corresponding to the joint is obtained;
[0316] Determine the joint posture information corresponding to the joint according to the target angular acceleration, and then obtain multiple joint posture information corresponding to all the non-root joints;
[0317] The multiple joint posture information corresponding to all the non-root joints are determined as the target posture information corresponding to the target object.
[0318] In a possible implementation, determining the joint posture information corresponding to the joint according to the target angular acceleration includes:
[0319] Determine the target linear velocity corresponding to the joint according to the preset matrix and the target angular acceleration;
[0320] Accumulating the target linear velocity to obtain second position information corresponding to the joint;
[0321] Accumulating the target angular acceleration to obtain the target angular velocity corresponding to the joint;
[0322] Accumulating the target angular velocity to obtain second posture information corresponding to the joint;
[0323] The second position information and the second posture information are determined as joint posture information corresponding to the joint.
[0324] The embodiment of the present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps in the above embodiment. If the components of the above-mentioned posture information determination device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium.
[0325] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The above-mentioned computer program product includes one or more computer instructions. When the above-mentioned computer program instructions are loaded and executed on a computer, the above-mentioned process or function according to the embodiment of the present application is generated in whole or in part. The above-mentioned computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium or transmitted by the above-mentioned computer-readable storage medium. The above-mentioned computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The above-mentioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The above-mentioned available media can be magnetic media (for example, floppy disks, hard disks, tapes), optical media (for example, digital versatile discs (DVD)), or semiconductor media (for example, solid state disks (SSD)), etc.
[0326] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes. In the absence of conflict, the technical features in this embodiment and the implementation scheme can be combined arbitrarily.
[0327] The above-mentioned embodiments are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements made to the technical solutions of the present application by ordinary technicians in this field should fall within the protection scope determined by the claims of the present application.
Claims
1. A method for determining posture information, characterized in that: include: Acquire relative inertia data of a plurality of preset joints of a target object relative to a root joint, and acquire head velocity information of the target object; Processing the relative inertial data through a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object; Processing the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain joint velocity information and joint ground contact state information corresponding to the target object; Based on a preset optimization algorithm, the head velocity information, the joint posture information, the joint velocity information and the joint ground contact state information are processed to obtain target posture information corresponding to the target object.
2. The method according to claim 1, characterized in that The joint position information corresponding to the target object includes first position information of each joint of all non-root joints of the target object relative to the root joint; The joint posture information includes first posture information of each joint relative to the parent joint of the joint; The joint speed information includes first speed information of each joint relative to the root joint; The joint contact state information includes the contact probability corresponding to each joint, and the contact probability is used to indicate the probability of the corresponding joint contacting the ground.
3. The method according to claim 1, characterized in that The plurality of preset joints include the root joint, and the step of obtaining relative inertia data of the plurality of preset joints of the target object relative to the root joint includes: Collecting global inertial data using inertial sensors worn at the plurality of preset joints of the target object, wherein the global inertial data includes global rotation information and global acceleration information corresponding to each of the plurality of preset joints; Determining root rotation information and root acceleration information corresponding to the root joint from the global inertial data; For each preset joint, determining relative rotation information corresponding to the preset joint based on the root rotation information and the global rotation information corresponding to the preset joint; Determine the relative acceleration information corresponding to the preset joint based on the root acceleration information and the global acceleration information corresponding to the preset joint, and then obtain a plurality of relative rotation information and a plurality of relative acceleration information corresponding to the plurality of preset joints; The plurality of relative rotation information and the plurality of relative acceleration information are determined as relative inertia data of the plurality of preset joints of the target object relative to the root joint.
4. The method according to claim 1, characterized in that The acquiring the head speed information of the target object includes: Using a camera disposed on the head of the target object to collect a current image; Acquire head relative rotation information and head relative acceleration information corresponding to the head of the target object at the moment of capturing the current image; Determining a current position of the camera when acquiring the current image based on the current image, the head relative rotation information, and the head relative acceleration information; Obtaining a historical position of the camera when capturing a previous frame of the current image; The head speed information of the target object is determined according to the current position, the historical position and a preset acquisition cycle.
5. The method according to claim 2, characterized in that The posture information estimation neural network includes a fully convolutional neural network and a recurrent neural network. The posture information estimation neural network trained in advance processes the relative inertial data to obtain joint position information and joint posture information corresponding to the target object, including: Inputting the relative inertia data into a pre-trained posture information estimation neural network, using the fully convolutional neural network to predict the positions of all non-root joints of the target object according to the relative inertia data, and obtaining corresponding joint position information of the target object output by the fully convolutional neural network; The relative inertia data and the joint position information are input into the recurrent neural network, and the recurrent neural network is used to predict the postures of all non-root joints to obtain the joint posture information corresponding to the target object.
6. The method according to claim 5, characterized in that The method further comprises: Get training data; Inputting the initial posture information in the training data into the fully convolutional neural network to obtain a hidden state vector output by the fully convolutional neural network; The hidden state vector is used as an initial hidden state for initial recurrent neural network training to train the initial recurrent neural network.
7. The method according to claim 2, characterized in that The physical information estimation network includes a joint contact reasoning network and a joint linear velocity reasoning network; The pre-trained physical information estimation network processes the joint position information and the relative inertia data to obtain joint velocity information and joint contact state information corresponding to the target object, including: Inputting the joint position information and the relative inertia data into a joint contact inference network in a pre-trained physical information estimation network to obtain joint contact state information corresponding to the target object; The joint position information and the relative inertia data are input into a joint linear velocity inference network in the physical information estimation network to obtain joint velocity information corresponding to the target object.
8. The method according to claim 2, characterized in that The step of processing the head velocity information, the joint posture information, the joint velocity information and the joint ground contact state information based on a preset optimization algorithm to obtain target posture information corresponding to the target object includes: For each of the joints, determining a constraint condition based on a ground contact probability corresponding to the joint; Using a preset optimization equation to process based on the constraint condition, the head velocity information, the joint posture information and the joint velocity information, to obtain a target angular acceleration corresponding to the joint; Determine the joint posture information corresponding to the joint according to the target angular acceleration, and then obtain multiple joint posture information corresponding to all non-root joints; The plurality of joint pose information corresponding to all the non-root joints are determined as the target pose information corresponding to the target object.
9. The method according to claim 8, characterized in that Determining joint posture information corresponding to the joint according to the target angular acceleration includes: Determine a target linear velocity corresponding to the joint according to a preset matrix and the target angular acceleration; Accumulating the target linear velocity to obtain second position information corresponding to the joint; Accumulating the target angular acceleration to obtain a target angular velocity corresponding to the joint; Accumulating the target angular velocity to obtain second posture information corresponding to the joint; The second position information and the second posture information are determined as joint posture information corresponding to the joint.
10. A device for determining posture information, characterized in that: include: A first acquisition module is used to acquire relative inertia data of a plurality of preset joints of a target object relative to a root joint, and to acquire head velocity information of the target object; A first processing module, configured to process the relative inertial data through a pre-trained posture information estimation neural network to obtain joint position information and joint posture information corresponding to the target object; A second processing module is used to process the joint position information and the relative inertia data through a pre-trained physical information estimation network to obtain joint velocity information and joint contact state information corresponding to the target object; The first determination module is used to process the head velocity information, the joint posture information, the joint velocity information and the joint ground contact state information based on a preset optimization algorithm to obtain the target posture information corresponding to the target object.
11. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method according to any one of claims 1 to 9.
12. A computer storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 9.
