Joint angle learning estimation system, joint angle learning system, joint angle estimation device, joint angle learning method, and computer program
Patent Information
- Application Number
- JP2024503259
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Priority Date
- 2023-02-24
- Filing Date
- 2023-02-24
- Publication Date
- 2024-12-06
AI Technical Summary
Conventional methods for estimating human joint angles during walking require attaching multiple sensors or markers, placing a significant burden on the subject and being less accurate.
A system using machine learning with inertial sensors attached to specific body parts, such as the waist, thighs, and shins, to estimate joint angles by measuring acceleration and angular velocity, reducing the number of sensors needed and improving accuracy.
Enables accurate joint angle estimation with reduced burden on the subject by using a smaller number of inertial sensors and improving the precision of joint angle identification during walking.
Abstract
Description
Joint angle learning and estimation system, joint angle learning system, joint angle estimation device, joint angle learning method, and computer program
[0001] The present invention relates to a technique for estimating joint angles of an individual, such as a human.
[0002] Techniques for identifying human walking movements are being utilized in various fields, including healthcare, sports, virtual reality, and animation. The identified movements are used, for example, in the following ways: in the healthcare field, for diagnosing patients; in the sports field, for checking athletes' form; in the virtual reality field, for replicating a user's movements with an avatar; and in the animation field, for replicating a user's movements with a character.
[0003] The following methods have been conventionally known as methods for identifying human movement while walking. One method involves attaching markers to multiple parts of the human body and capturing the trajectory of each marker while walking with a video camera to identify movement. Alternatively, sensors such as an inertial measurement unit (IMU) are attached instead of markers, and movement is identified based on the acceleration and angular velocity measured by each sensor. By identifying the positions and orientations of segments such as the hips, thighs, and shins using these methods, joint angles can be identified. For example, according to a method described in Patent Document 1, an inertial sensor is attached to each of the forearm and upper arm, and the angle of the elbow joint is calculated based on the angular velocity and acceleration measured by these inertial sensors.
[0004] In addition, the following method has been proposed as a method for acquiring the angles of joints while a person is walking. According to the method described in Patent Document 2, a wearing device having a relatively rotatable arm part, a waist-mounted part, and an angle sensor that detects the relative rotation angle between the two is worn on the human body, and the angles of the joints of the human body are identified.
[0005] JP 2021-503340 A JP 2018-134724 A
[0006] As mentioned above, there are several methods for determining the joint angles of a human walking. However, all of these methods require attaching numerous sensors to the body or wearing bulky equipment. Alternatively, they require attaching numerous markers and capturing the image with a video camera. As such, these conventional methods place a burden on the person whose joint angles are being determined.
[0007] In view of the above problems, the present invention aims to identify the angle of a joint while reducing the burden on the subject compared to conventional methods.
[0008] An individual angle learning and estimation system according to one embodiment of the present invention comprises a first inertial sensor attached to a predetermined part of a learning subject individual that is the subject of learning in order to determine the acceleration and angular velocity of the predetermined part, an acquisition means for acquiring the angle of the joint of the learning subject individual as a correct angle, a learning means for generating a learned model by performing machine learning using the acceleration and angular velocity of the predetermined part of the learning subject individual as input data and the correct angle as correct data, a second inertial sensor attached to the predetermined part of the estimation subject individual that is the subject of estimation in order to determine the acceleration and angular velocity of the predetermined part, and an estimation means for estimating the angle of the joint of the estimation subject individual by inputting the acceleration and angular velocity of the predetermined part of the estimation subject individual into the learned model.
[0009] The predetermined part is, for example, one of the waist, left thigh, right thigh, left shin, right shin, left foot, and right foot. There may be multiple predetermined parts. For example, the accelerations and angular velocities of three parts, the waist, left foot, and right foot, may be calculated.
[0010] According to the present invention, it is possible to identify the angle of a joint while reducing the burden on the subject compared to conventional techniques.
[0011] By attaching a second inertial sensor to any of the waist, left thigh, right thigh, left shin, right shin, left foot, and right foot and calculating the acceleration and angular velocity to estimate the joint angle of the target individual, the joint angle can be more accurately identified. Multiple second inertial sensors may be attached. In particular, attaching the second inertial sensors to the waist, left foot, and right foot can provide more accurate results.
[0012] 1 is a diagram showing an example of the overall configuration of a joint angle estimation system. FIG. 2 is a diagram showing an example of the hardware configuration of a learning inference device. FIG. 3 is a diagram showing an example of the functional configuration of a learning inference device. FIG. 4 is a flowchart explaining an example of the flow of a dataset preparation process. FIG. 5 is a diagram showing an example of the mounting positions of each of a plurality of inertial sensors. FIG. 6 is a diagram showing an example of changes in acceleration and angular velocity in a moving coordinate system of a waist inertial sensor, and an example of changes in the angles of four joints of a subject. FIG. 7 is a diagram showing an example of the flow of a machine learning process. FIG. 8 is a diagram showing an example of a neural network. FIG. 9 is a diagram showing an example of a trained neural network. FIG. 10 is a flowchart explaining an example of the overall processing flow by a joint angle estimation program. FIG. 11 is a diagram showing an example of MAE in each of the first to third patterns. FIG. 12 is a diagram showing an example of MAE when an inertial sensor is fixed to each part. FIG. 13 is a diagram showing an example of a method for controlling a prosthetic leg.
[0013] 1. Overall Configuration of the System Fig. 1 is a diagram showing an example of the overall configuration of a joint angle estimation system 1. Fig. 2 is a diagram showing an example of the hardware configuration of a learning and inference device 2. Fig. 3 is a diagram showing an example of the functional configuration of the learning and inference device 2.
[0014] 1 is a system that uses AI (artificial intelligence) technology to estimate the joint angles of a person 1A to be estimated while he or she is walking, and is composed of a learning and inference device 2 and a plurality of inertial sensors 3. The learning and inference device 2 and the inertial sensors 3 are connected wirelessly.
[0015] The learning and inference device 2 generates a learned model by machine learning, and estimates the angles of the joints of the person 1A to be inferred when the person 1A to be inferred is walking, based on the learned model, etc. Below, an example will be described in which a personal computer is used as the learning and inference device 2.
[0016] As shown in FIG. 2, the learning and inference device 2 is composed of a main processor 20, a RAM (Random Access Memory) 21, a ROM (Read Only Memory) 22, an auxiliary storage device 23, a network interface 24, a serial interface 25, a wireless communication device 26, a display 27, a keyboard 28, and a pointing device 29.
[0017] In addition to the operating system, computer programs such as a joint angle estimation program 40 and a sensor application 41 are installed in the ROM 22 or the auxiliary storage device 23 .
[0018] According to the joint angle estimation program 40, functions such as a shift window value extraction unit 401, a joint angle calculation unit 402, a data set registration unit 403, a data set storage unit 404, a machine learning unit 405, a learned model storage unit 406, a joint angle inference unit 407, and an inference result output unit 408 shown in FIG. 3 are realized.
[0019] The RAM 21 is the main memory of the learning and inference device 2. Computer programs such as a joint angle estimation program 40 and a sensor application 41 are loaded into the RAM 21 as needed.
[0020] The main processor 20 executes a computer program loaded into the RAM 21. As the main processor 20, a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit) is used.
[0021] The network interface 24 communicates with other devices using protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol). As the network interface 24, a NIC (Network Interface Card) or a communication device for Wi-Fi is used.
[0022] The serial interface 25 communicates with peripheral devices using a serial communication system, and is implemented as a board conforming to standards such as USB (Universal Serial Bus).
[0023] The wireless communication device 26 communicates with peripheral devices by short-range wireless communication, and is implemented as a board conforming to standards such as Bluetooth.
[0024] The serial interface 25 and the wireless communication device 26 are used particularly for communication with the inertial sensor 3. The serial interface 25 and the wireless communication device 26 are used depending on the standard of the inertial sensor 3.
[0025] The display 27 displays a screen for inputting commands or data, a screen showing the results of calculations performed by the main processor 20, and the like.
[0026] The keyboard 28 and pointing device 29 are input devices that allow the operator to input commands, data, and the like.
[0027] The inertial sensor 3 is a device that measures acceleration and angular velocity along three axes (X-axis, Y-axis, and Z-axis), and is generally called an "IMU (Inertial Measurement Unit)" or "Inertial Measurement Device." A commercially available IMU is used as the inertial sensor 3. The following description will be given taking as an example a case where an IMU provided by the MTw Awinda system of Xsens, a Dutch company, is used.
[0028] The MTw Awinda has the following functions: Multiple IMUs (up to 30) can measure acceleration and angular velocity using a unique protocol while maintaining synchronization accuracy of 1 / 100,000 of a second. As described below, in this embodiment, joint angles are calculated or estimated every 1 / 100 of a second, allowing the measurement results of these IMUs to be used in a nearly accurate synchronized manner. Each IMU can then transmit its measurement results to a personal computer via ZigBee.
[0029] Additionally, MTw Awinda provides a receiver and manager software for personal computers. The receiver is a ZigBee-compliant USB device for receiving data transmitted from the IMU. The manager software is a computer program for changing the settings and properties of the IMU. The receiver is connected to the serial interface 25, and the manager software is installed as a sensor application 41 in the auxiliary storage device 23.
[0030] The processing of each part of the learning and inference device 2 shown in FIG. 3 and the processing of the inertial sensor 3 shown in FIG. 1 will be explained below by roughly dividing it into a data set preparation phase, a machine learning phase, and an inference phase.
[0031] 2. Preparation of Data Set Figure 4 is a flowchart illustrating an example of the flow of a data set preparation process. Figure 5 is a diagram showing an example of the mounting positions of the multiple inertial sensors 31 to 37. Figure 6 is a diagram showing an example of changes in acceleration and angular velocity in the moving coordinate system of the waist inertial sensor 31, and an example of changes in the angles of the four joints of subject 1B.
[0032] The shift window value extraction unit 401, joint angle calculation unit 402, and dataset registration unit 403 (see FIG. 3) of the learning and inference device 2 execute the process of collecting information from multiple subjects 1B (1B1, 1B2, ...) with different genders, ages, weights, and heights and generating a dataset for machine learning according to the procedure shown in FIG. 4, based on operations by an operator of the joint angle estimation system 1. Below, an example will be described in which information is collected from a certain subject 1B1.
[0033] The operator inputs the age, sex, weight, and height as attributes of the subject 1B1 into the learning and inference device 2 using the keyboard 28 or pointing device 29. The data set registration unit 403 acquires an attribute vector C indicating these attributes (#101 in FIG. 4). Hereinafter, the attribute vector C of the v-th subject 1B will be referred to as the "attribute vector Cv."
[0034] Furthermore, the operator collects data on the state of the waist, left and right thighs, left and right shins, and left and right feet while the subject 1B1 is walking, as follows.
[0035] The operator prepares seven inertial sensors 3, one waist belt, two thigh belts, two shin belts, and two foot belts. These belts are exclusive to the MTw Awinda, and after initialization is performed with all seven inertial sensors 3 aligned in the same orientation, the inertial sensors 3 are attached one by one in a predetermined orientation. During this initialization, the orientation angles of all the inertial sensors are set to a certain orientation angle in a spatial (fixed) coordinate system, and generally, the orientation is aligned with the spatial coordinate system.
[0036] After attaching the inertial sensors 3 to these belts one by one, the subject 1B1 wears these belts on each segment (site) of the subject 1B1's waist, left and right thighs, left and right shins, and left and right feet according to the MTw Awinda manual. As a result, the inertial sensors 3 are fixed at specific positions on each segment, as shown in Figure 5.
[0037] Hereinafter, the inertial sensors 3 fixed to the respective segments of the waist, left thigh, right thigh, left shin, right shin, left foot, and right foot will be referred to as "inertial sensor 31," "inertial sensor 32," ..., "inertial sensor 37."
[0038] Identification numbers are assigned in advance to the inertial sensors 31, 32, ..., 37. In this embodiment, the inertial sensors 31, 32, ..., 37 are assigned identification numbers "1," "2," ..., "7," respectively. It can be said that the identification numbers "1," "2," ..., "7" identify the waist, left thigh, right thigh, left shin, right shin, left foot, and right foot, respectively.
[0039] Once these inertial sensors 3 are fixed at specific positions on each segment of the subject 1B1, the subject 1B1 stands upright so that the angles of the left hip joint, right hip joint, left knee joint, and right knee joint are all zero degrees. It is desirable to stand with the back of the head, back, buttocks, and calves against a vertical wall. This straight-standing state is the reference state.
[0040] Then, when the subject 1B1 is in a reference state, the operator calibrates the inertial sensors 31 to 37 by using the sensor application 41. This determines the position and posture of each of the seven segments (waist, left thigh, right thigh, left shin, right shin, left foot, and right foot) in the reference state, and the three-axis acceleration and angular velocity of each of these seven segments can be obtained based on the measurement results of the inertial sensors 31 to 37. The posture is expressed by a quaternion, but may also be expressed by Euler angles. Below, an example will be described in which the posture is expressed by a quaternion. Note that the traveling direction (forward direction) of the subject 1B1 can be determined by the position and posture of each of these seven segments, but it may also be determined by having the subject 1B1 walk a few steps.
[0041] Once the calibration is complete, the positions and postures of the seven segments, such as the waist, in the reference state are set in the learning and inference device 2. Then, the subject 1B1 continues walking for about 10 minutes.
[0042] The inertial sensors 31 to 37 measure the acceleration and angular velocity of each of the three axes at a sampling rate of 100 Hz while walking, and calculate the quaternion of the posture of the inertial sensors themselves. The inertial sensors 31 to 37 and the sensor application 41 then obtain the acceleration, angular velocity, and quaternion of each of the hips, left thigh, right thigh, left shin, right shin, left foot, and right foot at each time (every 1 / 100th of a second). These accelerations and angular velocities are input to a shift window value extraction unit 401 (see FIG. 3 ) as values in the local moving coordinate system for each inertial sensor 3, and the quaternions are input to a joint angle calculation unit 402.
[0043] Hereinafter, the accelerations at time t of the three axes of the moving coordinate system of a segment with identification number "s" will be described as "acceleration ax_s_t," "acceleration ay_s_t," and "acceleration az_s_t," the angular velocities at time t will be described as "angular velocity ωx_s_t," "angular velocity ωy_s_t," and "angular velocity ωz_s_t," and the quaternion will be described as "quaternion Qs_t." Furthermore, the time of the mth measurement will be represented as "m-1." Therefore, the times of the 1st, 2nd, 3rd, ... measurements will be "0," "1," "2," ...
[0044] While subject 1B1 is walking, the acceleration and angular velocity of each of the seven segments, including the waist, change. Furthermore, the angles of subject 1B1's joints also change. For example, the acceleration and angular velocity of the waist change as shown in FIG. 6(A). Furthermore, the angles of the left hip joint, right hip joint, left knee joint, and right knee joint change as shown in FIG. 6(B). These angles are calculated by the joint angle calculation unit 402, as will be described later.
[0045] In the learning and inference device 2, the acceleration and angular velocity of each segment are input to the shift window value extraction unit 401, and the quaternion of each segment is input to the joint angle calculation unit 402 (#102 in FIG. 4), and then the following process is performed.
[0046] The shift window value extraction unit 401 extracts values belonging to each of the shift windows 4A (4A1, 4A2, ...) of the multiple time periods from the input waist accelerations and angular velocities as follows (#103 to #106).
[0047] The u-th shift window 4A is a window that ranges from time "20(u-1)" to time "99+20(u-1)." Therefore, the first shift window 4A1 is a window that ranges from time 0 to 99, as shown in FIG. 6(A). The second shift window 4A2 is a window that ranges from time 20 to 119, that is, a range shifted 0.2 seconds to the right from the shift window 4A1. Similarly, the third and subsequent shift windows 4A are windows that range from the previous shift window 4A by 0.2 seconds to the right.
[0048] The shift window value extraction unit 401 sets the first shift window 4A as the current window (#103, #104). From the accelerations and angular velocities received from the inertial sensor 31, the accelerations and angular velocities that fit within the current window are extracted (#105). That is, first, the accelerations and angular velocities that fit within the shift window 4A1 are extracted.
[0049] The extracted acceleration and angular velocity are represented by a 100 x 6 matrix. For example, the acceleration and angular velocity extracted from the shift window 4A1 are Hereinafter, the extracted matrix will be referred to as "motion matrix B." In particular, the motion matrix B for the u-th time period (shift window 4A) will be referred to as "motion matrix Bu," and the motion matrix B for the u-th time period of the v-th subject 1B will be referred to as "motion matrix Bv_u."
[0050] The joint angle calculation unit 402 calculates the angles of the left hip joint, right hip joint, left knee joint, and right knee joint at the last time of the current window, that is, at the right end, as follows (#106).
[0051] It can be said that the angle of the left hip joint is the angle between the waist and the left thigh, and the angle of the right hip joint is the angle between the waist and the right thigh. Therefore, the joint angle calculation unit 402 calculates the angle between the waist and the left thigh as the left hip joint angle based on the quaternions at the end of the current window for each of the waist and the left thigh. For example, the angle of the left hip joint at the end of the first time period is calculated based on quaternions Q1_99 and Q2_99. Similarly, the angle of the right hip joint is calculated based on the quaternions at the end of the current window for each of the waist and the right thigh.
[0052] It can also be said that the angle of the left knee joint is the angle between the left thigh and left shin, and the angle of the right knee joint is the angle between the right thigh and right shin. Therefore, the joint angle calculation unit 402 calculates the angle between the left thigh and left shin segments as the angle of the left knee joint, based on the quaternions at the last time of the current window for each of the left thigh and left shin. It calculates the angle between the right knee joint segments, based on the quaternions at the last time of the current window for each of the right thigh and right shin.
[0053] Hereinafter, the calculated angles of the left hip joint, right hip joint, left knee joint, and right knee joint will be referred to as "angle θHL," "angle θHR," "angle θKL," and "angle θKR," respectively. In particular, the angles in the u-th time period will be referred to as "angle θHL_u," "angle θHR_u," "angle θKL_u," and "angle θKR_u," and the angles in the u-th time period of the v-th subject 1B will be referred to as "angle θHL_v_u," "angle θHR_v_u," "angle θKL_v_u," and "angle θKR_v_u."
[0054] The dataset registration unit 403 stores a dataset consisting of the attribute vector C acquired in step #101, the motion matrix Bu extracted by the shift window value extraction unit 401 in step #105, and the angles θHL_u, θHR_u, θKL_u, and θKR_u calculated by the joint angle calculation unit 402 in step #106 as dataset 50 in the dataset storage unit 404 (#107).
[0055] Then, the shift window value extraction unit 401, the joint angle calculation unit 402, and the data set registration unit 403 shift the current window to the right by 0.2 seconds (Yes in #108, #109), and perform the processes of steps #104 to #107. However, if the 100 x 6 accelerations and angular velocities are not complete even after the shift, that is, if there is no continuation (No in #108), the collection of the data set from subject 1B1 is terminated.
[0056] Furthermore, the shift window value extraction unit 401, the joint angle calculation unit 402, and the dataset registration unit 403 collect datasets 50 from each of the subjects 1B other than subject 1B1 using a similar method and store them in the dataset storage unit 404 (Yes in #110, #101 to #109).
[0057] Since the shift window 4A shifts by 0.2 seconds, about 3,000 shift windows 4A appear for one subject 1B during the walking time, i.e., 10 minutes. Therefore, about 3,000 sets of data sets 50 are obtained from one subject 1B. Therefore, if there are 200 subjects 1B, for example, about 600,000 sets of data sets 50 are obtained.
[0058] 7 is a diagram showing an example of the flow of a machine learning process, and FIG. 8 is a diagram showing an example of a neural network 60.
[0059] The machine learning unit 405 (see FIG. 3 ) generates a trained model for estimating the angles of the left hip joint, right hip joint, left knee joint, and right knee joint by performing machine learning based on the dataset 50 stored in the dataset storage unit 404. The machine learning method will be described below with reference to FIGS. 7 and 8 .
[0060] As shown in FIG. 7, the machine learning phase mainly includes a training step, a validation step, and a test step.
[0061] 8 is prepared in advance in the machine learning unit 405. The neural network 60 is made up of a first network 61, a second network 62, and a third network 63.
[0062] The first network 61 is a neural network of LSTM (Long Short-term Memory), which calculates and outputs a feature matrix indicating the features of the motion matrix B extracted by the shift window 4A.
[0063] The third network 63 is a network consisting of one or more connection layers, and outputs a row vector indicating the characteristics of attribute vector C. In this embodiment, it is configured by connection layer 63A. The connection layer 63A has 32×4 weighting coefficients and 32 bias coefficients, and when attribute vector C is input, it outputs a 32×1 row vector indicating the characteristics of attribute vector C.
[0064] The second network 62 is a network consisting of multiple fully connected layers, and in this embodiment, is composed of four fully connected layers 62A to 62D. The feature matrix output from the first network 61 is expanded into one dimension (256 x 1) and input to the fully connected layer 62A. The output values of all units in the fully connected layer 62A are input to each unit (node) of the fully connected layer 62B. The output values (128 x 1 row vector) from the fully connected layer 62B are merged with the output values from the connection layer 63A to form a 160 x 1 row vector. All values of the 160 x 1 row vector are then input to each unit of the fully connected layer 62C. In other words, the output values of all units in the connection layer 63A and all units in the fully connected layer 62B are input to each unit of the fully connected layer 62C. Furthermore, the output values of all units in the fully connected layer 62C are input to each unit of the fully connected layer 62D. Then, the angles ψHL, ψHR, ψKL, and ψKR of the left hip joint, right hip joint, left knee joint, and right knee joint, respectively, are output from the four units of the fully connected layer 62D.
[0065] A portion of the dataset 50 stored in the dataset storage unit 404 is selected as training data and used for learning the neural network 60. A portion of the remaining dataset 50 is selected as validation data and used for tuning the neural network 60. The remainder is selected as test data and used for final evaluation of the neural network 60. For example, 87% of the dataset 50 is selected as training data, 12% is selected as validation data, and 1% is selected as test data.
[0066] 7, the machine learning unit 405 alternately executes a training step and a validation step multiple times. In the training step, the neural network 60 is trained based on a data set 50 selected as training data. The validation may be performed using a holdout method, a cross-validation method, or any other known method.
[0067] The machine learning unit 405 uses, for example, a certain data set 50a to train the neural network 60 as follows: A motion matrix B included in the data set 50a is input as input data (explanatory variable data) to the first network 61, and an attribute vector C is input as input data to the third network 63, and calculation processing is performed on each layer of the first network 61, the second network 62, and the third network 63. As a result, angles ψHL, ψHR, ψKL, and ψKR of the left hip joint, right hip joint, left knee joint, and right knee joint, respectively, are calculated and output from the fully connected layer 62D.
[0068] Then, the machine learning unit 405 uses the data on the angles θHL, θHR, θKL, and θKR included in the data set 50a as teacher data (correct answer data) and adjusts each parameter (e.g., weighting coefficient or bias value) of the first network 61, the second network 62, and the third network 63. In other words, each parameter is adjusted so that the difference between the angle ψHL and the angle θHL, the difference between the angle ψHR and the angle θHR, the difference between the angle ψKL and the angle θKL, and the difference between the angle ψKR and the angle θKR are reduced.
[0069] The machine learning unit 405 trains the neural network 60 in a similar manner using other data sets 50 selected as training data.
[0070] In the verification step, the machine learning unit 405 verifies the output values (estimated values) of the trained neural network 60 using the dataset 50 selected as verification data, and tunes the hyperparameters of this neural network 60.
[0071] In the second training step, the machine learning unit 405 trains the neural network 60 tuned in the first verification step using the above-mentioned method, and in the second verification step, tunes the neural network 60 trained in the second training step using the above-mentioned method. The third and subsequent training and verification steps are also processed in the same manner.
[0072] Hereinafter, the neural network 60 trained and tuned by repeating such a set of training and validation steps (i.e., epochs) multiple times will be referred to as the "trained neural network 67." Then, in the test step, the trained neural network 67 is finally evaluated using the data set 50 selected as test data.
[0073] If the trained neural network 67 has a certain level of accuracy, the machine learning unit 405 stores the trained neural network 67 as a trained model in the trained model storage unit 406. If the trained neural network 67 does not have a certain level of accuracy, a new data set 50 may be prepared and each step may be repeated until a certain level of accuracy is achieved. Alternatively, the width of the shift window 4A may be changed, and the data set 50 may be collected again to perform the machine learning again.
[0074] 4. Inference FIG. 9 is a diagram showing an example of a trained neural network 67. As shown in FIG.
[0075] Once the machine learning is complete, the angles of the left hip joint, right hip joint, left knee joint, and right knee joint of the prediction target person 1A while he or she is walking can be estimated mainly by the trained neural network 67 and one inertial sensor 3. Hereinafter, this inertial sensor 3 will be referred to as "inertial sensor 38" to distinguish it from the inertial sensors 31 to 37. It is also assumed that the inertial sensor 38 is assigned the identification number "8."
[0076] Like the inertial sensor 37, the inertial sensor 38 is attached to a waist belt in a predetermined orientation. The person 1A to be inferred wears this belt so that the inertial sensor 38 is positioned at a specific position on the waist of the person 1A to be inferred. This fixes the inertial sensor 38 to a specific position on the waist of the person 1A to be inferred. This specific position is the same as the specific position when the dataset was prepared.
[0077] The person 1A to be inferred inputs his / her own attributes, namely, age, sex, weight, and height, to the learning and inference device 2. Hereinafter, the vector representing the input attributes will be referred to as "attribute vector C'".
[0078] When the person 1A begins walking, the inertial sensor 38 measures the acceleration and angular velocity along each of the three axes at a sampling rate of 100 Hz. The inertial sensor 38 or the sensor application 41 then obtains the acceleration and angular velocity of the hips at each time point (every 1 / 100th of a second) in the moving coordinate system, and inputs these values to the joint angle inference unit 407 (see FIG. 3 ).
[0079] Then, the joint angle inference unit 407 performs inference processing. The procedure of this processing will be described below with reference to FIG.
[0080] The joint angle inference unit 407 inputs a motion matrix B' representing the acceleration and angular velocity of the hip at the time when the joint angle is to be inferred and the acceleration and angular velocity of the hip for the most recent second to the first network 61 of the trained neural network 67, and inputs the attribute vector C' to the third network 63. The motion matrix B' is a 100 x 6 matrix, and the motion matrix B' representing the acceleration and angular velocity at time t and the acceleration and angular velocity for the most recent second is expressed as follows: is.
[0081] The joint angle inference unit 407 then obtains angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t from the fully connected layer 62D by performing calculations on the layers constituting each of the first network 61, the second network 62, and the third network 63. These angles are the angles of the left hip joint, right hip joint, left knee joint, and right knee joint of the person 1A to be inferred at time t, respectively.
[0082] The inference result output unit 408 outputs the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t at each time t obtained by the joint angle inference unit 407. For example, the inference result output unit 408 displays each angle as a line graph on the display 27. Alternatively, it generates table data showing each angle and transmits it to an external device.
[0083] Alternatively, the inference result output unit 408 may generate an animation of a pedestrian in which the left hip joint, right hip joint, left knee joint, and right knee joint change to angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t, respectively, as follows, and display it on the display 27:
[0084] The inference result output unit 408 prepares a three-dimensional model of the walker's body. While the walker is walking, angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t are obtained for each time t by the inertial sensor 38 or the like, and the three-dimensional model is modified so that the left hip joint, right hip joint, left knee joint, and right knee joint are at their respective angles. The modified three-dimensional model is then rendered to generate an image of the human body, which is then displayed on the display 27. This reproduces an animation.
[0085] It should be noted that the inertial sensor 38 can obtain the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t at a sampling rate of 100 Hz, but the three-dimensional model may be changed by thinning out the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t to match the refresh rate of the display 27.
[0086] Furthermore, the inference result output unit 408 may calculate the center of gravity of the person 1A to be inferred at each time t. The center of gravity may be calculated using a known method. For example, since the software of the MTw Awinda system described above has a function for calculating the center of gravity, the center of gravity may be calculated using this software.
[0087] Alternatively, the center of gravity may be calculated using AI technology, for example, as follows: Various postures of a person with an average build while walking are captured in advance, and the angles and center of gravity of each part (left hip joint, right hip joint, left knee joint, and right knee joint) in each posture are determined. A trained model for the center of gravity is generated by performing machine learning using the angles of each part as explanatory variable data and the center of gravity as training data.
[0088] Then, while the person 1A to be inferred is walking, the inference result output unit 408 inputs the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t into this trained model, thereby calculating the center of gravity at each time t.
[0089] Alternatively, a person with an average body type is made to assume various postures, and the postures are measured by motion capture, and the center of gravity for each posture is calculated. A trained model for the center of gravity is generated by performing machine learning using each posture as explanatory variable data and the center of gravity as training data.
[0090] Then, the inference result output unit 408 calculates the center of gravity at each time t by inputting the posture of the three-dimensional model for the animation described above into this trained model.
[0091] The center of gravity is defined as the position coordinate value of the foot that is on the ground on the reference coordinate system. However, when both feet are on the ground, the foot that landed first can be used as the reference. Also, the animation of the pedestrian may be displayed so that the position of the center of gravity is represented by a marker such as a point.
[0092] The inference result output unit 408 may further detect and output variations in the center of gravity and posture based on changes in each of these during walking.
[0093] 5. Overall Processing Flow and Effects of This Embodiment FIG. 10 is a flowchart illustrating an example of the overall processing flow of the joint angle estimation program 40. In FIG.
[0094] Next, the overall processing flow of the learning and inference device 2 will be described with reference to a flowchart.
[0095] The learning and inference device 2 executes the process of constructing an AI and the process of inferring a joint angle based on the joint angle estimation program 40 in the procedure shown in FIG.
[0096] In the dataset preparation phase, the learning and inference device 2 generates and stores a dataset 50 (#11 in FIG. 10). The method of these processes is as described in FIG. 4. By performing machine learning based on the dataset 50, the learning and inference device 2 generates and stores a trained neural network 67 (#12, #13). The method of these processes is as described in FIGS. 7 and 8.
[0097] Then, when the learning and inference device 2 acquires the hip acceleration, angular velocity, and attributes of the person to be inferred 1A (#14, #15), it infers the angles ψHL_t, ψHR_t, ψKL_t, and ψKR_t of the left hip joint, right hip joint, left knee joint, and right knee joint of the person to be inferred 1A while walking based on this information and the trained neural network 67 (#16) and outputs these (#17). The method of these processes is as described in Figure 9. Furthermore, an animation showing the appearance of the person to be inferred 1A may be displayed based on the inference result, or the center of gravity may be calculated and its position may be shown together with the animation.
[0098] According to this embodiment, the data required to generate a trained model for estimating four joint angles can be acquired by using seven inertial sensors 3 of the same type as sensors. This makes it possible to generate a trained model more efficiently than conventionally and identify the joint angles of an individual, such as a human, while they are walking. Furthermore, only one inertial sensor 3 needs to be attached to the person 1A to be inferred during inference. Therefore, it is possible to identify the joint angles with less burden on the person 1A to be inferred than conventionally.
[0099] 6. Modifications Fig. 11 is a diagram showing examples of MAE in each of the first to third patterns. Fig. 12 is a diagram showing an example of MAE when an inertial sensor 38 is fixed to each part. Fig. 13 is a diagram showing an example of a method for controlling a robotic prosthetic leg 71.
[0100] In this embodiment, the learning and inference device 2 performed machine learning using only the acceleration and angular velocity of the waist out of the seven segments as input data, but machine learning may also be performed using the acceleration and angular velocity of the left foot and the acceleration and angular velocity of the right foot as input data. In this case, instead of the motion matrix B shown in Figure 8, a 100 x 18 matrix as shown below, in which data for the waist, left foot, and right foot are arranged in the column direction, can be used as input data.
[0101] Furthermore, during inference, not only is an inertial sensor 38 attached to a specific position on the waist of the person 1A to be inferred, but one inertial sensor 3 is attached to each specific position on the left and right feet, and the learning and inference device 2 determines the acceleration and angular velocity of the waist, left foot, and right foot using these three inertial sensors 3. A 100×18 matrix of these accelerations and angular velocities is input into the first network 61 (see FIG. 9 ) of the trained neural network 67, thereby inferring the angles of each of the four joints of the person 1A to be inferred.
[0102] In this embodiment, the learning and inference device 2 uses four attributes, age, sex, weight, and height, as input data for machine learning, but any one to three of these attributes may be used. Alternatively, other attributes may be used. Alternatively, machine learning may be performed without using the attribute vector C. In this case, the third network 63 is not provided in the neural network 60 (see FIG. 8). Furthermore, only output values from the fully connected layer 62C are input to each unit of the fully connected layer 62D of the second network 62.
[0103] In this embodiment, an LSTM neural network is used as the first network 61 (see FIGS. 8 and 9 ). However, other types of neural networks may also be used. For example, a convolutional neural network (CNN), a bidirectional LSTM (BLSTM), or a wavelet neural network (WNN) may also be used. When a CNN is used, the motion matrix B is converted into a grayscale image in machine learning and input to the first network 61. Similarly, the motion matrix B′ is converted into a grayscale image in inference and input to the first network 61. Instead of a grayscale image, a line graph image such as that shown in the shift window 4A in FIG. 6A may also be used.
[0104] Here, the accuracy of inference will be examined based on the mean absolute error (MAE) of the inference for each of the following three patterns (1) to (3).
[0105] (1) First Pattern In the first pattern, machine learning is performed by acquiring data on the waist, left and right thighs, left and right shins, and left and right feet using seven inertial sensors 3 (31 to 37), and inference is performed by acquiring data on the waist using inertial sensor 38. The attribute vector C is not used.
[0106] Experiments were conducted using the first pattern using neural networks WNN, LSTM, BLSTM, and CNN, and the results shown in Figure 11(A) were obtained. The results show that the MAE of each of the four neural networks varies depending on the target of inference, but the MAE generally falls within the range of 6 to 9. However, overall, BLSTM appears to have the highest accuracy.
[0107] (2) Second Pattern The second pattern is a modification of the first pattern, in which attribute vectors (age, gender, weight, and height) are used in both machine learning and inference. Experiments were conducted using the second pattern with each of the four neural networks, and the results shown in FIG. 11(B) were obtained. The results show that the total MAE is lower than in the first pattern, indicating that using attribute vectors provides a certain level of effectiveness.
[0108] (3) Third Pattern The third pattern is a modification of the first pattern, in which inference is performed by adding inertial sensors 3 for the left and right feet. Experiments were conducted using the third pattern using each of the four neural networks, and the results shown in Figure 11(C) were obtained. The results show that all MAEs are lower than those of the first pattern.
[0109] These experiments show that the more types of data there are, the higher the accuracy becomes, but the more types of data there are, the longer it takes to prepare for inference. Therefore, it is best to use different patterns depending on the environment or purpose.
[0110] In this embodiment, the learning and inference device 2 performs machine learning using the acceleration and angular velocity of the waist as input data, but machine learning may alternatively be performed using the acceleration and angular velocity of any one of the left thigh, right thigh, left shin, right shin, left foot, and right foot as input data. For example, when the acceleration and angular velocity of the left thigh are used as input data, the following 100×6 matrix may be used as input data instead of the motion matrix B shown in FIG. 8 .
[0111] During inference, an inertial sensor 38 is fixed to a specific position on the left thigh of the person to be inferred 1A instead of a specific position on the waist, and the learning and inference device 2 finds the acceleration and angular velocity of the waist using this inertial sensor 38. A 100×6 matrix of these accelerations and angular velocities is input to the first network 61 (see FIG. 9 ) of the trained neural network 67, thereby inferring the angles of each of the four joints of the person to be inferred 1A.
[0112] The MAEs obtained when machine learning and inference are performed using BLSTM with the inertial sensors 38 fixed to the waist, thighs, shins, and feet are shown in Fig. 12. The MAE corresponding to the waist among the MAEs shown in Fig. 12 is the same as the MAE corresponding to BLSTM among the MAEs shown in Fig. 11(A).
[0113] 12, it can be seen that fixing the inertial sensor 38 to the thigh, shin, or foot results in better inference than fixing it to the waist. Fixing it to the shin is the best. However, since it is easier to fix it to the waist than to the shin, it is best to choose the fixing location depending on the environment or purpose.
[0114] However, it may be difficult for subject 1B to stand so that the angle of his or her left hip joint is zero degrees. Therefore, the learning and inference device 2 may set the actual angle of the left hip joint during calibration as the reference angle, and the difference between the reference angle and the actual angle of the left hip joint while walking may be displayed as the left hip joint angle in the data set 50. The same applies to the right hip joint, left knee joint, and right knee joint.
[0115] In this embodiment, the learning and inference device 2 generates a versatile trained neural network 67 by performing machine learning using a dataset 50 collected from multiple subjects 1B. However, a trained neural network 67 dedicated to a specific subject 1B may also be generated by collecting a dataset 50 from one specific subject 1B and performing machine learning on that dataset. In this case, there is also no need to provide a third network 63 in the neural network 60, and only output values from the fully connected layer 62C are input to each unit of the fully connected layer 62D of the second network 62.
[0116] In this embodiment, an example has been described in which a neural network (trained model) is generated for inferring the angles of the hip joint and the knee joint, but according to the present invention, it is also possible to generate a neural network for inferring the angles of other joints.
[0117] For example, a neural network for inferring the angles of each of the left and right ankle joints (ankle joints) may be generated. In this case, in the phase of preparing a data set, the learning and inference device 2 acquires the angle of the left ankle joint as correct answer data based on the quaternions of the left shin and left foot, and acquires the angle of the right ankle joint as correct answer data based on the quaternions of the right shin and right foot, and performs machine learning.
[0118] Alternatively, a neural network for inferring the leg spread angle in the forward-backward direction (the angle of both legs) may be generated. In this case, in the phase of preparing a data set, the learning and inference device 2 acquires the leg spread angle in the forward-backward direction as correct answer data based on the quaternions of the left thigh and the right thigh, and performs machine learning.
[0119] Alternatively, a neural network for inferring the angles of the left and right shoulder joints in the front-to-back direction may be generated. In this case, one inertial sensor 3 is fixed to each of the left and right upper arms, the angle of the left shoulder joint is obtained as correct data based on the quaternions of the waist and the left upper arm, and the angle of the right shoulder joint is obtained as correct data based on the quaternions of the waist and the right upper arm, and machine learning may be performed.
[0120] The learning and reasoning device 2 can be used in various fields such as medicine, sports, and entertainment. For example, the learning and reasoning device 2 can be used as follows for the rehabilitation of a patient.
[0121] A medical institution staff member fixes an inertial sensor 38 to a specific position on the patient's waist and inputs the patient's attribute values of age, sex, weight, and height into the learning and inference device 2. When the patient starts walking, the acceleration and angular velocity of the waist at each time point (for example, every 1 / 100th of a second) in the moving coordinate system are obtained and input into the learning and inference device 2.
[0122] Then, the learning and inference device 2 estimates (infers) the angles HL_t, ψHR_t, ψKL_t, and ψKR_t of the patient's left hip joint, right hip joint, left knee joint, and right knee joint at each time t using the method illustrated in Figure 9. Furthermore, the learning and inference device 2 calculates the center of gravity. The patient's walking state is then displayed as an animation. At this time, changes in the position of the center of gravity are also reproduced. Furthermore, variations in posture and center of gravity during walking are inspected and output.
[0123] The staff member looks at the animation and the variations in posture and center of gravity together with the patient, and gives the patient advice on a desirable way of walking. The learning and inference device 2 may further display an animation showing an ideal way of walking next to the animation of the patient.
[0124] A unique ID may be assigned to each patient, and patient information (such as name, address, contact details, facial photograph, medical records, information on prosthetic devices such as artificial limbs or legs, rehabilitation history, signed consent to disclaimers, or code information to prevent tampering of this information) may be associated with the ID and managed in the learning and inference device 2. By referring to such information, staff can support rehabilitation more efficiently and reliably than before.
[0125] The angle inferred by the learning and inference device 2 may be used to control the prosthetic leg. Here, the control method will be described using as an example a case where the person to be inferred 1A2 uses a robotic prosthetic leg 71 for the left leg, as shown in Figure 13.
[0126] The robotic prosthetic leg 71 corresponds to the part of the left foot from just below the knee to the toes of a human, and has a joint at the part that corresponds to the ankle. The joint has roughly the same range of motion as the ankle joint of a human left foot, and can bend and extend using a motor and a controller that controls the motor.
[0127] The inertial sensor 38 and the robotic prosthetic leg 71 are fixed to predetermined positions on the person 1A2 to be estimated, and the attributes of the person 1A2, namely, age, sex, weight, and height, are input to the learning and inference device 2 in advance.
[0128] When the person 1A2 to be inferred starts walking, the acceleration and angular velocity of the hips at each time in the moving coordinate system are obtained and input to the learning and inference device 2. The learning and inference device 2 infers the angle of each joint at each time t using the method illustrated in Fig. 9 and transmits the angle of the left ankle joint to the controller of the robotic prosthetic leg 71.
[0129] The controller of the robotic prosthetic leg 71 then controls the motor based on the transmitted angle (command angle). For example, the motor is controlled so that the joint of the robotic prosthetic leg 71 forms the command angle. Driving the joint of the robotic prosthetic leg 71 in this way enables natural walking.
[0130] In this embodiment, as shown in Figures 6(A) and (B), the learning and inference device 2 performed machine learning using the acceleration and angular velocity at each time in the last second, including the time at which the joint angle was formed, as input data that forms a pair with the joint angle (correct answer data).However, machine learning may also be performed using the acceleration and angular velocity at each time in the immediately following time period, or may also be performed using the acceleration and angular velocity at each time in both the immediately preceding and immediately following time periods.
[0131] That is, for example, if the time when the joint angle is formed is "T", In this case, the acceleration and angular velocity at each time in the immediately following time period are input to the trained neural network 67 in the same way as in the case of inference.
[0132] or may be used as input data, where α+β=100. In this case, the acceleration and angular velocity at each time in both the immediately preceding and following time periods are input to the trained neural network 67 in the same way during inference.
[0133] In this embodiment, as shown in FIG. 6 , the learning and inference device 2 uses a shift window with a time width of 1 second and a shift width of 0.2 seconds as the shift window 4A to acquire the data set 50. However, shift windows with other time widths or shift widths may also be used. For example, a shift window with a time width of 1.2 seconds and a shift width of 0.5 seconds may be used. In this case, a matrix corresponding to the shift width (a matrix of acceleration and angular velocity) is used as the motion matrix B' in the inference. In other words, if the time width of the shift window 4A is 1.2 seconds, a 120×6 matrix is used.
[0134] In this embodiment, the inertial sensors 31, 38 are attached in a predetermined orientation to a dedicated belt, and the belt is worn at a specific position on the waist of the subject 1B or the person being estimated 1A, so that the inertial sensors 31, 38 are fixed in a specific orientation at a specific position on the waist. However, the inertial sensors 31, 38 may be fixed slightly shifted or tilted from the specific position. Therefore, during machine learning and inference, calibration may be performed so that the positions and orientations of the inertial sensors 31, 38 in a reference state are regarded as initial positions.
[0135] In this embodiment, the learning and inference device 2 acquires the angles of the left hip joint, right hip joint, left knee joint, and right knee joint based on the position and orientation of each segment determined by the inertial sensors 31 to 35, but these angles may be acquired by other methods. For example, markers may be attached to each segment and the position and orientation of each segment may be determined by photographing the markers with one or more cameras, and each angle may be calculated. In other words, these angles may be calculated by so-called optical motion capture. Alternatively, the three-dimensional position of each segment may be determined with one or more depth cameras, and each angle may be calculated. Alternatively, each angle may be acquired using a wearing device such as that described in Patent Document 2.
[0136] In this embodiment, a neural network for inferring joint angles during walking was generated as the trained neural network 67, but it is also possible to generate a neural network for inferring angles during other actions such as jumping, running, climbing stairs, descending stairs, cycling, and skiing.
[0137] In this embodiment, a neural network for inferring the angles of human joints was generated as the trained neural network 67, but it is also possible to generate a neural network for inferring the angles of joints of animals other than humans, such as dogs, cats, and horses, as long as they have a torso and limbs.
[0138] In this embodiment, the functions shown in FIG. 3 are provided in the learning and inference device 2, but they may be distributed across multiple devices. For example, the shift window value extraction unit 401, joint angle calculation unit 402, dataset registration unit 403, and dataset storage unit 404 may be provided in a first computer, the machine learning unit 405 may be provided in a second computer, and the learned model storage unit 406, joint angle inference unit 407, and inference result output unit 408 may be provided in a third computer. The first computer, the second computer, and the third computer are connected via a communication line such as the Internet, a LAN (Local Area Network), or a public line. The first computer, the second computer, and the third computer perform the following processes based on the first computer program, the second computer program, and the third computer program, respectively.
[0139] The first computer transmits the dataset 50 stored in the dataset storage unit 404 to the second computer along with a machine learning command. The second computer then generates a trained neural network 67 by performing machine learning using the received dataset 50, and transmits the trained neural network 67 to the third computer. The third computer stores the trained neural network 67 in the trained model storage unit 406. The trained neural network 67 is then used to estimate the four joint angles of the person 1A to be estimated.
[0140] In addition, the overall or individual configurations of the joint angle estimation system 1 and the learning and inference device 2, the processing content, the processing order, the data set configuration, the range of the shift window 4A, etc. can be modified as appropriate in accordance with the spirit of the present invention.
[0141] 1 Joint angle estimation system 1B Subject (learning subject individual) 2 Learning and inference device (individual angle learning system, individual angle estimation device) 31 Inertial sensor (first inertial sensor) 37 Inertial sensor (second inertial sensor) 401 Shift window value extraction unit (first acquisition means) 402 Joint angle calculation unit (acquisition means, second acquisition means) 405 Machine learning unit (learning means) 407 Joint angle inference unit (estimation means) 50 Data set (input data, correct answer data) 67 Trained neural network (trained model)
Claims
1. A joint angle learning and estimation system for estimating a joint angle between a first predetermined portion and a second predetermined portion of an individual, comprising: a first inertial sensor attached to a third predetermined portion, which is neither the first predetermined portion nor the second predetermined portion, of a learning subject individual that is a learning subject; an acquisition means for acquiring an angle of a joint between a first predetermined portion and a second predetermined portion of the training object as a correct angle; A learning means for generating a trained model by performing machine learning using the acceleration and angular velocity acquired by the first inertial sensor as input data and the correct angle acquired by the acquisition means as correct data; a second inertial sensor attached to a third predetermined portion of the estimation target individual that is the subject of estimation; an estimation means for estimating an angle of a joint between a first predetermined portion and a second predetermined portion of the estimation target individual by inputting the acceleration and angular velocity acquired by the second inertial sensor into the trained model; A joint angle learning and estimation system comprising:
2. A joint angle learning system that generates a trained model for estimating a joint angle between a first predetermined portion and a second predetermined portion of an individual, comprising: a first acquisition means for acquiring acceleration and angular velocity of a third predetermined portion, which is neither the first predetermined portion nor the second predetermined portion, of a learning subject individual that is a learning subject; A second acquisition means for acquiring an angle of a joint between a first predetermined portion and a second predetermined portion of the learning object as a correct angle; A learning means for generating the trained model by performing machine learning using the acceleration and angular velocity acquired by the first acquisition means as input data and the correct angle acquired by the second acquisition means as correct data; A joint angle learning system comprising:
3. The learning means generates the trained model by performing machine learning using, as input data, the acceleration and angular velocity in a time period of a predetermined length immediately before or after the time when the correct angle is formed. The joint angle learning system according to claim 2 .
4. (delete)
5. (delete)
6. (delete)
7. The learning means generates the trained model by performing machine learning using the age, sex, weight, or height of the training subject individual as the input data.
21. A joint angle learning system according to claim 2, 3, or 13 to 20.
8. (delete)
9. a prediction means for predicting a joint angle between a first predetermined part and a second predetermined part of an individual to be predicted by inputting an acceleration and an angular velocity of a third predetermined part, which is neither the first predetermined part nor the second predetermined part, of the individual to be predicted to a learned model generated by the joint angle learning system of any one of claims 2, 3, 13 to 16; A joint angle estimation device comprising:
10. A joint angle learning method for generating a trained model for estimating an angle of a joint between a first predetermined portion and a second predetermined portion of an individual, comprising: acquiring acceleration and angular velocity of a third predetermined portion, which is neither the first predetermined portion nor the second predetermined portion, of a learning subject individual that is a learning subject, using a first inertial sensor attached to the third predetermined portion, which is neither the first predetermined portion nor the second predetermined portion; acquiring an angle of a joint between a first predetermined portion and a second predetermined portion of the learning object as a correct angle; Generate a trained model by having a computer perform the machine learning using the acquired acceleration and angular velocity as input data and the acquired correct angle as correct data. A joint angle learning method comprising:
11. A computer program for use in a computer generating a trained model for estimating a joint angle between a first predetermined portion and a second predetermined portion of an individual, comprising: The computer includes: executing a first acquisition process for acquiring acceleration and angular velocity of a third predetermined portion, which is neither the first predetermined portion nor the second predetermined portion, of a learning subject individual that is a learning subject; executing a second acquisition process for acquiring an angle of a joint between a first predetermined portion and a second predetermined portion of the learning object as a correct angle; Executing a learning process to generate the trained model by having a machine learning system perform machine learning using the acceleration and angular velocity acquired in the first acquisition process as input data and the correct angle acquired in the second acquisition process as correct data; A computer program comprising:
12. On the computer, execute a process of acquiring acceleration and angular velocity of a third predetermined part of the estimation target individual that is the subject of estimation, which is neither the first predetermined part nor the second predetermined part; and executing a process of estimating a joint angle between a first predetermined portion and a second predetermined portion of the estimation target individual by inputting the acquired acceleration and angular velocity into a learned model generated by the joint angle learning system according to any one of claims 2, 3, 13 to 16. A computer program comprising:
13. the combination of the first and second predetermined parts is a combination of a left thigh and a left shin, a combination of a right thigh and a right shin, a combination of a left shin and a left foot, or a combination of a right shin and a right foot; The third predetermined region is the waist or the torso. The joint angle learning system according to claim 2 or 3.
14. The third predetermined part is the waist; The second acquisition means further acquires an angle of a joint between the left thigh or the right thigh and a waist of the training object individual as a second correct angle; The learning means further generates a second trained model for estimating an angle of a joint between the left thigh or the right thigh and a hip by performing machine learning using the second correct angle as correct answer data. The joint angle learning system according to claim 13.
15. the combinations of the first and second predetermined parts are a combination of a left thigh and a left shin, a combination of a right thigh and a right shin, a combination of a left shin and a left foot, and a combination of a right shin and a right foot; the third predetermined region is the waist or the torso; The learning means generates, as the trained models, trained models for estimating a joint between a left thigh and a left shin, a joint between a right thigh and a right shin, a joint between a left shin and a left foot, and a joint between a right shin and a right foot, The joint angle learning system according to claim 2 or 3.
16. The third predetermined part is the waist; The second acquisition means further acquires angles of joints between the waist and the left thigh and between the waist and the right thigh of the training object individual as second correct angles; The learning means further generates a second trained model for estimating each of an angle of a joint between a waist and a left thigh and an angle of a joint between a waist and a right thigh by performing machine learning using the second correct angle as correct answer data. The joint angle learning system according to claim 15.
17. the third predetermined part is any one of the left thigh, the right thigh, the left shin, the right shin, the left foot, and the right foot; When the third predetermined part is the left thigh, the combination of the first predetermined part and the second predetermined part is a combination of the waist and the right thigh, a combination of the right thigh and the right shin, a combination of the left shin and the left foot, or a combination of the right shin and the right foot; When the third predetermined part is the right thigh, the combination of the waist and the left thigh, the combination of the left thigh and the left shin, the combination of the left shin and the left foot, or the combination of the right shin and the right foot, When the third predetermined part is the left shin, the combination is a combination of the waist and the left thigh, a combination of the waist and the right thigh, a combination of the right thigh and the right shin, or a combination of the right shin and the right foot; When the third predetermined part is the right shin, the combination of the waist and the left thigh, the combination of the waist and the right thigh, the combination of the left thigh and the left shin, or the combination of the left shin and the left foot, When the third predetermined part is the left leg, the combination is a combination of the waist and the left thigh, a combination of the waist and the right thigh, a combination of the left thigh and the left shin, a combination of the right thigh and the right shin, or a combination of the right shin and the right leg; When the third predetermined part is the right leg, the combination is a combination of the waist and the left thigh, a combination of the waist and the right thigh, a combination of the left thigh and the left shin, a combination of the right thigh and the right shin, or a combination of the left shin and the left leg. The joint angle learning system according to claim 2 or 3.
18. When the third predetermined part is the left thigh, The second acquisition means further acquires an angle of a joint between a waist and a left thigh or an angle of a joint between a left thigh and a left shin of the learning object as a second correct angle; The learning means further generates a second trained model for estimating an angle of a joint between a waist and a left thigh or an angle of a joint between a left thigh and a left shin by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the right thigh, The second acquisition means further acquires an angle of a joint between a waist and a right thigh or an angle of a joint between a right thigh and a right shin of the learning object individual as a second correct angle; The learning means further generates a second trained model for estimating an angle of the joint between the waist and the right thigh or an angle of the joint between the right thigh and the right shin by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the left shin, The second acquisition means further acquires an angle of a joint between a left thigh and a left shin or an angle of a joint between a left shin and a left foot of the learning object as a second correct angle; The learning means further generates a second trained model for estimating an angle of a joint between the left thigh and the left shin or an angle of a joint between the left shin and the left foot by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the right shin, The second acquisition means further acquires an angle of a joint between a right thigh and a right shin or an angle of a joint between a right shin and a right foot of the learning subject individual as a second correct angle; The learning means further generates a second trained model for estimating an angle of the joint between the right thigh and the right shin or an angle of the joint between the right shin and the right foot by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the left foot, The second acquisition means further acquires an angle of a joint between a left shin and a left foot of the training object individual as a second correct angle; The learning means further generates a second trained model for estimating an angle of a joint between the left shin and the left foot by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the right foot, The second acquisition means further acquires an angle of a joint between a right shin and a right foot of the training object individual as a second correct angle; The learning means further generates a second trained model for estimating an angle of a joint between the right shin and the right foot by performing machine learning using the second correct angle as correct answer data. The joint angle learning system according to claim 17.
19. the third predetermined part is any one of the left thigh, the right thigh, the left shin, the right shin, the left foot, and the right foot; If the third predetermined part is the left thigh, the combinations of the first and second predetermined regions are a combination of the waist and the right thigh, a combination of the right thigh and the right shin, a combination of the left shin and the left foot, and a combination of the right shin and the right foot; The learning means generates, as the trained models, trained models for estimating a joint between a waist and a right thigh, a joint between the right thigh and a right shin, a joint between a left shin and a left foot, and a joint between the right shin and a right foot, If the third predetermined part is the right thigh, the combinations of the first and second predetermined regions are a combination of the waist and the left thigh, a combination of the left thigh and the left shin, a combination of the left shin and the left foot, and a combination of the right shin and the right foot; the learning means generates, as the trained models, trained models for estimating a joint between a waist and a left thigh, a joint between the left thigh and a left shin, a joint between the left shin and a left foot, and a joint between the right shin and a right foot; If the third predetermined part is the left shin, the combinations of the first and second predetermined regions are a combination of the waist and the left thigh, a combination of the waist and the right thigh, a combination of the right thigh and the right shin, and a combination of the right shin and the right foot; The learning means generates, as the trained models, trained models for estimating a joint between a waist and a left thigh, a joint between a waist and a right thigh, a joint between a right thigh and a right shin, and a joint between a right shin and a right foot, If the third predetermined part is the right shin, The combinations of the first and second predetermined regions are a combination of the waist and the left thigh, a combination of the waist and the right thigh, a combination of the left thigh and the left shin, and a combination of the left shin and the left foot, The learning means generates, as the trained models, trained models for estimating a joint between a waist and a left thigh, a joint between a waist and a right thigh, a joint between a left thigh and a left shin, and a joint between a left shin and a left foot, If the third predetermined part is the left foot, the combinations of the first and second predetermined regions are a combination of the waist and the left thigh, a combination of the waist and the right thigh, a combination of the left thigh and the left shin, a combination of the right thigh and the right shin, and a combination of the right shin and the right foot; the learning means generates, as the trained models, trained models for estimating a joint between a waist and a left thigh, a joint between a waist and a right thigh, a joint between a left thigh and a left shin, a joint between a right thigh and a right shin, and a joint between a right shin and a right foot; If the third predetermined part is the right foot, the combinations of the first and second predetermined regions are a combination of the waist and the left thigh, a combination of the waist and the right thigh, a combination of the left thigh and the left shin, a combination of the right thigh and the right shin, and a combination of the left shin and the left foot; The learning means generates, as the learned models, learned models for estimating a joint between a waist and a left thigh, a joint between a waist and a right thigh, a joint between a left thigh and a left shin, a joint between a right thigh and a right shin, and a joint between a left shin and a left foot, The joint angle learning system according to claim 2 or 3.
20. When the third predetermined part is the left thigh, The second acquisition means further acquires an angle of a joint between a waist and a left thigh and an angle of a joint between the left thigh and a left shin of the learning object individual as second correct angles; The learning means further generates a second trained model for estimating an angle of the joint between the waist and the left thigh and an angle of the joint between the left thigh and the left shin by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the right thigh, The second acquisition means further acquires an angle of a joint between a waist and a right thigh and an angle of a joint between the right thigh and a right shin of the learning object individual as second correct angles; The learning means further generates a second trained model for estimating an angle of the joint between the waist and the right thigh and an angle of the joint between the right thigh and the right shin by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the left shin, The second acquisition means further acquires an angle of a joint between a left thigh and a left shin and an angle of a joint between a left shin and a left foot of the training subject individual as a second correct angle; The learning means further generates a second trained model for estimating an angle of the joint between the left thigh and the left shin and an angle of the joint between the left shin and the left foot by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the right shin, The second acquisition means further acquires an angle of a joint between the right thigh and the right shin and an angle of a joint between the right shin and the right foot of the learning object individual as second correct angles; The learning means further generates a second trained model for estimating an angle of the joint between the right thigh and the right shin and an angle of the joint between the right shin and the right foot by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the left foot, The second acquisition means further acquires an angle of a joint between a left shin and a left foot of the training object individual as a second correct angle; The learning means further generates a second trained model for estimating an angle of a joint between the left shin and the left foot by performing machine learning using the second correct angle as correct answer data; When the third predetermined part is the right foot, The second acquisition means further acquires an angle of a joint between a right shin and a right foot of the training object individual as a second correct angle; The learning means further generates a second trained model for estimating an angle of a joint between the right shin and the right foot by performing machine learning using the second correct angle as correct answer data. The joint angle learning system according to claim 19.
