Posture estimation device, posture estimation system, and posture estimation method

The posture estimation device uses machine learning to estimate whole body posture from smart shoes sensors, addressing the challenges of continuous monitoring with fewer sensors, enabling daily tracking.

JP2024164991APending Publication Date: 2024-11-28ASICS CORP

Patent Information

Application Number
JP2023080778
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing methods for continuously measuring whole body posture, such as motion capture and wearable devices, face challenges in capturing the entire body range or requiring multiple sensors, making daily monitoring cumbersome.

Method used

A posture estimation device using machine learning to estimate whole body posture from data acquired by sensors attached to the subject, such as smart shoes, which include 3-axis acceleration and angular velocity sensors, and an estimation model trained on walking data to reduce the need for multiple sensors.

Benefits of technology

Enables continuous monitoring of whole body posture with a reduced number of sensors, allowing daily tracking without the burden of extensive equipment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164991000001_ABST
    Figure 2024164991000001_ABST
Patent Text Reader

Abstract

To provide a posture estimation device, posture estimation system, and posture estimation method capable of continuously grasping the full-body posture on a daily basis.SOLUTION: A posture estimation device 10 includes: at least one sensor module 21 which is attached to a subject P and acquires walking data based on the three-axis acceleration and three-axis angular velocity at least one location of a subject P; a computing device 1 which estimates the full-body posture of the subject P in a prescribed operation from the walking data acquired by the sensor module 21 by using an estimation model previously trained through machine learning; and an output interface which outputs the estimation data of the full-body posture of the subject P estimated by the computing device 1. The estimation model is generated in advance through training with machine learning on the basis of the walking data acquired by the sensor module 21 in the prescribed operation and posture data obtained by measuring the full-body posture of the subject P at the timing when the walking data was acquired.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a posture estimation device, a posture estimation system, and a posture estimation method. [Background technology]

[0002] In order to improve walking and running form, it is necessary to continuously measure the whole body posture while walking or running. In addition, research is being conducted to detect mental illness and estimate the risk of falling from walking posture, and these researches require daily measurement of posture while walking. Methods for measuring whole body posture include using motion capture, using moving images, and using wearable devices.

[0003] For example, JP 2023-16825 A (Patent Document 1) discloses a system for measuring whole-body posture using a wearable device. In this system, six sensor devices are attached to six parts of the subject's body, specifically, to joints that serve as reference points for the body (e.g., the waist and head), or near the extremities of the body (wrists, ankles, head, etc.), and the whole-body posture is measured based on the measured values ​​from the sensor devices. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-16825 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the method using motion capture or the method using moving images, it is necessary to keep the whole body in the shooting range at all times, and it was difficult to keep shooting the whole body in the shooting range at all times in order to grasp the whole body posture continuously on a daily basis. On the other hand, in the method using a wearable device, the posture can be measured by having the subject wear the wearable device, so it is suitable for grasping the posture continuously on a daily basis, but it is only possible to grasp the partial posture including the part where the wearable device is worn. Therefore, in order to grasp the whole body posture using a wearable device, it is necessary to wear many sensor devices near the joints and extremities of the body as shown in JP 2020-120908 A (Patent Document 1), and it was difficult to continue wearing many sensors to grasp the whole body posture continuously on a daily basis.

[0006] The present disclosure has been made to solve such problems, and its purpose is to provide a posture estimation device, a posture estimation system, and a posture estimation method that are capable of continuously grasping the entire body posture on a daily basis. [Means for solving the problem]

[0007] A posture estimation device according to an aspect of the present disclosure is a posture estimation device that estimates the whole body posture of a person being measured during a predetermined movement. The posture estimation device includes at least one sensor attached to the person being measured and acquiring measurement data based on three-axis acceleration and three-axis angular velocity of at least one location of the person being measured, a calculation circuit that estimates the whole body posture of the person being measured during a predetermined movement from the measurement data acquired by the sensor using an estimation model trained by machine learning, and an output device that outputs the estimated data of the whole body posture of the person being measured estimated by the calculation circuit. The estimation model is generated in advance by training using machine learning based on the measurement data acquired by the sensor during the predetermined movement and posture data measuring the whole body posture of the person being measured at the timing when the measurement data is acquired.

[0008] A posture estimation system according to one aspect of the present disclosure includes a measurement device that measures posture data and the posture estimation device described above, and uses the posture data measured by the measurement device when training an estimation model using machine learning.

[0009] A posture estimation method according to an aspect of the present disclosure is a posture estimation method for estimating the whole body posture of a person being measured during a predetermined movement. The posture estimation method includes the steps of acquiring measurement data based on three-axis acceleration and three-axis angular velocity of at least one location of the person being measured from at least one sensor attached to the person being measured, estimating the whole body posture of the person being measured during a predetermined movement from the measurement data acquired by the sensor using an estimation model trained by machine learning, and outputting the estimated data of the estimated whole body posture of the person being measured. The estimation model is generated in advance by training using machine learning based on the measurement data acquired by the sensor during the predetermined movement and posture data measuring the whole body posture of the person being measured at the time when the measurement data was acquired. Effect of the Invention

[0010] According to the present disclosure, an estimation model trained by machine learning is used to estimate the entire body posture of the subject during a specified movement from measurement data acquired by a sensor, making it possible to continuously grasp the entire body posture on a daily basis using a small number of sensors. [Brief description of the drawings]

[0011] [Figure 1] 1 is a schematic diagram illustrating a configuration of a posture estimation system including a posture estimation device according to an embodiment. [Diagram 2] 1A and 1B are schematic diagrams for explaining smart shoes according to an embodiment. [Diagram 3] 1 is a block diagram showing a configuration of a posture estimation device according to an embodiment. [Figure 4] 4 is a flowchart for explaining the processing of a posture estimation method executed by the posture estimation system according to the embodiment. [Diagram 5]1A and 1B are diagrams illustrating a training phase and an operation phase in a posture estimation system according to an embodiment. [Figure 6] FIG. 2 shows a schematic diagram of a calibration phase in the posture estimation system according to the embodiment. [Figure 7] FIG. 1 is a schematic diagram for explaining one walking cycle. [Figure 8] 5 is a diagram illustrating a schematic diagram of calculation of an estimation model in the posture estimation system according to the embodiment. FIG. [Figure 9] 1 is a diagram showing whole-body skeletal coordinate data estimated by a posture estimation device according to an embodiment, viewed from the front. [Figure 10] 1 is a diagram showing whole-body skeletal coordinate data estimated by a posture estimation device according to an embodiment, viewed from the side. FIG. [Figure 11] 11 is a graph showing a change in error over one walking period in arm skeletal coordinate data estimated by the posture estimation device according to the embodiment. [Figure 12] 11 is a graph showing a change in error over one walking period in lower body skeletal coordinate data estimated by the posture estimation device according to the embodiment. [Figure 13] 11 is a graph showing a change in error over one walking period in head skeletal coordinate data estimated by the posture estimation device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present disclosure provides a posture estimation device, a posture estimation system, and a posture estimation method capable of estimating the whole body posture during a predetermined motion (for example, walking motion, running motion (running motion)) from information of a sensor provided in a shoe. In the following description, a case where a sensor is provided in a shoe and the whole body posture during walking is estimated, but the motion of the whole body posture to be estimated is not limited to walking, and the position where the sensor is attached is not limited to the shoe. In particular, it is preferable that the part and the number of sensors are parts and numbers that do not burden the subject to be measured continuously on a daily basis. For example, a sensor may be attached to both arms, a sensor may be attached to one arm and a waist, or a sensor may be attached to one arm and a shoe. Therefore, the posture estimation device, the posture estimation system, and the posture estimation method according to the present disclosure can grasp the whole body posture during a predetermined motion while reducing the burden on the subject in daily activities, and can be used to improve the walking or running form.

[0013] Hereinafter, a posture estimation device, a posture estimation system, and a posture estimation method according to embodiments will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[0014] (Embodiment) [Configuration of posture estimation system] 1 is a schematic diagram showing the configuration of a posture estimation system 100 including a posture estimation device 10 according to embodiment 1. The posture estimation device 10 includes a calculation device 1 (an example of a calculation circuit) that estimates a whole-body posture during a predetermined movement from measurement data, and smart shoes 2 that include a sensor that measures the movement of a subject P when walking or running. The posture estimation system 100 includes the posture estimation device 10 and a measurement device 3 for acquiring posture data required for training, by machine learning, an estimation model that estimates a whole-body posture during a predetermined movement.

[0015] The posture estimation device 10 includes an estimation model that is generated in advance by machine learning training in order to estimate the whole-body posture of the subject P when walking from walking data (an example of measurement data) of the subject P measured by the smart shoes 2. In other words, the estimation model is generated by machine learning training before estimating the whole-body posture of the subject P when walking from the walking data. Therefore, in the posture estimation system 100, in order to train the estimation model, training data that associates the walking data measured by the smart shoes 2 when walking with posture data acquired by the measuring device 3, is collected for multiple subjects. In the present disclosure, an example of a posture estimation device 10 that trains an estimation model using training data of multiple subjects in the posture estimation system 100 and estimates the whole-body posture of the subject P using the trained estimation model.

[0016] Note that posture estimation device 10 may use an estimation model trained with training data of multiple subjects selected regardless of attributes (e.g., gender, age, race, etc.), or may use an estimation model trained with training data of subjects for each attribute. In other words, posture estimation device 10 may be configured to select an appropriate estimation model according to the attributes of subject P when estimating the whole-body posture of subject P. Furthermore, as described below, posture estimation system 100 may calibrate an estimation model trained with training data of multiple subjects with individual training data of subject P to generate a personalized estimation model.

[0017] First, a smart shoe 2 including a sensor for measuring movements when walking or running in the posture estimation system 100 will be described. Fig. 2 is a schematic diagram for explaining a smart shoe 2 according to an embodiment. A sensor module 21 (an example of a sensor) is incorporated in the smart shoe 2 shown in Fig. 2, and the sensor module 21 measures walking data of the subject P.

[0018] Although not shown, the sensor module 21 includes an acceleration sensor, an angular velocity sensor, a calculation circuit that calculates walking parameters from the measured values ​​of these sensors, and a communication circuit for wirelessly transmitting the walking parameters and measured values ​​calculated by the calculation circuit to the calculation device 1. The acceleration sensor can measure, for example, the acceleration of three axes, X, Y, and Z, and the angular velocity sensor can measure, for example, the angular velocity of each of the three axes, X, Y, and Z. Therefore, the sensor module 21 can obtain, as walking data of the subject P, walking parameters such as stride, pitch, walking speed, time taken for one step, stance phase period, swing phase period, toe angle at the time of contact with the ground, pronation, maximum foot height, and landing impact based on the measured values ​​of the acceleration sensor and the angular velocity sensor. Here, the landing impact is an impact applied to the foot when landing, and is an example of a walking parameter evaluated by the maximum value of vertical acceleration at the time of landing. In addition, while the subject P is running, the sensor module 21 can obtain running parameters such as stride, pitch, running speed, time taken for one step, stance phase duration, swing phase duration, toe angle at contact with the ground, pronation, maximum foot height, and landing impact as running data based on the measurement values ​​of the acceleration sensor and the angular velocity sensor.

[0019] Although the data transfer of the walking parameters and the like from the sensor module 21 to the arithmetic device 1 has been described as being performed by wireless communication, this is not limited thereto. For example, the data transfer of the walking parameters and the like from the sensor module 21 to the arithmetic device 1 may be performed by wired communication or by a recording medium (for example, a memory chip, a USB memory, etc.).

[0020] Also, although it has been described that the sensor module 21 calculates the walking parameters from the measurement values ​​of the acceleration sensor and the measurement values ​​of the angular velocity sensor, this is not limited to this. For example, the sensor module 21 may simply transmit the measurement values ​​of the acceleration sensor (acceleration values ​​on three axes) and the measurement values ​​of the angular velocity sensor (angular velocity values ​​on three axes) to the calculation device 1, or the calculation device 1 may calculate the walking parameters from the measurement values ​​of the acceleration sensor and the measurement values ​​of the angular velocity sensor acquired from the sensor module 21.

[0021] In the present disclosure, smart shoes 2 in which a sensor module 21 for measuring walking data is incorporated in shoes will be described as an example below, but the sensor module 21 for measuring the walking data of the subject P may be directly attached to the leg of the subject P using a support or the like. Furthermore, the sensor module 21 may be a portable device such as a smartphone or a smartwatch having an acceleration sensor capable of measuring acceleration on three axes and an angular velocity sensor capable of measuring angular velocity on three axes.

[0022] The posture estimation system 100 requires walking data as training data and posture data of the whole body posture measured when the walking data is measured, and therefore uses a measuring device 3 to acquire the posture data. The measuring device 3 is a motion capture camera (for example, Vicon (registered trademark), Kinect (registered trademark), etc.), and acquires skeletal coordinate data (one example of posture data) of the subject P while walking using the motion capture camera. Note that the measuring device 3 is not limited to a motion capture camera, and may be configured to acquire posture data from a moving image of the subject P while walking, or may be configured to acquire posture data using a motion sensor.

[0023] [Configuration of the computing device] In the present disclosure, the calculation device 1 constitutes part of the posture estimation system 100 as a device that performs calculations to train an estimation model with training data, and also constitutes part of the posture estimation device 10 as a device that estimates the whole-body posture of the subject P from walking data using the trained estimation model.

[0024] Fig. 3 is a block diagram showing the configuration of the arithmetic device 1 according to embodiment 1. As shown in Fig. 3, the arithmetic device 1 includes a processor 11, a memory 12, a storage 13, an input interface 14, an output interface 15, a communication interface 16, and a media reader 17. These components are connected via a processor bus 19.

[0025] The processor 11 is a computer that reads out programs (for example, an OS (Operating System) 130, a training program 131, and an estimation program 133) stored in the storage 13, and deploys the read out programs in the memory 12 and executes them. The processor 11 is configured, for example, with a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), a GPU (Graphics Processing Unit), or an MPU (Multi Processing Unit). The processor 11 may also be configured with a processing circuitry.

[0026] The memory 12 is composed of a volatile memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a non-volatile memory such as a read only memory (ROM) or a flash memory.

[0027] The storage 13 is configured by a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage 13 stores an OS 130, a training program 131, an estimation program 133, training data 132, an estimation model 134, etc.

[0028] The training program 131 is a program for generating an estimation model 134 through machine learning training using training data 132. The estimation program 133 is a program for estimating the whole-body posture of the subject P from walking data of the subject P using the estimation model 134. The training program 131 and the estimation model 134 are used together to execute a posture estimation method.

[0029] The input interface 14 accepts input operations from a keyboard, a mouse, a touch device, etc. The output interface 15 (an example of an output device) outputs estimated data of the whole body posture of the subject P estimated by the arithmetic device 1 to a display, a speaker, etc.

[0030] The communication interface 16 receives the input of walking data from the sensor module 21 and transmits and receives data to and from other devices by performing wired or wireless communication. The communication interface 16 may output the estimated data of the whole body posture of the subject P to the other devices by communicating with the other devices.

[0031] The media reading device 17 accepts a storage medium such as a removable disk 18, a memory chip, a USB memory, etc., and acquires data stored in the removable disk 18, the memory chip, the USB memory, etc. The media reading device 17 may read out training data stored in the removable disk 18, or may store estimated data of the whole body posture of the subject P in the removable disk 18, etc., and output the estimated data.

[0032] In the above description, the arithmetic device 1 generates the estimation model 134 through machine learning training using the training data 132, but if a generalized estimation model 134 is prepared in advance, it may be received from a server or the like via the communication interface 16. In addition, the arithmetic device 1 may read the estimation model 134 stored in a removable disk 18 or the like using the media reading device 17.

[0033] Furthermore, the posture estimation device 10 is configured by combining the calculation device 1 with the smart shoes 2 (sensor module 21), but the configuration of the posture estimation device 10 is not limited to this. For example, the posture estimation device 10 may be configured as a device in which the calculation device 1 and the sensor module 21 are integrated, or may be configured as only the calculation device 1 without the sensor module 21.

[0034] [Pose estimation method] Next, the processing of the posture estimation method executed by posture estimation system 100 will be described with reference to a flowchart. Fig. 4 is a flowchart for explaining the processing of the posture estimation method executed by posture estimation system 100 (particularly, arithmetic device 1) according to embodiment 1. Note that each step shown in Fig. 4 is realized by processor 11 of arithmetic device 1 executing training program 131 and estimation program 133.

[0035] First, the arithmetic device 1 determines whether or not to perform the training phase processing by the training program 131 (step S101). When the posture estimation system 100 does not yet have the estimation model 134, or when the estimation model 134 has not been sufficiently trained, the arithmetic device 1 performs the training phase processing. When the training phase processing is not to be performed (NO in step S101), the arithmetic device 1 advances the processing to step S107. On the other hand, when the training phase processing is to be performed (YES in step S101), the arithmetic device 1 executes the training phase processing A to generate the estimation model 134 for estimating the whole body posture of the subject P during walking from the walking data.

[0036] In the training phase process A, input of training data 132 for N people is accepted in order to generate an estimation model 134 (step S102). The training data 132 accepted in step S102 includes walking data of the person being measured using the smart shoes 2 and posture data obtained by the measuring device 3 of the whole body posture at the time of measuring the walking data. FIG. 5 shows a schematic diagram of the training phase and operation phase in the posture estimation system according to the embodiment. As in the training data collection process A1 shown in FIG. 5, in order to input the training data 132, N people being measured wearing the smart shoes 2 are asked to walk in front of the measuring device 3, and walking data and posture data are acquired.

[0037] Next, the arithmetic device 1 trains an estimation model 134 that estimates the whole body posture during walking by machine learning using the training data 132 (step S103). Here, the estimation model 134 includes a network structure such as a known neural network, a support vector machine (SVM), or a Bayesian network, and internal parameters used by the network structure, and the internal parameters are optimized (adjusted) by training based on estimated data of the whole body posture estimated from the walking data of the subject P and posture data of the whole body posture at the time of measuring the walking data. As in the estimation model training process A2 shown in FIG. 5, the estimation model 134 is trained using training data of N people in which walking data and posture data are associated with each other, thereby constructing an estimation model 134 that estimates posture data (whole body posture) from walking data.

[0038] Next, the calculation device 1 determines whether to perform individual calibration (step S104). If the calculation device 1 receives an input indicating that the subject P does not desire a personalized estimation model (NO in step S104), the calculation device 1 proceeds to step S107. On the other hand, if the calculation device 1 receives an input indicating that the subject P desires a personalized estimation model (YES in step S104), the calculation device 1 executes a calibration process B in which the estimation model 134 is retrained with the individual training data of the subject P.

[0039] In the calibration process B, in order to generate the estimation model 134, input of individual training data 132 of the subject P is accepted (step S105). The individual training data 132 of the subject P accepted in step S105 includes walking data of the subject P himself / herself measured by the smart shoes 2 and posture data of the subject P's whole body posture at the time of measuring the walking data, acquired by the measuring device 3. FIG. 6 is a diagram showing a schematic diagram of the calibration phase in the posture estimation system according to the embodiment. As in the corrected training data collection process B1 shown in FIG. 6, in order to input the individual training data 132 of the subject P, the subject P wearing the smart shoes 2 is asked to walk in front of the measuring device 3, and walking data and posture data are acquired.

[0040] Next, the calculation device 1 retrains the estimation model 134 that estimates the whole body posture by machine learning using the individual training data 132 of the subject P (step S106). As in the retraining process B2 shown in Fig. 6, by retraining the estimation model 134 using individual training data in which the walking data of the subject P and the posture data of the subject P are associated with each other, it is possible to construct the estimation model 134 that can accurately estimate the posture data (whole body posture) of the subject P from the walking data of the subject P. In other words, by retraining the estimation model 134 using the individual training data 132 of the subject P, it is possible to personalize the estimation model 134 for the subject P.

[0041] Next, the arithmetic device 1 judges whether or not to perform operation phase processing by the estimation program 133 (step S107). When a sufficiently trained estimation model 134 is generated in the training phase processing A, or when an estimation model 134 personalized for the subject P is generated in the calibration processing B, the arithmetic device 1 performs operation phase processing. When the operation phase processing is not performed (NO in step S107), the arithmetic device 1 advances the processing to step S112. On the other hand, when the operation phase processing is performed (YES in step S107), the arithmetic device 1 executes operation phase processing C in which the estimation model 134 is used to estimate the whole body posture of the subject P during walking from only the walking data.

[0042] In the operation phase process C, in order to estimate the whole body posture of the person P when walking, the input of walking data of the person P is accepted (step S108). Note that in step S108, only the walking data sent from the smart shoes 2 (sensor module 21) of the person P is required, so the measuring device 3 is not required. As in the measurement process C1 shown in FIG. 5, in order to input the walking data of the person P, the person P wearing the smart shoes 2 is asked to walk, and the walking data sent from the smart shoes 2 is accepted as an input by the arithmetic device 1. The arithmetic device 1 shown in FIG. 1 is a notebook computer, but in the operation phase process C, the trained estimation model 134 may be stored in a smartphone, and the smartphone may be used as the arithmetic device 1 to accept the input of the walking data sent from the smart shoes 2 and estimate the whole body posture of the person P when walking.

[0043] Next, the arithmetic device 1 estimates the whole body posture of the subject P when walking from the walking data of the subject P using the estimation model 134 (step S109). The arithmetic device 1 displays skeletal coordinate data (one example of estimated data) of the whole body posture of the subject P when walking estimated in step S109 as an image on a display connected to the arithmetic device 1. As in estimation process C2 shown in Fig. 5, the walking data of the subject P is input to the estimation model 134, and the skeletal coordinate data estimated as the whole body posture of the subject P when walking is displayed as an image on the display.

[0044] The arithmetic device 1 judges whether or not new walking data has been input from the smart shoes 2 (step S111). The arithmetic device 1 successively estimates the whole-body posture of the person P based on the walking data, and therefore updates the skeletal coordinate data of the person P estimated based on the newly input walking data. Therefore, when new walking data has been input (YES in step S111), the arithmetic device 1 returns the process to step S109 and estimates the whole-body posture of the person P while walking from the new walking data.

[0045] On the other hand, if the input of new walking data has not been received (NO in step S111), the arithmetic device 1 determines whether or not to end the processing of the posture estimation method (step S112). If the arithmetic device 1 has not received an input to end the processing of the posture estimation method (NO in step S112), the arithmetic device 1 returns the process to step S101. On the other hand, if the arithmetic device 1 has received an input to end the processing of the posture estimation method (YES in step S112), the arithmetic device 1 ends the processing of the posture estimation method.

[0046] Next, the processing of the posture estimation method will be described in more detail. In this disclosure, estimation of the whole body posture during walking motion from walking data acquired from the smart shoes 2 will be described. However, in the processing of the posture estimation method, the target of the motion for estimating the whole body posture is not limited to walking motion, and can be similarly applied to any motion in which the whole body is coordinated (for example, coordinated motion (locomotion) such as running). In addition, even for motions other than motions in which the whole body is coordinated, it is possible to estimate the whole body posture during motion in which the lower body and the upper body are not coordinated, for example, by estimating the posture of the lower body from walking data acquired from the smart shoes 2 and estimating the posture of the upper body from measurement data acquired from another sensor attached to the arm or upper body. This is because, in the processing of the posture estimation method, an estimation model 134 is generated in which the correspondence between posture data of the whole body posture during a predetermined motion and measurement data of a sensor attached to a certain part of the body is trained by machine learning in the training phase processing A, and the whole body posture during a predetermined motion can be estimated from the measurement data of a sensor attached to a certain part of the body using the estimation model 134 in the operation phase processing C.

[0047] First, when the motion to be estimated for the whole body posture is walking, posture data of the subject is acquired by the measuring device 3 (motion capture camera) in units of one walking cycle. FIG. 7 is a schematic diagram for explaining one walking cycle. In FIG. 7, a series of motions from the heel-on-scratch timing of the subject's left foot to the heel-on-scratch timing of the right foot and back to the heel-on-scratch timing of the left foot is regarded as one walking cycle. Walking motion is a motion in which the pattern shown in FIG. 7 is repeated. Furthermore, the whole body posture of the subject is expressed as skeletal coordinate data of 21 points, which are characteristic points. The subject is asked to perform walking motion (walking about 1m to 4.5m) that constitutes one walking cycle at a position about 5m away from the measuring device 3, and the measuring device 3 acquires the three-dimensional coordinates of the 21 points of the whole body posture as data for 101 frames. Note that expressing the skeletal coordinate data as 21 body parts is just an example, and the number of points may be increased from 21 points to obtain more accurate whole body posture information, and the number of points may be decreased from 21 points to reduce calculation processing. Similarly, obtaining one walking cycle as 101 frames of data is just one example, and the number of frames can be increased from 101 to obtain more accurate whole-body posture information, or the number of frames can be decreased from 101 to further reduce calculation processing.

[0048] In the above example, the skeleton coordinate data consists of 21 points x 3 coordinates x 101 frames, making a total of 6,363 pieces of data. However, if these 6,363 pieces of data were used as training data as is, the computational processing load would be large, so dimensionality reduction is performed using principal component analysis. Here, principal component analysis is a statistical method for reducing dimensions by synthesizing a small number of uncorrelated variables (principal components) that best represent the overall variance from a large number of correlated variables. Of course, if the computational processing load is not a concern, the 6,363 pieces of data may be used as training data as is.

[0049] The dimension compression process of the skeletal coordinate data will be described more specifically. The arithmetic device 1 extracts data of one walking cycle from the skeletal coordinate data of the walking motion of N people acquired from the measurement device 3, resamples one walking cycle to 0 to 100%, and normalizes the time. That is, the arithmetic device 1 normalizes one walking cycle as skeletal coordinate data of 101 frames. The arithmetic device 1 also calculates the sum of the distances from a certain reference coordinate to each coordinate as norm d, and normalizes the position by dividing each coordinate by norm d. The arithmetic device 1 performs principal component analysis on the 6363 pieces of skeletal coordinate data thus obtained, and performs dimension compression to compress the information of components (for example, 16 variables) that have a high contribution rate (for example, 95% or more) among the principal components (variables) that explain the walking motion. That is, the arithmetic device 1 calculates the 6363 pieces of skeletal coordinate data as posture data that has been dimension-compressed to 16 components (variables). Of course, the number of components in the principal component analysis of the skeletal coordinate data is an example, and is not limited to 16 components.

[0050] In the present disclosure, the posture data dimensionally compressed to 16 components and the walking data corresponding to the posture data are adopted as training data. The walking data adopts 10 walking parameters, including stride, pitch, walking speed, time taken for one step, stance phase period, swing phase period, toe angle at contact, pronation, maximum foot height, and landing impact. Of course, the walking data may not be the 10 walking parameters, but may be the measured values ​​of the acceleration sensor and the angular velocity sensor themselves. By adopting 10 walking parameters as the walking data, the burden of calculation processing is reduced, and machine learning of the estimation model is performed based on the walking parameters that represent the characteristics of the walking motion, making it easier to converge the estimation model. Also, the 10 walking parameters are an example, and fewer or more walking parameters may be used as the walking data.

[0051] Fig. 8 is a diagram illustrating the calculation of an estimation model in posture estimation system 100 according to an embodiment. As shown in Fig. 8, posture estimation system 100 estimates posture data of 16 components from walking data including 10 walking parameters using estimation model 134. Here, estimation model 134 has a neural network structure including three 100-dimensional fully connected layers. In posture estimation system 100, machine learning is performed so as to minimize a loss function between posture data of 16 components of training data and posture data of estimated 16 components, and trained estimation model 134 is constructed.

[0052] By using the trained estimation model 134, the posture estimation device 10 can estimate posture data of 16 components from walking data of the subject P measured by the sensor module 21, and reconstructs skeletal coordinates from the estimated 16 components to obtain skeletal coordinate data of the whole-body posture of the subject P as estimated data. In this way, by using the trained estimation model 134, the posture estimation device 10 can continuously grasp the whole-body posture of the subject P while walking on a daily basis, simply by having the subject P walk while wearing the smart shoes 2, without using the measuring device 3 (motion capture camera).

[0053] The accuracy of the whole-body posture of subject P during walking estimated by posture estimation device 10 using trained estimation model 134 will be described. Fig. 9 is a diagram showing whole-body skeletal coordinate data estimated by posture estimation device 10 according to an embodiment from the front. Fig. 10 is a diagram showing whole-body skeletal coordinate data estimated by posture estimation device 10 according to an embodiment from the side. In Figs. 9 and 10, skeletal coordinate data I of the whole-body posture of subject P estimated by posture estimation device 10 is shown by a solid line. In Figs. 9 and 10, skeletal coordinate data II of the whole-body posture of subject P measured by measuring device 3 is also shown by a dashed line for comparison.

[0054] In Fig. 9, which is displayed from the front, there is no difference between the solid line skeletal coordinate data I and the dashed line skeletal coordinate data II, and it can be seen that the posture estimation device 10 has accurately estimated the skeletal coordinate data I of the subject P. On the other hand, in Fig. 10, which is displayed from the side, a comparison of the solid line skeletal coordinate data I and the dashed line skeletal coordinate data II shows differences in the arms and legs, but it can be seen that the posture estimation device 10 has appropriately estimated the skeletal coordinate data I of the subject P.

[0055] The accuracy of the whole-body posture of subject P estimated by posture estimation device 10 will be described in more detail. Fig. 11 is a graph showing the change in error in one walking cycle of arm skeletal coordinate data estimated by posture estimation device 10 according to the embodiment. Fig. 12 is a graph showing the change in error in one walking cycle of lower body skeletal coordinate data estimated by posture estimation device 10 according to the embodiment. Fig. 13 is a graph showing the change in error in one walking cycle of head skeletal coordinate data estimated by posture estimation device 10 according to the embodiment.

[0056] The graphs shown in Figs. 11 to 13 show the results of accuracy verification by comparing the feature points (21 points) of the whole body posture of the subject P estimated by the posture estimation device 10 with the feature points (21 points) of the whole body posture of the subject P measured by the measurement device 3. In the graphs shown in Figs. 11 to 13, the horizontal axis indicates one walking cycle, and the vertical axis indicates error. In addition, in the graphs shown in Figs. 11 to 13, the average error of each of the arms, lower body, and head (trunk) is indicated by a solid line, and the standard deviation of each is indicated by a dashed line. Since the posture estimation device 10 estimates the whole body posture only from the information of the sensor module 21 provided in the smart shoes 2, it is considered that there is a possibility that differences in accuracy may be observed depending on the body part. Therefore, the feature points (21 points) of the whole body posture are divided into three parts, namely the head (trunk), lower body, and arms, and the average error is obtained, and the graphs in Figs. 11 to 13 are shown in order to verify the change in the error over time.

[0057] From the graphs shown in Figs. 11 to 13, the average error of all the feature points (21 points) of the whole body posture was 3.8 cm (standard deviation: 3.3 cm), which is a highly accurate result. The average error of the arm shown in Fig. 11, where the largest error was observed, was 7.6 cm (standard deviation: 4.5 cm). The average stride length was about 70 cm, and the error in the stride length estimated by posture estimation device 10 was about 5%. It was found that the errors of the feature points estimated by posture estimation device 10 were largest in the order of arms, lower body, and head (trunk), and that the errors of the feature points of the lower body and the feature points of the arms had a peak value at the timing when the feet touched the ground.

[0058] [Variations] (1) In the above-described embodiment, the posture estimation device 10 estimates skeletal coordinate data (an example of estimated data) of the whole-body posture during walking from walking data (an example of measured data) of the sensor module 21. However, the estimated skeletal coordinate data may be a part of the body, such as the lower body or upper body.

[0059] (2) In the above embodiment, the subject's movements are measured using smart shoes 2 incorporating sensor modules 21 (including a three-axis acceleration sensor and a three-axis angular velocity sensor) in the shoes, but the sensor modules 21 may be attached at any position as long as they move in accordance with the subject's movements. In the above embodiment, a total of two sensor modules 21 are used, with one type of sensor provided in each of the left and right shoes, but one sensor module 21 provided in either the left or right shoe may be used, or multiple sensors of different types may be used. Furthermore, even if a total of two sensor modules 21 are provided in each of the left and right shoes, the posture estimation device 10 may be configured to use measurement data from either one of the sensor modules 21.

[0060] (3) In the above-described embodiment, ten walking parameters (also called gait indices) obtainable from the sensor module 21 were used as walking data, but any walking data derived from a three-axis acceleration sensor and a three-axis angular velocity sensor may be used, and the values ​​of the three-axis acceleration sensor and the three-axis angular velocity sensor themselves may be used.

[0061] (4) In the above embodiment, the subject of the motion for which the whole body posture is to be estimated is walking motion, but the present invention can be applied to any motion in which the whole body is coordinated (for example, locomotion such as running). In addition, even for specific motions other than those in which the whole body is coordinated (for example, periodic motions such as a golf swing), it is possible to estimate the whole body posture in a specific motion by providing the sensor modules 21 on each of the legs and arms, creating training data, and training an estimation model.

[0062] (5) In the above embodiment, the whole-body posture of the subject P while walking, estimated in the operational phase processing C, is displayed as an image on the display. However, other than an image, the whole-body posture of the subject P may be output as sound from a speaker or as vibration using a vibration mechanism, and so on, to provide feedback to the subject P.

[0063] (6) In the above-described embodiment, the posture data was regressed from the walking data acquired from the smart shoes 2 using the estimation model 134 having a neural network structure. However, other machine learning methods other than the neural network structure may also be used.

[0064] (7) In the above embodiment, the posture estimation device 10 estimated the whole-body posture of the subject P from the walking data acquired from the smart shoes 2, but the whole-body posture of the subject P at the time of a predetermined movement may be estimated from measurement data of the subject P's movements acquired from a device other than the smart shoes 2 (for example, a smartphone, a smartwatch, etc. with a built-in 3-axis acceleration sensor and a 3-axis angular velocity sensor). Furthermore, the posture estimation device 10 may estimate the whole-body posture of the subject P at the time of a predetermined movement from measurement data of the subject P's movements by combining the smart shoes 2 with a device such as a smartphone or a smartwatch.

[0065] (8) In the above-described embodiment, for convenience, it has been explained that both the training data 132 and the walking data of the subject P are acquired using smart shoes 2 including a sensor module 21, but the sensors that acquire the training data 132 and the walking data of the subject P may be different from each other.

[0066] [Aspects] (1) A posture estimation device according to the present disclosure is a posture estimation device that estimates a whole-body posture of a subject during a predetermined movement, At least one sensor attached to the subject to acquire measurement data based on three-axis acceleration and three-axis angular velocity of at least one location of the subject; a calculation circuit that estimates the whole-body posture of the subject during a predetermined movement from the measurement data acquired by the sensor using an estimation model trained by machine learning; an output device that outputs estimated data of the whole body posture of the subject estimated by the arithmetic circuit; The estimated model is The posture data is generated in advance through machine learning training based on measurement data acquired by a sensor during a specified movement and posture data that measures the subject's entire body posture at the time the measurement data was acquired.

[0067] As a result, the posture estimation device of the present disclosure uses an estimation model trained by machine learning to estimate the entire body posture of the person being measured during a specified movement from measurement data acquired by the sensors, making it possible to continuously grasp the entire body posture on a daily basis using a small number of sensors.

[0068] (2) A posture estimation device according to (1), wherein the estimation model is retrained by machine learning for each subject who uses the posture estimation device.

[0069] (3) In the posture estimation device according to (1) or (2), the predetermined movement is a movement in which the entire body of the person being measured is linked to the movement of the part of the person being measured to which the sensor is attached.

[0070] (4) In the posture estimation device according to any one of (1) to (3), the sensor is provided in a shoe worn by the person being measured.

[0071] (5) A posture estimation device according to (4), wherein the sensor acquires a plurality of parameters based on the walking or running of the person being measured, the parameters being calculated from three-axis acceleration and three-axis angular velocity, as measurement data.

[0072] (6) A posture estimation device as described in (5), wherein the multiple parameters based on the walking or running of the subject include at least one of stride, pitch, walking or running speed, time taken for one step, stance phase duration, swing phase duration, toe angle at contact with the ground, pronation, maximum foot height, and landing impact.

[0073] (7) A posture estimation system according to the present disclosure includes a measurement device that measures posture data, and the posture estimation device according to any one of (1) to (6), Posture data measured by a measuring device is used when training an estimation model using machine learning.

[0074] (8) The posture estimation system according to (7), The measuring device is a motion capture camera. The posture data is skeletal coordinate data acquired using a motion capture camera.

[0075] (9) The posture estimation system according to (8), The skeletal coordinate data is dimensionally compressed using principal component analysis, and an estimation model is trained using machine learning.

[0076] (10) A posture estimation method according to the present disclosure is a posture estimation method for estimating a whole-body posture of a subject during a predetermined movement, comprising: Obtaining measurement data based on three-axis acceleration and three-axis angular velocity of at least one location of the subject from at least one sensor attached to the subject; A step of estimating a whole-body posture of a subject during a predetermined movement from measurement data acquired by a sensor using an estimation model trained by machine learning; and outputting estimated data of the whole body posture of the subject. The estimation model is generated in advance through machine learning training based on measurement data acquired by a sensor during a specified movement and posture data measuring the subject's entire body posture at the time the measurement data was acquired.

[0077] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0078] 1 Calculation device, 2 Smart shoes, 3 Measurement device, 10 Posture estimation device, 11 Processor, 12 Memory, 13 Storage, 14 Input interface, 15 Output interface, 16 Communication interface, 17 Media reading device, 18 Removable disk, 19 Processor bus, 21 Sensor module, 100 Posture estimation system, 131 Training program, 132 Training data, 133 Estimation program, 134 Estimation model.

Claims

1. A posture estimation device that estimates a whole-body posture of a subject during a predetermined movement, comprising: At least one sensor attached to the subject and configured to acquire measurement data based on three-axis acceleration and three-axis angular velocity at at least one location of the subject; a calculation circuit that estimates a whole-body posture of the subject during the predetermined movement from the measurement data acquired by the sensor using an estimation model trained by machine learning; an output device that outputs estimated data of the whole body posture of the subject estimated by the arithmetic circuit, The estimation model is A posture estimation device that is generated in advance through machine learning training based on the measurement data acquired by the sensor during the specified movement and posture data that measures the subject's entire body posture at the time the measurement data was acquired.

2. The posture estimation device according to claim 1 , wherein the estimation model is retrained by machine learning for each of the subjects who use the posture estimation device.

3. The posture estimation device according to claim 1 , wherein the predetermined movement is a movement in which the entire body of the subject moves in conjunction with the movement of the part of the subject to which the sensor is attached.

4. 4. The posture estimation device according to claim 1, wherein the sensor is provided in a shoe worn by the person being measured.

5. The posture estimation device according to claim 4 , wherein the sensor acquires, as the measurement data, a plurality of parameters based on the walking or running of the person being measured, the parameters being calculated from three-axis acceleration and three-axis angular velocity.

6. 6. The posture estimation device of claim 5, wherein the plurality of parameters based on the walking or running of the subject include at least one parameter of stride, pitch, walking speed or running speed, time per step, stance phase duration, swing phase duration, toe angle at contact with the ground, pronation, maximum foot height, and landing impact.

7. A measuring device for measuring the posture data; The posture estimation device according to any one of claims 1 to 3, A posture estimation system that uses the posture data measured by the measurement device when training the estimation model using machine learning.

8. the measuring device is a motion capture camera; The posture estimation system according to claim 7 , wherein the posture data is skeletal coordinate data acquired using the motion capture camera.

9. The posture estimation system according to claim 8 , wherein the skeletal coordinate data is dimensionally reduced by principal component analysis, and the estimation model is trained by machine learning.

10. A posture estimation method for estimating a whole-body posture of a subject during a predetermined movement, comprising: acquiring measurement data based on three-axis acceleration and three-axis angular velocity of at least one location of the subject from at least one sensor attached to the subject; estimating a whole-body posture of the subject during the predetermined movement from the measurement data acquired by the sensor using an estimation model trained by machine learning; and outputting estimated data of the whole body posture of the subject. A posture estimation method in which the estimation model is generated in advance through machine learning training based on the measurement data acquired by the sensor during a specified movement and posture data that measures the subject's entire body posture at the time the measurement data was acquired.

Citation Information

Patent Citations

  • Operation state estimation apparatus, operation state estimation method, and program

    JP2019103609A

  • Estimation device, estimation method and program

    JP2022013405A

  • Information processing system, server, information processing method, and program

    JP2023037657A

  • Program, information processing device, and information processing method

    JP2023016825A

Cited By

  • Posture prediction device, posture prediction method, and program

    JP2026151634A