Information processing system, information processing method, and program

Abstract

To provide an information processing system, an information processing method, and a program for estimating a three-dimensional posture of an estimation object on the basis of measurement data of foot pressure by receiving measurement data of the foot pressure of the measurement object.SOLUTION: In an information processing system in which an information processing device 100 and a plurality of set of insole type sensors 110 are connected communicably through a network, the insole type sensors measure foot pressure of a measurement object person wearing shoes with the insole type sensors put therein, and transmit a measured result as measurement data to the information processing device through the network. A control part of the information processing device estimates a three-dimensional posture of the measurement object person on the basis of the measurement data of the foot pressure of the measurement object person received from the insole type sensors. The three-dimensional posture of the measurement object person is estimated at low cost because a camera system is not required, and the three-dimensional posture of the measurement object person can be estimated without being affected even by a photographing condition.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique for estimating the posture of a posture estimation target from a two-dimensional image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-85933 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, a camera is used to estimate the posture of the posture estimation target. Generally, posture estimation using a camera requires a specific camera system, and the cost of the entire system is high. In addition, there are problems such as the body of the target having to be clearly included within the field of view of the camera, and the effect of shooting conditions such as resolution, lighting, and motion blur. [Means for solving the problem]

[0005] According to one aspect of the present invention, there is provided an information processing system that receives foot pressure measurement data of an object to be estimated, and estimates a three-dimensional posture of the object based on the foot pressure measurement data. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 3]FIG. 3 is an activity diagram illustrating an example of information processing executed by an information processing device. [Figure 4] FIG. 4 is a conceptual diagram showing an example of the processing of activity A330. [Diagram 5] FIG. 5 is a diagram showing an example of a result of the processing according to the first embodiment. [Figure 6] FIG. 6 is an activity diagram illustrating an example of information processing executed by the information processing device of the first modification. [Figure 7] FIG. 7 is a conceptual diagram showing an example of the processing of activity A630. [Figure 8] FIG. 8 is a diagram showing an example of a result of the processing of the first modification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic features shown in the following embodiments can be combined with each other.

[0008] In this specification, the term "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, various information is handled in this embodiment, and this information is represented by high and low signal values ​​as a binary bit collection consisting of 0 or 1, and communication and calculation can be performed on the circuit in the broad sense.

[0009] In addition, a circuit in the broad sense is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, a memory, etc. In other words, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0010] In addition, the program for realizing the software appearing in the embodiments may be implemented in a form that is downloadable from a server, the program may be executed on a cloud computer, or the program may be stored in a non-volatile or volatile non-transitory storage medium and distributed.

[0011] <Embodiment 1> 1. Information Processing System 1 is a diagram showing an example of a system configuration of an information processing system 1000. The information processing system 1000 includes, as a system configuration, an information processing device 100 and a pair of insole type sensors 110 (hereinafter simply referred to as the insole type sensors 110). The information processing device 100 and the insole type sensors 110 are communicatively connected via a network 150. Although only one insole type sensor 110 is connected to the network 150 in FIG. 1, a plurality of insole type sensors 110 may be connected to the network 150.

[0012] The information processing device 100 executes the processing described in embodiment 1 (including embodiment 1 and the modified examples described below; the same applies below). In FIG. 1, a PC (Personal Computer) is shown as an example of the information processing device 100, but the information processing device 100 is not limited to a PC. The information processing device 100 may be a tablet PC, a smartphone, a server device, or a cloud system. In other words, the information processing device 100 may be any device or system that can execute the processing of embodiment 1.

[0013] The insole-type sensor 110 measures in real time the foot pressure of a subject (hereinafter simply referred to as the subject) wearing shoes containing the insole-type sensor 110, and transmits the measurement results as measurement data to the information processing device 100 via the network 150. The subject is an example of an object whose three-dimensional posture is to be estimated.

[0014] Here, the information processing system described in the claims may be composed of multiple devices or may be composed of one device. When the information processing system described in the claims is composed of one device, an example of the device is the information processing device 100, etc. When the information processing system described in the claims is composed of multiple devices, an example of the multiple devices is the information processing device 100 and the insole type sensor 110, etc. However, the configuration of the information processing system is not limited to these, and the information processing system may be composed of any device or any combination of devices as long as it can execute the processes described below.

[0015] 2. Hardware Configuration 2 is a diagram showing an example of a hardware configuration of the information processing device 100. The information processing device 100 includes a control unit 210, a storage unit 220, an input unit I / F 230, an output unit I / F 240, and a communication unit 250 as the hardware configuration.

[0016] The control unit 210 is a CPU (Central Processing Unit) or the like, and controls the entire information processing device 100. The control unit 210 executes processing based on a program stored in the storage unit 220, thereby realizing the functions of the information processing device 100 and information processing of activity diagrams such as those shown in Figs. 3 and 6, which will be described later.

[0017] The storage unit 220 is any one of a hard disk drive (HDD), a read only memory (ROM), a random access memory (RAM), and a solid state drive (SSD), or any combination thereof, and stores a program and data (for example, measurement data of foot pressure, an image and an estimation result generated based on the measurement data of foot pressure, which will be described later) used when the control unit 210 executes a process based on the program. The control unit 210 executes a process based on the program stored in the storage unit 220, thereby realizing the functions of the information processing device 100 and the process of the activity diagram shown in FIG. 4, which will be described later. The storage unit 220 is an example of a storage medium. Note that, in the first embodiment, the data used when the control unit 210 executes a process based on the program is described as being stored in the storage unit 220, but the data may be stored in a storage unit of another device that can communicate with the information processing device 100. In other words, the data may be stored in a storage unit of any device as long as the control unit 210 can refer to the data.

[0018] The input unit I / F 230 is an interface that connects an input unit to the information processing device 100. The input unit may be, for example, a keyboard and / or a mouse. The output unit I / F 240 is an interface that connects an output unit to the information processing device 100. The output unit may be, for example, a display.

[0019] The communication unit 250 is a NIC (Network Interface Card) or the like, and connects the information processing device 100 to the network 150 and controls communication with other devices.

[0020] 3. Information Processing (1) Overview of the process The control unit 210 receives the measurement data of the subject's foot pressure from the insole-type sensor 110, and estimates the three-dimensional posture of the subject based on the received measurement data of the foot pressure.

[0021] According to this processing, since a camera system is not required, the three-dimensional posture of the measurement subject can be estimated inexpensively. Also, since a camera system is not required, the three-dimensional posture of the measurement subject can be estimated without being affected by the shooting conditions. (2) Details of the processing FIG. 3 is an activity diagram showing an example of information processing executed by information processing device 100. As shown in FIG. In activity A310, the control unit 210 receives measurement data of the subject's foot pressure from the insole type sensor 110. More specifically, the insole type sensor 110 measures data of pressure distribution on the sole of the subject's foot as the measurement data of foot pressure. The control unit 210 receives the data of pressure distribution on the sole of the subject's foot from the insole type sensor 110. In activity A320, the control unit 210 stores the received data of pressure distribution on the soles of the measurement subject in the storage unit 220, etc. The control unit 210 also generates an image showing pressure distribution on the soles of the measurement subject based on the data of pressure distribution on the soles of the measurement subject, and stores the generated image in the storage unit 220, etc. The control unit 210 associates information on the measurement date and time included in the measurement data with the image and stores it in the storage unit 220, etc. The control unit 210 repeatedly executes the processing of activity A310 and activity A320 while the measurement data is being sent from the insole type sensor 110.

[0022] In activity A330, the control unit 210 acquires a history image of the sole pressure distribution generated based on the data of the sole pressure distribution of the measurement subject from the storage unit 220 or the like. Here, the history image refers to a predetermined number of consecutive images in a time series of images generated based on the sole pressure distribution. The control unit 210 estimates the three-dimensional posture of the measurement subject based on the acquired history image of the sole pressure distribution. For example, the control unit 210 inputs the history image into a trained model. Here, the trained model is a trained model trained with the history image of the sole pressure distribution generated based on the data of the sole pressure distribution of the measurement subject as input data and the data of the three-dimensional posture of the measurement subject as output data. The control unit 210 acquires the three-dimensional posture of the measurement subject output from the trained model as an estimation result.

[0023] 4 is a conceptual diagram showing an example of the processing of activity A 330. An image 410 is an image of pressure distribution. The control unit 210 inputs a predetermined number of images 410 (e.g., five images) to a trained model 420, and obtains three-dimensional posture data 430 output from the trained model 420.

[0024] In activity A340, the control unit 210 displays the data of the estimation result of the three-dimensional posture of the person to be measured acquired in activity A330, for example, on a display or the like connected to the information processing device 100 via the output unit I / F 240. Displaying on a display or the like is one example of output. As another example of output, the control unit 210 may store a file or the like including the data of the estimation result in the storage unit 220 or the like, or may transmit a file or the like including the data of the estimation result to another device via the communication unit 250. The same applies hereinafter.

[0025] FIG. 5 is a diagram showing an example of a result of the processing according to the first embodiment. PoseFormer (RGB) is one of the known camera-based estimation methods. IMUPoser (2IMUs) is one of the known two-IMU-based estimation methods. SoleFormer (Pressure Only) 510 is the result of the estimation method of the first embodiment. InferenceTime is the inference time. Results w / o Arm and Wrist (13 Joints) shows the results of estimation of 13 joints excluding the arms and wrists. Results w / Arm and Wrist (17 Joints) shows the results of estimation of 17 joints including the arms and wrists. SP-S stands for SolePose-Sport, and is a dataset of 3D motion capture of dynamic sports movements such as golf swing, skiing, and snowboarding. SP-E stands for SolePose-Exercise, and is a dataset of 3D motion capture of daily exercises such as walking, jogging, squatting, and jumping. TMM100 is a dataset including foot pressure measurements specialized for Tai Chi movements. MPJPE is the mean joint position error. MPJAE is the mean joint angle error.

[0026] According to the first embodiment, the three-dimensional posture of the subject can be estimated from the data of the foot pressure of the subject, and the estimation result can be output. Since a camera system is not required, the three-dimensional posture of the subject can be estimated inexpensively. Furthermore, since a camera system is not required, the three-dimensional posture of the subject can be estimated without being affected by the shooting conditions. Furthermore, since the three-dimensional posture of the subject can be estimated from the data of the foot pressure, there is no need to go to a predetermined place where a camera system or the like is installed, and there is no need to go to the trouble of moving or wear a marker or the like. Furthermore, since the three-dimensional posture of the subject is estimated based on the data from the insole type sensor 110, the posture of the subject in a natural state can be estimated. Furthermore, by applying the method of the first embodiment to an application or the like, sports analysis, evaluation of rehabilitation exercise, and the like can be performed.

[0027] (Variation 1) Modification 1 of embodiment 1 will be described below. In modification 1, differences from embodiment 1 will be described. The control unit 210 of the first modification (hereinafter simply referred to as the control unit 210) receives the measurement data of the three-dimensional inertial motion of the subject, and estimates the three-dimensional posture of the subject based on the measurement data of the foot pressure and the measurement data of the inertial motion. This process achieves the effects of the first embodiment described above, and also makes it possible to estimate the three-dimensional posture of the subject with higher accuracy.

[0028] The insole type sensor 110 of the first modification (hereinafter simply referred to as the insole type sensor 110) measures the acceleration of the subject with six degrees of freedom (hereinafter simply referred to as the acceleration) in real time, and transmits it to the information processing device 100 via the network 150. Note that in the first modification, the insole type sensor 110 will be described as measuring the acceleration of the subject and transmitting it to the information processing device 100 in real time. However, a sensor other than the insole type sensor 110 may measure the acceleration of the subject and transmit it to the information processing device 100 in real time. The acceleration of the subject is an example of measurement data of the three-dimensional inertial motion of the subject.

[0029] FIG. 6 is an activity diagram showing an example of information processing executed by information processing device 100 of the first modification. In activity A 610 , the control unit 210 receives the acceleration of the measurement subject from the insole type sensor 110 . In activity A620, the control unit 210 stores the acceleration in the storage unit 220 or the like.

[0030] In activity A630, the control unit 210 estimates the three-dimensional posture of the subject based on the historical image of the pressure distribution of the sole of the foot and the acceleration. For example, the control unit 210 inputs the historical image of the pressure distribution of the sole of the foot and the acceleration generated based on the data of the pressure distribution of the sole of the foot of the subject to the trained model. Here, the trained model is a trained model trained using the historical image of the pressure distribution of the sole of the foot generated based on the data of the pressure distribution of the sole of the foot of the subject and the three-dimensional acceleration of the subject as input data and the data of the three-dimensional posture of the subject as output data. The control unit 210 acquires the three-dimensional posture of the subject output from the trained model as an estimation result.

[0031] 7 is a conceptual diagram showing an example of the processing of activity A 630. Measurement data 710 is acceleration data. Control unit 210 inputs a predetermined number of images 410 (e.g., five images) and measurement data 710 to trained model 720, and obtains three-dimensional posture data 730 output from trained model 720.

[0032] In activity A640, the control unit 210 displays the data of the estimation result of the three-dimensional posture of the person to be measured acquired in activity A630 on a display or the like connected to the information processing device 100 via the output unit I / F 240, for example.

[0033] FIG. 8 is a diagram showing an example of a result of the processing of the first modification. SoleFormer(Ours) is the result of the estimation method of Modification 1. As shown in Fig. 8, according to the estimation method of Modification 1, it is possible to estimate the three-dimensional posture data of the measurement subject more quickly and with less error than the conventional estimation method.

[0034] According to the process of the first modification, the effects of the first embodiment can be achieved, and the three-dimensional posture of the subject can be estimated more accurately and more quickly.

[0035] <Additional Notes> The invention may be provided in each of the following aspects:

[0036] (1) An information processing system that receives measurement data of foot pressure of an object to be estimated, and estimates a three-dimensional posture of the object to be estimated based on the measurement data of foot pressure.

[0037] (2) In the information processing system described above in (1), the foot pressure measurement data is data on pressure distribution of the sole of the subject of estimation.

[0038] (3) In the information processing system described in (2) above, a historical image of the pressure distribution of the sole of the foot is generated based on the data of the pressure distribution of the sole of the foot, and a three-dimensional posture of the estimation target is estimated based on the generated historical image.

[0039] (4) In the information processing system described in (3) above, a historical image of the pressure distribution of the sole of the foot generated based on data of the pressure distribution of the sole of the foot to be estimated is input to a trained model, and the trained model is a trained model trained using the historical image of the pressure distribution of the sole of the foot generated based on the data of the pressure distribution of the sole of the foot to be estimated as input data and data of the three-dimensional posture of the target to be estimated as output data, and the three-dimensional posture of the target to be estimated output from the trained model is obtained as a result of estimation.

[0040] (5) In the information processing system described in (1) above, an information processing system receives measurement data of three-dimensional inertial motion of the estimation object, and estimates a three-dimensional posture of the estimation object based on the measurement data of the foot pressure and the measurement data of the inertial motion.

[0041] (6) In the information processing system described in (5) above, the foot pressure measurement data is data on a pressure distribution on the sole of the subject to be estimated, a historical image of the pressure distribution on the sole of the foot is generated based on the data on the pressure distribution on the sole of the foot, and a three-dimensional posture of the subject to be estimated is estimated based on the generated historical image and the measurement data of the inertial motion.

[0042] (7) In the information processing system described in (6) above, a historical image of the pressure distribution on the sole of the foot generated based on data of the pressure distribution on the sole of the subject to be estimated and the measurement data of the inertial motion are input into a trained model, and the trained model is a trained model trained using the historical image of the pressure distribution on the sole of the foot generated based on the data of the pressure distribution on the sole of the subject to be estimated and the measurement data of the three-dimensional inertial motion of the subject to be estimated as input data and data of the three-dimensional posture of the subject to be estimated as output data, and the three-dimensional posture of the subject to be estimated output from the trained model is obtained as an estimation result.

[0043] (8) In the information processing system according to any one of (1) to (7) above, the information processing system receives measurement data of the foot pressure of the subject to be estimated from an insole-type sensor.

[0044] (9) In the information processing system according to any one of (5) to (6) above, measurement data of the three-dimensional inertial motion of the estimation target is received from an insole-type sensor.

[0045] (10) An information processing system according to any one of (1) to (9) above, which outputs data on the three-dimensional posture of the estimation target.

[0046] (11) An information processing method executed by an information processing system, comprising: receiving measurement data of foot pressure of an object to be estimated; and estimating a three-dimensional posture of the object to be estimated based on the measurement data of foot pressure.

[0047] (12) A program for causing a computer to function as the information processing system according to any one of (1) to (10) above. Of course, this is not the case.

[0048] For example, the above-mentioned program may be provided as a computer-readable non-transitory storage medium that stores the program.

[0049] Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The new embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments and their modifications are within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0050] 100: Information processing device 110: Insole-type sensor 150: Network 210: Control section 220: Storage section 230: Input I / F 240: Output I / F 250: Communications Department 410:Image 420: Model 430: Data 710: Measurement data 720: Model 730: Data 1000: Information processing systems

Claims

1. An information processing system, Receive measurement data of the foot pressure to be estimated; estimating a three-dimensional posture of the estimation target based on the foot pressure measurement data; Information processing system.

2. 2. The information processing system according to claim 1, The foot pressure measurement data is data on pressure distribution of the sole of the subject. Information processing system.

3. 3. The information processing system according to claim 2, generating a history image of the pressure distribution of the sole of the foot based on the data of the pressure distribution of the sole of the foot, and estimating a three-dimensional posture of the estimation target based on the generated history image; Information processing system.

4. 4. The information processing system according to claim 3, A historical image of the pressure distribution of the sole of the foot, which is generated based on the data of the pressure distribution of the sole of the foot to be estimated, is input to the trained model; The trained model is a trained model trained using, as input data, a history image of a pressure distribution on the sole of a target foot, the pressure distribution being generated based on data of the pressure distribution on the sole of the target foot, and using, as output data, data of a three-dimensional posture of the target foot, Obtaining the three-dimensional posture of the estimation target output from the trained model as an estimation result; Information processing system.

5. 2. The information processing system according to claim 1, receiving measurement data of three-dimensional inertial motion of the estimation target; estimating a three-dimensional posture of the estimation target based on the foot pressure measurement data and the inertial motion measurement data; Information processing system.

6. 6. The information processing system according to claim 5, the foot pressure measurement data is pressure distribution data of a sole of the subject of estimation, generating a history image of the pressure distribution of the sole of the foot based on the data of the pressure distribution of the sole of the foot, and estimating a three-dimensional posture of the estimation target based on the generated history image and the measurement data of the inertial motion; Information processing system.

7. 7. The information processing system according to claim 6, inputting a history image of the pressure distribution of the sole of the foot generated based on the pressure distribution data of the sole of the foot to be estimated and the measurement data of the inertial motion into a trained model; The trained model is a trained model trained using a history image of a pressure distribution on the sole of a target object generated based on pressure distribution data on the sole of the target object and measurement data of a three-dimensional inertial motion of the target object as input data, and using data of a three-dimensional posture of the target object as output data, Obtaining the three-dimensional posture of the estimation target output from the trained model as an estimation result; Information processing system.

8. 2. The information processing system according to claim 1, receiving measurement data of the foot pressure of the subject from an insole type sensor; Information processing system.

9. 6. The information processing system according to claim 5, receiving measurement data of three-dimensional inertial motion of the estimation target from an insole-type sensor; Information processing system.

10. 2. The information processing system according to claim 1, outputting data on the three-dimensional posture of the object to be estimated; Information processing system.

11. An information processing method executed by an information processing system, comprising: Receive measurement data of the foot pressure to be estimated; estimating a three-dimensional posture of the estimation target based on the foot pressure measurement data; Information processing methods.

12. A program, A program for causing a computer to function as the information processing system according to any one of claims 1 to 10.

Images