Information processing device, electronic device, information processing system, information processing method, and program

Abstract

This information processing device comprises a control unit. The control unit acquires, using sensor data indicating the movement of at least some of body portions of a user and a learning model, an estimated value of a posture angle of at least one of a plurality of body portions of the user. The learning model is trained so as to, when sensor data is inputted, output an estimated value of a posture angle.

Description

Cross-reference to related applications

[0001] This application claims priority to Patent Application No. 2021-201275, filed in Japan on December 10, 2021, the entire disclosure of which is incorporated herein by reference. [Technical Field]

[0002] The present disclosure relates to an information processing device, an electronic device, an information processing system, an information processing method, and a program. [Background technology]

[0003] Conventionally, there is known a technique for detecting a user's movement using a camera. For example, Patent Document 1 describes a motion capture system that estimates the movement of an object based on images captured by a plurality of cameras. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-201183 Summary of the Invention

[0005] An information processing device according to an embodiment of the present disclosure includes: a control unit that acquires an estimated value of a posture angle of at least one of a plurality of body parts of the user based on sensor data indicating a movement of at least a part of the body part of the user and a learning model; The learning model is trained to output the estimated value of the attitude angle when the sensor data is input.

[0006] An electronic device according to an embodiment of the present disclosure includes: The information processing device includes an output unit that outputs the data of the walking model generated by the information processing device.

[0007] An information processing system according to an embodiment of the present disclosure includes: an information processing device that acquires an estimated value of a posture angle of at least one of a plurality of body parts of a user using sensor data indicating a movement of at least a part of the body part of the user and a learning model; The learning model is trained to output the estimated value of the attitude angle when the sensor data is input.

[0008] An information processing method according to an embodiment of the present disclosure includes: obtaining an estimated value of a posture angle of at least one of a plurality of body parts of the user using sensor data indicating a movement of at least a part of the body part of the user and a learning model; The learning model is trained to output the estimated value of the attitude angle when the sensor data is input.

[0009] A program according to an embodiment of the present disclosure includes: On the computer, obtaining an estimated value of a posture angle of at least one of a plurality of body parts of the user using sensor data indicating a movement of at least a part of the body part of the user and a learning model; The learning model is trained to output the estimated value of the attitude angle when the sensor data is input. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a schematic configuration of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram for explaining a local coordinate system and a global coordinate system. [Figure 3] FIG. 2 is a block diagram showing a configuration of the information processing system shown in FIG. [Figure 4] FIG. 1 is a block diagram showing the configuration of a transformer. [Figure 5] FIG. 1 is a block diagram showing the configuration of "Multi-Head Attention." [Figure 6] FIG. 1 is a block diagram showing the configuration of "Scaled Dot-Product Attention." [Figure 7] FIG. 10 is a diagram illustrating an example of a combination of sensor data. [Figure 8] 10 is a graph showing the evaluation results. [Figure 9] FIG. 1 is a diagram showing a subject. [Figure 10] 1 is a graph of the posture angle of the subject's neck. [Figure 11] 1 is a graph of the posture angle of the subject's chest. [Figure 12] 1 is a graph of the posture angle of the subject's right upper arm. [Figure 13] 1 is a graph of the posture angle of the left upper arm of a subject. [Figure 14] 1 is a graph of the posture angle of the subject's right forearm. [Figure 15] 1 is a graph of the posture angle of the left forearm of a subject. [Figure 16] 1 is a graph of the postural angle of the subject's right thigh. [Figure 17] 1 is a graph of the posture angle of the left thigh of a subject. [Figure 18] 10 is a graph of the posture angle of the right lower leg of the subject. [Figure 19] 10 is a graph of the posture angle of the left lower leg of the subject. [Figure 20] 1 is a graph of the posture angle of the subject's right foot. [Figure 21] 1 is a graph of the posture angle of the left foot of a subject. [Figure 22] This is a graph of the posture angle of the right thigh of subjects who received a high evaluation of center of gravity movement. [Figure 23] This is a graph of the posture angle of the right thigh of subjects who received a high evaluation of center of gravity movement. [Figure 24] This is a graph of the posture angle of the right thigh of subjects who received a high evaluation of center of gravity movement. [Figure 25] This is a graph of the posture angle of the right thigh of subjects who received a high evaluation of center of gravity movement. [Figure 26] 10 is a graph showing the posture angle of the right thigh of a subject who received a low evaluation of center of gravity movement. [Figure 27] 10 is a graph showing the posture angle of the right thigh of a subject who received a low evaluation of center of gravity movement. [Figure 28] 10 is a graph showing the posture angle of the right thigh of a subject who received a low evaluation of center of gravity movement. [Figure 29] 10 is a graph showing the posture angle of the right thigh of a subject who received a low evaluation of center of gravity movement. [Figure 30] 10 is a graph showing the posture angle of the right upper arm of subjects who received a high evaluation of center of gravity movement. [Figure 31] 10 is a graph showing the posture angle of the right upper arm of subjects who received a high evaluation of center of gravity movement. [Figure 32] 10 is a graph showing the posture angle of the right upper arm of subjects who received a high evaluation of center of gravity movement. [Figure 33] 10 is a graph showing the posture angle of the right upper arm of subjects who received a high evaluation of center of gravity movement. [Figure 34] 10 is a graph showing the posture angle of the right upper arm of a subject who received a low evaluation of center of gravity movement. [Figure 35] 10 is a graph showing the posture angle of the right upper arm of a subject who received a low evaluation of center of gravity movement. [Figure 36] 10 is a graph showing the posture angle of the right upper arm of a subject who received a low evaluation of center of gravity movement. [Figure 37] 10 is a graph showing the posture angle of the right upper arm of a subject who received a low evaluation of center of gravity movement. [Figure 38] 4 is a flowchart showing the operation of a posture angle estimation process executed by the electronic device shown in FIG. [Figure 39] FIG. 10 is a block diagram showing a configuration of an information processing system according to another embodiment of the present disclosure. [Figure 40] FIG. 40 is a sequence diagram showing the operation of the estimation process executed by the information processing system shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] There is a need for an improved technique for detecting user movement. For example, conventional techniques require the installation of a camera. According to an embodiment of the present disclosure, an improved technique for detecting user movement can be provided.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the following drawings, the same components are denoted by the same reference numerals.

[0013] (System configuration) An information processing system 1 as shown in FIG. 1 can estimate posture angles of any body part of a user performing periodic motion. For example, the information processing system 1 can generate a model such as a 3D animation showing the user performing periodic motion by estimating posture angles of body parts throughout the user's entire body. The periodic motion may be any motion. For example, the periodic motion is walking, running, pedaling a bicycle, or the like. In this embodiment, the periodic motion is assumed to be walking. That is, in this embodiment, the information processing system 1 estimates posture angles of body parts of a user performing periodic motion. For example, a user may walk as an exercise in their daily life.

[0014] The information processing system 1 includes a sensor device 10A, a sensor device 10B, a sensor device 10C, sensor devices 10D-1 and 10D-2, sensor devices 10E-1 and 10E-2, sensor devices 10F-1 and 10F-2, and an electronic device 20. However, the information processing system 1 does not need to include all of the sensor devices 10A, 10B, 10C, 10D-1, 10D-2, 10E-1, 10E-2, 10F-1, and 10F-1. The information processing system 1 only needs to include at least one of the sensor devices 10A, 10B, 10C, 10D-1, 10D-2, 10E-1, 10E-2, 10F-1, and 10F-1.

[0015] Hereinafter, when the sensor devices 10D-1 and 10D-2 are not particularly distinguished from one another, they are also collectively referred to as "sensor device 10D." When the sensor devices 10E-1 and 10E-2 are not particularly distinguished from one another, they are also collectively referred to as "sensor device 10E." When the sensor devices 10F-1 and 10F-2 are not particularly distinguished from one another, they are also collectively referred to as "sensor device 10F." When the sensor devices 10A to 10D are not particularly distinguished from one another, they are also collectively referred to as "sensor device 10."

[0016] The sensor device 10 and the electronic device 20 can communicate with each other via a communication line. The communication line may be wired or wireless.

[0017] As shown in FIG. 2, the local coordinate system is a coordinate system based on the position of the sensor device 10. In FIG. 2, the position of the sensor device 10A is indicated by a dashed line as an example of the position of the sensor device 10. The local coordinate system is composed of, for example, an x-axis, a y-axis, and a z-axis. The x-axis, y-axis, and z-axis are perpendicular to one another. The x-axis is parallel to the front-to-back direction as seen from the sensor device 10. The y-axis is parallel to the left-to-right direction as seen from the sensor device 10. The z-axis is parallel to the up-down direction as seen from the sensor device 10. The positive and negative directions of the x-axis, y-axis, and z-axis may be set according to the configuration of the information processing system 1, etc.

[0018] As shown in FIG. 2 , the global coordinate system is a coordinate system based on a position in the space where the user walks. The global coordinate system is composed of, for example, an X-axis, a Y-axis, and a Z-axis. The X-axis, the Y-axis, and the Z-axis are perpendicular to each other. The X-axis is parallel to the front-to-back direction as seen from the user. In this embodiment, the positive direction of the X-axis is the direction from the rear of the user to the front of the user. The negative direction of the X-axis is the direction from the front of the user to the rear of the user. The Y-axis is parallel to the up-down direction as seen from the user. In this embodiment, the positive direction of the Y-axis is the direction from the bottom of the user to the top of the user. The negative direction of the Y-axis is the direction from the top of the user to the bottom of the user. The Z-axis is parallel to the left-to-right direction as seen from the user. In this embodiment, the positive direction of the Z-axis is the direction from the left to the right of the user. The negative direction of the Z-axis is the direction from the right to the left of the user. However, the positive and negative directions of the X-axis, Y-axis, and Z-axis may be set according to the configuration of the information processing system 1, etc.

[0019] As shown in Fig. 1, the sensor device 10 is attached to at least a part of a body part of a user. The sensor device 10 detects sensor data indicating the movement of the body part of the user to which the sensor device 10 is attached. The sensor data is data in a local coordinate system. The sensor data is data indicating the movement of at least a part of the body part of the user.

[0020] The sensor device 10A is worn on the user's head. For example, the sensor device 10A is worn on the user's ear. The sensor device 10A may be a wearable device. The sensor device 10A may be an earphone or may be included in an earphone. Alternatively, the sensor device 10A may be a device that can be retrofitted to existing glasses, earphones, or the like. The sensor device 10A may be worn on the user's head by any method. The sensor device 10A may be worn on the user's head by being attached to a hair accessory such as a hairband or a hairpin, earrings, a helmet, a hat, a hearing aid, dentures, an implant, or the like.

[0021] The sensor device 10A may be attached to the user's head so that the x-axis of a local coordinate system based on the position of the sensor device 10A is parallel to the front-to-back direction of the head as seen by the user, the y-axis of the local coordinate system is parallel to the left-to-right direction of the head as seen by the user, and the z-axis of the local coordinate system is parallel to the up-to-down direction of the head as seen by the user. However, the x-axis, y-axis, and z-axis of the local coordinate system based on the position of the sensor device 10A do not necessarily correspond to the front-to-back direction, left-to-right direction, and up-to-down direction of the head as seen by the user, respectively. In this case, the relative orientation of the sensor device 10A with respect to the user's head may be initialized or known as appropriate. The initialization or knowing of the relative orientation may be performed using information on the shape of a jig used to attach the sensor device 10A to the user's head or image information generated by capturing an image of the user's head wearing the sensor device 10A.

[0022] The sensor device 10A detects sensor data indicating the movement of the user's head. The sensor data detected by the sensor device 10A includes, for example, at least one of data on the speed of the user's head, the acceleration of the user's head, the angle of the user's head, the angular velocity of the user's head, the temperature of the user's head, and the geomagnetism at the position of the user's head.

[0023] The sensor device 10B is worn on the user's forearm. For example, the sensor device 10B is worn on the user's wrist. The sensor device 10B may be worn on the user's left forearm or on the user's right forearm. The sensor device 10B may be a wristwatch-type wearable device. The sensor device 10B may be worn on the user's forearm by any method. The sensor device 10B may be attached to a band, bracelet, friendship bracelet, glove, ring, false nail, prosthetic arm, or the like, and thereby worn on the user's forearm. The bracelet may be worn by the user for decorative purposes, or may be used to attach a key to a locker or the like to the wrist.

[0024] The sensor device 10B may be worn on the user's forearm so that the x-axis of a local coordinate system based on the position of the sensor device 10B is parallel to the front-to-back direction of the wrist as seen by the user, the y-axis of the local coordinate system is parallel to the left-to-right direction of the wrist as seen by the user, and the z-axis of the local coordinate system is parallel to the rotation direction of the wrist as seen by the user. The rotation direction of the wrist is, for example, the direction in which the wrist twists and rotates.

[0025] The sensor device 10B detects sensor data indicating the movement of the user's forearm. For example, the sensor device 10B detects sensor data indicating the movement of the wrist. The sensor data detected by the sensor device 10B includes, for example, at least one of the following data: the speed of the user's forearm, the acceleration of the user's forearm, the angle of the user's forearm, the angular velocity of the user's forearm, the temperature of the user's forearm, and the geomagnetism at the position of the user's forearm.

[0026] The sensor device 10C is attached to the user's waist. The sensor device 10C may be a wearable device. The sensor device 10C may be attached to the user's waist by a belt, a clip, or the like.

[0027] The sensor device 10C may be attached to the waist of the user so that the x-axis of a local coordinate system based on the position of the sensor device 10C coincides with the front-to-back direction of the waist as seen by the user, the y-axis of the local coordinate system coincides with the left-to-right direction of the waist as seen by the user, and the z-axis of the local coordinate system coincides with the rotation direction of the waist as seen by the user. The rotation direction of the waist is, for example, the direction in which the waist twists and rotates.

[0028] The sensor device 10C detects sensor data indicating the movement of the user's waist. The sensor data detected by the sensor device 10C includes, for example, at least one of data on the speed of the user's waist, the acceleration of the user's waist, the angle of the user's waist, the angular velocity of the user's waist, the temperature of the user's waist, and the geomagnetism at the position of the user's waist.

[0029] The sensor device 10D-1 is worn on the user's left thigh. The sensor device 10D-2 is worn on the user's right thigh. The sensor device 10D may be a wearable device. The sensor device 10D may be worn on the user's thigh by any method. The sensor device 10D may be worn on the user's thigh by a belt, a clip, or the like. The sensor device 10D may be worn on the user's thigh by being placed in a pocket near the thigh of pants worn by the user. The sensor device 10D may be worn on the user's thigh by being attached to pants, underwear, shorts, a support, a prosthetic limb, an implant, or the like.

[0030] The sensor device 10D may be attached to the user's thigh so that the x-axis of a local coordinate system based on the position of the sensor device 10D is parallel to the front-to-back direction of the thigh as seen by the user, the y-axis of the local coordinate system is parallel to the left-to-right direction of the thigh as seen by the user, and the z-axis of the local coordinate system is parallel to the rotation direction of the thigh as seen by the user. The rotation direction of the thigh is, for example, the direction in which the thigh twists and rotates.

[0031] The sensor device 10D-1 detects sensor data indicating the movement of the user's left thigh. The sensor device 10D-2 detects sensor data indicating the movement of the user's right thigh. The sensor data detected by the sensor device 10D includes, for example, at least one of the following data: the speed of the user's thigh, the acceleration of the user's thigh, the angle of the user's thigh, the angular velocity of the user's thigh, the temperature of the user's thigh, and the geomagnetism at the position of the user's thigh.

[0032] The sensor device 10E-1 is worn on the user's left ankle. The sensor device 10E-2 is worn on the user's right ankle. The sensor device 10E may be a wearable device. The sensor device 10E may be worn on the user's ankle by any method. The sensor device 10E may be worn on the user's ankle by a belt, a clip, or the like. The sensor device 10E may be worn on the user's ankle by being attached to an anklet, a band, a friendship bracelet, a tattoo sticker, a support, a cast, a sock, a prosthetic limb, an implant, or the like.

[0033] The sensor device 10E may be attached to the ankle of the user so that the x-axis of a local coordinate system based on the position of the sensor device 10E coincides with the front-to-back direction of the ankle as seen by the user, the y-axis of the local coordinate system coincides with the left-to-right direction of the ankle as seen by the user, and the z-axis of the local coordinate system coincides with the rotation direction of the ankle as seen by the user. The rotation direction of the ankle is, for example, the direction in which the ankle twists and rotates.

[0034] The sensor device 10E-1 detects sensor data indicating the movement of the user's left ankle. The sensor device 10E-2 detects sensor data indicating the movement of the user's right ankle. The sensor data detected by the sensor device 10E includes, for example, at least one of the following data: the speed of the user's ankle, the acceleration of the user's ankle, the angle of the user's ankle, the angular velocity of the user's ankle, the temperature of the user's ankle, and the geomagnetism at the position of the user's ankle.

[0035] The sensor device 10F-1 is worn on the user's left foot. The sensor device 10F-2 is worn on the user's right foot. In this embodiment, the foot is the part of the user from the ankle to the toes. The sensor device 10F may be a shoe-shaped wearable device. The sensor device 10F may be provided in a shoe. The sensor device 10F may be attached to the user's foot by any method. The sensor device 10F may be attached to the user's foot by being attached to an anklet, band, friendship bracelet, false nail, tattoo sticker, supporter, cast, sock, insole, prosthetic limb, ring, implant, or the like.

[0036] The sensor device 10F may be attached to the user's foot so that the x-axis of a local coordinate system based on the position of the sensor device 10F is parallel to the front-to-back direction of the foot as seen by the user, the y-axis of the local coordinate system is parallel to the left-to-right direction of the foot as seen by the user, and the z-axis of the local coordinate system is parallel to the up-down direction of the foot as seen by the user.

[0037] The sensor device 10F-1 detects sensor data indicating the movement of the user's left foot. The sensor device 10F-2 detects sensor data indicating the movement of the user's right ankle. The sensor data detected by the sensor device 10F includes, for example, at least one of the following data: the speed of the user's foot, the acceleration of the user's foot, the angle of the user's foot, the angular velocity of the user's foot, the temperature of the user's foot, and the geomagnetism at the position of the user's foot.

[0038] The electronic device 20 is carried by, for example, a user who is walking. The electronic device 20 is, for example, a mobile device such as a mobile phone, a smartphone, or a tablet.

[0039] In this embodiment, the electronic device 20 functions as an information processing device and acquires an estimated value of the posture angle of a user's body part using sensor data detected by the sensor device 10 and a learning model (described later). In this embodiment, the posture angle of a body part is the angle of the body part in the global coordinate system. Hereinafter, regarding the posture angle of a body part, the angle at which the body part rotates around the X-axis is also referred to as the "posture angle θX." The angle at which the body part rotates around the Y-axis is also referred to as the "posture angle θY." The angle at which the body part rotates around the Z-axis is also referred to as the "posture angle θZ." In this embodiment, the positive direction of the posture angle θX is the direction of clockwise rotation around the X-axis when viewed from the negative direction of the X-axis. The negative direction of the posture angle θX is the direction of counterclockwise rotation around the X-axis when viewed from the negative direction of the X-axis. The positive direction of the posture angle θY is the direction of clockwise rotation around the Y-axis when viewed from the negative direction of the Y-axis. The negative direction of the attitude angle θY is the direction of counterclockwise rotation around the Y axis when viewed from the negative direction of the Y axis. The positive direction of the attitude angle θZ is the direction of clockwise rotation around the Z axis when viewed from the negative direction of the Z axis. The negative direction of the attitude angle θZ is the direction of counterclockwise rotation around the Z axis when viewed from the negative direction of the Z axis.

[0040] [Sensor equipment configuration] 3, the sensor device 10 includes a communication unit 11, a sensor unit 12, a notification unit 13, a storage unit 15, and a control unit 16. However, the sensor devices 10C to 10F do not necessarily include the notification unit 13.

[0041] The communication unit 11 includes at least one communication module capable of communicating with the electronic device 20 via a communication line. The communication module is a communication module that complies with the standard of the communication line. The standard of the communication line is, for example, a short-range wireless communication standard including Bluetooth (registered trademark), infrared, and NFC (Near Field Communication).

[0042] The sensor unit 12 is configured to include any sensor corresponding to the sensor data to be detected by the sensor device 10. The sensor unit 12 is configured to include at least one of, for example, a three-axis motion sensor, a three-axis acceleration sensor, a three-axis speed sensor, a three-axis gyro sensor, a three-axis geomagnetic sensor, a temperature sensor, an inertial measurement unit (IMU), and a camera. When the sensor unit 12 is configured to include a camera, the camera can capture an image of a body part of the user and analyze the generated image to detect the movement of the body part.

[0043] When the sensor unit 12 is configured to include an acceleration sensor and a geomagnetic sensor, data detected by each of the acceleration sensor and the geomagnetic sensor may be used to calculate the initial angle of the body part to be detected by the sensor device 10. In addition, data detected by each of the acceleration sensor and the geomagnetic sensor may be used to correct the angle data detected by the sensor device 10.

[0044] When the sensor unit 12 is configured to include a gyro sensor, the angle of the body part to be detected by the sensor device 10 may be calculated by integrating the angular velocity detected by the gyro sensor over time.

[0045] The notification unit 13 notifies information. In this embodiment, the notification unit 13 includes an output unit 14. However, the notification unit 13 is not limited to the output unit 14. The notification unit 13 may include any component capable of notifying information.

[0046] The output unit 14 is capable of outputting data. The output unit 14 includes at least one output interface capable of outputting data. The output interface is, for example, a display or a speaker. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0047] When the output unit 14 is included in the sensor device 10A, the output unit 14 may be configured to include a speaker. When the output unit 14 is included in the sensor device 10B, the output unit 14 may be configured to include a display.

[0048] The storage unit 15 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a random access memory (RAM) or a read-only memory (ROM). The RAM is, for example, a static random access memory (SRAM) or a dynamic random access memory (DRAM). The ROM is, for example, an electrically erasable programmable read-only memory (EEPROM). The storage unit 15 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 15 stores data used in the operation of the sensor device 10 and data obtained by the operation of the sensor device 10. For example, the storage unit 15 stores system programs, application programs, embedded software, etc.

[0049] The control unit 16 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 16 controls each part of the sensor device 10 and executes processes related to the operation of the sensor device 10.

[0050] The control unit 16 receives a signal from the electronic device 20 via the communication unit 11 instructing the start of data detection. Upon receiving this signal, the control unit 16 starts data detection. For example, the control unit 16 acquires data detected by the sensor unit 12 from the sensor unit 12. The control unit 16 transmits the acquired data as sensor data to the electronic device 20 via the communication unit 11. The signal instructing the start of data detection is transmitted as a broadcast signal from the electronic device 20 to the multiple sensor devices 10. By transmitting the signal instructing the start of data detection to the multiple sensor devices 10 as a broadcast signal, the multiple sensor devices 10 can simultaneously start data detection.

[0051] The control unit 16 acquires data from the sensor unit 12 at a preset time interval and transmits the acquired data as sensor data via the communication unit 11. This time interval may be set based on the walking speed of a typical user, etc. This time interval may be the same for each of the multiple sensor devices 10. By setting this time interval to be the same for the multiple sensor devices 10, the timing at which each of the multiple sensor devices 10 detects data can be synchronized.

[0052] [Electronic device configuration] As shown in FIG. 3, the electronic device 20 includes a communication unit 21, an input unit 22, a notification unit 23, a storage unit 26, and a control unit 27.

[0053] The communication unit 21 includes at least one communication module capable of communicating with the sensor device 10 via a communication line. The communication module is at least one communication module compatible with the standard of the communication line. The standard of the communication line is, for example, a short-range wireless communication standard including Bluetooth (registered trademark), infrared, and NFC.

[0054] The communication unit 21 may further include at least one communication module connectable to a network 2 as shown in Fig. 39 described below. The communication module is a communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation).

[0055] The input unit 22 can accept input from a user. The input unit 22 includes at least one input interface that can accept input from a user. The input interface is, for example, a physical key, a capacitance key, a pointing device, a touch screen that is integrated with a display, a microphone, or the like.

[0056] The notification unit 23 notifies information. In this embodiment, the notification unit 23 includes an output unit 24 and a vibration unit 25. However, the notification unit 23 is not limited to the output unit 24 and the vibration unit 25. The notification unit 23 may include any components capable of notifying information. The output unit 24 and the vibration unit 25 may be mounted on the electronic device 20, or may be arranged near any of the sensor devices 10B to 10F.

[0057] The output unit 24 is capable of outputting data. The output unit 24 includes at least one output interface capable of outputting data. The output interface is, for example, a display or a speaker. The display is, for example, an LCD or an organic EL display.

[0058] The vibration section 25 is capable of vibrating the electronic device 20. The vibration section 25 includes a vibration element. The vibration element is, for example, a piezoelectric element.

[0059] The storage unit 26 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 26 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 26 stores data used in the operation of the electronic device 20 and data obtained by the operation of the electronic device 20. For example, the storage unit 26 stores system programs, application programs, embedded software, and the like. For example, the storage unit 26 stores data of a transformer 30 and data used by the transformer 30, as shown in FIG. 4 (described later).

[0060] The control unit 27 is configured to include at least one processor, at least one dedicated circuit, or a combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA or an ASIC. The control unit 27 controls each unit of the electronic device 20 and executes processes related to the operation of the electronic device 20. The control unit 27 may execute processes executed by a transformer 30 as shown in FIG. 4, which will be described later.

[0061] The control unit 27 receives an input instructing the execution of a posture angle estimation process via the input unit 22. This input causes the electronic device 20 to execute a process of estimating posture angles of body parts of the user. This input is, for example, input from the input unit 22 by a user wearing the sensor device 10. The user inputs this input from the input unit 22, for example, before starting to walk. The control unit 27 may also receive an input indicating the user's height via the input unit 22 along with the input instructing the execution of this estimation process. When the control unit 27 receives an input indicating the user's height via the input unit 22, it may store the received data on the user's height in the storage unit 26.

[0062] When the control unit 27 receives an input instructing execution of an attitude angle estimation process via the input unit 22, the control unit 27 transmits a signal instructing the start of data detection as a broadcast signal to the plurality of sensor devices 10 via the communication unit 21. After the signal instructing the start of data detection is transmitted to the plurality of sensor devices 10, sensor data is transmitted from at least one sensor device 10 to the electronic device 20.

[0063] The control unit 27 receives sensor data from at least one sensor device 10 via the communication unit 21. The control unit 27 acquires the sensor data from the sensor device 10 by receiving the sensor data from the sensor device 10.

[0064] The control unit 27 acquires an estimated value of the posture angle of at least one of the plurality of body parts of the user based on the sensor data and the learning model. When using sensor data in the global coordinate system, the control unit 27 may acquire the sensor data in the global coordinate system by performing coordinate transformation on the sensor data in the local coordinate system acquired from the sensor device 10.

[0065] The learning model is machine-learned to output an estimated value of at least one posture angle of a body part of a user when sensor data is input. The body part for which the learning model outputs an estimated value may be set appropriately depending on the application. In this embodiment, the control unit 27 uses the Transformer described in "Ashish Vaswani et al., "Attention Is All You Need," June 12, 2017, arXiv:1706.03762v5 [cs.CL]" as the learning model. The Transformer can process time-series data. The Transformer will be described later with reference to FIG. 4. However, the learning model is not limited to the Transformer. The control unit 27 may use a learning model generated by machine learning based on any machine learning algorithm.

[0066] The control unit 27 may acquire time-series data of estimated values ​​of posture angles of body parts throughout the user's entire body using a learning model. The control unit 27 may generate a gait model using the time-series data of estimated values ​​of posture angles of body parts throughout the user's entire body and time-series data of the movement speed of the user's waist. The movement speed of the user's waist is a speed in a global coordinate system. The control unit 27 may acquire the time-series data of the movement speed of the user's waist by converting sensor data detected by the sensor device 10C into data in the global coordinate system. Alternatively, the control unit 27 may acquire the time-series data of the movement speed of the user's waist using a learning model. When a transformer (described later) is used as the learning model, the control unit 27 may acquire a normalized speed of the user's waist (described later) from the transformer and multiply the acquired normalized speed by the user's height to calculate data of the movement speed of the user's waist. Here, the generated gait model is a model that represents the user's walking. The control unit 27 may generate the gait model as a 3D animation. The control unit 27 may generate a walking model by scaling a humanoid model of a predetermined size according to the height of the user. The body parts of the user's entire body used to generate the walking model may be set appropriately around the waist. For example, the body parts of the user's entire body used to generate the walking model include the user's head, neck, chest, lumbar spine, pelvis, right and left thighs, right and left lower legs, right and left feet, right and left upper arms, and right and left forearms, as shown in FIG. 2. However, the body parts of the user's entire body may be set appropriately.

[0067] The control unit 27 may output data of the generated walking model to the output unit 24. If the output unit 24 is configured to include a display, the control unit 27 may display the generated walking model on the display of the output unit 24. With such a configuration, the user can understand how he or she is walking.

[0068] The control unit 27 may display the generated walking model as a three-dimensional animation on the display of the output unit 24. In this case, the control unit 27 may display the three-dimensional animation of the walking model as a free viewpoint video on the display of the output unit 24 based on the user's input received by the input unit 22. With this configuration, the user can grasp in detail how he or she is walking.

[0069] The control unit 27 may transmit data on estimated values ​​of the user's body parts obtained by the learning model or data on the generated gait model to an external device via a network 2 as shown in FIG. 39 (described later) using the communication unit 21. For example, the user may be receiving instruction on his / her walking style from an instructor. In this case, by transmitting data on estimated values ​​of the user's body parts or data on the gait model to the external device, the instructor can grasp the user's walking style via the external device and provide instruction on walking to the user. Alternatively, the user may be the instructor. In this case, by transmitting data on estimated values ​​of the user's body parts or data on the gait model to the external device, students can view the instructor's walking style as a model to follow via the external device.

[0070] [Transformer configuration] The transformer 30 shown in FIG. 4 can be trained to output time series data of estimated values ​​of posture angles of preset body parts of the user when multiple time-series sensor data are input. The transformer 30 can also be trained to output time series data of normalized waist velocity in addition to the time series data of estimated values ​​of posture angles of the user's body parts. The normalized waist velocity of the user is normalized by dividing the movement velocity of the user's waist by the user's height. The time range and time interval of the time-series sensor data input to the transformer 30 may be set according to the desired estimation accuracy, etc.

[0071] As shown in FIG. 4, transformer 30 includes encoder 40 and decoder 50. Encoder 40 includes functional unit 41, functional unit 42, and N-stage layer 43. Layer 43 includes functional unit 44, functional unit 45, functional unit 46, and functional unit 47. Decoder 50 includes functional unit 51, functional unit 52, N-stage layer 53, functional unit 60, and functional unit 61. Layer 53 includes functional unit 54, functional unit 55, functional unit 56, functional unit 57, functional unit 58, and functional unit 59. The number of stages of layers 43 included in encoder 40 and the number of stages of layers 53 included in decoder 50 are both N (N is a natural number).

[0072] The function unit 41 is also referred to as "Input Embedding." A sequence of multiple pieces of sensor data along a time series is input to the function unit 41. For example, the sensor data at time ti (0≦i≦n) is expressed as "D ti ", the sequence of the sensor data input to the function unit 41 is (D t0 ,D t1 ,…,D tn ) The function unit 41 may receive an array that combines multiple types of sensor data. For example, two pieces of sensor data at different times ti (0≦i≦n) may be expressed as "Da ti " and "Db ti ", the array of sensor data input to the function unit 41 is (Da t0 ,Da t1 ,…,Da tn ,Db t0 ,Db t1 ,…,Db tn )

[0073] The function unit 41 generates a distributed vector by converting each element of the array of input sensor data into a multidimensional vector. The number of dimensions of the multidimensional vector may be set in advance.

[0074] The function unit 42 is also referred to as “Positional Encoding.” The function unit 42 adds position information to the distributed vector.

[0075] The function unit 42 calculates and adds position information for each element of the distributed representation of the vector. The position information indicates the position of each element of the distributed representation of the vector in the array of sensor data input to the function unit 41 and the position in the element array of the distributed representation of the vector. The function unit 42 calculates position information PE of the (2×i)th element in the array of elements of the distributed representation of the vector using equation (1). The function unit 42 calculates position information PE of the (2×i+1)th element in the array of elements of the distributed representation of the vector using equation (2).

number

number

[0076] In the N-stage layer 43, the first-stage layer 43 receives a vector that has been assigned position information and is represented in a distributed manner from the function unit 42. The second and subsequent layers 43 receive a vector from the previous layer 43.

[0077] The function unit 44 is also referred to as "Multi-Head Attention." A Q (Query) vector, a K (Key) vector, and a V (Value) vector are input to the function unit 44. The Q vector is obtained by multiplying the vector input to the layer 43 by a weight matrix WQ. The K vector is obtained by multiplying the vector input to the layer 43 by a weight matrix WK. The V vector is obtained by multiplying the vector input to the layer 43 by a weight matrix WV. During learning, the transformer 30 learns the weight matrix WQ, the weight matrix WK, and the weight matrix WV.

[0078] 5, the functional unit 44 includes h functional units 70 and the functional units "Linear" and "Contact." The functional unit 70 is also referred to as "Scaled Dot-Product Attention." The functional unit 70 receives h divided Q vectors, K vectors, and V vectors.

[0079] The function unit 70 includes the function units "MatMul", "Scale", "Mask(opt.)", and "Softmax" as shown in Fig. 6. The function unit 70 calculates the scaled dot-product attention using the Q vector, K vector, and V vector and equation (3).

number

[0080] The function unit 44 calculates the multi-head attention by calculating the scale dot product attention using h function units 70 as shown in Fig. 5. The function unit 44 calculates the multi-head attention using equation (4).

number

[0081] The multi-head attention calculated by the function unit 44 is input to a function unit 45 as shown in FIG.

[0082] The functional unit 45 is also referred to as "Add & Norm." The functional unit 45 normalizes the vector input to the layer 43 by adding the multi-head attention calculated by the functional unit 44. The functional unit 45 inputs the normalized vector to the functional unit 46.

[0083] The functional unit 46 is also referred to as "Position-wise Feed-Forward Networks." The functional unit 46 generates an output using an activation function such as a ReLU (Rectified Linear Unit) and a vector input from the functional unit 45. The functional unit 46 uses a different FFN (Feed-Forward Network) for each position in the element array of the time-series sensor data before vectorization, i.e., the time-series sensor data input to the functional unit 41. If the vector input from the functional unit 45 to the functional unit 46 is referred to as "x," the functional unit 46 generates an output FFN(x) using equation (5).

number

[0084] The function unit 47 is also referred to as “Add & Norm.” The function unit 47 adds the output generated by the function unit 46 to the vector output by the function unit 45, thereby normalizing the vector.

[0085] The functional unit 51 is also referred to as "Input Embedding." Time series data, such as estimated values ​​of posture angles of body parts, output by the decoder 50 in the previous processing, is input to the functional unit 51. When the decoder 50 estimates data, such as posture angles of body parts at the initial time, preset data, such as dummy data, may be input to the functional unit 51. The functional unit 51, in the same way as or similar to the functional unit 41, converts each element of the input time series data into a multidimensional vector to generate a distributed representation vector. In the same way as or similar to the functional unit 41, the number of dimensions of the multidimensional vector may be preset.

[0086] The functional unit 52 is also referred to as "Positional Encoding." The functional unit 52 assigns position information to the distributed representation of the vector in the same way as or similar to the functional unit 42. That is, the functional unit 52 calculates and adds position information for each element of the distributed representation of the vector. The position information indicates the position of each element of the distributed representation of the vector in the array of time-series data input to the functional unit 51 and the position in the element array of the distributed representation of the vector.

[0087] In the N-stage layer 53, the first-stage layer 53 receives a vector that has been assigned position information and is represented in a distributed manner from the function unit 52. The second and subsequent layers 53 receive a vector from the previous layer 53.

[0088] Functional unit 54 is also referred to as "Masked Multi-Head Attention." Functional unit 54 receives Q, K, and V vectors in the same or similar manner as functional unit 44. The Q, K, and V vectors are obtained by multiplying the vectors input to layer 53 by the same or different weight matrices. Transformer 30 learns these weight matrices during training. Functional unit 54 calculates multi-head attention using the Q, K, and V vectors input in the same or similar manner as functional unit 44.

[0089] Here, when the transformer 30 is training, the correct time-series data such as posture angles of body parts is input all at once to the function unit 54. When the transformer 30 is training, the function unit 54 masks the time-series data such as posture angles of body parts after the time that the decoder 50 should estimate.

[0090] The function unit 55 is also referred to as “Add & Norm.” The function unit 55 normalizes the vector input to the layer 53 by adding the multi-head attention calculated by the function unit 54 to the vector.

[0091] The functional unit 56 is also referred to as "Multi-Head Attention." A Q vector, a K vector, and a V vector are input to the functional unit 56. The Q vector is a normalized vector input to the functional unit 56 by the functional unit 55. The K vector and the V vector are obtained by multiplying the vector output by the final layer 43 of the encoder 40 by the same or different weighting matrices. The functional unit 56 calculates multi-head attention using the input Q vector, K vector, and V vector in the same or similar manner as the functional unit 44.

[0092] The functional unit 57 is also referred to as "Add & Norm." The functional unit 57 adds the multi-head attention calculated by the functional unit 56 to the vector output by the functional unit 55, and normalizes the vector.

[0093] The functional unit 58 is also referred to as a “Position-wise Feed-Forward Network.” The functional unit 58 generates an output using an activation function such as ReLU and the vector input from the functional unit 57, in the same manner as or similar to the functional unit 46.

[0094] The functional unit 59 is also referred to as “Add & Norm.” The functional unit 59 adds the output generated by the functional unit 58 to the vector output by the functional unit 57, thereby normalizing the vector.

[0095] The functional unit 60 is also referred to as "Linear." The functional unit 61 is also referred to as "SoftMax." The output of the final layer 53 is normalized by the functional units 60 and 61, and then output from the decoder 50 as data of estimated values ​​of posture angles of body parts, etc.

[0096] Here, walking speeds vary from user to user. In other words, walking periods vary from user to user. A walking period is the period from when one of a user's two feet lands on the ground or the like until it lands on the ground or the like again. Even if walking periods vary from user to user, the characteristics of the user's walking will appear in the time series data as long as there is time series data of sensor data that is about half the average value of the walking period. Furthermore, even if the input time series data of sensor data is only part of the walking period, the transformer 30 can learn which part of the walking period the sensor data corresponds to. Therefore, the transformer 30 can be trained to output time series data of an estimated value of a posture angle when time series data of sensor data that is about half the average value of the walking period is input.

[0097] [Sensor data combination] The control unit 27 may use a transformer that has learned one type of sensor data, or may use a transformer that has learned a combination of multiple types of sensor data. Examples of combinations of multiple types of sensor data include cases C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, and C13 shown in FIG.

[0098] FIG. 7 shows an example of a combination of sensor data. Cases C1 to C13 are examples of the combination of sensor data. The control unit 27 may select one of the cases C1 to C13 depending on the type of sensor device 10 that transmitted the sensor data to the electronic device 20. The data of the transformer 30 used in each of the cases C1 to C13 may be associated with the case C1 to C13 and stored in the storage unit 26. The control unit 27 inputs the sensor data of one of the selected cases C1 to C13 to the transformer 30 corresponding to the selected case C1 to C13, thereby acquiring an estimated value of the posture angle of the user's body part.

[0099] <Case C1> When the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor device 10A, the control unit 27 may select case C1.

[0100] In case C1, sensor data indicating the movement of the user's head is used. In case C1, sensor data D10AG and sensor data D10AL are used.

[0101] The sensor data D10AG is sensor data that indicates the movement of the user's head in a global coordinate system. The sensor data D10AG includes velocity data and acceleration data of the user's head on the X-axis, the Y-axis, and the Z-axis of the global coordinate system. The control unit 27 acquires the sensor data D10AG by performing coordinate transformation on the sensor data in the local coordinate system acquired from the sensor device 10A.

[0102] The sensor data D10AL is sensor data that indicates the movement of the user's head in a local coordinate system based on the position of the sensor device 10A. The sensor data D10AL includes velocity data and acceleration data of the user's head on the x-axis, velocity data and acceleration data of the user's head on the y-axis, and velocity data and acceleration data of the user's head on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10AL from the sensor device 10A.

[0103] <Case C2> The control unit 27 may select case C2 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor device 10A and the sensor device 10E-1 or the sensor device 10E-2.

[0104] In case C2, sensor data indicating the movement of the user's head and sensor data indicating the movement of one of the user's two ankles are used. In case C2, sensor data D10AG, sensor data D10AL, and sensor data D10EL-1 or sensor data D10EL-2 are used.

[0105] The sensor data D10EL-1 is sensor data that indicates the movement of the user's left ankle in a local coordinate system based on the position of the sensor device 10E-1. The sensor data D10EL-1 includes velocity data and acceleration data of the user's left ankle on the x-axis, velocity data and acceleration data of the user's left ankle on the y-axis, and velocity data and acceleration data of the user's left ankle on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10EL-1 from the sensor device 10E-1.

[0106] The sensor data D10EL-2 is sensor data that indicates the movement of the user's right ankle in a local coordinate system based on the position of the sensor device 10E-2. The sensor data D10EL-2 includes velocity data and acceleration data of the user's right ankle on the x-axis, velocity data and acceleration data of the user's right ankle on the y-axis, and velocity data and acceleration data of the user's right ankle on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10EL-2 from the sensor device 10E-2.

[0107] <Case C3> The control unit 27 may select case C3 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor device 10A and the sensor device 10F-1 or the sensor device 10F-2.

[0108] In case C3, sensor data indicating the movement of the user's head and sensor data indicating the movement of one of the user's two feet are used. In case C3, sensor data D10AG, sensor data D10AL, and sensor data D10FL-1 or sensor data D10FL-2 are used.

[0109] The sensor data D10FL-1 is sensor data that indicates the movement of the user's left foot in a local coordinate system based on the position of the sensor device 10F-1. The sensor data D10FL-1 includes velocity data and acceleration data of the user's left foot on the x-axis, velocity data and acceleration data of the user's left foot on the y-axis, and velocity data and acceleration data of the user's left foot on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10FL-1 from the sensor device 10F-1.

[0110] The sensor data D10FL-2 is sensor data that indicates the movement of the user's right foot in a local coordinate system based on the position of the sensor device 10F-2. The sensor data D10FL-2 includes velocity data and acceleration data of the user's right foot on the x-axis, velocity data and acceleration data of the user's right foot on the y-axis, and velocity data and acceleration data of the user's right foot on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10FL-2 from the sensor device 10F-2.

[0111] <Case C4> The control unit 27 may select case C4 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor device 10A and the sensor device 10D-1 or the sensor device 10D-2.

[0112] In case C4, sensor data indicating the movement of the user's head and sensor data indicating the movement of one of the user's two thighs are used. In case C4, sensor data D10AG, sensor data D10AL, and sensor data D10DL-1 or sensor data D10DL-2 are used.

[0113] The sensor data D10DL-1 is sensor data that indicates the movement of the user's left thigh in a local coordinate system based on the position of the sensor device 10D-1. The sensor data D10DL-1 includes velocity data and acceleration data of the user's left thigh on the x-axis, velocity data and acceleration data of the user's left thigh on the y-axis, and velocity data and acceleration data of the user's left thigh on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10DL-1 from the sensor device 10D-1.

[0114] The sensor data D10DL-2 is sensor data that indicates the movement of the user's right thigh in a local coordinate system based on the position of the sensor device 10D-2. The sensor data D10DL-2 includes velocity data and acceleration data of the user's right thigh on the x-axis, velocity data and acceleration data of the user's right thigh on the y-axis, and velocity data and acceleration data of the user's right thigh on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10DL-2 from the sensor device 10D-2.

[0115] <Case C5> When the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor device 10A and the sensor device 10B, the control unit 27 may select case C5.

[0116] In case C5, sensor data indicating the movement of the user's head and sensor data indicating the movement of one of the user's two wrists are used. In case C5, sensor data D10AG, sensor data D10AL, and sensor data D10BL are used.

[0117] The sensor data D10BL is sensor data indicating the movement of the user's wrist in a local coordinate system based on the position of the sensor device 10B. In this embodiment, the sensor data D10BL is assumed to be sensor data indicating the movement of the user's left wrist. However, the sensor data D10BL may also be sensor data indicating the movement of the user's right wrist.

[0118] The sensor data D10BL includes velocity data and acceleration data of the user's wrist on the x-axis, velocity data and acceleration data of the user's wrist on the y-axis, and velocity data and acceleration data of the user's wrist on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10BL from the sensor device 10B.

[0119] <Case C6> The control unit 27 may select case C6 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor devices 10A and 10B and the sensor device 10E-1 or the sensor device 10E-2.

[0120] In case C6, sensor data indicating the movement of the user's head, sensor data indicating the movement of one of the user's two wrists, and sensor data indicating the movement of one of the user's two ankles are used. In case C6, sensor data D10AG, sensor data D10AL, sensor data D10BL, and sensor data D10EL-1 or sensor data D10EL-2 are used.

[0121] <Case C7> The control unit 27 may select case C7 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor devices 10A and 10B and the sensor device 10F-1 or the sensor device 10F-2.

[0122] In case C7, sensor data indicating the movement of the user's head, sensor data indicating the movement of one of the user's two wrists, and sensor data indicating the movement of one of the user's two feet are used. In case C7, sensor data D10AG, sensor data D10AL, sensor data D10BL, and sensor data D10FL-1 or sensor data D10FL-2 are used.

[0123] <Case C8> The control unit 27 may select case C8 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor devices 10A, 10B, 10F-1, and 10F-2.

[0124] In case C8, sensor data indicating the movement of the user's head, sensor data indicating the movement of one of the user's two wrists, and sensor data indicating the movement of each of the user's two feet are used. In case C8, sensor data D10AG, sensor data D10AL, sensor data D10BL, sensor data D10FL-1, and sensor data D10FL-2 are used.

[0125] <Case C9> The control unit 27 may select case C9 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor device 10F-1 and the sensor device 10F-2.

[0126] In case C9, sensor data indicating the movements of each of the user's two feet is used. In case C9, sensor data D10FL-1 and sensor data D10FL-2 are used.

[0127] <Case C10> The control unit 27 may select case C10 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor devices 10D-1 and 10D-2.

[0128] In case C10, sensor data indicating the movement of each of the user's two thighs is used. In case C10, sensor data D10DL-1 and sensor data D10DL-2 are used.

[0129] The sensor data D10DL-1 is sensor data that indicates the movement of the user's left thigh in a local coordinate system based on the position of the sensor device 10D-1. The sensor data D10DL-1 includes velocity data and acceleration data of the user's left thigh on the x-axis, velocity data and acceleration data of the user's left thigh on the y-axis, and velocity data and acceleration data of the user's left thigh on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10DL-1 from the sensor device 10D-1.

[0130] The sensor data D10DL-2 is sensor data that indicates the movement of the user's right thigh in a local coordinate system based on the position of the sensor device 10D-2. The sensor data D10DL-2 includes velocity data and acceleration data of the user's right thigh on the x-axis, velocity data and acceleration data of the user's right thigh on the y-axis, and velocity data and acceleration data of the user's right thigh on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10DL-2 from the sensor device 10D-2.

[0131] <Case C11> When the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor device 10C, the control unit 27 may select case C11.

[0132] In case C11, sensor data indicating the movement of the user's waist is used. In case C11, sensor data D10CG and sensor data D10CL are used.

[0133] The sensor data D10CG is sensor data that indicates the movement of the user's waist in a global coordinate system. The sensor data D10CG includes velocity data and acceleration data of the user's waist on the X-axis, velocity data and acceleration data of the user's waist on the Y-axis, and velocity data and acceleration data of the user's waist on the Z-axis of the global coordinate system. The control unit 27 may acquire the sensor data D10CG by performing coordinate transformation on the sensor data in the local coordinate system acquired from the sensor device 10C.

[0134] The sensor data D10CL is sensor data that indicates the movement of the user's waist in a local coordinate system based on the position of the sensor device 10C. The sensor data D10CL includes velocity data and acceleration data of the user's waist on the x-axis, velocity data and acceleration data of the user's waist on the y-axis, and velocity data and acceleration data of the user's waist on the z-axis of the local coordinate system. The control unit 27 acquires the sensor data D10CL from the sensor device 10C.

[0135] <Case C12> The control unit 27 may select case C12 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor device 10B and the sensor device 10C.

[0136] In case C12, sensor data indicating the movement of one of the user's two wrists and sensor data indicating the movement of the user's waist are used. In case C12, sensor data D10BL, sensor data D10CG, and sensor data D10CL are used.

[0137] <Case C13> The control unit 27 may select case C13 when the sensor devices 10 that have transmitted sensor data to the electronic device 20 include the sensor devices 10B, 10F-1, 10F-2, and 10C.

[0138] In case C13, sensor data indicating the movement of one of the user's two wrists, sensor data indicating the movement of each of the user's two feet, and sensor data indicating the movement of the user's waist are used. In case C13, sensor data D10BL, sensor data D10FL-1, sensor data D10FL-2, sensor data D10CG, and sensor data D10CL are used.

[0139] [Transformer generation and evaluation] The generation and evaluation of the transformer will be described below. The inventors generated and evaluated a transformer that outputs an estimate of the normalized velocity of the waist in addition to estimates of the posture angles of the user's entire body parts. The user's entire body parts include the user's head, neck, chest, lumbar spine, pelvis, right and left thighs, right and left lower legs, right and left feet, right and left upper arms, and right and left forearms.

[0140] The transformer was generated using a subject's walking database. The subject's walking database was found in "2019: AIST Walking Database 2019," by Yoshiyuki Kobayashi, Naoto Hida, Kanako Nakajima, Masahiro Fujimoto, and Masaaki Mochimaru, [Online], [Retrieved November 11, 2021], Internet.<https: / / unit.aist.go.jp / harc / ExPART / GDB2019_e.html> " was used. This gait database contains gait data for multiple subjects. The gait data for the subjects includes data showing the subject's movements while walking and data on the ground reaction forces acting on the subject while walking. The data showing the subject's movements while walking was detected by a motion capture system. The data on the ground reaction forces acting on the subject while walking was detected by a ground reaction force meter.

[0141] From the data indicating the subject's movements detected by the motion capture system of the gait database described above, data corresponding to the sensor data, data on the posture angle of the subject's body parts, and data on the movement speed of the waist were acquired. Data on the normalized velocity of the subject's waist was calculated by dividing the movement speed of the subject's waist by the subject's height. Data sets were generated by associating the data corresponding to the sensor data with the posture angle data of the body parts and the normalized velocity data of the waist. Data sets corresponding to Cases C1 to C13 described above with reference to FIG. 7 were generated. Sensor data indicating the movement of the subject's left wrist was used as sensor data D10BL for Case C6 described above with reference to FIG. 7. Furthermore, in Cases C2 and C6, sensor data D10EL-1 indicating the movement of the user's left ankle was used from sensor data D10EL-1 and sensor data D10EL-2. Furthermore, in Cases C3 and C7, sensor data D10FL-1 indicating the movement of the left foot was used from sensor data D10FL-1 and sensor data D10FL-2. In case C4, of the sensor data D10DL-1 and the sensor data D10DL-2, the sensor data D10DL-1 indicating the movement of the left thigh was used.

[0142] The generated dataset was used to train the transformer. During the training of the transformer, approximately 10% noise was added to the dataset to prevent overfitting.

[0143] The inventors evaluated the trained transformers using a dataset that was not used to train the transformers, and obtained evaluation results for cases C1 to C13 described above with reference to FIG.

[0144] Figure 8 shows a graph of the evaluation results. As the evaluation results, Figure 8 also shows a bar graph of the mean squared error (MSE) of the estimated values ​​of posture angles and other body parts across the entire body of the subject in each of cases C1 to C13. The mean squared error data shown in Figure 8 is data obtained from the subject shown in Figure 9, which will be described later. The mean squared error was calculated using the estimated values ​​of posture angle and waist movement speed obtained by the transformer and the actual measured values ​​of posture angle and waist movement speed obtained from the dataset. The mean squared error was calculated using the following equation (6).

number

[0145] As shown in Figure 8, in case C1, the mean square error is 6.328[(deg) 2 In case C1, only sensor data indicating the user's head movement is used. The results of case C1 show that the posture angles of the user's entire body parts can be estimated with a certain degree of accuracy using only sensor data indicating the user's head movement. This is presumably because the vertical movement of the user while walking is reflected in the head movement.

[0146] As shown in FIG. 8, the mean squared errors in Cases C2 to C8 were smaller than the mean squared error in Case C1. That is, the estimation accuracy of the posture angles of the user's body parts was improved in Cases C2 to C8 compared to Case C1. In Cases C2 to C8, as described above, in addition to sensor data indicating the movement of the user's head, sensor data indicating the movement of at least one of the user's wrists, ankles, feet, and thighs is used. That is, in Cases C2 to C8, in addition to sensor data indicating the movement of the user's trunk including the head, sensor data indicating the movement of the user's limbs including at least one of the user's wrists, ankles, feet, and thighs is used. The sensor data indicating the movement of the user's limbs and the sensor data indicating the movement of the user's trunk have significantly different patterns in one walking cycle. For example, due to the bilateral symmetry of the user's body, the sensor data indicating the movement of the user's limbs has one pattern in one walking cycle. In contrast, the sensor data indicating the movement of the user's trunk has two patterns in one walking cycle. In cases C2 to C8, it is presumed that the accuracy of estimating posture angles of body parts is improved compared to case C1 by using sensor data having different patterns in one walking cycle.

[0147] As shown in Figure 8, in case C9, the mean square error is 4.173[(deg) 2 ]. In case C10, the mean square error was 2.544[(deg) 2 In Cases C9 and C10, sensor data indicating the movement of the user's feet and sensor data indicating the movement of the user's thighs are used, respectively. The feet and thighs are body parts of the user that are highly relevant to walking. In Cases C9 and C10, sensor data indicating the movement of the feet or thighs, which are body parts highly relevant to walking, is used, so it is inferred that the posture angles of the body parts could be estimated with a certain degree of accuracy.

[0148] As shown in Figure 8, in case C11, the mean square error is 2.527[(deg) 2In case C11, only sensor data indicating the movement of the waist is used. The results of case C11 show that the posture angles of the user's body parts can be estimated with a certain degree of accuracy even with only sensor data indicating the movement of the user's waist. The reason for this is presumed to be that the vertical movement of the user while walking is reflected in the movement of the trunk, including the waist.

[0149] As shown in FIG. 8 , the mean squared errors of cases C12 and C13 were smaller than the mean squared error of case C11. That is, in cases C12 and C13, the estimation accuracy of posture angles of body parts was improved compared to case C11. As described above, in cases C12 and C13, in addition to sensor data indicating the movement of the user's waist, sensor data indicating the movement of at least one of the user's wrists and ankles is used. That is, in cases C12 and C13, in addition to sensor data indicating the movement of the user's trunk including the waist, sensor data indicating the movement of the user's limbs including at least one of the user's wrists and ankles is used. As described above, the sensor data indicating the movement of the user's limbs and the sensor data indicating the movement of the user's trunk have significantly different patterns in one gait cycle. It is inferred that in cases C12 and C13, the estimation accuracy of posture angles of body parts was improved compared to case C11 by using sensor data having different patterns in one gait cycle.

[0150] As shown in Figure 8, the mean square error for C06, C07, C08, and C13 is 1.0[(deg) 2 The estimation accuracy of the attitude angle in C06, C07, C08, and C13 was the highest among the cases C1 to C13.

[0151] Next, the results of comparing the measured and estimated values ​​of posture angles of the subject's body parts will be described. First, the subject used in the comparison will be described with reference to FIG.

[0152] An example of subjects is shown in Figure 9. The subjects have a variety of physical characteristics.

[0153] The subject SU1 is a male, 33 years old, 171 cm tall, and weighs 100 kg. The physical characteristics of the subject SU1 are that he is a heavy male.

[0154] The subject SU2 is a female, 70 years old, 151 cm tall, and weighs 39 kg. The physical characteristics of the subject SU2 are that she is a light-weight woman.

[0155] The subject SU3 is a female, 38 years old, 155 cm tall, and weighs 41 kg. The physical characteristics of the subject SU3 are that she is light in weight and young in age.

[0156] Subject SU4 is a female, 65 years old, 149 cm tall, and weighs 70 kg. Subject SU4 has physical characteristics of being a heavy female.

[0157] Subject SU5 is a male, 22 years old, 163 cm tall, and weighs 65 kg. Subject SU5's physical characteristics are that of an average male with average height and weight.

[0158] Subject SU6 is a female, 66 years old, 149 cm tall, and weighs 47 kg. Subject SU6's physical characteristics are that she is a short woman.

[0159] Subject SU7 is a female, 65 years old, 148 cm tall, and weighs 47 kg. Subject SU7's physical characteristics are that she is a short woman.

[0160] Subject SU8 is a male, 57 years old, 178 cm tall, and weighs 81 kg. Subject SU8's physical characteristics are that he is a tall man.

[0161] <Comparison result 1> As a comparison result 1, the actual measured values ​​and estimated values ​​of posture angles of different body parts of one subject SU6 when sensor data of case C6 is used will be described.

[0162] 10 to 21 show graphs of the measured and estimated values ​​of posture angles of body parts of subject SU6. The horizontal axis in Fig. 10 to Fig. 21 represents time [s]. The vertical axis in Fig. 10 to Fig. 21 represents posture angle [deg].

[0163] 10 to 15 are graphs of the posture angles of the upper body parts of subject SU6. Specifically, FIG. 10 is a graph of the posture angle of the neck of subject SU6. FIG. 11 is a graph of the posture angle of the chest of subject SU6. FIG. 12 is a graph of the posture angle of the right upper arm of subject SU6. FIG. 13 is a graph of the posture angle of the left upper arm of subject SU6. FIG. 14 is a graph of the posture angle of the right forearm of subject SU6. FIG. 15 is a graph of the posture angle of the left forearm of subject SU6.

[0164] 16 to 21 are graphs of the posture angles of the body parts of the lower body of subject SU6. Specifically, FIG. 16 is a graph of the posture angle of the right thigh of subject SU6. FIG. 17 is a graph of the posture angle of the left thigh of subject SU6. FIG. 18 is a graph of the posture angle of the right lower leg of subject SU6. FIG. 19 is a graph of the posture angle of the left lower leg of subject SU6. FIG. 20 is a graph of the posture angle of the right foot of subject SU6. FIG. 21 is a graph of the posture angle of the left foot of subject SU6.

[0165] In the following drawings, the attitude angle θXr is the measured value of the attitude angle θX described above, the attitude angle θYr is the measured value of the attitude angle θY described above, and the attitude angle θZr is the measured value of the attitude angle θZ described above.

[0166] In the following drawings, the attitude angle θXe is an estimated value of the attitude angle θX described above, the attitude angle θYe is an estimated value of the attitude angle θY described above, and the attitude angle θZe is an estimated value of the attitude angle θZ described above.

[0167] As shown in Figures 10 to 15, the estimated values ​​of posture angles of the upper body parts of subject SU6 were in relatively good agreement with the actual measurements. As described above, sensor data indicating the movement of the subject's left wrist was used as sensor data D10BL in case C6, as shown in Figure 7. In other words, in case C6, sensor data indicating the movement of the subject's right wrist was not used. Nevertheless, as shown in Figure 12, the estimated values ​​of posture angles of subject SU6's right upper arm were in agreement with the actual measurements with the same or similar accuracy as those of the left upper arm shown in Figure 13. Furthermore, as shown in Figure 14, the estimated values ​​of posture angles of subject SU6's right forearm were in agreement with the actual measurements with the same or similar accuracy as those of the left forearm shown in Figure 15.

[0168] As shown in Figures 16 to 21, the estimated values ​​of posture angles of the lower body parts of subject SU6 were in relatively good agreement with the actual measurements. As described above, in case C6, sensor data indicating the movement of the subject's right wrist was not used. Nevertheless, as shown in Figure 16, the estimated values ​​of posture angles of subject SU6's right thigh were consistent with the actual measurements with the same or similar accuracy as those of the left thigh shown in Figure 17. Furthermore, as shown in Figure 18, the estimated values ​​of posture angles of subject SU6's right lower leg were consistent with the actual measurements with the same or similar accuracy as those of the left lower leg shown in Figure 19. As shown in Figure 20, the estimated values ​​of posture angles of subject SU6's right foot were consistent with the actual measurements with the same or similar accuracy as those of the left foot shown in Figure 21.

[0169] <Comparison result 2> As comparison result 2, we will explain the actual measured values ​​and estimated values ​​of the posture angle of the right thigh for subjects with a high evaluation of center of gravity shift and subjects with a low evaluation of center of gravity shift when using case C6. A subject with a high evaluation of center of gravity shift means a subject whose center of gravity shifts significantly in the up-down direction while walking. A subject with a low evaluation of center of gravity shift means a subject whose center of gravity shifts significantly in the up-down direction while walking.

[0170] Fig. 22 is a graph of the posture angle of the right thigh of subject SU7, who received a high evaluation for center of gravity shift. Fig. 23 is a graph of the posture angle of the right thigh of subject SU1, who received a high evaluation for center of gravity shift. Fig. 24 is a graph of the posture angle of the right thigh of subject SU3, who received a high evaluation for center of gravity shift. Fig. 25 is a graph of the posture angle of the right thigh of subject SU6, who received a high evaluation for center of gravity shift.

[0171] Fig. 26 is a graph of the posture angle of the right thigh of subject SU5, who received a low evaluation for center of gravity shift. Fig. 27 is a graph of the posture angle of the right thigh of subject SU2, who received a low evaluation for center of gravity shift. Fig. 28 is a graph of the posture angle of the right thigh of subject SU4, who received a low evaluation for center of gravity shift. Fig. 29 is a graph of the posture angle of the right thigh of subject SU8, who received a low evaluation for center of gravity shift.

[0172] 22 to 29, the horizontal axis represents time [s], and the vertical axis represents the attitude angle [deg].

[0173] As mentioned above, in case C6, sensor data showing the movement of the subject's right wrist was not used. Nevertheless, as shown in Figures 22 to 25, for subjects who received high ratings for center of gravity movement, the estimated values ​​of the posture angle of the right thigh were in relatively good agreement with the actual measured values.

[0174] As mentioned above, in case C6, sensor data showing the movement of the subject's right wrist was not used. Nevertheless, as shown in Figures 26 to 29, for subjects with low evaluations of center of gravity movement, the estimated values ​​of the posture angle of the right thigh were in relatively good agreement with the actual measurements.

[0175] Subjects with low ratings of center of gravity shift have smaller vertical center of gravity shifts and movements than subjects with high ratings of center of gravity shift. Therefore, subjects with low ratings of center of gravity shift have less vertical movement during walking reflected in the sensor data than subjects with high ratings of center of gravity shift. Nevertheless, as shown in Figures 22 to 29, the estimated values ​​of the posture angle of the right thigh for subjects with low ratings of center of gravity shift matched the actual measured values ​​with the same or similar accuracy as subjects with high ratings of center of gravity shift. This indicates that by selecting Case C6, it is possible to estimate the posture angle of the right thigh with the same or similar accuracy as subjects with high ratings of center of gravity shift, even for subjects with low ratings of center of gravity shift.

[0176] <Comparison result 3> As a comparison result 3, the actual measured values ​​and estimated values ​​of the posture angle of the right upper arm for the subjects who received a high evaluation of the center of gravity shift and the subjects who received a low evaluation of the center of gravity shift when case C6 was used will be described.

[0177] Fig. 30 is a graph of the posture angle of the right upper arm of subject SU7, who received a high evaluation for center of gravity shift. Fig. 31 is a graph of the posture angle of the right upper arm of subject SU1, who received a high evaluation for center of gravity shift. Fig. 32 is a graph of the posture angle of the right upper arm of subject SU3, who received a high evaluation for center of gravity shift. Fig. 33 is a graph of the posture angle of the right upper arm of subject SU6, who received a high evaluation for center of gravity shift.

[0178] Fig. 34 is a graph of the posture angle of the right upper arm of subject SU5, who received a low evaluation for center of gravity shift. Fig. 35 is a graph of the posture angle of the right upper arm of subject SU2, who received a low evaluation for center of gravity shift. Fig. 36 is a graph of the posture angle of the right upper arm of subject SU4, who received a low evaluation for center of gravity shift. Fig. 37 is a graph of the posture angle of the right upper arm of subject SU8, who received a low evaluation for center of gravity shift.

[0179] The horizontal axis in Figures 30 to 37 represents time [s], and the vertical axis in Figures 30 to 37 represents the attitude angle [deg].

[0180] As mentioned above, in case C6, sensor data showing the movement of the subject's right wrist was not used. Nevertheless, as shown in Figures 30 to 37, the times when the maximum and minimum values ​​appeared in the estimated values ​​of the posture angle of the right upper arm were in relatively good agreement with the actual measurements.

[0181] As described above, subjects with a low evaluation of center of gravity shift have smaller vertical movements, so the vertical movements of the subjects while walking are less likely to be reflected in the sensor data than subjects with a high evaluation of center of gravity shift. Nevertheless, as shown in Figures 30 to 37, the estimated posture angle of the right upper arm for subjects with a low evaluation of center of gravity shift matched the actual measured value with the same or similar accuracy as subjects with a high evaluation of center of gravity shift. This shows that by selecting case C6, the posture angle of the right upper arm can be estimated with the same or similar accuracy as subjects with a high evaluation of center of gravity shift, even for subjects with a low evaluation of center of gravity shift.

[0182] [Evaluation decision process] The control unit 27 may determine an evaluation of the user's walking based on the estimated values ​​of the posture angles of the user's body parts. As an example, the control unit 27 may estimate the amount of movement of the user's center of gravity in the up and down direction based on the estimated values ​​of the posture angles of the user's body parts, and determine an evaluation of the movement of the user's center of gravity.

[0183] The control unit 27 may notify the user of the determined evaluation via the notification unit 23. For example, the control unit 27 may cause the display of the output unit 24 to display information about the determined evaluation, or may cause the speaker of the output unit 24 to output the information about the determined evaluation as sound. Alternatively, the control unit 27 may cause the vibration unit 25 to vibrate with a vibration pattern according to the determined evaluation.

[0184] The control unit 27 may generate an evaluation signal indicating the determined evaluation. The control unit 27 may transmit the generated evaluation signal to any external device via the communication unit 21. As an example, the control unit 27 may transmit the evaluation signal via the communication unit 21 to any sensor device 10 having a notification unit 13 as an external device. In this case, in the sensor device 10, the control unit 16 receives the evaluation signal via the communication unit 11. The control unit 16 causes the notification unit 13 to notify the information indicated by the evaluation signal. For example, the control unit 16 causes the output unit 14 to output the information indicated by the evaluation signal. With this configuration, the user can grasp the evaluation of their own walking.

[0185] For example, when the sensor device 10A is an earphone or is included in an earphone, the control unit 27 may transmit the evaluation signal to the earphone as an external device via the communication unit 21. In this case, when the control unit 16 of the sensor device 10A receives the evaluation signal via the communication unit 11, the control unit 16 may output information indicated by the evaluation signal as sound from the speaker of the output unit 14. With this configuration, the user can be notified of the evaluation information regarding their walking by sound. Notifying the user by sound reduces the possibility of interrupting the user's walking.

[0186] (System Operation) 38 is a flowchart showing the operation of the attitude angle estimation process executed by the electronic device 20 shown in FIG. This operation corresponds to an example of the information processing method according to the present embodiment. For example, when the user inputs an input instructing the execution of the attitude angle estimation process from the input unit 22, the control unit 27 starts the process of step S1.

[0187] The control unit 27 receives an input instructing execution of a posture angle estimation process via the input unit 22 (step S1).

[0188] The control unit 27 transmits a signal instructing the start of data detection as a broadcast signal to the plurality of sensor devices 10 via the communication unit 21 (step S2). After the process of step S2 is executed, sensor data is transmitted from at least one sensor device 10 to the electronic device 20.

[0189] The control unit 27 receives sensor data from at least one sensor device 10 via the communication unit 21 (step S3).

[0190] The control unit 27 selects one of the cases C1 to C13 depending on the type of the sensor device 10 that transmitted the sensor data to the electronic device 20 (step S4). The control unit 27 acquires data of the transformer 30 used in the cases C1 to C13 selected in the processing of step S4 from the storage unit 26 (step S5).

[0191] Control unit 27 inputs the sensor data of cases C1 to C13 selected in the process of step S4 to the transformer that acquired the data in the process of step S5. Control unit 27 inputs the sensor data to the transformer, and acquires, from the transformer, time series data of estimated values ​​of posture angles of body parts across the user's entire body and time series data of the movement speed of the user's waist (step S6).

[0192] The control unit 27 generates a walking model from the time series data of the estimated values ​​of posture angles of body parts throughout the user's entire body and the time series data of the moving speed of the user's waist, which are acquired in the process of step S6 (step S7). The control unit 27 causes the output unit 24 to output the data of the walking model generated in the process of step S7 (step S8).

[0193] After executing the process of step S8, the control unit 27 ends the estimation process. After ending the estimation process, the control unit 27 may execute the estimation process again. When executing the estimation process again, the control unit 27 may start from the process of step S3. The control unit 27 may repeatedly execute the estimation process until an input instructing the end of the estimation process is received from the input unit 22. The input instructing the end of the estimation process is, for example, input from the input unit 22 by the user. For example, when the user finishes walking, the user inputs an input instructing the end of the estimation process from the input unit 22. When the control unit 27 receives the input instructing the end of the estimation process, the control unit 27 may transmit a signal instructing the end of data detection as a broadcast signal to the multiple sensor devices 10 via the communication unit 21. In the sensor device 10, when the control unit 16 receives the signal instructing the end of data detection via the communication unit 11, the control unit 16 may end the data detection.

[0194] In this way, in electronic device 20 serving as an information processing device, control unit 27 obtains estimated values ​​of posture angles of the user's body parts using sensor data and a learning model. Obtaining estimated values ​​of posture angles of the user's body parts makes it possible to detect the user's movements. Furthermore, from the results shown in FIGS. 8 to 37, it can be seen that it is possible to obtain an estimated value of posture angle of one of the user's left and right body parts using sensor data indicating the movement of the other body part and a learning model.

[0195] As a comparative example, consider a case where a camera captures an image of a user walking, analyzes the captured data, and detects the user's movements. In this case, it is necessary to install a camera. Installing a camera is time-consuming. Furthermore, the camera can only capture an image of the user walking at the location where it is installed. Furthermore, depending on the clothing worn by the user, the user's body movements may be hidden from the outside by the clothing. If the user's body movements are hidden from the outside, the camera may not be able to capture the user's body movements.

[0196] In contrast to such comparative examples, in this embodiment, it is not necessary to install a camera, and therefore it is possible to detect the user's movements more easily. Furthermore, wherever the user walks, as long as the user wears the sensor device 10, it is possible to detect sensor data, and therefore it is possible to detect the user's movements. Furthermore, even if the user's body movements are not visible from the outside due to clothing, as long as the user wears the sensor device 10, it is possible to detect sensor data, and therefore it is possible to detect the user's movements. Furthermore, as long as the user wears the sensor device 10, it is possible to detect the user's movements, regardless of where the user walks and when the user walks.

[0197] Therefore, according to this embodiment, an improved technique for detecting the user's movement can be provided.

[0198] Furthermore, in this embodiment, the transformer may be trained to output an estimated value of the posture angle of a user's body part when sensor data from the case C1 is input. The sensor data from the case C1 is detected by the sensor device 10A. With this configuration, even when the user wears only the sensor device 10A, an estimated value of the user's body part can be obtained. Furthermore, since the user only needs to wear the sensor device 10A, user convenience can be improved. Furthermore, if the sensor device 10A is an earphone or is included in an earphone, the user can easily wear the sensor device 10A on their head. Since the user can easily wear the sensor device 10A on their head, user convenience can be further improved. Furthermore, by using only the sensor data detected by the sensor device 10A, it is not necessary to synchronize the timing at which each of the multiple sensor devices 10 detects data. Since it is not necessary to synchronize the timing at which each of the multiple sensor devices 10 detects data, it is possible to more easily obtain an estimated value of the posture angle.

[0199] In this embodiment, the transformer may be trained to output an estimated value of the posture angle of a body part of a user when sensor data of any of cases C2 to C5 is input. The sensor data of case C2 is detected by the sensor device 10A and the sensor device 10E-1 or the sensor device 10E-2, i.e., by two sensor devices 10. The sensor data of case C3 is detected by the sensor device 10A and the sensor device 10F-1 or the sensor device 10F-2, i.e., by two sensor devices 10. The sensor data of case C4 is detected by the sensor device 10A and the sensor device 10D-1 or the sensor device 10D-2, i.e., by two sensor devices 10. The sensor data of case C5 is detected by the sensor device 10A and the sensor device 10B, i.e., by two sensor devices 10. In this way, in cases C2 to C5, sensor data is detected by two sensor devices 10, so the user only needs to wear two sensor devices 10. This improves user convenience. Also, as described above with reference to FIG. 8, the accuracy of estimating posture angles of the user's body parts is improved in cases C2 to C5 compared to case C1. Therefore, by using the sensor data in cases C2 to C5, the posture angles of the user's body parts can be estimated with high accuracy.

[0200] In addition, in this embodiment, the transformer may be trained to output an estimated value of the posture angle of a user's body part when sensor data of either Case C6 or C7 is input. The sensor data of Case C6 is detected by the sensor device 10A, the sensor device 10B, and the sensor device 10E-1 or the sensor device 10E-2, i.e., three sensor devices 10. The sensor data of Case C7 is detected by the sensor device 10A, the sensor device 10B, and the sensor device 10F-1 or the sensor device 10F-2, i.e., three sensor devices 10. In this way, in Cases C6 and C7, the sensor data is detected by three sensor devices 10, so the user only needs to wear three sensor devices 10. This improves user convenience. Furthermore, as described above with reference to FIG. 8, the accuracy of estimating the posture angle of the user's body part is improved in Cases C6 and C7 compared to Case C1. Therefore, by using the sensor data in cases C6 and C7, the posture angles of the user's body parts can be estimated with high accuracy.

[0201] (Other system configurations) FIG. 39 is a functional block diagram showing a configuration of an information processing system 101 according to another embodiment of the present disclosure.

[0202] The information processing system 101 includes a sensor device 10, an electronic device 20, and a server 80. In the information processing system 101, the server 80 functions as an information processing device, and acquires estimated values ​​of posture angles of body parts of a user using sensor data detected by the sensor device 10 and a learning model.

[0203] The electronic device 20 and the server 80 can communicate with each other via a network 2. The network 2 may be any network including a mobile communication network, the Internet, and the like.

[0204] The control unit 27 of the electronic device 20 receives sensor data from the sensor device 10 via the communication unit 21 in the same manner as or similar to the information processing system 1. In the information processing system 101, the control unit 27 transmits the sensor data to the server 80 via the network 2 via the communication unit 21.

[0205] The server 80 is, for example, a server that belongs to a cloud computing system or another computing system. The server 80 includes a communication unit 81, a storage unit 82, and a control unit 83.

[0206] The communication unit 81 is configured to include at least one communication module connectable to the network 2. The communication module is, for example, a communication module compatible with standards such as a wired LAN (Local Area Network) or a wireless LAN. The communication unit 81 is connected to the network 2 via the wired LAN or wireless LAN by the communication module.

[0207] The storage unit 82 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM or a ROM. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The storage unit 82 may function as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 82 stores data used in the operation of the server 80 and data obtained by the operation of the server 80. For example, the storage unit 82 stores system programs, application programs, embedded software, and the like. For example, the storage unit 82 stores data of the transformer 30 shown in FIG. 4 and data used by the transformer 30, and the like.

[0208] The control unit 83 is configured to include at least one processor, at least one dedicated circuit, or a combination of these. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. The dedicated circuit is, for example, an FPGA or an ASIC. The control unit 83 controls each unit of the server 80 and executes processes related to the operation of the server 80. The control unit 83 may execute processes executed by the transformer 30 shown in FIG. 4.

[0209] The control unit 83 receives sensor data from the electronic device 20 via the network 2 using the communication unit 81. The control unit 83 executes processing that is the same as or similar to the processing performed by the control unit 27 of the electronic device 20 described above, thereby obtaining estimated values ​​of posture angles of the user's body parts using the sensor data and a learning model.

[0210] (Other system behavior) Fig. 40 is a sequence diagram showing the operation of the estimation process executed by the information processing system 101 shown in Fig. 39. This operation corresponds to an example of the information processing method according to the present embodiment. For example, when a user inputs an input instructing the execution of the posture angle estimation process from the input unit 22 of the electronic device 20, the information processing system 101 starts the process of step S1.

[0211] In the electronic device 20, the control unit 27 receives an input instructing execution of a posture angle estimation process via the input unit 22 (step S11). The control unit 27 transmits a signal instructing start of data detection as a broadcast signal to the plurality of sensor devices 10 via the communication unit 21 (step S12).

[0212] In the sensor device 10, the control unit 16 receives a signal from the electronic device 20 via the communication unit 11 instructing the start of data detection (step S13). Upon receiving this signal, the control unit 16 starts data detection. The control unit 16 acquires data detected by the sensor unit 12 from the sensor unit 12. The control unit 16 transmits the acquired data as sensor data to the electronic device 20 via the communication unit 11 (step S14).

[0213] In the electronic device 20, the control unit 27 receives the sensor data from the sensor device 10 via the communication unit 21 (step S15). The control unit 27 transmits the sensor data to the server 80 via the network 2 via the communication unit 21 (step S16).

[0214] In the server 80, the control unit 83 receives sensor data from the electronic device 20 via the network 2 through the communication unit 81 (step S17). The control unit 83 selects one of cases C1 to C13 depending on the type of sensor device 10 that transmitted the sensor data to the server 80 through the electronic device 20 (step S18). The control unit 83 acquires data of the transformer 30 used in the cases C1 to C13 selected in the processing of step S18 from the storage unit 82 (step S19). The control unit 83 inputs the sensor data of the cases C1 to C13 selected in the processing of step S18 to the transformer that acquired the data in the processing of step S19. The control unit 83 inputs the sensor data to the transformer and acquires, from the transformer, time series data of estimated values ​​of posture angles of body parts across the user's entire body and time series data of the movement speed of the user's waist (step S20). The control unit 83 generates a walking model from the time series data of the estimated values ​​of posture angles of body parts throughout the user's body and the time series data of the movement speed of the user's waist, which are acquired in the process of step S20 (step S21). The control unit 83 transmits the data of the walking model generated in the process of step S21 to the electronic device 20 via the network 2 by the communication unit 81 (step S22).

[0215] In the electronic device 20, the control unit 27 receives the gait model data from the server 80 via the network 2 through the communication unit 21 (step S23). The control unit 27 causes the output unit 24 to output the received gait model data (step S24).

[0216] After executing the processing of step S24, the information processing system 101 ends the estimation processing. After finishing the estimation processing, the information processing system 101 may execute the estimation processing again. When executing the estimation processing again, the information processing system 101 may start from the processing of step S14. The information processing system 101 may repeatedly execute the estimation processing until the electronic device 20 receives an input from the input unit 22 instructing the end of the estimation processing. As described above, when the electronic device 20 receives an input instructing the end of the estimation processing, the electronic device 20 may transmit a signal instructing the end of data detection as a broadcast signal to the multiple sensor devices 10. As described above, when the sensor device 10 receives the signal instructing the end of data detection, the sensor device 10 may end data detection.

[0217] The information processing system 101 can achieve the same or similar effects as the information processing system 1.

[0218] While the present disclosure has been described based on various drawings and examples, it should be noted that those skilled in the art would easily be able to make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present disclosure. For example, the functions contained in each functional unit can be rearranged so as not to cause logical inconsistencies. Multiple functional units can be combined into one or separated. The above-described embodiments of the present disclosure are not limited to faithful implementation of each of the described embodiments, but can be implemented by combining features or omitting some features as appropriate. In other words, those skilled in the art can make various modifications and alterations based on the present disclosure. Therefore, these modifications and alterations are within the scope of the present disclosure. For example, in each embodiment, each functional unit, means, step, etc. can be added to other embodiments so as not to cause logical inconsistencies, or can be replaced with each functional unit, means, step, etc. of other embodiments. Furthermore, in each embodiment, multiple functional units, means, steps, etc. can be combined into one or separated. Furthermore, each of the above-described embodiments of the present disclosure is not limited to being implemented faithfully according to each of the described embodiments, but can also be implemented by combining each feature or omitting some of them as appropriate.

[0219] For example, the electronic device 20 or the server 80 may include a filter that can be applied to the data output from the learning model, such as a Butterworth filter.

[0220] For example, in the above-described embodiment, the periodic motion is described as walking. Furthermore, the learning model is described as having been trained to output estimated values ​​of posture angles of body parts of a user while walking. However, the periodic motion is not limited to walking. The information processing system of the present disclosure can obtain estimated values ​​of posture angles of body parts of a user while performing any periodic motion. In other words, the learning model can be trained to output estimated values ​​of posture angles of body parts while the user is performing any periodic motion.

[0221] For example, in the above-described embodiment, the information processing system 1, 101 is described as estimating the posture angle of a user who walks as exercise in daily life. However, the use of the information processing system of the present disclosure is not limited to this.

[0222] As another example of application, the information processing system of the present disclosure may be used to allow other customers at an event venue to view how a customer walks. In this case, in the information processing system 1 shown in FIG. 3 , the control unit 27 of the electronic device 20 may transmit data of the generated gait model to a projection device at the event venue as an external device via the network 2 or short-range wireless communication using the communication unit 21. In the information processing system 101 shown in FIG. 39 , the control unit 83 of the server 80 may transmit data of the generated gait model to a projection device at the event venue as an external device via the network 2 using the communication unit 21. With this configuration, the projection device at the event venue can project a gait model showing how a customer walks onto a screen or the like.

[0223] As yet another example of an application, the information processing system of the present disclosure may be used to generate images of characters walking using the generated walking model in movies, games, etc. By attaching the sensor device 10 to various users, various walking models can be generated.

[0224] For example, the communication unit 11 of the sensor device 10 may further include at least one communication module connectable to a network 2 as shown in Fig. 39. The communication module is, for example, a communication module compatible with a mobile communication standard such as LTE, 4G, or 5G. In this case, in the information processing system 101 as shown in Fig. 39, the control unit 16 of the sensor device 10 may directly transmit data detected by the sensor device 10 to a server 80 via the network 2 using the communication unit 11.

[0225] For example, in the above-described embodiment, cases C5 to C8, C12, and C13 are described as including sensor data indicating the movement of the user's wrist. However, in cases C5 to C8, C12, and C13, sensor data indicating the movement of a part of the user's forearm other than the wrist may be used instead of the sensor data indicating the movement of the user's wrist.

[0226] For example, in the above-described embodiment, the sensor device 10 has been described as including a communication unit 11 as shown in FIGS. 3 and 39. However, the sensor device 10 does not have to include a communication unit 11. In this case, sensor data detected by the sensor device 10 may be transferred to a device such as the electronic device 20 or server 80 that estimates the attitude angle via a storage medium such as an SD (Secure Digital) memory card. An SD memory card is also called an "SD card." The sensor device 10 may be configured so that a storage medium such as an SD memory card can be inserted therein.

[0227] For example, an embodiment is also possible in which a general-purpose computer functions as the electronic device 20 according to the present embodiment. Specifically, a program describing the processing content for realizing each function of the electronic device 20 according to the present embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the configuration according to the present embodiment can also be realized as a program executable by a processor or a non-transitory computer-readable medium storing the program. [Explanation of symbols]

[0228] 1,101 Information Processing Systems 2 Network 10, 10A, 10B, 10C, 10D, 10D-1, 10D-2, 10E, 10E-1, 10E-2, 10F, 10F-1, 10F-2 Sensor equipment 11 Communications Department 12 Sensor section 13. Information Department 14 Output section 15 Storage section 16 Control Unit 20 Electronic equipment (information processing equipment) 21 Communications Department 22 Input section 23 Information Department 24 Output section 25 Vibration unit 26 Memory section 27 Control Unit 30 Transformers 40 Encoder 41, 42, 44, 45, 46, 47 Functional section 43 Layers 50 decoders 51, 52, 54, 55, 56, 57, 58, 59, 60, 61, 70 Functional section 53 Layers 80 Server (information processing device) 81 Communications Department 82 Memory section 83 Control Unit C1,C2,C3,C4,C5,C6,C7,C8,C9,C10,C11,C12,C13 case D10AG, D10AL, D10BL, D10CG, D10CL, D10DL-1, D10DL-2, D10EL-1, D10EL-2, D10FL-1, D10FL-2 sensor data

Claims

1. a control unit that acquires an estimated value of a posture angle of at least one of a plurality of body parts of the user based on sensor data indicating a movement of at least a part of the body part of the user and a learning model; the plurality of body parts of the user include at least a first body part that is the head of the user and a second body part; The information processing device, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when at least the sensor data indicating the movement of the first body part is input.

2. The plurality of body parts of the user includes a third body part which is one of the right arm and the left arm of the user; the second body part is the other of the right arm and the left arm of the user, 2. The information processing device according to claim 1, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when sensor data indicating the movement of the first body part and sensor data indicating the movement of the third body part are input.

3. The plurality of body parts of the user includes a third body part which is one of the right foot and the left foot of the user, the second body part is the other of the right foot and the left foot of the user, 2. The information processing device according to claim 1, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when sensor data indicating the movement of the first body part and sensor data indicating the movement of the third body part are input.

4. The plurality of body parts of the user includes a third body part which is one of the right thigh and the left thigh of the user; the second body part is the other of the right thigh and the left thigh of the user, 2. The information processing device according to claim 1, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when sensor data indicating the movement of the first body part and sensor data indicating the movement of the third body part are input.

5. The information processing device according to claim 1 , wherein the learning model is a Transformer.

6. an information processing device that acquires an estimated value of a posture angle of at least one of a plurality of body parts of a user using sensor data indicating a movement of at least a part of the body part of the user and a learning model; the plurality of body parts of the user include at least a first body part that is the head of the user and a second body part; An information processing system, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when at least the sensor data indicating the movement of the first body part is input.

7. obtaining an estimated value of a posture angle of at least one of a plurality of body parts of the user using sensor data indicating a movement of at least a part of the body part of the user and a learning model; the plurality of body parts of the user include at least a first body part that is the head of the user and a second body part; An information processing method, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when at least the sensor data indicating the movement of the first body part is input.

8. On the computer, obtaining an estimated value of a posture angle of at least one of a plurality of body parts of the user using sensor data indicating a movement of at least a part of the body part of the user and a learning model; the plurality of body parts of the user include at least a first body part that is the head of the user and a second body part; The learning model is a program that has been trained to output an estimated value of the posture angle of the second body part when at least the sensor data indicating the movement of the first body part is input.

9. A control unit that obtains an estimate of a posture angle of at least one of a plurality of body parts of a user based on sensor data indicating the movement of at least a part of the body part of the user and a learning model, the plurality of body parts of the user include at least a first body part that is a waist region of the user and a second body part; The information processing device, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when at least the sensor data indicating the movement of the first body part is input.

10. The plurality of body parts of the user includes a third body part which is one of the right arm part and the left arm part of the user, the second body part is the other of the right arm and the left arm of the user, 10. The information processing device according to claim 9, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when sensor data indicating the movement of the first body part and sensor data indicating the movement of the third body part are input.

11. The plurality of body parts of the user includes a third body part which is one of the right foot and the left foot of the user, the second body part is the other of the right foot and the left foot of the user, 10. The information processing device according to claim 9, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when sensor data indicating the movement of the first body part and sensor data indicating the movement of the third body part are input.

12. An information processing device described in any one of claims 9 to 11, wherein the learning model is a transformer.

13. An information processing device that obtains an estimate of a posture angle of at least one of a plurality of body parts of a user by using sensor data indicating movement of at least a part of the body part of the user and a learning model, the plurality of body parts of the user include at least a first body part that is a waist region of the user and a second body part; An information processing system, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when at least the sensor data indicating the movement of the first body part is input.

14. The method includes obtaining an estimate of a posture angle of at least one of a plurality of body parts of the user using sensor data indicating the movement of at least a part of the body part of the user and a learning model; the plurality of body parts of the user include at least a first body part that is a waist region of the user and a second body part; An information processing method, wherein the learning model is trained to output an estimated value of the posture angle of the second body part when at least the sensor data indicating the movement of the first body part is input.

15. A computer comprising: obtaining an estimated value of a posture angle of at least one of a plurality of body parts of the user using sensor data indicating a movement of at least a part of the body part of the user and a learning model; the plurality of body parts of the user include at least a first body part that is a waist region of the user and a second body part; The learning model is a program that has been trained to output an estimated value of the posture angle of the second body part when at least the sensor data indicating the movement of the first body part is input.

Images