Program, method, and information processing device

Abstract

To provide another technology for a user himself / herself to easily check his or her body motion.SOLUTION: There is provided a program for causing a computer to be operated. The program causes a processor of the computer to execute the steps of: sequentially sensing a body motion of a user; presenting to the user an avatar corresponding to the user in a mode to cause the shape of the body surface of the user to be perceived instead of an actually taken image when the user's external appearance is photographed, on the basis of a result of the sensing; specifying a position for drawing a graphic object as an indication of the body motion of the user in association with the avatar on the basis of the result of the sensing; and displaying the graphic object at the specified position.SELECTED DRAWING: Figure 37

Description

[Technical field]

[0001] The present disclosure relates to a program, a method, and an information processing device. [Background technology]

[0002] There is known a technique for capturing a user's movements using a camera or the like and evaluating the user's movements based on the captured video images.

[0003] JP 2016-81504 A describes technology relating to a device that uses a depth camera to recognize athletic forms based on the human skeleton, and uses this to automatically evaluate a user's movements and determine whether or not they will be promoted. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-81504 A Summary of the Invention [Problem to be solved by the invention]

[0005] There is a need for alternative technology that can enable users to easily monitor their own physical movements. [Means for solving the problem]

[0006] According to one embodiment, a program for operating a computer is provided, the program causes a processor of the computer to execute the steps of: sequentially sensing a user's body movement, presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive the shape of the user's body surface instead of a real image of the user's appearance based on the sensing result, specifying a position for drawing a graphical object serving as a guide for the user's body movement in association with the avatar based on the sensing result, and displaying the graphical object at the specified position. Effect of the Invention

[0007] According to the present disclosure, the user can recognize the avatar presented to the user based on the sensing results of the user's body movements, while also recognizing graphical objects that serve as an indication of the user's body movements, allowing the user to easily check their own body movements. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the overall configuration of a measurement system 1. [Diagram 2] 1 is a block diagram of a terminal device 10 constituting a measurement system 1 according to a first embodiment. [Diagram 3] FIG. 2 is a diagram showing the functional configuration of a server 20. [Figure 4] 2 shows the data structures of a user information database 281, measurement item information 282, a measurement history database 283, and a friend list 284 stored in the server 20. FIG. [Diagram 5] 1 shows a situation in which a user measures the movement of his or her body using the measurement system 1. [Figure 6] FIG. 2 is a diagram showing details of a mat 32. [Figure 7] 13 is a flowchart showing a process in which the terminal device 10 performs a ROM check and updates the measurement history. [Figure 8]13 is a flowchart showing a process in which the terminal device 10 evaluates a range of motion of a joint by measuring a stepping motion of a user. [Figure 9] 13 is a flowchart showing a process in which the terminal device 10 evaluates a range of motion of a joint by measuring a user's twisting motion of the upper body. [Figure 10] FIG. 11 is a diagram showing an example of the notification content of the terminal device 10 in a situation where the user's body movement is measured. [Figure 11] 13 shows a situation in which a user's walking motion is measured by the measurement system 1 in the second embodiment. [Figure 12] Based on the measurement results of the measuring device 31, the terminal device 10 identifies each part of the user's body and indicates it. [Figure 13] FIG. 13 is a diagram showing the data structures of a measurement result database 287 and an evaluation result database 288 stored in the server 20 in the second embodiment. [Figure 14] 13 is a flowchart showing a process in which the terminal device 10 measures and evaluates the walking movement of a user in the second embodiment. [Figure 15] 13A to 13C are diagrams illustrating an example of an output when the terminal device 10 outputs a measurement result of measuring a user's walking motion and an evaluation result of evaluating the user's walking motion based on the measurement result in the second embodiment. [Figure 16] 13A to 13C are diagrams illustrating an example of an output when the terminal device 10 outputs a measurement result of measuring a user's walking motion and an evaluation result of evaluating the user's walking motion based on the measurement result in the second embodiment. [Figure 17] 14A and 14B are diagrams illustrating an example in which the terminal device 10 outputs an evaluation result of the strain on the knees during the walking motion of the user, which corresponds to step S1409 in FIG. [Figure 18] 13 is a flowchart showing a process in which the terminal device 10 measures and evaluates the balance of the user's trunk in the third embodiment. [Figure 19] 13A to 13C are diagrams illustrating an example in which the terminal device 10 outputs the measurement results of measuring the balance of the user's trunk and the evaluation results of the trunk balance in the third embodiment. [Figure 20] FIG. 13 is a diagram showing data structures of various data stored in a server 20 in the third embodiment. [Figure 21] 13 is a screen example in which the terminal device 10 presents possible physical concerns of the user based on the measurement results and evaluation results of the user's physical movement, and suggests an exercise menu to address the concerns. [Figure 22] 13 shows the flow of processing in each device in the fourth embodiment. [Figure 23] FIG. 13 is a diagram showing data structures of various data stored in the server 20 in a modified example of the fourth embodiment. [Figure 24] FIG. 13 is a diagram showing an operational flow in a modified example of the fourth embodiment. [Diagram 25] This is an example screen of a situation where the user answers a questionnaire in which the terminal device 10 presents possible physical concerns of the user based on the measurement and evaluation results of the user's physical movements and suggests an exercise menu to address the concerns. [Figure 26] FIG. 26 is a diagram showing the data structures of a trainee information database 2601, a trainer information database 2602, and a training history database 2603, which are stored by the server 20 in the fifth embodiment. [Figure 27] 11 is a flowchart showing a series of processes for sensing the body movement of a first user as point cloud data and transmitting information based on the point cloud data of the first user obtained by sensing to a terminal of a second user. [Figure 28] 13 is a flowchart showing a process flow for accepting voice input of a trainer from the trainer's terminal and transmitting it to the trainee's terminal. [Figure 29] This is a flowchart showing a series of processes for sensing the body movements of a first user as point cloud data, and identifying instruction content for the first user based on information based on the point cloud data of the first user obtained by sensing and a trained model. [Diagram 30]11 is an example of a screen displayed when a terminal device 10A of a first user senses the body movements of the first user as point cloud data, and transmits information based on the point cloud data of the first user obtained by sensing to a terminal of a second user. [Diagram 31] This is an example of the screen of a first user and a second user, including the overall configuration, when the body movements of a first user are sensed as point cloud data, and information based on the point cloud data of the first user obtained by sensing is transmitted to a terminal of a second user. [Diagram 32] This is an example screen when the body movements of multiple users are sensed as point cloud data, and information based on the point cloud data of each user obtained by sensing is transmitted to a defined virtual space. [Diagram 33] FIG. 13 is a diagram showing a concept of a process in which the terminal device 10A draws auxiliary lines based on the estimation result of the user's posture. [Diagram 34] An example of a screen presented to the user is shown. [Diagram 35] FIG. 13 is a diagram showing an example of a screen that allows the user to adjust the tilt in the forward / backward and left / right directions while in a standing position on the terminal device 10A. [Diagram 36] 13 is a diagram showing a process of specifying a position for drawing a graphical object that serves as a guide for the user's body movement, based on a result of estimating the position of a specific part of the user. FIG. [Figure 37] FIG. 13 is a diagram showing an example of a screen displayed by the terminal device 10A for prompting the user to perform a lateral lifting motion. [Figure 38] 13 is a diagram showing a process of specifying a position for drawing a graphical object that serves as a guide for the user's body movement, based on a result of estimating the position of a specific part of the user. FIG. [Figure 39] FIG. 1 is a diagram showing a process in which a user is moving his / her body. [Diagram 40] FIG. 13 is a diagram showing a state in which the user has fully raised both arms, which is the peak timing of the lateral lifting motion. [Diagram 41] FIG. 13 is a diagram showing an example in which a user saves sensing results at the peak timing of a lateral lift movement and displays them as a comparable avatar. [Diagram 42] FIG. 13 is a diagram showing an example of a screen displayed by the terminal device 10A to prompt the user to perform a crouching motion. [Diagram 43] 13 is a diagram showing a process of specifying a position for drawing a graphical object that serves as a guide for the user's body movement, based on a result of estimating the position of a specific part of the user. FIG. [Diagram 44] FIG. 1 is a diagram showing a process in which a user is moving his / her body. [Diagram 45] FIG. 13 is a diagram showing a state in which the user has fully crouched, which is the peak timing of the crouching motion. [Figure 46] FIG. 13 is a diagram showing an example in which sensing results at the peak timing of a user's crouching movement are saved and displayed as comparable avatars. [Figure 47] FIG. 13 is a diagram showing an example of a screen displayed on the terminal device 10A to prompt the user to perform a motion of stepping forward and lowering the hips. [Figure 48] 13A and 13B are diagrams illustrating an example of a screen in which a user is notified that the user's initial posture is appropriate. [Figure 49] FIG. 2 illustrates a situation in which measurement of a user's movement is started. [Figure 50] FIG. 13 is a diagram showing a situation in which it is determined whether a user's action is appropriate or not, and the determination result is presented to the user. [Figure 51] FIG. 13 is a diagram showing a situation in which it is determined whether a user's action is appropriate or not, and the determination result is presented to the user. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

[0010] <First embodiment> <Summary> In the following embodiment, a measurement system 1 is described that evaluates the flexibility of a user's body by measuring the balance and movement of the user's body, and provides an exercise menu to enable the user to acquire beautiful posture (e.g., beautiful walking posture).

[0011] If the user's walking posture is not good, the user will perform movements that are not necessarily required for walking, which will consume the user's strength unnecessarily and lead to the accumulation of fatigue. By improving the user's walking posture, the user can move the body functionally, which will result in a walking style that causes less fatigue. Therefore, the measurement system 1 (1) measures the user's walking movements and evaluates the walking posture (evaluates whether the user can walk beautifully and in a balanced manner), (2) measures the flexibility of each of the user's joints and performs a ROM check to evaluate the range of motion of the joints (ROM: Range of motion), and (3) evaluates the user's muscle strength and provides a training menu.

[0012] When a user walks in a well-balanced manner (no bias in the left and right movements of the body, ideal muscle movements during walking, etc.), the body movement is functional. Here, functional movement means that the user can move the muscles to walk without consuming unnecessary force. In other words, the user moves the muscles to obtain propulsive force for walking, and the muscle movement is highly efficient. As a result, the user can give the impression of walking beautifully. In order to enable the user to learn such a well-balanced walking style, it is effective to evaluate the range of motion of the joints, and the measurement system 1 provides the user with muscle training necessary to make the walking posture look beautiful, based on the evaluation results of the range of motion of the joints, in addition to evaluating the range of motion of the joints.

[0013] As described above, the measurement system 1 measures (checks) the user's physical movement and provides exercises for moving the user's body.

[0014] The measurement system 1 may be installed, for example, in the offices of a business company, in the facilities of a commercial facility, in a store space, in a studio, in an individual's home, etc. For example, a business company may introduce the measurement system 1 to further encourage the health management of employees. Also, by deploying the measurement system 1 in a store of a commercial facility, health management services may be provided to consumers.

[0015] <1 Overall system configuration> FIG. 1 is a diagram showing the overall configuration of a measurement system 1. As shown in FIG.

[0016] As shown in FIG. 1, the measurement system 1 is composed of multiple measurement units (in the illustrated example, measurement units 30A, 30B, and 30C are shown. Hereinafter, each measurement unit may be referred to as "measurement unit 30") composed of various sensors that measure the user's body movements, a terminal device 10 that controls the operation of the measurement units (in the illustrated example, terminal device 10 is shown. Hereinafter, when describing terminal devices of multiple users, they may be described as "terminal device 10A", "terminal device 10B", etc. Also, each element (such as a processor) that constitutes the terminal device may be described in a similar manner), and a server 20 that accumulates the measurement results of the user's body movements.

[0017] The measurement unit 30 is connected to the terminal device 10 by wire or wirelessly. As shown in the figure, a plurality of measurement units 30 may be connected to one terminal device 10, or a terminal device 10 may be connected to each measurement unit 30 (i.e., one terminal device 10 is prepared for one measurement unit 30 to measure the user).

[0018] The measuring unit 30 includes a measuring device (in the illustrated example, measuring devices 31A, 31B, and 31C are shown. These may be collectively referred to as "measuring device 31"), a mat (in the illustrated example, mats 32A, 32B, and 32C are shown. These may be collectively referred to as "mat 32"), and a pole which is a three-dimensional object (in the illustrated example, poles 33A, 33B, and 33C are shown. These may be collectively referred to as "pole 33").

[0019] The measuring device 31 is for measuring the movement of the user's body, and detects the distance and direction to a measurement target by, for example, an RGB-D camera (depth sensor). The measuring device 31 may also include a microphone array and detect the user's voice and the direction from which the voice comes. For example, the measuring device 31 may be a device capable of receiving a user's gesture and an operation by voice recognition, such as Kinect (registered trademark). The measuring device 31 detects the user's body to detect the position of the user and the positions of each joint of the user's body. As a result, the measuring device 31 sets virtual bones that virtually connect each joint constituting the user, and measures the movement of the user's body. In addition to Kinect (registered trademark), the measuring device 31 may also be a terminal such as a smartphone. For example, if a smartphone is provided with a depth camera, the smartphone can acquire the user's body as point cloud data.

[0020] Furthermore, even if the measuring device 31 does not have a function for measuring distance, it may acquire the distance to the subject (for example, the distance to each joint of the subject) based on the captured image by using a trained model that identifies the subject based on the captured image and estimates the distance between the subject and the camera. Even if the measuring device 31 cannot measure the distance (such as an RGB camera), it may identify each part of the subject based on the captured image and estimate the skeleton (bones). Furthermore, the measuring device 31 may be provided with multiple cameras and estimate the distance to the subject by obtaining captured images with parallax.

[0021] The mat 32 is for displaying a guide for the user's standing position in order to measure the user's body movements. The mat 32 is placed at a position that has a predetermined positional relationship with the measuring device 31. In other words, the mat 32 is placed within a range where the measuring device 31 can measure the user's body movements. As described later, the mat 32 includes a plurality of patterns for guiding the user's standing position. The user adjusts the position of his / her feet or takes a step forward while viewing the patterns.

[0022] The pole 33 is used together with the mat 32 to measure the movement of the user's body, for example, the movement of the user twisting the body. The pole 33 is placed on the mat 32 at a position associated with a pattern indicating the standing position of the user's feet. The user twists the body while keeping both feet on the mat 32 (i.e., without moving the standing positions of both feet), and moves to touch a pair of poles placed on both sides of the mat 32 with their hands. Thus, the mat 32 is used as a target point to move the arms of the user to perform the movement of twisting the body. The mat 32 may be set to a height according to the height of the user so that the user can hold it with his / her hands when twisting the body.

[0023] As described above, the combination of the measuring device 31, mat 32, and pole 33 that constitute the measuring unit 30 has been described. In the following description, it is assumed that one user uses one measuring unit 30. However, one measuring device 31 may measure the body movements of multiple users. In other words, multiple users may move their bodies within the measurement range of one measuring device 31.

[0024] The terminal device 10 and the server 20 are connected for communication via a network 80. In this embodiment, each device (terminal device, server, etc.) can also be regarded as an information processing device. That is, a collection of each device can be regarded as one "information processing device", and the measurement system 1 may be formed as a collection of multiple devices. The method of allocating multiple functions required to realize the measurement system 1 according to this embodiment to one or multiple pieces of hardware can be appropriately determined in consideration of the processing capacity of each piece of hardware and / or the specifications required for the measurement system 1.

[0025] The terminal device 10 is a device operated by each user. The terminal device 10 may be, for example, a desktop PC (Personal Computer) or a laptop PC. Alternatively, the terminal device 10 may be realized by a mobile terminal such as a smartphone or tablet compatible with a mobile communication system. As shown as the terminal device 10B in FIG. 1, the terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage unit 16, and a processor 19. The server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29.

[0026] The terminal device 10 is communicatively connected to the server 20 via a network 80. The terminal device 10 supports communication standards such as 5G and LTE (Long Term Evolution) and wireless LAN (Local Area Network) standards such as IEEE (Institute of Electrical and Electronics Engineers) 802.11, and is connected to the network 80 by communicating with a communication device.

[0027] The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 can communicate with an external device. The input device 13 is an input device (e.g., a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.) for receiving an input operation from a user. The output device 14 is an output device (a display, a speaker, etc.) for presenting information to a user. The memory 15 is for temporarily storing programs and data processed by the programs, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage unit 16 is a storage device for saving data, and is, for example, a flash memory or a HDD (Hard Disc Drive). The processor 19 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, etc.

[0028] As shown in the figure, the terminal device 10 is connected by wire or wirelessly to a measurement unit 30. The measurement unit 30 measures the movement of the user's body, making it possible to evaluate the flexibility of the user's body and the degree of joint motion range.

[0029] The server 20 manages information about each user. The server 20 manages, as the user information, parameters (for example, height) that are referenced when measuring the physical movement of each user.

[0030] The server 20 performs processes for encouraging interaction between users, specifically, matching of users to support multiple users in exercising and measuring their physical movements for health management. The server 20 also transmits and receives messages between users.

[0031] The server 20 may manage virtual points owned by each user. For example, assume that a business company has an in-house point system that each employee can use like a virtual currency. The points can be used for in-house purchasing (for example, can be used for payment at vending machines in the company) or can be given to other employees like a virtual currency. For example, the server 20 may give the points to a user when the user performs a measurement using the measurement system 1, or when the user performs a measurement using the measurement system 1 together with another user (for example, performing a measurement at the same place, performing a measurement at the same place at the same time, etc.), or when the evaluation result of the range of motion of the joint exceeds a predetermined value.

[0032] The communication IF 22 is an interface for inputting and outputting signals so that the server 20 can communicate with an external device. The input / output IF 23 functions as an interface with an input device for receiving input operations from a user and an output device for presenting information to a user. The memory 25 is for temporarily storing programs and data processed by the programs, etc., and is a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for saving data, such as a flash memory or a HDD (Hard Disc Drive). The processor 29 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, etc.

[0033] <1.1 Configuration of the terminal device 10> FIG. 2 is a block diagram of the terminal device 10 constituting the measurement system 1 of the first embodiment. As shown in FIG. 2, the terminal device 10 includes a plurality of antennas (antenna 111, antenna 112), wireless communication units (first wireless communication unit 121, second wireless communication unit 122) corresponding to the respective antennas, an operation reception unit 130 (including a touch-sensitive device 131. In addition, it may include various pointing devices such as a mouse and a touchpad, and may include a keyboard and a gesture detection device), a display 132, a voice processing unit 140, a microphone 141, a speaker 142, a position information sensor 150, a camera 160, a storage unit 180, and a control unit 190. The terminal device 10 also has functions and configurations (for example, a battery for holding power, a power supply circuit for controlling the supply of power from the battery to each circuit, etc.) that are not particularly shown in FIG. 2. As shown in FIG. 2, each block included in the terminal device 10 is electrically connected by a bus or the like.

[0034] The antenna 111 emits a signal generated by the terminal device 10 as a radio wave. The antenna 111 also receives a radio wave from space and provides the received signal to the first wireless communication unit 121.

[0035] The antenna 112 radiates a signal generated by the terminal device 10 as a radio wave. The antenna 112 also receives the radio wave from space and provides the received signal to the second radio communication unit 122.

[0036] The first wireless communication unit 121 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 111 so that the terminal device 10 can communicate with other wireless devices. The second wireless communication unit 122 performs modulation and demodulation processing for transmitting and receiving signals via the antenna 112 so that the terminal device 10 can communicate with other wireless devices. The first wireless communication unit 121 and the second wireless communication unit 122 are communication modules including a tuner, a Received Signal Strength Indicator (RSSI) calculation circuit, a Cyclic Redundancy Check (CRC) calculation circuit, a high-frequency circuit, and the like. The first wireless communication unit 121 and the second wireless communication unit 122 perform modulation and demodulation and frequency conversion of wireless signals transmitted and received by the terminal device 10, and provide the received signals to the control unit 190.

[0037] The operation reception unit 130 has a mechanism for receiving an input operation of a user. Specifically, the operation reception unit 130 includes a touch-sensitive device 131 as a pointing device. The touch-sensitive device 131 receives an input operation of a user of the terminal device 10. The touch-sensitive device 131 detects a touch position of the user on the touch panel by using, for example, a capacitive touch panel. The touch-sensitive device 131 outputs a signal indicating the touch position of the user detected by the touch panel to the control unit 190 as an input operation. The terminal device 10 may receive an input operation of the user by communicating with the measurement device 31 and receiving a measurement result of the measurement device 31. For example, when the measurement device 31 acquires point cloud data, the measurement device 31 detects the user's body and each joint constituting the user's body based on the point cloud data, and receives a gesture of the user as an input based on the detection result. Gestures that can be input by the user include body movements such as the user's hand movement and head movement. For example, the operation receiving unit 130 may detect when the user takes a predetermined pose (raising both hands horizontally to take a "T" pose), or may detect when the user makes a predetermined shape with their hands.

[0038] The display 132 displays data such as images, videos, and texts in response to control by the control unit 190. The display 132 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. The terminal device 10 and the output device (display 132, speaker 142) may be realized as separate devices.

[0039] The audio processing unit 140 modulates and demodulates an audio signal. The audio processing unit 140 modulates a signal provided from the microphone 141 and provides the modulated signal to the control unit 190. The audio processing unit 140 also provides the audio signal to the speaker 142. The audio processing unit 140 is realized by, for example, a processor for audio processing. The microphone 141 accepts audio input and provides an audio signal corresponding to the audio input to the audio processing unit 140. The speaker 142 converts the audio signal provided from the audio processing unit 140 into audio and outputs the audio to the outside of the terminal device 10.

[0040] The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals from at least three or four satellites are received, and the current position of the terminal device 10 equipped with the GPS module is detected based on the received signals. For example, in the measurement system 1, when the location where the user performed the measurement is recorded, the detection result of the position information sensor 150 (latitude and longitude information, information on the facility specified by the latitude and longitude and map information, information on the business associated with the facility, etc.) may be associated with the measurement result and stored in the server 20. In other words, the terminal device 10 may determine the location where the user performed the measurement based on the detection result of the position information sensor 150 without the user inputting the location.

[0041] The camera 160 is a device for receiving light by a light receiving element and outputting it as a captured image. The camera 160 is, for example, a depth camera that can detect the distance from the camera 160 to a subject to be captured. In this embodiment, the measuring device 31 may be realized by the camera 160 of the terminal device 10. That is, in addition to realizing the terminal device 10 and the measuring device 31 as separate devices, the terminal device 10 may also be equipped with the function of the measuring device 31.

[0042] The storage unit 180 is configured, for example, with a flash memory or the like, and stores data and programs used by the terminal device 10. In one aspect, the storage unit 180 stores user information 181, measurement item information 182, measurement history information 183, a friend list 184, and training item information 185. These will be described in detail later, but an overview is as follows. The terminal device 10 may acquire this information from various databases of the server 20, which will be described later.

[0043] User information 181 is information about a user in the measurement system 1. The user information includes information for identifying the user, the user's name, the user's affiliation, and the user's physical information (height, weight, age, etc.).

[0044] The measurement item information 182 is information on each measurement item when a ROM check is performed. The measurement item information 182 includes information on audio guidance to be output to the user for each measurement item.

[0045] The measurement history information 183 indicates a history of the measurement results of measuring the user's body movements.

[0046] The friend list 184 shows information about each user who is registered as a friend by the user. Users who are registered as friends may be able to send and receive messages between each other.

[0047] The training item information 185 includes information on each training item when a muscle training menu is provided to the user. For example, it includes the name of the training and video and audio information that guides the user to move his / her body. The terminal device 10 uses the measuring device 31 to measure how the user is moving his / her body based on the video and audio that guides the user, and sequentially displays the measurement results on the display 132. The terminal device 10 may also compare the ideal way to move the body with the measurement results (movement of each joint) when the user is moving his / her body on the display 132, to encourage the user to move his / her body in an ideal way.

[0048] The control unit 190 reads a program stored in the storage unit 180 and executes instructions included in the program to control the operation of the terminal device 10. The control unit 190 is, for example, an application processor. The control unit 190 operates according to the program to fulfill the functions of an input operation reception unit 191, a transmission / reception unit 192, a data processing unit 193, and a notification control unit 194.

[0049] The input operation reception unit 191 performs a process of receiving a user's input operation on an input device such as the touch-sensitive device 131. The input operation reception unit 191 determines the type of operation, such as whether the user's operation is a flick operation, a tap operation, or a drag (swipe) operation, based on information on the coordinates where the user touches the touch-sensitive device 131 with a finger or the like.

[0050] The transmission / reception unit 192 performs processing for the terminal device 10 to transmit and receive data to and from external devices such as the server 20 and the measurement unit 30 in accordance with a communication protocol.

[0051] The data processing unit 193 performs a process of performing calculations on the data that the terminal device 10 accepts as input in accordance with a program, and outputs the calculation results to a memory or the like.

[0052] The notification control unit 194 performs a process of displaying a display image on the display 132, a process of outputting audio to the speaker 142, etc. For example, the notification control unit 194 performs a process of outputting audio guidance during a ROM check, and presents the user with audio guidance and an image showing exemplary body movements when providing a training menu for exercise.

[0053] <1.2 Functional configuration of server 20> 3 is a diagram showing a functional configuration of the server 20. As shown in FIG. 3, the server 20 fulfills the functions of a communication unit 201, a storage unit 202, and a control unit 203.

[0054] The communication unit 201 performs processing for the server 20 to communicate with external devices such as the terminal device 10 .

[0055] The storage unit 202 stores data and programs used by the server 20. The storage unit 202 stores a user information database 281, measurement item information 282, a measurement history database 283, a friend list 284, and the like.

[0056] The user information database 281 indicates information about each user who performs measurement using the measurement system 1. Details will be described later.

[0057] The measurement item information 282 indicates information on each of a plurality of measurement items, as will be described in detail later.

[0058] The measurement history database 283 shows the history of measurements taken by each user, as will be described in detail later.

[0059] The friend list 284 is a database for holding information about users who are registered as friends by each user.

[0060] Training item information 285 includes information on each training item when a training menu for exercise is provided to the user. The training menu includes information on multiple training items according to the load on the user's body, the part of the body to be trained, etc.

[0061] The control unit 203 exerts functions shown as various modules by the processor of the server 20 performing processes according to the programs.

[0062] The operation content acquisition module 2041 acquires the operation content of the user. For example, the operation content acquisition module 2041 accepts an operation by the user to acquire past measurement results by the measurement unit 30 and evaluation results of the user's body movement, an operation to send and receive messages with other users, an operation to apply to participate in a measurement when there is an opportunity to be measured by the measurement unit 30, and the like.

[0063] The reception control module 2042 controls the process in which the server 20 receives a signal from an external device in accordance with a communication protocol.

[0064] The transmission control module 2043 controls the process in which the server 20 transmits a signal to an external device in accordance with a communication protocol.

[0065] The matching module 2044 matches users with each other. For example, the matching module 2044 may detect that multiple users have performed measurements using the measurement system 1 at the same measurement venue, match the multiple users, and encourage them to register as friends. The matching module 2044 may extract and match users who have a similar history of performing measurements using the measurement system 1, or extract and match users who have a similar result of a ROM check using the measurement unit 30. By matching multiple users in this way and encouraging them to register as friends, it is possible to encourage multiple users to train together to achieve a beautiful posture, and therefore it is possible to support the measurement system 1 in continuing the measurement.

[0066] The measurement progress module 2045 progresses the process of measuring the user's body movements by the measurement unit 30. The measurement progress module 2045 reads out each measurement item indicated in the measurement item information 282, and sequentially outputs audio guidance or the like for measurements according to the measurement items to the user. For example, the server 20 may transmit each measurement item to the terminal device 10, and sequentially measure the user's body movements according to each measurement item.

[0067] The training progress module 2046 performs a process of providing the user with a training menu by video and audio. The training progress module 2046 reads out each training item shown in the training item information 285, and sequentially outputs audio guidance, video guidance, and the like for training according to the training items to the user.

[0068] <2 Data Structure> FIG. 4 is a diagram showing the data structures of user information database 281, measurement item information 282, measurement history database 283, and friend list 284 stored in server 20. As shown in FIG.

[0069] As shown in FIG. 4, each record in the user information database 281 includes an item "user identification information (user ID)", an item "user name", an item "gender", an item "height", an item "weight", an item "affiliation", an item "evaluation value", an item "company points", etc.

[0070] The item "user identification information (user ID)" is information for identifying each user.

[0071] The item "user name" is the name set by the user.

[0072] The item "gender" indicates the gender of the user. For example, the gender may be set when the user attempts to start using the measurement system 1 and registers user information.

[0073] The item "height" indicates the height of the user as information to be referenced when measuring the user's physical movements.

[0074] The item "weight" indicates information about the weight of the user. In cases where weight information is not referenced in measuring the user's physical movements, data for the item "weight" may not be stored.

[0075] The item "Affiliation" indicates information about the user's affiliation, such as when the user belongs to an organization. For example, the item "Affiliation" includes information such as the company name, organization name, and business establishment name. The item "Affiliation" may also include information about the location of the office where the user works.

[0076] The item "evaluation value" includes information on an evaluation value evaluated based on the measurement result of a ROM check performed by the user in the measurement system 1 and an evaluation value based on the measurement result of measuring the user's posture while walking. The evaluation value includes, for example, information on the classification result when the terminal device 10 classifies the result of the ROM check of the user based on the measurement result by the measurement unit 30.

[0077] The item "internal points" indicates the amount of virtual points held by the user that can be used in the business company to which the user belongs, if any.

[0078] Each record of measurement item information 282 includes the items "measurement item number (measurement item No.)", "measurement part", "guidance voice", "foot position on mat", and "body part to be measured".

[0079] The item "measurement item number" is information for identifying each measurement item.

[0080] The item "measurement site" indicates the body site to be measured. For example, the following sites are used to evaluate the range of motion of the joints:

[0081] (i) Measurement of the upper limbs Scapula: Movement can be flexed, extended, raised, or lowered. Shoulder: Directions of movement include flexion, extension, abduction, adduction, external rotation, and internal rotation. Elbow: Movement can be in the direction of flexion or extension. Forearm: Direction of movement is pronation and supination. Hand: Directions of movement include flexion, extension, flexion, and ulnar flexion.

[0082] (ii) Hand and finger measurements Thumb: Directions of movement include flexible abduction, ulnar adduction, volar abduction, volar adduction, flexion, and extension. Fingers: Directions of movement include flexion, extension, abduction, and adduction.

[0083] (iii) Measurement of the lower limbs Hip: Directions of movement include flexion, extension, abduction, adduction, external rotation, and internal rotation. Knee: Movement can be in the direction of flexion or extension. · Feet: Directions of movement include flexion and extension. · Foot: Directions of movement include eversion, inversion, abduction, and adduction. Thumb: Movement can be in the direction of flexion or extension. Toes: Movement can be in the direction of flexion or extension.

[0084] (iv) Measurement of the trunk · Neck: Directions of movement include flexion, extension, rotation (left rotation, right rotation), and lateral bending (left side bending, right side bending). Thoracic and lumbar region: Directions of movement include flexion, extension, rotation (left rotation, right rotation), and lateral bending (left side bending, right side bending).

[0085] The item "Guide voice" indicates the contents of the voice notifying the user when performing the measurement indicated in the measurement item. The voice guide includes voice when the user does not place his / her feet in the specified position on the mat 32 (e.g., "Place both feet in the specified position on the mat," etc.), voice when the terminal device 10 detects that the user has placed his / her feet in the specified position on the mat 32 for measurement and prompts the user to move his / her body (e.g., "Twist your body while keeping both feet in the specified position on the mat," etc.), and voice when measurement data of the user's body movement is obtained based on the measurement result of the measuring device 31 (e.g., "Measurement was successful," etc.).

[0086] The item "Feet position on mat" includes information on which position on the mat 32 the user places both feet when performing the measurement (the position on the mat 32 where the user places the feet). For example, the terminal device 10 detects whether or not the user's both feet are placed in the specified positions on the mat based on the information on the item "Feet position on mat". Upon this detection, the terminal device 10 outputs a sound to the user indicating the content of the body movement for the measurement.

[0087] The item "body part to be measured" indicates the part of the user's body that is the subject of measurement in the measurement item.

[0088] Each record in the measurement history database 283 includes an item "measurement number (measurement No.)", an item "date and time", an item "user identification information (ID)", an item "measurement location", an item "measurement data", and an item "evaluation result". The server 20 receives the measurement results of the user's body movements from the terminal device 10. The server 20 updates the records in the measurement history database 283 based on the information received from the terminal device 10.

[0089] The item “measurement number (measurement No.)” indicates each measurement result when the user measures the body movement using the measurement unit 30.

[0090] The item "Date and time" indicates the timing when the user performed the measurement using the measurement unit 30.

[0091] The item “user identification information (ID)” indicates information for identifying the user whose body movement is measured by the measurement unit 30.

[0092] The item "measurement location" indicates the location where the user performed the measurement using the measurement unit 30. When the server 20 receives a measurement result from the terminal device 10, the server 20 may refer to the user information associated with the measurement result and identify the location where the user performed the measurement based on the information associated with the user (user information database 281). For example, based on the information of the user's affiliation shown in the user information database 281, the information of the location where the affiliation is located may be set as the location where the measurement was performed by the measurement unit 30. In addition, the terminal device 10 may acquire location information using the location information sensor 150, and the location of the terminal device 10 may be set as the location where the measurement was performed by the measurement unit 30.

[0093] The item "measurement data" indicates the measurement results of the user's body movements measured by the measurement unit 30. For example, when the user's body movements are measured as time-series point cloud data, the time-series point cloud data may be the measurement results in the item "measurement data". In addition, for the user's body measured by the measurement device 31, the terminal device 10 may identify the positions of the user's joints based on the measurement results, and the movements of each joint may be stored in the server 20 as the measurement results of the user's body movements. For example, the terminal device 10 can measure the time-series movements of the user's joints by detecting specific parts of the body, such as shoulders and elbows, based on the captured images of the user's body.

[0094] The item "Evaluation result" indicates an evaluation result of the level of the user's body movement based on the measurement result shown in the item "Measurement data". For example, when the terminal device 10 holds time-series data of the user's joint movements in the item "Measurement data", it evaluates the user's body movement by comparing it with reference body movement data. For example, the terminal device 10 holds multiple levels of data according to the degree of goodness of the user's body movement as reference data, and can identify the reference data of the movement that is most similar to the measurement result of the user's body movement.

[0095] Each record of the friend list 284 includes an item "User 1", an item "User 2", an item "friend registration date", an item "same timing measurement history", an item "message sending / receiving history", and the like.

[0096] The item "user 1" indicates one of the users who are registered as friends of each other.

[0097] The item "user 2" indicates the other user among the users who are registered as friends of each other.

[0098] The item "friend registration date" indicates the date and time when the users were registered as friends.

[0099] The item "same-timing measurement history" indicates a history of measurements taken by each user registered as a friend while satisfying at least one of the following conditions: the same location or the same timing. For example, when the measurement unit 30 is placed in an office space of a business establishment, multiple users may take measurements during a certain period of time, so that the multiple users may take measurements at the same timing. The item "same-timing measurement history" may include information on the location where multiple users took measurements. For example, the server 20 can refer to the friend list 284 to identify the frequency with which multiple users measure their body movements using the measurement unit 30 and train using a training menu.

[0100] The item "Message transmission / reception history" shows the history of messages sent and received between users registered as friends.

[0101] <3. Configuration of the measuring unit 30> Fig. 5 shows a situation in which a user measures the body movement by the measurement system 1. Fig. 5 shows an example in which a plurality of measurement units 30 are installed and a plurality of users perform measurements. Also, an example in which one terminal device 10 is connected to a plurality of measurement units 30 is shown.

[0102] The measuring device 31A measures the body movement of the first user User_A in a sensing detection range 38A. In the detection range 38A, a mat 32A is installed at a position that has a predetermined positional relationship with the measuring device 31A. Here, the position of the mat 32A may be, for example, a position at a certain distance (for example, about several meters) from the front of the measuring device 31A. The mat 32A may be installed relative to the measuring device 31A so that the measuring device 31A can measure the whole body movement of the user when the first user User_A stands on the mat 32A. Poles 33A are installed on the right and left sides of the mat 32A. For example, when the mat 32A has a pattern indicating a predetermined position (which may be called a home position) for the user to place both feet, a pair of poles 33A may be arranged on the outside of the mat 32A based on the pattern. The pole 33A is installed as a guide for the user to grasp when the user twists the upper limbs on the mat 32A with both feet in the specified positions. This encourages the user to move the hand toward the pole 33A when twisting the upper limbs, so that the form of the upper limb movement can be stabilized to some extent even when the number of measurement opportunities is increased. In other words, it is possible to suppress the variation in the time-series measurement results when measuring the user's body movement.

[0103] The measuring device 31B measures the body movement of the second user User_B in a sensing-enabled detection range 38B. In the detection range 38B, a mat 32B is installed at a position that has a predetermined positional relationship with the measuring device 31B. Here, the position of the mat 32B may be, for example, a position at a certain distance (about several meters) from the front of the measuring device 31B. The mat 32B may be installed relative to the measuring device 31B so that when the second user User_B stands on the mat 32B, the measuring device 31B can measure the user's entire body movement. Poles 33B are installed on the right and left sides of the mat 32B, respectively.

[0104] The first user User_A visually checks the image and sound displayed on the display 132A, checks the position of his / her feet using the pattern on the mat 32A as a guide, and moves his / her body. Similarly, the second user User_B moves his / her body following the guide by the image and sound displayed on the display 132B. In this way, the body movements of the first user User_A and the second user User_B can be measured by the measuring devices 31A and 31B, respectively.

[0105] In the illustrated example, a first user User_A is about to start bending and stretching exercises after placing both feet on the home position on the mat 32A based on the output from the display 132A. A second user User_B is stepping out with his right foot based on the output from the display 132B.

[0106] 6 is a diagram showing details of the mat 32. As shown in the figure, patterns serving as guides for placing the user's right and left feet are arranged in multiple places.

[0107] For example, multiple positions for placing the right foot are shown as guide positions for the user to place both feet across multiple measurement items, such as a first home position 41R, a second home position 42R, and a third home position 43R for the right foot.

[0108] For the left foot, a plurality of positions for placing the left foot are shown, such as a first home position 41L, a second home position 42L, and a third home position 43L.

[0109] In the following description, home positions 41R and 41L may be collectively referred to as home position 41, etc. Similarly, home positions 42R and 42L may be collectively referred to as home position 42, and home positions 43R and 43L may be collectively referred to as home position 43.

[0110] For each measurement item (measurement item information 282), the user specifies the distance between the right and left feet, and the terminal device 10 guides the user to perform a predetermined body movement. For example, information on the location of the home position to be used may be provided in the audio guidance accompanying the measurement of the measurement item.

[0111] The terminal device 10 determines whether the user is using a predetermined home position when measuring a measurement item based on the measurement result of the measuring device 31. For example, when the terminal device 10 identifies each pattern of the home position shown on the mat 32 based on the sensing result of the measuring device 31, it may be possible to specify the pattern on which both feet of the user are placed based on the detection result of the user's ankles, etc. When the user places their feet in a position different from the expected home position when measuring a measurement item, the user may be notified by audio guide, video guide, etc., to prompt them to place their feet in an appropriate position.

[0112] Furthermore, the terminal device 10 includes, as a measurement item, measurement of the movement of each joint when the user steps out with the measurement unit 30. The terminal device 10 can evaluate, based on the measurement items, the balance of the user's body when stepping out, and the like.

[0113] When the measurement item includes a stepping out motion, the mat 32 includes a pattern group 45R that serves as a target for stepping out with the right foot, and similarly, a pattern group 45L that serves as a target for stepping out with the left foot.

[0114] These pattern groups 45R and 45L are sometimes collectively referred to as "pattern group 45."

[0115] The pattern group 45 has a number of patterns arranged as guides for stepping positions according to the height of the user. In the illustrated example, the patterns are arranged in 10 cm increments in the range of the user's height from "140 cm" to "200 cm". The patterns are shaped to resemble the toes of the feet, so that the user can easily recognize that they are the target positions for stepping out. The user places both feet at the home positions 41R and 41L, or the patterns 44R and 44L (which may be collectively referred to as "pattern 44") as initial positions, and steps out one foot at a time to the pattern position that is the guide according to the user's height. The terminal device 10 measures the user's body movement (joint movement) when performing the stepping out movement by the measurement unit 30. When measuring the user's stepping out movement, the terminal device 10 may specify the home positions 41 to 43 or the pattern 44 as the initial position of the foot according to the distance the user is desired to step out, and guide the user.

[0116] In the illustrated example, the position of the pole 33 is also shown. For example, the pole 33 (right side and left side) may be arranged so as to be aligned in a straight line with the home positions 41, 42, and 43. For example, the user places both feet on one of the home positions 41, 42, and 43 and twists his / her upper limbs to try to grab the pole 33.

[0117] As shown in the figure, the pattern groups 45R and 45L may have arrow patterns. In the example shown in the figure, the arrow patterns are arranged overlapping with the patterns corresponding to the heights of the pattern groups 45R and 45L. This can make the user aware of the action of stepping forward with his / her foot, and can reduce the possibility of the user being confused about how to move his / her body during measurement when looking at the mat 32.

[0118] As shown in the figure, the size of the pattern is intended as a guide for the user to place their foot on, so in the illustrated example, the length in the vertical direction (the direction in which the foot steps out) may be approximately the same as the size of a normal foot (approximately 22 cm to 28 cm in length). The shape of the pattern may be a footprint or a shape similar to a footprint, so that the user can easily understand that they will place their foot on the pattern.

[0119] <4 Actions> The measurement process in which the terminal device 10 measures the body movement of the user will be described below.

[0120] FIG. 7 is a flowchart showing a process in which the terminal device 10 performs a ROM check and updates the measurement history.

[0121] In step S700, the control unit 190 of the terminal device 10 displays a screen for the user to log in on the display 132 and accepts the operation for logging in from the user. The terminal device 10 authenticates the user and acquires information about the user, for example, by reading information from a recording medium such as an IC card carried by each user, or by reading a two-dimensional code displayed on a communication terminal carried by the user. For example, the terminal device 10 communicates with the server 20 to acquire information about the user, such as the user's height (user information database 281), past measurement history (measurement history database 283), and information about other users who are registered as friends of the user (friend list 284). The terminal device 10 also acquires information about measurement items (measurement item information 282) from the server 20 and stores it as measurement item information 182.

[0122] In step S701, the terminal device 10 detects that the mat 32 and the pole 33 are installed within the measurement range of the measuring device 31 based on the measurement results of the measuring device 31. For example, the terminal device 10 detects the mat 32 and the pole 33 by performing image recognition on the measurement results of the measuring device 31 as a two-dimensional image. The terminal device 10 detects that the mat 32 and the pole 33 are in predetermined positions relative to the measuring device 31. In other words, the terminal device 10 detects that the mat 32 and the pole 33 are in positions suitable for measurement and are in positions where the user can easily confirm the audio and video of the guide.

[0123] In step S703, the terminal device 10 refers to the measurement item information 182 and performs a process of measuring each measurement item in sequence. The terminal device 10 reads out the first measurement item and outputs information serving as a guide for the measurement (such as the position where the user places his / her feet on the mat 32) by audio and video. The terminal device 10 detects the user's joints based on the measurement results of the measuring device 31, and detects that both of the user's feet are placed in predetermined positions on the mat (positions defined by the measurement items).

[0124] In step S705, the terminal device 10 outputs a voice prompting the user to perform a predetermined body movement. The terminal device 10 measures the user's body movement by detecting the user's body movement with the measurement device 31.

[0125] In step S707, the terminal device 10 sequentially reads out the measurement items from the measurement item information 182, outputs audio guidance, detects that the user's feet are in the specified positions on the mat 32, and measures the user's body movements.

[0126] In step S709, when the terminal device 10 completes the measurement for each measurement item, it evaluates the range of motion of the user's joints based on the measurement results of the user's body movement. Specifically, the terminal device 10 identifies the position of each of the user's joints based on the measurement results of the measurement device 31, and evaluates the range of motion of the body part to be measured based on the time-series data of the joint positions (the movement of the joint positions that changes over time). The terminal device 10 outputs the evaluation results of the range of motion of the joints on the display 132 or the like, and transmits them to the server 20.

[0127] In step S753, the server 20 receives the result of the ROM check of the user from the terminal device 10, and updates the databases (the measurement history database 283 and the user information database 281) based on the measurement result of the user.

[0128] Next, the process of each device when measuring the user's stepping out motion as a measurement item will be described.

[0129] FIG. 8 is a flowchart showing a process in which the terminal device 10 evaluates the range of motion of a joint by measuring the user's stepping motion.

[0130] In step S801, the terminal device 10 detects that both feet of the user are placed in the prescribed positions on the mat 32 (the item "Feet Position on Mat" in the measurement item information 282) based on the measurement result of the measurement device 31. If both feet of the user are not placed in the prescribed positions, the terminal device 10 may guide the user to place the feet in the prescribed positions.

[0131] In step S803, the terminal device 10 outputs audio and video to the user to guide the user to step out with his / her right foot from a specified position to a stepping position according to the user's height.

[0132] In step S805, the terminal device 10 detects that the user's right foot has stepped from a specified position on the mat to another position, based on the measurement result by the measurement device 31, and records the measurement result of the user's stepping out with the right foot. When the user's right foot steps out to a position different from the target position to be stepped out, the terminal device 10 may guide the user to step out with the right foot to the target position again in order to perform the measurement again.

[0133] In step S807, the terminal device 10 outputs a voice guiding the user to step out with his / her left foot from a specified position to a stepping position according to the user's height (stride length).

[0134] In step S809, based on the measurement results of the measuring device 31, the terminal device 10 detects that the user's left foot has stepped from a specified position on the mat to another position, and records the measurement results of the user's stepping out with his / her left foot.

[0135] In step S811, the terminal device 10 evaluates the range of motion of the joints of the lower limbs based on the measurement results of the user's stepping out movement on the mat 32, outputs the evaluation result to the display 132 or the like, and transmits it to the server 20.

[0136] Next, the processing of each device when measuring the user's torso twisting motion will be described.

[0137] FIG. 9 is a flowchart showing a process in which the terminal device 10 evaluates the range of motion of a joint by measuring a user's twisting motion of the upper body.

[0138] In step S901, the terminal device 10 detects, based on the measurement result of the measuring device 31, that both feet of the user are placed in the specified positions on the mat.

[0139] In step S903, the terminal device 10 determines whether the height of the pole 33 is appropriate based on the information on the height of the user. For example, the terminal device 10 stores information on the height of the pole 33 that is appropriate according to the height of the user in association with the height of the user, thereby determining whether the height of the pole 33 detected by the measurement result of the measuring device 31 is appropriate. For example, the appropriate height of the pole 33 may be determined based on the position of the user's hands when the user stands upright. Here, the position of the hands when the user stands upright may vary depending on the height of the user. In other words, it is assumed that the higher the height, the higher the position of the hands when standing upright. In addition, the terminal device 10 may detect the position of the user's hands (height from the ground) when standing upright based on the measurement result of the measuring device 31, and determine the appropriate height of the pole 33 based on the position of the hands. In other words, the height of the pole 33 may be adjusted according to the measurement result of the user's body by the measuring device 31.

[0140] The height of the pole 33 may be determined, for example, based on the measurement results of the measuring device 31, by identifying a first position where the pole 33 contacts the ground and a second position which is the top of the pole 33, and based on information on the distance from the position of the measuring device 31 to the first position and the second position of the pole 33. In addition, if the pole 33 has a structure that allows it to be extended, the pattern on the pole 33 may include one indicating the height of the pole 33, and the terminal device 10 may detect the pattern indicating the height of the pole 33.

[0141] The terminal device 10 may determine whether or not the position at which the pole 33 is installed is appropriate. For example, the terminal device 10 may detect a pattern indicating a home position or the like on the pole 33, and determine whether or not the position of the pole 33 is appropriate based on the positional relationship between the position of the pattern and the position at which the pole 33 is in contact with the ground.

[0142] In step S905, the terminal device 10 outputs audio and video guiding the user to twist his / her body to the right while keeping both feet in the specified positions on the mat, and to grasp the right pole (pole 33 placed on the right side) with the left hand and the left pole (pole 33 placed on the left side) with the right hand.

[0143] In step S907, the terminal device 10 records the measurement result of the user twisting his / her body to the right based on the measurement result of the measurement device 31.

[0144] In step S909, the terminal device 10 outputs a voice guiding the user to twist his / her body to the left while keeping both feet on the mat at the specified positions, and to grasp the left pole with his / her right hand and the right pole with his / her left hand.

[0145] In step S911, the terminal device 10 records the measurement result of the user twisting his / her body to the left based on the measurement result.

[0146] In step S913, the terminal device 10 evaluates the range of motion of each joint of the body based on the measurement result of the user's twisting motion, and outputs the evaluation result. Whether or not the user can grasp the pole 33 by twisting the upper body may also be used as an evaluation item for the evaluation of the range of motion of the joints.

[0147] As described above, by setting the height of the pole 33 to be approximately the same as the height of the user's hands when standing upright, when the user twists his / her upper body, the user turns around and tries to move his / her hands and arms toward the visible pole 33. This makes it possible to reduce the amount of shaking when the user twists his / her upper body, and makes it easier to compare the time series data of the measurement results when the terminal device 10 accumulates the measurement results.

[0148] <5 Screen example> FIG. 10 is a diagram showing an example of the notification content of the terminal device 10 in a situation where the user's body movement is measured.

[0149] The example screen in Fig. 10(A) shows a situation where the terminal device 10 accepts a login operation from a user to perform a measurement. This corresponds to the processing of step S700 in Fig. 7. For example, when the measurement unit 30 is installed in a business office, a store space, etc. of a business company and multiple users use it, the users are first authenticated.

[0150] As shown in FIG. 10(A), the terminal device 10 displays the notification section 132L on the display 132. The terminal device 10 displays the operation content for the user to log in on the notification section 132L. In the illustrated example, the measurement unit 30 is installed in the business establishment, and the notification section 132L indicates that the user can log in by reading the employee ID card of the user. The terminal device 10 displays in the notification section 132C that another user is going to perform a measurement at the same time. As in the example of FIG. 5, when multiple measurement units 30 are installed and multiple users perform a measurement together, the terminal device 10 displays in the notification section 132C the progress of the other user performing the measurement. In the example of FIG. 10(A), the terminal device 10 displays in the notification section 132C that another user performing a measurement at the same time has already completed the login process.

[0151] 10(B) shows a screen example in which the user's stepping motion is measured on the terminal device 10. This corresponds to the processing of steps S801 to S811 in FIG.

[0152] As shown in FIG. 10(B), the terminal device 10 displays on the display 132 a notification section 132D, a notification section 132E, a guidance output section 132F, and a detection result display section 132G.

[0153] The terminal device 10 displays information about the logged-in user (i.e., the user performing the measurement) in the notification unit 132D. This allows the user to easily check whether or not the user is performing the measurement by mistake as another user.

[0154] The terminal device 10 displays the history of measurements taken by the user in the notification section 132E. In the illustrated example, the number of times the user has taken measurements using the measurement unit 30 is displayed, but in addition to this, evaluation results of past measurements, etc. may also be displayed. This allows the user to easily confirm that he or she is continuing to take measurements, which may motivate the user to continue taking measurements.

[0155] The terminal device 10 notifies the user of information on a target pattern for stepping out according to the user's height through a guide output unit 132F by video and audio. This allows the user to easily visualize stepping out toward the target pattern, and easily perform the action for measurement.

[0156] Terminal device 10 displays the detection results by measuring device 31 on detection result display section 132G. For example, terminal device 10 displays information such as that both feet of the user are in the prescribed positions for measurement (both feet are placed in the home position) on detection result display section 132G. This allows the user to easily recognize that the measurement is being performed accurately according to the procedure, and allows the user to perform the measurement with a sense of security that there will be no measurement errors.

[0157] The example screen of FIG. 10(C) shows a situation in which the user is measuring a motion of twisting his / her upper body on the terminal device 10.

[0158] 10(C), the terminal device 10 detects that the height of the pole 33 is not appropriate, and displays a message in the guide output unit 132F urging the user to extend the pole 33 to a predetermined height. This corresponds to the process of step S903 in FIG. 9.

[0159] The example screen of FIG. 10(D) shows a situation in which the terminal device 10 notifies the user that the measurement has been completed, the evaluation result based on the measurement result, and encourages the user to register other users as friends.

[0160] 10(D), the terminal device 10 displays, in the guide output unit 132F, that the measurements for all of the multiple measurement items have been completed, and the results of the evaluation of the user's body movements for the measurement items. The terminal device 10 refers to the user's past measurement history, and displays the results of comparison with past measurement results (such as changes in the range of motion of the joints compared to the previous time) in the guide output unit 132F.

[0161] The terminal device 10 displays, in the notification section 132M, a display encouraging the user to register the other user as a friend, and the circumstances under which the other user is proposed as a friend candidate (such as measurements being taken at the same location).

[0162] <Summary> (1) As described above, according to this embodiment, there is no need to apply a physical measuring tool such as a goniometer to the user's body when checking the ROM, and the user can easily perform the measurement by himself / herself with the guidance of the terminal device 10. In addition, by proceeding with the measurement according to the measurement items, the range of motion of each joint of the user's lower body and upper body can be evaluated.

[0163] (2) When measuring the user's body movements, the mat 32 and the pole 33 can be used to allow the user to recognize the target point for the user to move the body. This stabilizes the form in which the user moves the body, and the measurement results can be accumulated and compared, making it easier for the user to check changes in the body movements. For example, by using the mat 32, (i) it is possible to prescribe the stride, such as the step length and range of movement, when the user steps out, and to encourage the user to perform the prescribed movement in the measurement. When continuously measuring the user's body movements, it is possible to easily compare the measurement results over time by prescribing the movements made by the user during the measurement. In other words, by prescribing the user's body movements, it is possible to further facilitate evaluation of how the user's body movements have changed. In addition, by using the mat 32, (ii) it is possible to prescribe the user's standing position relative to the measuring device 31. The terminal device 10 can determine whether the mat 32 is placed at a predetermined position relative to the measuring device 31 based on the measurement results of the measuring device 31. Therefore, when the user's body movements are measured using the mat 32, the user's standing position is prescribed by each pattern on the mat 32. This makes it easier for the user's body movement to fall within the measurement range of the measuring device 31. For example, even if the user thrusts his / her arm straight up or to the side, by placing the mat 32 in a predetermined position relative to the measuring device 31 in advance, the user's body movement can be prevented from going outside the measurement range of the measuring device 31. In addition, by using the mat 32, (iii) when the user is made to perform a specific movement, the user can be made to assume an appropriate posture before measurement. For example, the home position can specify the user's standing position, and when the user is made to perform a specific movement (such as squatting, lifting one leg), measurement can be started in an appropriate posture (such as a predetermined foot width).

[0164] This makes it possible to easily evaluate the flexibility of each joint in the user's entire body.

[0165] (3) Furthermore, in the first embodiment, the terminal device 10 may suggest an exercise suitable for the user according to the measurement result of the user's body motion (such as the result of the ROM check). For example, the terminal device 10 holds an exercise menu (training item information 185, 285) in association with the classification result of the evaluation result of the ROM check (measurement history information 183, measurement history database 283). That is, the terminal device 10 holds information on a training menu to be proposed in advance in association with the measurement result and evaluation result of the user's body motion. The terminal device 10 may refer to the measurement history database 283, and specify an exercise menu to be proposed to the user based on the measurement result and evaluation result of the user's body motion such as the ROM check, and present the specified exercise menu to the user by displaying it on the display 132, etc. For example, the range of motion of each joint of the user may be evaluated based on the result of the ROM check. The server 20 may predetermine the width of the range of motion of the joints required for the user to perform the exercise in the training item information 285. For example, if the result of the ROM check shows that the range of motion of the hip joint of the user is smaller than a certain level (if the hip joint is stiff), a specific training item may not be provided to the user as an exercise (the terminal device 10 may not be able to execute processing such as displaying an image based on the training item). Also, based on the result of the ROM check, an exercise menu according to the evaluation result of each joint of the user's body may be provided to the user.

[0166] The terminal device 10 also measures the body movements of the user while performing the training indicated in the training items, and judges whether the user is able to move the body as specified in the training. As a result of the user continuing the exercise, it is expected that the muscle strength of each part of the body for a beautiful walking posture will gradually improve, and the user will be able to perform the movements required for the exercise. As a result, the terminal device 10 may judge that the user has completed the exercise menu, and present the user with a different exercise menu.

[0167] In addition, an exercise menu may be suggested to the user based on the history of exercise menus performed by other users who have similar evaluation results of the measurement results of the user's physical movements.

[0168] (4) In the above description of the first embodiment, an example has been described in which the measurement device 31, the mat 32, and the pole 33 are installed in the real world to measure and evaluate the user's body movements.

[0169] In addition, objects corresponding to the mat 32 and the pole 33 may be placed in the virtual space to measure the user's body movements. For example, the user may view the display 132 or wear a head-mounted display (HMD). The terminal device 10 displays a field of view image according to the settings of the virtual camera in the virtual space on the display 132 or the HMD. The terminal device 10 tracks the user's body movements by attaching motion sensors to each part of the user's body (head, shoulders, elbows, hands, waist, knees, feet, etc.). The user moves the user's body while viewing each object placed in the virtual space. This makes it possible to measure and evaluate the user's body movements without placing the mat 32 and the pole 33 in the real world.

[0170] In addition to placing the mat 32 and the pole 33 in the real world, an image corresponding to the mat 32 may be projected onto the floor or the like, and an image (such as a three-dimensional image) corresponding to the pole 33 may be displayed in the real world. For example, in the real world, light may be projected from a laser, a projector, or the like to display the virtual mat 32 and the pole 33 so that they can be visually recognized by the user.

[0171] <Second embodiment> The measurement system 1 in the second embodiment will be described. In the second embodiment, the measurement system 1 will be described for measuring the user's walking movements and evaluating the walking posture, thereby supporting the user in acquiring a walking style that is beautiful and allows the user to easily exercise their physical functions.

[0172] FIG. 11 shows a situation in which the walking motion of a user is measured by the measurement system 1 in the second embodiment.

[0173] <1 Overview> The measuring device 31A measures the walking movement of the first user User_A from a predetermined position to the measuring device 31A in the detection range 38A. The terminal device 10A notifies the first user User_A on the display 132A that the walking state is measured and how to move the body when walking. As shown in the figure, the terminal device 10A notifies the user to walk while putting weight on the front foot when walking.

[0174] In the example of FIG. 11, the coordinate axes are defined as follows. That is, with the position of the first user User_A as a reference, the walking direction is defined as "forward", the direction opposite to the walking direction is defined as "backward", and the axis determined by the walking direction is defined as the "front-rear axis". Meanwhile, with the position of the first user User_A as a reference, the direction perpendicular to the walking direction on the surface where the walking foot touches the ground is defined as the "left-right axis", the direction of the user's right half body is defined as the "right direction", and the direction of the user's left half body is defined as the "left direction". The axis in the direction of the torso from the feet of the first user User_A to the top of the head is defined as the "vertical axis".

[0175] The terminal device 10A notifies the first user User_A via the display 132A to walk toward the measuring device 31A from a position that is a predetermined distance away from the position of the measuring device 31A. Here, the starting point where the first user User_A starts walking may be, for example, a position that is several meters away from the measuring device 31A, and may be set at a position where the walking cycle of the first user User_A (a stepping out of the right foot and a stepping out of the left foot are regarded as one cycle) can be repeated multiple times. The measuring device 31A can measure the walking cycle of the first user User_A multiple times. When the terminal device 10A measures the walking motion of the user by the measuring device 31A, if the first user User_A is not located at the starting point, the terminal device 10A may notify the first user User_A via the display 132A or the like to prompt the first user User_A to move to the starting point. In response to the presence of the first user User_A at the starting point, the terminal device 10A may notify the first user to start measuring the walking motion (or may notify the first user to prompt the user to start walking motion).

[0176] FIG. 12 shows each part of the user's body identified by the terminal device 10 based on the measurement results of the measuring device 31. The terminal device 10 identifies the joints of the limbs, the head, shoulders, trunk, pelvis, feet, heels, toes, ribs, anterior superior iliac spine, etc. as each point characteristic of the user's body. The terminal device 10 identifies the thoracic vertebrae (center of the trunk), the center of the pelvis, etc. For example, the terminal device 10 may prompt the user to take a predetermined pose (such as a pose with both arms thrust out to the sides to form a T-shape, a pose with both arms touching the torso, etc.), and based on the measurement results of the measuring device 31 in these predetermined poses, the terminal device 10 may identify the positions of each joint of the user in the measurement results of the measuring device 31.

[0177] <2 Data Structure> FIG. 13 is a diagram showing the data structure of measurement result database 287 and evaluation result database 288 stored in server 20 in the second embodiment.

[0178] The measurement result database 287 is a database that holds the measurement results of the physical movements of each user measured by the measuring device 31 (for example, walking movements, movements to balance the trunk, etc.).

[0179] Each record in the measurement result database 287 includes an item "measurement data ID", an item "date and time", an item "user ID", an item "time elapsed from start of measurement", and an item "measurement data".

[0180] The item "measurement data ID" indicates information for identifying each piece of measurement data measured by the measuring device 31. In the second embodiment, the terminal device 10 starts and ends measurement of the user's body movement (e.g., measurement of a walking cycle), thereby causing the server 20 to store information for identifying each piece of measurement data in association with the measurement data.

[0181] The item “date and time” indicates the date and time when the measurement device 31 measured the user's body movement.

[0182] The item “user ID” is information for identifying each user whose physical movements are measured by the measurement device 31.

[0183] The item "time elapsed since start of measurement" indicates the time elapsed since the measurement device 31 started measuring the user's body movement.

[0184] The item "measurement data" indicates the position of each part of the user's body (head, shoulders, ...) at the timing indicated in the item "elapsed time from start of measurement" as described in Fig. 12. For example, the item "measurement data" may store the measurement results of measuring the distance from the position of measuring device 31 to each part of the user's body in a coordinate system based on the position of measuring device 31.

[0185] The evaluation result database 288 is a database that holds the evaluation results of evaluating the user's body movements (for example, walking movements) based on the measurement results of the measuring device 31 (measurement result database 287).

[0186] Each record in the evaluation result database 288 includes an item "evaluation result ID", an item "measurement data ID", an item "measurement subject", and an item "evaluation result".

[0187] The item “evaluation result ID” is information for identifying each evaluation result obtained by evaluating the measurement results of the user's body movement measured by the measuring device 31.

[0188] The item "measurement data ID" is information that identifies each measurement result (measurement result database 287) of measuring device 31 that is the subject of evaluation.

[0189] The item "measurement target" indicates the content to be measured as the user's body motion. The measurement targets include, for example, the user's posture during walking motion, the balance of the user's trunk, and the like.

[0190] The item "Evaluation result" indicates the evaluation result of the user's body movement based on the measurement result of the measuring device 31. For example, when evaluating the posture of the user during walking, the evaluation targets include "the user's stepping out," "whether the weight is transferred to the stepping front foot," and "whether the weight is transferred onto the front foot." Details will be described later. Also, when evaluating the balance of the user's trunk, the evaluation targets include "whether the position of the shoulders, hips, and feet is misaligned." Details will be described later.

[0191] <3 operations> Hereinafter, a process in which the terminal device 10 measures the body movement of the user and performs an evaluation based on the measurement result in the second embodiment will be described.

[0192] FIG. 14 is a flowchart showing a process in which the terminal device 10 measures and evaluates the walking movement of the user in the second embodiment.

[0193] In step S 1403 , the terminal device 10 detects, based on the measurement result of the measurement device 31 , that the user is at a predetermined position within the measurement range of the measurement device 31 .

[0194] In step S1405, the terminal device 10 outputs a guide (audio, video) to the user to walk. The terminal device 10 detects each joint of the user based on the measurement result of the measurement device 31, and records the position of each joint during the user's walking motion.

[0195] In step S1407, the terminal device 10 classifies the cycle of the user's walking motion into a plurality of phases based on the measurement result of the measurement device 31. The terminal device 10 determines an evaluation value that evaluates the walking motion according to the evaluation items of each phase of the user's walking motion.

[0196] In step S1409, the terminal device 10 outputs the measurement results of the measuring device 31 and the evaluation results of the walking movement together with the ideal walking posture model to the display 132 etc. The terminal device 10 transmits the measurement results and the evaluation results to the server 20.

[0197] In step S1453, the server 20 updates the measurement result database 287 and the evaluation result database 288 based on the measurement results and evaluation results of the user's walking movement.

[0198] <4. Example of measurement and evaluation results> 15 is a diagram showing an example of output when the terminal device 10 outputs the measurement results of measuring the walking movement of the user and the evaluation results of evaluating the walking movement of the user based on the measurement results in the second embodiment. In the example shown in the figure, the terminal device 10 displays the measurement results and evaluation results on the display 132, but the terminal device 10 may output the measurement results and evaluation results by printing them on a paper medium or the like using a printer or the like.

[0199] 15(A) shows a situation in which the terminal device 10 prompts the measuring device 31 to walk a certain number of steps (for example, about a few meters) in order to measure the walking movement of the user. This corresponds to the processes of steps S1403 and S1405 in FIG.

[0200] The terminal device 10 displays, on the display 132, a notification to encourage the user to "walk with weight on the front foot that steps out" when walking, on the display unit 132P1. The terminal device 10 displays, on the display units 132P2 and 132P3, images showing the specific positions of each part of the user's body (each part of the body exemplified in FIG. 12) as the measurement results of the measuring device 31. The terminal device 10 displays the measurement results of the user's body in a manner that makes it easy for the user to visually recognize them, by connecting each part of the user's body in a line. In the example of FIG. 15, each part of the right half of the user's body is connected by a dotted line, and each part of the left half of the body is connected by a solid line. This allows the user to easily recognize whether the measurement results of the measuring device 31 are the movement of the right half of the user's body or the movement of the left half of the body.

[0201] Based on the measurement results of the measuring device 31, the terminal device 10 displays on the display unit 132P2 the results of identifying the positions of each part of the user when viewed from the front of the user (the front-rear axis shown in FIG. 11) at any time.

[0202] The terminal device 10 displays, on the display unit 132P3, the results of identifying the positions of each part of the user when viewed from the side of the user (the left-right axis shown in FIG. 11) based on the measurement results of the measuring device 31 at any time. That is, the terminal device 10 displays the measurement results of the measuring device 31 on the display units 132P2 and 132P3 in real time. This allows the user to obtain feedback of the measurement results of the user's own body movements in real time.

[0203] The terminal device 10 notifies the user to walk closer to the measuring device 31. Here, the terminal device 10 may display the measurement results on the display units 132P2 and 132P3 as if the user's body movement was measured at a position at a certain distance from the user's body. For example, the terminal device 10 generates a three-dimensional virtual section, and arranges objects corresponding to the positions of each part of the user's body in the three-dimensional virtual space based on the measurement results of the measuring device 31 (that is, the distance from the position of the measuring device 31 to the position of each part of the first user User_A). For example, the terminal device 10 identifies coordinates corresponding to the measuring device 31 in the three-dimensional virtual space, and arranges objects corresponding to each part at the coordinates corresponding to the position of each part based on the measurement results of the measuring device 31 for the coordinates corresponding to the measuring device 31. The terminal device 10 holds information on the positions of the objects corresponding to each part in the three-dimensional virtual space in association with the elapsed time associated with the measurement by the measuring device 31. That is, by reproducing the trajectory of the movement of the object corresponding to the position of each part of the user in the three-dimensional virtual space, the walking motion of the user can be reproduced as the trajectory of the position of each part. The terminal device 10 may set a virtual camera at coordinates that have a certain positional relationship with respect to the group of objects corresponding to the positions of each part of the user (for example, a position at a certain distance from the "center of the shoulder" on the front-back axis, or a position at a certain distance from the chest and waist on the left-right axis), and move the virtual camera in conjunction with the movement of the object corresponding to the position of each part.

[0204] The terminal device 10 notifies the display unit 132P4 of information for appropriately performing measurement by the measuring device 31. This corresponds to step S1403 in Fig. 14. For example, when the terminal device 10 measures the user's walking motion by the measuring device 31, it determines whether the position from which the user starts walking is at a predetermined position, and notifies the user to start walking from the predetermined position. As shown in the figure, if the position from which the user starts walking is too close to the measuring device 31, the terminal device 10 notifies the user to start walking further back.

[0205] 15B shows a situation where the terminal device 10 outputs the measurement results and evaluation results for a certain phase among a plurality of phases of the user's walking movement, which corresponds to the process of step S1409 in FIG.

[0206] In Figures 15(B) and 15(C), the measurement results of the user's body are displayed based on the vertical axis, which makes it possible to present to the user how the user's body is rotating based on the vertical axis.

[0207] The terminal device 10 outputs, on the display 132, the evaluation results of the balance of shoulder rotation and the balance of hip rotation during walking as the evaluation results of the beauty of the user's walking posture. The terminal device 10 divides the walking movement into a plurality of phases, and outputs the evaluation results of the balance of shoulder rotation, the balance of hip rotation, etc. for each phase. In this way, by presenting the measurement results and evaluation results for a plurality of phases of the walking movement to the user, it is possible to provide suggestions for making the user's walking movement more beautiful.

[0208] The terminal device 10 displays, on the display unit 132P5, the inclination of shoulder rotation in each of left and right rotations during a walking cycle (one cycle is a movement in which the right foot steps out, then the left foot steps out).

[0209] The terminal device 10 displays, on the display unit 132P6, a "line connecting both shoulders" determined by the positions of the right and left shoulders, a "line connecting both hip joints" determined by the positions of both hip joints, a "center position of the shoulders" that is the center of the positions of the right and left shoulders, and a "center position of the waist" determined by the positions of the hip joints (for example, the positions of the right and left pelvis). The terminal device 10 evaluates the degree of left rotation of the shoulder with respect to the vertical axis (how far the shoulder has rotated) based on the position of the left shoulder (or both shoulders) and the position of the hip joint when the user steps out with the left foot in one cycle of walking motion. For example, the terminal device 10 draws a "line connecting both shoulders" based on the amount of rotation of the shoulder due to left rotation in one cycle with the position of the waist as the reference. The terminal device 10 displays a score (score "4.8 points") as the evaluation result on the display unit 132P6.

[0210] The terminal device 10 displays "a line connecting both shoulders," "a line connecting both hip joints," "center position of the shoulders," and "center position of the waist" on the display unit 132P7. The terminal device 10 evaluates the degree of left rotation of the shoulder with respect to the vertical axis based on the position of the right shoulder (or both shoulders) and the position of the hip joint when the user steps out with the right foot during one cycle of walking. The terminal device 10 draws the "line connecting both shoulders" based on the amount of rotation of the shoulder due to right rotation during one cycle, using the position of the waist as a reference. The terminal device 10 displays a score (score "4.5 points") as the evaluation result on the display unit 132P7, which is a lower score than the left rotation.

[0211] The terminal device 10 displays the "center position of the shoulders" and the "center position of the waist" in the action of stepping out the left foot and the action of stepping out the right foot in one cycle of the walking action on the display units 132P6 and 132P7. This allows the user to recognize how the upper body is tilted with respect to the position of the waist during the walking action. For example, the user can recognize the degree to which the upper body is moving on the left-right axis and the degree to which the upper body is moving on the front-back axis during the user's walking action. In the illustrated example, in the display unit 132P6, the degree of overlap between the "center position of the shoulders" and the "center position of the waist" is large during the left rotation, and the upper body is not shaken, whereas in the display unit 132P7, the degree of overlap between the "center position of the shoulders" and the "center position of the waist" is relatively small during the right rotation, and the upper body is shaken more laterally than during the left rotation. Therefore, the score, which is the evaluation result, is lower during the right rotation than during the left rotation.

[0212] 15C shows a phase in which the terminal device 10 outputs the measurement results and evaluation results when the front feet of the user's walking motion are put on the ground, which corresponds to step S1409 in FIG.

[0213] The terminal device 10 displays, on the display unit 132P8, the measurement result and the evaluation result of the tilt of the rotation of the hips when the user places the front feet on the ground during walking.

[0214] The terminal device 10 displays, on the display unit 132P9, a "line connecting both hip joints" when the left foot is placed on the ground as the front foot, etc. As shown in the figure, the "line connecting both hip joints" is inclined, which indicates that the user is able to rotate the hips when the left foot is placed on the ground.

[0215] The terminal device 10 displays, on the display unit 132P10, a "line connecting both hip joints" when the right foot is placed on the ground as a front foot, etc. As shown in the figure, the "line connecting both hip joints" is inclined, which indicates that the user is able to rotate the hips when the right foot is placed on the ground.

[0216] 16 is a diagram showing an example of an output when the terminal device 10 outputs a measurement result of measuring the walking motion of the user and an evaluation result of evaluating the walking motion of the user based on the measurement result in the second embodiment. This corresponds to the processing of steps S1407 and S1409 shown in FIG.

[0217] In the example of Fig. 16, the terminal device 10 displays the measurement results and evaluation results of one cycle of the user's walking movement in a manner that allows comparison with a reference example (model example, example) for making the walking posture look beautiful. The output example shown in Fig. 16 may also be output by printing it on a paper medium using a printer or the like.

[0218] The terminal device 10 divides one cycle of walking movement into a plurality of steps on the display 132, and displays, for each step, a model for making the walking posture look beautiful, the measurement results of the user's walking movement, and an evaluation result for the measurement results.

[0219] In the example of Figure 16, one cycle of walking motion is divided into the first step, "stepping forward with the foot," the second step, "trying to put weight on the stepping front foot (trying to get on the front foot)," and the third step, "putting all weight on the stepping front foot (getting on the front foot)."

[0220] Here, the items evaluated by the terminal device 10 in the first step, the second step, and the third step include items common to each of these steps and items not common to each of the steps. (1) Evaluation items for the first step - Upper body inclination during walking: Are the parts of the upper body (head, shoulders, etc.) inclined forward or backward in relation to the position of the waist? For example, is the angle of inclination of the line determined by the position of the pelvis, the position of the chest and waist, and the position of the center of the shoulders relative to the vertical axis within a certain range? - The distance between the front and back of the feet when stepping forward: The degree of distance between the front and back of the feet. For example, is the distance between the ankle of the front foot and the ankle of the back foot that is stepping forward greater than a certain distance? Hip position when stepping forward: When stepping forward, are your hips left behind? For example, is your pelvis positioned forward or backward in relation to your shoulders, chest and hips? (2) Evaluation items for the second step · Upper body inclination at the moment the front foot touches the ground: Are the parts of the upper body (head, shoulders, etc.) leaning forward or backward in relation to the position of the waist? - Knee bending when the front foot touches the ground: Are each part of the front foot (ankle, knee, pelvis) bent too much or not? For example, is the angle formed by the line connecting the ankle and knee and the line connecting the knee and pelvis within a certain angle? (3) Evaluation items for the third step The inclination of the upper body when the weight is fully transferred to the front foot: Are the parts of the upper body (head, shoulders, etc.) leaning forward or backward in relation to the position of the waist? - Extension of the supporting leg when putting all weight on the front leg: When putting weight on the supporting leg, using the front leg that stepped out and landed as the supporting leg, are each part of the supporting leg (ankle, knee, pelvis) extended or bent? For example, is the angle formed by the line connecting the ankle and knee and the line connecting the knee and pelvis within a certain angle? Specifically, the terminal device 10 displays a model for the first step on the display unit 132P11. The terminal device 10 displays the measurement results of the user's body in the first step on the display unit 132P21. The terminal device 10 displays the evaluation results of the user's body movement in the first step on the display unit 132P31.

[0221] The terminal device 10 displays on the display units 132P11, 132P12, and 132P13 the positions of the various body parts (joints) when these evaluation items are favorable for a beautiful walking posture. For example, the display unit 132P11 shows the positions of the various body parts when the upper body is not tilted, the front legs and the back legs are spaced apart to a certain degree (the stride is not too small), and the position of the waist is appropriate (not leaning backward).

[0222] In the illustrated example, the display unit 132P31 displays, as evaluation results, the evaluation result RF_ic of the right foot when the right foot steps out and the evaluation result LF_ic of the left foot when the left foot steps out, superimposed on a predetermined bar, for the evaluation item "upper body tilt". The predetermined bar is drawn according to the range that can be taken as an evaluation result. The terminal device 10 displays, in the predetermined bar, a region where the evaluation result is good and a region where the evaluation result is not good, and in the illustrated example, the region where the evaluation result is good is highlighted. For the evaluation item "upper body tilt", the terminal device 10 displays, in association with the evaluation result RF_ic of the right foot and the evaluation result LF_ic of the left foot, in a good range, assuming that the evaluation result is neither too much forward leaning nor too much backward leaning.

[0223] On the display unit 132P31, the terminal device 10 displays the evaluation results for the right leg and the left leg in association with a specified bar for the evaluation items "leg width" and "waist position."

[0224] The terminal device 10 displays the example for the second step on the display unit 132P12. The terminal device 10 displays the measurement results of the user's body in the second step on the display unit 132P22. The terminal device 10 displays the evaluation results of the user's body movement in the second step on the display unit 132P32.

[0225] The terminal device 10 displays the example in the third step on the display unit 132P13. The terminal device 10 displays the measurement results of the user's body in the third step on the display unit 132P23. The terminal device 10 displays the evaluation results of the user's body movement in the third step on the display unit 132P33.

[0226] 17 is a diagram showing an example in which the terminal device 10 outputs an evaluation result of the strain on the knees during the user's walking motion, which corresponds to step S1409 in FIG.

[0227] 17(A) shows an example of outputting measurement results of hip adduction and abduction during a user's walking motion and evaluation results. The terminal device 10 displays the measurement results of the user's walking motion on the display 132 based on the anterior-posterior axis (as if the user were viewed from the front), allowing the user to easily check the bending of the knee during walking motion.

[0228] The terminal device 10 displays the measurement results during walking on the display unit 132P41. The terminal device 10 displays the measurement results when walking with the right foot as the pivot foot on the display unit 132P42. The terminal device 10 displays the measurement results when walking with the left foot as the pivot foot on the display unit 132P43.

[0229] The terminal device 10 displays on the display unit 132P44 a preferable measurement result pattern ("both the supporting leg and the free leg are straight") and a measurement result pattern that may cause knee injury ("possibility of injury to the outside of the knee", "possibility of injury to the inside of the knee") for the user's reference. In other words, if the hip joint is too adducted or too abducted during walking, there is a risk of injuring the knee.

[0230] As a result, by referring to the output result of the terminal device 10, the user can be encouraged to walk in a way that does not hurt his / her knees.

[0231] 17(B) is a diagram showing an example of outputting an evaluation result as to whether or not the knee is moving too far ahead of the ankle during the user's walking motion. The terminal device 10 displays the measurement result of the user's walking motion on the display 132 based on the left-right axis (as if the user were seen from the side), allowing the user to easily check whether or not the knee is moving too far ahead during walking motion.

[0232] The terminal device 10 displays the measurement results during walking on the display unit 132P51. The terminal device 10 displays the measurement results when walking with the right foot as the pivot foot on the display unit 132P52. The terminal device 10 displays the measurement results when walking with the right foot as the pivot foot on the display unit 132P53. The terminal device 10 displays on the display unit 132P51 the evaluation result of the user's walking posture that "the forward load on the knee is small."

[0233] The terminal device 10 displays on the display unit 132P54 a favorable measurement result pattern ("Good") and a measurement result pattern that may injure the knee ("Knee is too far forward") for the user's reference. In other words, if the knee is too far ahead of the ankle when the user tries to step out with the front foot, the knee may be injured. The terminal device 10 evaluates whether the knee is too far ahead when the user tries to step out with the front foot based on whether the angle at which the line connecting the position of the ankle and the position of the knee intersects with the line connecting the position of the knee and the position of the pelvis is equal to or greater than a predetermined angle (whether the area from the knee to the ankle is forward).

[0234] As described above, the walking motion of the user is evaluated for multiple phases of the walking cycle, such as stepping out with the supporting foot, landing with the supporting foot, and trying to put weight on the supporting foot. This encourages the user to walk more beautifully by referring to the evaluation results of each phase. In addition, the user is encouraged to walk closer to a beautiful walking posture by being able to refer to the comparison between the model body movement and the measurement results of the user's body movement.

[0235] <Third embodiment> The measurement system 1 in the third embodiment will be described. In the third embodiment, a process for displaying the measurement result of the balance of the user's trunk and enabling the user to achieve a balanced state of the trunk will be described.

[0236] <Operation> Hereinafter, a process in which the terminal device 10 measures the user's body movements and performs an evaluation based on the measurement results will be described in the third embodiment.

[0237] FIG. 18 is a flowchart showing a process in which the terminal device 10 measures and evaluates the balance of the user's trunk in the third embodiment.

[0238] In step S1803, the terminal device 10 outputs guidance (audio, video) to measure the balance of the user's trunk (shoulders, waist, legs) based on the measurement results of the measurement device 31. The terminal device 10 displays the measurement results by displaying the positions of the user's shoulders, waist, and legs as circles based on the vertical axis. The terminal device 10 outputs guidance to encourage the user to balance the trunk according to the degree of overlap of the positions of the user's shoulders, waist, and legs.

[0239] In step S1805, the terminal device 10 evaluates the balance of the user's trunk based on the measurement results of the measurement device 31, and outputs an evaluation value. The terminal device 10 transmits the measurement results of the measurement device 31 and the evaluation results to the server 20.

[0240] In step S1853, the server 20 updates the measurement result database 287 and the evaluation result database 288 based on the measurement results and evaluation results of the user's trunk balance.

[0241] <Output example> FIG. 19 is a diagram showing an example in which the terminal device 10 outputs the measurement result of measuring the balance of the user's trunk and the evaluation result of the balance of the trunk in the third embodiment.

[0242] Fig. 19(A) shows an example of a screen in which the terminal device 10 outputs the measurement results in real time in order to measure the balance of the user's trunk using the measurement device 31. This corresponds to the processing of step S1803 in Fig. 18. In the example of Fig. 19(A), circular figures (such as a perfect circle or an ellipse) corresponding to the shoulders, waist, and legs of the user's body are displayed on the display unit 132P61.

[0243] The terminal device 10 displays the measurement results of each part of the user's body on the display unit 132P62 based on the anterior-posterior axis.

[0244] The terminal device 10 displays the positions of the shoulders, waist, and legs of the user's body based on the vertical axis on the display unit 132P63, based on the measurement results of the measuring device 31. The terminal device 10 displays the measurement results of the measuring device 31 on the display unit 132P63 at any time. Therefore, if the user's body is shaken, the circular figures corresponding to the shoulders, waist, and legs will not overlap, causing a misalignment. The example screen of FIG. 19(A) shows that the parts of the user's body are not aligned along the vertical axis and are misaligned. The user is prompted to move his or her body to balance the trunk while watching the display on the display unit 132P63 of the display 132.

[0245] The terminal device 10 notifies the user on the display unit 132P64 to encourage the user to balance the trunk. Based on the measurement results of the measurement device 31, the terminal device 10 identifies the body movements of the user required to balance each part of the user's body. In the illustrated example, the head and feet are located behind the position of the user's waist. In other words, the position of the user's waist is protruding forward. The terminal device 10 displays on the display unit 132P64 a message encouraging the user to adjust the inclination of the body in the front-back direction. This allows the user to easily recognize the body movements required to balance the trunk.

[0246] Fig. 19(B) shows an example of the measurement result display when the user's trunk is well-balanced. On the display unit 132P63, the user's shoulders, waist, and legs are aligned along the vertical axis, and the trunk is well-balanced. On the display unit 132P64, the terminal device 10 notifies the user that the trunk is well-balanced.

[0247] 19C shows an example of a display when the measurement of trunk balance is completed and the evaluation results are output, which corresponds to the process of step S1805 in FIG.

[0248] The terminal device 10 displays on the display unit 132P65 a message indicating that the measurement of trunk balance has been completed, and outputs an evaluation result based on the measurement result.

[0249] Based on the evaluation result of the trunk balance, the terminal device 10 determines a training menu for adjusting the trunk balance as a training menu to be proposed to the user. In the illustrated example, the terminal device 10 displays, on the display unit 132P65, the evaluation result of the trunk balance of the user and candidates for a training menu to be proposed to the user according to the evaluation result. On the display unit 132P66, the terminal device 10 accepts an operation for starting the training menu to be proposed to the user. In response to the user's input operation on the display unit 132P66, the terminal device 10 reads out the corresponding training menu.

[0250] The terminal device 10 accepts a training menu designation from the user on the display unit 132P67. That is, the display unit 132P67 accepts an input operation for starting other training, regardless of the training menu that the terminal device 10 proposes to the user based on the evaluation result of the balance of the trunk.

[0251] The features described in each of the above embodiments may be combined with each other.

[0252] <Fourth embodiment> In the fourth embodiment, a technique for proposing a training menu suitable for a user based on the measurement results of the user's body movements, such as a ROM check (first embodiment), a walking posture check (second embodiment), and a measurement of trunk balance (third embodiment), will be described. By measuring and evaluating the user's body movements as described above, the terminal device 10 can identify which parts of the user's body should be trained, and propose an appropriate training menu based on the identified results.

[0253] With this configuration, when adding new training menu content, it is possible to provide the user with content that corresponds to the measurement results of the user's body movements by associating the training menu content with which part of the body it is suitable for and with which of the user's possible physical problems it addresses, making it even easier to add content.

[0254] (1) As described above, the terminal device 10 may propose an exercise menu to the user based on the measurement results and evaluation results in each step of the user's walking cycle. For example, the terminal device 10 compares the model body movement with the user's body movement (example of FIG. 16) to identify muscle parts that the user should improve. For example, when the evaluation result of the first step (movement of stepping forward) as the user's walking movement shows that the front and back of the feet are small, the server 20 refers to the training item information 285 to identify an exercise menu that is expected to enable the user to take a large step forward. That is, the server 20 may previously associate and hold the proposed exercise menu according to the evaluation results of the evaluation items of each step of the walking movement and the evaluation item of the balance of the trunk.

[0255] (2) The terminal device 10 may propose a training menu to the user so that the left-right difference in the user's body movements can be eliminated. As described above, the terminal device 10 evaluates the left-right balance of the user's body movements by performing a ROM check and a walking posture check. For example, the terminal device 10 evaluates the left-right balance of the rotational movements based on the vertical axis based on the evaluation results of the shoulder rotation and waist rotation during the user's walking movements. The terminal device 10 identifies the muscle parts that are the cause of the user's body movements being asymmetrical. For example, the terminal device 10 holds in advance information on the muscle parts that are the cause of the left-right difference in the amount of shoulder rotation. As described above, the terminal device 10 can identify the muscle parts that should be trained to eliminate the left-right balance difference by previously storing and associating the target of evaluation during walking movements with the information on the muscle parts that are the cause of the left-right difference.

[0256] (3) Based on the evaluation result of the user's physical movement, the terminal device 10 may predict a possible future state of the user's body and, based on the prediction result, may suggest a training menu to the user.

[0257] For example, the terminal device 10 stores (i) items of possible physical problems of the user, such as "stiff neck and shoulders," "sagging upper arms," ​​"rolled shoulders, forward head posture," "lower back pain," "swelling legs," and "stiff arms," ​​in association with (ii) information on parts of the user's body, as predetermined information. The terminal device 10 also stores information on training menus to be proposed as training, in association with each of the above items of possible physical problems of the user.

[0258] The terminal device 10 performs a ROM check and a walking movement check, and identifies muscle parts that are desirable to be improved based on the measurement results and evaluation results. For example, when the twisting of the body is not good in the ROM check, which muscle parts are associated with it is stored in advance in the terminal device 10. For example, in the ROM check, the terminal device 10 stores muscle information in association with the evaluation items of the range of motion of the shoulders, waist, etc.

[0259] The terminal device 10 identifies possible future problems by referring to the predetermined information, such as by identifying muscle parts that are stiff. Based on the identification result, the terminal device 10 determines a training menu to be proposed to the user so that the user can deal with the possible future problems in the user's body.

[0260] As described above, the information on each muscle part is associated with the evaluation items of balance during walking and body twisting, and the information on each muscle part is associated with information on each item of possible physical troubles of the user. This allows the terminal device 10 to propose a training menu customized for each user based on the evaluation results of the user's physical movements.

[0261] <Data Structure in the Fourth Embodiment> FIG. 20 is a diagram showing the data structure of various data stored in the server 20 in the fourth embodiment.

[0262] In the measurement item information 282, the server 20 holds, for each measurement item, an item "Possible future concerns" and an item "Related muscle area."

[0263] The item "Possible future problems" is information (for example, tags such as "stiff shoulders" or "stiff neck") that identifies problems that may occur to the user's body in the future depending on the evaluation results of the measurement results for each measurement item.

[0264] The item "Related muscle parts" is information on the body parts (muscle parts) that are recommended to be trained depending on the evaluation results of the measurement results for each measurement item. In the item "Related muscle parts", information on body parts may be stored in stages according to the level of the evaluation results.

[0265] In the training item information 285, the server 20 holds, for each training item, an item "contents," an item "corresponding concern," and an item "related muscle part."

[0266] The item "content" indicates content (video, motion data that serves as a reference for the user's body movements, audio data, etc.) that is presented to the user when performing training.

[0267] The item "addressed problem" is information (for example, tags such as "stiff shoulders" and "stiff neck") that identifies a problem that may occur to the user's body in the future depending on the evaluation result for each training item.

[0268] The item "Related muscle parts" is information on the body parts (muscle parts) that are desirable to train for each training item depending on the evaluation results. In the item "Related muscle parts", information on body parts may be stored in stages according to the level of the evaluation results.

[0269] The server 20 holds the item "Potential future worries" in the evaluation result database 288. The item "Potential future worries" is information that specifies worries that may occur to the user's body in the future depending on each evaluation result (for example, tags such as "stiff shoulders" and "stiff neck").

[0270] As described above, the server 20 updates the evaluation result database 288 according to the evaluation result of the user's body movement. For example, the server 20 may specify information on possible future worries about the user's body according to the measurement result of the ROM check (measurement item information 282). The server 20 specifies information on possible future worries about the user by referring to the evaluation result database 288, and extracts training items to be suggested to the user by referring to the training item information 285. When extracting multiple training items to be suggested to the user, the server 20 sets priorities based on a predetermined rule. For example, the server 20 may prioritize training items that can eliminate worries corresponding to the unfavorable evaluation result of the user's body movement according to the degree to which the evaluation result is unfavorable.

[0271] <Example of a screen in the fourth embodiment> FIG. 21 is an example of a screen in which the terminal device 10 presents possible physical concerns of the user based on the measurement results and evaluation results of the user's physical movement, and suggests an exercise menu to address the concerns.

[0272] FIG. 21(A) shows the terminal device 10 displaying the results of identifying possible physical concerns of the user based on the measurement results and evaluation results of the user's physical movements.

[0273] The terminal device 10 displays on the display unit 132P71 a message encouraging the user to take measurements via the terminal device 10 every day, while displaying on the display unit 132P72 a condition that may occur in the user's body in the future. In the illustrated example, the operation reception unit 132P73 displays that a "neck and shoulder stiffness" problem may occur, while displaying a training menu corresponding to the "neck and shoulder stiffness" problem in response to the user's input operation on the operation reception unit 132P73. In addition, the terminal device 10 displays on the operation reception unit 132P74 that a "sagging upper arms" problem may occur, while displaying a training menu corresponding to the "sagging upper arms" problem in response to the user's input operation on the operation reception unit 132P74. In addition, the terminal device 10 displays in the operation reception section 132P75 that the problem of "rolled shoulders and forward head posture" may occur, and in response to the user's input operation to the operation reception section 132P75, displays a training menu corresponding to the problem of "rolled shoulders and forward head posture."

[0274] The terminal device 10 displays, on the display unit 132P76, an operation reception unit 132P77 for starting a training menu that is smaller than the number of training menus displayed on the display unit 132P76 among the training menu candidates proposed to the user for a user who is short on time. The terminal device 10 sets priorities, etc., for the training menu candidates proposed to the user based on a predetermined rule, and displays them on the display unit 132P76.

[0275] The terminal device 10 accepts an operation from the user on the display unit 132P78 to edit a training menu that the user wishes to execute, such as "favorites."

[0276] The terminal device 10 presents a questionnaire to the user and accepts responses to the questionnaire by accepting an input operation to the operation acceptance unit 132P91. The terminal device 10 may identify a concern of the user based on the input result of the questionnaire and propose a training menu corresponding to the identified concern.

[0277] FIG. 21(B) shows a situation where a recommended training menu is proposed to the user. The terminal device 10 displays, on the display unit 132P81, information on body parts (such as "Are you suffering from stiff neck or shoulders") for the user's concerns identified based on the measurement results and evaluation results. The terminal device 10 displays, on the display unit 132P81, the cause of the user's physical concerns (such as "Your shoulders tend to be positioned forward") based on the measurement results and evaluation results. The terminal device 10 displays, on the display unit 132P81, guidelines for resolving the user's physical concerns (such as "It is expected to improve blood flow to the upper body" and "Stretch the back of the thighs") based on the measurement results and evaluation results, and suggests exercises corresponding to the guidelines.

[0278] For example, the terminal device 10 displays operation acceptance sections 132P83 to 132P87 for starting training using a recommended training menu, as shown on the display section 132P82, in response to an input operation to the operation acceptance section 132P73. The terminal device 10 may also play a video for starting training in response to an input operation to these operation acceptance sections 132P83 to 132P87.

[0279] In response to an input operation to the operation receiving portion 132P77, the terminal device 10 displays the operation receiving portions 132P83 to 132P85 in a state in which the number of training menus to be proposed is narrowed down for users who are short on time, as shown on the display portion 132P88.

[0280] This makes it possible to encourage the user to continue training according to the user's condition at any given time.

[0281] <Example of operation> The operation of each device in the fourth embodiment will now be described.

[0282] FIG. 22 shows the flow of processing in each device in the fourth embodiment.

[0283] In step S2203, the terminal device 10 evaluates at least one of the user's range of motion, walking posture, and trunk balance based on the measurement results of the user's body movement, and outputs the evaluation results. The terminal device 10 transmits the measurement results and the evaluation results to the server 20.

[0284] In step S2253, the server 20 updates a database (such as the evaluation result database 288) based on various measurement results of the user, such as the measurement results and evaluation results received from the terminal device .

[0285] In step S2255, the server 20 identifies a physical problem that may occur to the user in the future, based on the evaluation result database 288 etc. The server 20 selects a training menu related to the body part corresponding to the identified problem from among a plurality of training menus, based on the training item information 285, and responds with the selected training menu to the terminal device 10 of the user.

[0286] In step S2205, the terminal device 10 displays to the user a training menu that addresses possible future physical concerns.

[0287] <Modification> In the fourth embodiment, questions may be presented to the user, and the priority of the training menu to be proposed to the user may be set according to the answer results (questionnaire results) inputted and accepted from the user.

[0288] This allows the server 20 to propose a training menu based on the user's physical measurement results, as well as present the user with questions about the user's body and propose a training menu based on the results of receiving answers to the questions from the user. That is, the server 20 updates the priority of the training menu proposed to the user based on the answers to the questions. The server 20 presents the user with a training menu with a high priority based not only on the physical evaluation results but also on the answers from the user to the questions. This allows the server 20 to propose a training menu that is more suitable for the user's physical condition.

[0289] For example, the terminal device 10 can receive from the user information on muscle parts of the user's body that have not yet been measured (evaluation based on the measurement results has not yet been performed) by the measurement unit 30 in the measurement system 1, by receiving a questionnaire about the user's body from the user. The terminal device 10 can also identify future concerns about muscle parts that have not yet been evaluated by the measurement system 1, based on the questionnaire results entered by the user. The terminal device 10 determines a training menu to be proposed to the user so that the user can appropriately deal with future concerns about the user's body, based on the concerns identified based on the measurement results of body movement and the questionnaire responses from the user.

[0290] <Data structure in modified example> FIG. 23 is a diagram showing the data structure of various data stored in the server 20 in a modification of the fourth embodiment.

[0291] In the user questionnaire information 289, the server 20 holds, for each questionnaire result, an item "answer options," an item "current worries," an item "possible future worries," and an item "associated muscle parts."

[0292] The item "answer options" includes information on options for the user to answer questions (questionnaires) presented to the user of the terminal device 10. For example, if the question concerns a user about physical discomfort, options for possible body parts that may cause discomfort are presented to the user. In this case, the user's concerns about body parts can be identified according to the results of the user's selection of each option.

[0293] The item "Current Concerns" is information (e.g., tags such as "lower back pain" or "knee pain") that identifies physical concerns that the user may currently be experiencing based on the evaluation results of the questionnaire responses for each questionnaire result.

[0294] The item "Possible future worries" is information (for example, tags such as "stiff shoulders" or "stiff neck") that identifies worries that may occur to the user's body in the future depending on the evaluation results of the questionnaire responses for each questionnaire result.

[0295] The item "Related muscle parts" is information on the body parts (muscle parts) that are desirable to train depending on the evaluation results of the questionnaire answers for each questionnaire result. In the item "Related muscle parts", information on body parts may be stored in stages according to the level of the answers.

[0296] The server 20 holds the item "current concern" in the evaluation result database 288. The item "current concern" is information (for example, tags such as "lower back pain" or "knee pain") that identifies a physical concern that the user may currently be having depending on the evaluation results obtained by evaluating the questionnaire response contents for each evaluation result.

[0297] <Operation in the modified example> As described above, the server 20 updates the evaluation result database 288 in response to the evaluation results of the user's physical movement and the user questionnaire information 289. For example, information on possible future physical problems of the user can be identified in response to the measurement results of the ROM check (measurement item information 282). For example, information on physical problems that the user currently has or may have in the near future can be identified in response to the results of a questionnaire about the user's body (user questionnaire information 289). The server 20 refers to the evaluation result database 290 to identify information on possible future problems of the user, and refers to the training item information 285 to extract training items to be suggested to the user.

[0298] When extracting a plurality of training items to be proposed to the user, the server 20 sets priorities based on a predetermined rule according to the degree to which the evaluation result of the user's physical movement is not good. For example, the server 20 may prioritize training items that can resolve concerns corresponding to the evaluation result of the user's physical movement that is not good according to the degree to which the evaluation result of the user's physical movement is not good. The server 20 corrects and sets priorities of training items to be proposed to the user based on the questionnaire results. For example, the server 20 may prioritize training items that can resolve concerns of the user's body identified from the questionnaire results according to the degree to which the evaluation result of the body related to the contents of the questionnaire answer is not good compared to the evaluation result of the ROM check.

[0299] FIG. 24 is a diagram showing an operational flow in a modified example of the fourth embodiment.

[0300] As shown in FIG. 24, the terminal device 10 presents a question (questionnaire) to the user of the terminal device 10 at any timing, such as step S2202 or step S2204.

[0301] The process of step S2202 will be described. For example, in response to the user completing the process of logging in to the measurement system 1 in step S700, the terminal device 10 displays a question on the display 132. The terminal device 10 accepts input of an answer to the question from the user. The terminal device 10 transmits the answer result for which the input was accepted to the server 20.

[0302] In step S 2252 , the server 20 updates the user survey information 289 based on the information received from the terminal device 10 .

[0303] In step S2253, in response to receiving from the terminal device 10 the measurement results by the measurement unit 30 etc. in step S2203 and the evaluation results based on the measurement results, the server 20 updates various databases such as the measurement result database 287.

[0304] The process of step S2204 will be described. For example, in response to completing the measurement of the user's body movement in step S2203, the terminal device 10 presents a question to the user (for example, displays the question on the display 132). The terminal device 10 transmits the answer result, which has been input, to the server 20.

[0305] In step S 2254 , the server 20 updates the user survey information 289 based on the information received from the terminal device 10 .

[0306] In step S2256, the server 20 identifies the user's current worries and possible future physical worries based on the user questionnaire information 289, the evaluation result database 288, and the like. The server 20 selects a training menu corresponding to the identified worries from among a plurality of training menus based on the training item information 285, and responds with the selected training menu to the user's terminal device. For example, the server 20 identifies the user's worries based on the results of the user's answers to questions (user questionnaire information 289), and identifies a training menu to be proposed to the user according to the identified worries (training item information 285). The server 20 also identifies a training menu to be proposed to the user based on the evaluation result database 288 as described above. The server 20 responds to the terminal device 10 with one or more training menus identified in this way. At this time, the server 20 may set priorities for the identified plurality of training menus based on a predetermined rule, and respond with the training menu to the terminal device 10 according to the priorities.

[0307] <Screen Example 1 in the Fourth Embodiment> 25 is an example of a screen in which the user answers a questionnaire when the terminal device 10 presents possible physical concerns of the user based on the measurement results and evaluation results of the user's physical movements and proposes an exercise menu to address the concerns. For example, the terminal device 10 displays the screen (display unit 132P71) shown in FIG. 21(A) by accessing the server 20, etc.

[0308] The terminal device 10 displays the display unit 132PA2 shown in FIG. 25(A) in response to the satisfaction of certain conditions, such as the timing at which the screen is displayed (the timing at which information for displaying the screen is received from the server 20 and displayed on a browser, etc.), the timing at which a certain time has elapsed since the screen was displayed, or the user performing a certain operation on the screen, and so on, to prompt the user to answer the questionnaire.

[0309] FIG. 25(A) shows a display section 132PA2 for the user to answer a questionnaire, which is overlaid on a screen (display section 132P71 described in FIG. 21(A)) on which the terminal device 10 displays the results of identifying possible physical problems of the user based on the measurement and evaluation results of the user's physical movements.

[0310] The terminal device 10 displays a question for the user ("What kind of physical problems do you have?") on the display unit 132PA3 of the display unit 132PA2. The server 20 determines the content of the question to be presented to the user on the display unit 132PA3. For example, the server 20 holds a plurality of questions in advance, and holds information on the body parts of the user and information on candidate answers to the questions in association with each question. The server 20 may refer to the measurement result database 287 and the evaluation result database 288, and determine the question to be presented to the user based on the history of measurements performed by the user, and display the determined question on the terminal device 10. For example, the server 20 may present the user with a question (such as "What kind of physical problems do you have?") about whether or not the user is aware of any discomfort in each body part when a certain period of time has passed since the user last measured, based on the history of measurements performed by the user using the measurement system 1 (such as the time interval between measurements).

[0311] Also, for example, the server 20 may determine the body part to be included in the question to be presented to the user based on the measurement result of the body part of the user. For example, if there is a body part of the user that has not been measured for at least a certain period of time (for example, there is no history of waist measurement, or a certain period of time has passed since the waist measurement), the body part may be included in the question (such as "How is your waist?").

[0312] In the example of Fig. 25(A), the terminal device 10 displays answer candidates to the question on the display unit 132PA4 and accepts a selection from the user. In the illustrated example, the display unit 132PA4 displays, as answer candidates to the question, "lower back problems" for specifying a lower back ailment, "knee problems" for specifying a knee ailment, and "***" for displaying answer candidates for other body parts. The terminal device 10 accepts the designation of "lower back problems" from the user, and highlights the candidate to allow the user to easily recognize that the user has made the designation.

[0313] The terminal device 10 displays a question about the degree of the concern (degree of discomfort in a body part) on the display unit 132PA5.

[0314] The terminal device 10 displays, on the display unit 132PA6, possible answers to the question displayed on the display unit 132PA5 and accepts a selection from the user. In the illustrated example, the terminal device 10 displays, on the display unit 132PA6, "mild" indicating a light degree of worry, "medium" indicating a medium degree of worry, and "heavy" indicating a heavy degree of worry as options. The terminal device 10 accepts the selection of "medium" from the user and highlights the selected option.

[0315] The terminal device 10 accepts an operation from the user on the display unit 132PA1 indicating that the answer to the question has been completed. As a result, the terminal device 10 transmits the user's answer result to the server 20. The server 20 updates the user survey information 289 by receiving the user's answer result. For example, if the user specifies information related to a body part, such as "back pain", as the answer result, the server 20 updates the user survey information 289 based on the user's pain "back pain".

[0316] FIG. 25(B) is an example of a screen in which a training menu is proposed to the user based on the answer to the question received from the user and the measurement and evaluation results of each part of the user's body.

[0317] Based on the answers received from the user to the question (user questionnaire information 289), the server 20 displays information about the user's body part trouble (e.g., information including the body part and the degree of trouble, such as "It seems that your knees are under moderate strain") on the display unit 132P81. The terminal device 10 displays the cause of the user's body trouble (e.g., "Are you maintaining an unnatural posture for a long time?") on the display unit 132P81. The terminal device 10 displays guidelines for resolving the user's body trouble (e.g., "Relax the muscles around your waist") and suggests exercises corresponding to the guidelines on the display unit 132P81.

[0318] <Other Modifications> (1) Ask users questions about their exercise experience In the above embodiment, the server 20 presents a question to the user and suggests a training menu based on the answer to the question. The server 20 may ask the user about the user's exercise experience as the question to be presented to the user.

[0319] For example, the server 20 may present the user with questions about the user's past sports experience and exercise habits. The server 20 accepts answers to the questions from the user (such as the sports the user has played in the past and whether the user has exercise habits). Based on the user's answers, if the user has relatively little exercise experience (such as the period during which the user played sports is less than a certain period, or the habit of exercising is less than a certain frequency), the server 20 may suggest to the user training items included in the training item information 285 that impose a relatively small physical load or a relatively low level of difficulty. For example, for each training item of the training item information 285, information on the degree of difficulty (such as the lower the difficulty of a training that a person who is not accustomed to exercise can perform without problems) and the level of physical load (such as the different physical loads imposed by each training method targeting a specific part (such as the upper arm)) is held in advance. This allows a training menu to be presented based on the user's exercise experience.

[0320] According to the above example, by proposing a relatively easy training menu to a user who is not accustomed to exercise, the user can easily feel a sense of accomplishment by completing the training menu. This can provide the user with the motivation to continue training.

[0321] (2) Evaluate whether the user is accustomed to exercise based on the results of measuring the user's physical movements. In addition, the server 20 may evaluate the user's exercise experience based on the measurement results of the user's body movement by the measurement unit 30, and determine a training menu to present to the user based on the evaluation results.

[0322] For example, the server 20 holds in advance a pattern of the measurement results when the user is not accustomed to exercise and a pattern of the measurement results when the user is accustomed to exercise, in association with the measurement results of the user's body movements such as the range of motion of joints, walking posture, and trunk balance. The server 20 evaluates the degree to which the user is accustomed to exercise based on the measurement results of the user's body movements. For example, the server 20 compares the above patterns with the measurement results of the user's body movements and evaluates the degree of agreement, thereby outputting an evaluation value of whether the user is accustomed to exercise or not. For example, the server 20 presents the user with a model body movement, and evaluates that the user is accustomed to exercise by the user being able to move his or her body to the same extent as the model body movement. At this time, if there is a discrepancy between the model body movement and the measurement results of the user's body movement, a voice output may be output to the user to correct the discrepancy. For example, if the position of the hand is different from the model movement, a voice output may be output to encourage the user to move the hand position to follow the model movement (e.g., "move the hand position in a specified direction"). In addition, if the user's body movements are consistent with the model movements (i.e., if the user can perform movements equivalent to those of the model), an audio or other output will be provided to indicate that the user's body movements are appropriate.

[0323] For example, when measuring the user's body movement, the server 20 may output an evaluation value indicating whether the user is accustomed to exercise, based on whether the user can move his / her body in accordance with the video or audio guide presented to the user. For example, the server 20 may evaluate the user as accustomed to exercise when the user can move his / her body in accordance with the audio output.

[0324] The server 20 determines a training menu to be proposed to the user based on the evaluation value.

[0325] As described above, the server 20 may store the history of the measurement results by the measurement unit 30 in the measurement history database 283, and may display to the user the time when the user was evaluated as to whether or not he / she was accustomed to exercise, the evaluation result, and the history of the training menu proposed to the user. This allows the user to confirm that he / she is gradually becoming accustomed to exercise and that the training menu is changing, and thus realize a sense of self-efficacy.

[0326] (3) Ask users questions about continuing their training In addition, the server 20 may present the user with questions regarding continuing training. For example, questions may be presented regarding the user's personality, such as whether the user has been able to continue training in the past, and the period until the user stops training.

[0327] The server 20 determines the training menu to be presented to the user based on the answer to the question. For example, as described in the above-mentioned embodiment 4, etc., it is assumed that a problem that may occur in the user's body is identified, and a training menu related to the body part associated with the problem is presented to the user. If the user answers negatively about whether or not he / she can continue the training (e.g., he / she tends not to continue the training), the server 20 may change the training menu proposed to the user according to the problem. For example, if there are multiple training menus related to the body part associated with the problem, a different training menu is proposed to the user. This is expected to make it easier for the user to engage in the training without getting bored, and to continue the training.

[0328] The server 20 identifies the intervals at which the user performed training based on the measurement history database 283 and the like. That is, it identifies the frequency at which the user performs training. The server 20 may present the user with a question about continuing the training based on the length of the intervals. For example, when the frequency at which the user performs training is decreasing, the server 20 may propose a different training menu to the user so that the user does not get bored by accepting a question about continuing the training from the user. The server 20 may also change the training menu proposed to the user based on the length of the intervals. This allows the user to perform training without getting bored, and the user can continue to work on the training.

[0329] <Fifth embodiment> <Summary> In the fifth embodiment, a technique will be described that enables a user to more easily check whether or not an ideal movement is being performed by moving the body.

[0330] (1) By visually reducing the amount of information about the user's body movements, it becomes easier to check the user's body movements. Specifically, when a user performs a specific exercise (for example, an exercise for evaluating the movement of the trunk muscles and the range of motion of joints, such as lateral lifts, or an exercise for strength training), the user's body movements are sensed as point cloud data, and an avatar corresponding to the user and auxiliary lines corresponding to the user's body movements are drawn based on the point cloud data and presented to the user. This allows the user to more easily check whether each part of the body is moving as intended.

[0331] (2) Make it easier for trainers to instruct trainees online by using video conferencing tools, etc. In the fifth embodiment, based on the above technology, a situation will be described in which a trainee (first user) who receives instruction on how to move his / her body and a trainer (second user) who instructs the trainee on how to move his / her body provide instruction online.

[0332] Specifically, by allowing the trainer to check an avatar that represents the trainee's body movements as point cloud data on the trainer's terminal, the trainer can check not only an image of the trainee's body, but also the movements of an avatar with less visual information than the image. This makes it even easier for the trainer to give the trainee advice on how to move their body online.

[0333] That is, in the fifth embodiment, a series of techniques will be described for executing various processes, such as (1) a first processor of a terminal of a trainee (first user) sensing the body movements of the trainee (first user) as point cloud data using a measurement unit 30 or the like, and (2) the first processor transmitting information based on the point cloud data of the trainee (first user) (for example, an image of the trainee's avatar based on the point cloud data, or the point cloud data itself, etc.) to a terminal of a trainer (second user).

[0334] Here, the body movements of the trainee (first user) are sensed as point cloud data, and the first processor transmits information based on the point cloud data of the trainee (first user) to the terminal of the trainer (second user). Examples of devices that perform various processes include terminal devices such as a computer, smartphone, or tablet terminal owned by the user.

[0335] <Data Structure in the Fifth Embodiment> FIG. 26 is a diagram showing the data structures of a trainee information database 2601, a trainer information database 2602, and a training history database 2603 stored by the server 20 in the fifth embodiment. Although not shown in the figure, the server 20 also manages a database that associates information on the types of body movements performed by the user (movements for evaluating joint range of motion, such as lateral raises, movements for strength training, such as push-ups, pre-defined body movements, such as yoga, and other types of movements), information on the types of auxiliary lines to be used when displaying the user's body movements as an avatar that is a collection of point cloud data, and information on the positions at which to draw the auxiliary lines (positions associated with the parts of the user's body represented by the avatar).

[0336] The trainee information database 2601 is a database for managing information on users who will become trainees. The trainer information database 2602 is a database for managing information on users who will become trainers. The training history database 2603 is a database for managing the history of training of trainees under the guidance of trainers.

[0337] As shown in FIG. 26, the trainee information database 2601 includes the item "Trainee ID", the item "Name", the item "Age", the item "Gender", the item "Weight", the item "Body fat percentage", the item "Objective", the item "Ideal appearance", the item "Screen display", the item "Physical fitness level", the item "Notes", etc.

[0338] The item "Trainee ID" is information for identifying each user who receives training.

[0339] The item "Name" is information indicating the name of each user receiving the training.

[0340] The item "age" is information indicating the age of each user undergoing training.

[0341] The item "gender" is information indicating the gender of each user undergoing training.

[0342] The item "weight" is information indicating the weight of each user undergoing training.

[0343] The item "body fat percentage" is information indicating the body fat percentage of each user undergoing training.

[0344] The item "Objective" is information indicating the objective of each user undergoing training. Specifically, a user identified by the name "Mori" with a user ID "U001" has the objective of training to "get in shape," and a user identified by the name "Sato" with a user ID "U002" has the objective of training to "diet." Here, in a certain aspect, a user (trainer) in charge of instructing a user (trainee) undergoing training may present a training menu to the user (trainee) according to the objective. In another aspect, the terminal device 10 or the server 20 may specify a training menu to present to the user based on a trained model, which will be described later, or the like.

[0345] The item "ideal appearance" is information indicating the ideal appearance of each user undergoing training. Specifically, a user identified by the name "Mori" with user ID "U001" has set the ideal appearance as "not specified", and a user identified by the name "Sato" with user ID "U002" has specified "XX" as the ideal appearance. Here, the ideal appearance may be a real celebrity or a fictional entity (for example, if the ideal body shape is like a cartoon character, the user may specify that character). In a certain aspect, the server 20 may identify and present a training menu based on the contents of the "ideal appearance" and the "purpose".

[0346] The item "screen display" is information indicating the display mode when the user is photographed by a camera or the like. The display mode may be to display the captured image generated by photographing the user as it is when the user is photographed by an RGB camera or the like, or to measure the user's body movements as point cloud data using a sensor capable of measuring distance (depth camera, LiDAR (light detection and ranging), etc.) and display the user as an avatar based on the point cloud data. Specifically, the user identified by the name "Mori" with user ID "U001" has set the screen display to "avatar", which indicates that when a video image of the user photographed by a camera or the like is displayed on the screen, the video image of the user is not displayed as it is, but is displayed as some kind of avatar (for example, an avatar using a character) based on the captured video image.

[0347] This allows the user to receive training with peace of mind, as it can meet their needs, for example, if it is their first training and they do not want others to see their face or body, or if they want to protect their privacy.

[0348] The item "physical fitness level" is information indicating a physical fitness level that evaluates the physical athletic ability of each user undergoing training in stages. Specifically, when sensing the physical movement of the user undergoing training, a physical fitness level is specified based on the information on the user's physical movement at the same time, which is a criterion for judging how much exercise the user is likely to be able to perform. A user identified by the name "Mori" with user ID "U001" has a physical fitness level of "5", and a user identified by the name "Sato" with user ID "U002" has a physical fitness level of "2". In this case, as a method of specifying the physical fitness level, the terminal device 10 or the server 20 may specify the physical fitness level of the user based on a learned model or the like when sensing the user's physical movement, or a user (trainer) who is in a position to instruct the user undergoing training (trainee) may specify the physical fitness level of the trainee based on the results of sensing.

[0349] This allows users to receive a training menu suited to their own physical fitness level, allowing them to continue training without strain. In addition, in the case of group training, the server 20 can identify users with the same physical fitness level and match users (trainers, trainees) so that the identified users train together. For example, when users (trainees) with different physical fitness levels train together, there may be differences in the content of the trainer's instruction and the content of the trainee's training, which can lower motivation, but this can be prevented as described above.

[0350] The item "remarks" is information indicating circumstances that should be specially considered for each user receiving training. Circumstances that should be specially considered include matters that should be considered based on safety when training (safety from a medical perspective, safety of the training content itself, etc.). For example, if a user has been advised by a doctor not to train excessively due to high blood pressure, such matters are written in the remarks. The server 20 may specify an optimal training menu for the user receiving training based on the contents written in the item "remarks". In addition, the trainer's terminal device 10B may obtain information in the item "remarks" from the server 20 and present it to the trainee, thereby providing information that the trainer can use as a reference when deciding on a training menu for the trainee.

[0351] As a result, even if the user has a chronic illness, the server 20 or the like can identify the training menu to present to the user based on the "Notes" item, or the trainer can select a training menu based on the "Notes" item, allowing the user to continue training without straining themselves.

[0352] Trainer information database 2602 includes items such as "Trainer ID," "Name," "Gender," "Years of experience," "Area of ​​expertise," "Recommended physical fitness level," "Type of training available," and "Notes."

[0353] The item "Trainer ID" is information for identifying each user who is in a position to train the user receiving training.

[0354] The item "Name" is information indicating the name of each user who is in a position to instruct the user receiving the training.

[0355] The item "gender" is information indicating the gender of each user who is in a position to instruct the user receiving the training.

[0356] The item "years of experience" is information indicating the number of years of experience in training for each user who is in a position to instruct users receiving training. Specifically, the user identified by the name "Tanaka" with trainer ID "T001" indicates that he has "5 years" of experience, and the user identified by the name "Kato" with trainer ID "T003" indicates that he has "10 years" of experience.

[0357] As a result, for example, in a certain situation, when a user (trainee) receiving training designates a user (trainer) who will provide instruction, the user can receive training with more peace of mind by designating the user with reference to the years of experience. The user (trainer) who will provide instruction can also improve his / her own recognition by being designated as a user who will provide instruction based on the information.

[0358] The item "area of ​​expertise" is information indicating the area of ​​expertise in training instruction of each user (trainer) who is in a position to instruct users (trainees) receiving training. Specifically, it indicates that the user identified by the name "Tanaka" with trainer ID "T001" is good at "yoga" and "exercise", and the user identified by the name "Saito" with trainer ID "T002" is good at "muscle training" and "exercise".

[0359] In this way, the server 20 or the like may specify a user (trainer) who is in a position to instruct the user based on information such as the items "Objective" and "Ideal Appearance" in the trainee information database 2601 of the user receiving the training. This makes it possible to match a trainer who matches the objective of the trainee receiving the training. For example, if the trainee aims to exercise, the server 20 or the like may assign a trainer who is good at training that matches the objective as the instructor.

[0360] The item "recommended physical fitness level" indicates the physical fitness level of the trainee that is recommended for training by the user (trainer), which is defined for each user (trainer) who is in a position to instruct the user (trainee) receiving training. Specifically, the recommended physical fitness level for training from the user (trainer) identified by the name "Tanaka" with trainer ID "T001" is "3 or more", and since the user (trainee) identified by the name "Mori" with the above-mentioned user ID "U001" has a physical fitness level of "5", it is understood that the user has reached the physical fitness level required to receive instruction from the user, while the user (trainee) identified by the name "Sato" with user ID "U002" has a physical fitness level of "2", it is understood that the user has not reached the physical fitness level required to receive instruction from the user.

[0361] This allows the user undergoing training to receive training suited to his / her own physical fitness level. Also, since the user instructing the user undergoing training can provide training to users who are at or above a certain level, it is possible to reduce variations in training menus and progress among users, and it is expected that the motivation of the user undergoing training will be maintained. The server 20 etc. may match trainers and trainees based on the trainer's item "recommended physical fitness level" and the trainee's item "physical fitness level".

[0362] The item "Instructable Type" indicates the type of training that a user (trainer) who is in a position to instruct a user who is receiving training can instruct. Specifically, the user identified by the name "Tanaka" with trainer ID "T001" can instruct types "individual" and "group", while the user identified by the name "Saito" with trainer ID "T002" can only instruct type "individual".

[0363] This allows the server 20, etc., to know whether the user (trainer) who will be instructing the user receiving the training is suited to individual instruction or group instruction, and therefore, when matching the trainer and the trainee, the server 20, etc., can assign one-on-one individual instruction training to one-on-multiple group instruction training based on the "instruction type." In addition, the "instruction type" of the trainer may be presented to the trainee on the trainee's terminal device 10.

[0364] The "Notes" item is information such as other special notes for each user (trainer) who is in a position to instruct the user receiving the training. Special notes may include information that objectively evaluates the training results by the trainer. Specifically, it can be seen that the user identified by the name "Tanaka" with trainer ID "T001" is a "** Grand Prix winner."

[0365] As a result, in a certain situation, when a user (trainee) receiving training designates a user (trainer) who is in a position to instruct the user receiving training, the trainee can designate a trainer taking the information into consideration by presenting the information in the "Notes" item to the trainee on the trainee's terminal device 10. By entering the information, users (trainers) who are in a position to instruct users receiving training can also expect to have more opportunities to be designated as instructors.

[0366] The training history database 2603 includes an item "Training ID", an item "Date and time", an item "Meeting ID", an item "Trainee ID", an item "Trainer ID", an item "Sensing results", an item "Instruction content", an item "Trainer evaluation", an item "Matching ID", etc.

[0367] The item "Training ID" is information for identifying each of the training sessions by the trainer and the trainee.

[0368] The item "date and time" indicates the date and time when the training was performed.

[0369] The item "Meeting ID" is information for identifying each communication session when the trainer's terminal device 10B and the trainee's terminal device 10A communicate with each other to provide online training instruction. For example, the information for identifying the communication session includes information such as a URL for an online meeting.

[0370] The item "Trainee ID" is information for identifying the trainer who performed the training.

[0371] The item "Trainer ID" is information for identifying a trainee who has received training guidance from a trainer.

[0372] The item "Sensing results" is the results (point cloud data) of the trainer's body movements during training measured by the terminal device 10A using the measurement unit 30, or an image in which the point cloud data is placed in a virtual space and rendered based on the settings of the virtual camera (including an avatar that represents the trainee's body movements as a collection of point cloud data, and auxiliary lines drawn according to the trainee's body movements (the type of body movements presented to the user, such as lateral raises)).

[0373] The item "instruction content" indicates the instruction content that the trainer instructs the trainee on how to move his / her body. For example, when the trainer instructs the trainee on how to move his / her body by voice (including advice on how to move the body to a desired posture, for example, advice from the trainer to encourage a body part (arm, leg, etc.) to be in the desired posture when it is bent), the voice including the instruction content or the analysis result of the voice is stored in the item "instruction content".

[0374] The item "Trainer Evaluation" indicates an evaluation of the trainee by the trainee. For example, after the online training, the trainee's terminal device 10A presents the trainee with a screen for inputting the trainer's evaluation, thereby accepting the trainer's evaluation from the trainee.

[0375] The item "matching ID" is information that identifies each match when the server 20 matches a trainer and a trainee. When a match is made, training between the trainer and the trainee may start, but training may not be conducted even if a match is made.

[0376] <Example of operation in the fifth embodiment> Below, in the fifth embodiment, a series of processes will be described in which a first processor of a terminal of a first user (trainee) in system 1 senses the body movements of the first user as point cloud data, and the first processor transmits information based on the point cloud data of the first user to a terminal of a second user (trainer) and executes various processes.

[0377] (Sense the first user's movements and send them to the second user's device) FIG. 27 is a flowchart showing a series of processes for sensing the body movement of a first user as point cloud data and transmitting information based on the point cloud data of the first user obtained by sensing to a terminal of a second user.

[0378] 1, the terminal device 10 of the first user will be referred to as "terminal device 10A" and the configuration of the terminal device 10A will be referred to as "processor 19A" etc. Similarly, the terminal device 10 of the second user will be referred to as "terminal device 10B" and the configuration of the terminal device 10B will be referred to as "processor 19B" etc.

[0379] In the example of FIG. 27, a terminal device 10A of a first user and a terminal device 10B of a second user establish communication, thereby transmitting and receiving images captured by the camera 160 of each terminal and audio picked up by the microphone 141 to each other.

[0380] For example, terminal device 10A and terminal device 10B are executing an application for conducting an online video meeting, and establish communication based on identification information for participating in the meeting (e.g., a meeting ID, URL, etc. issued by the application).

[0381] The terminal device 10A acquires the body movement of the first user as point cloud data by the measurement unit 30 or the like. The terminal device 10A arranges the point cloud data in a virtual space, arranges virtual cameras in the virtual space at a position in front of the body of the first user (front-back axis in the description of FIG. 11) and at a position on the side of the body of the first user (left-right axis in the description of FIG. 11), respectively, performs rendering, and displays the rendering result. The rendering result includes an avatar as a collection of point cloud data of the first user viewed from the front, and an avatar as a collection of point cloud data of the first user viewed from the side. The terminal device 10A draws auxiliary lines corresponding to the user's body movement in association with an area displaying the above rendering results (the avatar viewed from the front, the avatar viewed from the side).

[0382] The terminal device 10A displays the image with the avatar and auxiliary lines on the screen, and shares the image with the terminal device 10B by an application for performing an online video meeting. As a result, the video of the first user captured by the camera 160A, the audio collected by the microphone 141, and the image with the avatar and auxiliary lines are transmitted from the terminal device 10A to the terminal device 10B. The second user of the terminal device 10B visually checks the video captured by the camera 160A and the image (the image with the avatar and auxiliary lines) being shared on the screen, thereby confirming the body movement of the first user and notifying the first user of advice to the first user by audio or the like.

[0383] In step S2701, the processor 19A of the terminal device 10A of the first user senses the body movement of the first user as point cloud data using the measurement unit 30 or the like.

[0384] Specifically, the processor 19A displays to the first user a menu of body movements to be performed by the first user, and prompts the first user to perform a specific movement for sensing the body movement (for example, a movement such as a lateral lift, a movement that is a training item (squats, etc.)). The processor 19A of the terminal device 10A of the first user captures the specific movement using the measurement unit 30 or the like, thereby sensing the body movement of the first user as point cloud data.

[0385] In step S2703, the processor 19A generates an image including an avatar representing the first user (trainee) and auxiliary lines corresponding to the type of body movement performed by the first user (type of exercise menu presented to the first user) as information based on point cloud data of the first user obtained by sensing by the measurement unit 30 or the like.

[0386] In step S2705, the processor 19A displays the image generated in step S2703 on the display 132A of the terminal device 10A. The processor 19A transmits the displayed image (image including the avatar and auxiliary lines) and a captured image (not a collection of point cloud data, but an image captured by an RGB camera) generated by the camera 160A capturing the first user separately to the terminal device 10B of the second user.

[0387] In step S2751, the processor 19B of the terminal device 10B of the second user receives, from the terminal device 10A of the first user, a video (including an avatar and auxiliary lines) generated by the terminal device 10A as information based on the point cloud data of the first user. The processor 19B receives the captured image of the first user (a captured image that is not a collection of point cloud data) generated in step S2705.

[0388] In step S2753, processor 19B displays an image including a first avatar corresponding to the body movement of the first user as a set of point cloud data on display 132B, etc., based on the image (including the avatar and auxiliary lines) received in step S2751 and the captured image. Processor 19B displays the captured image of the first user (not a set of point cloud data) on display 132B, etc. This presents the first avatar and the captured image of the first user to the second user as information on the body movement of the first user.

[0389] Here, in step S2703, the processor 19A may specify various information regarding the exercise of the first user (e.g., which part of the body is easy or difficult to move), a physical fitness level, and the like, based on the trainee information database 2601 of the server 20. At this time, the processor 19A may store the specified information in the storage unit 180 of the terminal device 10A of the first user, or may transmit the information to the server 20 and store it in the trainee information database 2601 of the storage unit 202 of the server 20.

[0390] In addition, the processor 19A may refer to the trainer information database 2602 based on the results of sensing the body movements of the first user by the measurement unit 30 or the like, and identify the second user (trainer) to which the information is to be transmitted.

[0391] This allows the user, who is a trainee, to receive training from an optimal instructor based on the information on the body movements at the start of training.

[0392] Here, in step S2751, data sensed by the measurement unit 30 or the like may be received from the terminal device 10A of the first user, and rendering may be performed based on the point cloud data in the terminal device 10B based on the received information. That is, the first avatar representing the body movement of the first user may be rendered in the terminal device 10B, not in the terminal device 10A, based on the point cloud data measured by the terminal device 10A. The terminal device 10B may generate an image including the first avatar, and display the generated image on the display 132B or the like. At this time, the body movement of the first user may be drawn from a plurality of viewpoints, and an image in which the first avatars drawn for each viewpoint are arranged may be presented to the second user.

[0393] In one aspect, terminal device 10B may draw an auxiliary line at a position based on a body part for evaluating whether the body movement performed by the first user is ideal or not, and present the image with the auxiliary line drawn to the second user. For example, when a trainer evaluates whether the body movement of a user is ideal or not based on whether the back is straightened and the knees are at a predetermined angle with respect to the ground in an exercise to be performed by the user, terminal device 10B displays an auxiliary line corresponding to the shape when the back is straightened near the back part of the avatar. When terminal device 10B evaluates whether the knees are at a predetermined angle with respect to the ground, it displays an auxiliary line at the predetermined angle near the knee part of the avatar (or in a region displaying the knee part).

[0394] In addition, the display manner of the body surface positions other than the joint positions may be changed based on the body parts for evaluating whether the body movements performed by the first user are ideal or not, and the image with the changed display manner of the body surface may be presented to the second user.

[0395] The surface of the body that is not a joint position is, for example, a surface that can be recognized as the external appearance of the body, such as the surface of the abdomen. For example, in the terminal device 10B, the color displayed may be changed depending on whether the body movement of the first user is too large or too small compared to the ideal movement. For example, when the abdomen is protruding too far forward compared to the ideal movement, the abdomen is displayed in a warm color, and when the abdominal movement is smaller than the ideal movement, the abdomen is displayed in a cool color, and other colors are changed. This allows the first user or the second user to easily confirm how to move the user's body to achieve the ideal movement. This also allows the user to train while understanding how much the movement of his or her body is different from the ideal movement. In addition, by changing the display mode of the body surface, it is possible to understand the movement of muscles on the body surface that are not joints.

[0396] In the above example, with regard to the video including the first avatar of the first user displayed on the display 132B, etc., both the image of the first user taken when generating the first avatar and the video of the first avatar have been described as being presented to the second user. In addition, the trainer's terminal device 10B may refer to the trainee information database 2601 and display a screen according to the specification of the item "screen display" on the display 132, etc.

[0397] For example, in the terminal device 10B, the body movement of the first user may be displayed as a first avatar (i.e., a collection of point cloud data), while the captured image of the first user (not a collection of point cloud data) may be hidden. In addition, when a plurality of trainees are trained by a trainer, the captured image of the trainee (not a collection of point cloud data) may be made displayable on the terminal device 10 of the trainer, while being hidden on the terminal devices 10 of the other trainees who are simultaneously training with the trainer. In this way, the trainee's sensing results, both the point cloud data and the captured image, may be disclosed to the trainer, while the avatar (a collection of point cloud data) may be shared between the trainees, but the captured image may not be shared.

[0398] This allows users to train while protecting their privacy.

[0399] In addition, in one aspect, at least one of processor 19A, processor 19B, or processor 29 of server 20 may define a virtual space and place in the virtual space a first object for displaying an image including a first avatar associated with a first user and a second object for displaying an image including a second avatar associated with a third user who trains together with the first user, and processor 19A or processor 19B may display the image including the first object and the second object in the virtual space on display 132A or display 132B, etc.

[0400] This allows users to experience training in the same virtual space when training with others, giving them a sense of unity and accomplishment in the training, which is expected to help maintain and increase their motivation.

[0401] Next, with reference to FIG. 28, a process flow when a trainer gives online instruction to a trainee will be described.

[0402] In the following, an example will be described in which the trainer gives voice guidance to the trainee while controlling an application running on the trainee's terminal device 10 with the trainer's voice, thereby enabling the trainer to easily obtain information suitable for guidance. Specifically, the trainer can control, by voice input, the drawing of an avatar based on point cloud data, the drawing of auxiliary lines, the determination of the type of body movement to be imposed on the trainee, and the like on the trainee's terminal.

[0403] FIG. 28 is a flowchart showing the flow of a process for accepting voice input from the trainer's terminal and transmitting it to the trainee's terminal.

[0404] In step S2851, the processor 19B of the terminal device 10B of the second user accepts voice input by the second user via the microphone 141B or the like, and transmits the voice of the second user from the terminal device 10B of the second user to the terminal device 10A of the first user.

[0405] In step S2801, the processor 19A of the terminal device 10A of the first user outputs the voice of the second user received from the terminal device 10B to the first user via the speaker 142A or the like.

[0406] In step S2803, the processor 19A analyzes the voice of the second user received from the terminal device 10B by voice recognition. The processor 19A identifies the content of the voice of the second user based on the result of the voice recognition analysis. The processor 19A identifies the content of instructions to an application that draws an avatar based on the point cloud data of the first user based on the identified content of the voice of the second user. The processor 19A controls the application based on the identified content of instructions to the application.

[0407] Specifically, the contents of the voice of the second user identified by the processor 19A are as follows: (i) There is content for making the first user's body movements ideal. Processor 19A recognizes, for example, by voice recognition, words related to the user's body parts (knees, elbows, arms, shoulders, etc.) from the voice of the second user. Processor 19A also recognizes words from the voice of the second user that indicate how to move the first user's body parts to make them move ideally (for example, "bring your knees closer to your body," "keep your legs horizontal," "straighten your back," etc.).

[0408] (ii) The content of the voice identified by processor 19A includes content indicating the type of exercise (such as a training menu) that the first user is to undertake for the first user's training.

[0409] When the content of the voice of the second user includes a type of exercise, the processor 19A sets the type of exercise to be imposed on the first user to the exercise designated by the second user through voice, and sets the auxiliary line to be drawn in association with the avatar corresponding to the first user to correspond to the designated type of exercise. This makes it easier for the second user to evaluate the body movement of the first user.

[0410] (iii) The contents of the voice identified by the processor 19A include contents that cause an application that runs on the terminal device 10A and draws the avatar and auxiliary lines of the first user to perform a specific process. For example, commands indicated by the voice of the second user in the application include the following: Avatar rendering mode (including the position at which the virtual camera is placed relative to the point cloud data representing the body movements of the first user, i.e., whether the avatar corresponding to the first user is rendered from the front, side, or other direction, etc.) How to draw auxiliary lines (including whether to draw auxiliary lines parallel to the user's body movements, whether to draw auxiliary lines along the shape of the user's body, whether to draw auxiliary lines at all, etc.) How to render your avatar when there are deviations from ideal body movements (including rendering specific body parts in different colors when their movements are greater or less than ideal body movements); Instructions to suspend the application, and other content

[0411] When the processor 19A recognizes the speaker as a trainer (for example, the trainer is registered in the trainer information database 2602) by voice recognition, the processor 19A may control an application that draws an avatar of the first user based on the voice of the speaker. This can prevent the application from being unintentionally controlled by a user other than the trainer. In addition, when the processor 19A communicates with another terminal device 10 such as the terminal device 10B by a web conference tool or the like, the processor 19A may control an application that draws an avatar of the first user based on information of the speaker identified by the web conference tool or the like. For example, in a web conference tool or the like, information (icons, etc.) that identifies each user participating in a meeting may be displayed, and the information that identifies each user may be displayed separately for a user who is speaking. When an icon, etc. corresponding to the trainer is displayed separately from the icons, etc. of other users in the web conference tool or the like, the processor 19A may perform the above process by assuming that the speaker is the trainer.

[0412] For example, when multiple trainees receive instruction from a trainer, depending on the trainees' voices, the application can be made to not perform processing related to the training (processing to change the type of exercise to be performed by the trainees, processing to change the auxiliary lines drawn together with the avatar in accordance with the change, etc.), but to have the trainer perform the processing. This allows the training to be performed as intended by the trainer, even if each user speaks by voice during training.

[0413] As described above, as a result of the second user providing audio instruction to the first user, the display manner of the video including the first avatar is changed, and the second user can continue to provide instruction to the first user based on the video including the changed avatar.

[0414] (1) Sensing the user's movements and determining the instruction content based on the trained model) 29 is a flowchart showing a series of processes for sensing the body movement of a first user as point cloud data, and specifying the instruction content for the first user based on the information based on the point cloud data of the first user obtained by sensing and the trained model. This allows the trainee to check how to move his / her own body using the trained model even without a trainer.

[0415] In step S2901, the processor 19A of the terminal device 10A of the first user senses the body movement of the first user as point cloud data using the measurement unit 30 or the like.

[0416] In step S2902, the processor 19A identifies instruction content for the first user based on information based on the point cloud data of the first user obtained by sensing and the trained model.

[0417] Here, we will provide an example of how to create a trained model. (i) For example, after obtaining information on point cloud data of one or more users, a trained model is created by associating the point cloud data information of one or more users with the training content based on various information (e.g., age, weight, purpose, etc.), information on the user (trainer) in charge of instructing the user receiving the training, the content of the training actually received by the trainee, and information such as a history of changes in various trainee information due to training.

[0418] In other words, the trained model learns the trainee's information and the training content by the trainer as time-series data, and outputs how much the trainee changes as a result of receiving training.

[0419] The trained model may be associated with information of a user who has acquired information based on point cloud data, which is training data for the trained model (i.e., information of the trainee), or information of a trainer who has instructed the trainee. This allows the trainee or trainer to refer to the information of the trainee or the trainer associated with the trained model when specifying a trained model to be used by the trainee. Alternatively, the processor 19A or the like may identify a trained model to be proposed to a user who is a trainee, based on information of the user associated with the trained model.

[0420] (ii) Alternatively, the trained model may generate time series data as learning data by having the trainer tag the time series data (point cloud data, photographed images of the trainee) obtained by measuring the trainee's body movements (e.g., comments on how to move body parts in an ideal way, the trainee should take a breath, or that the trainee's heart rate is too high and the trainee should take a break).

[0421] For example, in a certain situation, based on the measurement results of the trainee's physical movements and the learned model, a training menu suited to the trainee user's physical constitution can be presented, and the timing of when to breathe during training, when to increase the heart rate, when to take fluids, when to take a break, etc. can be identified and presented to the user.

[0422] For example, in a certain situation, a trainee may wear a wearable device (wristwatch-type, eyeglass-type, ring-type, etc.) and measure the trainee user's heart rate, breathing timing, etc., while using the above-mentioned trained model, the trained model may output the content to be taught to the trainee.

[0423] In another aspect, the user's exercise intensity (how intense the exercise is), including the amount of calories burned, can be identified from information regarding the point cloud data obtained from the user, and advice such as taking a break and drinking water can be provided to the user. (iii) As a method for generating a trained model, a trained model may be generated using the measurement data (measurement history database 283) accumulated as a result of the ROM check as teacher data as described in the first embodiment. For example, a trained model may be generated based on the measurement data, a training menu performed by the user, and the results of responses to a questionnaire or the like entered by the user. As a result, the type of exercise suggested to the user, the content of advice to the user, etc. may be output from a trained model generated based on the measurement data of the ROM check.

[0424] As a method for generating a trained model, as described in the second embodiment, a trained model may be generated using the measurement data (measurement result database 287) accumulated as a result of evaluating the walking posture of the user as teacher data. For example, a trained model may be generated based on the measurement data, a training menu performed by the user, and answers to a questionnaire or the like entered by the user. As a result, the type of exercise suggested to the user, the content of advice to the user, etc. may be output by a trained model generated based on measurement data for walking posture evaluation.

[0425] For example, a method for generating a trained model may be to generate the trained model based on measurement data accumulated as a result of estimating the user's posture (joint positions, etc.). Specifically, a method for generating a trained model in the server 20, etc. may be to generate a trained model based on the result of estimating the user's joint positions, the training menu performed by the user, and the contents of advice from a trainer, and store the generated trained model.

[0426] As described above, the server 20 may identify instruction contents for the user based on information based on the estimation result of the user's posture and the trained model, and present the identified instruction contents to the user.

[0427] In addition, the server 20 etc. may store the trained model in association with information about the user who acquired the training data for the trained model (the result of estimating the user's posture).

[0428] The server 20 may also specify the exercise intensity of the user based on the information based on the estimation result of the user's posture and the trained model in the same manner as described above. The server 20 may specify the instruction content for the user based on the result of specifying the exercise intensity of the user, and present the specified instruction content to the user.

[0429] In step S2903, the processor 19A presents the instruction content identified by the trained model to the first user, for example by displaying it on the display 132A.

[0430] <Example of a screen in the fifth embodiment> 30 to 32 are diagrams showing examples of screens displayed on the terminal device 10. FIG.

[0431] FIG. 30 is an example screen displayed when the body movements of a first user are sensed as point cloud data on a terminal device 10A of a first user, and information based on the point cloud data of the first user obtained by sensing is transmitted to a terminal of a second user.

[0432] In FIG. 30, the camera 2901 of the measurement unit 30 represents a device for sensing the body movements of the first user.

[0433] Specifically, the camera 2901 captures an image of the first user through a capturing unit provided in the camera 2901. At this time, the camera 2901 is configured to be able to measure distance and direction, and is configured using, for example, an RGB-D camera and LIDAR (Laser Imaging Detection and Ranging) technology.

[0434] The previous screen 2902 is a screen that displays, when the first user performs training, an image including an avatar of the first user when the first user previously performed the same type of training (i.e., an image drawn based on point cloud data). The terminal device 10A, for example, refers to the training history database 2603 of the server 20 to acquire point cloud data indicating the body movements of the first user when the first user previously performed training.

[0435] In the illustrated example, the avatar of the first user is shown on the previous screen 2902 as a collection of figures such as circles, but it may be displayed as a collection of point cloud data measured by the measurement unit 30.

[0436] By displaying an avatar based on point cloud data of the first user's past training, the first user and the second user can easily check how much the first user's body movement has changed due to training. For example, when the first user hugs his / her knees and bends over, a previous screen 2902, which is a previous training screen, is displayed on the terminal device 10A. The first user or the second user can compare the previous screen 2902 with the display content of a current screen 2903, which will be described later. This allows the first user to check the difference in his / her own body movement.

[0437] Furthermore, the terminal device 10A may always display the previous screen 2902, or may display or turn off the display according to an instruction from the first user or the second user. At this time, the terminal device 10A may display an image of the first user drawn from a plurality of viewpoints, such as the front, side, and top, as an avatar representing the first user. Also, auxiliary lines may be drawn at positions based on body parts for evaluating whether the body movement performed by the first user is ideal or not. In the illustrated example, auxiliary lines are drawn in the past screen 2902 in association with an area displaying the avatar of the first user.

[0438] The current screen 2903 is a screen that displays an image of the avatar of the first user based on the sensing result of the measurement unit 30 while the first user is performing training.

[0439] In the illustrated example, a first user 2094 is shown, and a state in which the first user is actually training on a mat or the like is shown. The terminal device 10A captures the movement of the first user with a camera 2901, and acquires information on the point cloud data of the first user.

[0440] The first user image 2905 shows a captured image of the first user captured by the camera 160 or the like. In the first user image 2905, the first user is not depicted as a collection of point cloud data, but an image captured by a 2D camera or the like is displayed. That is, in the first user image 2905, a captured image equivalent to what a human can see with the naked eye is displayed.

[0441] Thereby, the terminal device 10A allows the first user to compare an actual image of the first user himself with an image based on the information of the point cloud data. In addition, the terminal device 10 may display an icon, character string, or the like that allows a second user, etc., to identify the first user as the first user, while not displaying an image captured by the first user in the first user image 2905, in response to a designation by the first user.

[0442] The second user image 2906 shows an image of a second user who is in a position to instruct the user undergoing training. The terminal device 10A displays the captured image of the second user in the second user image 2906 by receiving the captured image of the second user from the terminal device 10B of the second user. This allows the first user and the second user to train while viewing each other's images and communicating with each other through a video call or the like between the terminals via the network 80 or the like.

[0443] Figure 31 shows example screens of a first user and a second user, including the overall configuration, when the body movements of a first user are sensed as point cloud data and information based on the point cloud data of the first user obtained by sensing is transmitted to a terminal of a second user.

[0444] In FIG. 31, a camera 3001, like the camera 2901 in FIG. 30, indicates a device for sensing the body movement of a first user as point cloud data.

[0445] Current images 3002 and 3006 are screens that display an image of the first user as a collection of point cloud data when the first user is training, similar to the current screen 2903 in Fig. 30. At this time, the current image 3002 is displayed on the display 132A or the like of the terminal device 10A of the first user, and the current image 3006 is displayed on the display 132B or the like of the terminal device 10B of the second user.

[0446] First user 3003 is a diagram showing a state in which a first user is actually performing training on a mat or the like, similar to first user 2904 in FIG.

[0447] 30, the first user images 3004 and 3007 show actual captured images, not images including an avatar, of the first user captured by the camera 2901. At this time, the first user image 3004 is displayed on the display 132A or the like of the terminal device 10A of the first user, and the first user image 3007 is displayed on the display 132B or the like of the terminal device 10B of the second user.

[0448] 30, the second user images 3005 and 3008 show images of a second user who is in a position to instruct a user undergoing training. At this time, the second user image 3005 is displayed on the display 132A or the like of the terminal device 10A of the first user, and the second user image 3008 is displayed on the display 132B or the like of the terminal device 10B of the second user. In addition, in a certain aspect, as in the screen example, the voice of the second user (such as "Sit down more!") may be output through the speaker 142A or the like of the terminal device 10A in response to a voice input operation from the second user.

[0449] The second user 3009 visually checks the screen displayed by the terminal device 10B on the display 132B or the like, and gives guidance based on the image of the first user. At this time, the second user can give guidance to the first user by voice. The second user can give advice on the body movement of the first user while checking both or either of an image based on point cloud data and an image equivalent to that seen with the naked eye as an image of the first user captured. By reducing the amount of information for the second user as a collection of point cloud data compared to the image of the first user shown by the actual captured image, it becomes easier for the second user to determine the body movement of the first user, and easier to evaluate and give advice.

[0450] FIG. 32 shows an example of a screen when the body movements of multiple users are sensed as point cloud data, and information based on the point cloud data of each user obtained by sensing is transmitted to a defined virtual space.

[0451] In FIG. 32, a virtual space 3101 shows a space in which one or more users undergoing training can place objects for displaying an image including an avatar associated with the user.

[0452] A trainer image 3102 shows an image of a second user who is in a position to instruct the user undergoing training. The terminal device 10B of the second user photographs the second user and transmits the photographed image to a computer that manages the virtual space as needed.

[0453] The user image 3103 is a screen that displays an image of each user when one or more users are training. At this time, the image displayed may be only an image including an avatar as a collection of point cloud data associated with the user, or an actual captured image of each user, not including an avatar, may be displayed together.

[0454] This allows each user to place an image including an avatar corresponding to his / her body movement in the virtual space, and each user can visually recognize the image of the avatar of each user placed in the virtual space, so that it is possible to obtain a sense of as if multiple users are training in the same place. This allows each user to obtain a sense of unity and accomplishment in the training, and provides each user with a motivation to continue training. In response to matching each trainee who receives training from the trainer, the server 20 may enable communication between the terminal of each trainee and the terminal of the trainer, and may construct a virtual space in either the server 20 or the terminal of each user. In other words, the server 20 may specify a computer to be a host when each user communicates.

[0455] <6 Variations> A modified example of this embodiment will be described below. That is, the following aspects may be adopted.

[0456] (1) In the example of Figures 31 and 32 above, an avatar based on the sensing results of a user when the user received training in the past and an avatar based on the current sensing results when the user is receiving training from a trainer are displayed side by side.

[0457] In addition, avatars corresponding to the trainees may be displayed side by side based on time-series data of point cloud data of the trainees sensed in time series during training with the trainer.

[0458] In other words, an avatar based on point cloud data sensed at the time training begins or within a certain time after training begins, and an avatar based on the results of real-time sensing based on the trainer's instruction or sensing results at a certain time after the trainer's instruction begins, may be displayed side by side on terminal device 10A.

[0459] In addition, during the course of one training session, a part of the trainee's body movement that is particularly close to the ideal body movement may be highlighted and reflected in the avatar in comparison with other parts. For example, the processor 19A may evaluate how close the trainee's body movement is to the above-mentioned auxiliary line drawn together with the avatar, and draw the screen according to the evaluation result. For example, when extending the knee in a specific direction is an ideal movement, the processor 19A may draw an auxiliary line along the specific direction. The processor 19A may change the drawing mode of the avatar's knee part or the auxiliary line associated with the knee part as the knee movement approaches a direction along the auxiliary line in the trainee's body movement.

[0460] This allows the trainee to easily check how much they have changed before and after a single training session, or during the training process, making it easier for the trainee to check how much they have changed (e.g., how close they are to reaching their goal), which further increases the trainee's motivation to continue training.

[0461] When training online, visual information is reduced compared to face-to-face training (when training face-to-face, it can be seen with the naked eye, whereas when training online, it is seen as a two-dimensional display image), and there is a risk that the quality of the communication experience will be reduced. In addition, when training online, it is very difficult for the trainer to physically touch the trainee. Therefore, by providing the trainee with feedback on the training results as described above, it is possible to motivate the trainee to continue training online.

[0462] As described above, if the user's body movements change to ideal movements after one training session, a trained model may be generated based on the sensing results.

[0463] (2) In the above embodiment, an example has been described in which an avatar is displayed based on the results of sensing the user's body movements, and an avatar corresponding to the trainer is displayed to the trainee in online training between the trainer and the trainee.

[0464] In addition, the processor 19A may train the user while checking the way of moving his / her body as the above-mentioned avatar, without relying on communication between the trainer and the trainee. For example, the user may set the way of moving his / her body in the terminal device 10A, and while drawing auxiliary lines or the like according to the set way of moving his / her body, notify the user of information on the way of moving his / her body (guide the user on how to move his / her body), and sense the way of moving the user's body with the measurement unit 30 or the like.

[0465] (2-1) Method for sensing user's body movements The terminal device 10A presents the user with the results of sensing how the user moves his / her body as an avatar, etc. Also, as described in the first and second embodiments, etc., the terminal device 10A can identify the positions of the joints of the user's body and estimate the skeleton (bones) based on the results of sensing the user's body (FIG. 12).

[0466] The terminal device 10A may estimate the posture of the user based on the estimation results of the positions of each joint and the skeleton of the user as follows. Based on the shooting result (two-dimensional moving image) of the user as a subject, the terminal device 10A estimates the posture of the user shown in the moving image without using point cloud data. That is, the terminal device 10A may photograph the user using a camera that does not have a configuration for optically measuring the distance to the subject, such as LiDAR, and estimate the position of the joints, bone structure, etc. as the user's posture in the captured image based on the photographing results. For example, by preparing a trained model that takes a captured image as input and outputs the positions of the joints, the posture of the skeleton, etc. of the person who is the subject of the captured image, the terminal device 10A can estimate the posture of the user based on the user's photographing results and the trained model. Such a trained model can be generated by preparing learning data in advance that associates information such as the positions of joints and skeletal structure in a captured image with the person who is the subject of the captured image, and having a computer train the data. As described in the other embodiments, the terminal device 10A senses the user's body movements as point cloud data based on a camera or a measurement unit 30 mounted on the terminal device 10A, etc., and estimates the positions of the joints, the skeleton, etc. based on the sensing results.

[0467] (2-2) Drawing auxiliary lines based on joint positions and skeletal structure estimation results The terminal device 10A may draw an auxiliary line that serves as a guide for the user when moving the body based on the estimation result of the user's posture. The terminal device 10A may draw the auxiliary line based on the estimation result of the user's posture as follows. The terminal device 10A or the like determines the position at which to draw the auxiliary lines based on the type of body movement presented to the user (for example, exercises for evaluating trunk muscle movement and joint range of motion, such as lateral raises, or exercises for muscle training) and the estimated results of the positions of the joints and the skeleton. For example, some types of body movements include those in which the user moves the body by a certain amount (for example, a stepping motion of the front foot to evaluate the range of motion of a joint). While displaying an avatar to the user, the terminal device 10A draws auxiliary lines as a guide for the amount of body movement based on an estimation result of the position of a specific joint of the user. For example, when the user steps out with the front foot, the terminal device 10A first prompts the user to assume a predetermined posture (for example, to stand upright before stepping out) by audio, video, or the like. The terminal device 10A estimates the amount of the user's stride based on the positions of the joints (for example, shoulder joints, hip joints, knee joints) in that posture (for example, the amount of the stride the user should take can be specified based on the height of the user and the length of the user's lower limbs), and determines the position at which the auxiliary line should be drawn based on the estimation result of the joint positions based on the estimated amount of the stride. The terminal device 10A may store in advance, in association with each other, the positions of the joints to be referred to in order to draw the auxiliary lines and the positional relationship in which the auxiliary lines are drawn with respect to the positions of the joints, depending on the type of body movement performed by the user.

[0468] FIG. 33 is a diagram showing a concept of a process in which the terminal device 10A draws an auxiliary line based on the estimation result of the user's posture.

[0469] In the example (A) of FIG. 33, the terminal device 10A senses the body movement of the user as point cloud data. The terminal device 10A stores three-dimensional object data of an avatar AVT_A corresponding to the user based on the sensing result. The terminal device 10A estimates the positions of the user's joints (for example, each joint shown in FIG. 12) of the three-dimensional object data of the avatar AVT_A based on the three-dimensional object data of the avatar AVT_A. In the example (A), when the three-dimensional object data of the avatar AVT_A is placed in a virtual space, the terminal device 10A specifies a joint position JNT_01A that is the coordinate of the result of estimating the position of the user's hip joint in the avatar. In the example (A) of FIG. 33, the terminal device 10A determines that the three-dimensional object data of the auxiliary line LN_01A is to be placed at a position that is a certain distance DIST_01A away from the joint position JNT_01A according to the type of body movement of the user. The terminal device 10A may set the certain distance based on the user's physique (obtained by measuring the user's height, sitting height, shoulder width, etc. based on the sensing result of the user's body movement). The amount that serves as a guide for the user's body movement may be determined according to the user's physique. In the example (A), the terminal device 10A places the three-dimensional object data of the auxiliary line LN_01A so as to be parallel to the ground when the user moves his / her body.

[0470] In the example (B) of Figure 33, the terminal device 10A positions the three-dimensional object data of the auxiliary line LN_02A based on the joint position JNT_01A so that it overlaps with the joint position JNT_01A and is perpendicular to the ground when the user moves his or her body.

[0471] FIG. 34 shows an example of a screen presented to the user. As shown in FIG. 34, the terminal device 10A, as a result of sensing the user's body movements, arranges on the screen 3403 the results of drawing an avatar corresponding to the user from multiple viewpoints, such as left side 3403A, front 3403B, and right side 3403C.

[0472] For example, the terminal device 10A places three-dimensional object data (that can be recognized by the user as an avatar) corresponding to the sensing result (e.g., point cloud data) of the user's body movement in a virtual space. The terminal device 10A sets multiple virtual cameras (e.g., a virtual camera corresponding to the right side, a virtual camera corresponding to the front, and a virtual camera corresponding to the left side) for the three-dimensional object data corresponding to the user's sensing result in the virtual space. This allows the terminal device 10A to present to the user the results of drawing the avatar corresponding to the user from multiple viewpoints.

[0473] As shown in the figure, the terminal device 10A may present an avatar to the user without presenting the result of skeletal (bone) estimation. For example, it is assumed that the user has few opportunities to visually recognize the skeleton (bone) estimation result itself with the naked eye on a daily basis, compared to viewing himself / herself in a mirror or the like. Therefore, by the terminal device 10A presenting an avatar to the user without displaying the skeleton (bone) estimation result, the user will recognize an avatar on the screen that is similar to the user's own body shape that can be viewed with the naked eye on a daily basis, and the user may be immersed in the avatar projected on the screen as if it were the user himself / herself.

[0474] The terminal device 10A places the three-dimensional object data of the auxiliary lines LN_01A and LN_02A in the virtual space in association with the three-dimensional object data corresponding to the user. Therefore, the terminal device 10A can display the auxiliary lines together with the avatar corresponding to the user by performing rendering according to the settings of the virtual camera.

[0475] Specifically, the terminal device 10A draws auxiliary lines LN_01A and LN_02A on the left side 3403A in association with the avatar AVT_A and displays them to the user. The terminal device 10A draws auxiliary lines LN_01B and LN_02B on the front side 3403B in association with the avatar AVT_B and displays them to the user. The terminal device 10A draws auxiliary lines LN_01C and LN_02C on the right side 3403C in association with the avatar AVT_C and displays them to the user.

[0476] (2-3) Method for evaluating user's body movements (i) As described above, terminal device 10A displays an avatar without displaying the estimated results of the skeleton (bone) to the user, thereby making the user recognize that the avatar is a projection of the user, and immersing the user in moving the body, training, etc. Meanwhile, terminal device 10A may use the estimated results of the skeleton (bone) of the user to evaluate the user's body movement, in order to evaluate whether the user's posture is desirable and whether the user's body movement is ideal depending on the type of body movement.

[0477] For example, in the example of FIG. 34, if it is considered desirable for the user's knees to be vertical, the terminal device 10A can determine whether the user's posture is desirable or not based on the estimation result of the knees of the user's skeleton (bones) (depending on whether the knees are vertical or not). (ii) Terminal device 10A may evaluate the user's body movements based on three-dimensional object data (e.g., point cloud data) constituting an avatar, which is the result of sensing the user's body movements, and three-dimensional object data corresponding to auxiliary lines.

[0478] For example, the terminal device 10A may evaluate whether the user's body movement is ideal by determining that three-dimensional object data constituting the user's avatar (e.g., avatar AVT_A on the left side 3403A) has collided with three-dimensional object data corresponding to an auxiliary line (e.g., auxiliary line LN_01A on the left side 3403A) or has come within a certain distance.

[0479] When sensing the user's body movement without using point cloud data (for example, when photographing the user as a two-dimensional image), the terminal device 10A may generate three-dimensional object data corresponding to the user's body based on the sensing result (two-dimensional photographed image data). The terminal device 10A may evaluate whether the user's body movement is ideal or not based on whether the three-dimensional object data and the three-dimensional object data corresponding to the auxiliary line collide with each other, approach each other within a certain distance, or the like.

[0480] When sensing the user's body movement without using point cloud data (when capturing the user as a two-dimensional image), the terminal device 10A may estimate the user's posture (joint positions, skeleton) in the two-dimensional image as described above (using a trained model for estimating posture as described above), and may draw auxiliary lines as object data in the two-dimensional image based on the estimation result. For example, the terminal device 10A determines the position at which to draw auxiliary lines as object data in the two-dimensional image in accordance with the type of exercise performed by the user from the positions of the user's joints (hip joints, shoulder joints, etc.) estimated in the two-dimensional image. The terminal device 10A may evaluate whether the user's body movement is ideal or not based on whether the two-dimensional image corresponding to the user and the auxiliary lines drawn as object data collide or approach within a certain distance, etc.

[0481] (2-4) Example of a screen presented to the user to measure specific body movements (A) Standing Alignment For example, by allowing the user to adjust the user's body so that it does not tilt forward, backward, left or right while the user is in a particular posture, such as standing, the user's posture can be maintained in a good state.

[0482] FIG. 35 is a diagram showing an example of a screen that the terminal device 10A displays to allow the user to adjust the tilt in the forward / backward and left / right directions while in a standing position.

[0483] FIG. 36 is a diagram showing a process of specifying a position for drawing a graphical object serving as a guide for the user's body movement, based on a result of estimating the position of a specific part of the user.

[0484] 35, terminal device 10A, as a result of sensing the user's body movement, arranges on screen 3500 the results of drawing an avatar corresponding to the user from multiple viewpoints, such as left side 3501A and front 3501B. Terminal device 10A displays on screen 3501C the positions of multiple parts of the user's body so that their positional relationships on the horizontal plane can be compared.

[0485] The terminal device 10A displays an avatar AVT_35A (corresponding to the user as viewed from the left side) corresponding to the user on the left side 3501A. The terminal device 10A displays an auxiliary line VL_35A as a graphical object together with the avatar AVT_35A.

[0486] Here, a process in which the terminal device 10A draws the auxiliary line VL_35A will be described. As shown in the example (A) of FIG. 36, the terminal device 10A estimates an ankle (heel) position JNT_35D as a specific part of the user's body.

[0487] The terminal device 10A can estimate the positions of the right ankle, right foot, right toe, left ankle, left foot, and left toe based on point cloud data obtained by measuring the user's body (for example, the example of FIG. 12). For example, based on the estimation result of the position of the right ankle or the left ankle, the terminal device 10A may specify a position for drawing an auxiliary line VL_35A as a graphic object so that the auxiliary line passes through the estimation result of the position of the ankle and is perpendicular to the ground.

[0488] Thus, the terminal device 10A allows the user to recognize the positional relationship with the auxiliary line VL_35A while updating the display of the avatar AVT_35A (i.e., while drawing the avatar AVT_35A based on point cloud data obtained by measuring the user's body in real time) every time the sensing result of the user's body movement is obtained. The user can recognize whether the body is misaligned forward, backward, left or right by comparing the avatar AVT_35A with the auxiliary line VL_35A. For example, if the avatar AVT_35A is significantly out of the position of the auxiliary line VL_35A, it may be that the user's body is significantly misaligned in either the forward, backward, left or right direction.

[0489] In addition, in order for the terminal device 10A to draw the auxiliary line VL_35A so that the user can compare it with the avatar AVT_35A viewed from the left side as described above, the terminal device 10A may draw the auxiliary line VL_35A based on the results of estimating the position of a specific part of the user's body other than the ankle (heel) (for example, based on the results of estimating the position of the pelvis).

[0490] As shown in a front view 3501B of Fig. 35, the terminal device 10A displays an avatar AVT_35B (corresponding to a case where the user is viewed from the front) corresponding to the user. The terminal device 10A displays an auxiliary line LN_35A, an auxiliary line LN_35B, an auxiliary line LN_35C, and an auxiliary line VL_35B as graphic objects in association with the avatar AVT_35B.

[0491] Here, a process in which the terminal device 10A draws the auxiliary lines LN_35A and the like on the front surface 3501B will be described.

[0492] As shown in the example (B) of FIG. 36, the terminal device 10A estimates the head position JNT_35A ("head" in the example of FIG. 12), the shoulder position JNT_35B ("center of shoulder" in the example of FIG. 12), and the waist position JNT_35C ("center of pelvis", "right pelvis", "left pelvis", etc. in the example of FIG. 12) as specific parts of the user's body. The terminal device 10A specifies the position at which the auxiliary line VL_35B is drawn so that the auxiliary line VL_35B passes through at least one of the head position JNT_35A, the shoulder position JNT_35B, and the waist position JNT_35C and is perpendicular to the ground. Here, the terminal device 10A may specify the position at which the auxiliary line VL_35B is drawn based on the position of a specific part other than these (for example, a line that passes through the midpoint between the right heel position and the left heel position and is perpendicular to the ground).

[0493] The terminal device 10A specifies the positions at which to draw the auxiliary lines LN_35A, etc., based on the estimation results of the positions of these specific parts. For example, the terminal device 10A specifies the position at which to draw the auxiliary line LN_35A to be a position that is a predetermined distance (distance DIST_35A in the example of FIG. 36) away in the horizontal direction (when the direction toward the ground is vertical) from the head position JNT_35A, which is the estimation result of the position of the user's head. The terminal device 10A specifies that the auxiliary line LN_35B is to be drawn at a position that is a predetermined distance away in the horizontal direction from the shoulder position JNT_35B. The terminal device 10A specifies that the auxiliary line LN_35C is to be drawn at a position that is a predetermined distance away in the horizontal direction from the waist position JNT_35C.

[0494] The terminal device 10A draws these auxiliary lines LN_35A, LN_35B, and LN_35C in different ways (for example, different colors, different shapes, etc.). This allows the user to easily recognize which part of the user's body each drawing way corresponds to. Here, the terminal device 10A draws auxiliary circles RD_35A, RD_35B, and RD_35C on the screen 3501C according to the drawing ways of these auxiliary lines LN_35A, LN_35B, and LNN_35C.

[0495] The terminal device 10A draws an auxiliary circle RD_35D, which is a graphic object based on the position of the heel of the user, on the screen 3501C. Example (C) of FIG. 36 shows a state in which the auxiliary circles RD_35A, RD_35B, and RD_35C are not drawn on the screen 3501C of FIG. 35. The terminal device 10A draws an auxiliary circle RD_35A corresponding to the position of the user's head, an auxiliary circle RD_35B corresponding to the shoulder, and an auxiliary circle RD_35C corresponding to the waist on the screen 3501C. These auxiliary circles RD_35A, RD_35B, and RD_35C and the auxiliary circle RD_35D may have the same shape or may have similar shapes. The terminal device 10A displays these auxiliary circles RD_35A, RD_35B, and RD_35C at the relative positions of the head, shoulders, and waist to be compared, in comparison with the position of a specific part on the horizontal plane as a reference. This allows the user to recognize whether his or her body is leaning forward, backward, left or right, and prompts him or her to align multiple graphical objects (auxiliary circles RD_35A, RD_35B, RD_35C) corresponding to multiple body parts with the reference auxiliary circle RD_35D.

[0496] As described above, the terminal device 10A sequentially senses the user's body movements as point cloud data or the like, and estimates the positions of at least three specific parts of the user (in the examples of Figs. 35 and 36 above, the positions of four parts, the head, shoulders, waist, and heels, are estimated) based on the results of sensing the user's body movements. The terminal device 10A determines the position for drawing a graphical object showing the positional relationship of the other specific parts on the horizontal plane using one of the at least three specific parts as a reference (in the examples of Figs. 35 and 36, the position of the heel is used as a reference, but it may be any part along the body axis, such as the waist), and displays the graphical object at the determined position.

[0497] The terminal device 10A displays an estimate of the remaining time during which the user performs an action to adjust the front-back, left-right balance in a standing state in the object 3501D in Fig. 35. This encourages the user to adjust the front-back, left-right balance as much as possible within the remaining time, and the user's posture can be further improved.

[0498] (a) Lateral lift The quality of the user's posture can be evaluated by measuring the movement of the user raising both arms while standing, without extending them beyond the sides of the body. In the lateral raising movement, the timing when the user fully raises both arms while standing is described as the peak of the movement.

[0499] FIG. 37 is a diagram showing an example of a screen displayed by the terminal device 10A to prompt the user to perform a lateral lifting motion.

[0500] FIG. 38 is a diagram showing a process of specifying a position for drawing a graphical object serving as a guide for the user's body movement, based on a result of estimating the position of a specific part of the user. FIG. 39 is a diagram showing the process in which a user moves his / her body. FIG. 40 is a diagram showing a state in which the user has fully raised both arms, which corresponds to the peak timing of the lateral lifting motion. FIG. 41 is a diagram showing an example in which the user saves sensing results at the peak timing of the lateral lifting movement and displays them as comparable avatars.

[0501] 37, before starting the lateral lifting motion, the user stands with both arms at the sides of the body to assume an initial posture. The terminal device 10A may notify the user by voice or image to assume the initial posture.

[0502] The terminal device 10A draws a graphical object based on the initial posture of the user and the result of estimating the position of a specific part. In the illustrated example, the terminal device 10A displays an avatar AVT_37A (corresponding to a case where the user is viewed from the left side) corresponding to the user on the left side 3701A of the screen 3700. The terminal device 10A draws the following graphical objects on the left side 3701A.

[0503] Auxiliary line VL_37A is a graphic object that helps the user easily perceive the shift in front and behind. It remains fixed while the user moves their body. Auxiliary line VL_37D indicates the surface position of the user's head in the initial posture. By allowing the user to perceive the positional relationship between the avatar AVT_37A, which is the sensing result of the user's body movement, and the auxiliary line VL_37D while the user is performing a lateral lifting motion, it becomes easier to recognize whether the user's body is moving forward or backward by focusing on the head. While the user is moving his or her body, the auxiliary line VL_37D is fixedly positioned without moving. Auxiliary line VL_37E indicates the position of the surface of the user's abdomen in the initial posture. By allowing the user to perceive the positional relationship between the avatar AVT_37A, which is the sensing result of the user's body movement, and the auxiliary line VL_37E while the user is performing the lateral lifting motion, it becomes easier to recognize whether the user's body is moving forward or backward by focusing on the abdomen. It is placed fixedly without moving while the user is moving his or her body. An auxiliary line HL_37A indicates the length between auxiliary lines VL_37A and VL_37D. This line remains fixed while the user moves his / her body. Auxiliary line HL_37B to make it easier to recognize the position of the user's shoulders in the initial posture. It remains fixed while the user moves his / her body. An auxiliary line HL_37C indicates the length between the auxiliary line VL_37A and the auxiliary line VL_37E. This line is fixed and does not move while the user is moving his / her body.

[0504] The terminal device 10A displays, on the right side surface 3701C, auxiliary lines VL_37C, VL_37F, VL_37G, HL_37D, HL_37E, and HL_37F together with an avatar AVT_37C (corresponding to the case where the user is viewed from the right side surface). The auxiliary line VL_37C corresponds to the auxiliary line VL_37A. The auxiliary line VL_37F corresponds to the auxiliary line VL_37D. The auxiliary line VL_37G corresponds to the auxiliary line VL_37E. The auxiliary line HL_37D corresponds to the auxiliary line HL_37A. The auxiliary line HL_37E corresponds to the auxiliary line HL_37B. The auxiliary line HL_37F corresponds to the auxiliary line HL_37C.

[0505] The terminal device 10A displays an avatar AVT_37B (corresponding to the case where the user is viewed from the front) corresponding to the user in the front face 3701B. The terminal device 10A displays the following graphic objects in the front face 3701B. Auxiliary line VL_37B is a graphic object that helps the user easily perceive left-right misalignment. It remains fixed while the user moves their body. Auxiliary lines LN_37L (corresponding to the user's left arm) and LN_37R (corresponding to the user's right arm) are associated with the avatar AVT_37B to facilitate the user's perception of whether the elbow is bent while performing a lateral lifting motion. The terminal device 10A updates the positions at which the graphic objects are drawn in conjunction with the sensing results of the user's body movements. That is, the terminal device 10A updates the positions at which the auxiliary lines LN_37L and LN_37R are drawn in conjunction with the movement of the avatar. Polygon QB_37A (corresponding to the left arm) indicating the end point of auxiliary line LN_37L, and polygon QB_37B (corresponding to the right arm) indicating the end point of auxiliary line LN_37R. If the user bends his / her elbow during a lateral movement, these polygons QB_37B or QB_37A will be drawn in a position outside of the avatar AVT_37B, so the user can easily recognize the bending of the elbow by recognizing the positional relationship between the avatar AVT_37V and the polygon QB_37A or QB_37B. Polygon QB_37C (corresponding to the left arm) and polygon QB_37D (corresponding to the right arm) for allowing the user to recognize whether the user is shrugging their shoulders (whether the user is using the shoulder joints to raise their arms) when the user fully raises both arms. As will be described later, when the user fully raises both arms and does not shrug their shoulders, the terminal device 10A draws polygon QB_37C and polygon QB_37D at positions where polygon QB_37C and polygon QB_37A overlap for the left arm and where polygon QB_37D and polygon QB_37B overlap for the right arm.

[0506] When prompting the user to perform the lateral lifting motion, the terminal device 10A displays an icon 3701D indicating that the user is in the initial posture, an icon 3701E indicating that the lateral lifting motion is in progress, and an icon 3701F indicating that the measurement has ended.

[0507] The process of drawing these graphic objects will now be described. As shown in the example (A) of Figure 38, the terminal device 10A estimates the waist position JNT_38B (e.g., "center of pelvis" in the example of Figure 12), the position between the waist and shoulder JNT_38C (e.g., "thorax / waist" in the example of Figure 12), the shoulder position JNT_37D, and the head position JNT_38E as specific parts of the user's body.

[0508] The terminal device 10A specifies the position at which the auxiliary line VL_37A is drawn so that the auxiliary line passes through the positions of the waist position JNT_38B and the like and is perpendicular to the ground.

[0509] The terminal device 10A specifies the drawing position of the auxiliary line VL_37D based on the position of the body surface when the auxiliary line passes through the head position JNT_38E and moves horizontally from behind to in front of the user. That is, the terminal device 10A specifies the drawing position of the graphic object based on the result of estimating the positions of the body parts and the sensing result of the user's body movement sensed as point cloud data.

[0510] The terminal device 10A determines the position to draw the auxiliary line VL_37E based on the position of the body surface when the line passes through a position midway between the waist position JNT_38B and the position JNT_38C between the waist and shoulder, and moves horizontally from behind to in front of the user. The method of drawing each auxiliary line shown in example (C) may be the same as that in example (A).

[0511] As shown in example (B), the terminal device 10A draws the auxiliary line VL_37B based on the result of estimating the positions of the user's body parts, such as the waist and pelvis. Here, the terminal device 10A may draw the auxiliary line VL_37B so as to pass through the midpoint between the positions of the user's right heel and the position of the user's left heel and be perpendicular to the ground based on the result of estimating the positions of the heel and the left heel.

[0512] Based on the results of estimating the user's wrist position JNT_38F (left wrist) and the user's elbow position JNT_38G (left elbow), the terminal device 10A specifies the position at which to draw the auxiliary line LN_37L, starting from the left wrist and extending the distance from the left wrist to the elbow to the end point (polygon QB_37A).

[0513] The terminal device 10A specifies the position at which to draw the auxiliary line LN_37R by specifying the length of the auxiliary line in the same manner as the auxiliary line LN_37L based on the results of estimating the position of the user's wrist JNT_38I (right wrist) and the position of the user's elbow JNT_38J (right elbow).

[0514] The terminal device 10A identifies coordinates that are a predetermined distance above and outside the user's body based on the shoulder position JNT_38H (left shoulder). The coordinates are set so as to correspond to the position of the shoulder to which the user moves when the user reaches the peak of the lateral lifting motion. The terminal device 10A may identify the coordinates based on the user's physical size (e.g., the user's height, shoulder width, etc.). The terminal device 10A identifies the position to draw the polygon QB_37C based on the identified coordinate values.

[0515] The terminal device 10A specifies the position at which to draw the polygon QB_37D based on the user's shoulder position JNT_38K (right shoulder) in a manner similar to the method for specifying the position at which to draw the polygon QB_37C.

[0516] As shown in FIG. 39, when the user raises his / her arm, the terminal device 10A displays the trajectory of the hand movement as trajectory LN_39A (trajectory of left hand movement) and trajectory LN_39C (trajectory of right hand movement) based on the history of the sensing results of the user's body movement and the history of identifying the wrist position.

[0517] As shown in FIG. 40, assume that the user has raised both arms to the maximum and reached a peak. The terminal device 10A detects that the user's lateral lifting motion has reached a peak (for example, detects that the user has raised both arms based on the estimation result of the position of a specific part of the user), and draws an avatar corresponding to the sensing result (user's posture) at that timing. Here, when a polygon QB_37A whose drawing is updated in conjunction with the user's body movement and a polygon QB_37C which is a guide for the motion when both arms are fully raised are in a specific positional relationship (for example, when they overlap), at least one of the polygon QB_37A and the polygon QB_37C may be displayed in a manner different from when they are not overlapped. This allows the terminal device 10A to easily allow the user to perceive that the lateral lifting motion is appropriate (shoulders are not shrugged, etc.). Similarly, when the polygon QB_37B and the polygon QB_37D are in a specific positional relationship, the terminal device 10A may change the display mode compared to when they are not in a specific positional relationship.

[0518] As shown in FIG. 41, the terminal device 10A displays historical avatars AVT_37D and AVT_37E, which are avatar states at their peaks, alongside avatars AVT_37A and AVT_37C, whose rendering is updated in conjunction with the sensing results of the user's body movements.

[0519] As shown in Figure 41, when a user performs a lateral raising motion, an avatar with both arms fully raised, the trajectory of the user raising both arms (trajectories LN_39A, LN_39C), and avatars AVT_37A, AVT_37C representing the user's real-time posture are displayed side by side, providing the user with material to consider how to perform the motion.

[0520] (U) Crouching FIG. 42 is a diagram showing an example of a screen displayed by the terminal device 10A to prompt the user to perform a crouching motion. FIG. 43 is a diagram showing a process of specifying a position for drawing a graphical object serving as a guide for the user's body movement, based on a result of estimating the position of a specific part of the user. FIG. 44 is a diagram showing the process in which a user moves his / her body. FIG. 45 is a diagram showing a state in which the user has crouched down to the maximum, which is the peak timing of the crouching motion. FIG. 46 is a diagram showing an example in which sensing results at the peak timing of a user's crouching movement are saved and displayed as comparable avatars.

[0521] As shown in FIG. 42, in an initial posture before starting a crouching motion, the terminal device 10A draws a graphical object based on the initial posture and a result of estimating the position of a specific part.

[0522] In the illustrated example, the terminal device 10A draws the following auxiliary lines while drawing the avatar AVT_42A on the left side 4201A of the screen 4200. Auxiliary line LN_42A for visualizing the angle of the user's back · LN_42B, a helper line to visualize the user's shin angle

[0523] The terminal device 10A draws the following auxiliary lines while drawing the avatar AVT_42C on the right side surface 4201C. Auxiliary line VL_42C to help users perceive whether their posture is leaning forward or backward. It is displayed in a fixed position while measuring the user's body movements.

[0524] The terminal device 10A draws the following auxiliary lines while drawing the avatar AVT_42B in the front view 4201B. Auxiliary line VL_42B to help the user perceive whether their posture is leaning to the left or right. It is displayed in a fixed position while measuring the user's body movements. Auxiliary line LN_37L to make the user perceive whether their left foot is open to the left or right Auxiliary line LN_37R to make the user perceive whether their right foot is open to the left or right

[0525] The terminal device 10A displays an object 4201D indicating that the state is before the start of the squatting motion, an object 4201E indicating that the squatting motion is being measured, and an object 4201F indicating that the measurement has ended.

[0526] As shown in the example (A) of Figure 43, the terminal device 10A estimates the positions of the user's body parts, and based on the heel position JNT_43A and the knee position JNT_43B, draws a line segment starting from the heel position JNT_43A and passing through the knee position JNT_43B as an auxiliary line LN_42B.

[0527] Based on the waist (pelvis, etc.) position JNT_43C and the shoulder position JNT_43D, the terminal device 10A draws a line segment starting from the waist position JNT_43C and passing through the shoulder position JNT_43D as an auxiliary line LN_42A. Here, the terminal device 10A may determine the lengths of the auxiliary lines LN_42A and LN_42B so that the heights of the end points of these auxiliary lines LN_42A and LN_42B are the same.

[0528] As shown in example (C), the terminal device 10A specifies the position at which to draw the auxiliary line VL_42C so that the auxiliary line VL_42C is perpendicular to the ground, based on the heel position JNT_43J. Here, the terminal device 10A may draw the auxiliary line VL_42C based on the position of the waist, etc., separately from the heel position JNT_43J.

[0529] As shown in example (B), based on the heel position JNT_43E (left heel) and the knee position JNT_43F (left knee), the terminal device 10A measures the length from the heel to the knee, sets the heel position as the starting point, and determines the position for drawing the auxiliary line LN_37L by measuring the length from the heel to the knee and setting the end point at a point further extended from the knee by that length.

[0530] The terminal device 10A specifies the position at which the auxiliary line VL_42B is drawn so that the auxiliary line VL_42B passes through the waist position JNT_43I and is perpendicular to the ground. Note that the terminal device 10A may specify the position at which the auxiliary line VL_42B is drawn based on the positions of the left and right heels (JNT_43E, JNT_43G) so that the auxiliary line VL_42B passes through a position midway between them and is perpendicular to the ground.

[0531] 44, when the user starts to squat, the terminal device 10A draws a trajectory LN_42M of the user squatting in front of the terminal device 10A 4201B. For example, the terminal device 10A may draw the trajectory LN_42M by storing a result of estimating the position of the user's neck (or head or shoulders) as a history.

[0532] The terminal device 10A is configured to detect the trajectory LN_42 of the user crouching on the right side surface 4201C. For example, the terminal device 10A may draw the trajectory LN_42N by retaining the result of estimating the position of the user's neck, etc., as a history.

[0533] This allows the user to easily recognize whether they are crouching without shaking left and right by comparing the trajectory LN_42M with the auxiliary line VL_42B, and whether they are crouching without shaking front and back by comparing the trajectory LN_42N with the auxiliary line VL_42C.

[0534] 45, the crouching motion reaches its peak when the user crouches down completely. The terminal device 10A detects that the user's crouching motion has reached its peak (for example, detects that the user's crouching motion has reached its peak by comparing the user's posture when the crouching motion has reached its peak based on a result of estimating the position of a specific part), and holds the sensing result (point cloud data, etc.) at the timing of the peak.

[0535] Here, in a fully crouched state, the terminal device 10A may determine whether the auxiliary line LN_42B (i.e., the angle of the knee) and the auxiliary line LN_42A (i.e., the angle of the back) are apart. In the example of FIG. 45, the auxiliary line LN_42B and the auxiliary line LN_42A are apart when the starting point (heel or waist) is used as a reference, and it may be determined that the user is not leaning forward and is in a good state. In addition, the terminal device 10A may draw the end point of the auxiliary line LN_42B (when the starting point is the heel) and the end point of the auxiliary line LN_42A (when the starting point is the waist) so that they are at the same height from the ground when the user is fully crouched. In addition, the terminal device 10A may determine whether the user's movements are good depending on whether the distance between the end points of these auxiliary lines LN_42A and LN_42B is a certain distance or more (determining that the movements are good if the user is familiar with the situation).

[0536] As shown in FIG. 46, when the user stands up after crouching down, the terminal device 10A displays the sensing results of the user in the crouched state as avatar AVT_42D on the left side 4201A, avatar AVT_42E on the front 4201B, and avatar AVT_42F on the right side 4201C.

[0537] The terminal device 10A displays auxiliary lines LN_42A and LN_42B in a fully crouched state together with the avatar AVT_42D on the left side 4201A.

[0538] The terminal device 10A draws, on the front surface 4201B, a trajectory LN_42P (for example, a trajectory is drawn based on the position of the neck) when standing up after crouching down to the fullest (after reaching the peak timing).

[0539] The terminal device 10A draws, on the right side surface 4201C, a trajectory LN_42Q (for example, a trajectory is drawn based on the position of the neck) when standing up after crouching down to the end (after reaching the peak timing).

[0540] (D) Stepping forward and lowering the hips FIG. 47 is a diagram showing an example of a screen displayed by the terminal device 10A to prompt the user to step forward and lower his hips. FIG. 48 is a diagram showing an example of a screen in which a message is displayed to the user indicating that the user's initial posture is appropriate. FIG. 49 shows a situation in which measurement of a user's movement starts. FIG. 50 is a diagram showing a situation in which it is determined whether a user's action is appropriate or not, and the determination result is presented to the user. FIG. 51 is a diagram showing a situation in which it is determined whether a user's action is appropriate or not, and the determination result is presented to the user.

[0541] 47, the terminal device 10A displays an avatar AVT_47B (corresponding to the front) corresponding to the user on a front side 4701B of the screen 4700. The terminal device 10A displays an avatar AVT_47C (right side) corresponding to the user and auxiliary lines LN_47A and LN_47B serving as a guide for the position where the user steps out with both feet on a right side 4701C.

[0542] In the illustrated example, the terminal device 10A displays an arrow-shaped object that prompts the user to lower his or her hind legs further back.

[0543] In the example of FIG. 47, the terminal device 10A measures the length of the user's legs based on the result of estimating the position of a specific part of the user. For example, the terminal device 10A measures the length of the user's legs based on the position of the user's heel and the position of the hips (pelvis). In this way, the terminal device 10A specifies the distance by which the user should spread his / her legs when performing a motion. The terminal device 10A may adjust the distance by which the user should spread his / her legs according to the user's physique (e.g., the length of the legs). The terminal device 10A determines the distance between the auxiliary lines LN_47A and LN_47B based on the result of measuring the length of the user's legs, and draws it on the right side surface 4701C.

[0544] In the illustrated example, the terminal device 10A displays an object 4701D for visualizing the rotation of the user's waist and shoulders on the screen 4700. The terminal device 10A identifies the rotation of the user's waist based on the result of estimating the position of the pelvis (the positions of the "left pelvis" and "right pelvis" in the example of FIG. 12). The terminal device 10A identifies the rotation of the user's shoulders based on the result of estimating the position of the shoulders (the "left shoulder" and "right shoulder" in the example of FIG. 12).

[0545] The terminal device 10A displays an auxiliary line LN_47SB indicating the rotation of the hips and an auxiliary line LN_47SH indicating the rotation of the shoulders in the object 4701D, for example, based on the position of the hips (such as the center of the pelvis). This makes it easy for the user to perceive the degree to which the hips and shoulders are rotating when lowering the hips while stepping forward.

[0546] As shown in FIG. 48, before measuring the movement of the user spreading their legs, lowering their hips and moving them up and down, the terminal device 10A determines whether or not both of their legs are sufficiently spread apart.

[0547] The terminal device 10A may determine that the user's legs are appropriately spread apart in the following manner. Based on the results of estimating the positions of the user's body parts, the angle at which the legs are apart (such as the angle formed by the line segment from the left knee to the pelvis and the line segment from the right knee to the pelvis) is greater than or equal to a certain value. When a specific part of the foot (such as the heel) when the user steps forward or backward has a predetermined positional relationship with auxiliary line LN_47A and auxiliary line LN_47B. For example, when the result of estimating the position of the heel of the user's front foot overlaps with auxiliary line LN_47A or is within a certain distance. Similarly, when the result of estimating the position of the heel of the user's back foot overlaps with auxiliary line LN_47B or is within a certain distance.

[0548] When the front and back legs are properly spread apart, the terminal device 10A displays an object OBJ_47A and an object OBJ_47B indicating that fact. When the legs are not properly spread apart, the terminal device 10A displays a message to that effect (for example, "NG") in the object OBJ_47A or the object OBJ_47B.

[0549] The terminal device 10A may draw the object OBG_47A or the object OBJ_47B based on the result of estimating the position of a specific part of the user. For example, the terminal device 10A may draw the object OBJ_47A in association with the coordinates of the result of estimating the position of the user's heel (for example, the speech bubble of the object OBJ_47A is drawn based on the position of the user's heel).

[0550] As described above, the terminal device 10A may notify the user by voice or image to start the movement (for example, a notification reminding the user of the specific movement content, such as "Slowly lower your hips and then return to the original position") in response to detecting that the user's posture before making a movement is appropriate. This allows the user to confirm that the posture before making a movement is appropriate, making it easier to start the movement of raising and lowering the hips.

[0551] As shown in FIG. 49, when the user moves his / her hips up and down, the terminal device 10A draws, in front 4701B, an avatar AVT_47B corresponding to the user, as well as auxiliary lines VL_47B and VL_47E that are displayed fixedly (at fixed positions) regardless of the user's movements.

[0552] The auxiliary line VL_47B is an auxiliary line for allowing the user to perceive whether the user is shaking left or right while the user is moving his or her hips up and down. For example, the terminal device 10A draws the auxiliary line VL_47B based on the result of estimating the position of the user's hips before the user starts the motion of lowering the hips.

[0553] The auxiliary line VL_47E is an auxiliary line for making the user perceive whether the user's foot (for example, the front foot) is shaking left and right while the user is lowering the hips. For example, before the user starts the motion of lowering the hips, the terminal device 10A draws the auxiliary line VL_47B based on the result of estimating the positions of the heel and knee of the user's front foot.

[0554] The auxiliary line VL_47F is an auxiliary line for making the user perceive whether the user's shoulders are shaking left and right while the user is lowering his / her hips. The terminal device 10A draws the auxiliary line VL_47F in conjunction with the user's motion of lowering his / her hips.

[0555] The terminal device 10A displays, on the right side surface 4701C, an avatar AVT_47C corresponding to the user, as well as auxiliary lines VL_47C and LN_47C that are fixedly displayed regardless of the user's movements.

[0556] The auxiliary line VL_47C is an auxiliary line for allowing the user to recognize whether the user is lowering the hips vertically without shaking forward and backward while the user is moving the hips up and down. For example, the terminal device 10A draws the auxiliary line VL_47C based on the result of estimating the position of the user's pelvis, etc., before the user starts the motion of lowering the hips.

[0557] The auxiliary line LN_47C is an auxiliary line for allowing the user to recognize whether the user is lowering his / her hips too far forward or backward (for example, whether the knees are protruding too far forward) while the user is lowering his / her hips. For example, the terminal device 10A draws the auxiliary line LN_47C based on the result of estimating the position of the user's heels, etc., before the user starts the motion of lowering his / her hips.

[0558] The terminal device 10A draws an object OBJ_47E and an object OBJ_47F, which indicate the position and speed at which the user lowers his / her hips, in conjunction with the speed at which the user raises and lowers his / her hips. These objects OBJ_47E and objects OBJ_47F move along the auxiliary line VL_47C in conjunction with the speed at which the user raises and lowers his / her hips.

[0559] Object OBJ_47E is an object for allowing the user to perceive the approximate positions of the neck and shoulders while the user is moving his / her hips up and down.

[0560] Object OBJ_47F is an object for allowing the user to perceive the position of the waist guideline while the user is moving the waist up and down.

[0561] 50, it is assumed that the user is trying to lower his / her hips and is leaning slightly forward. As shown in the figure, the terminal device 10A renders, on the right side surface 4701C, an object OBJ_47J indicating the actual position where the user is moving his / her body (an estimated result of the positions of the neck, shoulders, head, etc.) with respect to an object OBJ_47E that is a guide for ideal movement.

[0562] The terminal device 10A renders an object OBJ_47K indicating the actual position where the user is moving his / her body (estimated result of the positions of the waist, pelvis, etc.) in relation to an object OBJ_47F that serves as a guide for ideal movement.

[0563] In order to allow the user to easily perceive whether or not the user is vertically moving his / her hips up and down, when the object OBJ_47E and the object OBJ_47J are in a predetermined positional relationship (overlapping, within a certain distance, etc.), the terminal device 10A may draw them in a display mode different from when the object OBJ_47E and the object OBJ_47J are not in the predetermined positional relationship. Similarly, when the object OBJ_47F and the object OBJ_47K are in a predetermined positional relationship, the terminal device 10A may draw them in a display mode different from when the object OBJ_47F and the object OBJ_47K are not in the predetermined positional relationship. This makes it easy for the user to perceive in real time whether the user is moving his / her hips up and down with an ideal movement and whether the user is moving his / her body in accordance with the guideline for the ideal movement.

[0564] The terminal device 10A determines whether the front legs are being moved appropriately and displays the determination result by the object OBJ_47P. For example, the terminal device 10A may determine whether the front legs are being moved appropriately as follows. Determine whether the user is moving the forefoot properly (e.g., leaning too far forward) based on the angle of the user's forefoot, including the heel, knee, and hip It is determined whether the estimated position of the user's lower body parts (heels, knees, etc.) and auxiliary line LN_47C have a predetermined positional relationship. (For example, if the position of the heel or knee exceeds the auxiliary line LN_47C, the user is leaning forward. Also, if the line segment connecting the position of the heel and the position of the knee (knee angle) is within a predetermined angle with the auxiliary line LN_47C, the hips are lowered properly.)

[0565] The terminal device 10A may specify the position at which the object OBJ_47P is drawn based on the result of estimating the position of the user's body part. For example, the terminal device 10A may draw the object OBJ_47P based on the position of the user's knee. This allows the object OBJ_47P to be drawn while moving in real time along with the user's body movement, making it easy for the user to perceive whether or not the user is moving his or her body appropriately.

[0566] The terminal device 10A determines whether the hind legs are being moved appropriately, and displays the result of the determination by the object OBJ_47N. As in the case of the front legs described above, it may be determined whether the hind legs are being moved appropriately.

[0567] The terminal device 10A may determine whether the front legs are not wobbling left and right in front 4701B, and display the determination result by the object OBJ_47M. For example, based on the result of estimating the positions of the heel and the knee, when the angle of the knee is not perpendicular (for example, when the angle is a predetermined angle or more with respect to the auxiliary line VL_47E), the terminal device 10A may determine that the front legs are wobbling left and right. In the example of FIG. 50, the front legs are not wobbling left and right, and "OK" is displayed in the object OBJ_47M.

[0568] As shown in FIG. 51, on the right side 4701C, the user is able to move his / her hips up and down vertically and satisfactorily, and "OK" is displayed on the objects OBJ_47P and OBJ_47N.

[0569] (3) Supplementary Notes The matters explained in the above embodiment will be supplemented below.

[0570] (Appendix 1) A program for operating a computer, the program causing a computer processor to execute the following steps: sequentially sensing a user's body movements; based on the sensing results, presenting an avatar corresponding to the user in a manner that allows the user to perceive the shape of the user's body surface rather than a live image of the user's appearance when photographed; based on the sensing results, specifying a position for drawing a graphical object that is associated with the avatar and serves as an indication of the user's body movements; and displaying the graphical object at the specified position. (Appendix 2) The program described in Appendix 1, wherein in the step of presenting an avatar, the avatar is presented to the user in a form that contains less information than a live-action image. (Appendix 3) 3. The program according to claim 1 or 2, wherein in the step of presenting an avatar, an avatar is presented to the user each time a sensing result is obtained. (Appendix 4) A program described in any of Appendices 1 to 3, in which in the step of presenting an avatar, an avatar depicting the user from a first direction and an avatar depicting the user from a second direction different from the first direction are drawn side by side based on the sensing results, thereby presenting an avatar to the user. (Appendix 5) A program described in any one of Appendices 1 to 4, wherein in the sensing step, the user's body movements are sensed as point cloud data, and in the avatar presenting step, the avatar is presented to the user by rendering the avatar as a collection of point cloud data. (Appendix 6) The program described in any of Appendices 1 to 5, further causing the processor to execute the steps of: estimating the positions of multiple specific body parts of the user based on the results of sensing the user's body movements; and drawing the avatar while not drawing the results of the estimation of the positions of the specific body parts, and evaluating the user's body movements based on the results of the estimation of the positions of the specific body parts. (Appendix 7) A program described in any of Appendices 1 to 6, wherein in the step of presenting an avatar, the drawing of the avatar is updated in accordance with the sensing results of the user's body movements, while in the step of displaying, a graphical object is displayed at a fixed position regardless of the user's body movements. (Appendix 8) The program described in Appendix 7 further causes the processor to execute the steps of estimating the position of a specific part of the user based on the results of sensing the user's body movements, and depicting a sensing history in chronological order along with an avatar for the position of a first part of the user's body that is the target for measuring the user's body movements. (Appendix 9) The program according to claim 7, wherein in the identifying step, based on the results of sensing the user, a linear graphical object is drawn parallel to the user's body axis at a fixed position regardless of the user's body movement. (Appendix 10) The program described in Appendix 1, in which in the identification step, a position for drawing a graphical object that serves as a guide when the user performs a specific movement is identified based on the sensing result of the user's first posture before the user starts the specific movement. (Appendix 11) The specific movement is a predetermined movement for measuring the user's body movement, and in the identifying step, a position for drawing a graphical object that indicates the user's body movement at a predetermined timing of the specific movement is identified based on the first posture. (Appendix 12) The program described in Appendix 10, in which in the identifying step, a position for drawing a graphical object that serves as a guide for at least one of a group consisting of front-to-back balance and left-to-right balance of the user's body is identified based on the first posture. (Appendix 13) The program described in Appendix 10, wherein in the identifying step, a position for drawing a graphical object that indicates a second posture that the user can take while performing a specific action is identified based on the first posture. (Appendix 14) The specific movement is a movement performed by the user with both feet apart and stepping forward, and in the identifying step, the position at which to draw a graphical object that indicates when the user has stepped forward with their front feet is identified based on the first posture. (Appendix 15) The program described in Appendix 14 further executes a step of estimating the position of the user's lower limbs based on the results of sensing the user's body movement, and in the identification step, identifies a position for drawing a graphical object that serves as an indication of the position where the user will step with both feet apart based on the estimation result. (Appendix 16) The program described in Appendix 13 executes the steps of: estimating the position of a specific part of the user based on the results of sensing the user's body movements; and evaluating whether the user is properly assuming the second posture based on the positional relationship between the estimated position of the specific part and the position of a graphical object. (Appendix 17) The program according to claim 16, wherein if the evaluating step evaluates that the second posture is being properly performed, a message to that effect is displayed to the user. (Appendix 18) The specific movement involves a body movement performed by the user in the second posture, and in the evaluating step, in response to evaluating that the second posture is being properly performed, measurement of the specific movement is started and a notification to the user that the measurement has started is presented. (Appendix 19) The program described in Appendix 1 further executes a step of estimating the positions of multiple specific body parts of the user based on the results of sensing the user's body movements, and in the identifying step, identifies a position for drawing a linear graphical object that allows the user to perceive whether the specific body parts are moving appropriately during the user's body movement based on the estimated positions of the specific body parts. (Appendix 20) 20. The program according to claim 19, wherein in the displaying step, a line segment extending from a surrounding area of ​​a specific area toward the specific area is displayed as a graphical object together with the avatar. (Appendix 21) The program according to claim 20, wherein in the displaying step, a graphical object having a size larger than the width of the line segment is drawn and displayed at the end point of the line segment. (Appendix 22) The program described in Appendix 21, in which, in the display step, a graphical object corresponding to the end point of the line segment is displayed so as to move in conjunction with the sensing results of the user's body movement, and the display mode of the graphical object is changed and displayed based on the positional relationship between the graphical object that is fixedly positioned regardless of the user's body movement and the graphical object that corresponds to the end point. (Appendix 23) The program described in Appendix 19 further executes a step of estimating the positions of multiple specific body parts of the user based on the results of sensing the user's body movements, and in the step of displaying, draws and displays a graphical object indicating the inclination of a first body part of the user and a graphical object indicating the inclination of a second body part different from the first body part in association with the first body part and the second body part, respectively. (Appendix 24) The program described in Appendix 19 further executes a step of estimating the position of a specific part of the user based on the results of sensing the user's body movements, and in the identification step, identifies a position for drawing a graphical object based on the estimated position of the specific part and the shape of the user's body surface. (Appendix 25) The specific movement is a predetermined movement for measuring the user's body movement, and in the presenting step, an avatar is presented to the user each time a sensing result is obtained, while the sensing result of the user at a predetermined timing that marks the division of the specific movement is stored in a memory unit, thereby presenting the avatars at the predetermined timing in a line to the user. (Appendix 26) The program described in Appendix 1, which executes the steps of: in the step of presenting an avatar, estimating the positions of at least three specific body parts of the user based on the results of sensing the user's body movements; determining a position for drawing an object indicating the positional relationship of the other specific body parts in a horizontal plane based on one of the at least three specific body parts, and displaying the object at the determined position; and in the step of identifying, identifying a position for drawing a graphical object in association with the avatar based on the estimation results of the positions of the at least three body parts. (Appendix 27) A program for operating a computer, the program causing a processor of the computer to execute the steps of sequentially sensing the movements of a user's body and estimating the positions of at least three specific body parts of the user based on the results of sensing the movements of the user's body, determining a position for drawing a graphical object indicating the positional relationship of other specific body parts in a horizontal plane based on one of the at least three specific body parts, and displaying the graphical object at the determined position. (Appendix 28) 28. The program according to claim 27, wherein in the step of displaying the graphical objects, the graphical objects representing specific parts that are not the reference are displayed as graphical objects each including a figure having the same or a similar shape. (Appendix 29) 1. A method for operating a computer, the method comprising: a processor of the computer comprising: A method comprising the steps of: sequentially sensing a user's body movements; based on the sensing results, presenting an avatar corresponding to the user in a manner that allows the user to perceive the shape of the user's body surface rather than a live image of the user's appearance when photographed; based on the sensing results, specifying a position for drawing a graphical object that is associated with the avatar and serves as an indication of the user's body movements; and displaying the graphical object at the specified position. (Appendix 30) An information processing device, the information processing device executing the following steps: a control unit sequentially senses the body movements of a user; based on the sensing results, presenting an avatar corresponding to the user in a manner that allows the user to perceive the shape of the user's body surface rather than a live image of the user's appearance when photographed; based on the sensing results, specifying a position to draw a graphical object that is associated with the avatar and serves as an indication of the user's body movements; and displaying the graphical object at the specified position. [Explanation of symbols]

[0571] 10 terminal device, 20 server, 30 measurement unit, 31 measurement device, 32 mat, 33 pole, 132 display, 181 user information, 182 measurement item information, 183 measurement history information, 184 friend list, 185 training item information, 281 user information database, 282 measurement item information, 283 measurement history database, 284 friend list, 285 training item information, 287 measurement result database, 288 evaluation result database, 289 user questionnaire information, 290 evaluation result database, 2601 trainee information database, 2602 trainer information database, 2603 training history database

Claims

1. A program for operating a computer, the program causing a processor of the computer to: A step of sequentially sensing a user's body movement; presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive a shape of a body surface of the user, rather than a real-life image of the user's appearance, based on a result of the sensing; determining a position for drawing a graphical object serving as an indication of a body movement of the user in association with the avatar based on the result of the sensing; displaying the graphical object at the identified position; Run the command, In the sensing step, a body movement of the user is sensed as point cloud data; A program that, in the step of presenting an avatar, presents the avatar to the user by rendering the avatar as a collection of the point cloud data.

2. The program according to claim 1 , wherein in the step of presenting the avatar, the avatar is presented to the user in a form in which an amount of information is reduced compared to the actual image.

3. 3. The program according to claim 1, wherein in the step of presenting an avatar, the avatar is presented to the user every time a result of the sensing is obtained.

4. In the step of presenting the avatar, A program as described in any one of claims 1 to 3, which presents an avatar to the user by drawing an avatar depicting the user from a first direction and an avatar depicting the user from a second direction different from the first direction side by side based on the sensing results.

5. The program further causes the processor to estimating positions of a plurality of specific parts of the user based on a result of sensing a body movement of the user; 5. The program according to claim 1, further comprising: a step of drawing the avatar while not drawing the result of estimating the position of the specific body part, and evaluating the user's body movements based on the result of estimating the position of the specific body part.

6. In the step of presenting the avatar, a drawing of the avatar is updated in accordance with a sensing result of a body movement of the user, 6. The program according to claim 1, wherein in said displaying step, said graphical object is displayed at a fixed position regardless of a body movement of said user.

7. The program further causes the processor to estimating a position of a specific part of the user based on a result of sensing a body movement of the user; 7. The program according to claim 6, further comprising: a step of drawing, in chronological order, the sensing history of the position of the first part of the user that is the target for measuring the body movement, together with the avatar.

8. 7. The program according to claim 6, wherein in the identifying step, it is identified that the linear graphical object is to be drawn parallel to the user's body axis and at a fixed position regardless of the user's body movement, based on the results of sensing the user.

9. The program according to claim 1 , wherein in the identifying step, a position at which to draw the graphical object that serves as a guide when the user performs a specific movement is identified based on a sensing result of a first posture of the user before the user starts the specific movement.

10. the specific motion is a predetermined motion for measuring a body movement of the user, 10. The program according to claim 9, wherein in the specifying step, a position for drawing the graphical object serving as an indication of the user's body movement at a predetermined timing of the specific action is specified based on the first posture.

11. The program according to claim 9 , wherein in the identifying step, a position for drawing the graphical object that serves as a guide for at least one of a group consisting of front-to-back balance and left-to-right balance of the user's body is identified based on the first posture.

12. 10. The program according to claim 9, wherein in the identifying step, a position at which to draw the graphical object, which is an indication of a second posture that the user can take while performing the specific action, is identified based on the first posture.

13. The specific motion is a motion performed by the user with both feet apart and stepping forward, 13. The program according to claim 12, wherein in the specifying step, a position at which the graphic object serving as a guide for a state in which the user has stepped forward with his / her front foot is drawn is specified based on the first posture.

14. further executing a step of estimating a position of a part of a lower limb of the user based on a result of sensing the body movement of the user; 14. The program according to claim 13, wherein in the specifying step, a position for drawing the graphic object serving as a guide for a position where the user will step with both feet apart is specified based on the result of the estimation.

15. estimating a position of a specific part of the user based on a result of sensing a body movement of the user; and evaluating whether the user is appropriately assuming the second posture based on a positional relationship between the estimated position of the specific part and a position of the graphical object.

16. The program according to claim 15 , wherein when it is evaluated in the evaluating step that the second posture is appropriately performed, a message to that effect is presented to the user.

17. the specific motion involves a body movement performed by the user in the second posture, The program according to claim 15, wherein in the evaluating step, in response to evaluating that the second posture is being properly performed, measurement of the specific movement is started and the user is notified that the measurement is about to start.

18. and further executing a step of estimating positions of a plurality of specific parts of the user based on a result of sensing the body movement of the user. The program according to claim 1, wherein in the identifying step, a position for drawing the linear graphical object is identified based on the estimated position of the specific part, in order to allow the user to perceive whether the specific part is moving appropriately during the user's physical movement.

19. 20. The program according to claim 18, wherein in the displaying step, line segments extending from areas surrounding the specific part toward the specific part are displayed as the graphic object together with the avatar.

20. 20. The program according to claim 19, wherein in the displaying step, the graphic object is drawn and displayed at an end point of the line segment, the graphic object being large compared to a width of the line segment.

21. In the displaying step, the graphical object corresponding to the end point of the line segment is displayed so as to move in conjunction with a sensing result of a body movement of the user; 21. The program according to claim 20, further comprising: changing and displaying a display mode of the graphical object based on a positional relationship between the graphical object that is fixedly positioned regardless of the user's body movement and the graphical object that corresponds to the end point.

22. and further executing a step of estimating positions of a plurality of specific parts of the user based on a result of sensing the body movement of the user.

20. The program according to claim 18, wherein in the displaying step, the graphical object indicating the inclination of a first part of the user and the graphical object indicating the inclination of a second part different from the first part are drawn and displayed in association with the first part and the second part, respectively.

23. The method further includes the step of estimating a position of a specific part of the user based on a result of sensing the body movement of the user; 20. The program according to claim 18, wherein in the specifying step, a position at which to draw the graphical object is specified based on the estimated position of the specific part and a shape of the user's body surface.

24. the specific motion is a predetermined motion for measuring a body movement of the user, The program of claim 9, wherein in the presenting step, the avatar is presented to the user each time the sensing result is obtained, while the sensing result of the user at a predetermined timing that marks the division of the specific action is stored in a memory unit, so that the avatars at the predetermined timing are lined up and presented to the user.

25. In the step of presenting the avatar, and estimating positions of at least three specific parts of the user based on the results of sensing the body movement of the user. determining a position for drawing an object indicating a positional relationship of the other specific body parts on a horizontal plane based on one of the at least three specific body parts, and displaying the object at the determined position; The program according to claim 1 , wherein in the specifying step, a position for drawing the graphic object in association with the avatar is specified based on the estimation results of the positions of the at least three body parts.

26. and estimating positions of at least three specific parts of the user based on the results of sensing the body movement of the user. The program according to claim 1 , further comprising a step of: determining a position for drawing an object indicating the positional relationship of another specific part in a horizontal plane based on one of the at least three specific parts; and displaying the object at the determined position.

27. In the step of displaying the object, 27. The program according to claim 26, wherein the objects representing the specific portions other than the reference are displayed as objects including figures having the same or similar shapes.

28. 28. The program according to claim 1, wherein in the sensing step, the sensing means sequentially senses the user's body movements without requiring the exercise equipment used by the user to be placed in a fixed position relative to the sensing means.

29. 1. A method for operating a computer, the method comprising: a processor of the computer comprising: A step of sequentially sensing a user's body movement; presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive a shape of a body surface of the user, rather than a real-life image of the user's appearance, based on a result of the sensing; determining a position for drawing a graphical object serving as an indication of a body movement of the user in association with the avatar based on the result of the sensing; displaying the graphical object at the identified position; Run In the sensing step, a body movement of the user is sensed as point cloud data; The method of claim 1, wherein in the step of presenting an avatar, the avatar is presented to the user by rendering the avatar as a collection of the point cloud data.

30. An information processing device, the information processing device comprising: A step of sequentially sensing a user's body movement; presenting an avatar corresponding to the user to the user in a manner that allows the user to perceive a shape of a body surface of the user, rather than a real-life image of the user's appearance, based on a result of the sensing; determining a position for drawing a graphical object serving as an indication of a body movement of the user in association with the avatar based on the result of the sensing; displaying the graphical object at the identified position; Run In the sensing step, a body movement of the user is sensed as point cloud data; An information processing device, in the step of presenting an avatar, presents the avatar to the user by rendering the avatar as a collection of the point cloud data.

Images