Information processing device, method, program, and system

Abstract

To provide a technique for enriching the types of exercise available for effective and safe exercise therapy. [Solution] A program of one embodiment of the present disclosure causes a computer to function as a means for acquiring sensing data regarding a user performing a first exercise type among a predetermined number of exercise types, a means for analyzing the sensing data, a means for determining whether the user is performing the first exercise type as defined based on the analysis results of the sensing data and exercise definition information that defines the first exercise type, and a means for presenting information to the user according to the result of the determination.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a method, a program, and a system. [Background technology]

[0002] The importance of exercise therapy, which utilizes exercise for the treatment and prevention of disorders and diseases, is increasing. For example, cardiac rehabilitation aims to help patients with cardiac disease recover their physical strength and self-confidence, return to a comfortable home and social life, and prevent recurrence of cardiac disease or re-hospitalization through a comprehensive activity program that includes exercise therapy. The core of exercise therapy is aerobic exercise, such as walking, jogging, cycling, and aerobics. To perform aerobic exercise more safely and effectively, it is preferable for patients to exercise at an intensity close to their own anaerobic threshold (AT).

[0003] The anaerobic metabolic threshold is an example of an evaluation index of exercise tolerance, and corresponds to a change point of the cardiopulmonary function state, that is, an exercise intensity near the boundary between aerobic exercise and anaerobic exercise. The anaerobic metabolic threshold is generally determined by a CPX test (cardiopulmonary exercise stress test) in which an examinee is given a gradually increasing exercise load while collecting and analyzing exhaled gas (see Non-Patent Document 1). In the CPX test, the anaerobic metabolic threshold is determined based on the results measured by exhaled gas analysis (e.g., oxygen intake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof). In addition to the anaerobic metabolic threshold, the CPX test can also determine the maximum oxygen intake equivalent to an exercise intensity near the maximum exercise tolerance.

[0004] In order to ensure the efficacy and safety of exercise training, it is important to properly manage the exercise intensity of the subjects.

[0005] Patent Document 1 discloses a technical idea of ​​acquiring the pace and pitch of a user's running, determining whether or not it is within an acceptable range, and performing a notification operation based on the determination result. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2014-045782 A [Non-patent literature]

[0007] [Non-Patent Document 1] Saito, Muneyasu, Cardiac Rehabilitation, Physical Therapy, 1997, Vol. 24, No. 8, p. 414-418 Summary of the Invention [Problem to be solved by the invention]

[0008] The technical idea of ​​Patent Document 1 cannot be simply applied to other types of exercise than running. Even if a specific type of exercise is specified to subject a subject to a specific exercise intensity, it is not easy for the subject to accurately understand and practice the ideal movements of the type of exercise in detail, so there is a risk that the subject may be subjected to a load that exceeds the expected load, or conversely, may not be subjected to the expected load. In other words, the exercise intensity may be too high or too low.

[0009] An object of the present disclosure is to provide technology for enriching the types of exercise available for effective and safe exercise therapy. [Means for solving the problem]

[0010] A program of one embodiment of the present disclosure causes a computer to function as a means for acquiring sensing data regarding a user performing a first exercise type among a predetermined number of exercise types, a means for analyzing the sensing data, a means for determining whether the user is performing the first exercise type as defined based on the analysis results of the sensing data and exercise definition information that defines the first exercise type, and a means for presenting information to the user according to the result of the determination. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing a configuration of a client device according to the present embodiment. [Diagram 3] FIG. 2 is a block diagram showing a configuration of a server according to the present embodiment. [Figure 4] 1 is a block diagram showing a configuration of a wearable device according to an embodiment of the present invention. [Diagram 5] FIG. 1 is an explanatory diagram of one aspect of the present embodiment. [Figure 6] FIG. 2 is a diagram showing a data structure of an exercise event database according to the present embodiment. [Figure 7] FIG. 4 is a diagram showing a data structure of a user profile database according to the present embodiment. [Figure 8] 4 is a flowchart of an exercise support process according to the present embodiment. [Figure 9] 13 is a flowchart of a scoring process in the first modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and the repeated description will be omitted.

[0013] (1) Information Processing System Configuration The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to the present embodiment.

[0014] As shown in FIG. 1, the information processing system 1 includes a client device 10, a server 30, and a wearable device 50.

[0015] Here, the number of client devices 10 and wearable devices 50 varies depending on, for example, the number of users. Therefore, the number of client devices 10 and wearable devices 50 may each be two or more. Furthermore, a terminal of a person who plans or instructs exercise therapy may also be included in the information processing system 1. The person who plans or instructs exercise therapy may include, for example, a medical professional (e.g., a doctor, a nurse, a pharmacist, a physical therapist, an occupational therapist, or a clinical laboratory technician), a nutritionist, or a trainer.

[0016] The client device 10 and the server 30 are connected via a network (eg, the Internet or an intranet) NW. The client device 10 and the wearable device 50 are connected via a wireless channel, for example using Bluetooth (registered trademark) technology.

[0017] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0018] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a server computer.

[0019] The wearable device 50 is an example of an information processing device that can be worn on the user's body (for example, on the arm, hand, or head).

[0020] (1-1) Client device configuration The configuration of the client device will now be described with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0021] 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0022] The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a Read Only Memory (ROM), a Random Access Memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0023] The programs include, for example, the following programs: ·OS (Operating System) programs · Programs for applications that process information (e.g. web browsers, therapeutic apps, rehabilitation apps, or fitness apps) Here, the diseases that are the target of therapeutic or rehabilitation apps are diseases in which exercise may contribute to improving symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

[0024] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0025] The processor 12 is a computer that realizes the functions of the client device 10 by running a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Gate Array)

[0026] The input / output interface 13 is configured to obtain information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10.

[0027] The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0028] The communication interface 14 is configured to control communications between the client device 10 and external devices (eg, another client device 10, a server 30, and a wearable device 50). Specifically, the communication interface 14 may include a module for communication with the server 30 (e.g., a WiFi module, a mobile communication module, or a combination thereof). The communication interface 14 may include a module for communication with the wearable device 50 (e.g., a Bluetooth module).

[0029] The display 15 is configured to display an image (still image or moving image). The display 15 is, for example, a liquid crystal display or an organic EL display.

[0030] The camera 16 is configured to capture images and generate image signals.

[0031] The depth sensor 17 is, for example, a light detection and ranging (LIDAR) sensor. The depth sensor 17 is configured to measure a distance (depth) from the depth sensor 17 to a surrounding object (for example, a user).

[0032] The microphone 18 is configured to receive sound waves and generate a sound signal, and is preferably placed in the vicinity of the user's body (particularly the respiratory tract), such as an earphone microphone.

[0033] The acceleration sensor 19 is configured to detect acceleration.

[0034] (1-2) Server configuration The configuration of the server will now be described with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0035] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0036] The storage device 31 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage.

[0037] The programs include, for example, the following programs: -OS programs Application programs that perform information processing

[0038] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0039] The processor 32 is a computer that realizes the functions of the server 30 by starting a program stored in the storage device 31. The processor 32 is, for example, at least one of the following: ·CPU GPU ·ASIC FPGA

[0040] The input / output interface 33 is configured to obtain information (eg, a user's instruction) from an input device connected to the server 30 and to output information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0041] The communication interface 34 is configured to control communications between the server 30 and external devices (eg, the client device 10).

[0042] (1-3) Configuration of wearable devices The configuration of the wearable device will now be described with reference to Fig. 4, which is a block diagram showing the configuration of the wearable device of this embodiment.

[0043] 4, the wearable device 50 includes a storage device 51, a processor 52, an input / output interface 53, and a communication interface 54. The wearable device 50 is connected to a display 55, a heart rate sensor 56, and an acceleration sensor 57.

[0044] The storage device 51 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage device.

[0045] The programs include, for example, the following programs: -OS programs · Programs for applications that process information (e.g., medical, rehabilitation, or fitness apps)

[0046] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0047] The processor 52 is a computer that realizes the functions of the wearable device 50 by starting the programs stored in the storage device 51. The processor 52 is, for example, at least one of the following: ·CPU GPU ·ASIC FPGA

[0048] The input / output interface 53 is configured to acquire information (e.g., user instructions, sensing results) from an input device connected to the wearable device 50, and to output information (e.g., images, commands) to an output device connected to the wearable device 50.

[0049] The input device is, for example, a heart rate sensor 56, an acceleration sensor 57, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 55, a speaker, or a combination thereof.

[0050] The communication interface 54 is configured to control communication between the wearable device 50 and an external device (eg, the client device 10). In particular, the communication interface 54 may include a module for communicating with the client device 10 (eg, a Bluetooth module).

[0051] The display 55 is configured to display an image (still image or moving image). The display 55 is, for example, a liquid crystal display or an organic EL display.

[0052] The heart rate sensor 56 is configured to measure the heart rate and generate a sensing signal. As an example, the heart rate sensor 56 measures the heart rate using an optical measurement technique.

[0053] The acceleration sensor 57 is configured to detect acceleration.

[0054] (2) One aspect of the embodiment An embodiment of the present invention will now be described with reference to Fig. 5, which is an explanatory diagram of an embodiment of the present invention.

[0055] As shown in FIG. 5, the client device 10 and the wearable device 50 sense the user US1 during exercise. The type of exercise performed by the user US1 (hereinafter referred to as the "target exercise type") may be fixed or may be arbitrarily selected. In the latter case, the target exercise type may be determined by the user, a person who plans or instructs the exercise therapy of the user, or an algorithm. The target exercise type is stored in an exercise type database, which will be described later, and has exercise definition information regarding the definition of the exercise type. The user US1 is typically a person who undergoes exercise therapy, and is, for example, a participant in a (cardiac) rehabilitation program or an exercise instruction program. In the example of FIG. 5, the exercise type is gymnastics, but the exercise type may include any exercise (aerobic exercise or anaerobic exercise).

[0056] As an example, the camera 16 captures an external view (e.g., the whole body) of the user US1 during exercise from the front or oblique front at a distance of, for example, about 2 m. The camera 16 may be installed at an appropriate height using a tripod or other height adjustment means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. It is also possible to generate three-dimensional video data by combining, for example, video data (two-dimensional) generated by the camera 16 and, for example, depth data generated by the depth sensor 17.

[0057] The acceleration sensor 57 of the wearable device 50 measures the acceleration of the user US1 while the user US1 is exercising, and transmits the measurement result to the client device 10.

[0058] The client device 10 acquires various sensing data and analyzes the sensing data. As an example, the client device 10 may refer to video data acquired from the camera 16 to analyze the body movement of the user US1 during exercise (particularly, the movement of the skeleton or other feature points over multiple points of time, or the state of the skeleton or other feature points at a single point of time). The client device 10 may further refer to depth data acquired from the depth sensor 17 or acceleration data acquired from the acceleration sensor 57 to analyze the body movement of the user US1 during exercise.

[0059] The client device 10 judges whether the user US1 is correctly performing the motion defined as the target exercise type based on the analysis result of the sensing data and the definition information of the target exercise type. Then, the client device 10 feeds back information according to the judgment result to the user US1. As a result, the user can be guided to optimize the load (exercise intensity) on the user US1. For example, if the user US1 is performing the target exercise type with a form that places a load on the user US1 or at a faster pace than the defined form, the user can be prompted to change the form or pace to reduce the exercise intensity to an appropriate level. On the other hand, if the user US1 is performing the target exercise type with a form that places a load on the user US1 or at a slower pace than the defined form, the user can be prompted to change the form or pace to increase the exercise intensity to an appropriate level. Therefore, according to this information processing system 1, it is possible to control the exercise intensity even for exercise types that have a large variation in exercise intensity depending on the method, such as gymnastics, dance, and strength training, so that the types of exercise available for effective and safe exercise therapy can be enriched.

[0060] (3) Database The databases of this embodiment will be described below. The following databases are stored in the storage device 11 or the storage device 31.

[0061] (3-1) Sports event database The exercise event database of this embodiment will be described below. Fig. 6 is a diagram showing the data structure of the exercise event database of this embodiment.

[0062] The exercise type database stores exercise type information. The exercise type information is information about the exercise type available to the user. Here, in this embodiment, the exercise types include exercise types that can be performed without using a device capable of adjusting the exercise load, such as gymnastics, bodyweight training, dancing, walking, running, and treadmills. These exercise types are rich in variation because they include types performed in a standing position. In addition, the exercise types available in this embodiment may include types consisting of repeated exercises in a specific range of motion without using a machine (especially a machine with a function for fixing the range of motion). Furthermore, in this embodiment, even an exercise type that is generally considered to be one type, such as squats, is subdivided and defined as multiple exercise types with different exercise intensities by providing differences in variations such as form (e.g., positional relationship between parts, angle of parts, or range of motion of parts moved during exercise), pace, number of repetitions, or time or number of rests. However, the exercise type of this embodiment may further include an exercise type performed using a device capable of adjusting the exercise load, such as an ergometer or strength training using training equipment.

[0063] 6, the exercise event database includes an "ID" field, a "name" field, an "exercise intensity" field, and an "exercise definition" field. Each field is associated with the others.

[0064] The "ID" field stores an exercise type ID. The exercise type ID is information for identifying the exercise type corresponding to the record.

[0065] The "name" field stores information about the name of the exercise type. The exercise type name information is information about the name of the exercise type corresponding to the record.

[0066] In the "exercise intensity" field, exercise intensity information is stored. The exercise intensity information is information on the standard exercise intensity of the exercise type corresponding to the record. The standard exercise intensity may mean, for example, the exercise intensity when a person with standard physical functions performs the corresponding exercise type as defined. As an example, such exercise intensity may be derived by actually measuring the exercise intensity (e.g., average oxygen consumption) when one or more people perform the corresponding exercise type as defined, for example, by performing an exhaled gas analysis (or estimating by analyzing a video of a user exercising), and statistically processing the results (e.g., averaging). Here, whether or not a person is performing the exercise type as defined may be determined by applying this embodiment, or may be determined by a human being.

[0067] The standard exercise intensity may be derived by, for example, measuring (or estimating) the exercise intensity in each section constituting the corresponding exercise type when one or more people perform the corresponding exercise type as defined, applying the measurement results of each section for each person to a predetermined calculation formula, and further performing statistical processing (e.g., averaging) of the values ​​obtained by this calculation formula among the people. Here, a section is a constituent unit of an exercise type, and when an exercise type is composed of multiple movement patterns, each movement pattern may correspond to a section. For example, when the exercise type is dance, each choreography may correspond to a section. Also, when the exercise type is squats, the transition from a standing state to a crouching state and the transition from a crouching state to a standing state may each correspond to a section.

[0068] The "exercise definition" field stores exercise definition information. The exercise definition information is information regarding the definition of the exercise event corresponding to the corresponding record. The exercise definition information may include information regarding at least one of the following elements. - The pace of movement (for example, the time required for one cycle of movement, the angle, angular velocity, acceleration, or change in these of the joint related to the part being moved, the velocity, acceleration, or change in these of the part, or a combination of these) - Tolerance of positional relationships between parts (e.g. foot width, etc.) - The allowable range of angles of parts (which may be one or more) (for example, the direction of the knee, the angle between the upper arm and the forearm) The allowable range of motion of the body parts (which may be one or more) that are moved during exercise (for example, the range of motion of each body part during one cycle of movement) Number of reps Break time Exercise load (for example, the magnitude of the external load set when exercising using an ergometer or other equipment that allows the exercise load to be adjusted)

[0069] Even for an exercise that is generally recognized as a single event (e.g., squats), multiple exercise events with slightly different exercise intensities can be defined by specifying and defining the details of the pace, form (particularly, range of motion), number of repetitions, rest time, and exercise load. For example, multiple exercise events with differences of 0.2 METs can be defined for an exercise that is generally recognized as a single event.

[0070] (3-2) User profile database The user profile database of this embodiment will now be described with reference to Fig. 7, which is a diagram showing the data structure of the user profile database of this embodiment.

[0071] The user profile database stores user profile information. The user profile information is information about a profile of a user (that is, a person who exercises) of the information processing system 1.

[0072] 7, the user profile database includes an "ID" field, a "name" field, a "target strength" field, and a "body" field. Each field is associated with the others.

[0073] The "ID" field stores a user ID. The user ID is information for identifying the user corresponding to the corresponding record.

[0074] The "name" field stores user name information. The user name information is information about the name of the user corresponding to the corresponding record (for example, name, account name, etc.).

[0075] In the "target intensity" field, target intensity information (an example of "predetermined exercise intensity") is stored. The target intensity information is information about a target value of exercise intensity (for example, oxygen consumption, energy consumption, heart rate, or a combination thereof) set for the user corresponding to the corresponding record. As an example, the target intensity information is specified by a person (for example, a doctor) who plans or instructs exercise therapy based on the results of measuring the exercise tolerance of the user by CPX when using an ergometer (for example, oxygen consumption and heart rate at the anaerobic metabolic threshold (AT)). However, CPX is not essential, and the target value may be specified at the doctor's discretion. As another example, the target intensity information is determined by an algorithm based on the results of measuring the exercise tolerance of the user by, for example, CPX. In order to perform aerobic exercise more safely and effectively, it is preferable to perform exercise at an intensity near the anaerobic metabolic threshold, so the target value of exercise intensity is, for example, an exercise intensity corresponding to the anaerobic metabolic threshold, but is not limited to this. In addition, an ergometer is usually used as the exercise type when measuring CPX, but when a treadmill is used, the oxygen consumption at the anaerobic metabolic threshold is about 1.2 to 1.3 times that when an ergometer is used. This is considered to be due to the fact that the total muscle mass used in the treadmill exceeds the total muscle mass used in the ergometer. Therefore, as a first example, the target value based on the result measured by CPX when using the ergometer may be corrected to about 1.2 to 1.3 times, or a value obtained by further subtracting a predetermined value (for example, 1 METs) may be used as the target intensity information. As a second example, the target value based on the result measured by CPX when using the ergometer may be used as the target intensity information without correction. This makes it possible to prevent the target value from becoming excessively high when an exercise type in which the total muscle mass used is relatively small, such as an ergometer, is selected. As a third example, the target value for each exercise type may be corrected, for example, by a coefficient according to the total muscle mass used.

[0076] In addition, a doctor may set an upper limit on exercise intensity for a user (an example of exercise prescription). In this case, the user is not permitted to select an exercise type that exceeds the upper limit of the prescribed exercise intensity. For exercise prescription, a UI (User Interface) screen for exercise prescription may be displayed on the display of the terminal used by the doctor. Such a UI screen may include, for example, the following information: User CPX data -Display area for sample videos of multiple selectable sports Here, the display area of ​​the sample video of each exercise type may be arranged according to the exercise intensity information corresponding to the exercise type. For example, when 3.6 METs is recommended as the upper limit based on the CPX data, a group of display areas of sample videos of exercise types corresponding to 3.4 METs, a group of display areas of sample videos of exercise types corresponding to 3.6 METs, and a group of display areas of sample videos of exercise types corresponding to 3.8 METs may be arranged on the UI screen. Note that the information of the exercise type (e.g., sample videos) may be arranged in units smaller than 0.2 METs or larger. When the doctor selects one of the display areas, the exercise intensity associated with the corresponding exercise type is set as the upper limit.

[0077] In addition, the raising or lowering of the upper limit specified by the doctor through the exercise prescription may be performed by a medical professional under the supervision of a doctor during regular (e.g., every two weeks) medical guidance or doctor rounds.

[0078] The "physical" field stores physical information. The physical information is information about the body (functions) of the user corresponding to the corresponding record. As an example, the physical information may include information about the user's age, sex, weight, height, illnesses, etc.

[0079] Additionally, the user profile database may store the following information: Information indicating the person who planned or instructed the user's exercise regimen Information identifying the user's doctor

[0080] (4) Information processing The information processing of this embodiment will be described.

[0081] (4-1) Exercise support processing The exercise support processing of this embodiment will be described with reference to the flowchart of FIG 8.

[0082] The exercise support process starts when, for example, any one of the following start conditions is met. The exercise support process was called by another process. The user or a person who plans or instructs the user's exercise therapy performs an operation to call up the exercise support process. The client device 10 enters a predetermined state (for example, a predetermined application is started). The appointed time has arrived. A certain amount of time has passed since a certain event.

[0083] As shown in FIG. 8, the client device 10 executes acquisition of sensing data (S110). Specifically, the client device 10 may start taking a video of the user exercising (hereinafter referred to as "user video") by enabling the operation of the camera 16. The client device 10 may also start measuring the distance from the depth sensor 17 to each part of the user exercising (hereinafter referred to as "user depth") by enabling the operation of the depth sensor 17.

[0084] Furthermore, the client device 10 may cause the wearable device 50 to start measuring the heart rate (hereinafter referred to as the "user's heart rate") using the heart rate sensor 56. Furthermore, the client device 10 may enable any sensor of the client device 10 or the wearable device 50 (for example, the acceleration sensor 19 or the acceleration sensor 57).

[0085] The client device 10 acquires sensing data from each sensor when the user is performing a target exercise. The target exercise is selected from a plurality of predetermined exercises, for example, exercises registered in an exercise database (FIG. 6). Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 may acquire user video data from the camera 16, acquire user depth data from the depth sensor 17, acquire user heart rate data from the wearable device 50, and acquire user acceleration data relating to the acceleration of the user (hereinafter, “user acceleration”) from at least one of the acceleration sensor 19 or the wearable device 50.

[0086] The client device 10 can repeatedly obtain the sensing data (S110) before proceeding to the next step S111.

[0087] After step S110, the client device 10 performs an analysis of the sensing data (S111). Specifically, the client device 10 analyzes the sensing data acquired in step S110 to analyze the user's body movement during exercise. As an example, the client device 10 estimates the user's form or pace. The user's form may be the position or angle of one or more parts of the user's body at one time or multiple time points, the range of motion of the moving parts, or a combination thereof. The user's pace may be the speed, acceleration, angular velocity, or angular acceleration of one or more parts of the user's body (moving parts) at one time or multiple time points, the time required for one cycle of movement, or a combination thereof. The client device 10 mainly refers to the user video data to analyze the user's body movement during exercise. The client device 10 may refer to user depth data, user acceleration data, or a combination thereof in addition to or instead of the user video data.

[0088] The body movements (characteristics) analyzed by the client device 10 may differ depending on the target exercise type. For example, when an exercise type is performed in which the form for the arms is not defined, the client device 10 may omit analyzing the position and angle of the arms. The target exercise type may be specified by an argument (for example, an exercise type ID) given at the start of the exercise support process.

[0089] Information on the skeleton of a user during exercise (hereinafter referred to as "user skeleton information") is data such as feature quantities. The user skeleton information includes, for example, information on the position, speed, or acceleration of each part of the user's body (which may include information on changes in the parts of the muscles used by the user, or information on shaking of the user's trunk). The user skeleton information can be acquired by analyzing the skeleton of a user during exercise with reference to user video data (or user video data and user depth data). As an example, Vision, which is an SDK for iOS (registered trademark) 14, or other skeleton detection algorithms (e.g., OpenPose, PoseNet, MediaPipe Pose) can be used for skeleton analysis.

[0090] After step S111, if the target exercise event is being performed continuously, the client device 10 executes a determination of the accuracy of the movement (S112). Specifically, the client device 10 judges whether or not the user can perform the movement of the target exercise event accurately (i.e., as defined by the exercise definition information) based on the analysis result in step S111 and the exercise definition information of the target exercise event. The judgment may be a binary judgment (a two-level evaluation of accurate or inaccurate) or a multi-level judgment (an evaluation of three or more levels including accurate, inaccurate, or intermediate states). The judgment may be based on logic (e.g., a judgment formula) or may be made using a trained model.

[0091] As a first example of determining the accuracy of a movement (S112), the client device 10 determines whether the user's movement form is accurate based on whether the estimated user's movement form matches the form of the target exercise defined by the exercise definition information corresponding to the target exercise. For example, the client device 10 may determine whether the user's movement form is accurate based on at least one of the following: -Whether the range of motion of the body parts moving in the target exercise is within the allowable range defined by the exercise definition information -Whether the relative positions of the body parts involved in the target exercise are within the allowable range defined by the exercise definition information -Whether the angles of the body parts involved in the target exercise are within the allowable range defined by the exercise definition information

[0092] As a second example of determining the accuracy of the movement (S112), the client device 10 determines whether the pace of the user's movement is accurate based on whether the estimated pace of the user matches the pace of the target exercise defined by the exercise definition information corresponding to the target exercise. For example, the client device 10 may determine whether the pace of the user's movement is accurate based on at least one of the following: -Whether the time required for one cycle of the exercise is within the allowable range defined by the exercise definition information Whether the angle, angular velocity, angular acceleration, or the amount of change of these of the joint related to the body part moving in the target exercise, the velocity, acceleration, or the amount of change of these of the body part, or a combination of these, is within the allowable range defined by the exercise definition information The client device 10 may perform a composite judgment based on a plurality of parameters regarding the speed (i.e., rhythm) of the movement of the body part in the target exercise. For example, the client device 10 may judge whether the movement of the body part of the user is smooth or awkward based on a combination of parameters (angle, angular velocity, angular acceleration, or the amount of change thereof) of a plurality of joints related to the body part in the target exercise.

[0093] The third example of determining the accuracy of the movement (S112) is a combination of the above first and second examples.

[0094] If it is determined in step S112 that the user's motion is correct, the client device 10 re-executes acquisition of sensing data (S110). On the other hand, if it is determined in step S112 that the user's motion is not correct, the client device 10 generates feedback information (S113). Specifically, the client device 10 generates feedback information in accordance with the result of the determination in step S112.

[0095] As a first example of the generation of feedback information (S113), the client device 10 generates feedback information on a body part of the user related to the target exercise that is determined in step S112 not to conform to the form of the target exercise defined by the exercise definition information. The feedback information on the body part may include information indicating the body part, information indicating an improvement guideline for the movement of the body part, or a combination thereof. The feedback information may be expressed as an image (which may include a moving image), text, sound, or a combination thereof. For example, when the width of the feet is out of the allowable range, feedback information indicating the feet or feedback information urging the user to make the feet about shoulder width may be generated. When the angle of the back is out of the allowable range, feedback information indicating the back or feedback information urging the user to straighten the back may be generated. Also, when the range of motion of the arms is out of the allowable range, feedback information indicating the arms or feedback information urging the user to straighten the arms may be generated.

[0096] As a second example of generating feedback information (S113), the client device 10 generates feedback information that allows comparison of the form (correct form) of a target exercise defined by the exercise type information of the target exercise with the user's form. The correct form may be expressed by a numerical value, or may be expressed using a model video or a still image captured from the model video, or may be expressed by a combination of these. The user's form may be expressed by a numerical value, or may be expressed using a user video or a still image captured from the user video, or may be expressed by a combination of these.

[0097] As a third example of generating feedback information (S113), when the estimated range of motion of a body part of the user to be moved in the target exercise deviates from the allowable range, the client device 10 may select another exercise from a plurality of predetermined exercises in which the range of motion of the body part of the user falls within the corresponding allowable range (e.g., the requirement for the range of motion of the body part is less stringent).Then, the client device 10 generates feedback information indicating the selected another exercise.

[0098] As a fourth example of generating feedback information (S113), the client device 10 may include information on a comparison result between the estimated pace of the user and the pace of the target exercise event defined by the exercise definition information, or on an improvement guideline derived from the comparison result. For example, the feedback information may generate feedback information indicating that the pace of the user is inappropriate (e.g., slow, fast, or the speed of the movement of the body part is unnatural). Alternatively, the feedback information may include information encouraging the user to move faster or slower. The feedback information may also include information indicating a part where the pace is slow or fast or the speed of the movement is unnatural, information indicating a section where the pace is slow or fast or the speed of the movement of the body part is unnatural, etc.

[0099] As a fifth example of generating feedback information (S113), the client device 10 generates feedback information that allows comparison between the pace of a target exercise event (correct pace) defined by the exercise event information of the target exercise event and the user's pace. The correct pace may be expressed by a numerical value, or may be expressed using a model video or a still image captured from the model video, or may be expressed by a combination of these. The user's pace may be expressed by a numerical value, or may be expressed using a user video or a still image captured from the user video, or may be expressed by a combination of these.

[0100] As a sixth example of generating feedback information (S113), the client device 10 may select another exercise type from a plurality of predetermined exercise types in which the user's pace falls within the corresponding allowable range (e.g., the requirement for pace is less severe) when the estimated user's pace deviates from the allowable range of pace defined by the exercise type information for the target exercise type. Then, the client device 10 generates feedback information indicating the selected another exercise type.

[0101] The seventh example of the generation of feedback information (S113) is a combination of multiple of the above first to sixth examples.

[0102] After step S113, the client device 10 executes presentation of feedback information (S114). Specifically, the client device 10 presents the feedback information generated in step S113 to the user. As a first example of presenting the feedback information (S114), the client device 10 displays a screen based on the feedback information on the display 15. As a second example of presenting the feedback information (S114), the client device 10 outputs a sound based on the feedback information from a speaker. As a third example of presenting the feedback information (S114), the client device 10 transmits the feedback information to the wearable device 50, and the wearable device 50 outputs a screen, sound, light, or vibration based on the feedback information. A fourth example of presenting the feedback information (S114) is a combination of multiple of these first to third examples. After step S114, the client device 10 may return to acquiring sensing data (S110) or may end the exercise support process of Fig. 8. For example, when the number or frequency of the movements determined to be inaccurate reaches or exceeds a threshold, the client device 10 may end the exercise support process of Fig. 8 to stop or restart the current target exercise event, or to select another exercise event.

[0103] After step S111, if the target exercise event is completed, the client device 10 executes generation of user data (S115). Specifically, the client device 10 generates user data based on the sensing data acquired in step S110, the analysis result in step S111, the determination result in step S112, or a combination of these. The user data may include at least one of the following: The sensing data acquired in step S110 (for example, user video data, user depth data, user heart rate data, or user acceleration data) Data obtained by processing the data acquired in step S110 Analysis results in step S111 The result of determination in step S112 Information that can identify the target sport - Information showing the user's subjective assessment of the exercise intensity of the completed target exercise event (hereinafter referred to as "perceived exercise intensity").

[0104] After step S115, the client device 10 executes sending of user data (S116). Specifically, the client device 10 transmits the user data generated in step S115 to the server 30. The server 30 can perform processes such as estimating exercise intensity and determining the next exercise type recommended to the user based on the user data. Description of the processes on the server 30 side will be omitted.

[0105] In the example of FIG. 8, the client device 10 generates user data (S115) and transmits the user data (S116) after the target exercise event is completed, but the client device 10 may repeatedly generate user data (S115) and transmit the user data (S116) while the target exercise event is being performed, and may omit the processes of generating user data (S115) and transmitting the user data (S116) if no processing is performed on the server 30 side.

[0106] (5) Summary As described above, the client device 10 of the present embodiment acquires sensing data on a user who performs a first exercise type among a plurality of predetermined exercise types, and analyzes the sensing data. Based on the analysis result of the sensing data and the exercise definition information that defines the first exercise type, the client device 10 judges whether the user is performing the first exercise type as defined, and presents information according to the judgment result to the user. This allows the user to be guided to optimize the load (exercise intensity) on the user. Therefore, it is possible to control the exercise intensity even for exercise types that have a large variation in exercise intensity depending on the method, such as gymnastics, dance, and strength training, so that the types of exercise available for effective and safe exercise therapy can be enriched.

[0107] The exercise definition information may define the form of the first exercise type. The client device 10 may estimate the user's form based on the analysis result of the sensing data, and determine whether the user is performing the first exercise type as defined based on whether the estimated user's form matches the form of the first exercise type defined by the exercise definition information. This makes it possible to prevent the user from being subjected to an excessive or insufficient load (exercise intensity) due to exercising in a form different from the defined form.

[0108] The client device 10 may present to the user information indicating the body parts related to the first exercise that are determined not to conform to the form of the first exercise defined by the exercise definition information, thereby allowing the user to recognize which body part's form should be improved.

[0109] The client device 10 may present to the user information indicating a guideline for improving the movement of a body part determined not to conform to the form of the first exercise defined by the exercise definition information among the body parts related to the first exercise type. This allows the user to recognize how to improve the form of which body part.

[0110] The client device 10 may present the user with information that allows the user to compare the form of the first exercise event defined by the exercise definition information with the user's form, thereby allowing the user to compare his or her own form with the correct form and recognize the gap between the correct form and the actual form.

[0111] The exercise definition information may define an allowable range of the range of motion of the body part to be moved in the first exercise type. The client device 10 may estimate the range of motion of the user's body part based on the analysis result of the sensing data, and determine whether the user is performing the first exercise type as defined based on whether the estimated range of motion of the user's body part is within the allowable range defined by the exercise definition information. This makes it possible to prevent the user from being subjected to an excessive or insufficient load (exercise intensity) due to exercising in a range of motion different from the defined range of motion.

[0112] When the range of motion of the body part of the user to be moved in the first exercise type deviates from the allowable range of the estimated range of motion of the body part of the user, the client device 10 may select a second exercise type from among a plurality of exercise types in which the range of motion of the body part of the user falls within the corresponding allowable range. The client device 10 may present information indicating the second exercise type to the user. This makes it possible to suggest an exercise type in which the range of motion that the user can handle is defined, making it easier to control the load (exercise intensity) on the user.

[0113] The exercise definition information may define the pace of the first exercise type. The client device 10 may estimate the pace of the first exercise type by the user based on the sensing data, and determine whether the user is performing the first exercise type as defined based on whether the estimated pace of the user matches the pace of the first exercise type defined by the exercise definition information. This makes it possible to prevent the user from being subjected to an excessive or insufficient load (exercise intensity) due to exercising at a pace different from the defined pace.

[0114] The client device 10 may present to the user information indicating that the user's pace is inappropriate, thereby allowing the user to recognize that his / her pace is slower or faster than the correct pace, or that the speed and speed of the movement of his / her body parts is unnatural.

[0115] The client device 10 may present the user with information indicating a guideline for improving the user's pace, thereby allowing the user to recognize how to improve the pace.

[0116] The client device 10 may present the user with information that allows the user to compare the pace of the first exercise defined by the exercise definition information with the estimated pace of the user, thereby allowing the user to compare his or her own pace with the correct pace and recognize the gap between the correct pace.

[0117] The exercise definition information may define an acceptable range of the pace of the first exercise type. When the estimated user's pace deviates from the acceptable range of the pace of the first exercise type, the client device 10 may select a second exercise type from among a plurality of exercise types in which the user's pace falls within the corresponding acceptable range. The client device 10 may present the second exercise type to the user. This makes it possible to suggest an exercise type in which a pace within the range that the user can handle is defined, making it easier to control the load (exercise intensity) on the user.

[0118] (6) Variations (6-1) Variation 1 The first modification is an example in which the score is calculated so that the more correctly the user performs the exercise, the higher the score. Fig. 9 is a flowchart of the scoring process of the first modification.

[0119] The scoring process starts, for example, when any of the following start conditions is met: The scoring process was invoked by another process. The user or a person who plans or instructs the user's exercise therapy performed an operation to call up the scoring process. The client device 10 enters a predetermined state (for example, a predetermined application is started). The appointed time has arrived. A certain amount of time has passed since a certain event.

[0120] Note that the first modified example can be combined with this embodiment. That is, both the generation and presentation of feedback information and scoring may be performed.

[0121] As shown in FIG. 9, the client device 10 acquires sensing data (S110) and analyzes the sensing data (S111) in the same manner as in FIG.

[0122] After step S111, if the target exercise event is being performed continuously, the client device 10 executes a determination of the accuracy of the movement (S112) in the same manner as in FIG.

[0123] After step S112, the client device 10 executes score calculation (S213). Specifically, the client device 10 calculates a score indicating how accurately the user is performing the target exercise event based on the result of the determination in step S112. For example, if the result of the determination in step S112 is positive, the client device 10 may increase the score. On the other hand, if the result of the determination in step S112 is negative, the client device 10 may decrease the score. The method of calculating the score can be determined arbitrarily.

[0124] After step S213, the client device 10 may return to acquiring sensing data (S110) or may end the scoring process of Fig. 9. For example, when the number or frequency of movements determined to be inaccurate reaches or exceeds a threshold, the client device 10 may end the scoring process of Fig. 9 to allow the user to stop or restart the current target exercise event, or to select another exercise event.

[0125] Step S213 may be performed each time step S112 is executed as shown in FIG. 9, or may be performed each time step S112 is executed multiple times based on the judgment results, or may be performed based on all the judgment results after the target exercise event is completed.

[0126] After step S111, if the target exercise event is completed, the client device 10 executes presentation of the score (S214). Specifically, the client device 10 presents to the user the score obtained by the most recently executed score calculation (S213). As a first example of score presentation (S214), the client device 10 displays a screen based on the score on the display 15. As a second example of score presentation (S214), the client device 10 outputs a sound based on the score from a speaker. As a third example of score presentation (S214), the client device 10 transmits the score to the wearable device 50, and the wearable device 50 outputs a screen, sound, light, or vibration based on the score. A fourth example of score presentation (S214) is a combination of multiple of these first to third examples.

[0127] Note that step S214 may be performed only once after the target exercise event is completed as shown in FIG. 9, or may be performed each time step S213 is performed, or may be performed each time step S213 is performed multiple times.

[0128] The scores that the user has obtained in each athletic event can be utilized in various ways as the user's achievements. For example, the scores can be used as a tool for managing the user's goals or for competition between users. Alternatively, the scores can be exchanged for goods, services, or assets of monetary value.

[0129] As described above, the client device 10 of the first modification repeats judgments while the user is performing the first exercise, calculates a score based on the results of the multiple judgments, and presents information corresponding to the score to the user. This motivates the user to perform the exercise correctly and encourages the optimization of the exercise intensity.

[0130] The score calculated by the first modification can also be applied to a video game in which the game progress is controlled according to the player's body movements. The video game may be a mini-game that can be played while the above-mentioned treatment app, rehabilitation app, or fitness app is being executed, or may be configured as a standalone app. As an example, the information processing system 1 may determine one of the following depending on the score. This can enhance the effect of the video game on improving the user's health. The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) offered to the user The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) provided to users Game parameters relating to the progression of a video game (e.g. game score, damage)

[0131] (7) Other modifications The storage device 11 may be connected to the client device 10 via a network NW. Each input device or output device may be integrated with the client device 10. The storage device 31 may be connected to the server 30 via the network NW. Each input device or output device may be integrated with the wearable device 50.

[0132] In the embodiment, the information processing system 1 is implemented by a client / server type system. However, the information processing system 1 may be implemented by a peer-to-peer type system or a standalone computer. As an example, the client device 10 may transmit sensing data (or an analysis result thereof) to the server 30, and the server 30 may determine the accuracy of the operation. The client device 10 and the wearable device 50 may be configured as a single device.

[0133] Each step of the above information processing can be executed by any of the client device 10, the server 30, and the wearable device 50. Furthermore, one or more steps of the above information processing may be executed using a trained model.

[0134] In the above description, an example has been given in which a user video is captured using the camera 16 of the client device 10. However, the user video may be captured using a camera other than the camera 16. An example has been given in which a user depth is measured using the depth sensor 17 of the client device 10. However, the user depth may be measured using a depth sensor other than the depth sensor 17.

[0135] Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited to the above embodiment. Furthermore, the above embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above embodiment and the modified examples can be combined. [Explanation of symbols]

[0136] 1: Information processing system 10: Client device 11:Storage device 12: Processor 13: Input / Output Interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 30: Server 31: Storage device 32 : Processor 33: Input / Output Interface 34: Communication interface 50: Wearable devices 51 :Storage device 52: Processor 53: Input / Output Interface 54: Communication interface 55: Display 56: Heart rate sensor 57: Acceleration sensor

Claims

1. Computer, A means for acquiring sensing data relating to a user performing a first exercise type among a plurality of predetermined exercise types; means for estimating a range of motion of a body part of the user by analyzing the sensing data; a means for determining whether the user is performing the first exercise type as defined based on whether the estimated range of motion of the user's body part is within an allowable range of the range of motion of the part to be moved in the first exercise type defined by exercise definition information corresponding to the first exercise type; a means for selecting a second exercise type from the plurality of exercise types in which the range of motion of the body part of the user falls within a corresponding allowable range when the estimated range of motion of the body part of the user to be moved in the first exercise type deviates from the allowable range of the range of motion of the body part of the user; A means for presenting information indicating the second exercise event to the user; A program that functions as a

2. A computer, A means for acquiring sensing data relating to a user performing a first exercise type among a plurality of predetermined exercise types; means for estimating a pace of the first exercise event performed by the user by analyzing the sensing data; a means for determining whether the user is performing the first exercise type as defined by exercise definition information corresponding to the first exercise type, based on whether the estimated pace of the first exercise type by the user is within an allowable range of the pace of the first exercise type defined by exercise definition information corresponding to the first exercise type; a means for selecting a second exercise type from the plurality of exercise types in which the pace of the user falls within a corresponding allowable range when the estimated pace of the user deviates from an allowable range of the pace of the first exercise type; A means for presenting information indicating the second exercise event to the user; A program that functions as a

3. The determining means repeatedly performs the determination while the user is performing the first exercise event; causing the computer to function as a means for calculating a score based on a plurality of results of the judgments; The presenting means presents information corresponding to the score to the user. The program according to claim 1.

4. A means for acquiring sensing data regarding a user who is performing a first exercise event among a plurality of predetermined exercise events; means for estimating a range of motion of a body part of the user by analyzing the sensing data; a means for determining whether the user is performing the first exercise type as defined by determining whether the estimated range of motion of the user's body part is within an allowable range of the range of motion of the body part to be moved in the first exercise type defined by exercise definition information corresponding to the first exercise type; a means for selecting a second exercise type from the plurality of exercise types in which the range of motion of the body part of the user falls within a corresponding allowable range when the estimated range of motion of the body part of the user to be moved in the first exercise type deviates from the allowable range of the range of motion of the body part of the user; means for presenting information indicating the second exercise event to the user; An information processing device comprising:

5. The computer acquiring sensing data relating to a user performing a first exercise event among a plurality of predetermined exercise events; estimating a range of motion of the user's body part by analyzing the sensing data; determining whether the user is performing the first exercise type as defined by determining whether the estimated range of motion of the user's body part is within an allowable range of the range of motion of the part to be moved in the first exercise type defined by exercise definition information corresponding to the first exercise type; a means for selecting a second exercise type from the plurality of exercise types in which the range of motion of the body part of the user falls within a corresponding allowable range when the estimated range of motion of the body part of the user to be moved in the first exercise type deviates from the allowable range of the range of motion of the body part of the user; presenting information indicating the second exercise event to the user; A method for performing.

6. A system including a plurality of information processing devices, A means for acquiring sensing data regarding a user who is performing a first exercise event among a plurality of predetermined exercise events; means for estimating a range of motion of a body part of the user by analyzing the sensing data; a means for determining whether the user is performing the first exercise type as defined by determining whether the estimated range of motion of the user's body part is within an allowable range of the range of motion of the body part to be moved in the first exercise type defined by exercise definition information corresponding to the first exercise type; a means for selecting a second exercise type from the plurality of exercise types in which the range of motion of the body part of the user falls within a corresponding allowable range when the estimated range of motion of the body part of the user to be moved in the first exercise type deviates from the allowable range of the range of motion of the body part of the user; means for presenting information indicating the second exercise event to the user; A system comprising:

Images