Information processing device, method, program, and system

Abstract

To provide a technique for fulfilling exercise types that can be utilized for effective and safe exercise therapy.SOLUTION: A program allows a computer to function as means for acquiring sensing data regarding a user who performs a first exercise type from among multiple predetermined exercise types, means for analyzing sensing data, means for determining whether the user is performing the first exercise type as defined on the basis of an analysis result of sensing data, and exercise definition information defining a first exercise type, and means for presenting information corresponding to a determination result to the user.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a method, a program, and a system. [Background technology]

[0002] Exercise therapy, which utilizes exercise for the treatment and prevention of disorders and diseases, is becoming increasingly important. For example, cardiac rehabilitation aims to help patients with heart disease regain their physical strength and confidence, return to a comfortable home and social life, and prevent recurrence of heart disease or re-hospitalization through a comprehensive activity program that includes exercise therapy. Exercise therapy focuses on aerobic exercise, such as walking, jogging, cycling, and aerobics. To perform aerobic exercise safely and effectively, patients should preferably exercise at an intensity near their anaerobic threshold (AT).

[0003] The anaerobic metabolic threshold is an example of an evaluation index of exercise tolerance and corresponds to a change point in cardiopulmonary function, i.e., an exercise intensity near the boundary between aerobic exercise and anaerobic exercise. The anaerobic metabolic threshold is generally determined by a cardiopulmonary exercise test (CPX test), in which a test subject is subjected to a gradually increasing exercise load while exhaled gas is collected and analyzed (see Non-Patent Document 1). In a CPX test, the anaerobic metabolic threshold is determined based on the results measured by exhaled gas analysis (e.g., oxygen intake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof). In addition to the anaerobic metabolic threshold, a CPX test can also determine the maximum oxygen intake, which corresponds to an exercise intensity near the maximum exercise tolerance.

[0004] To ensure the effectiveness and safety of exercise therapy, it is important to properly manage the exercise intensity of the subject.

[0005] Patent Document 1 discloses a technical idea of acquiring the pace and pitch of a user's running, determining whether or not it is within an acceptable range, and performing a notification operation based on the determination result. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-045782 [Non-patent literature]

[0007] [Non-Patent Document 1] Muneyasu Saito, Cardiac Rehabilitation, Physical Therapy, 1997, Vol. 24, No. 8, pp. 414-418 Summary of the Invention [Problem to be solved by the invention]

[0008] The technical idea of Patent Document 1 cannot be simply applied to exercises other than running. Furthermore, even if a specific exercise type is designated to subject a specific exercise intensity, it is not easy for the subject to accurately understand and practice the ideal movements of that exercise type down to the smallest detail. Therefore, the subject may be subjected to a load that exceeds the expected load, or conversely, may not be subjected to the expected load. In other words, the exercise intensity may be too high or too low.

[0009] An object of the present disclosure is to provide a technique for expanding the range of exercise types available for effective and safe exercise therapy. [Means for solving the problem]

[0010] A program according to one embodiment of the present disclosure causes a computer to function as a means for acquiring sensing data relating to a user performing a first exercise type among a plurality of predetermined exercise types, a means for analyzing the sensing data, a means for determining whether the user is performing the first exercise type as defined based on the analysis results of the sensing data and exercise definition information that defines the first exercise type, and a means for presenting information to the user according to the results of the determination. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing the configuration of a client device according to the present embodiment. [Figure 3] FIG. 2 is a block diagram showing the configuration of a server according to the present embodiment. [Figure 4] FIG. 1 is a block diagram showing the configuration of a wearable device according to an embodiment of the present invention. [Figure 5] FIG. 1 is an explanatory diagram of one aspect of the present embodiment. [Figure 6] FIG. 2 is a diagram showing the data structure of an exercise event database according to the present embodiment. [Figure 7] FIG. 2 is a diagram showing the data structure of a user profile database according to the present embodiment. [Figure 8] 10 is a flowchart of an exercise support process according to the present embodiment. [Figure 9] 10 is a flowchart of a scoring process according to the first modification. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0013] (1) Information processing system configuration The configuration of the information processing system will now be described with reference to Fig. 1, which is a block diagram showing the configuration of the information processing system according to this embodiment.

[0014] As shown in FIG. 1, the information processing system 1 includes a client device 10, a server 30, and a wearable device 50.

[0015] Here, the number of client devices 10 and wearable devices 50 varies depending on, for example, the number of users. Therefore, the number of client devices 10 and wearable devices 50 may each be two or more. Furthermore, a terminal of a person who plans or instructs exercise therapy may also be included in the information processing system 1. The person who plans or instructs exercise therapy may include, for example, a medical professional (e.g., a doctor, nurse, pharmacist, physical therapist, occupational therapist, or clinical laboratory technician), a nutritionist, or a trainer.

[0016] The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW. The client device 10 and the wearable device 50 are connected via a wireless channel using, for example, Bluetooth (registered trademark) technology.

[0017] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0018] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a server computer.

[0019] The wearable device 50 is an example of an information processing device that can be worn on the user's body (for example, on the arm, hand, or head).

[0020] (1-1) Client device configuration The configuration of the client device will now be described with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0021] 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0022] The storage device 11 is configured to store programs and data, and is, for example, a combination of a read-only memory (ROM), a random access memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0023] The programs include, for example, the following programs: OS (Operating System) programs · Programs for applications that process information (e.g., web browsers, therapy apps, rehabilitation apps, or fitness apps) Here, the diseases that are the target of therapeutic or rehabilitation apps are, for example, heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), obesity, and other diseases in which exercise may contribute to improving symptoms.

[0024] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0025] The processor 12 is a computer that implements the functions of the client device 10 by running a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Gate Array)

[0026] The input / output interface 13 is configured to acquire information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10.

[0027] The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0028] The communication interface 14 is configured to control communication between the client device 10 and external devices (eg, another client device 10, a server 30, and a wearable device 50). Specifically, the communication interface 14 may include a module (e.g., a WiFi module, a mobile communication module, or a combination thereof) for communication with the server 30. The communication interface 14 may include a module (e.g., a Bluetooth module) for communication with the wearable device 50.

[0029] The display 15 is configured to display an image (a still image or a moving image). The display 15 is, for example, a liquid crystal display or an organic EL display.

[0030] The camera 16 is configured to take pictures and generate image signals.

[0031] The depth sensor 17 is, for example, a light detection and ranging (LIDAR) sensor. The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to a surrounding object (for example, a user).

[0032] The microphone 18 is configured to receive sound waves and generate sound signals, and is preferably placed near the user's body (particularly the respiratory tract) as an earphone microphone, for example.

[0033] The acceleration sensor 19 is configured to detect acceleration.

[0034] (1-2) Server configuration The configuration of the server will now be described with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0035] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0036] The storage device 31 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0037] The programs include, for example, the following programs: OS programs Application programs that perform information processing

[0038] The data includes, for example, the following data: Databases referenced in information processing - Results of information processing

[0039] The processor 32 is a computer that implements the functions of the server 30 by running a program stored in the storage device 31. The processor 32 is, for example, at least one of the following: ·CPU GPU ASIC FPGA

[0040] The input / output interface 33 is configured to obtain information (for example, a user's instruction) from an input device connected to the server 30 and to output information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0041] The communication interface 34 is configured to control communications between the server 30 and an external device (eg, the client device 10).

[0042] (1-3) Wearable device configuration The configuration of the wearable device will now be described with reference to Figure 4. Figure 4 is a block diagram showing the configuration of the wearable device of this embodiment.

[0043] 4, the wearable device 50 includes a storage device 51, a processor 52, an input / output interface 53, and a communication interface 54. The wearable device 50 is connected to a display 55, a heart rate sensor 56, and an acceleration sensor 57.

[0044] The storage device 51 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0045] The programs include, for example, the following programs: OS programs · Programs for applications that process information (e.g., therapeutic, rehabilitation, or fitness apps)

[0046] The data includes, for example, the following data: Databases referenced in information processing - Results of information processing

[0047] The processor 52 is a computer that executes the programs stored in the storage device 51 to realize the functions of the wearable device 50. The processor 52 is, for example, at least one of the following: ·CPU GPU ASIC FPGA

[0048] The input / output interface 53 is configured to acquire information (e.g., user instructions, sensing results) from an input device connected to the wearable device 50, and to output information (e.g., images, commands) to an output device connected to the wearable device 50.

[0049] The input device is, for example, a heart rate sensor 56, an acceleration sensor 57, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 55, a speaker, or a combination thereof.

[0050] The communication interface 54 is configured to control communication between the wearable device 50 and an external device (eg, the client device 10). Specifically, the communication interface 54 may include a module for communication with the client device 10 (eg, a Bluetooth module).

[0051] The display 55 is configured to display an image (still image or moving image). The display 55 is, for example, a liquid crystal display or an organic EL display.

[0052] The heart rate sensor 56 is configured to measure the heart rate and generate a sensing signal. As an example, the heart rate sensor 56 measures the heart rate using an optical measurement technique.

[0053] The acceleration sensor 57 is configured to detect acceleration.

[0054] (2) One aspect of the embodiment An example of this embodiment will now be described with reference to Fig. 5, which is an explanatory diagram of this example.

[0055] As shown in FIG. 5, the client device 10 and the wearable device 50 sense the user US1 while he or she is exercising. The type of exercise performed by the user US1 (hereinafter referred to as the "target exercise type") may be fixed or may be arbitrarily selected. In the latter case, the target exercise type may be determined by the user, a person planning or instructing the user's exercise therapy, or an algorithm. Exercise definition information regarding the definition of the target exercise type is stored in an exercise type database (described later). The user US1 is typically a person undergoing exercise therapy, such as a participant in a (cardiac) rehabilitation program or an exercise instruction program. In the example of FIG. 5, the exercise type is gymnastics, but the exercise type may include any exercise (aerobic exercise or anaerobic exercise).

[0056] As an example, the camera 16 captures an appearance (e.g., the entire body) of the user US1 while exercising from a front or oblique front view at a distance of, for example, about 2 m. The camera 16 may be installed at an appropriate height using a tripod or other height adjustment means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. It is also possible to generate three-dimensional video data by combining, for example, video data (two-dimensional) generated by the camera 16 with, for example, depth data generated by the depth sensor 17.

[0057] The acceleration sensor 57 of the wearable device 50 measures the acceleration of the user US1 while the user US1 is exercising and transmits the measurement result to the client device 10.

[0058] The client device 10 acquires various types of sensing data and analyzes the sensing data. As an example, the client device 10 may refer to video data acquired from the camera 16 to analyze the body movements of the user US1 during exercise (particularly, the movements of the skeleton or other feature points over multiple points in time, and the states of the skeleton or other feature points at a single point in time). The client device 10 may further refer to depth data acquired from the depth sensor 17 or acceleration data acquired from the acceleration sensor 57 to analyze the body movements of the user US1 during exercise.

[0059] The client device 10 determines whether the user US1 is correctly performing the movements defined as the target exercise type based on the analysis results of the sensing data and the definition information of the target exercise type. The client device 10 then provides feedback to the user US1 based on the determination result. As a result, the user US1 can be guided to optimize the load (exercise intensity) on the user US1. For example, if the user US1 is performing the target exercise type with a form that places more strain on the user US1 or at a faster pace than the defined form, the user US1 can be prompted to change his / her form or pace to reduce the exercise intensity to an appropriate level. On the other hand, if the user US1 is performing the target exercise type with a form that places less strain on the user US1 or at a slower pace than the defined form, the user US1 can be prompted to change his / her form or pace to increase the exercise intensity to an appropriate level. Therefore, this information processing system 1 makes it possible to control the exercise intensity even for exercise types that vary greatly in exercise intensity depending on the method, such as gymnastics, dance, or strength training. This allows for the expansion of available exercise types for effective and safe exercise therapy.

[0060] (3) Database The databases of this embodiment will be described below. The following databases are stored in the storage device 11 or the storage device 31.

[0061] (3-1) Sports event database The exercise event database of this embodiment will be described below with reference to Fig. 6, which is a diagram showing the data structure of the exercise event database of this embodiment.

[0062] The exercise type database stores exercise type information. The exercise type information is information about exercise types available to the user. In this embodiment, the exercise types include exercise types that can be performed without using devices with adjustable exercise loads, such as gymnastics, bodyweight training, dancing, walking, running, and treadmills. These exercise types are rich in variety because they include types performed in a standing position. Furthermore, the exercise types available in this embodiment may also include types consisting of repeated exercises within a specific range of motion without using a machine (especially a machine with a function to fix the range of motion). Furthermore, in this embodiment, even an exercise type that is generally considered to be a single type, such as squats, is subdivided and defined as multiple exercise types with different exercise intensities by varying the form (e.g., the positional relationship between body parts, the angle of the body parts, the range of motion of the body parts moved during exercise, etc.), pace, number of repetitions, or the duration or number of rest periods. However, the exercise types of this embodiment may further include exercise types performed using equipment capable of adjusting the exercise load, such as an ergometer or strength training using training equipment.

[0063] 6, the exercise event database includes an "ID" field, a "name" field, an "exercise intensity" field, and an "exercise definition" field. Each field is associated with the others.

[0064] The "ID" field stores an exercise type ID. The exercise type ID is information that identifies the exercise type corresponding to the record.

[0065] The "Name" field stores information about the name of the exercise type. The exercise type name information is information about the name of the exercise type corresponding to the record.

[0066] The "exercise intensity" field stores exercise intensity information. The exercise intensity information is information related to the standard exercise intensity of the exercise type corresponding to the record. Standard exercise intensity may refer to the exercise intensity when, for example, a person with standard physical function performs the corresponding exercise type as defined. As an example, such exercise intensity may be derived by actually measuring the exercise intensity (e.g., average oxygen consumption) when one or more people perform the corresponding exercise type as defined, for example, by performing exhaled gas analysis (or estimating it by analyzing video footage of the user exercising), and then statistically processing the results (e.g., averaging). Here, whether a person is performing the exercise type as defined may be determined by applying this embodiment or by a human.

[0067] Note that the standard exercise intensity may be derived, for example, by measuring (or estimating) the exercise intensity of one or more people performing a corresponding exercise type as defined for each section of the exercise type, applying the measurement results for each section to a predetermined calculation formula for each person, and then statistically processing (e.g., averaging) the values obtained by this calculation formula across people. Here, a section is a component unit of an exercise type, and if the exercise type consists of multiple movement patterns, each movement pattern may correspond to a section. For example, if the exercise type is dance, each choreography may correspond to a section. Also, if the exercise type is squats, the transition from a standing position to a crouching position and the transition from a crouching position to a standing position may each correspond to a section.

[0068] The "exercise definition" field stores exercise definition information. The exercise definition information is information about the definition of the exercise event corresponding to the record. The exercise definition information can include information about at least one of the following elements: - The pace of movement (for example, the time required for one cycle of movement, the angle, angular velocity, acceleration, or the amount of change of these, of the joint related to the part being moved, the speed, acceleration, or the amount of change of these, of the part in question, or a combination of these) - Tolerance of positional relationships between parts (e.g., foot width) - The allowable range of angles (for example, the direction of the knee, the angle between the upper arm and the forearm) of the part (which may be one or more) The range of motion (for example, the range of movement of each part during one cycle of movement) of the part (which may be one or more) that moves during exercise Number of reps Break time Exercise load (for example, the magnitude of the external load set when exercising using an ergometer or other device that allows for adjustable exercise load)

[0069] Even for an exercise that is generally recognized as a single event (such as squats), multiple exercise events with slightly different exercise intensities can be defined by specifying details of pace, form (particularly range of motion), number of repetitions, rest time, and exercise load. For example, for an exercise that is generally recognized as a single event, multiple exercise events with differences in exercise intensity of 0.2 METs can be defined.

[0070] (3-2) User profile database The user profile database of this embodiment will now be described with reference to Fig. 7, which shows the data structure of the user profile database of this embodiment.

[0071] The user profile database stores user profile information, which is information relating to the profile of a user of the information processing system 1 (i.e., a person who exercises).

[0072] 7, the user profile database includes an "ID" field, a "name" field, a "target strength" field, and a "body" field. Each field is associated with the others.

[0073] The "ID" field stores a user ID. The user ID is information that identifies the user corresponding to the corresponding record.

[0074] The "Name" field stores user name information. The user name information is information about the name of the user corresponding to the record (for example, name, account name, etc.).

[0075] The "target intensity" field stores target intensity information (an example of a "predetermined exercise intensity"). The target intensity information is information about a target value of exercise intensity (e.g., oxygen consumption, energy consumption, heart rate, or a combination thereof) set for the user corresponding to the record. As an example, the target intensity information is specified by a person (e.g., a doctor) who plans or instructs exercise therapy based on the results of measuring the user's exercise tolerance, for example, by CPX while using an ergometer (e.g., oxygen consumption and heart rate at the anaerobic threshold (AT)). However, CPX is not required, and the target value may be specified at the doctor's discretion. As another example, the target intensity information is determined by an algorithm based on the results of measuring the user's exercise tolerance, for example, by CPX. In order to perform aerobic exercise more safely and effectively, it is preferable to exercise at an intensity near the anaerobic threshold. Therefore, the target value of exercise intensity is, for example, but not limited to, an exercise intensity corresponding to the anaerobic threshold. While an ergometer is typically used as the exercise type for CPX measurement, the oxygen consumption at the anaerobic threshold when using a treadmill, for example, is approximately 1.2 to 1.3 times that when using an ergometer. This is thought to be because the total muscle mass used on the treadmill exceeds the total muscle mass used on the ergometer. Therefore, as a first example, the target intensity information may be a value obtained by correcting the target value based on the results measured by CPX when using an ergometer to approximately 1.2 to 1.3 times the original value, or by further subtracting a predetermined value (e.g., 1 MET). As a second example, the target intensity information may be a value obtained by correcting the target value based on the results measured by CPX when using an ergometer without correction. This prevents the target value from becoming excessively high when selecting an exercise type that uses a relatively small amount of muscle mass, such as an ergometer. As a third example, the target value for each exercise type may be corrected, for example, using a coefficient corresponding to the total muscle mass used.

[0076] The doctor may also set an upper limit on the exercise intensity for the user (an example of exercise prescription). In this case, the user is not permitted to select an exercise type that exceeds the prescribed upper limit of exercise intensity. For exercise prescription, a UI (User Interface) screen for exercise prescription may be displayed on the display of the terminal used by the doctor. Such a UI screen may include, for example, the following information: User CPX data -Display area for sample videos of multiple exercises that can be selected Here, the display areas of the sample videos for each exercise type may be arranged according to the exercise intensity information corresponding to the exercise type. For example, if 3.6 METs is recommended as the upper limit based on the CPX data, a group of display areas of sample videos for exercise types corresponding to 3.4 METs, a group of display areas of sample videos for exercise types corresponding to 3.6 METs, and a group of display areas of sample videos for exercise types corresponding to 3.8 METs may be arranged on the UI screen. Note that the information on exercise types (e.g., sample videos) may be arranged in units smaller than or larger than 0.2 METs. When the doctor selects one of the display areas, the exercise intensity associated with the corresponding exercise type is set as the upper limit.

[0077] In addition, the upper limit specified by a doctor through exercise prescription may be raised or lowered by a medical professional under the supervision of a doctor during regular (for example, every two weeks) medical guidance or doctor rounds.

[0078] The "Physical" field stores physical information. The physical information is information about the body (functions) of the user corresponding to the record. As an example, the physical information may include information about the user's age, sex, weight, height, illnesses, etc.

[0079] Additionally, the user profile database may store the following information: Information indicating the person who planned or instructed the user's exercise regimen Information indicating the user's doctor

[0080] (4) Information processing The information processing of this embodiment will be described.

[0081] (4-1) Exercise support processing The exercise support process of this embodiment will be described with reference to Fig. 8, which is a flowchart of the exercise support process of this embodiment.

[0082] The exercise support process starts when, for example, any of the following start conditions is met. The exercise support process was called by another process. The user or the person who plans or instructs the user's exercise therapy performs an operation to call up the exercise support process. The client device 10 enters a predetermined state (for example, a predetermined application is started). The appointed date and time has arrived. A certain amount of time has passed since a certain event.

[0083] As shown in FIG. 8, the client device 10 acquires sensing data (S110). Specifically, the client device 10 may start capturing a video of the user exercising (hereinafter referred to as "user video") by enabling the operation of the camera 16. The client device 10 may also start measuring the distance from the depth sensor 17 to each part of the user exercising (hereinafter referred to as "user depth") by enabling the operation of the depth sensor 17.

[0084] Furthermore, the client device 10 may cause the wearable device 50 to start measuring the heart rate (hereinafter referred to as the "user's heart rate") using the heart rate sensor 56. Furthermore, the client device 10 may enable any sensor of the client device 10 or the wearable device 50 (for example, the acceleration sensor 19 or the acceleration sensor 57).

[0085] The client device 10 then acquires sensing data from each sensor while the user is performing the target exercise. The target exercise is selected from a plurality of predetermined exercises, for example, exercises registered in an exercise database (FIG. 6). Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 may acquire user video data from the camera 16, acquire user depth data from the depth sensor 17, acquire user heart rate data from the wearable device 50, and acquire user acceleration data related to the acceleration of the user (hereinafter, "user acceleration") from at least one of the acceleration sensor 19 or the wearable device 50.

[0086] The client device 10 can repeatedly acquire the sensing data (S110) before proceeding to the next step S111.

[0087] After step S110, the client device 10 analyzes the sensing data (S111). Specifically, the client device 10 analyzes the sensing data acquired in step S110 to analyze the user's body movements during exercise. As an example, the client device 10 estimates the user's form or pace. The user's form may be the position or angle of one or more parts of the user's body at a single point in time or multiple points in time, the range of motion of the moving parts, or a combination thereof. The user's pace may be the velocity, acceleration, angular velocity, or angular acceleration of one or more parts of the user's body (moving parts) at a single point in time or multiple points in time, the time required for one cycle of movement, or a combination thereof. The client device 10 primarily references user video data to analyze the user's body movements during exercise. The client device 10 may reference user depth data, user acceleration data, or a combination thereof in addition to or instead of the user video data.

[0088] The body movements (features) analyzed by the client device 10 may differ depending on the type of exercise. For example, when an exercise type for which the arm form is not defined is performed, the client device 10 may omit analyzing the position and angle of the arms. The type of exercise can be identified by an argument (for example, an exercise type ID) provided at the start of the exercise support process.

[0089] Information about the user's skeleton during exercise (hereinafter referred to as "user skeleton information") is data such as feature quantities. The user skeleton information includes, for example, information about the position, speed, or acceleration of each part of the user's body (which may include information about changes in the parts of the muscles used by the user or shaking of the user's trunk). The user skeleton information can be obtained by analyzing the user's skeleton during exercise with reference to user video data (or user video data and user depth data). As an example, Vision, an SDK for iOS (registered trademark) 14, or other skeleton detection algorithms (e.g., OpenPose, PoseNet, MediaPipe Pose) can be used for skeleton analysis.

[0090] After step S111, if the target exercise event is being performed continuously, the client device 10 executes a determination of the accuracy of the movement (S112). Specifically, the client device 10 determines whether the user is performing the movements of the target exercise event accurately (i.e., as defined by the exercise definition information) based on the analysis result in step S111 and the exercise definition information of the target exercise event. The determination may be a binary determination (a two-level evaluation of accurate or inaccurate) or a multi-level determination (an evaluation of three or more levels including accurate, inaccurate, or intermediate states). The determination may be based on logic (e.g., a determination formula) or may be performed using a trained model.

[0091] As a first example of determining the accuracy of the movement (S112), the client device 10 determines whether the user's movement form is accurate based on whether the estimated user's movement form matches the movement form of the target exercise defined by the exercise definition information corresponding to the target exercise. For example, the client device 10 may determine whether the user's movement form is accurate based on at least one of the following: -Whether the range of motion of the body parts moving in the target exercise is within the allowable range defined by the exercise definition information -Whether the positional relationships between body parts involved in the target exercise are within the allowable range defined by the exercise definition information -Whether the angles of the body parts involved in the target exercise are within the allowable range defined by the exercise definition information

[0092] As a second example of determining the accuracy of the movement (S112), the client device 10 determines whether the pace of the user's movement is accurate based on whether the estimated pace of the user matches the pace of the target exercise defined by the exercise definition information corresponding to the target exercise. For example, the client device 10 may determine whether the pace of the user's movement is accurate based on at least one of the following: Whether the time required for one cycle of the exercise is within the allowable range defined by the exercise definition information Whether the angle, angular velocity, angular acceleration, or the amount of change of these of the joints related to the body parts moving in the target exercise, the velocity, acceleration, or the amount of change of these of the body parts, or a combination of these, is within the allowable range defined by the exercise definition information The client device 10 may perform a composite determination based on a plurality of parameters regarding the speed (i.e., rhythm) of the movement of the body part being moved in the target exercise. For example, the client device 10 may determine whether the movement of the user's body part is smooth or awkward based on a combination of parameters (angle, angular velocity, angular acceleration, or changes therein) of a plurality of joints related to the body part being moved in the target exercise.

[0093] The third example of determining the accuracy of the movement (S112) is a combination of the first and second examples.

[0094] If it is determined in step S112 that the user's motion is correct, the client device 10 re-executes the acquisition of sensing data (S110). On the other hand, if it is determined in step S112 that the user's motion is not correct, the client device 10 generates feedback information (S113). Specifically, the client device 10 generates feedback information in accordance with the determination result in step S112.

[0095] As a first example of generating feedback information (S113), the client device 10 generates feedback information about a body part of the user involved in the target exercise that is determined in step S112 not to conform to the form of the target exercise defined by the exercise definition information. The feedback information about the body part may include information indicating the body part, information indicating guidelines for improving the movement of the body part, or a combination thereof. The feedback information may be expressed as an image (which may include a moving image), text, audio, or a combination thereof. For example, if the width of the feet is outside the allowable range, feedback information indicating the feet or feedback information urging the user to keep their feet approximately shoulder-width apart may be generated. If the angle of the back is outside the allowable range, feedback information indicating the back or feedback information urging the user to straighten their back may be generated. Furthermore, if the range of motion of the arms is outside the allowable range, feedback information indicating the arms or feedback information urging the user to straighten their arms may be generated.

[0096] As a second example of generating feedback information (S113), the client device 10 generates feedback information that allows a comparison between the form (correct form) of a target exercise event defined by the exercise event information of the target exercise event and the user's form. The correct form may be expressed numerically, or may be expressed using a model video or a still image captured from the model video, or a combination thereof. The user's form may be expressed numerically, or may be expressed using a user video or a still image captured from the user video, or a combination thereof.

[0097] As a third example of generating feedback information (S113), when the estimated range of motion of a body part of the user to be moved in the target exercise deviates from the allowable range of the range of motion of the body part, the client device 10 may select another exercise from a plurality of predetermined exercises in which the range of motion of the body part of the user falls within the corresponding allowable range (for example, with a more relaxed requirement for the range of motion of the body part).The client device 10 then generates feedback information indicating the selected another exercise.

[0098] As a fourth example of generating feedback information (S113), the client device 10 may include information on a comparison result between the estimated user's pace and the pace of the target exercise event defined by the exercise definition information, or on an improvement guideline derived from the comparison result. For example, the feedback information may generate feedback information indicating that the user's pace is inappropriate (e.g., slow, fast, or the speed and slowness of the body parts are unnatural). Alternatively, the feedback information may include information urging the user to move faster or slower. The feedback information may also include information indicating parts where the pace is slow or fast or the speed and slowness of the body parts are unnatural, information indicating sections where the pace is slow or fast or the speed and slowness of the body parts are unnatural, etc.

[0099] As a fifth example of generating feedback information (S113), the client device 10 generates feedback information that allows a comparison between the pace of a target exercise event (correct pace) defined by exercise event information of the target exercise event and the user's pace. The correct pace may be expressed numerically, or may be expressed using a model video or a still image captured from the model video, or a combination thereof. The user's pace may be expressed numerically, or may be expressed using a user video or a still image captured from the user video, or a combination thereof.

[0100] As a sixth example of generating feedback information (S113), when the estimated user's pace deviates from the allowable pace range defined by the exercise type information for the target exercise type, the client device 10 may select another exercise type from a plurality of predetermined exercise types in which the user's pace falls within the corresponding allowable range (e.g., with a gentler pace requirement).The client device 10 then generates feedback information indicating the selected another exercise type.

[0101] The seventh example of the generation of feedback information (S113) is a combination of multiple of the above first to sixth examples.

[0102] After step S113, the client device 10 presents feedback information (S114). Specifically, the client device 10 presents the feedback information generated in step S113 to the user. As a first example of presenting the feedback information (S114), the client device 10 displays a screen based on the feedback information on the display 15. As a second example of presenting the feedback information (S114), the client device 10 outputs a sound based on the feedback information from a speaker. As a third example of presenting the feedback information (S114), the client device 10 transmits the feedback information to the wearable device 50, and the wearable device 50 outputs a screen, sound, light, or vibration based on the feedback information. A fourth example of presenting the feedback information (S114) is a combination of multiple of these first to third examples. After step S114, the client device 10 may return to acquiring sensing data (S110) or may end the exercise support process of Fig. 8. For example, if the number or frequency of times that the movement is determined to be inaccurate is equal to or exceeds a threshold, the client device 10 may end the exercise support process of Fig. 8 to allow the user to stop or restart the current target exercise event, or to select another exercise event.

[0103] After step S111, if the target exercise event has been completed, the client device 10 generates user data (S115). Specifically, the client device 10 generates user data based on the sensing data acquired in step S110, the analysis result in step S111, the determination result in step S112, or a combination thereof. The user data may include at least one of the following: The sensing data acquired in step S110 (for example, user video data, user depth data, user heart rate data, or user acceleration data) Data obtained by processing the data acquired in step S110 Analysis results in step S111 The determination result in step S112 Information that can identify the target sport Information indicating the user's subjective assessment of the exercise intensity of the completed exercise event (hereinafter referred to as "Perceived Exertion Intensity").

[0104] After step S115, the client device 10 transmits the user data (S116). Specifically, the client device 10 transmits the user data generated in step S115 to the server 30. Based on the user data, the server 30 can perform processes such as estimating exercise intensity and determining the next type of exercise to recommend to the user. A description of the processes performed by the server 30 will be omitted.

[0105] In the example of Figure 8, the client device 10 generates user data (S115) and sends the user data (S116) after the target exercise is completed, but the client device 10 may repeatedly generate user data (S115) and send the user data (S116) while the target exercise is being performed, or may omit the processes of generating user data (S115) and sending the user data (S116) if no processing is performed on the server 30 side.

[0106] (5) Summary As described above, the client device 10 of this embodiment acquires sensing data about a user performing a first exercise type among a plurality of predetermined exercise types and analyzes the sensing data. Based on the analysis results of the sensing data and the exercise definition information defining the first exercise type, the client device 10 determines whether the user is performing the first exercise type as defined and presents information corresponding to the determination result to the user. This allows the user to be guided to optimize the load (exercise intensity) imposed on the user. Therefore, it is possible to control the exercise intensity even for exercise types that vary greatly in exercise intensity depending on the method, such as gymnastics, dance, and strength training, thereby expanding the range of exercise types available for effective and safe exercise therapy.

[0107] The exercise definition information may define a form for the first exercise type. The client device 10 may estimate the user's form based on the analysis results of the sensing data and determine whether the user is performing the first exercise type as defined based on whether the estimated user's form matches the form for the first exercise type defined by the exercise definition information. This makes it possible to prevent the user from being subjected to excessive or insufficient load (exercise intensity) due to exercising in a form different from the defined form.

[0108] The client device 10 may present to the user information indicating the body parts related to the first exercise that are determined not to conform to the form of the first exercise defined by the exercise definition information, thereby allowing the user to recognize which body parts need to have their form improved.

[0109] The client device 10 may present the user with information indicating guidelines for improving the movement of body parts that are determined not to conform to the form of the first exercise defined by the exercise definition information, among the body parts related to the first exercise type. This allows the user to recognize how to improve the form of each body part.

[0110] The client device 10 may present the user with information that allows the user to compare the form of the first exercise event defined by the exercise definition information with the user's form, thereby allowing the user to compare their own form with the correct form and recognize any gaps between their own form and the correct form.

[0111] The exercise definition information may define an allowable range of motion for a body part to be moved in the first exercise type. The client device 10 may estimate the range of motion for the user's body part based on the analysis results of the sensing data, and determine whether the user is performing the first exercise type as defined based on whether the estimated range of motion for the user's body part is within the allowable range defined by the exercise definition information. This makes it possible to prevent the user from being subjected to excessive or insufficient load (exercise intensity) due to exercising with a range of motion different from the defined range of motion.

[0112] When the estimated range of motion of a body part of the user to be moved in the first exercise type deviates from the allowable range of motion of the body part, the client device 10 may select a second exercise type from a plurality of exercise types in which the range of motion of the body part of the user falls within the corresponding allowable range. The client device 10 may present information indicating the second exercise type to the user. This may suggest an exercise type with a defined range of motion that the user can handle, making it easier to control the load (exercise intensity) on the user.

[0113] The exercise definition information may define the pace of the first exercise type. The client device 10 may estimate the user's pace for the first exercise type based on the sensing data and determine whether the user is performing the first exercise type as defined based on whether the estimated user pace matches the pace for the first exercise type defined by the exercise definition information. This makes it possible to prevent the user from being subjected to excessive or insufficient load (exercise intensity) due to exercising at a pace different from the defined pace.

[0114] The client device 10 may present the user with information indicating that the user's pace is inappropriate, thereby making the user aware that his or her pace is slower or faster than the correct pace, or that the speed and speed of the movements of his or her body parts are unnatural.

[0115] The client device 10 may present the user with information indicating guidelines for improving the user's pace, thereby allowing the user to recognize how to improve the pace.

[0116] The client device 10 may present the user with information that allows the user to compare the pace of the first exercise defined by the exercise definition information with the estimated pace of the user, thereby allowing the user to compare their own pace with the correct pace and recognize the gap between the correct pace.

[0117] The exercise definition information may define an acceptable range of pace for the first exercise type. If the estimated user's pace deviates from the acceptable range of pace for the first exercise type, the client device 10 may select a second exercise type from multiple exercise types in which the user's pace falls within the corresponding acceptable range. The client device 10 may present the second exercise type to the user. This may suggest an exercise type that defines a pace within the range that the user can handle, making it easier to control the load (exercise intensity) on the user.

[0118] (6) Variations (6-1) Variation 1 9 is a flowchart of the scoring process of the first modification.

[0119] The scoring process starts when, for example, any of the following start conditions is met: The scoring process was called by another process. The user or the person who plans or instructs the user's exercise therapy performed an operation to invoke the scoring process. The client device 10 enters a predetermined state (for example, a predetermined application is started). The appointed date and time has arrived. A certain amount of time has passed since a certain event.

[0120] Note that Modification 1 can be combined with this embodiment, that is, both generation and presentation of feedback information and scoring may be performed.

[0121] As shown in FIG. 9, the client device 10 acquires sensing data (S110) and analyzes the sensing data (S111) in the same manner as in FIG.

[0122] After step S111, if the target exercise event is being performed continuously, the client device 10 determines the accuracy of the movement (S112) in the same manner as in FIG.

[0123] After step S112, the client device 10 calculates the score (S213). Specifically, the client device 10 calculates a score that indicates how accurately the user is performing the target exercise based on the result of the determination in step S112. For example, if the result of the determination in step S112 is positive, the client device 10 may increase the score. On the other hand, if the result of the determination in step S112 is negative, the client device 10 may decrease the score. The method of calculating the score can be determined arbitrarily.

[0124] After step S213, the client device 10 may return to acquiring sensing data (S110) or may end the scoring process of Fig. 9. For example, if the number or frequency of movements determined to be inaccurate reaches or exceeds a threshold, the client device 10 may end the scoring process of Fig. 9 to allow the user to stop or restart the current target exercise, or to select another exercise.

[0125] Step S213 may be performed each time step S112 is executed as shown in FIG. 9, or may be performed each time step S112 is executed multiple times based on the judgment results of those multiple times, or may be performed based on all judgment results after the target exercise event is completed.

[0126] After step S111, if the target exercise event has been completed, the client device 10 displays the score (S214). Specifically, the client device 10 presents to the user the score obtained by the most recently executed score calculation (S213). As a first example of score presentation (S214), the client device 10 displays a screen based on the score on the display 15. As a second example of score presentation (S214), the client device 10 outputs a sound based on the score from a speaker. As a third example of score presentation (S214), the client device 10 transmits the score to the wearable device 50, and the wearable device 50 outputs a screen, sound, light, or vibration based on the score. A fourth example of score presentation (S214) is a combination of multiple of these first to third examples.

[0127] Note that step S214 may be performed only once after the target exercise has been completed as shown in FIG. 9, or may be performed each time step S213 is performed, or may be performed each time step S213 is performed multiple times.

[0128] The scores that users obtain in each sport can be utilized in various ways as a record of their achievements. For example, the scores can be used as a tool for managing users' goals or for competitions between users. Alternatively, the scores can be exchanged for goods, services, or assets of monetary value.

[0129] As described above, the client device 10 of the first modification repeatedly performs judgments while the user is performing the first exercise, calculates a score based on the results of the multiple judgments, and presents information corresponding to the score to the user. This motivates the user to perform the exercise accurately and encourages the user to optimize the exercise intensity.

[0130] The score calculated by the first modification can also be applied to a video game in which the game progress is controlled according to the player's physical movements. The video game may be a mini-game that can be played while the aforementioned treatment app, rehabilitation app, or fitness app is running, or may be configured as a standalone app. As an example, the information processing system 1 may determine one of the following depending on the score. This can enhance the effect that the video game has on improving the user's health. The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) provided to users The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) provided to users Game parameters related to the progression of a video game (e.g., game score, damage)

[0131] (7) Other variations The storage device 11 may be connected to the client device 10 via a network NW. Each input device or output device may be integrated with the client device 10. The storage device 31 may be connected to the server 30 via the network NW. Each input device or output device may be integrated with the wearable device 50.

[0132] In the embodiment, the information processing system 1 is implemented as a client / server system. However, the information processing system 1 can also be implemented as a peer-to-peer system or a standalone computer. As an example, the client device 10 may transmit sensing data (or analysis results thereof) to the server 30, and the server 30 may determine the accuracy of the operation. Alternatively, the client device 10 and the wearable device 50 may be configured as a single device.

[0133] Each step of the above information processing can be executed by any of the client device 10, the server 30, and the wearable device 50. Furthermore, one or more steps of the above information processing may be performed using a trained model.

[0134] In the above description, an example has been given in which a user video is captured using the camera 16 of the client device 10. However, the user video may be captured using a camera other than the camera 16. In the above description, an example has been given in which the user depth is measured using the depth sensor 17 of the client device 10. However, the user depth may be measured using a depth sensor other than the depth sensor 17.

[0135] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above-described embodiments. Furthermore, the above-described embodiments can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above-described embodiments and modifications can be combined. [Explanation of symbols]

[0136] 1: Information processing system 10: Client device 11:Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 30: Server 31:Storage device 32: Processor 33: Input / output interface 34: Communication interface 50: Wearable devices 51:Storage device 52: Processor 53: Input / output interface 54: Communication interface 55: Display 56: Heart rate sensor 57: Acceleration sensor

Claims

1. Computer, means for acquiring sensing data relating to a user performing a first exercise type among a plurality of predetermined exercise types; means for analyzing the sensing data; a means for determining whether the user is performing the first exercise type as defined based on an analysis result of the sensing data and exercise definition information that defines the first exercise type; means for presenting information corresponding to the result of the determination to the user; A program that functions as a

2. The exercise definition information defines a form of the first exercise event, the analyzing means estimates the form of the user based on the analysis result of the sensing data; the determining means determines whether the user is performing the first exercise type as defined by the exercise definition information based on whether the estimated form of the user matches the form of the first exercise type defined by the exercise definition information. The program according to claim 1.

3. the presenting means presents to the user information indicating body parts that are determined to not conform to the form of the first exercise type defined by the exercise definition information, among the body parts related to the first exercise type; The program according to claim 2.

4. the presenting means presents to the user information indicating guidelines for improving the way of moving a body part determined to not conform to the form of the first exercise type defined by the exercise definition information, among the body parts related to the first exercise type; The program according to claim 2.

5. The presenting means presents to the user information that allows a comparison between the form of the first exercise event defined by the exercise definition information and the form of the user. The program according to claim 2.

6. The exercise definition information defines an allowable range of motion of a part to be moved in the first exercise event; the analyzing means estimates a range of motion of the user's body part based on an analysis result of the sensing data; the determining means determines whether the user is performing the first exercise as defined based on whether the estimated range of motion of the user's body part is within an allowable range defined by the exercise definition information. The program according to claim 2.

7. the computer is further configured to function as a means for selecting a second exercise type from the plurality of exercise types in which the range of motion of the body part of the user falls within a corresponding allowable range when the estimated range of motion of the body part of the user to be moved in the first exercise type deviates from the allowable range of the range of motion of the body part of the user to be moved in the first exercise type; The presenting means presents information indicating the second exercise event to the user. The program according to claim 6.

8. The exercise definition information defines a pace for the first exercise event; The analyzing means estimates a pace of the first exercise performed by the user based on the sensing data; the determining means determines whether the user is performing the first exercise type as defined based on whether the estimated pace of the user matches the pace of the first exercise type defined by the exercise definition information. The program according to claim 1.

9. the presenting means presents to the user information indicating that the user's pace is inappropriate. The program according to claim 8.

10. the presenting means presents to the user information indicating a guideline for improving the user's pace. The program according to claim 8.

11. the presenting means presents to the user information that allows a comparison between the pace of the first exercise type defined by the exercise definition information and the estimated pace of the user. The program according to claim 8.

12. The exercise definition information defines an acceptable range of pace for the first exercise event; causing the computer to function as a means for selecting, when the estimated pace of the user deviates from an allowable range of the pace of the first exercise type, a second exercise type from the plurality of exercise types in which the pace of the user falls within the corresponding allowable range; The presenting means presents the second exercise event to the user. The program according to claim 8.

13. the determining means repeatedly performs the determination while the user is performing the first exercise event; causing the computer to function as a means for calculating a score based on a plurality of results of the determination; the presenting means presents information according to the score to the user. The program according to claim 1.

14. a means for acquiring sensing data relating to a user performing a first exercise type among a plurality of predetermined exercise types; means for analyzing the sensing data; a means for determining whether the user is performing the first exercise type as defined based on an analysis result of the sensing data and exercise definition information that defines the first exercise type; means for presenting information according to the result of the determination to the user; An information processing device comprising:

15. The computer acquiring sensing data relating to a user performing a first exercise event among a plurality of predetermined exercise events; analyzing the sensing data; determining whether the user is performing the first exercise type as defined based on an analysis result of the sensing data and exercise definition information that defines the first exercise type; presenting information according to the result of the determination to the user; How to perform.

16. A system including a plurality of information processing devices, a means for acquiring sensing data relating to a user performing a first exercise type among a plurality of predetermined exercise types; means for analyzing the sensing data; a means for determining whether the user is performing the first exercise type as defined based on an analysis result of the sensing data and exercise definition information that defines the first exercise type; means for presenting information according to the result of the determination to the user; A system comprising:

Images