Information processing device, method, program, and system

Abstract

To provide decision-making materials for planning or guiding exercise therapy, or signals indicating the user's physical function.SOLUTION: A program of one embodiment causes a computer to function as: means for estimating the first exercise load while a first user is performing the target exercise activity on the basis of sensing data related to the first user; means for determining the first individual indicator of the first user for the exercise load of the target exercise activity by performing a predetermined calculation on the first exercise load; and means for calculating the first parameter representing the characteristics of the first user on exercise tolerance on the basis of the first individual indicator and a reference value for the exercise load of the target exercise activity.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to an information processing system, method, program, and system.

Background Art

[0002] Cardiac rehabilitation aims to enable patients with heart disease to recover physical strength and confidence, return to a comfortable home and social life, and prevent recurrence or readmission of heart disease through a comprehensive activity program including exercise therapy. The core of exercise therapy is aerobic exercise such as walking, jogging, cycling, aerobics, etc. In order to perform aerobic exercise more safely and effectively, it is preferable for patients to exercise at an intensity near their anaerobic threshold (AT).

[0003] The anaerobic threshold is an example of an evaluation index of exercise tolerance, and corresponds to the change point of the cardiopulmonary function state, that is, the exercise intensity near the boundary between aerobic exercise and anaerobic exercise. The anaerobic threshold is generally determined by a CPX test (cardiopulmonary exercise stress test) in which exhaled gas is collected and analyzed while gradually increasing the exercise load on the test subject (see Non-Patent Document 1). In the CPX test, the anaerobic threshold is determined based on the results measured by exhaled gas analysis (for example, oxygen uptake, carbon dioxide emission, tidal volume, respiratory rate, minute ventilation volume, or a combination thereof). According to the CPX test, in addition to the anaerobic threshold, the maximum oxygen uptake corresponding to the exercise intensity near the maximum exercise tolerance can also be determined.

[0004] Patent Document 1 describes determining whether or not the ventilation work threshold (VT) has been reached based on the pulsation information of the subject, and adjusting the exercise load of the exercise providing device according to the determination result.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Literature

[0006]

Non-Patent Literature 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] In the technical idea described in Patent Document 1, generally speaking, feedback control is performed such that the exercise load approaches the ventilatory work threshold (VT). However, the load on the person performing the exercise depends not only on the type of exercise but also on the person's physical function and daily physical condition. Therefore, even if such a technical idea is applied to exercise therapy, it is not possible to obtain materials for determining how much exercise load will be imposed on the patient when performing the specified type of exercise, or what type of exercise should be recommended to the patient in order to achieve the specified exercise load.

[0008] An object of the present disclosure is to provide materials for making decisions in planning or guiding exercise therapy, or signals indicating the physical function of a user.

Means for Solving the Problems

[0009] A program according to an aspect of the present disclosure causes a computer to function as means for estimating a first exercise load when a first user is performing a target exercise type based on sensing data related to the first user, means for determining a first individual index of the first user for the exercise load for the target exercise type by performing a predetermined calculation on the first exercise load, and means for calculating a first parameter representing a characteristic of the first user regarding exercise tolerance based on the first individual index and a reference value of the exercise load for the target exercise type.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same reference numerals are generally given to the same components, and the repeated description thereof is omitted.

[0012] (1) Configuration of the information processing system The configuration of the information processing system will be described. FIG. 1 is a block diagram showing the configuration of the information processing system according to the present embodiment.

[0013] As shown in FIG. 1, the information processing system 1 includes a client device 10, a server 30, and a wearable device 50.

[0014] Here, the numbers of the client device 10 and the wearable device 50 vary depending on, for example, the number of users. Therefore, the numbers of the client device 10 and the wearable device 50 may each be two or more. Furthermore, the terminal of a person who plans or guides exercise therapy may also be included in the information processing system 1. The person who plans or guides exercise therapy can include, for example, medical personnel (e.g., doctors, nurses, pharmacists, physical therapists, occupational therapists, clinical laboratory technicians), dietitians, or trainers.

[0015] The client device 10 and the server 30 are connected via a network (e.g., the Internet or an intranet) NW. The client device 10 and the wearable device 50 are connected via a wireless channel, for example, using Bluetooth (registered trademark) technology.

[0016] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0017] The server 30 is an example of an information processing device that provides a response corresponding to the request transmitted from the client device 10 to the client device 10. The server 30 is, for example, a server computer.

[0018] The wearable device 50 is an example of an information processing device that can be worn on a user's body (e.g., the arm).

[0019] (1-1) Configuration of the Client Device The configuration of the client device will be described. FIG. 2 is a block diagram showing the configuration of the client device of the present embodiment.

[0020] As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0021] The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage (e.g., a flash memory or a hard disk).

[0022] The program includes, for example, the following programs. · Program of an OS (Operating System) · Program of an application that executes information processing (e.g., a web browser, a therapeutic application, a rehabilitation application, or a fitness application) Here, the diseases targeted by the therapeutic application or the rehabilitation application are diseases in which exercise may contribute to the improvement of symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

[0023] The data includes, for example, the following data. · Database referred to in information processing · Data obtained by executing information processing (i.e., the execution result of information processing)

[0024] The processor 12 is a computer that realizes the functions of the client device 10 by starting the programs stored in the storage device 11. The processor 12 is, for example, at least one of the following. · CPU (Central Processing Unit) · GPU (Graphic Processing Unit) · ASIC (Application Specific Integrated Circuit) · FPGA (Field Programmable Gate Array)

[0025] The input / output interface 13 is configured to acquire information (e.g., user instructions, images, sounds) from an input device connected to the client device 10 and output information (e.g., images, commands) to an output device connected to the client device 10.

[0026] The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0027] The communication interface 14 is configured to control communication between the client device 10 and external devices (e.g., another client device 10, a server 30, and a wearable device 50). Specifically, the communication interface 14 can include a module for communication with the server 30 (e.g., a WiFi module, a mobile communication module, or a combination thereof). The communication interface 14 can include a module for communication with the wearable device 50 (e.g., a Bluetooth module).

[0028] The display 15 is configured to display an image (a still image or a moving image). The display 15 is, for example, a liquid crystal display or an organic EL display.

[0029] The camera 16 is configured to take a picture and generate an image signal.

[0030] The depth sensor 17 is, for example, LIDAR (Light Detection And Ranging). The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to surrounding objects (for example, the user).

[0031] The microphone 18 is configured to receive sound waves and generate a sound signal. The microphone 18 is preferably installed in the vicinity of the user's body (especially the respiratory organ), for example, like an earphone microphone.

[0032] The acceleration sensor 19 is configured to detect acceleration.

[0033] (1-2) Configuration of the server The configuration of the server will be described. FIG. 3 is a block diagram showing the configuration of the server of the present embodiment.

[0034] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface 34.

[0035] The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of a ROM, a RAM, and a storage.

[0036] The program includes, for example, the following programs. · Program of the OS · Program of an application that executes information processing

[0037] The data includes, for example, the following data. · Database referred to in information processing · Execution result of information processing

[0038] The processor 32 is a computer that realizes the functions of the server 30 by starting the programs stored in the storage device 31. The processor 32 is, for example, at least one of the following. · CPU ·GPU ·ASIC ·FPGA

[0039] The input / output interface 33 is configured to acquire information (e.g., a user instruction) from an input device connected to the server 30 and output the information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0040] The communication interface 34 is configured to control communication between the server 30 and an external device (e.g., the client device 10).

[0041] (1-3) Configuration of the Wearable Device The configuration of the wearable device will be described. FIG. 4 is a block diagram showing the configuration of the wearable device of the present embodiment.

[0042] As shown in FIG. 4, the wearable device 50 includes a storage device 51, a processor 52, an input / output interface 53, and a communication interface 54. The wearable device 50 is connected to a display 55, a heartbeat sensor 56, and an acceleration sensor 57.

[0043] The storage device 51 is configured to store programs and data. The storage device 51 is, for example, a combination of a ROM, a RAM, and a storage.

[0044] The program includes, for example, the following programs. · An OS program · A program of an application that executes information processing (e.g., a therapeutic application, a rehabilitation application, or a fitness application)

[0045] The data includes, for example, the following data. ·Database referenced in information processing ·Execution result of information processing

[0046] The processor 52 is a computer that realizes the functions of the wearable device 50 by starting the program stored in the storage device 51. The processor 52 is, for example, at least one of the following. ·CPU ·GPU ·ASIC ·FPGA

[0047] The input / output interface 53 is configured to acquire information (e.g., user instructions, sensing results) from an input device connected to the wearable device 50 and output information (e.g., images, commands) to an output device connected to the wearable device 50.

[0048] The input device is, for example, a heart rate sensor 56, an acceleration sensor 57, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 55, a speaker, or a combination thereof.

[0049] The communication interface 54 is configured to control communication between the wearable device 50 and an external device (e.g., the client device 10). Specifically, the communication interface 54 can include a module for communication with the client device 10 (e.g., a Bluetooth module).

[0050] The display 55 is configured to display an image (a still image or a moving image). The display 55 is, for example, a liquid crystal display or an organic EL display.

[0051] The heart rate sensor 56 is configured to measure the heart rate and generate a sensing signal. As an example, the heart rate sensor 56 measures the heart rate by an optical measurement technique.

[0052] The acceleration sensor 57 is configured to detect acceleration.

[0053] (2) One aspect of the embodiment One aspect of this embodiment will be described. FIG. 5 is an explanatory diagram of one aspect of this embodiment.

[0054] As shown in FIG. 5, the client device 10 and the wearable device 50 perform sensing of the user US1 during exercise. That is, the user is performing any one of a plurality of available exercise types (hereinafter referred to as the "target exercise type"). The available exercise types are those for which reference values (described later) of the exercise load amount corresponding to the exercise type are stored in the storage device 31 of the server 30. The user US1 is a person receiving exercise therapy and is, for example, a participant in a (cardiac) rehabilitation program or an exercise guidance program. In the example of FIG. 5, an example in which the user US1 performs gymnastic exercises is shown, but the user US1 can perform any exercise (aerobic exercise or anaerobic exercise) from among the plurality of available exercise types.

[0055] As an example, the camera 16 photographs the appearance (e.g., the whole body) of the user US1 during exercise from the front or diagonally in front at a distance of about 2 m, for example. The camera 16 may be installed at an appropriate height by a tripod or other height adjustment means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. It is also possible to generate three-dimensional video data by combining, for example, video data (two-dimensional) generated by the camera 16 and depth data generated by the depth sensor 17, for example. The microphone 18 receives sounds emitted from the user US1 during exercise (e.g., sounds generated by breathing or vocalization) and generates a sound signal.

[0056] The heart rate sensor 56 of the wearable device 50 measures the heart rate of the user US1 during exercise and transmits the measurement result to the client device 10. The acceleration sensor 57 measures the acceleration during the exercise of the user US1 and transmits the measurement result to the client device 10.

[0057] The client device 10 acquires various sensing data and performs analysis as necessary. As an example, the client device 10 may refer to the video data acquired from the camera 16 and analyze the physical state of the user US1 during exercise. The client device 10 may further refer to the depth data acquired from the depth sensor 17 in order to analyze the physical state of the user US1 during exercise. The client device 10 transmits user data including at least one of the sensing data or the analysis result of the sensing data to the server 30.

[0058] The server 30 estimates the exercise load of the user when performing the target exercise type based on the user data acquired from the client device 10. The exercise load is estimated at a plurality of time points rather than once for the target exercise type. Then, the server 30 performs a predetermined calculation on the exercise loads estimated at the plurality of time points to determine an individual index (described later) of the user US1 for the exercise load for the target exercise type. Further, the server 30 calculates an individual difference parameter (described later) representing the characteristics of the user US1 regarding exercise tolerance ability (for example, whether there is a tendency for the exercise load to be more likely to increase or decrease compared to a standard person) based on the determined individual index and a reference value (described later) of the exercise load for the target exercise type. For example, since the oxygen consumption during exercise depends on the muscle mass, individual differences in the exercise load will occur even for the same exercise type depending on how the muscles of the user US1 are built and the parts where the load is applied.

[0059] The server 30 stores the calculated individual difference parameters in the storage device 31. The server 30 can use the individual difference parameters to predict the amount of exercise load when the user US1 performs a different exercise type from the target exercise type, or estimate what exercise types the user US1 should perform to achieve a desired exercise load. Furthermore, by continuously monitoring the individual difference parameters, it is possible to grasp the trend of changes in the physical function of the user US1 (for example, improvement, maintenance, or decline), or the rate of change (for example, whether it is rapid). For example, when the physical function of the user US1 is rapidly declining, an alert can be output indicating that abnormalities such as the exacerbation of heart diseases such as heart failure, physical failure, or temporary deterioration of physical condition are suspected.

[0060] In this way, the information processing system 1 calculates individual difference parameters representing the characteristics of the user US1 regarding exercise tolerance based on the sensing data of the user US1 when performing the target exercise type. Therefore, according to this information processing system 1, such individual difference parameters can be used as a basis for judgment for planning or guiding the exercise therapy provided to the user US1, or as a signal indicating the physical function or physical condition of the user US1.

[0061] (3) Database The database of this embodiment will be described. The following database is stored in the storage device 31.

[0062] (3-1) Exercise Type Database The exercise type database of this embodiment will be described. FIG. 6 is a diagram showing the data structure of the exercise type database of this embodiment.

[0063] The sports event database stores sports event information. The sports event information is information about recommended sports events or candidate sports events for target sports events (i.e., the available sports events described above). Here, in this embodiment, the sports events include, for example, sports events that can be implemented without using a device capable of adjusting the exercise load, such as gymnastics, bodyweight training, dance, walking, running, treadmill, etc. Since these sports events include events performed in a standing position, they are rich in variations. Furthermore, in this embodiment, the exercise load of these sports events can be adjusted through form (e.g., the range of motion of a part, the degree of opening of the arms or legs, etc.), pace, number of reps, or the time or number of breaks. However, the sports events of this embodiment can further include sports events implemented using a device capable of adjusting the exercise load, such as an ergometer or strength training using training equipment.

[0064] As shown in FIG. 6, the sports event database includes an "ID" field, a "name" field, an "exercise load amount" field, and an "index reference value" field. Each field is associated with each other.

[0065] The "ID" field stores the sports event ID. The sports event ID is information that identifies the sports event corresponding to the relevant record.

[0066] The "name" field stores the sports event name information. The sports event name information is information about the name of the sports event corresponding to the relevant record.

[0067] In the "exercise load amount" field, exercise load amount information is stored. The exercise load amount information is information regarding the standard exercise load amount of the exercise type corresponding to the record in question. The standard load amount is, for example, information regarding the exercise load amount when a person with standard physical functions performs the corresponding exercise type. As an example, such an exercise load amount may be derived by, for example, actually measuring the exercise load amount (such as average oxygen consumption) when the corresponding exercise type is performed on a plurality of people through, for example, exhaled gas analysis, and statistically processing (such as averaging) the measurement results, or may be obtained by referring to the exercise load amount set for the exercise type by a third-party institution. The exercise load amount information may be managed in units finer than generally recognized exercise types. As an example, the exercise amount information may be managed for each variation such as the form, pace, number of reps, or rest time or number of times of each exercise type. That is, even for an exercise (such as leg raises) that is generally recognized as a single type, by specifying the form, pace, number of reps, and rest details, a plurality of exercise types with slightly different exercise loads can be defined. For example, for an exercise that is generally recognized as a single type, a plurality of exercise types differing by 0.2 METs can be defined.

[0068] In the "Index Reference Value" field, index reference value information is stored. The index reference value information is information regarding a value obtained by standardizing the amount of exercise load when a person with standard physical functions performs the type of exercise corresponding to the record. As an example, such a value may be derived, for example, by actually measuring the amount of exercise load when a plurality of persons perform the corresponding type of exercise in each section constituting the type of exercise, applying the measurement results of each section to a predetermined calculation formula for each person to calculate an individual index, and averaging the individual indices among the persons. Here, a section is a constituent unit of a type of exercise, and when a type of exercise consists of a plurality of movement patterns, each movement pattern may correspond to a section. For example, when the type of exercise is dance, each choreography may correspond to a section. Also, when the type of exercise is a squat, the transition from a standing state to a squatting state and the transition from a squatting state to a standing state may each correspond to a section. Note that the above exercise load information may be used as the index reference value information, and in this case, the "Index Reference Value" field may be omitted.

[0069] (3-2) User Profile Database The user profile database of the present embodiment will be described. FIG. 7 is a diagram showing the data structure of the user profile database of the present embodiment.

[0070] User profile information is stored in the user profile database. The user profile information is information regarding the profile of a user of the information processing system 1 (that is, a person who performs exercise).

[0071] As shown in FIG. 7, the user profile database includes an "ID" field, a "Name" field, a "Target Load" field, and a "Body" field. Each field is associated with each other.

[0072] A user ID is stored in the "ID" field. The user ID is information for identifying the user corresponding to the record.

[0073] In the "Name" field, user name information is stored. The user name information is information regarding the name of the user corresponding to the record (for example, name, account name, etc.).

[0074] In the "target load amount" field, target load amount information (an example of "predetermined exercise load amount") is stored. The target load amount information is information regarding the target value of the exercise load amount (for example, oxygen consumption, energy consumption, heart rate, or a combination thereof) set for the user corresponding to the relevant record. As an example, the target load amount information is information specified by a person (for example, a doctor) who plans or guides exercise therapy based on the result of measuring the user's exercise tolerance by CPX when using, for example, an ergometer (as an example, oxygen consumption and heart rate at the anaerobic metabolic threshold (AT)). However, CPX is not essential, and the target value may be specified at the doctor's discretion. As another example, the target load amount information is determined by an algorithm based on the result of measuring the user's exercise tolerance by CPX, for example. In order to perform aerobic exercise more safely and effectively, it is preferable to exercise at an intensity near the anaerobic metabolic threshold. Therefore, the target value of the exercise load amount is, for example, the exercise load amount corresponding to the anaerobic metabolic threshold, but is not limited thereto. Usually, an ergometer is adopted as the exercise type during CPX measurement. However, for example, the oxygen consumption at the anaerobic metabolic threshold when a treadmill is adopted is about 1.2 to 1.3 times the oxygen consumption at the anaerobic metabolic threshold when an ergometer is adopted. This is considered to be due to the fact that the total muscle mass used on the treadmill exceeds the total muscle mass used on the ergometer. Therefore, as a first example, the target value based on the result measured by CPX when using an ergometer may be used as the target load amount information after being corrected by about 1.2 to 1.3 times, or a value obtained by further subtracting a predetermined value (for example, 1 METs). As a second example, the target value based on the result measured by CPX when using an ergometer may be used as the target load amount information without correction. Thereby, when an exercise type with a relatively small total muscle mass used, such as an ergometer, is selected, it is possible to prevent the target value from becoming excessively high. As a third example, the target value may be corrected for each exercise type by a coefficient corresponding to the total muscle mass used, for example.

[0075] In addition, a doctor can also set an upper limit of the exercise load for the user (an example of an exercise prescription). In this case, the user is not allowed to select an exercise item that exceeds the prescribed upper limit of the exercise load. For exercise prescription, a UI (User Interface) screen for exercise prescription may be displayed on the display of the terminal used by the doctor. Such a UI screen can include, for example, the following information. · The user's CPX data · A display area for sample videos of a plurality of selectable exercise items Here, the display areas for the sample videos of each exercise item may be arranged in alignment according to the exercise load amount information corresponding to the exercise item. For example, when 3.6 METs is recommended as the upper limit based on the CPX data, a group of display areas for sample videos of exercise items corresponding to 3.4 METs, a group of display areas for sample videos of exercise items corresponding to 3.6 METs, and a group of display areas for sample videos of exercise items corresponding to 3.8 METs can be arranged on the UI screen. When the doctor selects any of the display areas, the exercise load amount associated with the corresponding exercise item is set as the upper limit.

[0076] In addition, the increase or decrease of the upper limit specified by the doctor through the exercise prescription may be performed by a medical staff under the supervision of the doctor in regular (for example, every two weeks) medical staff guidance or attending doctor consultation.

[0077] Body information is stored in the "Body" field. The body information is information regarding the user's body (function) corresponding to the relevant record. As an example, the body information may include information regarding the user's age, gender, weight, height, disease, and the like.

[0078] In addition, the following information may also be stored in the user profile database. · Information indicating the person who planned or guided the user's physical therapy · Information indicating the user's attending doctor

[0079] (3-3) Parameter Log Database The parameter log database of this embodiment will be described. FIG. 8 is a diagram showing the data structure of the parameter log database of this embodiment.

[0080] The parameter log database can be constructed for each user (that is, the person who performs the exercise) of the information processing system 1, for example. Alternatively, the parameter log database can be configured to store records including information (such as a user ID) that can identify the user.

[0081] Parameter log information is stored in the parameter log database. The parameter log information is information regarding the log of the individual difference parameters calculated for the user.

[0082] As shown in FIG. 8, the parameter log database includes a "date" field and a "parameter" field. Each field is associated with each other.

[0083] Date information is stored in the "date" field. The date information is information regarding the date (or date and time) when the individual difference parameter of the corresponding record was calculated.

[0084] Parameter information is stored in the "parameter" field. The parameter information is information regarding the value of the individual difference parameter of the corresponding record.

[0085] (4) Information processing The information processing of this embodiment will be described.

[0086] (4-1) Exercise type recommendation processing The exercise type recommendation processing of this embodiment will be described. FIG. 9 is a flowchart of the exercise type recommendation processing of this embodiment. FIG. 10 is a diagram showing an example of a screen displayed in the exercise type recommendation processing of this embodiment.

[0087] The exercise type recommendation processing starts, for example, in response to the establishment of any of the following start conditions. · The exercise item recommendation process was called by another process. · The user, or the person who plans or guides the user's exercise therapy, performed an operation to call the exercise item recommendation process. · The client device 10 reached a predetermined state (for example, the startup of a predetermined application). · A predetermined date and time arrived. · A predetermined time elapsed from a predetermined event.

[0088] As shown in FIG. 9, the client device 10 executes acquisition of sensing data (S110). Specifically, the client device 10 may start shooting a video of the user during exercise (hereinafter referred to as "user video") by enabling the operation of the camera 16. Further, the client device 10 may start measuring the distance from the depth sensor 17 to each part of the user during exercise (hereinafter referred to as "user depth") by enabling the operation of the depth sensor 17. The client device 10 may start collecting sound (for example, sound generated by the user's breathing or voice (hereinafter referred to as "user sound")) by enabling the operation of the microphone 18.

[0089] Furthermore, the client device 10 may cause the wearable device 50 to start measuring the heart rate (hereinafter referred to as "user heart rate") by the heart rate sensor 56. Furthermore, the client device 10 may enable any sensor of the client device 10 or the wearable device 50 (for example, the acceleration sensor 19 or the acceleration sensor 57).

[0090] Then, the client device 10 acquires sensing data from each sensor. Specifically, the client device 10 acquires the sensing results generated by various sensors enabled in step S110. For example, the client device 10 acquires user video data from the camera 16, acquires user depth data from the depth sensor 17, acquires user audio data from the microphone 18, acquires user heart rate data from the wearable device 50, and may acquire user acceleration data (hereinafter, referred to as "user acceleration") regarding the user's acceleration from at least one of the acceleration sensor 19 or the wearable device 50.

[0091] After step S110, the client device 10 executes generation of user data (S111). Specifically, the client device 10 generates user data based on the sensing data acquired in step S110. The user data can include at least one of the following. · Data acquired in step S111 (for example, user video data, user depth data, user audio data, user heart rate data, or user acceleration data) · Data obtained by processing the data acquired in step S111 · Data obtained by analyzing the user video data (or user video data and user depth data) acquired in step S111 (for example, skeleton data, expression data, skin color data, respiration data, or a combination thereof, which will be described later) · Information capable of specifying the type of exercise (target exercise type) that the user was performing in step S110 · Information indicating the user's subjective evaluation (hereinafter, referred to as "perceived exercise intensity") of the exercise load of the target exercise type

[0092] After step S111, the client device 10 executes transmission of user data (S112). Specifically, the client device 10 transmits the user data generated in step S111 to the server 30.

[0093] After step S112, the server 30 executes the estimation of the exercise load amount (S130). Specifically, the server 30 receives the user data transmitted by the client device 10 in step S112. The server 30 estimates the exercise load amount of the user when performing the target exercise type based on the user data acquired from the client device 10. The exercise load amount can be calculated, for example, as the energy consumption amount (e.g., METs), oxygen consumption amount, exercise intensity based on the heart rate (e.g., the exercise intensity calculated by the Karvonen method), or a combination thereof. The server 30 may refer to the user profile information stored in the user profile database (Figure 7) for the estimation of the exercise load amount.

[0094] As a first example of the estimation of the exercise load amount (S130), the server 30 performs the calculation of the Karvonen method based on the heart rate of the user measured at a plurality of time points during the target exercise and the age of the user, thereby estimating the exercise load amount of the user in each section constituting the target exercise.

[0095] Note that the client device 10 or the server 30 may specify to which section each sensing data is associated by analyzing, for example, user video data (and user depth data as necessary). Alternatively, when a fixed time is allocated to each section, such as in gymnastics or dance, the client device 10 or the server 30 may specify to which section each sensing data is associated based on such time allocation.

[0096] As a second example of the estimation of the exercise load amount (S130), the server 30 estimates the exercise load amount of the user in each section constituting the target exercise using the estimation model described later. Specifically, the server 30 estimates the exercise load amount by applying the estimation model to the input data based on the user data (e.g., skeleton data, facial expression data, skin color data, respiration data, heart rate data, or a combination thereof). The server 30 may combine the above first example and second example.

[0097] Note that since the estimated exercise load amount is not stable immediately after the start of exercise (for example, for 1 to 2 minutes), the estimation of the exercise load amount may be omitted for a predetermined period from the start of exercise, or the estimated exercise load amount may be discarded. In other words, the server 30 may estimate the exercise load amount only when it is determined that the user is in a plateau state.

[0098] After step S130, the server 30 executes the determination of individual indices (S131). Specifically, the server 30 performs a predetermined calculation using the exercise load amount estimated in step S130 to determine, for the user, an individual index of the exercise load amount for the target exercise type. As an example, the server 30 determines, as the individual index, a representative value (for example, an average value, a median value, a mode value, a maximum value, a minimum value, a first quartile, or a third quartile) of the exercise load amounts estimated for each section constituting the target exercise type. As another example, the server 30 may determine, as the individual index, a weighted sum of the exercise load amounts estimated for each section constituting the target exercise type. The calculation formula for the individual index may be defined for each exercise type, or may be commonly defined across a plurality of exercise types.

[0099] After step S131, the server 30 executes the calculation of individual difference parameters (S133). Specifically, the server 30 refers to the exercise type database (FIG. 6) to obtain index reference value information for the target exercise type. Then, the server 30 calculates an individual difference parameter representing the characteristics of the user regarding exercise tolerance based on the individual index determined in step S131 and the reference value of the exercise load amount for the target exercise type. The server 30 creates a record based on the calculated individual difference parameter and the calculation date (or calculation date and time), and adds it to the user's parameter log database (FIG. 8).

[0100] As a first example, the server 30 calculates an individual difference parameter by dividing an individual index by a reference value. In this case, the individual difference parameter corresponds to a prediction result of how much larger or smaller the exercise load is than that of a standard person when the user performs the target exercise type or another exercise type.

[0101] As a second example, the server 30 calculates an individual difference parameter by subtracting a reference value from an individual index. In this case, the individual difference parameter corresponds to a prediction result of how much larger or smaller the exercise load is than that of a standard person when the user performs the target exercise type or another exercise type.

[0102] After step S132, the server 30 executes the selection of the recommended exercise type (S133). Specifically, the server 30 refers to the user profile database (FIG. 7) and obtains the user's target load amount information (which is determined based on the result of measuring the user's exercise tolerance as described above). The server 30 selects an exercise type (hereinafter referred to as the "recommended exercise type") suitable for the characteristics of the user's exercise tolerance based on the obtained target load amount information, the individual difference parameter calculated in step S132, and the exercise load amount information of each exercise type stored in the exercise type database (FIG. 6). The recommended exercise type may be one or a plurality.

[0103] As a first example, the server 30 corrects the exercise load amount (hereinafter referred to as "standard load amount") indicated by the exercise load amount information of each exercise type by using the individual difference parameter. For example, the server 30 obtains a corrected load amount by multiplying or adding the individual difference parameter to the standard load amount. Then, the server 30 selects a recommended exercise type from the exercise types whose corrected load amount does not exceed the target value of the user's exercise load amount. For example, the server 30 may include, as the recommended exercise type, the exercise type having the maximum value within the range where the corrected load amount is less than or equal to the target value. Note that when selecting the recommended exercise type, the server 30 may use, instead of the target value, a value obtained by adding or subtracting a margin to or from the target value, or a value obtained by multiplying the target value by a positive coefficient different from 1. Alternatively, when selecting the recommended exercise type, the server 30 may use, instead of the corrected load amount, a value obtained by adding or subtracting a margin to or from the corrected load amount, or a value obtained by multiplying the corrected load amount by a positive coefficient different from 1.

[0104] As a second example, the server 30 corrects the target value of the user's exercise load amount by using the individual difference parameter. For example, the server 30 obtains a corrected target value by dividing the target value by the individual difference parameter or subtracting the individual difference parameter from the target value. Then, the server 30 selects a recommended exercise type from the exercise types whose standard load amount does not exceed the corrected target value. For example, the server 30 may include, as the recommended exercise type, the exercise type having the maximum value within the range where the standard load amount is less than or equal to the corrected target value. Note that when selecting the recommended exercise type, the server 30 may use, instead of the corrected target value, a value obtained by adding or subtracting a margin to or from the corrected target value, or a value obtained by multiplying the corrected target value by a positive coefficient different from 1. Alternatively, when selecting the recommended exercise type, the server 30 may use, instead of the standard load amount, a value obtained by adding or subtracting a margin to or from the standard load amount, or a value obtained by multiplying the standard load amount by a positive coefficient different from 1.

[0105] After step S133, the server 30 executes sending of the recommended exercise type information (S134). Specifically, the server 30 transmits information regarding the recommended exercise type selected in step S133 (hereinafter referred to as "recommended exercise type information") to the client device 10. The recommended exercise type information may be, for example, information for identifying the recommended exercise type, or may be screen information for displaying the recommended exercise type.

[0106] After step S134, the client device 10 executes screen display (S113). Specifically, the client device 10 receives the recommended exercise type information transmitted by the server in step S134. The client device 10 displays a screen based on the recommended exercise type information on the display 21.

[0107] For example, the client device 10 displays the screen of FIG. 10 on the display 21. The screen of FIG. 10 includes objects J20 to J23.

[0108] The object J20 displays the recommended exercise type. Also, the object J20 accepts a user instruction to start the recommended exercise type or a user instruction to play a demonstration video of the recommended exercise type. When the object J20 is selected, the client device 10 may re-execute the exercise type recommendation process of the present embodiment with the exercise type corresponding to the object J20 as a new target exercise type, or may play the demonstration video of the exercise type.

[0109] The object J21 accepts a user instruction to select an exercise type other than the recommended exercise type. When the object J21 is selected, the client device 10 may, for example, display a list of exercise types other than the recommended exercise type and accept a user instruction to select an exercise type. In response to receiving such a user instruction, the client device 10 may re-execute the exercise type recommendation process of the present embodiment with the selected exercise type as a new target exercise type, or may play the demonstration video of the exercise type.

[0110] The object J22 receives a user instruction to end the exercise. When the object J22 is selected, the client device 10 ends the exercise type recommendation process of the present embodiment.

[0111] After step S113, the client device 10 may end the exercise type recommendation process (FIG. 9).

[0112] (4-2) Parameter Monitoring Process The parameter monitoring process of the present embodiment will be described. FIG. 11 is a flowchart of the parameter monitoring process of the present embodiment.

[0113] The parameter monitoring process of the present embodiment may be started every time the exercise type recommendation process of the present embodiment (FIG. 9) (particularly, the calculation of individual difference parameters (S133)) is executed, or may be repeatedly executed for each user at a predetermined cycle.

[0114] As shown in FIG. 11, the server 30 executes the acquisition of parameter log information (S230). Specifically, the server 30 refers to the parameter log database of the user to be processed and acquires the parameter log information.

[0115] After step S230, the server 30 executes the determination of predetermined conditions (S231). Specifically, the server 30 determines whether a predetermined condition is satisfied for the parameter log information acquired in step S230. For example, the predetermined condition may include at least one of the following. · The representative value of a predetermined number (one or more) of recent individual difference parameters exceeds the threshold value (that is, it is predicted that the exercise load of the user is excessively large compared to a standard person). · The representative value of the individual difference parameters over a recent predetermined period exceeds the threshold value (that is, it is predicted that the exercise load of the user is excessively large compared to a standard person). · The increase rate of the individual difference parameter in the most recent predetermined number exceeds the threshold value (that is, it is predicted that the exercise load of the user for the same exercise type is rapidly increasing). · The increase rate of the individual difference parameter in the most recent predetermined period exceeds the threshold value (that is, it is predicted that the exercise load of the user for the same exercise type is rapidly increasing).

[0116] When it is determined in step S231 that a predetermined condition is satisfied, the server 30 executes the output of an alert (S232). Specifically, the server 30 transmits the alert to the user's client device 10 or a predetermined terminal. The predetermined terminal may be the terminal of a person designated by the user (for example, a family member), the terminal of the user's attending doctor, or the terminal of a person who planned or instructed the user's physical therapy.

[0117] The alert can include, for example, at least one of the following information. · The most recent predetermined number of individual difference parameters · The individual difference parameter over the most recent predetermined period · The increase rate of the individual difference parameter in the most recent predetermined number · The increase rate of the individual difference parameter in the most recent predetermined period · A message indicating that there is a suspicion of deterioration of heart disease in the user (for example, text, image, voice, or a combination thereof) · A message indicating that there is a suspicion of physical failure in the user · A message indicating that there is a suspicion of temporary physical deterioration in the user

[0118] The output destination of the alert (the user's client device 10 or a predetermined terminal) presents the information by image, voice, vibration, or a combination thereof based on the alert received from the server 30.

[0119] After step S232, the server 30 ends the parameter monitoring process of the present embodiment.

[0120] When it is determined in step S231 that a predetermined condition is not satisfied, the server 30 skips the output of an alert (S232) and ends the parameter monitoring process of the present embodiment.

[0121] (5) Parentheses As described above, the server 30 of the present embodiment estimates the amount of exercise load when the user is performing the target exercise type based on the sensing data regarding the user, and performs a predetermined calculation on the amount of exercise load to determine the individual index of the amount of exercise load for the target exercise type for the user. The server 30 calculates an individual difference parameter representing the characteristics of the user regarding exercise tolerance based on the individual index and the reference value of the amount of exercise load for the target exercise type. Thereby, the individual difference parameter can be used as a judgment material for planning or guiding an exercise therapy to be provided (which may include a prescription) to the user, or as a signal indicating the physical function or physical condition of the user.

[0122] The server 30 may select a recommended exercise type suitable for the exercise tolerance of the user from a plurality of exercise types based on the standard exercise load amounts of the respective plurality of exercise types, the individual difference parameter of the user, and the result of measuring the exercise tolerance of the user, and output information indicating the recommended exercise type. Thereby, it is possible to recommend an exercise type by further considering not only the matching between the measurement result of the user's exercise tolerance and the standard exercise load amount of each exercise type, but also the characteristics regarding the user's exercise tolerance.

[0123] The server 30 may select a recommended exercise type so that the corrected exercise load amount obtained by correcting the standard exercise load amounts of the respective plurality of exercise types using the individual difference parameter of the user does not exceed a predetermined exercise load amount. Thereby, it is possible to recommend an exercise type for which it is estimated that the exercise load amount when the user actually performs it does not exceed a predetermined exercise load amount.

[0124] The server 30 may select a recommended sport type such that the corrected exercise load obtained by correcting the standard exercise load for each of a plurality of sport types using the user's individual difference parameters is the largest within a range not exceeding a predetermined exercise load. As a result, it is possible to recommend a sport type in which the exercise load when the user actually performs the exercise does not exceed the predetermined exercise load and is estimated to be near the predetermined exercise load.

[0125] The predetermined exercise load may be determined according to the exercise load specified by a person or algorithm who plans or guides the user's exercise therapy based on the result of measuring the user's exercise tolerance. As a result, it is possible to recommend a sport type that conforms to the exercise load determined to be suitable for the user from a medical perspective.

[0126] The server 30 may output an alert when the user's individual difference parameters satisfy a predetermined condition. As a result, changes in the user's physical function and physical condition can be grasped early by the user himself or a person concerned (for example, family members or the attending doctor).

[0127] The user's individual difference parameter has a value corresponding to the excess of the user's individual index over the reference value of the amount of exercise for the target sport type, and the predetermined condition may be that the excess exceeds a threshold value. As a result, for example, it is possible for the user himself or a person concerned to grasp early that there is suspicion of an abnormality such as exacerbation of the user's heart disease, physical failure, or temporary deterioration of physical condition.

[0128] (6) Estimation model As described above, the server 30 may estimate the exercise load using an estimation model. In this case, the estimation model corresponds to a trained model created by supervised learning using the teacher dataset described below, or a derivative model or distilled model of the trained model. The estimation model may be constructed for each sport type or may be constructed commonly for a plurality of sport types.

[0129] (6-1) Teacher Dataset A teacher dataset available for supervised learning to build an estimation model will be described. FIG. 12 is a diagram showing the data structure of the teacher dataset available in this embodiment.

[0130] As shown in FIG. 12, the teacher dataset includes a plurality of teacher data. The teacher data is used for training or evaluation of a model to be learned (hereinafter referred to as the "target model"). The teacher data includes a sample ID, input data, and correct answer data.

[0131] The sample ID is information for identifying the teacher data.

[0132] The input data is data input to the target model during training or evaluation. The input data corresponds to an example used during training or evaluation of the target model. As an example, the input data is data related to the physical state of a subject during exercise (i.e., relatively dynamic data) and data related to the health state of the subject (i.e., relatively static data). At least a part of the data related to the physical state of the subject is obtained by analyzing the physical state of the subject with reference to subject video data (or subject video data and subject depth data).

[0133] The subject video data is data related to a subject video showing a subject during exercise. The subject video data can be obtained, for example, by photographing the appearance (e.g., the whole body) of a subject during an examination related to exhaled gas (e.g., a CPX examination) from the front or diagonally in front (e.g., 45 degrees forward) with a camera (e.g., a camera mounted on a smartphone).

[0134] The subject depth data is data related to the distance (depth) from a depth sensor to each part of a subject during exercise. The subject depth data can be obtained by operating a depth sensor when photographing the subject video.

[0135] The subject may be the same person as the user for whom an estimate of the exercise load based on the exercise tolerance is made during the operation of the information processing system 1, or may be a different person. By making the subject and the user the same person, the target model may learn the individuality of the user, and the estimation accuracy may be improved. On the other hand, allowing the subject to be a different person from the user has the advantage of facilitating the enrichment of the teacher dataset. Further, the subject may be composed of a plurality of people including the user, or a plurality of people not including the user.

[0136] In the example of FIG. 12, the input data includes skeletal data, facial expression data, skin color data, respiratory data, heart rate data, and health status data.

[0137] The skeletal data is data (e.g., feature amounts) related to the skeleton of the subject during exercise. The skeletal data includes, for example, data related to the speed or acceleration of each part of the subject (which may include data related to changes in the parts of the muscles used by the subject or the wobbling of the subject's physical sensation). The skeletal data can be obtained by analyzing the skeleton of the subject during exercise with reference to the subject video data (or the subject video data and the subject depth data). As an example, Vision, which is an SDK of iOS (registered trademark) 14, or other skeletal detection algorithms (e.g., OpenPose, PoseNet, MediaPipe Pose) can be used for the analysis of the skeleton. Alternatively, the skeletal data for the teacher dataset can be obtained, for example, by having the subject exercise while wearing motion sensors on each part of the subject.

[0138] The result of the skeletal detection can be used for quantitative evaluation, qualitative evaluation, or a combination of these of the exercise. As a first example, the result of the skeletal detection can also be used for counting the number of reps. As a second example, the result of the skeletal detection can be used for evaluating the form of the exercise or the appropriateness of the load applied by the exercise. For example, when the exercise type is a squat, the result of the skeletal detection can be used for evaluations such as whether the knee is protruding too far forward and is in a dangerous form, or whether the subject is squatting deeply enough and applying sufficient load.

[0139] Facial expression data is data (such as feature quantities) related to the facial expressions of a subject during exercise. The facial expression data can be analyzed by applying an algorithm or a trained model to the subject video data. Alternatively, the facial expression data for the training dataset can be obtained, for example, by a human who watches the subject video to perform labeling.

[0140] Skin color data is data (such as feature quantities) related to the skin color of a subject during exercise. The skin color data can be analyzed by applying an algorithm or a trained model to the subject video data. Alternatively, the skin color data for the training dataset can be obtained, for example, by a human who watches the subject video to perform labeling.

[0141] Respiration data is data (such as feature quantities) related to the respiration of a subject during exercise. The respiration data relates to, for example, the respiration rate per unit time or the respiration pattern. The respiration pattern can include at least one of the following. · Ventilation frequency · Ventilation volume · Ventilation speed (that is, ventilation volume or ventilation frequency per unit time) · Ventilation acceleration (that is, the time derivative of the ventilation speed) · Carbon dioxide emission concentration · Carbon dioxide emission amount (VCO2) · Oxygen uptake concentration · Oxygen uptake amount (VO2) The data related to the respiration pattern may include data that can be calculated based on a combination of the above-mentioned data such as the gas exchange ratio R (= VCO2 / VO2).

[0142] The respiration data can be obtained, for example, by analyzing the above-mentioned skeletal data. As an example, the following items can be analyzed from the skeletal data. · Movements (expansions) of the shoulders, chest (including the side chest), abdomen, or combinations thereof · Inhalation time · Exhalation time · Degree of use of the accessory respiratory muscles

[0143] The breathing data for the teacher dataset can be obtained, for example, from the results of an exhaled gas test performed on a subject during exercise. Details of the exhaled gas test that can be performed on a subject during exercise will be described later. Alternatively, among the breathing data for the teacher dataset, the ventilation rate, ventilation volume, ventilation speed, or ventilation acceleration can also be obtained, for example, from the results of a respiratory function test (e.g., a pulmonary function test or a vital capacity test) performed on the subject during exercise. In this case, the respiratory function test is not limited to medical devices, and commercially available test instruments may be used.

[0144] The heart rate data is data (e.g., feature quantities) related to the heart rate of a subject during exercise. The heart rate data can be obtained, for example, by analyzing subject video data or its analysis results (e.g., skin color data). Alternatively, the heart rate data for the teacher dataset may be obtained, for example, from the results of a test related to exhaled gas together with the breathing data described later. The subject heart rate data for the teacher dataset can also be obtained by having the subject perform the above exercise while wearing a heart rate sensor or electrodes for an electrocardiogram monitor.

[0145] The health status data is data related to the health status of the subject. The health status data can be obtained in various ways. The health status data of the subject may be obtained at any timing before, during, or after the subject's exercise. The health status data of the subject may be obtained based on a declaration from the subject or their attending physician, or by extracting information linked to the subject in a medical information system, or via the subject's app (e.g., a healthcare app).

[0146] The health status includes at least one of the following. · Age · Gender · Height · Weight · Body fat percentage · Muscle mass · Bone density · Medical history · Past history · Medication history · Surgical history · Life history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty score, etc.) · Family history · Results of pulmonary function tests · Test results other than pulmonary function tests (e.g., blood tests, urine tests, electrocardiogram tests (including Holter electrocardiogram tests), echocardiogram tests, X-ray tests, CT tests (including cardiac morphology CT and coronary artery CT), MRI tests, nuclear medicine tests, PET tests, etc.) · Data obtained during cardiac rehabilitation (including Borg index)

[0147] The correct data corresponds to the correct answer for the corresponding input data (example problem). The target model is trained (supervised learning) to produce an output closer to the correct data for the input data. As an example, the correct data represents the exercise load.

[0148] The exercise load is an index for quantitatively evaluating the load of exercise. The exercise load can be numerically represented using at least one of the following. · Energy (calorie) consumption · Oxygen consumption · Heart rate

[0149] The correct data can be obtained, for example, from the results of tests on exhaled gas performed on a subject during exercise. The first example of a test on exhaled gas is a test (typically a CPX test) performed while a subject wearing an exhaled gas analyzer is performing an incremental exercise (e.g., an ergometer). The second example of a test on exhaled gas is a test performed while a subject wearing an exhaled gas analyzer is performing an exercise with a constant or variable load (e.g., bodyweight exercise, gymnastics, strength training).

[0150] Alternatively, the correct data can also be obtained from the results of tests other than the exhaled gas test performed on the subject during exercise. Specifically, the correct data can be obtained from the results of a cardiopulmonary exercise load prediction test based on the measurement of the lactic acid concentration in the sweat or blood of the subject during exercise. A wearable lactic acid sensor may be used to measure the lactic acid concentration of the subject.

[0151] In addition, it is also possible to construct an estimation model for each of a plurality of health status categories based on (at least a part of) the health status of the subject. In this case, (at least a part of) the user's health status may be referred to in order to select the estimation model. In this further modification example, the input data of the estimation model may be data not based on the user's health status, or may be data based on the user's health status and the user video.

[0152] (7) Other modification examples The storage device 11 may be connected to the client device 10 via the network NW. Each input device or output device may be incorporated in the client device 10. The storage device 31 may be connected to the server 30 via the network NW. Each input device or output device may be incorporated in the wearable device 50.

[0153] An example of implementing the information processing system 1 of the embodiment by a client / server type system has been shown. However, the information processing system 1 of the embodiment can also be implemented by a peer-to-peer type system or a stand-alone computer. As an example, the client device 10 may estimate the exercise load.

[0154] Each of the above information processing steps can be executed by either the client device 10 or the server 30. As an example, instead of the client device 10, the server 30 may analyze the user video (or the user video and the user depth) to obtain at least a part of the user data or the user skeleton data.

[0155] One or more steps of the above information processing may be performed using a learned model.

[0156] In the above example, an example of selecting a recommended exercise type based on the standard exercise load for each of a plurality of exercise types, the user's individual difference parameters, and the result of measuring the user's exercise tolerance was shown. However, based on this information, the server 30 may determine the instruction content for adjusting the exercise load for the exercise type selected by the user, and output the information of the instruction content to the client device 10. The instruction content can include, for example, the form of the exercise type (e.g., the range of motion of the body part, the degree of opening of the arm or leg, etc.), the pace, the number of repetitions, or the time or number of rests. Thereby, for example, even when it is estimated that the exercise load when performing the exercise type selected by the user exceeds the above-mentioned target value, a form with a lighter load than the standard can be specified, the pace can be made slower than the standard, the number of repetitions can be reduced from the standard, or the time or number of rests can be increased, thereby preventing the situation where the user's exercise load deviates from the target value. Alternatively, even when it is estimated that the exercise load when performing the exercise type selected by the user is below the above-mentioned target value, a form with a heavier load than the standard can be specified, the pace can be made faster than the standard, the number of repetitions can be increased from the standard, or the time or number of rests can be reduced, thereby preventing the situation where the user's exercise load deviates from the target value.

[0157] In the above description, an example of selecting a recommended exercise type based on the standard exercise load for each of a plurality of exercise types, the user's individual difference parameters, and the result of measuring the user's exercise tolerance was shown. However, the server 30 may select a recommended exercise type based on at least one of the user's heart rate, respiratory rate, or perceived exercise intensity during exercise. Here, the perceived exercise intensity can be obtained, for example, by receiving the input of the Borg index from the user via the client device 10 during the rest period. When at least one of the heart rate, respiratory rate, or perceived exercise intensity exceeds the upper limit value, the server 30 may select, as the recommended exercise type, an exercise type with a lower standard exercise load compared to the case where this is not the case. On the other hand, when at least one of the heart rate, respiratory rate, or perceived exercise intensity falls below the lower limit value, the server 30 may select, as the recommended exercise type, an exercise type with a higher standard exercise load compared to the case where this is not the case. However, when an upper limit of the exercise load is set for the user by a doctor, an exercise type corresponding to an exercise load exceeding the upper limit is not selected. Furthermore, the user's heart rate, respiratory rate, or perceived exercise intensity during exercise may be used to adjust the exercise load of the next set when performing a plurality of sets of exercise types. For example, when at least one of the heart rate, respiratory rate, or perceived exercise intensity exceeds the upper limit value, the server 30 determines the instruction content (for example, form, pace, number of reps, or rest time or number of times) so that the exercise load of the next set is lower than the current set, and may output the information of the instruction content to the client device 10. On the other hand, when at least one of the heart rate, respiratory rate, or perceived exercise intensity falls below the lower limit value, the server 30 determines the instruction content so that the exercise load of the next set is higher than the current set, and may output the information of the instruction content to the client device 10. However, when an upper limit of the exercise load is set for the user by a doctor, an instruction content corresponding to an exercise load exceeding the upper limit is not selected. For example, when the average heart rate is 5 or more higher than the target heart rate, or the perceived exercise intensity is 14 or more, the server 30 may determine the recommended exercise type or instruction content so that the exercise load of the next set decreases by 0.2 METs. When the average heart rate is 10 or more higher than the target heart rate, or the perceived exercise intensity is 16 or more, the server 30 may determine the recommended exercise type or instruction content so that the exercise load of the next set decreases by 0.4 METs. Further, when the average heart rate is 5 or more lower than the target heart rate and the perceived exercise intensity is less than 10, the server 30 may determine the recommended exercise type or instruction content so that the exercise load of the next set increases by 0.2 METs.

[0158] The client device 10 may further execute the following processing, for example, when executing the acquisition of sensing data (S110) (in other words, during the user's exercise). Specifically, the client device 10 estimates the skeleton of the user during exercise based on the sensing data. Then, when the result of the estimation regarding the skeleton does not conform to at least one of the forms or paces defined for the exercise type the user is performing (for example, the movable range of the part is too narrow or too wide, the angle of the part deviates from the standard, or the pace is too fast or too slow, etc.), the client device 10 provides feedback to the user. The feedback can include at least one of the following. · Output of light, sound, or voice (for example, voice indicating the part the user is not performing correctly) · Display of an image (for example, an image indicating the part the user is not performing correctly) · Vibration of the wearable device Alternatively, when the number or frequency of times the result of the estimation regarding the skeleton does not conform to at least one of the forms or paces defined for the exercise type the user is performing exceeds a threshold, the client device 10 may recommend that the user repeat the same exercise type or perform another exercise type. In this case, the client device 10 may omit the processing after the generation of user data (S111).

[0159] Instead of the exercise type recommendation process, the information processing system 1 may perform a process for measuring the oxygen consumption at the anaerobic metabolism threshold. This process may be automatically selected, for example, within the first three days after the user starts the exercise therapy, or may be selected according to the instructions of the user or the person who plans or guides the exercise therapy. In this mode, the information processing system 1 sequentially selects exercise types so that the exercise load increases by 0.2 METs each until either of the conditions that the average heart rate is 5 or more higher than the target heart rate or the perceived exercise intensity is 14 or more is satisfied, and the user performs the exercise. The information processing system 1 treats the estimated oxygen consumption of the exercise type performed immediately before the exercise type for which the above condition is satisfied as the oxygen consumption at the anaerobic metabolism threshold of the user. Also, the information processing system 1 may treat the exercise load corresponding to the exercise type in which the average heart rate closest to the target heart rate is obtained as the optimal exercise load.

[0160] In the above description, an example of outputting an alert when the user's individual difference parameter satisfies a predetermined condition has been described. However, not limited to the individual difference parameter, alerts may be output with various triggers. As an example, the server 30 accumulates data on the exercise types the user has performed in the past and the user's heart rate, perceived exercise intensity, or respiratory rate when performing the exercise type. Then, when the heart rate, perceived exercise intensity, or respiratory rate of the user during exercise exceeds a threshold and increases (worsens) compared to the heart rate, perceived exercise intensity, or respiratory rate of the user when performing the same exercise type in the past, the server 30 may output an alert. Also, for example, if the degree of the user's distress (hereinafter referred to as "distress level") is quantified based on facial expression data or skeletal data, it can be used as a trigger similar to the above heart rate, perceived exercise intensity, or respiratory rate. In addition, the server 30 may use as a trigger the fact that there has been an input regarding the appearance or worsening of the user's weight gain, shortness of breath, edema, fatigue, loss of appetite, or insomnia in daily life, or the fact that there has been a sensor input suggesting the appearance or worsening of these symptoms.

[0161] In the above description, an example was shown in which the server 30 of the present embodiment estimates the exercise load of a user when performing a target exercise item, determines an individual index of the exercise load, and calculates an individual difference parameter. However, as another example, the server 30 acquires values of one or more variables related to the physiological reaction of the user when the user is performing the target exercise item (for example, heart rate, respiratory rate, perceived exercise intensity, or degree of distress based on facial expression or skeleton (body movement)). For example, the server 30 can acquire these values from the client device 10. Then, the server 30 determines an individual index of the variable for the target exercise item by performing a predetermined calculation on the acquired variables. Then, based on the determined individual index of the user and the reference value of the variable and the reference value of the exercise load for each of a plurality of exercise items including the target exercise item, the server 30 estimates at least one of the exercise item that results in a predetermined exercise load when the user performs it, or the exercise load when the user performs any of the exercise items. The reference value of the variable can be calculated in the same manner as the reference value of the exercise load. As a first example, the server 30 estimates the reference value of the exercise load of the exercise item corresponding to the reference value of the variable closest to the value of the individual index as the exercise load (Mt) when the user is performing the target exercise item, and estimates the exercise load (Mo) when performing other exercise items based on the reference value of the exercise load (Mrt) of the target exercise item and the reference value of the exercise load (Mro) of the other exercise item (for example, Mo = Mt * Mro / Mrt). As a second example, the server 30 estimates the reference value of the exercise load of the exercise item corresponding to the reference value of the variable closest to the value of the individual index as the exercise load (Mt) when the user is performing the target exercise item, and estimates, as the exercise item that results in a predetermined exercise load (Mp) when the user performs it, the one with the closest value within the range not exceeding the value (for example, Mrp = Mrt * Mp / Mt) based on the reference value of the exercise load (Mrt) of the target exercise item, the exercise load (Mt), and the predetermined exercise load (Mp).

[0162] In the above description, an example of capturing a user video using the camera 16 of the client device 10 was shown. However, the user video may be captured using a camera different from the camera 16. An example of measuring the user depth using the depth sensor 17 of the client device 10 was shown. However, the user depth may be measured using a depth sensor different from the depth sensor 17.

[0163] In the above description, an example of measuring the user's heart rate by the wearable device 50 was shown. However, the heart rate can be obtained by analyzing video data or its analysis results (for example, skin color data) (for example, rPPG (Remote Photo-plethysmography) analysis). The analysis of the heart rate may be performed by a trained model constructed using machine learning techniques. Alternatively, the electrocardiogram monitor may be able to measure the user's heart rate by having the user exercise while wearing electrodes for an electrocardiogram monitor. In these variations, the user does not need to wear the wearable device 50 for measuring the heart rate.

[0164] The wearable device 50 can be provided with sensors for measuring at least one of the following items instead of, or in addition to, the heart rate sensor 56 and the acceleration sensor 57. · Blood glucose level · Oxygen saturation The measurement results by each sensor can be appropriately used in the generation of input data, the estimation of the exercise load amount or ventilation index, the presentation of information based on the estimation results, or other scenarios. As an example, the measurement result of the blood glucose level can be referred to for evaluating, for example, the energy consumption amount or the exercise load amount converted into the oxygen consumption amount. As another example, the measurement result of the acceleration can be used, for example, for determining the score of the user's exercise (for example, gymnastics).

[0165] It is also possible to use acceleration data as part of the input data for the estimation model described in this embodiment or the modified example. Alternatively, the user's skeleton may be analyzed with reference to the acceleration data. The acceleration data may be acquired by, for example, the acceleration sensor 19 or the acceleration sensor 57 during the shooting of the user video.

[0166] It is also possible to use oxygen saturation data as part of the input data for the estimation model described in this embodiment or the modified example. The oxygen saturation data can be acquired, for example, by attaching a wearable device equipped with a sensor (e.g., an optical sensor) capable of measuring blood oxygen concentration or a pulse oximeter to the user during the shooting of the user video. The oxygen saturation data may be estimated, for example, by performing rPPG analysis on the user video data.

[0167] The information processing system 1 of this embodiment and each modified example is also applicable to a video game in which the progress of the game is controlled according to the movement of the player's body. The video game may be a mini-game that can be played during the execution of the aforementioned therapeutic application, rehabilitation application, or fitness application. As an example, during the game play, the information processing system 1 may estimate the amount of exercise load based on the user's exercise tolerance, and determine any one of the following according to the result of the estimation (for example, a numerical value indicating the amount of exercise load based on the user's exercise tolerance). Thereby, the effect of the video game on improving the user's health can be enhanced. · The quality (e.g., difficulty level) or quantity of the tasks (e.g., stages, missions, quests) related to the video game given to the user · The quality (e.g., type) or quantity of the benefits (e.g., in-game currency, items, bonuses) related to the video game given to the user · Game parameters (e.g., score, damage) related to the progress of the video game

[0168] The information processing system 1 of the present embodiment and each modification example is also applicable to a video game in which the progress of the game is controlled according to the movement of the player's body. The video game may be a mini-game that can be played during the execution of the aforementioned therapeutic application, rehabilitation application, or fitness application. As an example, during the game play, the information processing system 1 estimates the user's skeleton based on the user video. The estimation of the user's skeleton may be further performed based on at least one of the user depth or the user acceleration in addition to the user video. The information processing system 1 evaluates how well the posture of the user during exercise (e.g., gymnastics) conforms to an ideal posture (model) based on the result of the estimation of the user's skeleton. The information processing system 1 may determine any one of the following according to the result of this evaluation (e.g., a numerical value indicating the degree of conformity of the user's posture to the ideal posture). Thereby, the effect of the video game on improving the user's health can be enhanced. · The quality (e.g., difficulty level) or quantity of the tasks (e.g., stages, missions, quests) related to the video game given to the user · The quality (e.g., type) or quantity of the benefits (e.g., in-game currency, items, bonuses) related to the video game given to the user · Game parameters (e.g., score, damage) related to the progress of the video game

[0169] In addition to the microphone 18, or instead of the microphone 18, the microphone of the wearable device 50 (the microphone provided in the wearable device 50 or connected to the wearable device 50) may receive the sound waves emitted by the user during the shooting of the user video and generate sound data. The sound data may constitute input data for the estimation model described in the present embodiment or the modification example. The sound emitted by the user is, for example, at least one of the following. · Sound waves generated by the rotation of the user's legs (e.g., sounds generated from a pedal or a drive unit connected to the pedal) · Sounds generated along with the user's breathing or vocalization

[0170] In the above description, the CPX test was exemplified as an examination related to exhaled gas. In the CPX test, an incremental exercise load is applied to the subject. However, it is not necessary to increase the exercise load applied to the user during the shooting of the user video. Specifically, the real-time exercise load can be estimated even when a constant or changeable exercise load is applied to the user. For example, the exercise performed by the user may be bodyweight exercise, gymnastics, or strength training.

[0171] As described above in detail for the embodiments of the present invention, the scope of the present invention is not limited to the above embodiments. Also, the above embodiments can be variously improved and modified without departing from the gist of the present invention. Further, the above embodiments and modifications can be combined.

Explanation of Reference Numerals

[0172] 1: Information processing system 10: Client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 21: Display 30: Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface 50: Wearable device 51: Storage device 52: Processor 53: Input / output interface 54: Communication interface 55: Display 56: Heart rate sensor 57: Acceleration sensor

Claims

1. Computer, a means for estimating a first exercise load when a first user is performing a target exercise event based on sensing data related to the first user; a means for determining a first individual index of the first user regarding an exercise stress amount for the target exercise event by performing a predetermined calculation on the first exercise stress amount; a means for calculating a first parameter representing a characteristic of the first user related to exercise tolerance based on the first individual index and a reference value of an exercise load for the target exercise event; A program that functions as a

2. The computer, means for making an estimation regarding a skeleton of the first user when the first user is performing the target exercise event, based on sensing data regarding the first user; means for providing feedback to the first user when the result of the estimation regarding the skeleton does not conform to at least one of a form or a pace defined for the target exercise event; The program according to claim 1 ,

3. The computer, a means for selecting a recommended exercise type suitable for the exercise tolerance of the first user from among the plurality of exercise types based on a standard exercise load for each of the plurality of exercise types, the first parameter, and a result of measuring the exercise tolerance of the first user; A means for outputting information indicating the recommended exercise type; The program according to claim 1 ,

4. the selecting means selects the recommended exercise type such that a corrected exercise stress amount obtained by correcting a standard exercise stress amount of each of the plurality of exercise types using the first parameter does not exceed a predetermined exercise stress amount. The program according to claim 3.

5. the selecting means selects the recommended exercise types so as to include an exercise type for which a corrected exercise stress amount, obtained by correcting a standard exercise stress amount of each of the plurality of exercise types using the first parameter, is the largest within a range not exceeding the predetermined exercise stress amount. The program according to claim 4.

6. The predetermined exercise stress amount is determined according to an exercise stress amount designated by a person or an algorithm who plans or instructs an exercise therapy for the first user based on a result of measuring the exercise tolerance of the first user. The program according to claim 4.

7. and causing the computer to function as a means for outputting information on the form, pace, number of repetitions, or time or number of rest periods of the first exercise type based on a standard exercise load amount for the first exercise type, the first parameter, and the predetermined exercise load amount, when the first user performs the first exercise type after the first parameter is calculated. The program according to claim 4.

8. causing the computer to function as a means for outputting an alert when the first parameter satisfies a predetermined condition; The program according to claim 1.

9. the first parameter has a value corresponding to an excess of the first individual index over a reference value of the amount of exercise for the target exercise event; The predetermined condition is that the excess exceeds a threshold value. The program according to claim 8.

10. The plurality of exercise events include an exercise event that can be performed without using a device capable of adjusting the exercise load, The program according to claim 3.

11. The reference value of the exercise stress amount for the target exercise type is a representative value of the exercise stress amount for the target exercise type measured for a plurality of persons. The program according to claim 1.

12. Computer, means for acquiring values ​​of one or more variables related to a physiological response of a first user when the first user is performing a target exercise event; means for determining a first individual index of the first user of the variables for the target athletic event by performing a predetermined calculation on the variables; a means for estimating at least one of an exercise type that will result in a predetermined exercise stress amount when performed by the first user, or an exercise stress amount when the first user performs any of the exercise types, based on the first individual index and reference values ​​of the variables and reference values ​​of the exercise stress amount for each of a plurality of exercise types including the target exercise type; A program that functions as a

13. The computer estimating a first exercise load when the first user is performing a target exercise event based on sensing data related to the first user; determining a first individual index of the first user regarding an exercise load for the target exercise event by performing a predetermined calculation on the first exercise load; calculating a first parameter representing a characteristic of the first user related to exercise tolerance based on the first individual index and a reference value of an exercise load for the target exercise event; How to do it.

14. means for estimating a first exercise load when a first user is performing a target exercise event based on sensing data related to the first user; a means for determining a first individual index of the first user regarding an exercise stress amount for the target exercise event by performing a predetermined calculation on the first exercise stress amount; a means for calculating a first parameter representing a characteristic of the first user related to exercise tolerance based on the first individual index and a reference value of an exercise load for the target exercise event; An information processing device comprising:

15. A system including a first information processing device and a second information processing device, The first information processing device is means for acquiring sensing data relating to a first user from the second information processing device; means for estimating a first exercise stress amount when the first user is performing a target exercise event based on the sensing data; a means for determining a first individual index of the first user regarding an exercise stress amount for the target exercise event by performing a predetermined calculation on the first exercise stress amount; a means for calculating a first parameter representing a characteristic of the first user related to exercise tolerance based on the first individual index and a reference value of an exercise load for the target exercise event; Equipped with system.

Images