Information processing device, method, program, and system

Abstract

It provides information for planning or instructing an exercise regimen, or a signal indicating the subject's physical function. [Solution] A program of one embodiment of the present disclosure causes a computer to function as: means for acquiring a first exercise intensity when a user is performing a first type of exercise based on first data based on sensing results when the user is performing a first type of exercise; means for acquiring a second exercise intensity when the user is performing a second type of exercise different from the first type of exercise or when the user is at rest based on second data based on sensing when the user is performing a second type of exercise different from the first type of exercise or when the user is at rest; means for determining a personal index of the user by performing a predetermined calculation on multiple exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity; and means for calculating parameters representing the user's characteristics related to exercise tolerance based on the user's personal index and a reference value.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a method, a program, and a system. [Background technology]

[0002] The importance of exercise therapy, which utilizes exercise for the treatment and prevention of disorders and diseases, is increasing. For example, cardiac rehabilitation aims to help cardiac patients recover their physical strength and self-confidence, return to a comfortable home and social life, and prevent cardiac recurrence or re-hospitalization through a comprehensive activity program including exercise therapy. The core of exercise therapy is aerobic exercise such as walking, jogging, cycling, and aerobics. In order to perform aerobic exercise more safely and effectively, it is preferable for the subject to exercise at an intensity close to his or her own anaerobic threshold (AT).

[0003] The anaerobic metabolic threshold is an example of an evaluation index of exercise tolerance, and corresponds to a change point of the cardiopulmonary function state, that is, an exercise intensity near the boundary between aerobic exercise and anaerobic exercise. The anaerobic metabolic threshold is generally determined by a CPX test (cardiopulmonary exercise stress test) in which an examinee is given a gradually increasing exercise load while collecting and analyzing exhaled gas (see Non-Patent Document 1). In the CPX test, the anaerobic metabolic threshold is determined based on the results measured by exhaled gas analysis (e.g., oxygen intake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof). In addition to the anaerobic metabolic threshold, the CPX test can also determine the maximum oxygen intake equivalent to an exercise intensity near the maximum exercise tolerance.

[0004] In order to ensure the efficacy and safety of exercise training, it is important to properly manage the exercise intensity of the subjects.

[0005] Patent Document 1 describes a method for determining whether or not a ventilatory threshold (VT) has been reached based on a subject's pulsation information, and adjusting the exercise load of an exercise providing device according to the determination result. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2022-059494 A [Non-patent literature]

[0007] [Non-Patent Document 1] Saito, Muneyasu, Cardiac Rehabilitation, Physical Therapy, 1997, Vol. 24, No. 8, p. 414-418 Summary of the Invention [Problem to be solved by the invention]

[0008] In the technical idea described in Patent Document 1, feedback control is performed so that the exercise intensity approaches the ventilatory threshold (VT). However, the exercise intensity for the person performing the exercise depends not only on the type of exercise but also on the person's physical function and daily physical condition. Therefore, even if such a technical idea is applied to exercise therapy, it is not possible to obtain information on the level of exercise intensity for the subject when performing a specified type of exercise, or on what type of exercise should be recommended to the subject to achieve the specified exercise intensity. In addition, the technical idea described in Patent Document 1 cannot be simply applied to the case where, for example, a subject is made to exercise by specifying an exercise intensity lower than the ventilatory threshold (VT).

[0009] An object of the present disclosure is to provide information for planning or instructing an exercise regimen, or a signal indicating a subject's physical function. [Means for solving the problem]

[0010] A program according to one embodiment of the present disclosure causes a computer to function as: means for acquiring first data based on sensing results when a user is performing a first type of exercise; means for acquiring a first exercise intensity when the user is performing the first type of exercise based on the first data; means for acquiring second data based on sensing when the user is performing a second type of exercise different from the first type of exercise or at rest; means for acquiring a second exercise intensity when the user is performing the second type of exercise or at rest based on the second data; means for determining a first personal index of the user by performing a predetermined calculation on multiple exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity; and means for calculating a parameter representing the user's characteristics related to exercise tolerance based on the user's first personal index and a first reference value. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing a configuration of a client device according to the present embodiment. [Diagram 3] FIG. 2 is a block diagram showing a configuration of a server according to the present embodiment. [Figure 4] 1 is a block diagram showing a configuration of a wearable device according to an embodiment of the present invention. [Diagram 5] FIG. 1 is an explanatory diagram of one aspect of the present embodiment. [Figure 6] FIG. 2 is a diagram showing a data structure of an exercise event database according to the present embodiment. [Figure 7] FIG. 4 is a diagram showing a data structure of a user profile database according to the present embodiment. [Figure 8] FIG. 4 is a diagram illustrating a data structure of a parameter log database according to the present embodiment. [Figure 9] 13 is a flowchart of an exercise event recommendation process according to the present embodiment. [Figure 10] 11A and 11B are diagrams illustrating an example of a screen displayed in the exercise type recommendation process of the present embodiment. [Figure 11]4 is a flowchart of a parameter monitoring process according to the present embodiment. [Figure 12] FIG. 2 is a diagram showing the data structure of a teacher dataset that can be used in this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and the repeated description will be omitted.

[0013] (1) Information Processing System Configuration The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to the present embodiment.

[0014] As shown in FIG. 1, the information processing system 1 includes a client device 10, a server 30, and a wearable device 50.

[0015] Here, the number of client devices 10 and wearable devices 50 varies depending on, for example, the number of users. Therefore, the number of client devices 10 and wearable devices 50 may each be two or more. Furthermore, a terminal of a person who plans or instructs exercise therapy may also be included in the information processing system 1. The person who plans or instructs exercise therapy may include, for example, a medical professional (e.g., a doctor, a nurse, a pharmacist, a physical therapist, an occupational therapist, or a clinical laboratory technician), a nutritionist, or a trainer.

[0016] The client device 10 and the server 30 are connected via a network (eg, the Internet or an intranet) NW. The client device 10 and the wearable device 50 are connected via a wireless channel, for example using Bluetooth (registered trademark) technology.

[0017] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0018] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a server computer.

[0019] The wearable device 50 is an example of an information processing device that can be worn on the user's body (for example, on the arm, hand, or head).

[0020] (1-1) Client device configuration The configuration of the client device will now be described with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0021] 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0022] The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a Read Only Memory (ROM), a Random Access Memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0023] The programs include, for example, the following programs: ·OS (Operating System) programs · Programs for applications that process information (e.g. web browsers, therapeutic apps, rehabilitation apps, or fitness apps) Here, the diseases that are the target of therapeutic or rehabilitation apps are diseases in which exercise may contribute to improving symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

[0024] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0025] The processor 12 is a computer that realizes the functions of the client device 10 by running a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Gate Array)

[0026] The input / output interface 13 is configured to obtain information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10.

[0027] The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0028] The communication interface 14 is configured to control communications between the client device 10 and external devices (eg, another client device 10, a server 30, and a wearable device 50). Specifically, the communication interface 14 may include a module for communication with the server 30 (e.g., a WiFi module, a mobile communication module, or a combination thereof). The communication interface 14 may include a module for communication with the wearable device 50 (e.g., a Bluetooth module).

[0029] The display 15 is configured to display an image (still image or moving image). The display 15 is, for example, a liquid crystal display or an organic EL display.

[0030] The camera 16 is configured to capture images and generate image signals.

[0031] The depth sensor 17 is, for example, a light detection and ranging (LIDAR) sensor. The depth sensor 17 is configured to measure a distance (depth) from the depth sensor 17 to a surrounding object (for example, a user).

[0032] The microphone 18 is configured to receive sound waves and generate a sound signal, and is preferably placed in the vicinity of the user's body (particularly the respiratory tract), such as an earphone microphone.

[0033] The acceleration sensor 19 is configured to detect acceleration.

[0034] (1-2) Server configuration The configuration of the server will now be described with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0035] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0036] The storage device 31 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage.

[0037] The programs include, for example, the following programs: -OS programs Application programs that perform information processing

[0038] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0039] The processor 32 is a computer that realizes the functions of the server 30 by starting a program stored in the storage device 31. The processor 32 is, for example, at least one of the following. ·CPU GPU ·ASIC FPGA

[0040] The input / output interface 33 is configured to obtain information (eg, a user's instruction) from an input device connected to the server 30 and to output information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0041] The communication interface 34 is configured to control communications between the server 30 and external devices (eg, the client device 10).

[0042] (1-3) Configuration of wearable devices The configuration of the wearable device will now be described with reference to Fig. 4, which is a block diagram showing the configuration of the wearable device of this embodiment.

[0043] 4, the wearable device 50 includes a storage device 51, a processor 52, an input / output interface 53, and a communication interface 54. The wearable device 50 is connected to a display 55, a heart rate sensor 56, and an acceleration sensor 57.

[0044] The storage device 51 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage device.

[0045] The programs include, for example, the following programs: -OS programs · Programs for applications that process information (e.g., medical, rehabilitation, or fitness apps)

[0046] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0047] The processor 52 is a computer that realizes the functions of the wearable device 50 by starting the programs stored in the storage device 51. The processor 52 is, for example, at least one of the following: ·CPU GPU ·ASIC FPGA

[0048] The input / output interface 53 is configured to acquire information (e.g., user instructions, sensing results) from an input device connected to the wearable device 50, and to output information (e.g., images, commands) to an output device connected to the wearable device 50.

[0049] The input device is, for example, a heart rate sensor 56, an acceleration sensor 57, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 55, a speaker, or a combination thereof.

[0050] The communication interface 54 is configured to control communication between the wearable device 50 and an external device (eg, the client device 10). In particular, the communication interface 54 may include a module for communicating with the client device 10 (eg, a Bluetooth module).

[0051] The display 55 is configured to display an image (still image or video). The display 55 is, for example, a liquid crystal display or an organic EL display.

[0052] The heart rate sensor 56 is configured to measure the heart rate and generate a sensing signal. As an example, the heart rate sensor 56 measures the heart rate using an optical measurement technique.

[0053] The acceleration sensor 57 is configured to detect acceleration.

[0054] (2) One aspect of the embodiment An embodiment of the present invention will now be described with reference to Fig. 5, which is an explanatory diagram of an embodiment of the present invention.

[0055] As shown in FIG. 5, the client device 10 and the wearable device 50 sense the user US1 during exercise. The user US sequentially performs a plurality of exercise types. The combination of a plurality of exercise types (hereinafter referred to as a "target exercise type set") used to calculate these individual difference parameters (described later) may be fixed or may be arbitrarily selected. In the latter case, the exercise types constituting the target exercise type set may be determined by the user, a person who plans or instructs the exercise therapy of the user, or an algorithm. In addition, the plurality of exercise types constituting the target exercise type set may be selected from types having different standard exercise intensities. In any case, information on the standard exercise intensity corresponding to each of the exercise types constituting the target exercise type set is stored in an exercise type database described later. The user US1 is typically a person who undergoes exercise therapy, for example, a participant in a (cardiac) rehabilitation program or an exercise instruction program. In the example of FIG. 5, an example is shown in which the exercise type is gymnastics, but the exercise type may include any exercise (aerobic exercise or anaerobic exercise).

[0056] As an example, the camera 16 captures an appearance (e.g., the whole body) of the user US1 during exercise from the front or oblique front at a distance of, for example, about 2 m. The camera 16 may be installed at an appropriate height using a tripod or other height adjustment means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. It is also possible to generate three-dimensional video data by combining, for example, video data (two-dimensional) generated by the camera 16 and, for example, depth data generated by the depth sensor 17. The microphone 18 receives sounds (e.g., sounds generated by breathing or speaking) emitted by the user US1 during exercise and generates a sound signal.

[0057] The heart rate sensor 56 of the wearable device 50 measures the heart rate of the user US1 while exercising and transmits the measurement result to the client device 10. The acceleration sensor 57 measures the acceleration of the user US1 while exercising and transmits the measurement result to the client device 10.

[0058] The client device 10 acquires various sensing data and performs analysis as necessary. As an example, the client device 10 may refer to video data acquired from the camera 16 to analyze the physical state of the user US1 during exercise. The client device 10 may further refer to depth data acquired from the depth sensor 17 to analyze the physical state of the user US1 during exercise. The client device 10 transmits user data including at least one of the sensing data or the analysis result of the sensing data to the server 30.

[0059] Based on the user data acquired from the client device 10, the server 30 estimates the exercise intensity of the user when performing each of the multiple exercise events constituting the target event set. Then, the server 30 performs a predetermined calculation on multiple exercise intensities corresponding to each of the multiple exercise events to determine the individual index (described later) of the user US1 for the target event set. Furthermore, the server 30 calculates an individual difference parameter that represents the characteristics of the user US1 regarding exercise tolerance (e.g., whether the exercise intensity tends to be high or low compared to a standard person) based on the determined individual index and a reference value (described later) of the individual index for the target event set. For example, since the amount of oxygen consumption during exercise depends on muscle mass, there will be individual differences in exercise intensity even for the same exercise event depending on the muscle development of the user US1 and the part of the body to which the load is applied.

[0060] The server 30 stores the calculated individual difference parameters in the storage device 31. Using the individual difference parameters, the server 30 can predict the exercise intensity of the user US1 when performing an exercise type not included in the target exercise set, or estimate what kind of exercise type the user US1 should perform to achieve a desired exercise intensity. Furthermore, by continuously monitoring the individual difference parameters, it is possible to grasp the trend of change (e.g., improvement, maintenance, or decline) in the physical function of the user US1, or the rate of change (e.g., whether it is rapid or not), and when the physical function of the user US1 is rapidly declining, for example, an alert can be output indicating that an abnormality such as an exacerbation of a heart disease such as heart failure, a physical breakdown, or a temporary deterioration in physical condition is suspected.

[0061] In this way, the information processing system 1 calculates individual difference parameters that represent the characteristics of the user US1 related to exercise tolerance, based on data based on the sensing results of the user US1 while performing each of the exercise events constituting the target event set. Therefore, according to the information processing system 1, such individual difference parameters can be used as information for planning or instructing an exercise therapy to be provided to the user US1, or as a signal indicating the physical function or physical condition of the user US1.

[0062] (3) Database The databases of this embodiment are stored in the storage device 31.

[0063] (3-1) Sports event database The exercise event database of this embodiment will be described below. Fig. 6 is a diagram showing the data structure of the exercise event database of this embodiment.

[0064] The exercise type database stores exercise type information. The exercise type information is information about exercise types that are candidates for the recommended exercise type or the target exercise type set. Here, in this embodiment, the exercise types include exercise types that can be performed without using a device that can adjust the exercise load, such as gymnastics, bodyweight training, dancing, walking, running, and treadmills. These exercise types include types performed in a standing position, so they are rich in variation. Furthermore, in this embodiment, the exercise intensity of these exercise types can be adjusted through the form (for example, the positional relationship or angle of parts (which may include parts that are not moved during exercise), the range of motion of parts that are moved during exercise, etc.), pace, number of reps, or the time or number of rest periods. However, the exercise types in this embodiment can also include exercise types performed using a device that can adjust the exercise load, such as an ergometer or strength training using training equipment.

[0065] 6, the exercise event database includes an "ID" field, a "name" field, an "exercise intensity" field, and an "exercise definition" field. Each field is associated with the others.

[0066] The "ID" field stores an exercise type ID. The exercise type ID is information for identifying the exercise type corresponding to the record.

[0067] The "name" field stores information about the name of the exercise type. The exercise type name information is information about the name of the exercise type corresponding to the record.

[0068] The "exercise intensity" field stores exercise intensity information. The exercise intensity information is information on the standard exercise intensity of the exercise type corresponding to the record. Standard exercise intensity may mean, for example, the exercise intensity when a person with standard physical function performs the corresponding exercise type as defined. As an example, such exercise intensity may be derived by actually measuring the exercise intensity (e.g., average oxygen consumption) when one or more people perform the corresponding exercise type as defined, for example, by performing an exhaled gas analysis (or estimating by analyzing a video of the user exercising), and statistically processing the results (e.g., averaging).

[0069] In addition, the exercise intensity may be derived by, for example, measuring (or estimating) the exercise intensity in each section constituting the corresponding exercise type when one or more people perform the corresponding exercise type as defined, applying the measurement results of each section for each person to a predetermined calculation formula, and further performing statistical processing (e.g., averaging) of the values ​​obtained by this calculation formula among the people. Here, a section is a constituent unit of an exercise type, and when an exercise type is composed of multiple movement patterns, each movement pattern may correspond to a section. For example, when the exercise type is a dance, each choreography may correspond to a section. Also, when the exercise type is a squat, the transition from a standing state to a crouching state and the transition from a crouching state to a standing state may each correspond to a section.

[0070] The "exercise definition" field stores exercise definition information. The exercise definition information is information regarding the definition of the exercise event corresponding to the corresponding record. The exercise definition information may include information regarding at least one of the following elements. Exercise pace (for example, the time it takes to complete one cycle of movement) - Positional relationship of parts (which may be one or more) (e.g., foot width, etc.) The angle of a part (which may be one or more) (for example, the direction of the knee, the angle between the upper arm and the forearm) The range of motion of the body parts (which may be one or more) that are moved during exercise (for example, the range of movement of each body part during one cycle of movement) Number of reps Break time Exercise load (for example, the magnitude of the external load set when exercising using an ergometer or other equipment that allows the exercise load to be adjusted)

[0071] Even for an exercise that is generally recognized as a single event (e.g., squats), multiple exercise events with slightly different exercise intensities can be defined by specifying details of pace, form (especially range of motion), number of repetitions, rest time, and exercise load. For example, multiple exercise events with differences of 0.2 METs can be defined for an exercise that is generally recognized as a single event.

[0072] (3-2) User profile database The user profile database of this embodiment will now be described with reference to Fig. 7, which is a diagram showing the data structure of the user profile database of this embodiment.

[0073] The user profile database stores user profile information. The user profile information is information about a profile of a user (that is, a person who exercises) of the information processing system 1.

[0074] 7, the user profile database includes an "ID" field, a "name" field, a "target strength" field, and a "body" field. Each field is associated with the others.

[0075] The "ID" field stores a user ID. The user ID is information for identifying the user corresponding to the corresponding record.

[0076] The "name" field stores user name information. The user name information is information about the name of the user corresponding to the corresponding record (for example, name, account name, etc.).

[0077] In the "target intensity" field, target intensity information (an example of "predetermined exercise intensity") is stored. The target intensity information is information about a target value of exercise intensity (for example, oxygen consumption, energy consumption, heart rate, or a combination thereof) set for the user corresponding to the corresponding record. As an example, the target intensity information is specified by a person (for example, a doctor) who plans or instructs exercise therapy based on the results of measuring the exercise tolerance of the user by CPX when using an ergometer (for example, oxygen consumption and heart rate at the anaerobic metabolic threshold (AT)). However, CPX is not essential, and the target value may be specified at the doctor's discretion. As another example, the target intensity information is determined by an algorithm based on the results of measuring the exercise tolerance of the user by, for example, CPX. In order to perform aerobic exercise more safely and effectively, it is preferable to perform exercise at an intensity near the anaerobic metabolic threshold, so the target value of exercise intensity is, for example, an exercise intensity corresponding to the anaerobic metabolic threshold, but is not limited to this. In addition, an ergometer is usually used as the exercise type when measuring CPX, but when a treadmill is used, the oxygen consumption at the anaerobic metabolic threshold is about 1.2 to 1.3 times that when an ergometer is used. This is considered to be due to the fact that the total muscle mass used in the treadmill exceeds the total muscle mass used in the ergometer. Therefore, as a first example, the target value based on the result measured by CPX when using the ergometer may be corrected to about 1.2 to 1.3 times, or a value obtained by further subtracting a predetermined value (for example, 1 METs) may be used as the target intensity information. As a second example, the target value based on the result measured by CPX when using the ergometer may be used as the target intensity information without correction. This makes it possible to prevent the target value from becoming excessively high when an exercise type in which the total muscle mass used is relatively small, such as an ergometer, is selected. As a third example, the target value for each exercise type may be corrected, for example, by a coefficient according to the total muscle mass used.

[0078] In addition, a doctor may set an upper limit on exercise intensity for a user (an example of exercise prescription). In this case, the user is not permitted to select an exercise type that exceeds the upper limit of the prescribed exercise intensity. For exercise prescription, a UI (User Interface) screen for exercise prescription may be displayed on the display of the terminal used by the doctor. Such a UI screen may include, for example, the following information: User CPX data -Display area for sample videos of multiple selectable sports Here, the display area of ​​the sample video of each exercise type may be arranged according to the exercise intensity information corresponding to the exercise type. For example, when 3.6 METs is recommended as the upper limit based on the CPX data, a group of display areas of sample videos of exercise types corresponding to 3.4 METs, a group of display areas of sample videos of exercise types corresponding to 3.6 METs, and a group of display areas of sample videos of exercise types corresponding to 3.8 METs may be arranged on the UI screen. Note that the information of the exercise type (e.g., sample videos) may be arranged in units smaller than 0.2 METs or larger. When the doctor selects one of the display areas, the exercise intensity associated with the corresponding exercise type is set as the upper limit.

[0079] In addition, the raising or lowering of the upper limit specified by the doctor through the exercise prescription may be performed by a medical professional under the supervision of a doctor during regular (e.g., every two weeks) medical guidance or doctor rounds.

[0080] The "physical" field stores physical information. The physical information is information about the body (functions) of the user corresponding to the corresponding record. As an example, the physical information may include information about the user's age, sex, weight, height, illnesses, etc.

[0081] Additionally, the user profile database may store the following information: Information indicating the person who planned or instructed the user's exercise regimen Information identifying the user's doctor

[0082] (3-3) Parameter log database The parameter log database of this embodiment will be described below with reference to Fig. 8, which is a diagram showing the data structure of the parameter log database of this embodiment.

[0083] The parameter log database may be constructed, for example, for each user (i.e., person who exercises) of the information processing system 1. Alternatively, the parameter log database may be configured to store records including information that can identify a user (e.g., a user ID).

[0084] The parameter log database stores parameter log information, which is information relating to a log of individual difference parameters calculated for a user.

[0085] 8, the parameter log database includes a "date" field and a "parameter" field. Each field is associated with each other.

[0086] The "Date" field stores date information. The date information is information about the date (or date and time) when the individual difference parameter of the corresponding record was calculated.

[0087] The "parameter" field stores parameter information. The parameter information is information about the value of the individual difference parameter of the corresponding record.

[0088] (4) Information processing The information processing of this embodiment will be described.

[0089] (4-1) Exercise type recommendation processing The exercise type recommendation process of this embodiment will be described below. Fig. 9 is a flowchart of the exercise type recommendation process of this embodiment. Fig. 10 is a diagram showing an example of a screen displayed in the exercise type recommendation process of this embodiment.

[0090] The exercise type recommendation process starts when, for example, any one of the following start conditions is met. The exercise recommendation process was called by another process. The user or a person who plans or instructs the user's exercise therapy performs an operation to call up the exercise type recommendation process. The client device 10 enters a predetermined state (for example, a predetermined application is started). The appointed time has arrived. A certain amount of time has passed since a certain event.

[0091] As shown in FIG. 9, the client device 10 executes acquisition of sensing data (S110). Specifically, the client device 10 may enable the operation of the camera 16 to start taking a video of the user exercising (hereinafter referred to as a "user video"). The client device 10 may enable the operation of the depth sensor 17 to start measuring the distance from the depth sensor 17 to each part of the user exercising (hereinafter referred to as a "user depth"). The client device 10 may enable the operation of the microphone 18 to start collecting sound (for example, sound generated by the user's breathing or speaking (hereinafter referred to as a "user sound").

[0092] Furthermore, the client device 10 may cause the wearable device 50 to start measuring the heart rate (hereinafter referred to as the "user's heart rate") using the heart rate sensor 56. Furthermore, the client device 10 may enable any sensor of the client device 10 or the wearable device 50 (for example, the acceleration sensor 19 or the acceleration sensor 57).

[0093] The client device 10 then acquires sensing data from each sensor while the user is performing an exercise event (an exercise event constituting a target event set). Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 may acquire user video data from the camera 16, acquire user depth data from the depth sensor 17, acquire user sound data from the microphone 18, acquire user heart rate data from the wearable device 50, and acquire user acceleration data relating to the acceleration of the user (hereinafter, "user acceleration") from the acceleration sensor 19 or at least one of the wearable device 50. The client device 10 may also acquire user oxygen intake data relating to the oxygen intake of the user from a device that performs a test on exhaled gas (e.g., a CPX test).

[0094] The client device 10 can repeatedly acquire sensing data (S110) while performing one exercise event.

[0095] After step S110, the client device 10 executes generation of user data (S111). Specifically, the client device 10 generates user data based on the sensing data acquired in step S110. The user data may include at least one of the following: Data acquired in step S111 (for example, user video data, user depth data, user sound data, user heart rate data, user acceleration data, or user oxygen intake data) Data obtained by processing the data acquired in step S111 Data acquired by analyzing the user moving image data (or the user moving image data and the user depth data) acquired in step S111 (for example, skeletal data, facial expression data, skin color data, breathing data, or a combination thereof, which will be described later) Information that can identify the type of exercise the user was performing in step S110 (the types of exercise that constitute the target exercise set) - Information showing the user's subjective assessment of the exercise intensity of the completed exercise event (hereinafter referred to as "perceived exercise intensity");

[0096] The client device 10 may repeatedly generate user data (S111) while performing one exercise event, or may generate user data each time one exercise event is completed.

[0097] After step S111, the client device 10 executes transmission of user data (S112). Specifically, the client device 10 transmits to the server 30 the user data generated in step S111.

[0098] The client device 10 may transmit the user data (S112) repeatedly during the performance of one exercise event, may transmit the data each time an exercise event is completed, or may transmit the data only once after all the exercise events constituting the target exercise set have been completed.

[0099] After step S112, the server 30 executes estimation of exercise intensity (S130). Specifically, the server 30 receives the user data transmitted by the client device 10 in step S112. The server 30 estimates the exercise intensity of the user when performing each exercise event constituting the target event set, based on the user data acquired from the client device 10. The exercise intensity may be calculated as, for example, energy consumption (e.g., METs), oxygen consumption, exercise intensity based on heart rate (e.g., exercise intensity calculated by the Karvonen method), or a combination thereof. The server 30 may refer to user profile information stored in a user profile database (FIG. 7) to estimate the exercise intensity.

[0100] As a first example of estimating exercise intensity (S130), the server 30 estimates the exercise intensity when the user performs one of the exercise types constituting the target exercise set by performing a Karvonen method calculation based on the user's heart rate measured over multiple points in time while the user is performing the exercise type.

[0101] As a second example of estimating exercise intensity (S130), the server 30 estimates the exercise intensity when the user performs each exercise event constituting the target event set, using an estimation model described later. Specifically, the server 30 estimates the exercise intensity by applying the estimation model to input data based on user data (e.g., skeletal data, facial expression data, skin color data, respiration data, heart rate data, or a combination thereof).

[0102] As a third example of estimating exercise intensity (S130), the server 30 may calculate the exercise intensity when the user performs each exercise type that constitutes the target exercise set, based on the user's oxygen intake data when performing the exercise type.

[0103] As a fourth example of estimating exercise intensity (S130), the server 30 may use the user's audio data to have an expert, such as a physical therapist, conduct a talk test while the user is performing each of the exercise types that make up the target exercise set, and calculate the exercise intensity when the user performs each of the exercise types based on the results of the talk test.

[0104] As a fifth example of estimating exercise intensity (S130), the server 30 may acquire information on the perceived intensity of exertion (e.g., the Borg index) when performing each exercise type that constitutes the target exercise set, and calculate the exercise intensity when the user performs the exercise type based on the perceived intensity of exertion.

[0105] A sixth example of the estimation of exercise intensity (S130) is a combination of multiple of the above first to fifth examples.

[0106] The server 30 may estimate the exercise intensity for each section constituting the exercise type, and treat a representative value of the estimated result (e.g., average value, median value, mode value, maximum value, minimum value, first quartile, or third quartile) as the exercise intensity of the exercise type. The client device 10 or the server 30 may identify which section each sensing result is associated with, for example, by analyzing user video data (and user depth data as necessary). Alternatively, in the case where a fixed time is assigned to each section, such as in gymnastics or dance, the client device 10 or the server 30 may identify which section each sensing result is associated with based on the time assignment. Furthermore, the server 30 may be a combination of two or more of the above first to third examples as appropriate.

[0107] Since the estimated exercise intensity is not stable immediately after the start of exercise (for example, for 1 to 2 minutes), the estimation of the exercise intensity may be omitted for a predetermined period from the start of exercise, or the estimated exercise intensity may be discarded. In other words, the server 30 may estimate the exercise intensity only when it is determined that the exercise intensity is in a plateau state.

[0108] After step S130, the server 30 executes determination of the personal index (S131). Specifically, the server 30 performs a predetermined calculation using the exercise intensity estimated in step S130 to determine an individual index of exercise intensity for the target event set for the user. As an example, the server 30 treats the exercise intensity estimated for the multiple exercise events constituting the target event set, or the product or ratio of the exercise intensities, as elements (the elements may include values ​​(bias) that are not based on exercise intensity). The server 30 determines the individual index by multiplying the element by a predetermined coefficient or adding a bias, and then performing a predetermined calculation such as finding the sum, product, or ratio of these. For example, the value obtained by subtracting the exercise intensity of event B from the exercise intensity of event A may be used as the individual index. The calculation formula for the individual index may be defined for each target event set, or may be defined commonly across multiple target event sets.

[0109] After step S131, the server 30 executes calculation of individual difference parameters (S132). Specifically, the server 30 calculates individual difference parameters representing the user's characteristics related to exercise tolerance based on the individual indices determined in step S131 and the reference value of the exercise intensity for the target event set. The server 30 creates a record based on the calculated individual difference parameters and the calculation date (or calculation date and time), and adds it to the user's parameter log database (FIG. 8).

[0110] The reference value of the exercise intensity for the target event set can be determined based on information on the exercise intensity when one or more reference persons (e.g., persons with standard exercise tolerance) performed each of the exercise events constituting the target event set, and on the basis of the exercise intensity, calculated for each of the persons. Specifically, the exercise intensity when each person performed each exercise event is acquired (actually measured or estimated), an individual index is calculated using the same calculation formula as in step S131, and the reference value can be determined by performing statistical processing (e.g., calculation of a representative value such as averaging) on ​​the individual index. The reference value may be calculated in advance for each available target event set and stored in a database. The person referred to for calculating the reference value applied to a user may be determined to have at least one attribute (e.g., age group, sex, or a combination thereof) that is the same as that of the user.

[0111] As a first example, the server 30 calculates the individual difference parameter by dividing the individual index by the reference value. In this case, the individual difference parameter corresponds to a prediction result of the ratio of the exercise intensity to be larger or smaller than that of a standard person when the user performs the exercise type constituting the target exercise set or another exercise type.

[0112] As a second example, the server 30 calculates the individual difference parameter by subtracting the reference value from the individual index. In this case, the individual difference parameter corresponds to a prediction result of how much the exercise intensity will be larger or smaller than that of a standard person when the user performs the exercise type constituting the target exercise set or another exercise type.

[0113] After step S132, the server 30 executes the selection of a recommended exercise type (S133). Specifically, the server 30 refers to the user profile database (FIG. 7) and acquires the user's target intensity information (which is determined based on the results of measuring the user's exercise tolerance as described above). The server 30 selects an exercise type (hereinafter referred to as a "recommended exercise type") that is suitable for the characteristics of the user's exercise tolerance based on the acquired target intensity information, the individual difference parameters calculated in step S132, and the exercise intensity information of each exercise type stored in the exercise type database (FIG. 6). There may be one or more recommended exercise types.

[0114] As a first example, the server 30 corrects the target value of the exercise intensity of the user by the individual difference parameter. For example, the server 30 obtains the corrected target value by dividing the target value by the individual difference parameter or subtracting the individual difference parameter from the target value. Then, the server 30 selects the recommended exercise type from the exercise types whose exercise intensity (hereinafter referred to as "standard intensity") indicated by the corresponding exercise intensity information does not exceed the corrected target value. For example, the server 30 may include in the recommended exercise types the exercise types whose standard intensity is maximum within a range equal to or less than the corrected target value. Note that, in selecting the recommended exercise type, the server 30 may use, instead of the corrected target value, a value obtained by adding or subtracting a margin to the corrected target value, or a value obtained by multiplying the corrected target value by a positive coefficient different from 1. Alternatively, in selecting the recommended exercise type, the server 30 may use, instead of the standard intensity, a value obtained by adding or subtracting a margin to the standard intensity, or a value obtained by multiplying the standard intensity by a positive coefficient different from 1.

[0115] As a second example, the server 30 corrects the standard intensity of each exercise type by the individual difference parameter. For example, the server 30 obtains the corrected intensity by multiplying or adding the standard intensity by the individual difference parameter. Then, the server 30 selects the recommended exercise type from the exercise types whose corrected intensity does not exceed the target value of the exercise intensity of the user. For example, the server 30 may include in the recommended exercise types the exercise types whose corrected intensity is maximum within a range equal to or less than the target value. Note that, in selecting the recommended exercise type, the server 30 may use, instead of the target value, a value obtained by adding or subtracting a margin to the target value, or a value obtained by multiplying the target value by a positive coefficient different from 1. Alternatively, in selecting the recommended exercise type, the server 30 may use, instead of the corrected intensity, a value obtained by adding or subtracting a margin to the corrected intensity, or a value obtained by multiplying the corrected intensity by a positive coefficient different from 1.

[0116] After step S133, the server 30 transmits recommended exercise event information (S134). Specifically, the server 30 transmits information about the recommended exercise selected in step S133 (hereinafter referred to as "recommended exercise information") to the client device 10. The recommended exercise information may be, for example, information for identifying the recommended exercise, or may be screen information for displaying the recommended exercise.

[0117] After step S134, the client device 10 executes screen display (S113). Specifically, the client device 10 receives the recommended exercise information transmitted by the server in step S134. The client device 10 displays on the display 15 a screen based on the recommended exercise information.

[0118] For example, the client device 10 displays the screen of Fig. 10 on the display 15. The screen of Fig. 10 includes objects J20 to J23.

[0119] The object J20 displays a recommended exercise. The object J20 also accepts a user instruction to start the recommended exercise or to play a model video of the recommended exercise. When the object J20 is selected, the client device 10 may re-execute the exercise type recommendation process of this embodiment with the exercise type corresponding to the object J20 as an exercise type constituting a new target exercise set. Here, the server 30 may calculate the user's individual index for the new target exercise set, and calculate the user's individual difference parameters based on the individual index and a reference value corresponding to the new target exercise set. Alternatively, when the object J20 is selected, the client device 10 may play a model video of the exercise type corresponding to the object J20.

[0120] The object J21 accepts a user instruction to select an exercise type from among those other than the recommended exercise types. When the object J21 is selected, the client device 10 may, for example, display a list of exercise types other than the recommended exercise types and accept a user instruction to select an exercise type. In response to the acceptance of such a user instruction, the client device 10 may re-execute the exercise type recommendation process of this embodiment with the selected exercise type as an exercise type constituting a new target exercise set. Here, the server 30 may calculate the user's individual index for the new target exercise set, and calculate the user's individual difference parameters based on the individual index and a reference value corresponding to the new target exercise set. Alternatively, when any exercise type is selected, the client device 10 may play a model video of the exercise type.

[0121] The object J22 receives a user instruction to end the exercise. When the object J22 is selected, the client device 10 ends the exercise type recommendation process of this embodiment.

[0122] After step S113, the client device 10 may end the exercise event recommendation process (FIG. 9).

[0123] (4-2) Parameter monitoring process The parameter monitoring process of this embodiment will be described below with reference to a flowchart of FIG 11.

[0124] The parameter monitoring process of this embodiment may be started, for example, each time the exercise type recommendation process (FIG. 9) of this embodiment (particularly, the calculation of individual difference parameters (S133)) is performed, or may be repeatedly performed at a predetermined cycle for each user.

[0125] As shown in FIG. 11, the server 30 acquires parameter log information (S230). Specifically, the server 30 refers to the parameter log database of the user to be processed and acquires the parameter log information.

[0126] After step S230, the server 30 executes a determination of a predetermined condition (S231). Specifically, the server 30 determines whether or not a predetermined condition is satisfied for the parameter log information acquired in step S230. For example, the predetermined condition may include at least one of the following. The representative value of a certain number of recent individual difference parameters (which may be one or more) exceeds a threshold value (i.e., the user is predicted to exercise at an excessively high intensity compared to a typical person). The representative value of the individual difference parameter for the recent specified period exceeds a threshold value (i.e., the user is predicted to exercise at a higher intensity than the average person). The rate of increase in the individual difference parameter in the most recent specified number of times exceeds the threshold value (i.e., it is predicted that the user's exercise intensity for the same exercise type is rapidly increasing) The rate of increase in the individual difference parameter in the most recent specified period exceeds the threshold value (i.e., the user's exercise intensity for the same exercise type is predicted to increase rapidly)

[0127] If it is determined in step S231 that the predetermined condition is met, the server 30 executes output of an alert (S232). Specifically, the server 30 transmits the alert to the user's client device 10 or a predetermined terminal. The predetermined terminal may be a terminal of a person designated by the user (e.g., a family member), a terminal of the user's doctor, or a terminal of a person who plans or instructs the user's exercise therapy.

[0128] The alert may include, for example, at least one of the following information: -The most recent individual difference parameters Individual difference parameters over a recent specified period -Increase rate of individual difference parameters in the most recent specified number of -Increase in individual difference parameters in the most recent specified period A message (e.g., text, image, audio, or a combination thereof) informing the user that they may be experiencing a cardiac event. A message indicating that the user is suspected of having a physical injury A message indicating that the user may be experiencing a temporary illness

[0129] Based on the alert received from the server 30, the alert output destination (the user's client device 10 or a predetermined terminal) presents information by an image, sound, vibration, or a combination thereof.

[0130] After step S232, the server 30 ends the parameter monitoring process of this embodiment.

[0131] If it is determined in step S231 that the predetermined condition is not met, the server 30 skips outputting the alert (S232) and ends the parameter monitoring process of this embodiment.

[0132] (5) Summary As described above, the server 30 of this embodiment acquires first data based on the sensing results when the user is performing a first type of exercise, and acquires a first exercise intensity when the user is performing the first type of exercise based on the first data. The server 30 acquires second data based on the sensing results when the user is performing a second type of exercise different from the first type of exercise, and acquires a second exercise intensity when the user is performing the second type of exercise based on the second data. The server 30 determines a first individual index of the user by performing a predetermined calculation on multiple exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity, and calculates a parameter representing the user's characteristics related to exercise tolerance based on the first individual index of the user and the first reference value. This allows the individual difference parameter to be used as a decision-making material for planning or instructing an exercise therapy provided to the user (which may include a prescription), or as a signal indicating the user's physical function or physical condition. In addition, a more reliable individual index can be calculated by referring to the exercise intensities for multiple types of exercise.

[0133] The server 30 may use the parameters, the user's target exercise intensity, and a database that associates each of the multiple exercise types with a standard exercise intensity to select a recommended exercise type for the user from among multiple exercise types, and output information indicating the recommended exercise type. This allows the server 30 to recommend an exercise type by not only matching the measurement result of the user's exercise tolerance with the standard exercise intensity of each exercise type, but also taking into consideration the characteristics of the user's exercise tolerance.

[0134] The server 30 may select, as a recommended exercise type, an exercise type associated with an exercise intensity that does not exceed a corrected exercise intensity obtained by correcting the target exercise intensity using a parameter. This makes it possible to recommend an exercise type whose exercise intensity is expected not to exceed the target exercise intensity when actually performed by the user.

[0135] The server 30 may select the recommended exercise types so as to include an exercise type associated with a maximum exercise intensity that does not exceed the corrected exercise intensity among the multiple exercise types. This makes it possible to recommend an exercise type that is estimated to have an exercise intensity that does not exceed the target exercise intensity and is close to the target exercise intensity when actually performed by the user.

[0136] The target exercise intensity may be determined according to an exercise intensity designated by an algorithm or by a person who plans or instructs the user's exercise therapy based on the measurement result of the user's exercise tolerance. This makes it possible to recommend an exercise type that matches the exercise intensity determined to be suitable for the user from a medical point of view.

[0137] The first reference value may be a representative value of the first individual indexes of a plurality of persons calculated using the same formula as the first individual index of the user, thereby making it possible to obtain an individual difference parameter that appropriately represents the characteristics of the exercise tolerance of the user.

[0138] The multiple persons may be defined to have at least one attribute that is the same as the user, thereby obtaining individual difference parameters that more appropriately characterize the user's exercise tolerance.

[0139] The plurality of exercise types may include an exercise type that can be performed without using a device capable of adjusting an exercise load, thereby making it possible to enrich the types of exercise that can be used by the user.

[0140] The first exercise type may have a different standard exercise intensity from the second exercise type. This makes it easier to obtain a reasonable individual difference parameter, for example, when calculating an individual index using a difference or ratio between the exercise intensities.

[0141] The multiple exercise intensities may be expressed using at least one of oxygen consumption and energy consumption, thereby obtaining an individual difference parameter that represents the characteristics of the oxygen consumption or energy consumption of the user.

[0142] The server 30 may acquire third data based on the sensing result when the user is performing a third exercise type selected from the recommended exercise types, and may acquire a third exercise intensity when the user is performing the third exercise type based on the third data. The server 30 may determine a second individual index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, including the third exercise intensity, and calculate a parameter based on the second individual index and a second reference value. This allows individual difference parameters to be calculated for various combinations of exercise types.

[0143] (6) Estimation model As described above, the server 30 may estimate the exercise intensity using an estimation model. In this case, the estimation model corresponds to a trained model created by supervised learning using a teacher data set described below, or a derived model or distilled model of the trained model. The estimation model may be constructed for each type of exercise, or may be constructed in common across multiple types of exercise.

[0144] (6-1) Training Data Set A teacher data set that can be used in supervised learning for constructing an estimation model will be described below. Fig. 12 is a diagram showing the data structure of a teacher data set that can be used in this embodiment.

[0145] As shown in Fig. 12, the teacher data set includes multiple teacher data. The teacher data is used for training or evaluating a model to be learned (hereinafter, referred to as a "target model"). The teacher data includes a sample ID, input data, and correct answer data.

[0146] The sample ID is information for identifying the training data.

[0147] The input data is data that is input to the target model during training or evaluation. The input data corresponds to examples used during training or evaluation of the target model. As an example, the input data is data regarding the subject's physical state during exercise (i.e., relatively dynamic data) and data regarding the subject's health condition (i.e., relatively static data). At least a portion of the data regarding the subject's physical state is obtained by analyzing the subject's physical state with reference to the subject video data (or the subject video data and subject depth data).

[0148] The subject video data is data on a subject video of a subject during exercise. The subject video data can be acquired, for example, by capturing an external view (e.g., the whole body) of a subject during an examination related to exhaled gas (e.g., a CPX examination) from the front or diagonally in front (e.g., 45 degrees forward) with a camera (e.g., a camera mounted on a smartphone).

[0149] The subject depth data is data on the distance (depth) from the depth sensor to each part of the subject during exercise. The subject depth data can be acquired by operating the depth sensor when taking a video of the subject.

[0150] The subject may be the same person as the user for whom estimation of exercise intensity based on exercise tolerance is performed when the information processing system 1 is operated, or may be a different person. By making the subject and the user the same person, the target model may learn the personality of the user, and the estimation accuracy may be improved. On the other hand, allowing the subject to be a person different from the user has the advantage that it is easy to enrich the teacher data set. In addition, the subject may be composed of multiple people including the user, or multiple people excluding the user.

[0151] In the example of FIG. 12, the input data includes bone structure data, facial expression data, skin color data, respiration data, heart rate data, and health condition data.

[0152] The skeletal data is data (e.g., features) related to the skeleton of a subject during exercise. The skeletal data includes, for example, data related to the speed or acceleration of each part of the subject (which may include data related to changes in the parts of the muscles used by the subject, or data related to the subject's bodily sensation). The skeletal data can be obtained by analyzing the skeleton of a subject during exercise with reference to the subject video data (or the subject video data and the subject depth data). As an example, Vision, which is an SDK for iOS (registered trademark) 14, or other skeletal detection algorithms (e.g., OpenPose, PoseNet, MediaPipe Pose) can be used to analyze the skeleton. Alternatively, the skeletal data for the teacher dataset can be obtained, for example, by having the subject exercise with a motion sensor attached to each part of the subject.

[0153] The results of skeletal detection can be used for quantitative evaluation of exercise, qualitative evaluation, or a combination of both. As a first example, the results of skeletal detection can be used to count the number of reps. As a second example, the results of skeletal detection can be used to evaluate the form of an exercise or the appropriateness of the load applied by the exercise. For example, if the exercise is a squat, the results of skeletal detection can be used to evaluate whether the knees are sticking out too far forward, resulting in a dangerous form, or whether the hips are lowered firmly and deeply to apply sufficient load.

[0154] Facial expression data is data (e.g., features) about a subject's facial expressions while exercising. Facial expression data can be analyzed by applying an algorithm or trained model to the subject video data. Alternatively, facial expression data for a training dataset can be obtained, for example, by humans labeling the subject videos.

[0155] Skin color data is data (e.g., features) about the skin color of a subject during exercise. Skin color data can be analyzed by applying an algorithm or trained model to the subject video data. Alternatively, skin color data for a training dataset can be obtained, for example, by humans labeling the subject videos.

[0156] The respiratory data is data (e.g., features) related to the subject's breathing during exercise. The respiratory data relates, for example, to the number of breaths per unit time or the breathing pattern. The breathing pattern may include at least one of the following: Ventilation rate Ventilation volume Ventilation rate (i.e. the amount of ventilation per unit time, or the number of ventilations) Ventilation acceleration (i.e. the time derivative of the ventilation rate) Carbon dioxide emission concentration Carbon dioxide emissions (VCO2) Oxygen intake concentration Oxygen Uptake (VO2) The data regarding the breathing pattern may include data that can be calculated based on a combination of the above data, such as the gas exchange ratio R (=VCO2 / VO2).

[0157] The respiration data can be obtained, for example, by analyzing the skeletal data. As an example, the following items can be analyzed from the skeletal data: Movement (spreading) of the shoulders, chest (which may include the lateral chest), abdomen, or a combination thereof Inhalation time Exhalation time - Use of accessory respiratory muscles

[0158] The respiratory data for the teacher dataset can be obtained, for example, from the results of a test on exhaled gas performed on a subject during exercise. Details of the exhaled gas test that can be performed on a subject during exercise will be described later. Alternatively, the ventilation rate, ventilation volume, ventilation speed, or ventilation acceleration of the respiratory data for the teacher dataset can be obtained, for example, from the results of a respiratory function test (e.g., a pulmonary function test or a spirometry test) performed on a subject during exercise. In this case, the respiratory function test can be performed using not only medical equipment but also commercially available testing equipment.

[0159] The heart rate data is data (e.g., feature quantity) related to the heart rate of the subject during exercise. The heart rate data can be obtained, for example, by analyzing the subject video data or the analysis results thereof (e.g., skin color data). Alternatively, the heart rate data for the teacher dataset can be obtained, for example, from the results of a test related to exhaled gas together with the respiratory data described below. The subject heart rate data for the teacher dataset can also be obtained by having the subject perform the above exercise while wearing a heart rate sensor or electrodes for an electrocardiogram monitor.

[0160] The health condition data is data related to the subject's health condition. The health condition data can be obtained in various ways. The subject's health condition data may be obtained before, during, or after the subject's exercise. The subject's health condition data may be obtained based on a declaration from the subject or the subject's doctor, may be obtained by extracting information linked to the subject in a medical information system, or may be obtained via the subject's app (e.g., a health care app).

[0161] The health condition includes at least one of the following: ·age ·sex ·height ·body weight ·Body fat percentage Muscle mass ·Bone density History of current illness ·Past history Medication history Surgical history Life history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty score, etc.) Family history Respiratory function test results Results of tests other than respiratory function tests (e.g., blood tests, urine tests, electrocardiograms (including Holter ECGs), cardiac ultrasounds, X-rays, CT scans (including cardiac morphological CT and coronary artery CT), MRI scans, nuclear medicine tests, PET scans, etc.) Data obtained during cardiac rehabilitation (including Borg index)

[0162] The correct answer data is data that corresponds to the correct answer for the corresponding input data (example question). The target model is trained (supervised learning) to output the correct answer data as close as possible to the input data. As an example, the correct answer data represents the intensity of exercise.

[0163] Exercise intensity is an index for quantitatively evaluating the exercise load imposed on a subject. Exercise intensity can be expressed numerically using at least one of the following: Energy (calorie) consumption Oxygen consumption Heart rate

[0164] The correct answer data can be obtained, for example, from the results of a test on exhaled gas conducted on a subject during exercise. A first example of a test on exhaled gas is a test (typically a CPX test) conducted while a subject wearing an exhaled gas analyzer is performing exercise with incremental load (e.g., an ergometer). A second example of a test on exhaled gas is a test conducted while a subject wearing an exhaled gas analyzer is performing exercise of a constant or variable intensity (e.g., bodyweight exercise, gymnastics, strength training).

[0165] Alternatively, the correct answer data can be obtained from the results of a test other than the exhaled gas performed on the subject during exercise. Specifically, the correct answer data can be obtained from the results of a cardiopulmonary exercise load prediction test based on the lactate concentration measurement in the sweat or blood of the subject during exercise. A wearable lactate sensor may be used to measure the lactate concentration of the subject.

[0166] It is also possible to construct an estimation model for each of a plurality of health condition categories based on (at least a part of) the subject's health condition. In this case, (at least a part of) the user's health condition may be referenced to select the estimation model. In this further modification, the input data for the estimation model may be data that is not based on the user's health condition, or may be data that is based on the user's health condition and the user's video.

[0167] (7) Other modifications The storage device 11 may be connected to the client device 10 via a network NW. Each input device or output device may be integrated with the client device 10. The storage device 31 may be connected to the server 30 via the network NW. Each input device or output device may be integrated with the wearable device 50.

[0168] In the embodiment, the information processing system 1 is implemented by a client / server system. However, the information processing system 1 may be implemented by a peer-to-peer system or a standalone computer. As an example, the client device 10 may estimate the exercise intensity.

[0169] Each step of the above information processing can be executed by either the client device 10 or the server 30. As an example, the server 30, instead of the client device 10, may acquire at least a part of the user data or the user skeletal data by analyzing the user video (or the user video and the user depth).

[0170] One or more steps of the above information processing may be performed using a trained model.

[0171] In the above example, a recommended exercise type is selected based on the standard exercise intensity of each of a plurality of exercise types, the user's individual difference parameters, and the results of measuring the exercise tolerance of the user. However, the server 30 may determine instructions for adjusting the exercise intensity of the exercise type selected by the user based on these pieces of information (in other words, presenting an exercise type (variation) in which only a part of the exercise definition information is different from the exercise type selected by the user), and output the information of the instructions to the client device 10. The instructions may include, for example, the form of the exercise type (for example, the positional relationship or angle of a part (which may include a part that is not moved during exercise), or the range of motion of a part that is moved during exercise), pace, number of reps, or the time or number of rest periods. As a result, even if it is estimated that the exercise intensity of the exercise type selected by the user will exceed the above-mentioned target value, it is possible to prevent the user's exercise intensity from deviating from the target value by specifying a form with a lighter exercise intensity than the standard, slowing down the pace than the standard, reducing the number of reps than the standard, or increasing the time or number of rest periods. Alternatively, even if it is estimated that the exercise intensity of the user's selected exercise type will be below the aforementioned target value, the user can prevent the exercise intensity from deviating from the target value by specifying a form that results in a heavier exercise intensity than standard, setting a faster pace than standard, increasing the number of repetitions than standard, or reducing the time or number of rest periods.

[0172] In the above description, an example was shown in which a recommended type of exercise was selected based on the standard exercise intensity of each of a plurality of exercise types, the user's individual difference parameters, and the results of measuring the exercise tolerance of the user. However, the server 30 may select a recommended type of exercise based on at least one of the pulse rate, respiration rate, or perceived exercise intensity of the user during exercise. Here, the perceived exercise intensity can be acquired, for example, by receiving an input of the Borg index from the user via the client device 10 during a rest period. When at least one of the pulse rate, respiration rate, or perceived exercise intensity exceeds an upper limit, the server 30 may select an exercise type with a lower standard exercise intensity as a recommended type of exercise compared to other cases. On the other hand, when at least one of the pulse rate, respiration rate, or perceived exercise intensity falls below a lower limit, the server 30 may select an exercise type with a higher standard exercise intensity as a recommended type of exercise compared to other cases. However, when an upper limit of exercise intensity is set for the user by a doctor, an exercise type corresponding to an exercise intensity exceeding the upper limit is not selected. Furthermore, the pulse rate, respiration rate, or perceived exertion level of the user during exercise may be used to adjust the exercise intensity of the next set when performing an exercise event consisting of multiple sets. For example, when at least one of the pulse rate, respiration rate, or perceived exertion level exceeds an upper limit, the server 30 may determine instructions (e.g., form, pace, number of reps, or time or number of rest periods) so that the exercise intensity of the next set is lower than that of the current set, and output information on the instructions to the client device 10. On the other hand, when at least one of the pulse rate, respiration rate, or perceived exertion level falls below a lower limit, the server 30 may determine instructions so that the exercise intensity of the next set is higher than that of the current set, and output information on the instructions to the client device 10. However, when an upper limit of exercise intensity is set for the user by a doctor, instructions corresponding to an exercise intensity exceeding the upper limit are not selected. For example, the server 30 may determine a recommended type of exercise or instructions to reduce the exercise intensity of the next set by 0.2 METs when the average heart rate is 5 or more higher than the target heart rate or the perceived intensity of exercise is 14 or higher. The server 30 may determine a recommended type of exercise or instructions to reduce the exercise intensity of the next set by 0.4 METs when the average heart rate is 10 or more higher than the target heart rate or the perceived intensity of exercise is 16 or higher. The server 30 may also determine a recommended type of exercise or instructions to increase the exercise intensity of the next set by 0.2 METs when the average heart rate is 5 or more lower than the target heart rate and the perceived intensity of exercise is less than 10.

[0173] The client device 10 may further execute the following processes, for example, when acquiring the sensing data (S110) (in other words, while the user is exercising). Specifically, the client device 10 estimates the user's movements during exercise (for example, the movements of the skeleton or other feature points over multiple points of time, or the state of the skeleton or other feature points at a single point of time) based on the sensing data. Then, when the result of the estimation of the movements does not match at least one of the forms or paces defined for the type of exercise being performed by the user (for example, the range of motion of a part is too narrow or too wide, the angle of a part deviates from the standard, or the pace is too fast or too slow, etc.), the client device 10 provides feedback to the user. The feedback may include at least one of the following: Light, sound or voice output (e.g., voice that tells the user what they did wrong) Displaying images (e.g., images that tell the user what they did wrong) Vibration on wearable devices This allows the user's personal index to be estimated more accurately. Alternatively, the client device 10 may recommend to the user to repeat the same exercise type or to perform a different exercise type when the number or frequency of the results of the estimation regarding the movement not conforming to at least one of the form or pace defined for the exercise type being performed by the user exceeds a threshold value. In this case, the client device 10 may omit the processes subsequent to the generation of user data (S111).

[0174] Instead of the exercise type recommendation process, the information processing system 1 may perform a process for measuring the oxygen consumption at the anaerobic metabolic threshold. This process may be automatically selected, for example, during the first three days after the user starts the exercise therapy, or may be selected by the user or the person who plans or instructs the exercise therapy. In this mode, the information processing system 1 sequentially selects exercise types so that the exercise intensity increases by 0.2 METs each time until one of the following conditions is met: the average heart rate is 5 or more higher than the target heart rate, or the perceived intensity of exercise is 14 or more, and the user exercises. The information processing system 1 treats the oxygen consumption estimated for the exercise type performed immediately before the exercise type for which the above condition is met as the oxygen consumption at the user's anaerobic metabolic threshold. The information processing system 1 may also treat the exercise intensity corresponding to the exercise type for which the average heart rate closest to the target heart rate is obtained as the optimal exercise intensity.

[0175] In the above description, an example was described in which an alert is output when the user's individual difference parameters satisfy a predetermined condition. However, the alert may be output by various triggers other than the individual difference parameters. As an example, the server 30 accumulates data on the types of exercise the user has performed in the past and the user's heart rate, perceived intensity of exercise, or respiration rate when performing the type of exercise. The server 30 may output an alert when the user's heart rate, perceived intensity of exercise, or respiration rate during exercise exceeds a threshold value and increases (worsens) compared with the user's heart rate, perceived intensity of exercise, or respiration rate when performing the same type of exercise in the past. In addition, if the degree of distress of the user (hereinafter referred to as "distress degree") is quantified based on, for example, facial expression data or skeletal data, it can be used as a trigger similar to the above-mentioned heart rate, perceived intensity of exercise, or respiration rate. In addition, the server 30 may use as a trigger an input regarding the emergence or worsening of the user's subjective symptoms of weight gain, shortness of breath, edema, fatigue, loss of appetite, or insomnia in daily life, or a sensor input suggesting the emergence or worsening of these symptoms.

[0176] In the above description, an example of calculating the user's individual difference parameters in the exercise type recommendation process or parameter monitoring process has been shown. However, the user's individual difference parameters may be calculated based on data obtained during a CPX test, for example. For example, the individual difference parameters may be calculated based on the exercise intensity (actual measurement of oxygen intake, for example) when the user is rowing the ergometer with a first load (for example, 20 W) and the exercise intensity when the user is rowing the ergometer with a second load (for example, 40 W). When a doctor sets an upper limit of exercise intensity for a user (an example of exercise prescription), the exercise type to be presented to the doctor can be selected based on the CPX data and the individual difference parameters. As an example, when 3.6 METs is recommended as the upper limit based on the CPX, a group of display areas of sample videos of exercise types corresponding to the above-mentioned corrected intensity of 3.4 METs, a group of display areas of sample videos of exercise types corresponding to a corrected intensity of 3.6 METs, and a group of display areas of sample videos of exercise types corresponding to a corrected intensity of 3.8 METs may be arranged on the UI screen. Note that information on exercise events (eg, sample videos) may be arranged in units smaller than 0.2 METs or larger than 0.2 METs.

[0177] In the above embodiment, an example is shown in which the user's individual difference parameter is calculated based on the exercise intensity when the user is performing a plurality of different exercise types. However, instead of one of the exercise types, the user's exercise intensity in a resting state may be used. By defining a standard exercise intensity in a resting state and acquiring the user's exercise intensity in a resting state, the individual difference parameter can be calculated in the same manner.

[0178] In the above description, an example has been given in which a user video is captured using the camera 16 of the client device 10. However, the user video may be captured using a camera other than the camera 16. An example has been given in which a user depth is measured using the depth sensor 17 of the client device 10. However, the user depth may be measured using a depth sensor other than the depth sensor 17.

[0179] In the above description, an example has been given in which the wearable device 50 measures the user's heart rate. However, the heart rate can be acquired by analyzing (for example, rPPG (Remote Photoplethysmography) analysis) video data or the analysis results thereof (for example, skin color data). The analysis of the heart rate may be performed by a trained model constructed using machine learning technology. Alternatively, the user may be made to exercise while wearing electrodes for an electrocardiogram monitor, so that the electrocardiogram monitor can measure the user's heart rate. In these modified examples, the user does not need to wear the wearable device 50 to measure the heart rate.

[0180] The wearable device 50 may be equipped with a sensor for measuring at least one of the following items instead of the heart rate sensor 56 and the acceleration sensor 57, or in addition to the heart rate sensor 56 and the acceleration sensor 57. Blood sugar levels Oxygen saturation The measurement results of each sensor may be used as appropriate in generating input data, estimating exercise intensity or ventilation index, presenting information based on the estimation results, or in other situations. As one example, the measurement results of blood glucose level may be referenced to evaluate exercise intensity converted into, for example, energy consumption or oxygen consumption. As another example, the measurement results of acceleration may be used to determine the score of a user's exercise (e.g., gymnastics).

[0181] It is also possible to use the acceleration data as part of the input data for the estimation model described in the present embodiment or the modified example. Alternatively, the user's skeleton may be analyzed by referring to the acceleration data. The acceleration data may be acquired by the acceleration sensor 19 or the acceleration sensor 57 when the user's video is shot, for example.

[0182] It is also possible to use oxygen saturation data as part of the input data for the estimation model described in this embodiment or the modified example. The oxygen saturation data can be obtained, for example, by having the user wear a wearable device equipped with a sensor (e.g., an optical sensor) capable of measuring blood oxygen concentration when shooting a user video, or a pulse oximeter. The oxygen saturation data may be estimated, for example, by performing rPPG analysis on the user video data.

[0183] The information processing system 1 of this embodiment and each modified example is also applicable to a video game in which the game progress is controlled according to the player's body movements. The video game may be a mini-game that can be played while the above-mentioned treatment app, rehabilitation app, or fitness app is being executed, or may be configured as a standalone app. As an example, the information processing system 1 may estimate the user's exercise intensity during game play, and determine one of the following depending on the result of the estimation (e.g., a numerical value indicating the user's exercise intensity). This can enhance the effect of the video game on improving the user's health. The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) offered to the user The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) provided to users Game parameters related to the progression of a video game (e.g. score, damage)

[0184] The information processing system 1 of this embodiment and each modified example is also applicable to a video game in which the game progress is controlled according to the movement of the player's body. The video game may be a mini-game that can be played while the above-mentioned treatment app, rehabilitation app, or fitness app is being executed, or may be configured as a standalone app. As an example, the information processing system 1 performs an estimation regarding the user's skeleton based on the user's video during gameplay. The estimation regarding the user's skeleton may be performed based on at least one of the user depth or the user acceleration in addition to the user's video. The information processing system 1 evaluates the degree to which the user's posture during exercise (e.g., gymnastics) matches an ideal posture (model) based on the result of the estimation regarding the user's skeleton. The information processing system 1 may determine any one of the following depending on the result of this evaluation (e.g., a numerical value indicating the degree to which the user's posture matches the ideal posture). This can enhance the effect of the video game on improving the user's health. The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) offered to the user The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) provided to users Game parameters related to the progression of a video game (e.g. score, damage)

[0185] In addition to or instead of the microphone 18, a microphone of the wearable device 50 (a microphone provided in the wearable device 50 or connected to the wearable device 50) may receive sound waves emitted by the user when capturing a user video and generate sound data. The sound data may constitute input data for the estimation model described in this embodiment or the modified example. The sound emitted by the user is, for example, at least one of the following: Sound waves emitted by the rotation of the user's legs (e.g., sounds coming from the pedals or from the actuators connected to the pedals) - Sounds that occur when the user breathes or speaks

[0186] In the above description, the CPX test is exemplified as a test related to exhaled gas. In the CPX test, a gradually increasing exercise load is applied to the test subject. However, it is not necessary to gradually increase the exercise load applied to the user when shooting the user video. Specifically, the real-time exercise intensity can be estimated even when a constant or constantly changeable exercise load is applied to the user. For example, the exercise performed by the user may be bodyweight exercise, calisthenics, or muscle training.

[0187] Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited to the above embodiment. Furthermore, the above embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above embodiment and the modified examples can be combined. [Explanation of symbols]

[0188] 1: Information processing system 10: Client device 11:Storage device 12: Processor 13: Input / Output Interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 21: Display 30: Server 31: Storage device 32 : Processor 33: Input / Output Interface 34: Communication interface 50: Wearable devices 51 :Storage device 52: Processor 53: Input / Output Interface 54: Communication interface 55: Display 56: Heart rate sensor 57: Acceleration sensor

Claims

1. Computer, A means for acquiring first data based on a sensing result when a user is performing a first type of exercise; a means for acquiring a first exercise intensity when the user is performing the first exercise event based on the first data; A means for acquiring second data based on sensing while the user is performing a second exercise type different from the first exercise type or while the user is at rest; A means for acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data; a means for determining a first personal index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, the plurality of exercise intensities including the first exercise intensity and the second exercise intensity; A means for calculating a parameter representing a characteristic of the user related to exercise tolerance based on a first individual index of the user and a first reference value of exercise intensity for a target event set which may include a resting state; A program that functions as a

2. The computer, a means for selecting a recommended exercise type for the user from among the plurality of exercise types using the parameters, a target exercise intensity for the user, and a database that associates each of a plurality of exercise types with a standard exercise intensity; A means for outputting information indicating the recommended exercise type; To function as The program according to claim 1.

3. the selecting means selects, as the recommended exercise type, an exercise type associated with an exercise intensity not exceeding a corrected exercise intensity obtained by correcting the target exercise intensity using the parameter; The program according to claim 2.

4. The selecting means selects the recommended exercise type so as to include an exercise type associated with a maximum exercise intensity not exceeding the corrected exercise intensity among the plurality of exercise types. The program according to claim 3.

5. The target exercise intensity is determined according to an exercise intensity designated by a person who plans or instructs the exercise therapy of the user or an algorithm based on the measurement result of the exercise tolerance of the user. The program according to claim 2.

6. the first reference value is a representative value of first individual indicators of a plurality of persons calculated using the same calculation formula as the first individual indicator of the user; The program according to claim 1.

7. The plurality of persons are defined to have at least one attribute identical to that of the user. The program according to claim 6.

8. The computer, means for making an estimation regarding a movement of the user when performing the first type of exercise based on the first data; means for providing feedback to the user when the result of the estimation regarding the movement does not conform to at least one of a form or a pace defined for the first athletic event; The program according to claim 1 ,

9. The plurality of exercise events include an exercise event that can be performed without using a device capable of adjusting the exercise load, The program according to claim 2.

10. The first exercise event has a different standard exercise intensity from the second exercise event. The program according to claim 1.

11. The plurality of exercise intensities are expressed using at least one of oxygen consumption or energy consumption. The program according to claim 1.

12. The computer, a means for acquiring third data based on a sensing result when the user is performing a third exercise selected from the recommended exercises; a means for acquiring a third exercise intensity when the user is performing the third exercise event based on the third data; means for determining a second personal index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, including the third exercise intensity; A means for calculating the parameters based on the second individual index and a second reference value of exercise intensity for a target event set, which may include a resting state; To function as The program according to claim 2.

13. A means for acquiring first data based on a sensing result when a user is performing a first type of exercise; A means for acquiring a first exercise intensity when the user is performing the first exercise event based on the first data; A means for acquiring second data based on sensing while the user is performing a second exercise type different from the first exercise type or while the user is at rest; A means for acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data; a means for determining a first individual index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, the plurality of exercise intensities including the first exercise intensity and the second exercise intensity; a means for calculating a parameter representing a characteristic of the user related to exercise tolerance based on a first individual index of the user and a first reference value of exercise intensity for a target event set that may include a resting state; An information processing device comprising:

14. The computer acquiring first data based on a sensing result when a user is performing a first exercise event; acquiring a first exercise intensity when the user is performing the first exercise event based on the first data; acquiring second data based on sensing while the user is performing a second exercise type different from the first exercise type or while the user is at rest; acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data; determining a first personal index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, the plurality of exercise intensities including the first exercise intensity and the second exercise intensity; calculating a parameter representing a characteristic of the user related to exercise tolerance based on a first individual index of the user and a first reference value of exercise intensity for a target event set that may include a resting state; How to do it.

15. A system including a plurality of information processing devices, A means for acquiring first data based on a sensing result when a user is performing a first type of exercise; A means for acquiring a first exercise intensity when the user is performing the first exercise event based on the first data; A means for acquiring second data based on sensing while the user is performing a second exercise type different from the first exercise type or while the user is at rest; A means for acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data; a means for determining a first individual index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, the plurality of exercise intensities including the first exercise intensity and the second exercise intensity; a means for calculating a parameter representing a characteristic of the user related to exercise tolerance based on a first individual index of the user and a first reference value of exercise intensity for a target event set that may include a resting state; A system comprising:

Images