Information processing device, method, program, and system

Abstract

A program according to one aspect of the present disclosure causes a computer to function as: a means for acquiring, on the basis of first data based on a sensing result obtained when a user is performing a first exercise item, a first exercise intensity when the user is performing the first exercise item; a means for acquiring, on the basis of second data based on sensing when the user is performing a second exercise item different from the first exercise item or the user is at rest, a second exercise intensity at the time when the user is performing the second exercise item or the user is at rest; a means for determining a personal index of the user by performing a prescribed calculation on a plurality of exercise intensities which are acquired regarding the user and include the first exercise intensity and the second exercise intensity; and a means for calculating, on the basis of the personal index of the user and a reference value, parameters representing characteristics of the user regarding exercise tolerance.

Description

Information processing device, method, program, and system

[0001] The present disclosure relates to an information processing device, a method, a program, and a system.

[0002] The importance of exercise therapy, which utilizes exercise for the treatment and prevention of disorders and diseases, is increasing. For example, cardiac rehabilitation aims to help heart disease patients regain their physical strength and confidence, return to a comfortable home and social life, and prevent recurrence of heart disease or re-hospitalization through a comprehensive activity program including exercise therapy. The focus of exercise therapy is aerobic exercise, such as walking, jogging, cycling, and aerobics. To perform aerobic exercise more safely and effectively, it is preferable for the subject to exercise at an intensity near their anaerobic threshold (AT).

[0003] The anaerobic threshold is an example of an evaluation index of exercise tolerance and corresponds to a change point in cardiopulmonary function, i.e., an exercise intensity near the boundary between aerobic exercise and anaerobic exercise. The anaerobic threshold is generally determined by a cardiopulmonary exercise test (CPX), in which a test subject is subjected to a gradually increasing exercise load while exhaled gas is collected and analyzed (see Non-Patent Document 1). In a CPX test, the anaerobic threshold is determined based on the results measured by exhaled gas analysis (e.g., oxygen intake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof). In addition to the anaerobic threshold, a CPX test can also determine the maximum oxygen intake corresponding to an exercise intensity near maximum exercise tolerance.

[0004] To ensure the effectiveness and safety of exercise therapy, it is important to properly manage the exercise intensity of the subject.

[0005] Patent document 1 describes a method of determining whether or not a subject has reached the ventilatory threshold (VT) based on the subject's pulse information, and adjusting the exercise load of an exercise providing device according to the determination result.

[0006] Japanese Patent Publication No. 2022-059494

[0007] Saito, Muneyasu, Cardiac Rehabilitation, Physical Therapy, 1997, Vol. 24, No. 8, pp. 414-418

[0008] The technical concept described in Patent Document 1, in general, performs feedback control so that exercise intensity approaches the ventilatory threshold (VT). However, the exercise intensity for the exerciser is influenced not only by the type of exercise but also by the individual's physical function and daily physical condition. Therefore, even if this technical concept is applied to exercise therapy, it is not possible to obtain information on the level of exercise intensity that a specified type of exercise will require for the subject, or on what type of exercise should be recommended for the subject to achieve the specified exercise intensity. Furthermore, the technical concept described in Patent Document 1 cannot be simply applied, for example, to cases where a subject is required to exercise at an exercise intensity lower than the ventilatory threshold (VT).

[0009] An object of the present disclosure is to provide information for planning or guiding an exercise regimen, or a signal indicating a subject's physical function.

[0010] A program according to one aspect of the present disclosure causes a computer to function as: means for acquiring first data based on sensing results obtained when a user is performing a first type of exercise; means for acquiring a first exercise intensity obtained when the user is performing the first type of exercise based on the first data; means for acquiring second data based on sensing obtained when the user is performing a second type of exercise different from the first type of exercise or while at rest; means for acquiring a second exercise intensity obtained when the user is performing the second type of exercise or while at rest based on the second data; means for determining a first individual index of the user by performing a predetermined calculation on multiple exercise intensities obtained for the user, including the first exercise intensity and the second exercise intensity; and means for calculating a parameter representing the user's characteristics related to exercise tolerance based on the user's first individual index and a first reference value.

[0011] 1 is a block diagram showing the configuration of an information processing system of this embodiment. FIG. 2 is a block diagram showing the configuration of a client device of this embodiment. FIG. 3 is a block diagram showing the configuration of a server of this embodiment. FIG. 4 is a block diagram showing the configuration of a wearable device of this embodiment. FIG. 5 is an explanatory diagram of one aspect of this embodiment. FIG. 6 is a diagram showing the data structure of an exercise type database of this embodiment. FIG. 7 is a diagram showing the data structure of a user profile database of this embodiment. FIG. 8 is a diagram showing the data structure of a parameter log database of this embodiment. FIG. 9 is a flowchart of exercise type recommendation processing of this embodiment. FIG. 10 is a diagram showing an example of a screen displayed in the exercise type recommendation processing of this embodiment. FIG. 11 is a flowchart of parameter monitoring processing of this embodiment. FIG. 12 is a diagram showing the data structure of a teacher dataset that can be used in this embodiment.

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0013] (1) Configuration of Information Processing System The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to this embodiment.

[0014] As shown in FIG. 1 , the information processing system 1 includes a client device 10 , a server 30 , and a wearable device 50 .

[0015] Here, the number of client devices 10 and wearable devices 50 varies depending on, for example, the number of users. Therefore, the number of client devices 10 and wearable devices 50 may each be two or more. Furthermore, a terminal of a person who plans or instructs the exercise therapy may also be included in the information processing system 1. The person who plans or instructs the exercise therapy may include, for example, a medical professional (e.g., a doctor, nurse, pharmacist, physical therapist, occupational therapist, or clinical laboratory technician), a nutritionist, or a trainer.

[0016] The client device 10 and the server 30 are connected via a network (e.g., the Internet or an intranet) NW. The client device 10 and the wearable device 50 are connected via a wireless channel using, for example, Bluetooth (registered trademark) technology.

[0017] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0018] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a server computer.

[0019] The wearable device 50 is an example of an information processing device that can be worn on the user's body (for example, on the arm, hand, or head).

[0020] (1-1) Configuration of the Client Device The configuration of the client device will be described below with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0021] 2 , the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0022] The storage device 11 is configured to store programs and data, and is, for example, a combination of a read-only memory (ROM), a random access memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0023] The programs include, for example, the following: - An operating system (OS) program - An application program that executes information processing (for example, a web browser, a therapeutic app, a rehabilitation app, or a fitness app) Here, diseases that are the target of therapeutic apps or rehabilitation apps are diseases in which exercise may contribute to improving symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

[0024] The data includes, for example, the following data: Databases referenced in information processing Data obtained by executing information processing (i.e., execution results of information processing)

[0025] The processor 12 is a computer that executes the programs stored in the storage device 11 to realize the functions of the client device 10. The processor 12 is, for example, at least one of the following: - CPU (Central Processing Unit) - GPU (Graphic Processing Unit) - ASIC (Application Specific Integrated Circuit) - FPGA (Field Programmable Gate Array)

[0026] The input / output interface 13 is configured to acquire information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10.

[0027] The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0028] The communication interface 14 is configured to control communication between the client device 10 and external devices (e.g., another client device 10, the server 30, and the wearable device 50). Specifically, the communication interface 14 may include a module for communication with the server 30 (e.g., a WiFi module, a mobile communication module, or a combination thereof). The communication interface 14 may include a module for communication with the wearable device 50 (e.g., a Bluetooth module).

[0029] The display 15 is configured to display an image (still image or moving image) and is, for example, a liquid crystal display or an organic EL display.

[0030] The camera 16 is configured to take pictures and generate image signals.

[0031] The depth sensor 17 is, for example, a light detection and ranging (LIDAR) sensor. The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to a surrounding object (for example, a user).

[0032] The microphone 18 is configured to receive sound waves and generate sound signals, and is preferably placed near the user's body (particularly the respiratory tract) as an earphone microphone.

[0033] The acceleration sensor 19 is configured to detect acceleration.

[0034] (1-2) Server Configuration The server configuration will be described below with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0035] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0036] The storage device 31 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0037] The programs include, for example, the following programs: OS programs; application programs for executing information processing.

[0038] The data includes, for example, the following data: Databases referenced in information processing Execution results of information processing

[0039] The processor 32 is a computer that executes the programs stored in the storage device 31 to realize the functions of the server 30. The processor 32 is, for example, at least one of the following: CPU, GPU, ASIC, FPGA

[0040] The input / output interface 33 is configured to acquire information (e.g., user instructions) from an input device connected to the server 30, and to output information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0041] The communication interface 34 is configured to control communications between the server 30 and an external device (eg, the client device 10).

[0042] (1-3) Configuration of the Wearable Device The configuration of the wearable device will be described below. Fig. 4 is a block diagram showing the configuration of the wearable device of this embodiment.

[0043] 4 , the wearable device 50 includes a storage device 51, a processor 52, an input / output interface 53, and a communication interface 54. The wearable device 50 is connected to a display 55, a heart rate sensor 56, and an acceleration sensor 57.

[0044] The storage device 51 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0045] The programs include, for example, the following: - OS programs - Programs for applications that execute information processing (for example, medical applications, rehabilitation applications, or fitness applications)

[0046] The data includes, for example, the following data: Databases referenced in information processing Execution results of information processing

[0047] The processor 52 is a computer that executes the programs stored in the storage device 51 to realize the functions of the wearable device 50. The processor 52 is, for example, at least one of the following: a CPU, a GPU, an ASIC, or an FPGA.

[0048] The input / output interface 53 is configured to acquire information (e.g., user instructions, sensing results) from an input device connected to the wearable device 50, and to output information (e.g., images, commands) to an output device connected to the wearable device 50.

[0049] The input device is, for example, a heart rate sensor 56, an acceleration sensor 57, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 55, a speaker, or a combination thereof.

[0050] The communication interface 54 is configured to control communication between the wearable device 50 and an external device (e.g., the client device 10). Specifically, the communication interface 54 may include a module (e.g., a Bluetooth module) for communication with the client device 10.

[0051] The display 55 is configured to display an image (still image or moving image) and is, for example, a liquid crystal display or an organic EL display.

[0052] The heart rate sensor 56 is configured to measure the heart rate and generate a sensing signal. As an example, the heart rate sensor 56 measures the heart rate using an optical measurement technique.

[0053] The acceleration sensor 57 is configured to detect acceleration.

[0054] (2) One aspect of the embodiment One aspect of the present embodiment will be described below. Fig. 5 is an explanatory diagram of one aspect of the present embodiment.

[0055] As shown in FIG. 5 , the client device 10 and the wearable device 50 sense the user US1 while he or she exercises. The user US sequentially performs multiple exercises. The combination of multiple exercises (hereinafter referred to as the “target exercise set”) used to calculate these individual difference parameters (described below) may be fixed or arbitrarily selected. In the latter case, the exercises constituting the target exercise set may be determined by the user, the person planning or instructing the user's exercise therapy, or an algorithm. Furthermore, the multiple exercises constituting the target exercise set may be selected from exercises with different standard exercise intensities. In either case, information on the standard exercise intensity corresponding to each of the exercises constituting the target exercise set is stored in an exercise database (described below). The user US1 is typically a person undergoing exercise therapy, such as a participant in a (cardiac) rehabilitation program or an exercise instruction program. While the example in FIG. 5 shows gymnastics as the exercise type, the exercise type may include any type of exercise (aerobic or anaerobic).

[0056] As an example, the camera 16 captures the appearance (e.g., the entire body) of the user US1 while exercising from a front or oblique front view at a distance of, for example, about 2 m. The camera 16 may be installed at an appropriate height using a tripod or other height adjustment means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. Note that it is also possible to generate three-dimensional video data by combining, for example, two-dimensional video data generated by the camera 16 with depth data generated by, for example, the depth sensor 17. The microphone 18 receives sounds (e.g., sounds generated by breathing or speaking) emitted by the user US1 while exercising and generates a sound signal.

[0057] The heart rate sensor 56 of the wearable device 50 measures the heart rate of the user US1 while exercising and transmits the measurement results to the client device 10. The acceleration sensor 57 measures the acceleration of the user US1 while exercising and transmits the measurement results to the client device 10.

[0058] The client device 10 acquires various types of sensing data and analyzes them as necessary. As an example, the client device 10 may analyze the physical state of the user US1 while exercising by referring to video data acquired from the camera 16. The client device 10 may further refer to depth data acquired from the depth sensor 17 to analyze the physical state of the user US1 while exercising. The client device 10 transmits user data including at least one of the sensing data or the analysis results of the sensing data to the server 30.

[0059] Based on the user data acquired from the client device 10, the server 30 estimates the exercise intensity of the user US1 when performing each of the multiple exercise types constituting the target exercise set. The server 30 then performs a predetermined calculation on the multiple exercise intensities corresponding to each of the multiple exercise types to determine the user US1's personal index (described below) for the target exercise set. Furthermore, the server 30 calculates individual difference parameters that represent the user US1's characteristics related to exercise tolerance (e.g., whether the user US1 tends to perform exercise at higher or lower intensity compared to a typical person) based on the determined personal index and a reference value (described below) of the personal index for the target exercise set. For example, because oxygen consumption during exercise depends on muscle mass, individual differences in exercise intensity will occur even for the same exercise type depending on the user US1's muscle development and the areas of muscle load.

[0060] The server 30 stores the calculated individual difference parameters in the storage device 31. Using the individual difference parameters, the server 30 can predict the exercise intensity of the user US1 when performing an exercise not included in the target exercise set, or estimate what exercise the user US1 should perform to achieve a desired exercise intensity. Furthermore, by continuously monitoring the individual difference parameters, it is possible to determine the trend of change in the user US1's physical function (e.g., improvement, maintenance, or decline) and the rate of change (e.g., whether the change is rapid or not). For example, if the user US1's physical function is rapidly declining, an alert can be output indicating a suspected abnormality, such as an exacerbation of a heart disease such as heart failure, a physical breakdown, or a temporary deterioration in physical condition.

[0061] In this way, the information processing system 1 calculates individual difference parameters that represent the characteristics of the user US1 related to exercise tolerance based on data based on the sensing results of the user US1 while performing each of the exercise events that make up the target event set. Therefore, with this information processing system 1, these individual difference parameters can be used as information for planning or instructing an exercise therapy to be provided to the user US1, or as signals indicating the physical function or physical condition of the user US1.

[0062] (3) Database The database of this embodiment will be described. The following database is stored in the storage device 31.

[0063] (3-1) Exercise Event Database The exercise event database of this embodiment will be described below. Fig. 6 is a diagram showing the data structure of the exercise event database of this embodiment.

[0064] The exercise type database stores exercise type information. The exercise type information is information about exercise types that are candidates for recommended exercise types or target exercise types. In this embodiment, the exercise types include exercise types that can be performed without using equipment with adjustable exercise load, such as gymnastics, bodyweight training, dancing, walking, running, and treadmills. These exercise types are rich in variety because they include types performed in a standing position. Furthermore, in this embodiment, the exercise intensity of these exercise types can be adjusted through form (e.g., the positional relationship or angle of parts (which may include parts that are not moved during exercise), the range of motion of parts that are moved during exercise, etc.), pace, number of repetitions, or the duration or number of rest periods. However, the exercise types in this embodiment may also include exercise types performed using equipment with adjustable exercise load, such as an ergometer or strength training using training equipment.

[0065] 6, the exercise event database includes an "ID" field, a "name" field, an "exercise intensity" field, and an "exercise definition" field. Each field is associated with the others.

[0066] The "ID" field stores an exercise type ID. The exercise type ID is information for identifying the exercise type corresponding to the record.

[0067] The "Name" field stores information about the name of the exercise type. The exercise type name information is information about the name of the exercise type corresponding to the record.

[0068] The "exercise intensity" field stores exercise intensity information. The exercise intensity information is information related to the standard exercise intensity of the exercise type corresponding to the record. Standard exercise intensity may refer to the exercise intensity when a person with standard physical function performs the corresponding exercise type as defined. As an example, such exercise intensity may be derived by actually measuring the exercise intensity (e.g., average oxygen consumption) when one or more people perform the corresponding exercise type as defined, for example, by performing exhaled gas analysis (or estimating it by analyzing video footage of the user exercising), and then statistically processing the results (e.g., averaging).

[0069] For example, exercise intensity may be derived by measuring (or estimating) the exercise intensity of one or more individuals performing a corresponding exercise type as defined for each section of the exercise type, applying the measurement results for each section to a predetermined formula for each individual, and then statistically processing (e.g., averaging) the values ​​obtained by this formula across individuals. Here, a section is a component of an exercise type, and if the exercise type consists of multiple movement patterns, each movement pattern may correspond to a section. For example, if the exercise type is dance, each choreography may correspond to a section. Furthermore, if the exercise type is squats, the transition from a standing position to a crouching position and the transition from a crouching position to a standing position may each correspond to a section.

[0070] The "exercise definition" field stores exercise definition information. The exercise definition information is information related to the definition of the exercise type corresponding to the record. The exercise definition information can include information related to at least one of the following elements: - Pace of exercise (for example, the time required for one cycle of movement) - Positional relationship of body parts (which may be one or multiple) (for example, foot width, etc.) - Angle of body parts (which may be one or multiple) (for example, the direction of the knee, the angle formed by the upper arm and forearm) - Range of motion of body parts (which may be one or multiple) moved during exercise (for example, the range of movement of each body part during one cycle of movement) - Number of reps - Rest time - Exercise load (for example, the magnitude of the external load set when exercising using equipment with adjustable exercise load, such as an ergometer)

[0071] Even for an exercise that is generally recognized as a single event (e.g., squats), multiple events with slightly different exercise intensities can be defined by specifying and defining details of pace, form (especially range of motion), number of repetitions, rest time, and exercise load. For example, for an exercise that is generally recognized as a single event, multiple events with differences in exercise intensity of 0.2 METs can be defined.

[0072] (3-2) User Profile Database The user profile database of this embodiment will now be described with reference to Fig. 7, which shows the data structure of the user profile database of this embodiment.

[0073] The user profile database stores user profile information, which is information relating to the profile of a user of the information processing system 1 (i.e., a person who exercises).

[0074] 7, the user profile database includes an "ID" field, a "Name" field, a "Target Strength" field, and a "Body" field. Each field is associated with the others.

[0075] The "ID" field stores a user ID, which is information for identifying the user corresponding to the corresponding record.

[0076] The "Name" field stores user name information, which is information about the name of the user (for example, name, account name, etc.) corresponding to the corresponding record.

[0077] The "target intensity" field stores target intensity information (an example of a "predetermined exercise intensity"). The target intensity information is information regarding the target value of exercise intensity (e.g., oxygen consumption, energy consumption, heart rate, or a combination thereof) set for the user corresponding to the record. As an example, the target intensity information is specified by a person planning or instructing exercise therapy (e.g., a doctor) based on the results of measuring the user's exercise tolerance, for example, by CPX while using an ergometer (e.g., oxygen consumption and heart rate at the anaerobic threshold (AT)). However, CPX is not required, and the target value may be specified at the doctor's discretion. As another example, the target intensity information is determined by an algorithm based on the results of measuring the user's exercise tolerance, for example, by CPX. In order to perform aerobic exercise more safely and effectively, it is preferable to exercise at an intensity near the anaerobic threshold. Therefore, the target value of exercise intensity is, for example, but not limited to, an exercise intensity corresponding to the anaerobic threshold. While an ergometer is typically used as the exercise type for CPX measurements, the oxygen consumption at the anaerobic threshold when using a treadmill, for example, is approximately 1.2 to 1.3 times that when using an ergometer. This is thought to be because the total muscle mass used on the treadmill exceeds the total muscle mass used on the ergometer. Therefore, as a first example, the target intensity information may be a value obtained by correcting the target value based on the results of CPX measurements when using an ergometer to approximately 1.2 to 1.3 times the original value, or a value obtained by further subtracting a predetermined value (e.g., 1 MET). As a second example, the target intensity information may be a value obtained by correcting the target value based on the results of CPX measurements when using an ergometer without correcting it. This prevents the target value from being excessively high when selecting an exercise type that uses a relatively small amount of muscle mass, such as an ergometer. As a third example, the target value for each exercise type may be corrected, for example, by a coefficient corresponding to the total muscle mass used.

[0078] A doctor may also set an upper limit on exercise intensity for a user (an example of exercise prescription). In this case, the user is not permitted to select an exercise type that exceeds the prescribed upper limit of exercise intensity. For exercise prescription, a UI (User Interface) screen for exercise prescription may be displayed on the display of the doctor's device. Such a UI screen may include, for example, the following information: The user's CPX data; and a display area for sample videos of multiple selectable exercise types. The display area for each sample video of each exercise type may be arranged according to the exercise intensity information corresponding to the exercise type. For example, if 3.6 METs is recommended as the upper limit based on the CPX data, a group of display areas for sample videos of exercise types corresponding to 3.4 METs, a group of display areas for sample videos of exercise types corresponding to 3.6 METs, and a group of display areas for sample videos of exercise types corresponding to 3.8 METs may be arranged on the UI screen. Information on exercise types (e.g., sample videos) may be arranged in units smaller or larger than 0.2 METs. When the doctor selects one of the display areas, the exercise intensity associated with the corresponding exercise type is set as the upper limit.

[0079] In addition, the upper limit specified by the doctor through exercise prescription may be raised or lowered by a medical professional under the supervision of the doctor during regular (e.g., every two weeks) medical guidance or doctor rounds.

[0080] The "Physical" field stores physical information. The physical information is information about the body (functions) of the user corresponding to the record. For example, the physical information may include information about the user's age, gender, weight, height, illnesses, etc.

[0081] In addition, the user profile database may store the following information: Information indicating the person who planned or instructed the user's exercise therapy Information indicating the user's doctor

[0082] (3-3) Parameter Log Database The parameter log database of this embodiment will be described below with reference to Fig. 8, which shows the data structure of the parameter log database of this embodiment.

[0083] The parameter log database may be constructed, for example, for each user (i.e., person who exercises) of the information processing system 1. Alternatively, the parameter log database may be configured to store records including information that can identify a user (e.g., a user ID).

[0084] The parameter log database stores parameter log information, which is information relating to a log of individual difference parameters calculated for a user.

[0085] 8, the parameter log database includes a "date" field and a "parameter" field, each of which is associated with the other.

[0086] The "Date" field stores date information. The date information is information about the date (or date and time) when the individual difference parameters of the corresponding record were calculated.

[0087] The "parameter" field stores parameter information, which is information about the value of the individual difference parameter of the corresponding record.

[0088] (4) Information Processing Information processing in this embodiment will be described.

[0089] (4-1) Exercise Recommendation Processing The exercise recommendation processing of this embodiment will be described. Fig. 9 is a flowchart of the exercise recommendation processing of this embodiment. Fig. 10 is a diagram showing an example of a screen displayed in the exercise recommendation processing of this embodiment.

[0090] The exercise type recommendation process starts when, for example, one of the following start conditions is met: - The exercise type recommendation process is called by another process; - The user or a person who plans or instructs the user's exercise therapy performs an operation to call the exercise type recommendation process; - The client device 10 enters a predetermined state (for example, a predetermined app is launched); - A predetermined date and time has arrived; - A predetermined amount of time has passed since a predetermined event.

[0091] As shown in Fig. 9, the client device 10 acquires sensing data (S110). Specifically, the client device 10 may enable the operation of the camera 16 to start capturing a video of the user exercising (hereinafter referred to as "user video"). The client device 10 may also enable the operation of the depth sensor 17 to start measuring the distance from the depth sensor 17 to each part of the user exercising (hereinafter referred to as "user depth"). The client device 10 may enable the operation of the microphone 18 to start collecting sound (e.g., sound generated by the user's breathing or vocalization (hereinafter referred to as "user sound")).

[0092] Furthermore, the client device 10 may cause the wearable device 50 to start measuring the heart rate (hereinafter referred to as the "user's heart rate") using the heart rate sensor 56. Furthermore, the client device 10 may enable any sensor of the client device 10 or the wearable device 50 (for example, the acceleration sensor 19 or the acceleration sensor 57).

[0093] The client device 10 then acquires sensing data from each sensor while the user is performing an exercise event (the exercise event constituting the target exercise set). Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 may acquire user video data from the camera 16, user depth data from the depth sensor 17, user sound data from the microphone 18, user heart rate data from the wearable device 50, and user acceleration data related to the user's acceleration (hereinafter, "user acceleration") from the acceleration sensor 19 or at least one of the wearable device 50. The client device 10 may also acquire user oxygen intake data related to the user's oxygen intake from a device that performs a test related to exhaled gas (e.g., a CPX test).

[0094] The client device 10 can repeatedly acquire sensing data (S110) while performing one exercise event.

[0095] After step S110, the client device 10 generates user data (S111). Specifically, the client device 10 generates user data based on the sensing data acquired in step S110. The user data may include at least one of the following: Data acquired in step S111 (e.g., user video data, user depth data, user sound data, user heart rate data, user acceleration data, or user oxygen intake data); Data obtained by processing the data acquired in step S111; Data acquired by analyzing the user video data (or the user video data and user depth data) acquired in step S111 (e.g., skeletal data, facial expression data, skin color data, respiratory data, or a combination thereof, as described below); Information that can identify the exercise type that the user was performing in step S110 (the exercise types that make up the target exercise set); Information that indicates the user's subjective evaluation of the exercise intensity of the completed exercise type (hereinafter referred to as "perceived exercise intensity").

[0096] The client device 10 may generate user data (S111) repeatedly while performing one exercise event, or may generate user data each time one exercise event is completed.

[0097] After step S111, the client device 10 transmits the user data (S112). Specifically, the client device 10 transmits the user data generated in step S111 to the server 30.

[0098] The client device 10 may send user data (S112) repeatedly during the execution of one exercise event, may send the data each time an exercise event is completed, or may send the data only once after all the exercise events that make up the target exercise set have been completed.

[0099] After step S112, the server 30 estimates exercise intensity (S130). Specifically, the server 30 receives the user data transmitted by the client device 10 in step S112. Based on the user data acquired from the client device 10, the server 30 estimates the exercise intensity of the user when performing each exercise event constituting the target exercise set. The exercise intensity may be calculated, for example, as energy consumption (e.g., METs), oxygen consumption, exercise intensity based on heart rate (e.g., exercise intensity calculated using the Karvonen method), or a combination thereof. The server 30 may refer to user profile information stored in a user profile database ( FIG. 7 ) to estimate the exercise intensity.

[0100] As a first example of estimating exercise intensity (S130), the server 30 estimates the exercise intensity when the user performs one of the exercise types that make up the target exercise set by performing a Karvonen calculation based on the user's heart rate measured over multiple points in time while the user is performing the exercise type and the user's age.

[0101] As a second example of exercise intensity estimation (S130), the server 30 estimates the exercise intensity when the user performs each exercise event constituting the target event set, using an estimation model described below. Specifically, the server 30 estimates the exercise intensity by applying the estimation model to input data based on user data (e.g., skeletal data, facial expression data, skin color data, breathing data, heart rate data, or a combination thereof).

[0102] As a third example of estimating exercise intensity (S130), the server 30 may calculate the exercise intensity when the user performs each exercise type that constitutes the target exercise type set based on the user's oxygen intake data when performing that exercise type.

[0103] As a fourth example of estimating exercise intensity (S130), the server 30 may use user audio data to have an expert, such as a physical therapist, conduct a talk test while the user is performing each exercise that constitutes the target exercise set, and calculate the exercise intensity when the user performs that exercise based on the results of the talk test.

[0104] As a fifth example of estimating exercise intensity (S130), the server 30 may acquire information on the perceived intensity of exercise (e.g., the Borg index) when performing each exercise event that constitutes the target exercise set, and calculate the exercise intensity when the user performs that exercise event based on the perceived intensity of exercise.

[0105] The sixth example of exercise intensity estimation (S130) is a combination of multiple examples from the first to fifth examples.

[0106] The server 30 may estimate the exercise intensity for each section constituting an exercise type and treat a representative value of the estimated results (e.g., the average, median, mode, maximum, minimum, first quartile, or third quartile) as the exercise intensity for that exercise type. The client device 10 or the server 30 may identify which section each sensing result is associated with by, for example, analyzing user video data (and user depth data, if necessary). Alternatively, in cases where a fixed time is assigned to each section, such as in gymnastics or dance, the client device 10 or the server 30 may identify which section each sensing result is associated with based on the time assignment. The server 30 may also appropriately combine two or more of the first to third examples described above.

[0107] Since the estimated exercise intensity is not stable immediately after starting exercise (for example, within 1 to 2 minutes), the server 30 may omit estimating the exercise intensity for a predetermined period after starting exercise, or may discard the estimated exercise intensity. In other words, the server 30 may estimate the exercise intensity only when it determines that the exercise intensity is in a plateau state.

[0108] After step S130, the server 30 determines an individual index (S131). Specifically, the server 30 determines the user's individual index of exercise intensity for the target event set by performing a predetermined calculation using the exercise intensity estimated in step S130. As an example, the server 30 treats the exercise intensities estimated for the multiple event sets constituting the target event set, or the product or ratio of the exercise intensities, as elements (the elements may include values ​​(bias) not based on exercise intensity). The server 30 determines the individual index by performing a predetermined calculation, such as multiplying the element by a predetermined coefficient or adding a bias, and then calculating the sum, product, or ratio of these. For example, the individual index may be a value obtained by subtracting the exercise intensity of event B from the exercise intensity of event A. The calculation formula for the individual index may be defined for each target event set, or may be defined commonly across multiple target event sets.

[0109] After step S131, the server 30 calculates individual difference parameters (S132). Specifically, the server 30 calculates individual difference parameters that represent the user's characteristics related to exercise tolerance based on the individual indices determined in step S131 and the reference values ​​of exercise intensity for the target event set. The server 30 creates a record based on the calculated individual difference parameters and the calculation date (or calculation date and time), and adds the record to the user's parameter log database ( FIG. 8 ).

[0110] The reference value of exercise intensity for the target sports set can be determined based on information on the exercise intensity when one or more reference individuals (e.g., individuals with standard exercise tolerance) perform each of the sports that make up the target sports set, and on the individual indices calculated for each of the individuals based on the exercise intensity. Specifically, the exercise intensity when each individual performs each sports set is acquired (actually measured or estimated), an individual indices are calculated using the same formula as in step S131, and the reference value can be determined by performing statistical processing on the individual indices (e.g., calculating a representative value such as averaging). The reference value may be calculated in advance for each available target sports set and stored in a database. The individual referenced to calculate the reference value to be applied to a user may be determined to have at least one attribute (e.g., age group, gender, or a combination thereof) that is the same as that of the user.

[0111] As a first example, the server 30 calculates the individual difference parameter by dividing the individual index by the reference value. In this case, the individual difference parameter corresponds to a prediction result of the ratio of the exercise intensity to be higher or lower than that of a standard person when the user performs the exercise type constituting the target exercise set or another exercise type.

[0112] As a second example, the server 30 calculates the individual difference parameter by subtracting the reference value from the individual index. In this case, the individual difference parameter corresponds to a prediction result of how much the exercise intensity will be higher or lower than that of a standard person when the user performs the exercise events that make up the target exercise set or other exercise events.

[0113] After step S132, the server 30 selects a recommended exercise type (S133). Specifically, the server 30 references the user profile database (FIG. 7) and acquires the user's target intensity information (which is determined based on the results of measuring the user's exercise tolerance, as described above). The server 30 selects an exercise type (hereinafter referred to as a "recommended exercise type") that is suitable for the characteristics of the user's exercise tolerance based on the acquired target intensity information, the individual difference parameters calculated in step S132, and the exercise intensity information for each exercise type stored in the exercise type database (FIG. 6). The server 30 may recommend one or more exercise types.

[0114] As a first example, the server 30 corrects the target value of the user's exercise intensity using an individual difference parameter. For example, the server 30 obtains a corrected target value by dividing the target value by the individual difference parameter or by subtracting the individual difference parameter from the target value. The server 30 then selects recommended exercise types from exercise types whose exercise intensity indicated by the corresponding exercise intensity information (hereinafter referred to as "standard intensity") does not exceed the corrected target value. For example, the server 30 may include, in the recommended exercise types, exercise types whose standard intensity is the maximum within a range equal to or less than the corrected target value. Note that, when selecting recommended exercise types, the server 30 may use, instead of the corrected target value, a value obtained by adding or subtracting a margin to or from the corrected target value, or a value obtained by multiplying the corrected target value by a positive coefficient other than 1. Alternatively, when selecting recommended exercise types, the server 30 may use, instead of the standard intensity, a value obtained by adding or subtracting a margin to or from the standard intensity, or a value obtained by multiplying the standard intensity by a positive coefficient other than 1.

[0115] As a second example, the server 30 corrects the standard intensity for each exercise type using an individual difference parameter. For example, the server 30 obtains a corrected intensity by multiplying or adding the standard intensity by the individual difference parameter. The server 30 then selects recommended exercise types from exercise types whose corrected intensity does not exceed the user's target exercise intensity. For example, the server 30 may include, in the recommended exercise types, exercise types whose corrected intensity is the maximum within a range equal to or less than the target value. Note that, when selecting recommended exercise types, the server 30 may use, instead of the target value, a value obtained by adding or subtracting a margin to or from the target value, or a value obtained by multiplying the target value by a positive coefficient other than 1. Alternatively, when selecting recommended exercise types, the server 30 may use, instead of the corrected intensity, a value obtained by adding or subtracting a margin to or from the corrected intensity, or a value obtained by multiplying the corrected intensity by a positive coefficient other than 1.

[0116] After step S133, the server 30 transmits recommended exercise information (S134). Specifically, the server 30 transmits information about the recommended exercise selected in step S133 (hereinafter referred to as "recommended exercise information") to the client device 10. The recommended exercise information may be, for example, information that identifies the recommended exercise, or screen information for displaying the recommended exercise.

[0117] After step S134, the client device 10 executes screen display (S113). Specifically, the client device 10 receives the recommended exercise information transmitted by the server in step S134. The client device 10 displays a screen based on the recommended exercise information on the display 15.

[0118] For example, the client device 10 displays the screen of Fig. 10 on the display 15. The screen of Fig. 10 includes objects J20 to J23.

[0119] The object J20 displays recommended exercises. The object J20 also accepts user instructions to start the recommended exercises or to play a model video of the recommended exercises. When the object J20 is selected, the client device 10 may re-execute the exercise recommendation process of this embodiment, with the exercise corresponding to the object J20 as an exercise that constitutes a new target exercise set. The server 30 may then calculate the user's individual index for the new target exercise set and calculate the user's individual difference parameters based on the individual index and a reference value corresponding to the new target exercise set. Alternatively, when the object J20 is selected, the client device 10 may play a model video of the exercise corresponding to the object J20.

[0120] The object J21 accepts a user instruction to select an exercise type other than the recommended exercise types. When the object J21 is selected, the client device 10 may, for example, display a list of exercise types other than the recommended exercise types and accept a user instruction to select an exercise type. In response to such a user instruction, the client device 10 may re-execute the exercise type recommendation process of this embodiment, with the selected exercise type as an exercise type constituting a new target exercise type set. Here, the server 30 may calculate the user's individual index for the new target exercise type set and calculate the user's individual difference parameters based on the individual index and a reference value corresponding to the new target exercise type set. Alternatively, when an exercise type is selected, the client device 10 may play a model video of the selected exercise type.

[0121] The object J22 receives a user instruction to end the exercise. When the object J22 is selected, the client device 10 ends the exercise type recommendation process of this embodiment.

[0122] After step S113, the client device 10 may end the exercise event recommendation process (FIG. 9).

[0123] (4-2) Parameter Monitoring Process The parameter monitoring process of this embodiment will be described below with reference to the flowchart of FIG.

[0124] The parameter monitoring process of this embodiment may be started, for example, each time the exercise type recommendation process (FIG. 9) of this embodiment (particularly, the calculation of individual difference parameters (S133)) is executed, or may be executed repeatedly at a predetermined interval for each user.

[0125] 11, the server 30 acquires parameter log information (S230). Specifically, the server 30 refers to the parameter log database of the user to be processed and acquires the parameter log information.

[0126] After step S230, the server 30 executes a determination of a predetermined condition (S231). Specifically, the server 30 determines whether a predetermined condition is met for the parameter log information acquired in step S230. For example, the predetermined condition may include at least one of the following: A representative value of a predetermined number of recent individual difference parameters (which may be one or more) exceeds a threshold (i.e., it is predicted that the user's exercise intensity will be excessively high compared to a typical person). A representative value of the individual difference parameters over a predetermined period of time exceeds a threshold (i.e., it is predicted that the user's exercise intensity will be excessively high compared to a typical person). An increase rate of the individual difference parameter over a predetermined number of recent individual difference parameters exceeds a threshold (i.e., it is predicted that the user's exercise intensity for the same type of exercise is increasing rapidly). An increase rate of the individual difference parameter over a predetermined period of time exceeds a threshold (i.e., it is predicted that the user's exercise intensity for the same type of exercise is increasing rapidly).

[0127] If it is determined in step S231 that the predetermined condition is met, the server 30 outputs an alert (S232). Specifically, the server 30 transmits the alert to the user's client device 10 or a predetermined terminal. The predetermined terminal may be a terminal of a person designated by the user (e.g., a family member), a terminal of the user's doctor, or a terminal of a person who plans or instructs the user's exercise therapy.

[0128] The alert may include at least one of the following information, for example: A recent predetermined number of individual difference parameters An individual difference parameter over a recent predetermined period An increase rate of the recent predetermined number of individual difference parameters An increase rate of the individual difference parameter over a recent predetermined period A message (e.g., text, image, sound, or a combination thereof) indicating that the user is suspected of having an exacerbation of heart disease A message indicating that the user is suspected of having a physical malfunction A message indicating that the user is suspected of having a temporary deterioration in physical condition

[0129] The alert output destination (user's client device 10 or a predetermined terminal) presents information based on the alert received from the server 30 using an image, sound, vibration, or a combination thereof.

[0130] After step S232, the server 30 ends the parameter monitoring process of this embodiment.

[0131] If it is determined in step S231 that the predetermined condition is not met, the server 30 skips outputting the alert (S232) and ends the parameter monitoring process of this embodiment.

[0132] (5) Summary As described above, the server 30 of this embodiment acquires first data based on sensing results when the user is performing a first type of exercise, and acquires a first exercise intensity when the user is performing the first type of exercise based on the first data. The server 30 acquires second data based on sensing results when the user is performing a second type of exercise different from the first type of exercise, and acquires a second exercise intensity when the user is performing the second type of exercise based on the second data. The server 30 determines a first individual index for the user by performing a predetermined calculation on multiple exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity, and calculates a parameter representing the user's characteristics related to exercise tolerance based on the user's first individual index and a first reference value. This allows the individual difference parameter to be used as a decision-making tool for planning or instructing an exercise therapy (which may include a prescription) provided to the user, or as a signal indicating the user's physical function or physical condition. Furthermore, by referring to the exercise intensities for multiple types of exercise, a more reliable individual index can be calculated.

[0133] The server 30 may use the parameters, the user's target exercise intensity, and a database that associates each of a plurality of exercise types with a standard exercise intensity to select a recommended exercise type for the user from a plurality of exercise types and output information indicating the recommended exercise type. This allows the server 30 to recommend an exercise type not only by matching the measurement results of the user's exercise tolerance with the standard exercise intensity of each exercise type, but also by taking into consideration the characteristics of the user's exercise tolerance.

[0134] The server 30 may select, as a recommended exercise type, an exercise type associated with an exercise intensity that does not exceed a corrected exercise intensity obtained by correcting the target exercise intensity using a parameter. This makes it possible to recommend an exercise type whose exercise intensity when actually performed by the user is expected to not exceed the target exercise intensity.

[0135] The server 30 may select recommended exercise types so as to include, among the multiple exercise types, an exercise type associated with the maximum exercise intensity that does not exceed the corrected exercise intensity. This makes it possible to recommend exercise types that are estimated to result in exercise intensity that does not exceed the target exercise intensity when actually performed by the user and is close to the target exercise intensity.

[0136] The target exercise intensity may be determined based on the exercise intensity specified by an algorithm or by a person who plans or instructs the user's exercise therapy based on the results of the user's exercise tolerance measurement, thereby making it possible to recommend an exercise type that matches the exercise intensity determined to be suitable for the user from a medical perspective.

[0137] The first reference value may be a representative value of the first individual indexes of a plurality of people calculated using the same formula as that of the first individual index of the user, thereby obtaining an individual difference parameter that appropriately represents the characteristics of the exercise tolerance of the user.

[0138] The plurality of people may be determined to have at least one attribute that is the same as the user, thereby obtaining individual difference parameters that more appropriately represent the characteristics of the user's exercise tolerance.

[0139] The plurality of exercise types may include exercise types that can be performed without using a device that can adjust the exercise load, thereby expanding the range of exercise types available to the user.

[0140] The first exercise type may have a different standard exercise intensity from the second exercise type, which makes it easier to obtain a reasonable individual difference parameter when calculating an individual index using, for example, the difference or ratio between the exercise intensities.

[0141] The plurality of exercise intensities may be expressed using at least one of oxygen consumption and energy consumption, thereby obtaining an individual difference parameter that represents the characteristics of the user's oxygen consumption or energy consumption.

[0142] The server 30 may acquire third data based on the sensing results when the user is performing a third exercise type selected from the recommended exercise types, and may acquire a third exercise intensity when the user is performing the third exercise type based on the third data. The server 30 may determine a second individual index of the user by performing a predetermined calculation on the multiple exercise intensities acquired for the user, including the third exercise intensity, and calculate parameters based on the second individual index and a second reference value. This makes it possible to calculate individual difference parameters for various combinations of exercise types.

[0143] (6) Estimation Model As described above, the server 30 may estimate exercise intensity using an estimation model. In this case, the estimation model corresponds to a trained model created by supervised learning using a training data set, which will be described below, or a derived model or distilled model of the trained model. The estimation model may be constructed for each type of exercise, or may be constructed commonly across multiple types of exercise.

[0144] (6-1) Teacher Data Set A teacher data set that can be used in supervised learning to build an estimation model will be described. Fig. 12 is a diagram showing the data structure of the teacher data set that can be used in this embodiment.

[0145] As shown in Fig. 12, the training data set includes multiple training data. The training data is used to train or evaluate a model to be learned (hereinafter referred to as a "target model"). The training data includes a sample ID, input data, and correct answer data.

[0146] The sample ID is information that identifies the training data.

[0147] The input data is data input to the target model during training or evaluation. The input data corresponds to examples used during training or evaluation of the target model. As an example, the input data is data regarding the subject's physical state during exercise (i.e., relatively dynamic data) and data regarding the subject's health state (i.e., relatively static data). At least a portion of the data regarding the subject's physical state is obtained by analyzing the subject's physical state with reference to the subject video data (or the subject video data and subject depth data).

[0148] The subject video data is data related to a video of a subject during exercise. The subject video data can be obtained, for example, by capturing an external image (e.g., the entire body) of a subject undergoing an exhaled gas test (e.g., a CPX test) from the front or diagonally in front (e.g., 45 degrees forward) using a camera (e.g., a camera mounted on a smartphone).

[0149] The subject depth data is data relating to the distance (depth) from the depth sensor to each part of the subject during exercise. The subject depth data can be acquired by operating the depth sensor when shooting the subject video.

[0150] The subject may be the same person as the user whose exercise intensity is estimated based on exercise tolerance during operation of the information processing system 1, or may be a different person. By making the subject and the user the same person, the target model may learn the user's personality and improve estimation accuracy. On the other hand, allowing the subject to be a different person from the user has the advantage of making it easier to enrich the training data set. Furthermore, the subjects may be composed of multiple people, including the user, or multiple people excluding the user.

[0151] In the example of FIG. 12, the input data includes skeletal data, facial expression data, skin color data, respiration data, heart rate data, and health condition data.

[0152] Skeletal data is data (e.g., features) related to the subject's skeleton during exercise. Skeletal data includes, for example, data related to the speed or acceleration of each part of the subject (which may include data related to changes in the parts of the muscles used by the subject or data related to the subject's perceived shaking). Skeletal data can be obtained by analyzing the subject's skeleton during exercise with reference to subject video data (or subject video data and subject depth data). For example, Vision, an SDK for iOS 14, or other skeletal detection algorithms (e.g., OpenPose, PoseNet, MediaPipe Pose) can be used to analyze the skeleton. Alternatively, skeletal data for a training dataset can be obtained by, for example, having the subject exercise while wearing motion sensors on each part of the body.

[0153] The results of skeletal detection can be used for quantitative evaluation of exercise, qualitative evaluation, or a combination of these. As a first example, the results of skeletal detection can be used to count the number of reps. As a second example, the results of skeletal detection can be used to evaluate the form of an exercise or the appropriateness of the load applied by the exercise. For example, if the exercise is a squat, the results of skeletal detection can be used to evaluate whether the knees are protruding too far forward, resulting in a dangerous form, or whether the hips are lowered firmly and deeply to apply sufficient load.

[0154] Facial expression data is data (e.g., features) about a subject's facial expressions during exercise. Facial expression data can be analyzed by applying an algorithm or trained model to video data of the subject. Alternatively, facial expression data for a training dataset can be obtained by, for example, human labeling of the subject's videos.

[0155] Skin color data is data (e.g., features) about the skin color of a subject during exercise. Skin color data can be analyzed by applying an algorithm or trained model to video data of the subject. Alternatively, skin color data for a training dataset can be obtained, for example, by human labeling after watching videos of the subject.

[0156] The respiratory data is data (e.g., feature quantities) related to the subject's breathing during exercise. The respiratory data relates, for example, to the respiratory rate per unit time or the respiratory pattern. The respiratory pattern may include at least one of the following: ventilation rate; ventilation volume; ventilation rate (i.e., ventilation volume per unit time or ventilation rate); ventilation acceleration (i.e., time derivative of ventilation rate); carbon dioxide excretion concentration; carbon dioxide excretion (VCO2); oxygen uptake concentration; and oxygen uptake (VO2). The data related to the respiratory pattern may include data that can be calculated based on a combination of the above data, such as the gas exchange ratio R (=VCO2 / VO2).

[0157] The respiratory data can be obtained by, for example, analyzing the skeletal data. As an example, the following items can be analyzed from the skeletal data: Movement (expansion) of the shoulders, chest (which may include the lateral chest), abdomen, or a combination thereof; Inhalation time; Exhalation time; and The degree of use of accessory respiratory muscles.

[0158] The respiratory data for the training dataset can be obtained, for example, from the results of a test on exhaled gases performed on a subject while exercising. Details of exhaled gas tests that can be performed on a subject while exercising will be described later. Alternatively, the respiratory data for the training dataset, such as the ventilation rate, ventilation volume, ventilation rate, or ventilation acceleration, can be obtained, for example, from the results of a respiratory function test (e.g., a pulmonary function test or a spirometry test) performed on the subject while exercising. In this case, the respiratory function test can be performed using commercially available testing equipment, not limited to medical equipment.

[0159] Heart rate data is data (e.g., feature values) related to the heart rate of a subject during exercise. Heart rate data can be obtained, for example, by analyzing subject video data or the analysis results thereof (e.g., skin color data). Alternatively, heart rate data for the training dataset may be obtained, for example, from the results of a test on exhaled gas along with respiratory data, which will be described later. Subject heart rate data for the training dataset can also be obtained by having the subject perform the above exercise while wearing a heart rate sensor or electrodes for an electrocardiogram monitor.

[0160] Health status data is data related to the subject's health status. Health status data can be obtained in various ways. The subject's health status data may be obtained before, during, or after the subject's exercise. The subject's health status data may be obtained based on a report from the subject or their doctor, by extracting information linked to the subject in a medical information system, or via the subject's app (e.g., a healthcare app).

[0161] The health condition includes at least one of the following: Age Gender Height Weight Body fat percentage Muscle mass Bone density History of current illness Past medical history Medication history History of surgery Lifestyle history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty score, etc.) Family history Results of respiratory function tests Results of tests other than respiratory function tests (e.g., results of blood tests, urine tests, electrocardiograms (including Holter ECG), cardiac ultrasound, X-rays, CT scans (including cardiac morphological CT and coronary CT), MRI scans, nuclear medicine scans, PET scans, etc.) Data obtained during cardiac rehabilitation (including the Borg index)

[0162] The correct answer data is data that corresponds to the correct answer for the corresponding input data (example problem). The target model is trained (supervised learning) to output data that is as close as possible to the correct answer data for the input data. As an example, the correct answer data represents exercise intensity.

[0163] Exercise intensity is an index for quantitatively evaluating the exercise load on a subject. Exercise intensity can be expressed numerically using at least one of the following: Energy (calorie) consumption, Oxygen consumption, Heart rate

[0164] The correct answer data can be obtained, for example, from the results of a breath gas test conducted on a subject during exercise. A first example of a breath gas test is a test (typically a CPX test) conducted while a subject wearing a breath gas analyzer is performing exercise with increasing load (e.g., on an ergometer). A second example of a breath gas test is a test conducted while a subject wearing a breath gas analyzer is performing exercise of a constant or variable intensity (e.g., bodyweight exercise, gymnastics, strength training).

[0165] Alternatively, the correct data can be obtained from the results of tests other than exhaled gas tests conducted on the exercising subject. Specifically, the correct data can be obtained from the results of a cardiopulmonary exercise load prediction test based on the lactate concentration measurement in the sweat or blood of the exercising subject. A wearable lactate sensor can also be used to measure the lactate concentration of the subject.

[0166] It is also possible to construct an estimation model for each of a plurality of health condition categories based on (at least a part of) the subject's health condition. In this case, (at least a part of) the user's health condition may be referenced to select an estimation model. In this further variation, input data for the estimation model may be data that is not based on the user's health condition, or may be data that is based on the user's health condition and user video.

[0167] (7) Other Modifications The storage device 11 may be connected to the client device 10 via the network NW. Each input device or output device may be integrated with the client device 10. The storage device 31 may be connected to the server 30 via the network NW. Each input device or output device may be integrated with the wearable device 50.

[0168] In the embodiment, the information processing system 1 is implemented as a client / server system. However, the information processing system 1 may also be implemented as a peer-to-peer system or a standalone computer. For example, the client device 10 may estimate exercise intensity.

[0169] Each step of the above information processing can be executed by either the client device 10 or the server 30. As an example, the server 30, instead of the client device 10, may acquire at least a portion of the user data or the user skeletal data by analyzing the user video (or the user video and the user depth).

[0170] One or more steps of the above information processing may be performed using a trained model.

[0171] In the above example, a recommended exercise type is selected based on the standard exercise intensity for each of multiple exercise types, the user's individual difference parameters, and the results of measuring the user's exercise tolerance. However, the server 30 may also determine instructions for adjusting the exercise intensity of the exercise type selected by the user based on this information (in other words, presenting an exercise type (variation) whose exercise definition information differs only partially from the exercise type selected by the user) and output the instructions to the client device 10. The instructions may include, for example, the form of the exercise type (e.g., the positional relationship or angle of body parts (which may include body parts that are not moved during exercise), or the range of motion of body parts that are moved during exercise), pace, number of reps, or the duration or number of rest periods. This allows, for example, even if the exercise intensity of the user's selected exercise type is estimated to exceed the aforementioned target value, to prevent the user's exercise intensity from deviating from the target value by specifying an exercise type that results in a lighter exercise intensity than the standard, a slower pace than the standard, fewer reps than the standard, or longer rest periods or rest periods. Alternatively, even if it is estimated that the exercise intensity of the exercise type selected by the user will be lower than the target value, the user can prevent the exercise intensity from deviating from the target value by specifying a form that makes the exercise intensity heavier than standard, setting a faster pace than standard, increasing the number of repetitions than standard, or reducing the length or number of rest periods.

[0172] In the above description, an example was shown in which a recommended type of exercise is selected based on the standard exercise intensity of each of a plurality of exercise types, the user's individual difference parameters, and the results of measuring the user's exercise tolerance. However, the server 30 may also select a recommended type of exercise based additionally on at least one of the user's pulse rate, respiration rate, or perceived exertion during exercise. Here, the perceived exertion can be obtained, for example, by receiving a Borg index input from the user via the client device 10 during a rest period. If at least one of the pulse rate, respiration rate, or perceived exertion exceeds an upper limit, the server 30 may select a recommended type of exercise with a lower standard exercise intensity than in other cases. On the other hand, if at least one of the pulse rate, respiration rate, or perceived exertion falls below a lower limit, the server 30 may select a recommended type of exercise with a higher standard exercise intensity than in other cases. However, if a doctor has set an upper limit for the exercise intensity for the user, the server 30 will not select an exercise type corresponding to an exercise intensity exceeding that upper limit. Furthermore, the user's pulse rate, respiration rate, or perceived exertion level during exercise may be used to adjust the exercise intensity of the next set when performing an exercise event consisting of multiple sets. For example, if at least one of the pulse rate, respiration rate, or perceived exertion level exceeds an upper limit, the server 30 may determine instructions (e.g., form, pace, number of repetitions, or rest time or number) so that the exercise intensity of the next set is lower than that of the current set, and output information about the instructions to the client device 10. On the other hand, if at least one of the pulse rate, respiration rate, or perceived exertion level falls below a lower limit, the server 30 may determine instructions so that the exercise intensity of the next set is higher than that of the current set, and output information about the instructions to the client device 10. However, if a doctor has set an upper limit for the exercise intensity of the user, instructions corresponding to an exercise intensity exceeding the upper limit will not be selected. For example, the server 30 may determine the recommended exercise type or instructions so that the exercise intensity of the next set is reduced by 0.2 METs if the average heart rate is 5 or more higher than the target heart rate or the perceived exertion level is 14 or higher.The server 30 may determine a recommended type of exercise or instructions to reduce the exercise intensity of the next set by 0.4 METs when the average heart rate is 10 or more higher than the target heart rate or the perceived intensity of exertion is 16 or higher. The server 30 may also determine a recommended type of exercise or instructions to increase the exercise intensity of the next set by 0.2 METs when the average heart rate is 5 or more lower than the target heart rate and the perceived intensity of exertion is less than 10.

[0173] The client device 10 may further perform the following processes, for example, when acquiring sensing data (S110) (in other words, while the user is exercising). Specifically, the client device 10 estimates the user's movements during exercise (e.g., movements of the skeleton or other feature points over multiple time points, or the state of the skeleton or other feature points at a single point in time) based on the sensing data. Then, if the estimated movements do not match at least one of the form or pace defined for the type of exercise the user is performing (e.g., the range of motion of a body part is too narrow or too wide, the angle of a body part deviates from the standard, or the pace is too fast or too slow), the client device 10 provides feedback to the user. The feedback may include at least one of the following: Output of light, sound, or voice (e.g., voice informing the user of an incorrect performance); Display of an image (e.g., an image informing the user of an incorrect performance); Vibration of the wearable device. This allows for more accurate estimation of the user's personal index. Alternatively, if the number or frequency of occurrences of the motion estimation results not conforming to at least one of the form or pace defined for the type of exercise being performed by the user exceeds a threshold, the client device 10 may recommend that the user redo the same type of exercise or perform a different type of exercise. In this case, the client device 10 may omit the processes subsequent to the generation of user data (S111).

[0174] Instead of recommending an exercise type, the information processing system 1 may perform a process for measuring oxygen consumption at the anaerobic threshold. This process may be automatically selected, for example, during the first three days after the user begins an exercise regimen, or may be selected at the instruction of the user or the person planning or instructing the exercise regimen. In this mode, the information processing system 1 sequentially selects exercise types so that the exercise intensity increases by 0.2 METs until one of the following conditions is met: the average heart rate is 5 or more higher than the target heart rate, or the perceived intensity of exertion is 14 or higher. The information processing system 1 treats the oxygen consumption estimated for the exercise type performed immediately before the exercise type for which the above condition is met as the user's oxygen consumption at the anaerobic threshold. The information processing system 1 may also treat the exercise intensity corresponding to the exercise type that yields the average heart rate closest to the target heart rate as the optimal exercise intensity.

[0175] The above description describes an example in which an alert is output when a user's individual difference parameters satisfy a predetermined condition. However, the alert may be output based on various triggers other than individual difference parameters. For example, the server 30 stores data on the types of exercise the user has previously performed and the user's heart rate, perceived intensity of exertion, or respiration rate while performing the exercise. The server 30 may then output an alert when the user's heart rate, perceived intensity of exertion, or respiration rate during exercise increases (worsens) by more than a threshold value compared to the user's heart rate, perceived intensity of exertion, or respiration rate when performing the same type of exercise in the past. Furthermore, if the user's level of distress (hereinafter referred to as "distress level") is quantified based on facial expression data or skeletal data, for example, this can be used as a trigger similar to the heart rate, perceived intensity of exertion, or respiration rate. In addition, the server 30 may use as a trigger input regarding the onset or worsening of the user's subjective symptoms of weight gain, shortness of breath, edema, fatigue, loss of appetite, or insomnia in daily life, or sensor input suggesting the onset or worsening of these symptoms.

[0176] The above description illustrates an example in which a user's individual difference parameters are calculated during the exercise type recommendation process or parameter monitoring process. However, the user's individual difference parameters may also be calculated based on data obtained during a CPX test, for example. For example, the individual difference parameters may be calculated based on the exercise intensity (e.g., the actual oxygen intake value) when the user is rowing an ergometer at a first load (e.g., 20 W) and the exercise intensity when the user is rowing an ergometer at a second load (e.g., 40 W). When a doctor sets an upper limit on exercise intensity for a user (an example of an exercise prescription), the type of exercise to be presented to the doctor can be selected based on the CPX data and the individual difference parameters. For example, if 3.6 METs is recommended as the upper limit based on CPX, a group of display areas for sample videos of exercise types corresponding to the aforementioned corrected intensity of 3.4 METs, a group of display areas for sample videos of exercise types corresponding to a corrected intensity of 3.6 METs, and a group of display areas for sample videos of exercise types corresponding to a corrected intensity of 3.8 METs may be arranged on the UI screen. It should be noted that information on exercise events (e.g., sample videos) may be arranged in units smaller than or larger than 0.2 METs.

[0177] In the above embodiment, an example was shown in which a user's individual difference parameter was calculated based on the exercise intensity when the user was performing a plurality of different exercise types. However, instead of one of the exercise types, the user's exercise intensity when at rest may be used. By defining a standard exercise intensity when at rest and acquiring the user's exercise intensity when at rest, the individual difference parameter can be calculated in a similar manner.

[0178] In the above description, an example has been shown in which a user video is captured using the camera 16 of the client device 10. However, the user video may be captured using a camera other than the camera 16. In the above description, an example has been shown in which the user depth is measured using the depth sensor 17 of the client device 10. However, the user depth may be measured using a depth sensor other than the depth sensor 17.

[0179] In the above description, an example has been given in which the wearable device 50 measures the user's heart rate. However, the heart rate can also be obtained by analyzing video data or its analysis results (e.g., skin color data) (e.g., remote photoplethysmography (rPPG) analysis). The heart rate analysis may be performed using a trained model constructed using machine learning technology. Alternatively, the user may exercise while wearing electrodes for an electrocardiogram monitor, allowing the electrocardiogram monitor to measure the user's heart rate. In these variations, the user does not need to wear the wearable device 50 to measure the heart rate.

[0180] The wearable device 50 may be provided with a sensor for measuring at least one of the following items instead of or in addition to the heart rate sensor 56 and the acceleration sensor 57: blood glucose level; and oxygen saturation. The measurement results from each sensor may be used as appropriate for generating input data, estimating exercise intensity or ventilation index, presenting information based on the estimation results, or in other situations. As one example, the measurement results of blood glucose level may be referenced to evaluate exercise intensity converted into energy consumption or oxygen consumption. As another example, the measurement results of acceleration may be used, for example, to determine the score of the user's exercise (e.g., gymnastics).

[0181] The acceleration data may be used as part of the input data for the estimation model described in this embodiment or the modified example. Alternatively, the user's skeleton may be analyzed by referring to the acceleration data. The acceleration data may be acquired by the acceleration sensor 19 or the acceleration sensor 57 when capturing a user video.

[0182] Oxygen saturation data can also be used as part of the input data for the estimation model described in this embodiment or the modified example. The oxygen saturation data can be obtained, for example, by having the user wear a wearable device equipped with a sensor (e.g., an optical sensor) capable of measuring blood oxygen concentration or a pulse oximeter when capturing the user's video. The oxygen saturation data can be estimated, for example, by performing rPPG analysis on the user's video data.

[0183] The information processing system 1 of this embodiment and each modification can also be applied to a video game in which game progress is controlled according to the player's physical movements. The video game may be a mini-game that can be played while the aforementioned therapeutic app, rehabilitation app, or fitness app is being executed, or may be configured as a standalone app. As an example, the information processing system 1 may estimate the user's exercise intensity during gameplay and determine one of the following based on the result of the estimation (e.g., a numerical value indicating the user's exercise intensity). This can enhance the effect of the video game on improving the user's health. - The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) provided to the user. - The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) provided to the user. - Game parameters related to the progress of the video game (e.g., score, damage).

[0184] The information processing system 1 of this embodiment and each modification can also be applied to a video game in which game progress is controlled according to the player's physical movements. The video game may be a mini-game that can be played while the aforementioned therapeutic app, rehabilitation app, or fitness app is being executed, or may be configured as a standalone app. As an example, the information processing system 1 estimates the user's skeletal structure based on a user's video during gameplay. The estimation of the user's skeletal structure may be performed based on at least one of the user's depth and user acceleration in addition to the user's video. Based on the result of the estimation of the user's skeletal structure, the information processing system 1 evaluates how well the user's posture during exercise (e.g., gymnastics) matches an ideal posture (model). The information processing system 1 may determine one of the following depending on the result of this evaluation (e.g., a numerical value indicating the degree to which the user's posture matches the ideal posture). This can enhance the effect of the video game on improving the user's health. The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) given to the user. The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) given to the user. Game parameters related to the progress of the video game (e.g., score, damage).

[0185] In addition to or instead of microphone 18, a microphone of wearable device 50 (a microphone provided in wearable device 50 or connected to wearable device 50) may receive sound waves emitted by the user when capturing the user's video and generate sound data. The sound data may constitute input data for the estimation model described in this embodiment or the modified example. The sound emitted by the user is, for example, at least one of the following: - Sound waves emitted by the rotation of the user's legs (for example, sound generated from the pedals or a drive unit connected to the pedals) - Sound generated by the user's breathing or vocalization

[0186] In the above description, a CPX test is exemplified as a test related to exhaled gas. In a CPX test, a gradually increasing exercise load is applied to the test subject. However, it is not necessary to gradually increase the exercise load applied to the user when recording the user video. Specifically, real-time exercise intensity can be estimated even when the user is subjected to a constant or constantly changeable exercise load. For example, the exercise performed by the user may be bodyweight exercise, calisthenics, or strength training.

[0187] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above-described embodiments. Furthermore, the above-described embodiments can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above-described embodiments and modifications can be combined.

[0188] 1: Information processing system 10: Client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 21: Display 30: Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface 50: Wearable device 51: Storage device 52: Processor 53: Input / output interface 54: Communication interface 55: Display 56: Heart rate sensor 57: Acceleration sensor

Claims

1. A program that causes a computer to function as means for obtaining first data based on sensing results when a user is performing a first type of exercise, means for obtaining a first exercise intensity when the user is performing the first type of exercise based on the first data, means for obtaining second data based on sensing when the user is performing a second type of exercise different from the first type of exercise or at rest, means for obtaining a second exercise intensity when the user is performing the second type of exercise or at rest based on the second data, means for determining a first personal index of the user by performing a predetermined calculation on a plurality of exercise intensities obtained for the user including the first exercise intensity and the second exercise intensity, and means for calculating a parameter representing the characteristics of the user regarding exercise tolerance based on the first personal index of the user and a first reference value.

2. The program according to claim 1, that causes the computer to function as means for selecting a recommended exercise type for the user from the plurality of exercise types using the parameter, the target exercise intensity of the user, and a database associating each of the plurality of exercise types with a standard exercise intensity, and means for outputting information indicating the recommended exercise type.

3. The program according to claim 2, wherein the means for selecting selects, as the recommended exercise type, an exercise type associated with an exercise intensity that does not exceed a corrected exercise intensity obtained by correcting the target exercise intensity by the parameter.

4. The program according to claim 3, wherein the means for selecting selects the recommended exercise type to include an exercise type associated with the maximum exercise intensity that does not exceed the corrected exercise intensity among the plurality of exercise types.

5. The target exercise intensity is determined according to an exercise intensity designated by a person or an algorithm that plans or guides the user's exercise therapy based on the measurement result of the user's exercise tolerance. The program according to any one of claims 2 to 4.

6. The program according to any one of claims 1 to 5, wherein the first reference value is a representative value of the first personal indices of a plurality of persons calculated using the same calculation formula as the first personal index of the user.

7. The program according to claim 6, wherein the plurality of persons are defined such that at least one attribute is the same as that of the user.

8. The program according to any one of claims 1 to 7, wherein the computer is caused to function as means for estimating the movement of the user when performing the first sports event based on the first data, and means for providing feedback to the user when the result of the estimation of the movement does not conform to at least one of a form or a pace defined for the first sports event.

9. The program according to any one of claims 2 to 4, wherein the plurality of sports events include sports events that can be performed without using a device capable of adjusting the exercise load.

10. The program according to any one of claims 1 to 9, wherein the first sports event has a standard exercise intensity different from that of the second sports event.

11. The program according to any one of claims 1 to 10, wherein the plurality of exercise intensities are represented by using at least one of oxygen consumption or energy consumption.

12. The program according to any one of claims 2 to 4, wherein the computer is caused to function as means for acquiring third data based on a sensing result when the user is performing a third sports event selected from the recommended sports events, means for acquiring a third exercise intensity of the user when performing the third sports event based on the third data, means for determining a second personal index of the user by performing a predetermined calculation on the plurality of exercise intensities acquired for the user including the third exercise intensity, and means for calculating the parameter based on the second personal index and a second reference value.

13. Means for acquiring first data based on a sensing result when a user is performing a first exercise type, means for acquiring a first exercise intensity when the user is performing the first exercise type based on the first data, means for acquiring second data based on sensing when the user is performing a second exercise type different from the first exercise type or at rest, means for acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data, means for determining a first personal index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity, and means for calculating a parameter representing the characteristics of the user regarding exercise tolerance based on the first personal index of the user and a first reference value. An information processing apparatus comprising the above.

14. A method in which a computer executes steps of acquiring first data based on a sensing result when a user is performing a first exercise type, acquiring a first exercise intensity when the user is performing the first exercise type based on the first data, acquiring second data based on sensing when the user is performing a second exercise type different from the first exercise type or at rest, acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data, determining a first personal index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity, and calculating a parameter representing the characteristics of the user regarding exercise tolerance based on the first personal index of the user and a first reference value.

15. A system including a plurality of information processing devices, comprising: means for acquiring first data based on a sensing result when a user is performing a first type of exercise; means for acquiring a first exercise intensity when the user is performing the first type of exercise based on the first data; means for acquiring second data based on sensing when the user is performing a second type of exercise different from the first type of exercise or at rest; means for acquiring a second exercise intensity when the user is performing the second type of exercise or at rest based on the second data; means for determining a first individual index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity; and means for calculating a parameter representing a characteristic of the user regarding exercise tolerance based on the first individual index of the user and a first reference value.

Images