Information processing device, method, program, and system

Abstract

A program according to one aspect of the present disclosure causes a computer to function as: a means for presenting a message to a user during exercise; a means for acquiring a sound corresponding to the user's reaction to the message; a means for extracting a sound feature from the sound; and a means for determining the user's exercise intensity on the basis of the sound feature.

Description

Information processing device, method, program, and system

[0001] The present disclosure relates to an information processing device, a method, a program, and a system.

[0002] The importance of exercise therapy, which utilizes exercise for the treatment and prevention of disorders and diseases, is increasing. For example, cardiac rehabilitation aims to help heart disease patients regain their physical strength and confidence, return to a comfortable home and social life, and prevent recurrence of heart disease or re-hospitalization through a comprehensive activity program including exercise therapy. The focus of exercise therapy is aerobic exercise, such as walking, jogging, cycling, and aerobics. To perform aerobic exercise more safely and effectively, it is preferable for the subject to exercise at an intensity near their anaerobic threshold (AT).

[0003] The anaerobic threshold is an example of an evaluation index of exercise tolerance and corresponds to a change point in cardiopulmonary function, i.e., an exercise intensity near the boundary between aerobic exercise and anaerobic exercise. The anaerobic threshold is generally determined by a cardiopulmonary exercise test (CPX), in which a test subject is subjected to a gradually increasing exercise load while exhaled gas is collected and analyzed (see Non-Patent Document 1). In a CPX test, the anaerobic threshold is determined based on the results measured by exhaled gas analysis (e.g., oxygen intake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof). In addition to the anaerobic threshold, a CPX test can also determine the maximum oxygen intake corresponding to an exercise intensity near maximum exercise tolerance.

[0004] In particular, for high-risk patients with heart disease and other conditions, it is important to appropriately manage the exercise intensity of the subjects in order to ensure the effectiveness and safety of exercise therapy.

[0005] Patent Document 1 discloses a technical idea in the field of power-assisted bicycles in which exercise load is measured and exercise intensity is estimated based on the correspondence between the exercise load and exercise intensity.

[0006] Japanese Patent Publication No. 2022-189171

[0007] Saito, Muneyasu, Cardiac Rehabilitation, Physical Therapy, 1997, Vol. 24, No. 8, pp. 414-418

[0008] It is not clear how the technical idea of ​​Patent Document 1 can be applied to exercise other than pedaling an electrically assisted bicycle.

[0009] An object of the present disclosure is to provide a technology that contributes to the implementation of effective and safe exercise therapy in various types of exercise.

[0010] A program according to one aspect of the present disclosure causes a computer to function as a means for presenting a message to a user who is exercising, a means for acquiring a sound corresponding to the user's reaction to the message, a means for extracting sound features from the sound, and a means for determining the user's exercise intensity based on the sound features.

[0011] 1 is a block diagram showing the configuration of an information processing system according to the present embodiment; FIG. 2 is a block diagram showing the configuration of a client device according to the present embodiment; FIG. 3 is a block diagram showing the configuration of a server according to the present embodiment; FIG. 4 is a block diagram showing the configuration of a wearable device according to the present embodiment; FIG. 5 is an explanatory diagram of one aspect of the present embodiment; FIG. 6 is a diagram showing the data structure of an exercise event database according to the present embodiment; FIG. 7 is a diagram showing the data structure of a user profile database according to the present embodiment; FIG. 8 is a flowchart of exercise intensity estimation processing according to the present embodiment; and FIG. 9 is a flowchart of exercise therapy support processing according to the present embodiment.

[0012] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0013] (1) Configuration of Information Processing System The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to this embodiment.

[0014] As shown in FIG. 1 , the information processing system 1 includes a client device 10 , a server 30 , and a wearable device 50 .

[0015] Here, the number of client devices 10 and wearable devices 50 varies depending on, for example, the number of users. Therefore, the number of client devices 10 and wearable devices 50 may each be two or more. Furthermore, a terminal of a person who plans or instructs the exercise therapy may also be included in the information processing system 1. The person who plans or instructs the exercise therapy may include, for example, a medical professional (e.g., a doctor, nurse, pharmacist, physical therapist, occupational therapist, or clinical laboratory technician), a nutritionist, or a trainer.

[0016] The client device 10 and the server 30 are connected via a network (e.g., the Internet or an intranet) NW. The client device 10 and the wearable device 50 are connected via a wireless channel using, for example, Bluetooth (registered trademark) technology.

[0017] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0018] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a server computer.

[0019] The wearable device 50 is an example of an information processing device that can be worn on the user's body (for example, on the arm, hand, or head).

[0020] (1-1) Configuration of the Client Device The configuration of the client device will be described below with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0021] 2 , the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0022] The storage device 11 is configured to store programs and data, and is, for example, a combination of a read-only memory (ROM), a random access memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0023] The programs include, for example, the following: - An operating system (OS) program - An application program that executes information processing (for example, a web browser, a therapeutic app, a rehabilitation app, or a fitness app) Here, diseases that are the target of therapeutic apps or rehabilitation apps are diseases in which exercise may contribute to improving symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

[0024] The data includes, for example, the following data: Databases referenced in information processing Data obtained by executing information processing (i.e., execution results of information processing)

[0025] The processor 12 is a computer that executes the programs stored in the storage device 11 to realize the functions of the client device 10. The processor 12 is, for example, at least one of the following: - CPU (Central Processing Unit) - GPU (Graphic Processing Unit) - ASIC (Application Specific Integrated Circuit) - FPGA (Field Programmable Gate Array)

[0026] The input / output interface 13 is configured to acquire information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10.

[0027] The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0028] The communication interface 14 is configured to control communication between the client device 10 and external devices (e.g., another client device 10, the server 30, and the wearable device 50). Specifically, the communication interface 14 may include a module for communication with the server 30 (e.g., a WiFi module, a mobile communication module, or a combination thereof). The communication interface 14 may include a module for communication with the wearable device 50 (e.g., a Bluetooth module).

[0029] The display 15 is configured to display an image (still image or moving image) and is, for example, a liquid crystal display or an organic EL display.

[0030] The camera 16 is configured to take pictures and generate image signals.

[0031] The depth sensor 17 is, for example, a light detection and ranging (LIDAR) sensor. The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to a surrounding object (for example, a user).

[0032] The microphone 18 is configured to receive sound waves and generate sound signals, and is preferably placed near the user's body (particularly the respiratory tract) as an earphone microphone.

[0033] The acceleration sensor 19 is configured to detect acceleration.

[0034] (1-2) Server Configuration The server configuration will be described below with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0035] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0036] The storage device 31 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0037] The programs include, for example, the following programs: OS programs; application programs for executing information processing.

[0038] The data includes, for example, the following data: Databases referenced in information processing Execution results of information processing

[0039] The processor 32 is a computer that executes the programs stored in the storage device 31 to realize the functions of the server 30. The processor 32 is, for example, at least one of the following: CPU, GPU, ASIC, FPGA

[0040] The input / output interface 33 is configured to acquire information (e.g., user instructions) from an input device connected to the server 30, and to output information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0041] The communication interface 34 is configured to control communications between the server 30 and an external device (eg, the client device 10).

[0042] (1-3) Configuration of the Wearable Device The configuration of the wearable device will be described below. Fig. 4 is a block diagram showing the configuration of the wearable device of this embodiment.

[0043] 4 , the wearable device 50 includes a storage device 51, a processor 52, an input / output interface 53, and a communication interface 54. The wearable device 50 is connected to a display 55, a heart rate sensor 56, and an acceleration sensor 57.

[0044] The storage device 51 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0045] The programs include, for example, the following: - OS programs - Programs for applications that execute information processing (for example, medical applications, rehabilitation applications, or fitness applications)

[0046] The data includes, for example, the following data: Databases referenced in information processing Execution results of information processing

[0047] The processor 52 is a computer that executes the programs stored in the storage device 51 to realize the functions of the wearable device 50. The processor 52 is, for example, at least one of the following: a CPU, a GPU, an ASIC, or an FPGA.

[0048] The input / output interface 53 is configured to acquire information (e.g., user instructions, sensing results) from an input device connected to the wearable device 50, and to output information (e.g., images, commands) to an output device connected to the wearable device 50.

[0049] The input device is, for example, a heart rate sensor 56, an acceleration sensor 57, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 55, a speaker, or a combination thereof.

[0050] The communication interface 54 is configured to control communication between the wearable device 50 and an external device (e.g., the client device 10). Specifically, the communication interface 54 may include a module (e.g., a Bluetooth module) for communication with the client device 10.

[0051] The display 55 is configured to display an image (still image or moving image) and is, for example, a liquid crystal display or an organic EL display.

[0052] The heart rate sensor 56 is configured to measure the heart rate and generate a sensing signal. As an example, the heart rate sensor 56 measures the heart rate using an optical measurement technique.

[0053] The acceleration sensor 57 is configured to detect acceleration.

[0054] (2) One aspect of the embodiment One aspect of the present embodiment will be described below. Fig. 5 is an explanatory diagram of one aspect of the present embodiment.

[0055] As shown in Fig. 5, a user US1 performs some type of exercise. The user US1 is typically a person receiving exercise therapy, such as a participant in a (cardiac) rehabilitation program or an exercise instruction program. In the example of Fig. 5, the type of exercise is gymnastics, but the type of exercise may include any type of exercise (aerobic exercise or anaerobic exercise).

[0056] The client device 10 presents a message to the user US1 who is exercising. The client device 10 may output the message as audio through a speaker, or may display the message on the display 21. Typically, the client device 10 presents a message (e.g., a question or a conversation) that prompts the user US1 to respond by speech.

[0057] User US1 attempts to respond to the presented message by speaking. If the exercise intensity of user US1 is not excessive, user US1 can speak normally. On the other hand, if the exercise intensity of user US1 is excessive, user US1 may become short of breath, causing interruptions in speech or becoming unable to speak at all. In other words, the exercise intensity of user US1 is reflected in the characteristics of the sounds emitted by user US1 (particularly speech, sounds produced by the respiratory organs (e.g., shortness of breath), or a combination thereof). The client device 10 acquires sounds corresponding to the user's response to the message using the microphone 18.

[0058] The client device 10 extracts sound features from the acquired sound, and determines the exercise intensity of the user US1 based on the sound features.

[0059] In this way, the information processing system 1 requests the user US1 to respond by speech while exercising a given type of exercise and determines the exercise intensity of the user US1 based on the sound characteristics collected from the user US1. Therefore, the exercise intensity of the user US1 can be determined for various types of exercise, even in environments where there are no specialized equipment for measuring exercise intensity, such as devices for breath gas testing, or resources for specialists such as physical therapists (e.g., the user US1's home or a primary care physician's clinic). According to this embodiment, for example, by planning or managing exercise therapy taking into account the user US1's exercise intensity, the effectiveness and safety of the exercise therapy can be improved.

[0060] (3) Database The database of this embodiment will be described. The following database is stored in the storage device 11 or the storage device 31.

[0061] (3-1) Exercise Event Database The exercise event database of this embodiment will be described below. Fig. 6 is a diagram showing the data structure of the exercise event database of this embodiment.

[0062] The exercise type database stores exercise type information. The exercise type information is information about exercise types available to the user. In this embodiment, the exercise types include exercise types that can be performed without using devices with adjustable exercise loads, such as gymnastics, bodyweight training, dancing, walking, running, and treadmills. These exercise types are rich in variety because they include types performed in a standing position. Furthermore, the exercise types available in this embodiment may also include types consisting of repeated exercises within a specific range of motion without using a machine (especially a machine with a function to fix the range of motion). Furthermore, in this embodiment, even an exercise type that is generally considered to be a single type, such as squats, is subdivided and defined as multiple exercise types with different exercise intensities by varying the form (e.g., the positional relationship between body parts, the angle of the body parts, the range of motion of the body parts moved during exercise, etc.), pace, number of repetitions, or the duration or number of rest periods. However, the exercise types of this embodiment may further include exercise types performed using equipment capable of adjusting the exercise load, such as an ergometer or strength training using training equipment.

[0063] 6, the exercise event database includes an "ID" field, a "name" field, an "exercise intensity" field, and an "exercise definition" field. Each field is associated with the others.

[0064] The "ID" field stores an exercise type ID. The exercise type ID is information for identifying the exercise type corresponding to the record.

[0065] The "Name" field stores information about the name of the exercise type. The exercise type name information is information about the name of the exercise type corresponding to the record.

[0066] The "exercise intensity" field stores exercise intensity information. The exercise intensity information is information about the standard exercise intensity of the exercise type corresponding to the record. Standard exercise intensity may refer to, for example, the exercise intensity when a person with standard physical function performs the corresponding exercise type as defined. As an example, such exercise intensity may be derived by, for example, actually measuring the exercise intensity (e.g., average oxygen consumption) when one or more people perform the corresponding exercise type as defined (i.e., according to a specific pace or a specific range of motion) using, for example, exhaled gas analysis (or estimating it by analyzing video footage of the user exercising), and then statistically processing the results (e.g., averaging). Here, whether a person is performing the exercise type as defined may be determined by an algorithm based on the person's sensing results, or may be determined by a human.

[0067] For example, exercise intensity may be derived by measuring (or estimating) the exercise intensity of each section of a corresponding exercise type when one or more people perform the exercise defined for that type of exercise, applying the measurement results for each section to a predetermined formula for each person, and then statistically processing (e.g., averaging) the values ​​obtained by this formula across people. Here, a section is a component of an exercise type, and if an exercise type consists of multiple movement patterns, each movement pattern may correspond to a section. For example, if the exercise type is dance, each choreography may correspond to a section. Furthermore, if the exercise type is squats, the transition from a standing position to a crouching position and the transition from a crouching position to a standing position may each correspond to a section.

[0068] The "exercise definition" field stores exercise definition information. The exercise definition information is information related to the definition of the exercise type corresponding to the record. The exercise definition information can include information related to at least one of the following elements: - Acceptable range of exercise pace (for example, the time required for one cycle of movement, or the speed or acceleration of the moving body part) - Acceptable range of positional relationship between body parts (for example, foot width, etc.) - Acceptable range of angle of body part (which may be one or multiple) (for example, the direction of the knee, or the angle formed by the upper arm and forearm) - Acceptable range of motion of body part (which may be one or multiple) moved during exercise (for example, the range of movement of each body part during one cycle of movement) - Number of repetitions - Rest time - Exercise load (for example, the magnitude of the external load set in the case of exercise performed using equipment with adjustable exercise load, such as an ergometer)

[0069] Even for an exercise that is generally recognized as a single event (e.g., squats), multiple events with slightly different exercise intensities can be defined by specifying and defining details of the pace, form (particularly the range of motion), number of repetitions, rest time, and exercise load. For example, for an exercise that is generally recognized as a single event, multiple events with exercise intensities that differ by 0.2 METs can be defined.

[0070] (3-2) User Profile Database The user profile database of this embodiment will now be described. Fig. 7 is a diagram showing the data structure of the user profile database of this embodiment.

[0071] The user profile database stores user profile information, which is information relating to the profile of a user of the information processing system 1 (i.e., a person who exercises).

[0072] 7, the user profile database includes an "ID" field, a "Name" field, a "Target Strength" field, and a "Body" field. Each field is associated with the others.

[0073] The "ID" field stores a user ID, which is information for identifying the user corresponding to the corresponding record.

[0074] The "Name" field stores user name information, which is information about the name of the user (for example, name, account name, etc.) corresponding to the corresponding record.

[0075] The "target intensity" field stores target intensity information. The target intensity information is information regarding the target value of exercise intensity (e.g., oxygen consumption, energy consumption, heart rate, or a combination thereof) set for the user corresponding to the record. As an example, the target intensity information is specified by a person (e.g., a doctor) planning or instructing an exercise regimen based on the results of measuring the user's exercise tolerance, for example, by CPX while using an ergometer (e.g., oxygen consumption and heart rate at the anaerobic threshold (AT)). However, CPX is not required, and the target value may be specified at the doctor's discretion. As another example, the target intensity information is determined by an algorithm based on the results of measuring the user's exercise tolerance, for example, by CPX. To perform aerobic exercise more safely and effectively, it is preferable to exercise at an intensity near the anaerobic threshold. Therefore, the target value of exercise intensity is, for example, but not limited to, an exercise intensity corresponding to the anaerobic threshold. While an ergometer is typically used as the exercise type for CPX measurements, the oxygen consumption at the anaerobic threshold when using a treadmill, for example, is approximately 1.2 to 1.3 times that when using an ergometer. This is thought to be because the total muscle mass used on the treadmill exceeds the total muscle mass used on the ergometer. Therefore, as a first example, the target intensity information may be a value obtained by correcting the target value based on the results of CPX measurements when using an ergometer to approximately 1.2 to 1.3 times the original value, or a value obtained by further subtracting a predetermined value (e.g., 1 MET). As a second example, the target intensity information may be a value obtained by correcting the target value based on the results of CPX measurements when using an ergometer without correcting it. This prevents the target value from being excessively high when selecting an exercise type that uses a relatively small amount of muscle mass, such as an ergometer. As a third example, the target value for each exercise type may be corrected, for example, by a coefficient corresponding to the total muscle mass used.

[0076] A doctor may also set an upper limit on exercise intensity for a user (an example of exercise prescription). In this case, the user is not permitted to select an exercise type that exceeds the prescribed upper limit of exercise intensity. For exercise prescription, a UI (User Interface) screen for exercise prescription may be displayed on the display of the doctor's device. Such a UI screen may include, for example, the following information: The user's CPX data; and a display area for sample videos of multiple selectable exercise types. The display area for each sample video of each exercise type may be arranged according to the exercise intensity information corresponding to the exercise type. For example, if 3.6 METs is recommended as the upper limit based on the CPX data, a group of display areas for sample videos of exercise types corresponding to 3.4 METs, a group of display areas for sample videos of exercise types corresponding to 3.6 METs, and a group of display areas for sample videos of exercise types corresponding to 3.8 METs may be arranged on the UI screen. Information on exercise types (e.g., sample videos) may be arranged in units smaller or larger than 0.2 METs. When the doctor selects one of the display areas, the exercise intensity associated with the corresponding exercise type is set as the upper limit.

[0077] In addition, the upper limit specified by the doctor through exercise prescription may be raised or lowered by a medical professional under the supervision of the doctor during regular (e.g., every two weeks) medical guidance or doctor rounds.

[0078] The "Physical" field stores physical information. The physical information is information about the body (functions) of the user corresponding to the record. For example, the physical information may include information about the user's age, gender, weight, height, illnesses, etc.

[0079] In addition, the user profile database may store the following information: Information indicating the person who planned or instructed the user's exercise therapy Information indicating the user's doctor

[0080] (4) Information Processing Information processing in this embodiment will be described.

[0081] (4-1) Exercise Intensity Estimation Process The exercise intensity estimation process of this embodiment will be described. Fig. 8 is a flowchart of the exercise intensity estimation process of this embodiment. The exercise intensity estimation process can easily estimate exercise intensity based on the user's exercise tolerance (for example, exercise intensity equivalent to the user's anaerobic metabolic threshold) (without requiring special equipment such as an exhaled gas testing device or expert resources such as a physical therapist).

[0082] The exercise intensity estimation process starts when, for example, one of the following start conditions is met: - The exercise intensity estimation process is called by another process; - The user or a person who plans or instructs the user's exercise therapy performs an operation to call the exercise intensity estimation process; - The client device 10 enters a predetermined state (for example, a predetermined application is launched); - A predetermined date and time has arrived; - A predetermined amount of time has passed since a predetermined event.

[0083] As shown in FIG. 8 , the client device 10 executes the step of identifying an exercise type (S110). Specifically, the client device 10 identifies the type of exercise to be performed by the user. As a first example of identifying an exercise type (S110), the client device 10 identifies the type of exercise in response to instructions from the user or an instructor. As a second example of identifying an exercise type (S110), the client device 10 determines the type of exercise to be performed by the user using a predetermined algorithm. As a third example of identifying an exercise type (S110), the client device 10 identifies the type of exercise changed in the step of changing the exercise type (S115) described below. As a fourth example of identifying an exercise type (S110), the client device 10 identifies the type of exercise the user is performing by analyzing the user's movements after starting the exercise. In this step, the client device 10 may present the user with information about the identified exercise type (e.g., a demonstration video).

[0084] After step S110, the client device 10 executes message presentation (S111). Specifically, the client device 10 presents a message that requests a speech response from the user. The client device 10 may output the message aloud through a speaker or may display the message on the display 21. The client device 10 may present a fixed message, or may present a message selected randomly or by a predetermined algorithm from a plurality of predetermined messages. Alternatively, the client device 10 may generate a message each time, for example, on a rule-based basis or using a generative model (e.g., a large-scale language model). For example, the client device 10 may generate a message by providing a prompt based on the content spoken by the user to a generative model.

[0085] After step S111, the client device 10 performs sound acquisition (S112). Specifically, the microphone 18 collects sounds (particularly speech sounds, sounds produced by the respiratory organs (e.g., shortness of breath), or a combination thereof) emitted by the user as the user's reaction to the message presented in step S111, and generates a sound signal. The client device 10 acquires the sound signal from the microphone 18.

[0086] After step S112, the client device 10 extracts sound features (S113). Specifically, the client device 10 extracts sound features from the sound signal acquired in step S112. The sound features may include features representing the pitch (which may include the amount of change), the loudness (which may include the amount of change), or the timbre (which may include the amount of change) of the sound emitted by the user. As an example, the sound features may include features related to the spectral envelope, such as LPC (Linear Predictive Coding) coefficients or MFCC (Mel-Frequency Cepstrum Coefficient).

[0087] After step S113, the client device 10 executes an exercise intensity determination (S114). Specifically, the client device 10 determines the user's exercise intensity based on the sound features extracted in step S113. The client device 10 determines whether the user's exercise intensity exceeds the user's anaerobic metabolic threshold. Note that the trained model may determine the user's exercise intensity from three or more states, or from multiple states defined by boundaries different from the anaerobic metabolic threshold. The client device 10 may store the determination result in the storage device 11 in association with information about the type of exercise identified in step S110.

[0088] As a first example of determining the exercise intensity (S114), the client device 10 determines the exercise intensity of the user by applying a trained model to the model input based on the sound features extracted in step S131. The trained model determines, for example, which of a plurality of states the user's exercise intensity is in based on the user's exercise tolerance.

[0089] The trained model may be constructed by machine learning to imitate a talk test conducted by a specialist such as a physical therapist. The trained model may be constructed by machine learning (supervised learning) using training data including, for example, sound features extracted from training sounds and ground truth data indicating the judgment results of a talk test conducted by a specialist on the training sounds (e.g., "exceeds the anaerobic metabolic threshold" / "does not exceed the anaerobic metabolic threshold").

[0090] As a second example of the exercise intensity determination (S114), the client device 10 determines the exercise intensity of the user by applying pattern recognition to the sound feature extracted in step S131. For example, the client device 10 determines which of a plurality of states based on the exercise tolerance of the user the sound feature matches.

[0091] After step S114, the client device 10 determines whether a termination condition is met for the exercise intensity determination result (which may include past determination results if step S114 is repeatedly executed). For example, the termination condition is that a first determination result and a second determination result different from the first determination result are obtained. For example, the first determination result may correspond to a state in which the user's exercise intensity does not exceed a certain boundary (e.g., the anaerobic metabolic threshold), and the second determination result may correspond to a state in which the user's exercise intensity exceeds the boundary (the correspondence between the determination results and the states may be reversed from this example).

[0092] If the termination condition is not met, the client device 10 executes a change of exercise type (S115). Specifically, the client device 10 selects an exercise type different from the exercise type identified in step S110 from the exercise types registered in the exercise type database ( FIG. 6 ). As a first example of the change of exercise type (S115), the client device 10 may select an exercise type corresponding to a higher exercise intensity than the exercise type identified in step S110 when the only determination result obtained is that the user's exercise intensity does not exceed a certain boundary. As a second example of the change of exercise type (S115), the client device 10 may select an exercise type corresponding to a lower exercise intensity than the exercise type identified in step S110 when the only determination result obtained is that the user's exercise intensity exceeds a certain boundary. After step S115, the client device 10 returns to identifying the exercise type (S110).

[0093] The client device 10 may select and present multiple candidates to the user, and the user may select a new exercise type from the candidates. In this case, the client device 10 determines the exercise intensity required for the new exercise type and selects a candidate (exercise type) that corresponds to the exercise intensity.

[0094] On the other hand, if the termination condition is met, the client device 10 executes exercise intensity estimation (S116). Specifically, the client device 10 estimates the user's exercise intensity (particularly, exercise intensity based on the user's exercise tolerance) based on the exercise intensity determination results (i.e., determination results for multiple exercise types with different exercise intensities) executed multiple times in step S114. As an example, the client device 10 identifies the exercise intensity corresponding to the boundary between different states based on the state indicated by the exercise intensity determination result in step S114 and the exercise intensity associated with the determination result (i.e., the exercise intensity indicated by the exercise intensity information for the exercise type performed by the user). For example, if the exercise intensities associated with a determination result that the anaerobic metabolic threshold has not been exceeded are 3.0 METs, 3.2 METs, and 3.4 METs, and the exercise intensity associated with a determination result that the anaerobic metabolic threshold has been exceeded is 3.6 METs, the client device 10 may estimate a value in the range of 3.4 METs to 3.6 METs as the exercise intensity corresponding to the user's anaerobic metabolic threshold. The client device 10 may use the estimation result to update the corresponding item in the user profile database ( FIG. 7 ). After step S116, the client device 10 may terminate the exercise intensity estimation process of this embodiment.

[0095] (4-2) Exercise Therapy Support Processing The exercise therapy support processing of this embodiment will be described. Fig. 9 is a flowchart of the exercise therapy support processing of this embodiment. The exercise therapy support processing can support the safe implementation of exercise therapy so that the user's exercise intensity does not exceed a threshold (e.g., anaerobic metabolic threshold) for a long period of time.

[0096] The exercise therapy support process starts when, for example, any of the following start conditions is met: - The exercise therapy support process is called by another process; - The user or a person who plans or instructs the user's exercise therapy performs an operation to call the exercise therapy support process; - The client device 10 enters a predetermined state (for example, a predetermined application is launched); - A predetermined date and time has arrived; - A predetermined amount of time has passed since a predetermined event.

[0097] As shown in Figure 9, the client device 10 performs the same steps as in Figure 8, such as identifying the type of exercise (S110), presenting a message (S111), acquiring sound (S112), extracting sound features (S113), and determining the exercise intensity (S114).

[0098] After step S114, the client device 10 determines whether an adjustment condition is met based on the exercise intensity determination result (which may include past determination results if step S114 is repeatedly executed). For example, the adjustment condition may be a determination result indicating that the user's exercise intensity exceeds a certain boundary (e.g., the anaerobic metabolic threshold). Furthermore, the client device 10 may determine whether the adjustment condition is met by further referring to, in addition to the determination result of step S114, the perceived intensity of exercise assessed by the user for the type of exercise identified in step S110, or the results of sensing the user while performing the type of exercise (e.g., heart rate) or information based thereon (e.g., respiratory rate).

[0099] If the adjustment condition is not met, the client device 10 re-executes the message presentation (S111). On the other hand, if the adjustment condition is met, the client device 10 executes a load adjustment instruction (S215). Specifically, the client device 10 presents the user with instructions to adjust the load (exercise intensity) imposed on the user. As an example, the client device 10 may present instructions to reduce the load (exercise intensity). For example, the client device 10 may present instructions (e.g., an image, audio, text, or a combination thereof) including information indicating slowing the pace of the exercise type identified in step S111, changing the form (range of motion) to a variation with a lighter load, reducing the number of repetitions, increasing the number or duration of rest periods, reducing the external load, or changing to an exercise type corresponding to a lower exercise intensity.

[0100] Depending on the adjustment condition determined to be met before transitioning to step S215, the client device 10 may present different instructions (e.g., instructions to increase the load (exercise intensity)). The client device 10 may present instructions (e.g., images, audio, text, or a combination thereof) including information indicating to speed up the pace of the exercise type identified in step S111, change the form (range of motion) to a variation with a heavier load, increase the number of repetitions, reduce the number or duration of rest periods, increase the external load, or change to an exercise type corresponding to a higher exercise intensity. However, if a doctor has set an upper limit on the exercise intensity for the user, the client device 10 may determine instructions so as not to exceed that upper limit.

[0101] After step S215, the client device 10 executes the message presentation (S111). Alternatively, the client device 10 may execute the exercise type identification (S110) when the client device 10 has presented information indicating that the exercise type will be changed.

[0102] The client device 10 may terminate the exercise therapy support process of this embodiment in response to the completion of the exercise event currently being performed, or may re-execute the exercise therapy support process of this embodiment in response to the start of a new exercise event.

[0103] (5) Summary As described above, the client device 10 of this embodiment presents a message to a user while exercising, acquires a sound corresponding to the user's response to the message, extracts sound features from the sound, and determines the user's exercise intensity based on the sound features. This makes it possible to grasp the user's exercise intensity for various types of exercise, even in environments where there are no specialized equipment for measuring exercise intensity, such as devices for breath gas testing, or resources for specialists such as physical therapists (e.g., the user's home or a primary care physician's clinic). According to this embodiment, for example, by planning or managing exercise therapy taking into account the exercise intensity of the user US1, the effectiveness and safety of the exercise therapy can be improved.

[0104] The client device 10 may estimate the exercise intensity corresponding to the user's exercise tolerance based on the assessment results for a plurality of exercise types with different exercise intensities, thereby making it possible to grasp the exercise intensity corresponding to the user's exercise tolerance.

[0105] The client device 10 may estimate the exercise intensity corresponding to the user's exercise tolerance based on the exercise intensity associated with the exercise type for which a first determination result was obtained and the exercise intensity associated with the exercise type for which a second determination result different from the first determination result was obtained, thereby making it possible to grasp the exercise intensity that is the boundary between the state corresponding to the first determination result and the state corresponding to the second determination result.

[0106] The client device 10 may present instructions for adjusting the load on the user in accordance with the determination result, thereby ensuring at least safety during the exercise therapy.

[0107] The instructions may include information regarding pace, form, reps, rest, external resistance, or changes in exercise type, allowing fine tuning of the load on the user.

[0108] The client device 10 may determine the user's exercise intensity by applying a trained model to the model input based on the sound features. The trained model may be constructed using machine learning to mimic a talk test conducted by an expert. This allows the user's exercise intensity to be determined at a level similar to that of a talk test conducted by an expert, without having to secure the resources of an expert to conduct the talk test while the user is exercising.

[0109] The client device 10 may determine the user's exercise intensity by applying a trained model to a model input based on sound features. The trained model may be constructed by machine learning using training data including sound features extracted from training sounds and ground truth data indicating the results of a talk test conducted by an expert on the training sounds. This allows the user's exercise intensity to be determined appropriately.

[0110] The client device 10 may generate messages using a generative model, thereby preventing the user from becoming bored or feeling uncomfortable due to the constant presentation of standardized messages.

[0111] The client device 10 may generate a message by providing a generative model with prompts based on the content of the user's utterance, thereby allowing for the presentation of natural-sounding messages that take into account the flow of dialogue with the user.

[0112] (6) Other Modifications The storage device 11 may be connected to the client device 10 via the network NW. Each input device or output device may be integrated with the client device 10. The storage device 31 may be connected to the server 30 via the network NW. Each input device or output device may be integrated with the wearable device 50.

[0113] Each step of the above information processing can be executed by either the client device 10 or the server 30. As an example, the server 30 may execute some or all of the steps of the exercise intensity estimation process or the exercise therapy support process. One or more steps of the above information processing may be performed using a trained model.

[0114] In this embodiment, an example of determining a user's exercise intensity based on sound features has been described. However, the user's exercise intensity may also be determined based on image features in addition to sound features. Specifically, the client device 10 may acquire an image (still image, moving image, or a combination thereof) of the user exercising, captured by, for example, the camera 16, and extract image features from the image. The image features may represent, for example, the user's facial expression, complexion, or the movement of the clavicle or rib cage. As the exercise intensity increases, the user feels more strained and changes in facial expression or complexion appear. Furthermore, as the exercise intensity increases, the user's breathing becomes heavier, resulting in more intense movement of the clavicle or rib cage. The client device 10 may then determine the user's exercise intensity by applying a trained model (or performing pattern recognition) to the model input based on the sound features and image features. This allows for a more accurate determination of the user's exercise intensity. The trained model can be constructed by machine learning using training data including sound features and image features extracted from training sounds and images, and correct answer data indicating the results of a talk test conducted by an expert on the training sounds and images. The image features can include features based on depth measured by a depth sensor.

[0115] In the above description, an example has been shown in which an image is captured using the camera 16 of the client device 10. However, the image may be captured using a camera other than the camera 16. In the above description, an example has been shown in which the depth is measured using the depth sensor 17 of the client device 10. However, the depth may be measured using a depth sensor other than the depth sensor 17.

[0116] In the present embodiment, an example in which a message is presented unconditionally in the exercise intensity estimation process and the exercise therapy support process has been described. However, the message may be presented conditionally. Specifically, the client device 10 may omit presenting the message when the exercise intensity associated with the type of exercise being performed by the user is equal to or lower than a threshold. Here, the threshold may be determined based on the user's exercise tolerance (e.g., an exercise intensity obtained by subtracting a predetermined margin from the exercise intensity corresponding to the user's anaerobic metabolic threshold, or an exercise intensity obtained by multiplying the exercise intensity corresponding to the user's anaerobic metabolic threshold by a predetermined coefficient (less than 1 and greater than 0)). This reduces the burden of responding to messages when the user is exercising at an exercise intensity within a range that does not pose safety concerns. Alternatively, the client device 10 may present messages less frequently the lower the exercise intensity associated with the type of exercise being performed by the user.

[0117] The client device 10 may further perform the following processes, for example, while the user is exercising. Specifically, the client device 10 estimates the user's movements during exercise (e.g., movements of the skeleton or other feature points over multiple time periods, or the state of the skeleton or other feature points at a single time) based on the sensing data. Then, if the estimated movements do not match at least one of the form or pace specified for the type of exercise the user is performing (e.g., the range of motion of a body part is too narrow or too wide, the angle of a body part deviates from the standard, or the pace is too fast or too slow), the client device 10 provides feedback to the user. The feedback may include at least one of the following: - Output of light, sound, or voice (e.g., voice informing the user of an incorrect part of the exercise) - Display of an image (e.g., an image informing the user of an incorrect part of the exercise) - Vibration of the wearable device. This can prevent the user's exercise intensity from deviating significantly from the standard intensity for the recommended type of exercise. Alternatively, the client device 10 may recommend that the user perform a different type of exercise if the number or frequency at which the results of the movement estimation do not conform to at least one of the form or pace defined for the type of exercise the user is performing exceeds a threshold.

[0118] In the above description, an example has been given in which the wearable device 50 measures the user's heart rate. However, the heart rate can also be obtained by analyzing video data or its analysis results (e.g., skin color data) (e.g., remote photoplethysmography (rPPG) analysis). The heart rate analysis may be performed using a trained model constructed using machine learning technology. Alternatively, the user may exercise while wearing electrodes for an electrocardiogram monitor, allowing the electrocardiogram monitor to measure the user's heart rate. In these variations, the user does not need to wear the wearable device 50 to measure the heart rate.

[0119] The wearable device 50 may be provided with a sensor for measuring at least one of the following items instead of or in addition to the heart rate sensor 56 and the acceleration sensor 57: blood glucose level oxygen saturation The measurement results from each sensor may be used as appropriate to estimate exercise intensity or in other situations. For example, the measurement result of the blood glucose level may be referenced to evaluate exercise intensity converted into energy consumption or oxygen consumption.

[0120] In the above description, a CPX test is exemplified as a test related to exhaled gas. In a CPX test, a gradually increasing exercise load is applied to the test subject. However, the exercise intensity estimation process does not require a gradually increasing exercise load applied to the user. Specifically, exercise intensity can be estimated even when the user is subjected to a constant or constantly changeable exercise load. For example, the exercise performed by the user may be bodyweight exercise, calisthenics, or strength training.

[0121] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above-described embodiments. Furthermore, the above-described embodiments can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above-described embodiments and modifications can be combined.

[0122] 1: Information processing system 10: Client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 21: Display 30: Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface 50: Wearable device 51: Storage device 52: Processor 53: Input / output interface 54: Communication interface 55: Display 56: Heart rate sensor 57: Acceleration sensor

Claims

1. A program that causes a computer to function as means for presenting a message to a user during exercise, means for acquiring a sound corresponding to the user's reaction to the message, means for extracting a sound feature amount from the sound, and means for determining the exercise intensity of the user based on the sound feature amount.

2. The program according to claim 1, that causes the computer to function as means for estimating an exercise intensity corresponding to the exercise tolerance of the user based on the determination result by the means for determining for a plurality of exercise types with different exercise intensities.

3. The program according to claim 2, wherein the means for estimating estimates the exercise intensity corresponding to the exercise tolerance of the user based on the exercise intensity associated with the exercise type for which a first determination result is obtained and the exercise intensity associated with the exercise type for which a second determination result different from the first determination result is obtained.

4. The program according to any one of claims 1 to 3, that causes the computer to function as means for presenting an instruction for adjusting the load applied to the user according to the determination result by the means for determining.

5. The program according to claim 4, wherein the instruction includes information regarding a change in pace, form, number of reps, rest, external load, or exercise type.

6. The program according to any one of claims 1 to 5, wherein the means for determining determines the exercise intensity of the user by applying a learned model to a model input based on the sound feature amount, and the learned model is constructed by machine learning for mimicking a talk test by an expert.

7. The program according to any one of claims 1 to 6, wherein the means for determining determines the exercise intensity of the user by applying a learned model to a model input based on the sound feature amount, and the learned model is constructed by machine learning using learning data including the sound feature amount extracted from the learning sound and the correct answer data indicating the determination result of the talk test by an expert for the learning sound.

8. Cause the computer to function as means for acquiring an image of the user during exercise and means for extracting image feature amounts from the image, and the means for determination determines the exercise intensity of the user based on the sound feature amounts and the image feature amounts. The program according to any one of claims 1 to 7.

9. The means for presentation omits the presentation of the message when the exercise intensity associated with the type of exercise the user is performing is less than or equal to a threshold value. The program according to any one of claims 1 to 8.

10. The threshold value is determined according to the exercise tolerance of the user. The program according to claim 9.

11. Cause the computer to further function as means for generating the message using a generation model. The program according to any one of claims 1 to 10.

12. The means for generating generates the message by giving a prompt based on the content spoken by the user to the generation model. The program according to claim 11.

13. An information processing apparatus comprising means for presenting a message to a user during exercise, means for acquiring a sound corresponding to the user's reaction to the message, means for extracting sound feature amounts from the sound, and means for determining the exercise intensity of the user based on the sound feature amounts.

14. A method in which a computer executes steps of presenting a message to a user during exercise, acquiring a sound corresponding to the user's reaction to the message, extracting sound feature amounts from the sound, and determining the exercise intensity of the user based on the sound feature amounts.

15. A system including a plurality of information processing apparatuses, the system comprising means for presenting a message to a user during exercise, means for acquiring a sound corresponding to the user's reaction to the message, means for extracting sound feature amounts from the sound, and means for determining the exercise intensity of the user based on the sound feature amounts.

Images