Program, information processing device, and method for processing information

Abstract

To perform evaluation relating to user's exercise tolerability without causing any burden on the user.SOLUTION: A program runs a computer as means for acquiring a user's moving image in which a user in exercise is filmed and means for performing estimation relating to user's exercise tolerability on the basis of the user's moving image.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] Cardiac rehabilitation aims to help heart disease patients regain their strength and self-confidence, return to comfortable home and social life, and prevent recurrence of heart disease or re-hospitalization through a comprehensive activity program including exercise therapy. Exercise therapy focuses on aerobic exercise such as walking, jogging, cycling, and aerobics. To perform aerobic exercise safely and effectively, patients should exercise at an intensity close to their anaerobic threshold (AT).

[0003] The anaerobic metabolic threshold is an example of an evaluation index of exercise tolerance and corresponds to a change point in cardiopulmonary function, i.e., an exercise intensity near the boundary between aerobic exercise and anaerobic exercise. The anaerobic metabolic threshold is generally determined by a cardiopulmonary exercise test (CPX test), in which a test subject is subjected to a gradually increasing exercise load while exhaled gas is collected and analyzed (see Non-Patent Document 1). In a CPX test, the anaerobic metabolic threshold is determined based on the results measured by exhaled gas analysis (e.g., oxygen intake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof). In addition to the anaerobic metabolic threshold, a CPX test can also determine the maximum oxygen intake, which corresponds to an exercise intensity near the maximum exercise tolerance. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Muneyasu Saito, Cardiac Rehabilitation, Physical Therapy, 1997, Vol. 24, No. 8, pp. 414-418 Summary of the Invention [Problem to be solved by the invention]

[0005] CPX testing has problems such as placing a heavy physical burden on the test subject, and the testing equipment being expensive and limited to facilities where it can be performed. In addition, wearing an exhalation mask, which is required for the exhaled gas analysis essential for CPX testing, is uncomfortable for the test subject.

[0006] Forcing subjects to wear special equipment such as a respiratory mask during exercise may be uncomfortable, inconvenient, or burdensome, and may discourage them from participating in assessments of exercise tolerance indicators (e.g., anaerobic threshold or maximal oxygen uptake).

[0007] An object of the present disclosure is to evaluate a user's exercise tolerance without imposing a burden on the user. [Means for solving the problem]

[0008] A program according to one aspect of the present disclosure causes a computer to function as a means for acquiring a user video showing a user exercising, and a means for making an estimation regarding the exercise tolerance of the user based on the user video. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to evaluate a user's exercise tolerance without imposing a burden on the user. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing the configuration of a client device according to the present embodiment. [Figure 3] FIG. 2 is a block diagram showing the configuration of a server according to the present embodiment. [Figure 4] FIG. 1 is a block diagram showing the configuration of a wearable device according to an embodiment of the present invention. [Figure 5] FIG. 1 is an explanatory diagram of an overview of the present embodiment. [Figure 6] FIG. 2 is a diagram illustrating a data structure of a training data set according to the present embodiment. [Figure 7] 4 is a flowchart of information processing according to the present embodiment. [Figure 8] 10A and 10B are diagrams illustrating examples of screens displayed in the information processing of the present embodiment. [Figure 9] 10A and 10B are diagrams illustrating examples of screens displayed in the information processing of the present embodiment. [Figure 10] 10A and 10B are diagrams illustrating examples of screens displayed in the information processing of the present embodiment. [Figure 11] FIG. 10 is a diagram showing the data structure of a training data set according to the first modification. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0012] (1) Information processing system configuration The configuration of the information processing system will now be described with reference to Fig. 1, which is a block diagram showing the configuration of the information processing system according to this embodiment.

[0013] As shown in FIG. 1, the information processing system 1 includes a client device 10, a server 30, and a wearable device 50. The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW. The client device 10 and the wearable device 50 are connected via a wireless channel using, for example, Bluetooth (registered trademark) technology.

[0014] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0015] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a web server.

[0016] The wearable device 50 is an example of an information processing device that can be worn on the body (for example, on the arm) of a user.

[0017] (1-1) Client device configuration The configuration of the client device will now be described with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0018] 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, and a depth sensor 17.

[0019] The storage device 11 is configured to store programs and data, and is, for example, a combination of a read-only memory (ROM), a random access memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0020] The programs include, for example, the following programs: OS (Operating System) programs · Programs for applications that process information (e.g., web browsers, rehabilitation apps, or fitness apps)

[0021] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0022] The processor 12 is a computer that implements the functions of the client device 10 by running a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Gate Array)

[0023] The input / output interface 13 is configured to acquire information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10. The input device is, for example, a camera 16, a depth sensor 17, a microphone, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0024] The communication interface 14 is configured to control communication between the client device 10 and external devices (eg, the server 30 and the wearable device 50). Specifically, the communication interface 14 may include a module (e.g., a WiFi module, a mobile communication module, or a combination thereof) for communication with the server 30. The communication interface 14 may include a module (e.g., a Bluetooth module) for communication with the wearable device 50.

[0025] The display 15 is configured to display an image (a still image or a moving image). The display 15 is, for example, a liquid crystal display or an organic EL display.

[0026] The camera 16 is configured to take pictures and generate image signals.

[0027] The depth sensor 17 is, for example, a light detection and ranging (LIDAR) sensor. The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to a surrounding object (for example, a user).

[0028] (1-2) Server configuration The configuration of the server will now be described with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0029] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0030] The storage device 31 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0031] The programs include, for example, the following programs: OS programs Application programs that perform information processing

[0032] The data includes, for example, the following data: Databases referenced in information processing - Results of information processing

[0033] The processor 32 is a computer that implements the functions of the server 30 by running a program stored in the storage device 31. The processor 32 is, for example, at least one of the following: ·CPU GPU ASIC FPGA

[0034] The input / output interface 33 is configured to obtain information (for example, a user's instruction) from an input device connected to the server 30 and to output information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0035] The communication interface 34 is configured to control communications between the server 30 and an external device (eg, the client device 10).

[0036] (1-3) Wearable device configuration The configuration of the wearable device will now be described with reference to Figure 4. Figure 4 is a block diagram showing the configuration of the wearable device of this embodiment.

[0037] 4, the wearable device 50 includes a storage device 51, a processor 52, an input / output interface 53, and a communication interface 54. The wearable device 50 is connected to a display 55 and a heart rate sensor 56.

[0038] The storage device 51 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0039] The programs include, for example, the following programs: OS programs · Programs for applications that process information (e.g., rehabilitation apps or fitness apps)

[0040] The data includes, for example, the following data: Databases referenced in information processing - Results of information processing

[0041] The processor 52 is a computer that executes the programs stored in the storage device 51 to realize the functions of the wearable device 50. The processor 52 is, for example, at least one of the following: ·CPU GPU ASIC FPGA

[0042] The input / output interface 53 is configured to acquire information (e.g., user instructions, sensing results) from an input device connected to the wearable device 50, and to output information (e.g., images, commands) to an output device connected to the wearable device 50. The input device is, for example, a heart rate sensor 56, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 55, a speaker, or a combination thereof.

[0043] The communication interface 54 is configured to control communication between the wearable device 50 and an external device (eg, the client device 10). Specifically, the communication interface 54 may include a module for communication with the client device 10 (eg, a Bluetooth module).

[0044] The display 55 is configured to display an image (a still image or a moving image). The display 55 is, for example, a liquid crystal display or an organic EL display.

[0045] The heart rate sensor 56 is configured to measure the heart rate and generate a sensing signal. As an example, the heart rate sensor 56 measures the heart rate using an optical measurement technique.

[0046] (2) Overview of the embodiment An outline of this embodiment will be described below with reference to Fig. 5.

[0047] As shown in Fig. 5, the camera 16 of the client device 10 captures an external image (e.g., the entire body) of the user US1 while exercising. In the example of Fig. 5, the user US1 is exercising by bicycle, but the user US1 can perform any exercise (aerobic exercise or anaerobic exercise).

[0048] As an example, the camera 16 captures an image of the appearance of the user US1 from the front or obliquely in front. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. It is also possible to generate three-dimensional video data by combining, for example, video data (two-dimensional) generated by the camera 16 with, for example, depth data generated by the depth sensor 17.

[0049] The heartbeat sensor 56 of the wearable device 50 measures the heartbeat of the user US1 and transmits the measurement result to the client device 10.

[0050] The client device 10 analyzes the physical condition of the user while exercising by referring to at least the video data acquired from the camera 16. The client device 10 may further refer to the depth data acquired from the depth sensor 17 to analyze the physical condition of the user while exercising. The client device 10 transmits data (hereinafter referred to as "user data") related to the physical condition of the user US1 while exercising to the server 30, based on the analysis results of the video data (or the video data and the depth data) and the measurement results of the heart rate of the user US1 acquired from the wearable device 50.

[0051] The server 30 estimates the exercise tolerance of the user US1 by applying the learned model LM1 (an example of an "estimation model") to the acquired user data. The server 30 transmits the estimation result (e.g., a numerical value indicating the anaerobic metabolic threshold, the maximum oxygen uptake, or the real-time cardiopulmonary exercise load of the user US1) to the client device 10.

[0052] In this way, the information processing system 1 estimates the exercise tolerance of the user US1 based on the video (or video and depth) and heart rate of the user US1 during exercise. Therefore, the information processing system 1 can evaluate the exercise tolerance of the user US1 without imposing a burden on the user, such as wearing special equipment.

[0053] (3) Training dataset The teacher dataset of this embodiment will now be described with reference to Fig. 6, which is a diagram showing the data structure of the teacher dataset of this embodiment.

[0054] As shown in Figure 6, the training data set includes multiple training data. The training data is used for training or evaluating a target model. The training data includes a sample ID, input data, and correct answer data.

[0055] The sample ID is information that identifies the training data.

[0056] The input data is data that is input to the target model during training or evaluation. The input data corresponds to example problems used during training or evaluation of the target model. As an example, the input data is data regarding the physical state of a subject during exercise. At least a portion of the data regarding the subject's physical state is obtained by analyzing the subject's physical state with reference to the subject video data (or the subject video data and subject depth data).

[0057] The subject video data is data related to a video of a subject during exercise. The subject video data can be obtained, for example, by capturing an external image (e.g., the whole body) of a subject undergoing an exhaled gas test (e.g., a CPX test) from the front or diagonally in front (e.g., 45 degrees forward) using a camera (e.g., a camera mounted on a smartphone).

[0058] The subject depth data is data on the distance (depth) from the depth sensor to each part of the subject during exercise. The subject depth data can be acquired by operating the depth sensor when shooting the subject video.

[0059] The subject may be the same person as the user whose exercise tolerance is estimated when the information processing system 1 is in operation, or may be a different person. By making the subject and the user the same person, the target model may learn the user's personality and improve the estimation accuracy. On the other hand, allowing the subject to be a different person from the user has the advantage of making it easier to enrich the training data set. Furthermore, the subjects may be composed of multiple people, including the user, or multiple people excluding the user.

[0060] In the example of FIG. 6, the input data includes skeletal data, facial expression data, skin color data, respiration data, and heart rate data.

[0061] Skeletal data is data (e.g., features) related to the subject's skeleton during exercise. Skeletal data includes, for example, data related to the speed or acceleration of each part of the subject (which may include data related to changes in the parts of the muscles used by the subject or data related to the subject's perceived shaking). Skeletal data can be obtained by analyzing the subject's skeleton during exercise with reference to subject video data (or subject video data and subject depth data). For example, Vision, an SDK for iOS (registered trademark) 14, or other skeletal detection algorithms can be used to analyze the skeleton. Alternatively, skeletal data for a training dataset can be obtained by, for example, having the subject exercise while wearing motion sensors on each part of their body.

[0062] Facial expression data is data (e.g., features) about a subject's facial expressions while exercising. Facial expression data can be analyzed by applying an algorithm or trained model to video data of the subject. Alternatively, facial expression data for a training dataset can be obtained by, for example, human labeling of the subject's videos.

[0063] Skin color data is data (e.g., features) about the skin color of a subject during exercise. Skin color data can be analyzed by applying an algorithm or trained model to video data of the subject. Alternatively, skin color data for a training dataset can be obtained by, for example, human labeling of the subject's video.

[0064] The respiratory data is data (e.g., feature quantities) related to the subject's breathing during exercise. The respiratory data relates to, for example, the number of breaths per unit time or the breathing pattern. The breathing pattern may include at least one of the following: Ventilation rate Ventilation volume Ventilation rate (i.e., the amount of ventilation per unit time, or the number of ventilations) Ventilation acceleration (i.e., the time derivative of the ventilation rate) Carbon dioxide emission concentration Carbon dioxide emissions (VCO2) Oxygen intake concentration Oxygen uptake (VO2) The data relating to the breathing pattern may include data that can be calculated based on a combination of the above data, such as the gas exchange ratio R (=VCO2 / VO2).

[0065] The respiration data can be obtained by, for example, analyzing the skeletal data. As an example, the following items can be analyzed from the skeletal data: Movement (expansion) of the shoulders, chest (which may include the lateral chest), abdomen, or a combination thereof Inhalation time Exhalation time - Use of accessory respiratory muscles

[0066] The respiratory data for the training dataset can be obtained, for example, from the results of a test on exhaled gases performed on a subject while exercising. Details of exhaled gas tests that can be performed on a subject while exercising will be described later. Alternatively, the respiratory data for the training dataset, such as the ventilation rate, ventilation volume, ventilation rate, or ventilation acceleration, can be obtained, for example, from the results of a respiratory function test (e.g., a pulmonary function test or a spirometry test) performed on the subject while exercising. In this case, the respiratory function test can be performed using commercially available testing equipment, not limited to medical equipment.

[0067] Heart rate data is data (e.g., feature values) related to the heart rate of a subject during exercise. Heart rate data can be obtained, for example, by analyzing subject video data or the analysis results thereof (e.g., skin color data). Alternatively, heart rate data for the training dataset may be obtained, for example, from the results of a test on exhaled gas along with respiratory data, which will be described later. Subject heart rate data for the training dataset can also be obtained by having the subject perform the above exercise while wearing a heart rate sensor or electrodes for an electrocardiogram monitor.

[0068] The correct answer data is data corresponding to the correct answer for the corresponding input data (example question). The target model is trained (supervised learning) to output the input data in a way that is closer to the correct answer data. As an example, the correct answer data includes at least one of an exercise tolerance evaluation index, an index showing the relationship between the evaluation index and exercise load, or an index used to determine the exercise tolerance evaluation index. The anaerobic threshold (AT) and maximal oxygen uptake (Peak VO2) are examples of exercise tolerance evaluation indexes. Cardiopulmonary exercise load is an example of an index showing the relationship between an exercise tolerance evaluation index and exercise load. For example, cardiopulmonary exercise load can be calculated as the ratio of the (real-time) exercise load to the maximal oxygen uptake.

[0069] The correct answer data can be obtained, for example, from the results of a test on exhaled gases conducted on a subject during exercise. A first example of a test on exhaled gases is a test (typically a CPX test) conducted while a subject wearing an exhaled gas analyzer is performing exercise with a gradually increasing load (e.g., on an ergometer). A second example of a test on exhaled gases is a test conducted while a subject wearing an exhaled gas analyzer is performing exercise with a constant or variable load (e.g., bodyweight exercise, gymnastics, strength training).

[0070] Alternatively, the correct data can be obtained from the results of tests other than exhaled gas tests conducted on the exercising subject. Specifically, the correct data can be obtained from the results of a cardiopulmonary exercise load prediction test based on the lactate concentration measurement in the sweat or blood of the exercising subject. A wearable lactate sensor can also be used to measure the lactate concentration of the subject.

[0071] Exercise stress is an index for quantitatively evaluating the stress of exercise. Exercise stress can be expressed numerically using at least one of the following: Energy (calorie) consumption Oxygen consumption Heart rate

[0072] (4) Estimation model The estimation model used by the server 30 corresponds to a trained model created by supervised learning using a training dataset (FIG. 6), or a derived model or distilled model of the trained model.

[0073] (5) Information processing Information processing of this embodiment will be described. Fig. 7 is a flowchart of the information processing of this embodiment. Fig. 8 is a diagram showing an example of a screen displayed in the information processing of this embodiment. Fig. 9 is a diagram showing an example of a screen displayed in the information processing of this embodiment. Fig. 10 is a diagram showing an example of a screen displayed in the information processing of this embodiment.

[0074] The information processing starts when, for example, any of the following start conditions is met. - Information processing is invoked by another process. The user performed an operation to call up information processing. The client device 10 enters a predetermined state (for example, a predetermined application is started). The appointed date and time has arrived. A certain amount of time has passed since a certain event.

[0075] As shown in FIG. 7, the client device 10 performs sensing (S110). Specifically, the client device 10 enables the operation of the camera 16 to start capturing video of the user exercising (hereinafter referred to as "user video"). The client device 10 also enables the operation of the depth sensor 17 to start measuring the distance from the depth sensor 17 to each part of the user exercising (hereinafter referred to as "user depth"). The client device 10 also causes the wearable device 50 to start measuring the heart rate (hereinafter referred to as "user heart rate") using the heart rate sensor 56.

[0076] After step S110, the client device 10 executes data acquisition (S111). Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 acquires user video data from the camera 16, user depth data from the depth sensor 17, and user heart rate data from the wearable device 50.

[0077] After step S111, the client device 10 executes the request (S112). Specifically, the client device 10 references the data acquired in step S111 and generates a request. The client device 10 transmits the generated request to the server 30. The request may include, for example, at least one of the following: Data acquired in step S111 (e.g., user video data, user depth data, or user heart rate data) Data obtained by processing the data acquired in step S111 User data (for example, skeletal data, facial expression data, skin color data, breathing data, or a combination thereof) acquired by analyzing the user video data (or the user video data and the user depth data) acquired in step S111

[0078] After step S112, the server 30 performs estimation regarding exercise tolerance (S130). Specifically, the server 30 acquires input data for the estimation model based on a request received from the client device 10. The input data includes user data (skeleton data, facial expression data, skin color data, respiratory data, heart rate data, or a combination thereof) as well as training data. The server 30 estimates exercise tolerance by applying the estimation model to the input data. As an example, the server 30 estimates at least one of the following: Exercise tolerance index (e.g., anaerobic threshold or maximum oxygen uptake) The relationship between the user's (real-time) exercise load and the estimated evaluation index (e.g., magnitude relationship, difference, ratio (i.e., cardiopulmonary exercise load), or a combination thereof)

[0079] After step S130, the server 30 executes a response (S131). Specifically, the server 30 generates a response based on the result of the estimation in step S130. The server 30 transmits the generated response to the client device 10. As an example, the response may include at least one of the following: Data equivalent to the estimated results of exercise tolerance Data processed from the results of the estimation of exercise tolerance (for example, data on a screen to be displayed on the display 15 of the client device 10, or data referenced to generate that screen)

[0080] After step S131, the client device 10 executes information presentation (S113). Specifically, the client device 10 causes the display 15 to display information based on the response acquired from the server 30 (that is, the result of the estimation of the exercise tolerance of the user). However, the information may also be presented to the user's instructor (eg, medical professional or trainer) on a terminal used by the instructor instead of or in addition to the user.

[0081] In a first example of information presentation (S113), the client device 10 displays a screen P10 (FIG. 8) on the display 15. The screen P10 includes display objects A10a to A10b. The display object A10a displays information indicating an evaluation index of the exercise tolerance of the user. In the example of Fig. 8, the display object A10a displays the result of converting the anaerobic threshold of the user into METS. The display object A10b displays information about activities recommended to the user. The activities recommended to the user are activities that are different from the exercise performed by the user and have a load corresponding to the evaluation index of the user's exercise tolerance. The load corresponding to the evaluation index of the user's exercise tolerance is, for example, the same load as the evaluation index of the exercise tolerance, or a load slightly higher / lower than the evaluation index of the exercise tolerance. The activities recommended to the user may be selected based on the user's attributes (e.g., preferences). In the example of FIG. 8, the display object A10b displays three activities selected from activities having loads corresponding to the user's anaerobic metabolic threshold.

[0082] In a second example of information presentation (S113), the client device 10 displays a screen P11 (FIG. 9) on the display 15. The screen P11 includes display objects A11a to A11b. The display object A11a displays information indicating an evaluation index of the exercise tolerance of the user. In the example of Fig. 9, the display object A11a displays the result of converting the user's maximum oxygen uptake into METS and the change in the result from the previous time. The display object A11b displays a graph showing the change over time of the result of converting the evaluation index of the user's exercise tolerance into METS. In the example of Fig. 9, the display object A11b displays a graph showing the change over time of the result of converting the user's maximum oxygen uptake into METS.

[0083] In a third example of information presentation (S113), the client device 10 displays screens P12a to P12c (FIG. 10) on the display 15. The client device 10 displays screens P12a to P12c on the display 15 while the user is exercising. The screens P12a to P12c include display objects A12a to A12e. The display object A12a is an object that displays the remaining time of the exercise or the duration of the exercise. The length of the exercise time may be fixed or variable. The length of the exercise time may be set by the user, by the user's instructor, or by an administrator of the app executed by the client device 10. In the example of FIG. 10, the display object A12a displays a timer that indicates the remaining time of the exercise. The display object A12b displays a score that evaluates the quality of the user's exercise. As an example, the score is a cumulative value of points that is awarded so that the smaller the difference between the user's exercise load and the evaluation index of exercise tolerance, the higher the score. The display object A12c displays information relating to the relationship between the user's exercise load and the evaluation index of the user's exercise tolerance. In the example of Fig. 10, the display object A12c displays whether the user's exercise load is above or below the anaerobic metabolic threshold using an animation of a balance. The display object A12d displays information relating to the relationship between the user's exercise load and the evaluation index of the user's exercise tolerance. In the example of Fig. 10, the display object A12d displays, by the expression of an icon, whether the user's exercise load is above or below the anaerobic metabolic threshold. The display object A12e displays information regarding guidelines for adjusting the exercise load of the user. In the example of Fig. 10, the display object A12e displays a comment informing the user whether the exercise load of the user should be increased, maintained, or decreased.

[0084] After step S113, the client device 10 ends the information processing (FIG. 7). However, if the estimation of the user's exercise tolerance is performed in real time while the user is exercising, the client device 10 may return to acquiring data (S111) after step S113.

[0085] (6) Summary As described above, the information processing system 1 according to the embodiment estimates the exercise tolerance of a user based on a video (or video and depth) of the user exercising and the user's heart rate. This makes it possible to evaluate the exercise tolerance of the user without imposing a burden on the user, such as wearing special equipment.

[0086] The information processing system 1 may estimate the exercise tolerance of a user by applying an estimation model to input data based on a video (or video and depth) of the user exercising and their heart rate. This allows for a statistical estimation of the user's exercise tolerance in a short time. Furthermore, the estimation model may correspond to a trained model created by supervised learning using the training dataset (FIG. 6) described above, or a derived model or distilled model of the trained model. This allows for efficient construction of the estimation model. Furthermore, the input data to which the estimation model is applied may include user data related to the user's physical condition during exercise. This allows for improved accuracy of the estimation model. The user data may include data related to at least one of the user's bone structure, facial expression, skin color, breathing, or heart rate during exercise. This allows for improved accuracy of the estimation model. Furthermore, the subject may be the same person as the user. This allows for highly accurate estimation using a model that has learned the user's personality.

[0087] The information processing system 1 may estimate at least one of the user's anaerobic threshold, maximum oxygen uptake, or real-time cardiopulmonary exercise load, thereby enabling appropriate evaluation of the user's exercise tolerance or the user's real-time exercise load.

[0088] The information processing system 1 may present information based on the results of the estimation of the user's exercise tolerance. This allows the user or their instructor to receive advice based on the user's exercise tolerance. As a first example, the information processing system 1 may present an evaluation index of the user's exercise tolerance. This allows the recipient of the information to understand the user's current exercise tolerance. As a second example, the information processing system 1 may present information about activities other than the exercise the user has performed, which have a load corresponding to the evaluation index of the user's exercise tolerance. This increases the user's freedom of activity and improves their motivation to continue rehabilitation or fitness. As a third example, the information processing system 1 may present information about changes over time in the evaluation index of the user's exercise tolerance. This allows the user to sense the growth or decline of their exercise tolerance and improves their motivation to continue rehabilitation or fitness. As a fourth example, the information processing system 1 may present information regarding guidelines for adjusting the user's exercise load based on the results of the estimation of exercise tolerance. This allows the user to optimize the load by following the instructions presented during exercise. As a fifth example, information regarding the relationship between the user's exercise load and the estimated results of the evaluation index of the user's exercise tolerance may be presented, allowing the user to adjust and optimize the load themselves during exercise.

[0089] (7) Variation 1 A description will be given of Modification 1. Modification 1 is an example in which input data for an estimation model is modified.

[0090] (7-1) Overview of Modification 1

[0091] An overview of Modification 1 will be described. In this embodiment, an example has been shown in which an estimation model is applied to input data based on a user's video. In Modification 1, by applying the estimation model to input data based on both the user's video and the user's health status, it is also possible to estimate the exercise tolerance of the user.

[0092] The health condition includes at least one of the following: ·age ·sex ·height ·body weight ·Body fat percentage Muscle mass ·Bone density History of current illness ·Past history ·Medication history Surgical history Life history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty score, etc.) Family history Respiratory function test results Test results other than respiratory function tests (e.g., blood tests, urine tests, electrocardiograms (including Holter ECGs), cardiac ultrasounds, X-rays, CT scans (including cardiac morphological CT and coronary artery CT), MRI scans, nuclear medicine scans, PET scans, etc.) Data obtained during cardiac rehabilitation (including Borg index)

[0093] (7-2) Training Dataset A description will be given of the teacher dataset of Modification 1. Fig. 11 is a diagram showing the data structure of the teacher dataset of Modification 1.

[0094] 11, the training data set of the first modification includes a plurality of training data. The training data is used for training or evaluating a target model. The training data includes a sample ID, input data, and correct answer data.

[0095] The sample ID and the correct answer data are as described in this embodiment.

[0096] The input data is data that is input to the target model during training or evaluation. The input data corresponds to example problems used during training or evaluation of the target model. As an example, the input data is data regarding the subject's physical state during exercise (i.e., relatively dynamic data) and data regarding the subject's health state (i.e., relatively static data). The data regarding the subject's physical state is as described in this embodiment.

[0097] Data on the subject's health condition can be obtained in various ways. The data on the subject's health condition may be obtained before, during, or after the subject's exercise. The data on the subject's health condition may be obtained based on a report from the subject or their doctor, by extracting information linked to the subject in a medical information system, or via the subject's app (e.g., a healthcare app).

[0098] (7-3) Estimation model In the first modification, the estimation model used by the server 30 corresponds to a trained model created by supervised learning using the training data set (FIG. 11), or a derived model or distilled model of the trained model.

[0099] (7-4) Information Processing The information processing of the first modification will be described with reference to FIG.

[0100] In the first modification, the client device 10 performs sensing (S110) in the same manner as in FIG.

[0101] After step S110, the client device 10 executes data acquisition (S111). Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 acquires user video data from the camera 16, user depth data from the depth sensor 17, and user heart rate data from the wearable device 50.

[0102] Furthermore, client device 10 acquires data related to the user's health condition (hereinafter referred to as "user health condition data"). For example, client device 10 may acquire the user health condition data based on an operation (declaration) by the user or the user's doctor, or may acquire the user health condition data by extracting information linked to the user in a medical information system, or may acquire the user health condition data via the user's app (e.g., a healthcare app). However, client device 10 may acquire the user health condition data at a timing different from step S111 (e.g., before step S110, at the same timing as step S110, or after step S111).

[0103] After step S111, the client device 10 executes the request (S112). Specifically, the client device 10 references the data acquired in step S111 and generates a request. The client device 10 transmits the generated request to the server 30. The request may include, for example, at least one of the following: Data acquired in step S111 (e.g., user video data, user depth data, user heart rate data, or user health status data) Data obtained by processing the data acquired in step S111 User data (for example, skeletal data, facial expression data, skin color data, breathing data, or a combination thereof) acquired by analyzing the user video data (or the user video data and the user depth data) acquired in step S111

[0104] After step S112, the server 30 performs estimation regarding exercise tolerance (S130). Specifically, the server 30 acquires input data for the estimation model based on a request received from the client device 10. The input data includes user data (skeleton data, facial expression data, skin color data, respiratory data, heart rate data, or a combination thereof, and health condition data) as well as training data. The server 30 estimates exercise tolerance by applying the estimation model to the input data. As an example, the server 30 estimates at least one of the following: Exercise tolerance index (e.g., anaerobic threshold or maximum oxygen uptake) The relationship between the user's (real-time) exercise load and the estimated evaluation index (e.g., magnitude relationship, difference, ratio (i.e., cardiopulmonary exercise load), or a combination thereof)

[0105] After step S130, the server 30 executes a response (S131) ​​in the same manner as in FIG.

[0106] After step S131, the client device 10 performs information presentation (S113) in the same manner as in FIG.

[0107] (7-5) Summary As described above, the information processing system 1 of the first modification estimates the exercise tolerance of a user by applying an estimation model to input data based on both the user's video and the user's health condition. This allows for highly accurate estimation by further considering the user's health condition. For example, even if there is a difference between the user's health condition and the health condition of the subject from which the training data was derived, a valid estimation can be made.

[0108] (8) Other variations The storage device 11 may be connected to the client device 10 via a network NW. The display 15 may be built into the client device 10. The storage device 31 may be connected to the server 30 via the network NW.

[0109] The information processing system of the embodiment and the first modification has been described as being implemented as a client / server system. However, the information processing system of the embodiment and the first modification may also be implemented as a stand-alone computer. As an example, the client device 10 may independently perform estimation of exercise tolerance using an estimation model.

[0110] Each step of the above information processing can be performed by either the client device 10 or the server 30. As an example, the server 30, instead of the client device 10, may acquire at least a portion of the user data by analyzing the user video (or the user video and user depth).

[0111] In the above description, an example was shown in which the anaerobic threshold or maximum oxygen uptake was estimated as an index for evaluating exercise tolerance, and information on the results of the estimation was presented. However, instead of or in addition to these evaluation indices, estimation of other types of evaluation indices for exercise tolerance may be performed, and information on the results of the estimation may be presented.

[0112] In the above description, an example has been shown in which a screen based on a response from the server 30 is displayed on the display 15 of the client device 10. The screen based on the response from the server 30 may also be displayed on the display 55 of the wearable device 50.

[0113] In the above description, an example has been given in which a user video is captured using the camera 16 of the client device 10. However, the user video may be captured using a camera other than the camera 16. In the above description, an example has been given in which the user depth is measured using the depth sensor 17 of the client device 10. However, the user depth may be measured using a depth sensor other than the depth sensor 17.

[0114] In the above description, an example has been given in which the wearable device 50 measures the user's heart rate. However, the heart rate can also be obtained by analyzing video data or its analysis results (e.g., skin color data) (e.g., rPPG (Remote Photoplethysmography) analysis). The heart rate analysis may be performed using a trained model constructed using machine learning technology. Alternatively, the user may exercise while wearing electrodes for an electrocardiogram monitor, so that the electrocardiogram monitor can measure the user's heart rate. In these modified examples, the user does not need to wear the wearable device 50.

[0115] Instead of or in addition to the heart rate sensor 56, the wearable device 50 may be equipped with a sensor for measuring at least one of the following items: ·acceleration Blood sugar levels Oxygen saturation The measurement results from each sensor may be used as appropriate for generating input data, estimating exercise tolerance, presenting information based on the estimation results, or other purposes. For example, blood glucose level measurements may be referenced to evaluate exercise load converted into energy consumption or oxygen consumption. For another example, acceleration measurements may be used to determine a user's exercise (e.g., gymnastics) score.

[0116] The acceleration data may be used as part of the input data for the estimation model. Alternatively, the user's skeletal structure may be analyzed by referring to the acceleration data. The acceleration data may be obtained, for example, by having the user carry a client device 10 equipped with an acceleration sensor when recording the user's video.

[0117] Oxygen saturation data can also be used as part of the input data for the estimation model. For example, the oxygen saturation data can be obtained by having the user wear a pulse oximeter while recording the user's video. Alternatively, the oxygen saturation data can be estimated by performing rPPG analysis on the user's video data.

[0118] In the above description, an example was given in which information regarding a recommended activity was presented to the user. However, instead of or in addition to such information, information regarding the recommended exercise load may be presented to the user. As an example, information indicating the load (e.g., target speed, target number of rotations, etc.) corresponding to the evaluation index of the user's exercise tolerance for the same activity as the exercise performed by the user may be presented.

[0119] The information processing system 1 of this embodiment and Modification 1 can also be applied to a video game in which the progress of the game is controlled according to the player's physical movements. As an example, the information processing system 1 may estimate the user's exercise tolerance during game play and determine one of the following depending on the result of the estimation. This can enhance the effect that the video game has on improving the user's health. The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) provided to users The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) provided to users Game parameters related to the progression of a video game (e.g., score, damage)

[0120] A microphone on or connected to the client device 10 may receive sound waves emitted by the user during exercise (e.g., sounds associated with breathing or speaking) and generate sound data, which, together with the user data, may constitute input data for the estimation model.

[0121] In the above description, a CPX test is used as an example of a test related to exhaled gas. In a CPX test, a gradually increasing exercise load is applied to the test subject. However, it is not necessary to gradually increase the exercise load applied to the user when recording the user video. Specifically, the real-time cardiopulmonary exercise load can be estimated even when the user is subjected to a constant or constantly changeable exercise load. For example, the exercise performed by the user may be bodyweight exercise, calisthenics, or strength training.

[0122] In the first modification, an example was shown in which an estimation model was applied to input data based on a health condition. However, it is also possible to construct multiple estimation models based on (at least a part of) the subject's health condition. In this case, (at least a part of) the user's health condition may be referenced to select an estimation model. In this further modification, the input data for the estimation model may be data that is not based on the user's health condition, or may be data that is based on the user's health condition and a user video.

[0123] Although the embodiments and modifications of the present invention have been described in detail above, the scope of the present invention is not limited to the above-described embodiments and modifications. Furthermore, the above-described embodiments and modifications can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above-described embodiments and modifications can be combined. [Explanation of symbols]

[0124] 1: Information processing system 10: Client device 11:Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 30: Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface 50: Wearable devices 51 :Storage device 52: Processor 53: Input / output interface 54: Communication interface 55: Display 56: Heart rate sensor

Claims

1. Computer, means for acquiring a user video showing the user exercising; means for acquiring health status data relating to the health status of the user; A means for making an estimation regarding the exercise tolerance of the user based on the user video and the health condition data. A program that functions as a

2. The health status data includes at least one selected from the group consisting of current illness history, past medical history, oral medication history, surgical history, results of respiratory function tests, and data obtained during cardiac rehabilitation. The program according to claim 1.

3. The health condition data includes at least one selected from the group consisting of age, sex, height, weight, body fat percentage, muscle mass, bone density, and life history. The program according to claim 2.

4. the means for estimating exercise tolerance estimates the exercise tolerance of the user by applying an estimation model to input data based on the user video and the health condition data. The program according to claim 1.

5. The estimation model corresponds to a trained model created by supervised learning using a teacher dataset including input data including data on a subject video showing the subject exercising and data on the subject's health condition, and ground truth data associated with each of the input data, or a derived model or distilled model of the trained model. The program according to claim 4.

6. The estimation model is one estimation model selected from a plurality of estimation models constructed in advance based on the health condition data. The program according to claim 4.

7. the input data includes user data relating to the user's physical condition during the exercise; The user data is obtained by analyzing the user video. The program according to claim 4.

8. The user data includes data regarding at least one of the user's bone structure, facial expression, skin color, breathing, or heart rate during the exercise. The program according to claim 7.

9. the means for performing an estimation performs an estimation regarding at least one of the user's anaerobic threshold, maximum oxygen uptake, or real-time cardiopulmonary workload. The program according to claim 1.

10. and further causing the computer to function as a means for presenting information based on the result of the estimation regarding the exercise tolerance of the user. The program according to claim 1.

11. The presenting means presents an evaluation index of the exercise tolerance of the user. The program according to claim 10.

12. The presenting means presents information about an activity that has a load corresponding to the evaluation index of the exercise tolerance of the user and is different from the exercise performed by the user. The program according to claim 10.

13. The presenting means presents information regarding a change over time in the evaluation index of the exercise tolerance of the user. The program according to claim 10.

14. the presenting means presents information regarding a guideline for adjusting the exercise load of the user based on the result of the estimation regarding the exercise tolerance of the user. The program according to claim 10.

15. The presenting means presents information regarding a relationship between the exercise load of the user and an evaluation index of the exercise tolerance of the user. The program according to claim 10.

16. the acquiring means acquires a user video showing a user playing a video game in which the progress of the game is controlled according to the player's body movements; the estimation means estimates the exercise tolerance of the user while playing the video game; and further causing the computer to function as a means for determining at least one of a challenge or a reward related to the video game to be given to the user depending on the result of the estimation regarding the exercise tolerance of the user. The program according to claim 1.

17. means for acquiring a user video showing the user exercising; means for acquiring health status data relating to the health status of the user; means for making an estimation regarding the exercise tolerance of the user based on the user video and the health condition data; An information processing device comprising:

18. The computer acquiring a user video of the user exercising; acquiring health status data relating to a health status of the user; making an estimation regarding the exercise tolerance of the user based on the user video and the health condition data; A method comprising:

Images