Information processing device, method, program, and system

Abstract

To provide a technique capable of evaluating a symptom of abnormality of a user while suppressing a burden applied to the user.SOLUTION: A program of one embodiment causes a computer to function as: means for acquiring item information on a first item which is an exercise item selected from a plurality of exercise items by a user, a related party, or an algorithm; means for acquiring user movement information on the user's movement obtained by analyzing images photographing the user who executes an exercise of the first item; and means for evaluating the user about a symptom of a target disorder which is at least one of a cranial nerve disease or a peripheral nerve / muscle disorder associated with the first item, on the basis of at least the user movement information and characteristics on the symptoms of the target disorder.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, method, program, and system.

Background Art

[0002] For example, neurodegenerative diseases such as dementia may be slowed down or symptoms alleviated by early detection and initiation of treatment. Although the symptoms of neurodegenerative diseases are diverse, for example, the disease may affect the patient's walking and cause gait disorders.

[0003] Patent Document 1 describes a technical idea that aims to simply and quantitatively measure movement disorders that appear in at least one of posture, vibration, and walking among a patient's body movements for the automatic diagnosis of neurodegenerative diseases such as Parkinson's disease.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technical idea described in Patent Document 1, the patient is forced to wear a dozen or so acceleration sensors on the body for posture measurement. Therefore, it is difficult to say that it is realistic to routinely measure a patient's movement disorder according to this technical idea because the burden on the patient is large.

[0006] An object of the present disclosure is to provide a technique for evaluating signs of abnormalities in a user while suppressing the burden on the user.

Means for Solving the Problems

[0007] A program according to an aspect of the present disclosure causes a computer to function as means for acquiring event information regarding a first event, which is an event selected by a user, a person concerned, or an algorithm from a plurality of sports events; means for acquiring user motion information regarding the motion of a user obtained by analyzing an image of the user performing the first event; and means for evaluating the user based on at least the user motion information and characteristics regarding symptoms of a target disease, which is at least one of a cerebrovascular disease or a peripheral nerve / muscle disease associated with the first event.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Embodiments for Carrying Out the Invention

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally denoted by the same reference numerals, and the repeated description thereof will be omitted.

[0010] (1) Configuration of the information processing system The configuration of the information processing system will be described. FIG. 1 is a block diagram showing the configuration of the information processing system of the present embodiment.

[0011] As shown in FIG. 1, the information processing system 1 includes a client device 10 and a server 30.

[0012] Here, the number of client devices 10 varies depending on, for example, the number of users. Therefore, the number of client devices 10 may be two or more. Further, the terminals of the relatives of each user may also be included in the information processing system 1.

[0013] Relatives can include, for example, at least one of the following. · The user's family · Those who planned / perform the user's exercise therapy (typically, medical staff (e.g., doctors (including the user's attending doctor), nurses, pharmacists, physical therapists, occupational therapists, clinical laboratory technicians), or dietitians) · Those who guide the user's exercise therapy (typically, medical staff, dietitians, or trainers)

[0014] The client device 10 and the server 30 are connected via a network (e.g., the Internet or an intranet) NW.

[0015] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0016] Server 30 is an example of an information processing device that provides a response corresponding to a request transmitted from client device 10 to client device 10. Server 30 is, for example, a server computer.

[0017] (1-1) Configuration of the client device The configuration of the client device will be described. FIG. 2 is a block diagram showing the configuration of the client device of the present embodiment.

[0018] As shown in FIG. 2, client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. Client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0019] Storage device 11 is configured to store programs and data. Storage device 11 is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage (e.g., flash memory or hard disk).

[0020] The program includes, for example, the following programs. · Program of the OS (Operating System) · Program of an application that executes information processing (e.g., a web browser, a therapeutic application, a rehabilitation application, or a fitness application) Here, the diseases targeted by the therapeutic application or the rehabilitation application are diseases in which exercise may contribute to the improvement of symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

[0021] The data includes, for example, the following data. · Database referred to in information processing · Data obtained by executing information processing (i.e., the execution result of information processing)

[0022] Processor 12 is a computer that realizes the functions of client device 10 by starting the program stored in storage device 11. Processor 12 is, for example, at least one of the following. ·CPU (Central Processing Unit) ·GPU (Graphic Processing Unit) ·ASIC (Application Specific Integrated Circuit) ·FPGA (Field Programmable Gate Array)

[0023] Input / output interface 13 is configured to acquire information (for example, user instructions, images, sounds) from an input device connected to client device 10 and output information (for example, images, commands) to an output device connected to client device 10.

[0024] The input device is, for example, camera 16, depth sensor 17, microphone 18, acceleration sensor 19, keyboard, pointing device, touch panel, sensor, or a combination thereof. The output device is, for example, display 15, speaker, or a combination thereof.

[0025] Communication interface 14 is configured to control communication between client device 10 and an external device (for example, another client device 10 (as an example, a terminal of a person concerned), server 30). Specifically, communication interface 14 can include a module for communication (for example, a WiFi module, a mobile communication module, a Bluetooth (registered trademark) module, or a combination thereof).

[0026] Display 15 is configured to display an image (a still image or a moving image). Display 15 is, for example, a liquid crystal display or an organic EL display.

[0027] The camera 16 is configured to perform imaging and generate an image signal.

[0028] The depth sensor 17 is, for example, LIDAR (Light Detection And Ranging). The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to surrounding objects (e.g., the user). Note that the depth sensor 17 is not essential and may be removed from the client device 10.

[0029] The microphone 18 is configured to receive sound waves and generate a sound signal. The microphone 18 is preferably installed near the user's body (especially the respiratory organ), for example, like an earphone microphone.

[0030] The acceleration sensor 19 is configured to detect acceleration.

[0031] (1-2) Configuration of the server The configuration of the server will be described. FIG. 3 is a block diagram showing the configuration of the server according to the present embodiment.

[0032] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface 34.

[0033] The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of a ROM, a RAM, and a storage.

[0034] The program includes, for example, the following programs. · Program of the OS · Program of an application that executes information processing

[0035] The data includes, for example, the following data. · Database referred to in information processing · Execution result of information processing

[0036] The processor 32 is a computer that realizes the functions of the server 30 by starting the program stored in the storage device 31. The processor 32 is, for example, at least one of the following. · CPU · GPU · ASIC · FPGA

[0037] The input / output interface 33 is configured to acquire information (for example, a user's instruction) from an input device connected to the server 30 and output the information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0038] The communication interface 34 is configured to control communication between the server 30 and an external device (for example, the client device 10).

[0039] (2) One aspect of the embodiment One aspect of this embodiment will be described. FIG. 4 is an explanatory diagram of one aspect of this embodiment.

[0040] As shown in FIG. 4, the client device 10 senses the user US1 during exercise. Although not shown, a wearable device equipped with, for example, an acceleration sensor may also sense the user US1 during exercise. The user US1 is typically a person receiving physical therapy and is, for example, a participant in a (cardiac) rehabilitation program or an exercise guidance program, but is not limited thereto.

[0041] User US1 is performing a type of exercise (hereinafter referred to as the "first type") selected from a plurality of available exercise types by himself / herself, a related person, or an algorithm. Here, the available exercise types are exercise types associated with any disease in the storage device 31 of the server 30 or another storage device accessible by the server 30.

[0042] The diseases associated with the exercise types may be neurological diseases, peripheral nerve and muscle diseases, or a combination thereof. Neurological diseases refer to neurological diseases mainly involving intracranial diseases, and can include, for example, at least one of dementia (such as Alzheimer's type, vascular, or Lewy body type, etc.), normal pressure hydrocephalus, or cerebellar diseases. Peripheral nerve and muscle diseases can include, for example, at least one of Parkinson's disease, myositis, muscular dystrophy, or Charcot-Marie-Tooth disease.

[0043] As an example, the camera 16 captures the appearance (e.g., the whole body) of the exercising user US1 from the front or obliquely in front at a distance of about 2 m. The camera 16 may be installed at an appropriate height by a tripod or other height adjustment means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. It is also possible to generate three-dimensional video data by combining, for example, the video data (two-dimensional) generated by the camera 16 and the depth data generated by the depth sensor 17. The microphone 18 receives the sound emitted from the exercising user US1 (e.g., the sound generated by breathing or vocalization) and generates a sound signal. The acceleration sensor 19 measures the acceleration during the movement of the user US1.

[0044] The client device 10 acquires various sensing data and performs analysis as necessary. As an example, the client device 10 may refer to the video data acquired from the camera 16 and analyze the movement of the body of the user US1 during exercise (in particular, the movement of the skeleton or other feature points over multiple time points, or the state of the skeleton or other feature points at a single time point). The client device 10 may further refer to the depth data acquired from the depth sensor 17 in order to analyze the movement of the body of the user US1 during exercise. The client device 10 transmits user data including at least one of the sensing data or the analysis result of the sensing data to the server 30. However, part or all of such analysis may be performed by the server 30.

[0045] Based on the user data acquired from the client device 10, the server 30 evaluates the user US1 for at least one of the cranial nerve diseases or peripheral nerve / muscle diseases associated with the first category (hereinafter referred to as "target diseases"). Specifically, the server 30 performs the evaluation based on the characteristics related to the symptoms of the target disease (for example, a specific type of movement disorder) and the analysis result of the movement of the user's body.

[0046] The server 30 generates information based on the result of the evaluation and presents it to the user US1 via the client device 10. Alternatively, the server 30 may present it to the above-mentioned related person on behalf of the user US1. Thereby, when it is evaluated that the user US1 has signs of the target disease, for example, it is possible to encourage the user US1 to visit a medical institution or undergo an examination. In addition, according to the present embodiment, since the user US1 is not forced to wear a large number of sensors (for example, attach acceleration sensors all over the body) or to perform a very limited type of exercise (for example, only walking), the burden on the user US1 is light. Therefore, such an evaluation can be easily performed daily (for example, as part of physical therapy) and can contribute to the early detection of the target disease.

[0047] (3) Database The database of this embodiment will be described. The following database is stored in the storage device 31 or another storage device accessible by the server 30.

[0048] (3-1) Sports event database The sports event database of this embodiment will be described. FIG. 5 is a diagram showing the data structure of the sports event database of this embodiment.

[0049] Sports event information is stored in the sports event database. The sports event information is information regarding a sports event (for example, the available sports events described above). Here, in this embodiment, the sports events include sports events that can be implemented without using a device capable of adjusting the exercise load, such as gymnastics, bodyweight training, dance, walking, running, treadmill, etc. Since these sports events include events performed in a standing position, they are rich in variations. Further, in this embodiment, the exercise load of these sports events can be adjusted through form (for example, the range of motion of a part, the degree of opening of the arms or legs, etc.), pace, number of reps, or the time or number of breaks. However, the sports events of this embodiment can further include sports events that are implemented using a device capable of adjusting the exercise load, such as an ergometer or strength training using training equipment.

[0050] As shown in FIG. 5, the sports event database includes an "ID" field, a "name" field, and an "exercise load amount" field. Each field is associated with each other.

[0051] A sports event ID is stored in the "ID" field. The sports event ID is information for identifying the sports event corresponding to the record.

[0052] Sports event name information is stored in the "name" field. The sports event name information is information regarding the name of the sports event corresponding to the record.

[0053] In the "exercise load" field, exercise load information is stored. The exercise load information is information regarding the standard exercise load of the exercise type corresponding to the record. The standard load is, for example, information regarding the exercise load when a person with standard physical functions performs the corresponding exercise type. As an example, such exercise load may be derived by, for example, actually measuring the exercise load when the corresponding exercise type is performed on a plurality of people and statistically processing (e.g., averaging) the measurement results, or may be obtained by referring to the exercise load set for the exercise type by a third-party institution. The exercise load information may be managed in units finer than the exercise type. As an example, the exercise load information may be managed for each variation such as the form, pace, number of reps, or rest time or number of times of each exercise type. Note that the "exercise load" field is not essential and may be removed from the exercise type database.

[0054] (3-2) Model Database The model database of the present embodiment will be described. FIG. 6 is a diagram showing the data structure of the model database of the present embodiment.

[0055] Model information is stored in the model database. The model information is information regarding a learned model that makes inferences about signs of one or more diseases based on at least input data related to body movements (e.g., skeleton).

[0056] As shown in FIG. 6, the model database includes a "model ID" field, a "model details" field, and an "application conditions" field. Each field is associated with each other.

[0057] A model ID is stored in the "model ID" field. The model ID is information for identifying the learned model corresponding to the record.

[0058] In the "Model Details" field, model detail information is stored. The model detail information is information regarding the details of the learned model corresponding to the relevant record. The detail information includes information capable of specifying the structure of the learned model (e.g., the layer structure in a neural network, the connection relationship between nodes, and the weights of each edge). The detail information may include, for example, information indicating the values of each parameter defining the structure of the learned model, or may include information indicating the location where the values are stored.

[0059] In the "Applicable Conditions" field, applicable condition information is stored. The applicable condition information is information regarding the conditions under which the learned model corresponding to the relevant record is applicable. As an example, the applicable condition information can include information capable of specifying the type of exercise (e.g., the aforementioned exercise ID) and information capable of specifying a disease (i.e., the target disease). As information capable of specifying a disease, for example, a disease ID that uniquely identifies a single disease or a group of diseases consisting of a plurality of diseases can be defined.

[0060] (3-3) Other Databases Databases other than the above may be constructed. As an example, a user profile database will be described.

[0061] In the user profile database, user profile information is stored. The user profile information is information regarding the profile of a user of the information processing system 1 (e.g., a person receiving physical therapy).

[0062] The user profile database may store records including at least one of the following information, for example. · User ID · User name information · Physical information · Information indicating an appropriate exercise load for the user · Information indicating the user's relatives

[0063] The user ID is information that identifies the user corresponding to the relevant record. The user name information is information regarding the name of the user corresponding to the record (for example, name, account name, etc.). The physical information is information regarding the body (function) of the user corresponding to the record. As an example, the physical information may include information regarding the user's age, gender, weight, height, disease, etc.

[0064] (4) Information processing The information processing of this embodiment will be described.

[0065] (4-1) Symptom evaluation processing The symptom evaluation processing of this embodiment will be described. FIG. 7 is a flowchart of the symptom evaluation processing of this embodiment. FIG. 8 is a diagram showing an example of a screen displayed in the symptom evaluation processing of this embodiment. FIG. 9 is a diagram showing an example of a screen displayed in the symptom evaluation processing of this embodiment. FIG. 10 is a diagram showing an example of a screen displayed in the symptom evaluation processing of this embodiment.

[0066] The symptom evaluation processing starts, for example, in response to the establishment of any of the following start conditions. · The symptom evaluation processing is called by other processing. · The user, or a person related to the user, has performed an operation to call the symptom evaluation processing. · The client device 10 has entered a predetermined state (for example, the startup of a predetermined application). · A predetermined date and time has arrived. · A predetermined time has elapsed since a predetermined event.

[0067] As shown in FIG. 7, the client device 10 executes the acquisition of sensing data (S110). Specifically, the client device 10 may start capturing a video of a user in motion (hereinafter referred to as "user video") by enabling the operation of the camera 16. Further, the client device 10 may start measuring the distance from the depth sensor 17 to each part of the user in motion (hereinafter referred to as "user depth") by enabling the operation of the depth sensor 17. The client device 10 may start collecting sound (e.g., sound generated by the user's voice or breathing (hereinafter referred to as "user sound")) by enabling the operation of the microphone 18. The client device 10 may start measuring acceleration by enabling the operation of the acceleration sensor 19.

[0068] Then, the client device 10 acquires sensing data from each sensor. Specifically, the client device 10 acquires the sensing results generated by the various sensors enabled in step S110. For example, the client device 10 may acquire user video data from the camera 16, acquire user depth data from the depth sensor 17, acquire user sound data from the microphone 18, and acquire user acceleration data regarding the user's acceleration (hereinafter referred to as "user acceleration") from the acceleration sensor 19.

[0069] After step S110, the client device 10 executes generation of user data (S111). Specifically, the client device 10 generates user data based on the sensing data acquired in step S110. The user data can include at least one of the following. · Data acquired in step S111 (e.g., user video data, user depth data, user sound data, or user acceleration data) · Data obtained by processing the data acquired in step S111 · Data obtained by analyzing the data acquired in step S111 (e.g., movement information (such as skeleton data as an example), facial expression data, gaze data, tremor data, assistive device data, voice data, or a combination thereof) · Information capable of identifying the type of exercise (first type) the user was performing in step S110

[0070] After step S111, the client device 10 executes the transmission of user data (S112). Specifically, the client device 10 transmits the user data generated in step S111 to the server 30.

[0071] After step S112, the server 30 executes the evaluation of signs (S130). Specifically, the server 30 receives the user data transmitted by the client device 10 in step S112. The server 30 acquires user skeleton information based on the user data obtained from the client device 10. The user skeleton information is information regarding the skeleton of the user obtained by analyzing the user video data. In addition to the user video data, user depth data may be referred to for generating the user skeleton information. The user skeleton information may be included in the user data, or may be generated by the server 30 analyzing the data included in the user data. Note that the user skeleton information is an example of information regarding the movement of the user's body during exercise (hereinafter referred to as "user movement information"), and may be used together with information regarding the movement of other feature points, or may be replaced with this.

[0072] The user skeleton information is information regarding the skeleton of the user during exercise (such as data such as feature amounts). The user skeleton information includes, for example, information regarding the position, speed, or acceleration of each part of the user's body (which may include information regarding changes in the parts of the muscles used by the user, or information regarding the sway of the user's torso). The user skeleton information can be obtained by analyzing the skeleton of the user during exercise with reference to the user video data (or the user video data and the user depth data). As an example, Vision, which is an SDK of iOS (registered trademark) 14, or other skeleton detection algorithms (for example, OpenPose, PoseNet, MediaPipe Pose) can be used for the analysis of the skeleton.

[0073] Note that the results of motion detection such as skeleton detection can also be used for quantitative evaluation, qualitative evaluation, or a combination of these of the motion. As a first example, the results of motion detection can also be used for counting the number of reps. As a second example, the results of motion detection can be used for evaluating the form of the motion or the appropriateness of the load applied by the motion. For example, when the type of exercise is a squat, the results of motion detection can be used for evaluations such as whether the knees are protruding too far forward and in a dangerous form, or whether the user is squatting deeply enough and applying sufficient load.

[0074] Further, the server 30 acquires information related to the first type of exercise (for example, information that can identify the first type of exercise such as the exercise ID corresponding to the first type of exercise). Such information may be included in the user data. Alternatively, during the process in which the first type of exercise is selected by a person concerned or an algorithm, it may be transmitted from the terminal of the person concerned to the server 30, or may be generated by the server 30.

[0075] The server 30 evaluates the user based on at least the user motion information and the characteristics regarding the symptoms of the target disease, which is at least one of the cerebrovascular diseases or peripheral nerve / muscle diseases associated with the first type of exercise.

[0076] As a first example of the evaluation of signs (S130), the server 30 refers to the model database (Fig. 6) and extracts records including the value of the exercise ID corresponding to the first type of exercise as an application condition. In this case, the learned model corresponding to each record constituting the extraction result corresponds to the characteristics regarding the symptoms of the target disease. The server 30 evaluates the user regarding the signs of the target disease by applying such a learned model to the input data based on the user motion information.

[0077] The learned model focuses on the presence or absence of the posture or movement of characteristic skeletons or other characteristic points observed as symptoms of the corresponding disease, and makes inferences regarding the presence or absence of symptoms, or the degree of possibility of the presence of symptoms. The learned model receives input data based on movement information (e.g., skeleton information) and outputs the result of the evaluation. The result of the evaluation may be binary data indicating the presence or absence of symptoms of any disease, or may be multi-valued data representing the possibility of the presence of such symptoms in multiple levels. The learned model corresponds to a learned model created by supervised learning using a teacher dataset, or a derivative model or distilled model of the learned model.

[0078] As a second example of the symptom evaluation (S130), when the target disease is specified, the server 30 refers to the model database (Fig. 6) and extracts a record including the item ID corresponding to the first type as an application condition and the value of the disease ID corresponding to the specified target disease. In this case, the learned model corresponding to each record constituting the extraction result corresponds to the characteristics regarding the symptoms of the target disease. The server 30 evaluates the user regarding the symptoms of the target disease by applying such a learned model to the input data based on the user movement information. The target disease can be specified by the user, a person related to the user, or an algorithm. Information that can identify the specified target disease may be included in the user data, may be transmitted from the terminal of the related person to the server 30, or may be generated by the server 30.

[0079] After step S130, the server 30 executes information generation (S131). Specifically, the server 30 generates information to be presented to the user based on the result of the evaluation in step S130.

[0080] As a first example of information generation (S131), when an evaluation result indicating that there are signs of any target disease or the possibility of signs of any target disease exceeding a threshold is obtained (an example of "when the user's evaluation result satisfies a predetermined condition"), the server 30 may select an exercise type (hereinafter referred to as the "second type") recommended for the user to perform next, and generate information (for example, an exercise type ID) that can identify the selected second type. As an example, the server 30 may select, as the second type, a type other than the first type among the exercise types associated with the target disease evaluated as having signs or the possibility thereof exceeding the threshold in step S131. The server 30 may further select the second type in consideration of the exercise load amount suitable for the user and the exercise load amount of each exercise type. By sensing the user who is performing the second type recommended in this way again and performing the sign evaluation (S130) again, a more reliable evaluation result can be obtained. That is, it is possible to confirm whether the signs observed during the execution of the first type are just accidental or whether the same signs are observed regardless of the type.

[0081] As a second example of information generation (S131), when an evaluation result indicating that there are signs of any target disease or the possibility of signs of any target disease exceeding a threshold is obtained, the server 30 may generate information that can identify the medical institution associated with the target disease. As an example, for each target disease, information on medical institutions suitable for the diagnosis of the target disease is associated and pre-databaseized in advance, and the server 30 can generate information by referring to the database. The server 30 may narrow down the medical institutions in consideration of the user's place of residence or current location information in addition to the target disease.

[0082] As a third example of information generation (S131), when an evaluation result indicating that there are signs of any target disease or the possibility of having signs of any target disease exceeds a threshold is obtained, the server 30 may generate information recommending the implementation of an examination associated with the target disease (for example, a screening test such as the Hasegawa Dementia Scale). As an example, for each target disease, information on an examination suitable for the target disease is associated and pre-databaseized, and the server 30 can generate information by referring to the database.

[0083] As a fourth example of information generation (S131), when an evaluation result indicating that there are signs of any target disease or the possibility of having signs of any target disease exceeds a threshold is obtained, the server 30 may combine two or more of the above first to third examples. For example, the server 30 may perform the above first example one or more times, and when the total or ratio of the number of times an evaluation result indicating that there are signs of the target disease or the possibility thereof exceeds the threshold reaches a predetermined value, the server 30 may perform at least one of the second or third examples.

[0084] Note that when an evaluation result indicating that there are no signs of any target disease or the possibility of having signs of any target disease does not exceed the threshold is obtained, the server 30 may omit information generation (S131) to information presentation (S132).

[0085] After step S131, the server 30 executes information presentation (S132). Specifically, the server 30 transmits the information generated in step S131 to the client device 10. Note that the server 30 can also present information to the user's relatives in addition to the user or on behalf of the user. In this case, the server 30 may transmit the information generated in step S132 to the relative's terminal.

[0086] After step S132, the client device 10 executes screen display (S113). Specifically, the client device 10 receives the information transmitted by the server in step S131. The client device 10 displays on the display 21 a screen (not limited to the screen of the application, and may include notifications (notifications of the application, or messages such as e-mails and chats)) based on the received information.

[0087] As a first example of the screen display (S113), the client device 10 displays the screen of FIG. 8 on the display 21. The screen of FIG. 8 includes objects J20 to J22.

[0088] The object J20 displays the aforementioned second type of information. In addition, the object J20 receives a user instruction to start each exercise type or a user instruction to play the demonstration video of each exercise type. When the object J20 is selected, the client device 10 may re-execute the sign evaluation process of the present embodiment with the exercise type (second type) corresponding to the object J20 as a new first type, or may play the demonstration video of the exercise type.

[0089] The object J21 receives a user instruction to select an exercise type from other than the second type. When the object J21 is selected, the client device 10 may, for example, display a list of exercise types other than the second type and receive a user instruction to select an exercise type. In response to receiving such a user instruction, the client device 10 may play the demonstration video of the exercise type selected by the user, or may re-execute the sign evaluation process of the present embodiment with the exercise type as a new first type when the exercise type is associated with the target disease.

[0090] The object J22 receives a user instruction to end the exercise. When the object J22 is selected, the client device 10 ends the sign evaluation process of the present embodiment.

[0091] As a second example of the screen display (S113), the client device 10 displays the screen of FIG. 9 on the display 21. The screen of FIG. 9 includes objects J30 to J31.

[0092] The object J30 displays information recommending a medical institution visit. The object J31 receives a user instruction for viewing information regarding a medical institution associated with the target disease. In response to receiving such a user instruction, the client device 10 may place an object that displays information regarding a medical institution associated with the target disease on the screen, or may transition to a screen that displays such information.

[0093] As a third example of the screen display (S113), the client device 10 displays the screen of FIG. 10 on the display 21. The screen of FIG. 10 includes objects J40 to J41.

[0094] The object J40 displays information recommending the implementation of an examination regarding the target disease. The object J41 receives a user instruction for starting an examination associated with the target disease. In response to receiving such a user instruction, the client device 10 may place an object that displays information regarding an examination associated with the target disease (for example, questions included in the examination, or a link for accessing a website where the examination is received, etc.) on the screen, or may transition to a screen that displays such information.

[0095] After step S113, the client device 10 may end the symptom evaluation process (FIG. 7).

[0096] (5) Teacher dataset A teacher dataset that can be used for supervised learning to construct the learned model of the present embodiment will be described.

[0097] The teacher dataset includes a plurality of teacher data. The teacher data is used for training or evaluating a model to be learned (hereinafter referred to as the "target model"). The teacher data includes a sample ID, input data, and correct answer data.

[0098] The sample ID is information for identifying the teacher data.

[0099] The input data is data that is input to the target model during training or evaluation. The input data corresponds to an example used during training or evaluation of the target model. As an example, the input data is data related to the movement of the subject's body during exercise (i.e., while performing the exercise type corresponding to the target model) (i.e., relatively dynamic data), and data related to the subject's health status (i.e., relatively static data). At least a part of the data related to the movement of the subject's body is obtained by analyzing the movement of the subject's body with reference to subject video data (or subject video data and subject depth data).

[0100] The subject video data is data related to a subject video showing the subject during exercise. The subject video data can be obtained, for example, by photographing the appearance (e.g., the whole body) of the subject during an examination related to exhaled gas (e.g., a CPX examination) from the front or diagonally in front (e.g., 45 degrees forward) with a camera (e.g., a camera mounted on a smartphone).

[0101] The subject depth data is data related to the distance (depth) from a depth sensor to each part of the subject during exercise. The subject depth data can be obtained by operating the depth sensor when photographing the subject video.

[0102] Subjects typically include those diagnosed with a specific disease or its symptoms and those diagnosed without the disease or its symptoms. The diagnosis is usually made by a doctor, but it may be replaced by a determination by an algorithm (which may include a trained model). However, for the same subject, data collected when diagnosed with a specific disease or its symptoms and data collected when diagnosed without the disease or its symptoms can also be used as teacher data, respectively. Also, any of the subjects may be the same person as the user for whom an evaluation of the symptoms of a specific disease is performed during the operation of the information processing system 1.

[0103] In this example, the input data includes at least movement data. The movement data can include elements similar to the aforementioned user movement information (e.g., skeletal information) and can be obtained by analyzing the movement (e.g., skeleton) of the subject during movement with reference to the subject video data. In addition to the subject video data, at least one of the subject depth data or the acceleration data measured by a wearable device worn by the subject may be referred to for obtaining the movement data.

[0104] The correct data is data corresponding to the correct answer for the corresponding input data (example problem). The target model is trained (supervised learning) to produce an output closer to the correct data for the input data. As an example, the correct data represents the presence or absence of a specific disease or its symptoms.

[0105] The correct data represents, for example, the presence or absence of a specific disease or its symptoms for the subject from whom the corresponding input data was obtained. That is, the correct data corresponding to the input data obtained from a subject diagnosed with a specific disease or its symptoms has a value indicating the presence of the disease or its symptoms. On the other hand, the correct data corresponding to the input data obtained from a subject diagnosed without a specific disease or its symptoms has a value indicating the absence of the disease or its symptoms.

[0106] In addition to the motion data, at least one of the following may be added to the input data together with the motion data, or a trained model that performs the same inference on the input data including at least one of the following as elements may be constructed. In this case, the input data during the operation of the trained model will also include similar elements based on the user's sensing results. · Facial expression data (an example of "facial expression information") · Gaze data (an example of "gaze information") · Tremor data (an example of "tremor information") · Assistive device data (an example of "assistive device information") · Voice data (an example of "voice information") · Includes health status data (an example of "health status information").

[0107] Facial expression data is data (e.g., feature quantities) related to the facial expressions of a subject during exercise. Facial expression data can be analyzed by applying an algorithm or a trained model to the subject video data. Alternatively, facial expression data for a teacher dataset can be obtained, for example, by having a person who watches the subject video perform labeling.

[0108] Gaze data is data (e.g., feature quantities) related to the gaze of a subject during exercise. Gaze data can be analyzed by applying an algorithm or a trained model to the subject video data.

[0109] Tremor data is data (e.g., feature quantities) related to the tremor of a subject during exercise. Tremor data can be analyzed by applying an algorithm or a trained model to the subject video data. Alternatively, tremor data for a teacher dataset can be obtained, for example, by having a person who watches the subject video perform labeling.

[0110] The assistive device data is data (such as feature quantities) regarding how an assistive device (e.g., a walking assistive device such as a cane or other tools used in exercise) of a subject during exercise is used. The assistive device data can be analyzed by applying an algorithm or a trained model to the subject video data. Alternatively, the assistive device data for the teacher dataset can be obtained, for example, by a human who has watched the subject video for labeling.

[0111] The voice data is data (such as feature quantities) regarding the voice of a subject during exercise. The voice data can be analyzed by applying an algorithm or a trained model to the sound data collected by a microphone installed around the subject. Alternatively, the assistive device data for the teacher dataset can be obtained, for example, by a human who has listened to the subject's voice for labeling.

[0112] The health status data is data regarding the health status of the subject. The health status data can be obtained in various ways. The health status data of the subject may be obtained at any timing before, during, or after the subject's exercise. The health status data of the subject may be obtained based on a report from the subject or their attending doctor, or may be obtained by extracting information linked to the subject in a medical information system, or may be obtained via the subject's app (e.g., a healthcare app).

[0113] The health status includes at least one of the following. · Age · Gender · Height · Weight · Body fat percentage · Muscle mass · Bone density · Current medical history · Past medical history · Medication history · Surgical history · Life history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty score, etc.) · Family history · Results of pulmonary function tests · Test results other than respiratory function tests (e.g., blood tests, urine tests, electrocardiogram tests (including Holter electrocardiogram tests), echocardiograms, X-ray tests, CT scans (including cardiac morphology CT and coronary artery CT), MRI scans, nuclear medicine tests, PET tests, etc.) · Data obtained during cardiac rehabilitation (including Borg index)

[0114] Note that it is also possible to construct a learned model for each of a plurality of health state categories based on (at least a part of) the health state of the subject. In this case, (at least a part of) the user's health state may be referred to in order to select a learned model. In this variant, the input data of the learned model may be data not based on the user's health state, or data based on the user's health state and the user video.

[0115] (5) Parentheses As described above, the server 30 of the present embodiment acquires user motion information regarding the motion of a user who has performed a first type of exercise, which is an exercise selected from a plurality of exercise types by the user, a related person, or an algorithm, by analyzing an image of the user. The server 30 evaluates the user based on at least the user motion information and the characteristics (in this embodiment, a learned model) regarding the symptoms of a target disease, which is at least one of a cerebrovascular disease or a peripheral nerve / muscle disease associated with the first type of exercise. As a result, the burden on the user is suppressed, so that it becomes easier to perform an evaluation of the signs of the target disease on a daily basis, and it is possible to contribute to the early detection of the target disease.

[0116] The server 30 may evaluate the user for signs of the target disease by applying a learned model corresponding to the first type to input data based on the user movement information. As a result, a statistically valid evaluation result can be obtained without incorporating an evaluation algorithm. Also, by using a learned model corresponding to the first type, it becomes less susceptible to the influence of differences depending on the type of exercise such as the posture or movement of the skeleton or other feature points, so the accuracy of the evaluation can be improved.

[0117] The server 30 may evaluate the user for signs of the target disease by applying a learned model corresponding to the first type and the target disease to input data based on the user movement information. By using a learned model corresponding to the combination of the first type and the target disease, it becomes less susceptible to the influence of differences depending on at least one of the type of exercise or the disease such as the posture or movement of the skeleton or other feature points, so the accuracy of the evaluation can be improved.

[0118] The user movement information may indicate at least one of the position, velocity, or acceleration of the user's joints. Thereby, the posture or movement of the user's skeleton can be quantitatively expressed, and an objective evaluation can be performed.

[0119] The server 30 may acquire at least one of expression information regarding the user's expression, gaze information regarding the user's gaze, tremor information regarding the tremor of the user's hands and feet, or assistive device information regarding the usage of the user's walking assistive device obtained by analyzing the aforementioned image. The server 30 may further evaluate the user for signs of the target disease based on at least one of the expression information, gaze information, gaze information, and assistive device information. As a result, evaluation can be performed from viewpoints other than movement, so it becomes easier to prevent overlooking signs.

[0120] The server 30 may acquire voice information regarding the user's voice during the execution of the first type of exercise, and may further evaluate the user for signs of the target disease based on the voice information. As a result, evaluation can be performed from the viewpoints of movement and voice, so it becomes easier to prevent overlooking signs.

[0121] When the user's evaluation result satisfies a predetermined condition, the server 30 may select a second exercise type that is associated with the target disease and different from the first exercise type among a plurality of exercise types, and present information on the second exercise type to the user. Then, by similarly evaluating symptoms for the second exercise type, a more reliable evaluation result can be obtained.

[0122] When the user's evaluation result satisfies a predetermined condition, the server 30 may present information on a medical institution associated with the target disease to the user. Thereby, it is possible to encourage the user to visit a medical institution that is good at treating the target disease and thereby detect the target disease at an early stage.

[0123] When the user's evaluation result satisfies a predetermined condition, the server 30 may present information on an examination associated with the target disease to the user. Thereby, it is possible to encourage the user to perform an examination established for the target disease and thereby detect the target disease at an early stage.

[0124] (6) Variation A variation of the present embodiment will be described.

[0125] (6-1) Variation 1 Variation 1 will be described. Variation 1 is an example in which symptoms are evaluated using a pattern instead of a learned model.

[0126] In Variation 1, in addition to the model database (Fig. 6), or instead of the model database (Fig. 6), a pattern database is used. Fig. 11 is a diagram showing the data structure of the pattern database of Variation 1.

[0127] Pattern information is stored in the pattern database. The pattern information is information regarding a pattern of movement information (e.g., skeletal information) that serves as a criterion for evaluating signs of one or more diseases. Such a pattern can be determined, for example, by analyzing the characteristics of the skeleton or other feature points that appear when a patient with a specific disease or a person showing signs thereof performs each type of exercise.

[0128] As shown in FIG. 6, the pattern database includes a "Pattern ID" field, a "Pattern Details" field, and an "Applicable Conditions" field. Each field is associated with each other.

[0129] A pattern ID is stored in the "Pattern ID" field. The pattern ID is information for identifying the pattern corresponding to the relevant record.

[0130] Pattern details information is stored in the "Pattern Details" field. The pattern details information is information regarding the details of the pattern corresponding to the relevant record. The details information includes information that can identify the pattern (e.g., information defining conditions regarding a specific joint, or the position, velocity, or acceleration of a specific joint orientation, or time-series changes thereof, or information indicating the location where such information is stored).

[0131] Applicable conditions information is stored in the "Applicable Conditions" field. The applicable conditions information is information regarding the conditions under which the pattern corresponding to the relevant record is applicable. As an example, the applicable conditions information can include information that can identify the type of exercise (e.g., the aforementioned type ID) and information that can identify the disease (i.e., the target disease). As information that can identify the disease, for example, a disease ID that uniquely identifies a single disease or a group of diseases consisting of a plurality of diseases can be defined.

[0132] As a first example of the sign evaluation (step S130 in FIG. 7), the server 30 refers to the pattern database (FIG. 11) and extracts a record including the value of the item ID corresponding to the first item as an application condition. In this case, the pattern corresponding to each record constituting the extraction result corresponds to the characteristics related to the symptoms of the target disease. The server 30 evaluates the user regarding the signs of the target disease by comparing such a pattern with the user movement information.

[0133] As a second example of the sign evaluation (S130), when the target disease is specified, the server 30 refers to the pattern database (FIG. 11) and extracts a record including the item ID corresponding to the first item and the value of the disease ID corresponding to the specified target disease as an application condition. In this case, the pattern corresponding to each record constituting the extraction result corresponds to the characteristics related to the symptoms of the target disease. The server 30 evaluates the user regarding the signs of the target disease by comparing such a pattern with the user movement information. The target disease can be specified by the user, a person related to the user, or an algorithm. Information that can identify the specified target disease may be included in the user data, may be transmitted from the terminal of the related person to the server 30, or may be generated by the server 30.

[0134] As described above, the server 30 of Modification 1 may select a pattern corresponding to the first item and compare the pattern with the user movement information to evaluate the user regarding the signs of the target disease. By using the pattern corresponding to the first item, it becomes difficult to be affected by differences depending on the type of movement such as the posture or movement of the skeleton or other feature points, so the accuracy of the evaluation can be improved.

[0135] The server 30 may select a pattern corresponding to the first item and the target disease and compare the pattern with the user movement information to evaluate the user regarding the signs of the target disease. By using the pattern corresponding to the combination of the first item and the target disease, it becomes difficult to be affected by differences depending on at least one of the type of movement or the disease such as the posture or movement of the skeleton or other feature points, so the accuracy of the evaluation can be improved.

[0136] (7) Other modifications The storage device 11 may be connected to the client device 10 via the network NW. Each input device or output device may be incorporated in the client device 10. The storage device 31 may be connected to the server 30 via the network NW.

[0137] An example of implementing the information processing system 1 of the embodiment by a client / server type system has been shown. However, the information processing system 1 of the embodiment can also be implemented by a peer-to-peer type system or a stand-alone computer. As an example, the client device 10 may perform the evaluation of signs.

[0138] Each of the above information processing steps can be executed by either the client device 10 or the server 30. As an example, instead of the client device 10, the server 30 may acquire user movement information by analyzing sensing data.

[0139] In the above description, an example of capturing a user video using the camera 16 of the client device 10 has been shown. However, the user video may be captured using a camera different from the camera 16. An example of measuring the user depth using the depth sensor 17 of the client device 10 has been shown. However, the user depth may be measured using a depth sensor different from the depth sensor 17.

[0140] It is also possible to use acceleration data as part of the input data for the learned model described in this embodiment or the modification. Alternatively, the movement (e.g., skeleton) of the user may be analyzed with reference to the acceleration data. The acceleration data may be acquired, for example, by the acceleration sensor 19 at the time of capturing the user video or by an acceleration sensor mounted on a wearable device (not shown) worn by the user.

[0141] The information processing system 1 of the present embodiment and each modification example is also applicable to a video game in which the progress of the game is controlled according to the movement of the player's body. The video game may be a mini-game that can be played during the execution of the aforementioned therapeutic application, rehabilitation application, or fitness application. As an example, during the game play, the information processing system 1 estimates the movement (e.g., skeleton) of the user based on the user video. The estimation of the user's movement may be further performed based on at least one of the user depth or the user acceleration in addition to the user video. The information processing system 1 evaluates how well the posture of the user during exercise (e.g., gymnastics) conforms to an ideal posture (model) based on the result of the estimation of the user's movement. The information processing system 1 may determine any one of the following according to the result of this evaluation (e.g., a numerical value indicating the degree of conformity of the user's posture to the ideal posture). Thereby, the effect of the video game on the improvement of the user's health can be enhanced. · The quality (e.g., difficulty level) or quantity of the tasks (e.g., stages, missions, quests) related to the video game given to the user · The quality (e.g., type) or quantity of the benefits (e.g., in-game currency, items, bonuses) related to the video game given to the user · Game parameters (e.g., score, damage) related to the progress of the video game

[0142] In addition to the microphone 18, or instead of the microphone 18, a microphone of a wearable device (not shown) worn by the user (a microphone provided in the wearable device or connected to the wearable device) may receive the sound waves emitted by the user during the shooting of the user video and generate sound data. The sound data may constitute input data for the learned model described in the present embodiment or the modification example. The sound emitted by the user is, for example, a sound generated along with the user's breathing or vocalization.

[0143] As described above, the embodiments of the present invention have been described in detail, but the scope of the present invention is not limited to the above embodiments. Further, the above embodiments can be variously improved and modified without departing from the gist of the present invention. Also, the above embodiments and variations can be combined.

Explanation of Signs

[0144] 1: Information processing system 10: Client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 21: Display 30: Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface

Claims

1. A program for causing a computer to function as: means for acquiring event information related to a first event, which is an event selected from a plurality of sports events by a user, a person concerned, or an algorithm; means for acquiring user movement information related to the movement of the user obtained by analyzing an image of the user performing the first event; means for evaluating the user regarding signs of a target disease, which is at least one of a cerebrovascular disease or a peripheral nerve / muscle disease associated with the first event, based on at least the user movement information and characteristics related to symptoms of the target disease; a program.

2. The means for evaluating evaluates the user regarding signs of the target disease by applying a learned model corresponding to the first event to input data based on the user movement information. The program according to claim 1.

3. The means for evaluating evaluates the user regarding signs of the target disease by applying a learned model corresponding to the first event and the target disease to input data based on the user movement information. The program according to claim 2.

4. The means for evaluating evaluates the user regarding signs of the target disease by selecting a pattern corresponding to the first event and comparing the pattern with the user movement information. The program according to claim 1.

5. The means for evaluating evaluates the user regarding signs of the target disease by selecting a pattern corresponding to the first event and the target disease and comparing the pattern with the user movement information. The program according to claim 4.

6. The user movement information indicates at least one of the position, speed, or acceleration of the joints of the user. The program according to claim 1.

7. A program for causing the computer to function as means for acquiring at least one of expression information related to the expression of the user, gaze information related to the gaze of the user, tremor information related to the tremor of the hands and feet of the user, or auxiliary tool information related to the usage of a walking aid for the user, obtained by analyzing the image; The means for evaluating evaluates the user regarding signs of the target disease based on at least one of the expression information, the gaze information, the tremor information, and the auxiliary tool information. The program according to claim 1.

8. Function the computer as means for acquiring voice information regarding the user's voice during the execution of the first type of exercise. The means for evaluating evaluates the user regarding signs of the target disease based further on the voice information. The program according to claim 1.

9. The computer Function as means for selecting, when the evaluation result of the user satisfies a predetermined condition, a second type that is associated with the target disease and different from the first type among the plurality of exercise types. Function as means for presenting information on the second type to the user. The program according to claim 1.

10. Function the computer as means for presenting to the user information on a medical institution associated with the target disease when the evaluation result of the user satisfies a predetermined condition. The program according to claim 1.

11. Function the computer as means for presenting to the user information regarding an examination associated with the target disease when the evaluation result of the user satisfies a predetermined condition. The program according to claim 1.

12. Means for acquiring item information regarding a first type, which is an exercise type selected from a plurality of exercise types by a user, a related person, or an algorithm, Means for acquiring user movement information regarding the movement of the user obtained by analyzing an image of the user performing the first type of exercise, Means for evaluating the user regarding signs of a target disease, which is at least one of a cerebrovascular disease or a peripheral nerve / muscle disease associated with the first type, based on at least the user movement information and characteristics regarding symptoms of the target disease An information processing apparatus comprising.

13. The computer Executes steps of acquiring item information regarding a first type, which is an exercise type selected from a plurality of exercise types by a user, a related person, or an algorithm, Executes steps of acquiring user movement information regarding the movement of the user obtained by analyzing an image of the user performing the first type of exercise, Executes steps of evaluating the user regarding signs of a target disease, which is at least one of a cerebrovascular disease or a peripheral nerve / muscle disease associated with the first type, based on at least the user movement information and characteristics regarding symptoms of the target disease A method of execution.

14. A system comprising a first information processing device and a second information processing device, wherein the first information processing device includes means for acquiring event information regarding a first event which is an event selected from a plurality of event types by a user, a person concerned, or an algorithm, means for acquiring user movement information regarding the movement of the user obtained by analyzing an image of the user performing the movement of the first event, means for evaluating the user based on at least the user movement information and features regarding symptoms of a target disease which is at least one of a cerebrovascular disease or a peripheral nerve / muscle disease associated with the first event, and means for transmitting information based on the result of the evaluation of the user to the second information processing device. System. ​

Images