Information processing device, method, program, and system

Abstract

A program according to one aspect of the present disclosure causes a computer to function as: a means for acquiring category information relating to a first category that is an exercise category selected from a plurality of exercise categories by a user, a concerned party, or an algorithm; a means for acquiring user movement information relating to the user's movement obtained by analyzing an image capturing the user performing an exercise in the first category; and a means for evaluating the user with regard to a sign of a target disease, which is a cranial nerve disease and / or a peripheral nerve / muscle disease associated with the first category, on the basis of at least the user movement information and a feature relating to a symptom of the target disease.

Description

Information processing device, method, program, and system

[0001] The present disclosure relates to an information processing device, a method, a program, and a system.

[0002] For example, early detection and treatment of neurological diseases such as dementia may slow the progression of the disease and alleviate its symptoms. Symptoms of neurological diseases vary, but for example, the disease may affect the patient's walking, causing gait disorders.

[0003] Patent Document 1 describes a technical concept intended to automatically diagnose neurological disorders such as Parkinson's disease by focusing on posture, vibration, and gait among the patient's physical movements and simply and quantitatively measuring movement disorders that appear in at least one of these.

[0004] Japanese Patent Publication No. 2021-184964

[0005] The technical concept described in Patent Document 1 requires patients to wear over a dozen acceleration sensors on their bodies to measure their posture. Therefore, measuring a patient's movement disorder on a daily basis using this technical concept places a heavy burden on the patient and is therefore not practical.

[0006] An object of the present disclosure is to provide a technology for evaluating signs of abnormality in a user while minimizing the burden on the user.

[0007] A program according to one aspect of the present disclosure causes a computer to function as: a means for acquiring event information relating to a first event, which is an event selected from a plurality of events by a user, a related person, or an algorithm; a means for acquiring user movement information relating to the user's movements obtained by analyzing images of the user performing the first event; and a means for evaluating the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first event, based at least on the user movement information and characteristics relating to the symptoms of the target disease.

[0008] 1 is a block diagram showing the configuration of an information processing system of this embodiment. FIG. 2 is a block diagram showing the configuration of a client device of this embodiment. FIG. 3 is a block diagram showing the configuration of a server of this embodiment. FIG. 4 is an explanatory diagram of one aspect of this embodiment. FIG. 5 is a diagram showing the data structure of an exercise event database of this embodiment. FIG. 6 is a diagram showing the data structure of a model database of this embodiment. FIG. 7 is a flowchart of a sign evaluation process of this embodiment. FIG. 8 is a diagram showing an example of a screen displayed in the sign evaluation process of this embodiment. FIG. 9 is a diagram showing an example of a screen displayed in the sign evaluation process of this embodiment. FIG. 10 is a diagram showing an example of a screen displayed in the sign evaluation process of this embodiment. FIG. 11 is a diagram showing the data structure of a pattern database of variant example 1.

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0010] (1) Configuration of Information Processing System The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to this embodiment.

[0011] As shown in FIG. 1, the information processing system 1 includes a client device 10 and a server 30 .

[0012] The number of client devices 10 varies depending on the number of users, for example. Therefore, the number of client devices 10 may be two or more. Furthermore, the information processing system 1 may also include terminals of people related to each user.

[0013] The person concerned may include, for example, at least one of the following: - A family member of the user - A person who plans / performs the user's exercise therapy (typically, a medical professional (e.g., a doctor (which may include the user's attending physician), a nurse, a pharmacist, a physical therapist, an occupational therapist, a clinical laboratory technician), or a nutritionist) - A person who instructs the user's exercise therapy (typically, a medical professional, a nutritionist, or a trainer)

[0014] The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW.

[0015] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0016] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a server computer.

[0017] (1-1) Configuration of the Client Device The configuration of the client device will be described below with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0018] 2 , the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0019] The storage device 11 is configured to store programs and data, and is, for example, a combination of a read-only memory (ROM), a random access memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0020] The programs include, for example, the following: - An operating system (OS) program - An application program that executes information processing (for example, a web browser, a therapeutic app, a rehabilitation app, or a fitness app) Here, diseases that are the target of therapeutic apps or rehabilitation apps are diseases in which exercise may contribute to improving symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

[0021] The data includes, for example, the following data: Databases referenced in information processing Data obtained by executing information processing (i.e., execution results of information processing)

[0022] The processor 12 is a computer that executes the programs stored in the storage device 11 to realize the functions of the client device 10. The processor 12 is, for example, at least one of the following: - CPU (Central Processing Unit) - GPU (Graphic Processing Unit) - ASIC (Application Specific Integrated Circuit) - FPGA (Field Programmable Gate Array)

[0023] The input / output interface 13 is configured to acquire information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10.

[0024] The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0025] The communication interface 14 is configured to control communication between the client device 10 and an external device (e.g., another client device 10 (e.g., a terminal of a related party), or the server 30). Specifically, the communication interface 14 may include a module for communication (e.g., a Wi-Fi module, a mobile communication module, a Bluetooth (registered trademark) module, or a combination thereof).

[0026] The display 15 is configured to display an image (still image or moving image) and is, for example, a liquid crystal display or an organic EL display.

[0027] The camera 16 is configured to take pictures and generate image signals.

[0028] The depth sensor 17 is, for example, a light detection and ranging (LIDAR) sensor. The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to a surrounding object (e.g., a user). Note that the depth sensor 17 is not essential and may be removed from the client device 10.

[0029] The microphone 18 is configured to receive sound waves and generate sound signals, and is preferably placed near the user's body (particularly the respiratory tract) as an earphone microphone.

[0030] The acceleration sensor 19 is configured to detect acceleration.

[0031] (1-2) Server Configuration The server configuration will be described below with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0032] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0033] The storage device 31 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage.

[0034] The programs include, for example, the following programs: OS programs; application programs for executing information processing.

[0035] The data includes, for example, the following data: Databases referenced in information processing Execution results of information processing

[0036] The processor 32 is a computer that executes the programs stored in the storage device 31 to realize the functions of the server 30. The processor 32 is, for example, at least one of the following: CPU, GPU, ASIC, FPGA

[0037] The input / output interface 33 is configured to acquire information (e.g., user instructions) from an input device connected to the server 30, and to output information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0038] The communication interface 34 is configured to control communications between the server 30 and an external device (eg, the client device 10).

[0039] (2) One aspect of the embodiment One aspect of the present embodiment will be described below. Fig. 4 is an explanatory diagram of one aspect of the present embodiment.

[0040] 4, the client device 10 senses the user US1 while he or she is exercising. Although not shown, a wearable device equipped with, for example, an acceleration sensor may also sense the user US1 while he or she is exercising. The user US1 is typically a person undergoing exercise therapy, such as, but not limited to, a participant in a (cardiac) rehabilitation program or an exercise instruction program.

[0041] The user US1 is performing an exercise type (hereinafter referred to as the "first exercise type") selected by the user, a related party, or an algorithm from among a plurality of available exercise types, where the available exercise type is an exercise type associated with a disease in the storage device 31 of the server 30 or another storage device accessible to the server 30.

[0042] The disease associated with athletic events may be a cranial nerve disease, a peripheral nerve / muscle disease, or a combination thereof. The cranial nerve disease particularly refers to a cranial nerve disease, primarily an intracranial disease, and may include, for example, at least one of dementia (e.g., Alzheimer's disease, vascular dementia, or Lewy body dementia), normal pressure hydrocephalus, or cerebellar disease. The peripheral nerve / muscle disease may include, for example, at least one of Parkinson's disease, myositis, muscular dystrophy, or Charcot-Marie-Thu's disease.

[0043] As an example, the camera 16 captures the appearance (e.g., the entire body) of the user US1 while exercising from a front or oblique front view at a distance of, for example, about 2 m. The camera 16 may be installed at an appropriate height using a tripod or other height-adjusting means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. Note that it is also possible to generate three-dimensional video data by combining, for example, two-dimensional video data generated by the camera 16 with depth data generated by, for example, the depth sensor 17. The microphone 18 receives sounds (e.g., sounds generated by breathing or speaking) emitted by the user US1 while exercising and generates an audio signal. The acceleration sensor 19 measures the acceleration of the user US1 while exercising.

[0044] The client device 10 acquires various sensing data and analyzes it as necessary. For example, the client device 10 may refer to video data acquired from the camera 16 to analyze the body movements of the user US1 during exercise (particularly, the movements of the skeleton or other feature points over multiple time periods, and the state of the skeleton or other feature points at a single time). The client device 10 may further refer to depth data acquired from the depth sensor 17 to analyze the body movements of the user US1 during exercise. The client device 10 transmits user data including at least one of the sensing data or the analysis results of the sensing data to the server 30. However, some or all of the analysis may be performed by the server 30.

[0045] The server 30 evaluates the user US1 for at least one of cranial nerve disease or peripheral nerve / muscle disease (hereinafter referred to as "target disease") associated with the first category, based on the user data acquired from the client device 10. Specifically, the server 30 performs the evaluation based on the characteristics of the symptoms of the target disease (e.g., a specific type of movement disorder) and the analysis results of the user's body movements.

[0046] The server 30 generates information based on the evaluation results and presents it to the user US1 via the client device 10. Alternatively, the server 30 may present this information to the relevant parties on behalf of the user US1. This can encourage, for example, a medical visit or examination if the user US1 is evaluated as having symptoms of a target disease. Additionally, this embodiment reduces the burden on the user US1 because it does not require the user to wear multiple sensors (e.g., attach acceleration sensors to various parts of the body) or to perform only limited exercises (e.g., walking only). Therefore, such evaluations can be easily performed on a daily basis (e.g., as part of exercise therapy), contributing to the early detection of target diseases.

[0047] (3) Database The database of this embodiment will be described. The following database is stored in the storage device 31 or another storage device accessible by the server 30.

[0048] (3-1) Exercise Event Database The exercise event database of this embodiment will be described below. Fig. 5 is a diagram showing the data structure of the exercise event database of this embodiment.

[0049] The exercise type database stores exercise type information. The exercise type information is information about exercise types (e.g., the available exercise types described above). In this embodiment, the exercise types include exercise types that can be performed without using devices with adjustable exercise loads, such as gymnastics, bodyweight training, dancing, walking, running, and treadmills. These exercise types include those performed in a standing position, providing a wide variety of options. Furthermore, in this embodiment, the exercise load of these exercise types can be adjusted through form (e.g., range of motion of a body part, arm or leg spacing, etc.), pace, number of repetitions, or rest time or number of repetitions. However, the exercise types in this embodiment may also include exercise types performed using devices with adjustable exercise loads, such as an ergometer or strength training using training equipment.

[0050] 5, the exercise event database includes an "ID" field, a "name" field, and an "exercise load amount" field. Each field is associated with the other fields.

[0051] The "ID" field stores an exercise type ID. The exercise type ID is information for identifying the exercise type corresponding to the record.

[0052] The "Name" field stores information about the name of the exercise type. The exercise type name information is information about the name of the exercise type corresponding to the record.

[0053] The "exercise load" field stores exercise load information. The exercise load information is information about the standard exercise load for the exercise type corresponding to the record. The standard load is, for example, information about the exercise load when a person with standard physical function performs the corresponding exercise type. For example, the exercise load may be derived by measuring the exercise load when multiple people perform the corresponding exercise type and statistically processing (e.g., averaging) the measurement results, or by referencing the exercise load set for the exercise type by a third-party institution. The exercise load information may be managed in units smaller than the exercise type. For example, the exercise load information may be managed for each variation of the form, pace, number of reps, or rest time or number of repetitions for each exercise type. Note that the "exercise load" field is not required and may be removed from the exercise type database.

[0054] (3-2) Model Database The model database of this embodiment will now be described with reference to Fig. 6, which shows the data structure of the model database of this embodiment.

[0055] The model database stores model information, which is information about a trained model that makes inferences about one or more disease symptoms based on input data related to at least body movements (e.g., skeleton).

[0056] 6, the model database includes a "model ID" field, a "model details" field, and an "applicable conditions" field. Each field is associated with the other fields.

[0057] The “model ID” field stores a model ID. The model ID is information for identifying the trained model corresponding to the record.

[0058] The "Model Details" field stores detailed model information. The detailed model information is information about the details of the trained model corresponding to the record. The detailed model information includes information that can identify the structure of the trained model (e.g., the layer structure in a neural network, the connection relationships between nodes, and the weights of each edge). The detailed model information may include, for example, information indicating the values ​​of each parameter that defines the structure of the trained model, or may include information indicating the location where the values ​​are stored.

[0059] The "application condition" field stores application condition information. The application condition information is information about the conditions under which the trained model corresponding to the record in question can be applied. As an example, the application condition information can include information that can identify the type of exercise (e.g., the type ID mentioned above) and information that can identify the disease (i.e., the target disease). As information that can identify the disease, for example, a disease ID that uniquely identifies a single disease or a group of diseases consisting of multiple diseases can be defined.

[0060] (3-3) Other Databases Databases other than those described above may be constructed. As an example, a user profile database will be described.

[0061] The user profile database stores user profile information, which is information relating to the profile of a user of the information processing system 1 (for example, a person receiving exercise therapy).

[0062] The user profile database may store records including at least one of the following information, for example: User ID User name information Physical information Information indicating the amount of exercise load suitable for the user Information indicating the user's associates

[0063] The user ID is information that identifies the user corresponding to the corresponding record. The user name information is information about the name (e.g., name, account name, etc.) of the user corresponding to the corresponding record. The physical information is information about the body (functions) of the user corresponding to the corresponding record. As an example, the physical information may include information about the user's age, sex, weight, height, illness, etc.

[0064] (4) Information Processing Information processing in this embodiment will be described.

[0065] (4-1) Sign Evaluation Processing The sign evaluation processing of this embodiment will be described. Fig. 7 is a flowchart of the sign evaluation processing of this embodiment. Fig. 8 is a diagram showing an example of a screen displayed in the sign evaluation processing of this embodiment. Fig. 9 is a diagram showing an example of a screen displayed in the sign evaluation processing of this embodiment. Fig. 10 is a diagram showing an example of a screen displayed in the sign evaluation processing of this embodiment.

[0066] The symptom evaluation process starts when, for example, one of the following start conditions is met: - The symptom evaluation process is called by another process. - The user or a person related to the user performs an operation to call the symptom evaluation process. - The client device 10 enters a predetermined state (for example, a predetermined application is launched). - A predetermined date and time has arrived. - A predetermined amount of time has passed since a predetermined event.

[0067] As shown in FIG. 7 , the client device 10 acquires sensing data (S110). Specifically, the client device 10 may enable the operation of the camera 16 to start capturing video of the user exercising (hereinafter referred to as "user video"). The client device 10 may also enable the operation of the depth sensor 17 to start measuring the distance from the depth sensor 17 to each part of the user exercising (hereinafter referred to as "user depth"). The client device 10 may enable the operation of the microphone 18 to start collecting sound (e.g., sound generated by the user's vocalization or breathing (hereinafter referred to as "user sound")). The client device 10 may enable the operation of the acceleration sensor 19 to start measuring acceleration.

[0068] The client device 10 then acquires sensing data from each sensor. Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 may acquire user video data from the camera 16, user depth data from the depth sensor 17, user sound data from the microphone 18, and user acceleration data related to the user's acceleration (hereinafter, "user acceleration") from the acceleration sensor 19.

[0069] After step S110, the client device 10 generates user data (S111). Specifically, the client device 10 generates user data based on the sensing data acquired in step S110. The user data may include at least one of the following: Data acquired in step S111 (e.g., user video data, user depth data, user sound data, or user acceleration data); Data obtained by processing the data acquired in step S111; Data acquired by analyzing the data acquired in step S111 (e.g., movement information (skeletal data, as an example), facial expression data, gaze data, tremor data, assistive device data, voice data, or a combination thereof, as described below); Information that can identify the type of exercise (first type) performed by the user in step S110.

[0070] After step S111, the client device 10 transmits the user data (S112). Specifically, the client device 10 transmits the user data generated in step S111 to the server 30.

[0071] After step S112, the server 30 evaluates the symptoms (S130). Specifically, the server 30 receives the user data transmitted by the client device 10 in step S112. The server 30 acquires user skeletal information based on the user data acquired from the client device 10. The user skeletal information is information about the user's skeletal structure obtained by analyzing user video data. In addition to the user video data, user depth data may be referenced to generate the user skeletal information. The user skeletal information may be included in the user data, or may be generated by the server 30 analyzing data included in the user data. Note that the user skeletal information is an example of information about the user's body movements during exercise (hereinafter referred to as "user movement information"), and may be used together with or replace information about the movements of other feature points.

[0072] User skeletal information is information (e.g., data such as feature amounts) about the user's skeleton during exercise. The user skeletal information includes, for example, information about the position, speed, or acceleration of each part of the user's body (which may include information about changes in the parts of the muscles used by the user or shaking of the user's trunk). The user skeletal information can be obtained by analyzing the user's skeleton during exercise with reference to user video data (or user video data and user depth data). As an example, Vision, an SDK for iOS 14, or other skeletal detection algorithms (e.g., OpenPose, PoseNet, MediaPipe Pose) can be used for skeletal analysis.

[0073] The results of motion detection, such as skeletal detection, can also be used for quantitative evaluation, qualitative evaluation, or a combination of these for exercise. As a first example, the results of motion detection can be used to count the number of reps. As a second example, the results of motion detection can be used to evaluate the form of the exercise or the appropriateness of the load applied by the exercise. For example, when the exercise is squats, the results of motion detection can be used to evaluate whether the knees are protruding too far forward, resulting in a dangerous form, or whether the hips are lowered firmly and deeply to apply sufficient load.

[0074] The server 30 also acquires information about the first event (e.g., information that can identify the first event, such as an event ID corresponding to the first event). Such information may be included in the user data. Alternatively, the information may be transmitted from the terminal of the participant to the server 30 or generated by the server 30 during the process in which the first event is selected by the participant or an algorithm.

[0075] The server 30 evaluates the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first category, based on at least the user movement information and characteristics related to the symptoms of the target disease.

[0076] As a first example of symptom evaluation (S130), the server 30 references the model database ( FIG. 6 ) and extracts records that include an event ID value corresponding to the first event as an application condition. In this case, the trained model corresponding to each record constituting the extraction result corresponds to features related to symptoms of the target disease. The server 30 evaluates the user for symptoms of the target disease by applying the trained model to input data based on user movement information.

[0077] The trained model focuses on the presence or absence of characteristic skeletal or other feature postures or movements observed as symptoms of the corresponding disease, and makes inferences regarding the presence or absence of signs of the disease, or the degree of likelihood of the presence of signs of the disease. The trained model accepts input data based on movement information (e.g., skeletal information) and outputs evaluation results. The evaluation results may be binary data indicating the presence or absence of a disease sign, or multi-valued data indicating the likelihood of the presence of such signs in multiple stages. The trained model corresponds to a trained model created by supervised learning using a training dataset, or a derived model or distilled model of the trained model.

[0078] As a second example of symptom evaluation (S130), when a target disease is specified, the server 30 references the model database ( FIG. 6 ) and extracts records including, as application conditions, an event ID corresponding to the first event and a disease ID value corresponding to the specified target disease. In this case, the trained model corresponding to each record constituting the extraction result corresponds to features related to the symptoms of the target disease. The server 30 evaluates the user for symptoms of the target disease by applying the trained model to input data based on user movement information. The target disease may be specified by the user, a related party of the user, or an algorithm. Information capable of identifying the specified target disease may be included in the user data, transmitted from a related party's terminal to the server 30, or generated by the server 30.

[0079] After step S130, the server 30 generates information (S131). Specifically, the server 30 generates information to be presented to the user based on the results of the evaluation in step S130.

[0080] As a first example of information generation (S131), when an evaluation result is obtained indicating the presence of a symptom of any target disease or the likelihood of the presence of a symptom of any target disease exceeds a threshold (an example of "when the user's evaluation result satisfies a predetermined condition"), the server 30 may select an exercise type (hereinafter referred to as the "second exercise type") to recommend to the user and generate information (e.g., an exercise ID) that can identify the selected second exercise type. As an example, the server 30 may select as the second exercise type an exercise type other than the first exercise type from among the exercise types associated with the target disease for which symptoms were evaluated in step S131 as being present or the likelihood of such symptoms exceeding a threshold. The server 30 may further select the second exercise type by taking into account the exercise load appropriate for the user and the exercise load of each exercise type. By resensing the user while performing the recommended second exercise in this way and re-evaluating the symptoms (S130), more reliable evaluation results can be obtained. In other words, it is possible to determine whether the symptoms observed during the first exercise were merely coincidental or whether similar symptoms are observed regardless of the exercise type.

[0081] As a second example of generating information (S131), when an evaluation result indicating the presence of a symptom of any target disease or the possibility of the presence of a symptom of any target disease exceeds a threshold is obtained, the server 30 may generate information capable of identifying a medical institution associated with the target disease. As an example, information on medical institutions suitable for diagnosing each target disease is associated with each target disease and stored in a database in advance, and the server 30 can generate information by referring to the database. The server 30 may narrow down the medical institutions by taking into account information on the user's place of residence or current location in addition to the target disease.

[0082] As a third example of generating information (S131), when an evaluation result indicating the presence of a symptom of any target disease or the possibility of the presence of a symptom of any target disease exceeds a threshold is obtained, the server 30 may generate information recommending the implementation of a test associated with the target disease (for example, a screening test such as the Hasegawa Dementia Scale). As an example, information on tests appropriate for each target disease is associated with the target disease and stored in a database in advance, and the server 30 can generate information by referring to the database.

[0083] As a fourth example of generating information (S131), when an evaluation result indicating the presence of a symptom of any of the target diseases or the possibility of the presence of a symptom of any of the target diseases exceeds a threshold, the server 30 may combine two or more of the above-mentioned first to third examples. For example, the server 30 may perform the above-mentioned first example one or more times, and when the total number or percentage of times that an evaluation result indicating the presence of a symptom of the target disease or the possibility thereof exceeds a threshold reaches a predetermined value, the server 30 may perform at least one of the above-mentioned second example or third example.

[0084] In addition, if the evaluation result indicates that there are no symptoms of any of the target diseases or that the possibility of the presence of symptoms of any of the target diseases does not exceed a threshold, the server 30 may omit the steps from generating information (S131) ​​to presenting information (S132).

[0085] After step S131, the server 30 executes the presentation of information (S132). Specifically, the server 30 transmits the information generated in step S131 to the client device 10. The server 30 can also present information to a related party of the user in addition to the user or on behalf of the user. In this case, the server 30 may transmit the information generated in step S132 to the terminal of the related party.

[0086] After step S132, the client device 10 executes screen display (S113). Specifically, the client device 10 receives the information transmitted by the server in step S131. The client device 10 displays a screen (not limited to an application screen, but may also include notifications (application notifications, or messages such as emails or chats)) based on the received information on the display 21.

[0087] As a first example of the screen display (S113), the client device 10 displays the screen of Fig. 8 on the display 21. The screen of Fig. 8 includes objects J20 to J22.

[0088] The object J20 displays information about the second sport. The object J20 also accepts user instructions to start each sport or to play a demonstration video for each sport. When the object J20 is selected, the client device 10 may re-execute the symptom evaluation process of this embodiment, setting the sport (second sport) corresponding to the object J20 as the new first sport, or may play a demonstration video for the sport.

[0089] The object J21 accepts a user instruction to select an exercise type other than the second type. When the object J21 is selected, the client device 10 may, for example, display a list of exercise types other than the second type and accept a user instruction to select an exercise type. In response to accepting such a user instruction, the client device 10 may play a demonstration video of the exercise type selected by the user, or, if the exercise type is associated with the target disease, may re-execute the symptom evaluation process of this embodiment with the exercise type as a new first type.

[0090] The object J22 receives a user instruction to end the exercise. When the object J22 is selected, the client device 10 ends the symptom evaluation process of this embodiment.

[0091] As a second example of the screen display (S113), the client device 10 displays the screen of Fig. 9 on the display 21. The screen of Fig. 9 includes objects J30 to J31.

[0092] The object J30 displays information recommending a visit to a medical institution. The object J31 accepts a user instruction to view information about medical institutions associated with the target disease. In response to the user instruction, the client device 10 may place an object displaying information about medical institutions associated with the target disease on the screen or transition to a screen displaying such information.

[0093] As a third example of the screen display (S113), the client device 10 displays the screen of Fig. 10 on the display 21. The screen of Fig. 10 includes objects J40 to J41.

[0094] The object J40 displays information recommending the implementation of a test for the target disease. The object J41 accepts a user instruction to start a test associated with the target disease. In response to the user instruction, the client device 10 may place an object on the screen displaying information about the test associated with the target disease (e.g., questions included in the test or a link to a website where the test can be taken) or transition to a screen displaying such information.

[0095] After step S113, the client device 10 may end the symptom evaluation process (FIG. 7).

[0096] (5) Teacher Data Set A teacher data set that can be used for supervised learning to construct the trained model of this embodiment will be described.

[0097] The training dataset includes multiple training data sets. The training data sets are used to train or evaluate a model to be learned (hereinafter referred to as a "target model"). The training data sets include a sample ID, input data, and correct answer data.

[0098] The sample ID is information that identifies the training data.

[0099] The input data is data input to the target model during training or evaluation. The input data corresponds to examples used during training or evaluation of the target model. As an example, the input data is data (i.e., relatively dynamic data) regarding the subject's body movements during exercise (i.e., while performing the exercise type corresponding to the target model) and data (i.e., relatively static data) regarding the subject's health condition. At least a portion of the data regarding the subject's body movements is obtained by analyzing the subject's body movements with reference to subject video data (or subject video data and subject depth data).

[0100] The subject video data is data related to a video of a subject during exercise. The subject video data can be obtained, for example, by capturing an external image (e.g., the entire body) of a subject undergoing an exhaled gas test (e.g., a CPX test) from the front or diagonally in front (e.g., 45 degrees forward) using a camera (e.g., a camera mounted on a smartphone).

[0101] The subject depth data is data relating to the distance (depth) from the depth sensor to each part of the subject during exercise. The subject depth data can be acquired by operating the depth sensor when shooting the subject video.

[0102] The subjects typically include those diagnosed with a specific disease or its symptoms and those diagnosed as having or not having the disease or its symptoms. Diagnosis is usually performed by a doctor, but may be substituted by a determination using an algorithm (which may include a trained model). However, for the same subject, data collected when the subject was diagnosed with a specific disease or its symptoms and data collected when the subject was diagnosed as having or not having the disease or its symptoms can also be used as training data. Furthermore, one of the subjects may be the same person as the user whose symptoms of the specific disease are evaluated when the information processing system 1 is in operation.

[0103] In this example, the input data includes at least motion data. The motion data may include elements similar to the user motion information (e.g., skeletal information) described above and may be obtained by analyzing the motion (e.g., skeletal) of the subject during exercise with reference to subject video data. In addition to the subject video data, at least one of subject depth data and acceleration data measured by a wearable device attached to the subject may be referenced to determine the motion data.

[0104] The correct answer data is data that corresponds to the correct answer for the corresponding input data (example question). The target model is trained (supervised learning) to output the input data in a way that is closest to the correct answer data. For example, the correct answer data indicates the presence or absence of a specific disease or its symptoms.

[0105] The correct answer data indicates, for example, the presence or absence of a specific disease or its symptoms for the subject from whom the corresponding input data was obtained. That is, the correct answer data corresponding to input data obtained from a subject diagnosed with a specific disease or its symptoms has a value indicating the presence of the disease or its symptoms. On the other hand, the correct answer data corresponding to input data obtained from a subject diagnosed as not having a specific disease or its symptoms has a value indicating the absence of the disease or its symptoms.

[0106] In addition to the movement data, at least one of the following may be added to the input data together with the movement data, or a trained model may be constructed that performs similar inference on input data that includes at least one of the following as an element. In this case, the input data when the trained model is in operation will also include similar elements based on the user's sensing results. Included are: facial expression data (an example of "facial expression information"); gaze data (an example of "gaze information"); tremor data (an example of "tremor information"); assistive device data (an example of "assistive device information"); voice data (an example of "voice information"); and health condition data (an example of "health condition information").

[0107] Facial expression data is data (e.g., features) about a subject's facial expressions during exercise. Facial expression data can be analyzed by applying an algorithm or trained model to video data of the subject. Alternatively, facial expression data for a training dataset can be obtained by, for example, human labeling of the subject's videos.

[0108] Gaze data is data (e.g., features) about the gaze of a subject during exercise. The gaze data can be analyzed by applying an algorithm or trained model to the subject's video data.

[0109] Tremor data is data (e.g., features) about a subject's tremor during exercise. Tremor data can be analyzed by applying an algorithm or trained model to subject video data. Alternatively, tremor data for a training dataset can be obtained, for example, by human labeling of subject videos.

[0110] Assistive device data is data (e.g., features) about how a subject uses an assistive device (e.g., a walking aid such as a cane, or other equipment used in exercise) during exercise. Assistive device data can be analyzed by applying an algorithm or trained model to subject video data. Alternatively, assistive device data for a training dataset can be obtained, for example, by human labeling after viewing subject videos.

[0111] Voice data is data (e.g., features) about a subject's voice while exercising. Voice data can be analyzed by applying an algorithm or trained model to sound data collected by microphones placed around the subject. Alternatively, aid data for a training dataset can be obtained by, for example, human labeling of the subject's voice.

[0112] Health status data is data related to the subject's health status. Health status data can be obtained in various ways. The subject's health status data may be obtained before, during, or after the subject's exercise. The subject's health status data may be obtained based on a report from the subject or their doctor, by extracting information linked to the subject in a medical information system, or via the subject's app (e.g., a healthcare app).

[0113] The health condition includes at least one of the following: Age Gender Height Weight Body fat percentage Muscle mass Bone density History of current illness Past medical history Medication history History of surgery Lifestyle history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty score, etc.) Family history Results of respiratory function tests Results of tests other than respiratory function tests (e.g., results of blood tests, urine tests, electrocardiograms (including Holter ECG), cardiac ultrasound, X-rays, CT scans (including cardiac morphological CT and coronary CT), MRI scans, nuclear medicine scans, PET scans, etc.) Data obtained during cardiac rehabilitation (including the Borg index)

[0114] It is also possible to construct a trained model for each of multiple health condition categories based on (at least a portion of) the subject's health condition. In this case, (at least a portion of) the user's health condition may be referenced to select the trained model. In this variation, the input data for the trained model may be data that is not based on the user's health condition, or may be data that is based on the user's health condition and user videos.

[0115] (5) Summary As described above, the server 30 of this embodiment acquires user movement information related to the user's movements by analyzing images of the user performing a first type of exercise, which is a type of exercise selected from multiple types of exercise by the user, a related party, or an algorithm. The server 30 evaluates the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first type, based on at least the user movement information and features related to the symptoms of the target disease (in this embodiment, a trained model). This reduces the burden on the user, making it easier to evaluate signs of the target disease on a daily basis and contributing to early detection of the target disease.

[0116] The server 30 may evaluate the user for signs of a target disease by applying a trained model corresponding to the first sport to input data based on user movement information. This allows for statistically valid evaluation results to be obtained without creating an evaluation algorithm. Furthermore, using a trained model corresponding to the first sport reduces the influence of differences depending on the sport, such as the posture or movement of the skeleton or other feature points, thereby improving the accuracy of the evaluation.

[0117] The server 30 may evaluate the user for symptoms of the target disease by applying a trained model corresponding to the first sport and the target disease to input data based on the user movement information. Using a trained model corresponding to the combination of the first sport and the target disease can improve the accuracy of the evaluation because it is less affected by differences that depend on at least one of the sport or disease, such as the posture or movement of skeletal or other feature points.

[0118] The user movement information may indicate at least one of the position, velocity, and acceleration of the user's joints, thereby allowing the posture or movement of the user's skeleton to be quantitatively expressed and objectively evaluated.

[0119] The server 30 may acquire at least one of facial expression information related to the user's facial expression, gaze information related to the user's gaze, tremor information related to the tremors of the user's limbs, or assistive device information related to the user's use of a walking aid, all of which are obtained by analyzing the images. The server 30 may also evaluate the user for signs of the target disease based on at least one of the facial expression information, gaze information, gaze information, and assistive device information. This allows evaluation from perspectives other than movement, making it easier to prevent signs from being overlooked.

[0120] The server 30 may acquire voice information about the user's voice while performing the first type of exercise, and may evaluate the user for signs of the target disease based on the voice information. This allows evaluation from the perspectives of movement and voice, making it easier to prevent signs from being overlooked.

[0121] If the user's evaluation results satisfy predetermined conditions, the server 30 may select a second type of exercise from the plurality of exercise types that is associated with the target disease and different from the first type, and present information about the second type to the user. Then, by similarly evaluating the symptoms of the second type, more reliable evaluation results can be obtained.

[0122] If the user's evaluation result satisfies a predetermined condition, the server 30 may present the user with information about medical institutions associated with the target disease, thereby encouraging the user to visit a medical institution that specializes in the target disease and thereby detect the target disease early.

[0123] If the user's evaluation result satisfies a predetermined condition, the server 30 may present the user with information about tests associated with the target disease, thereby encouraging the user to perform established tests for the target disease and thereby detect the target disease early.

[0124] (6) Modifications A modification of this embodiment will be described.

[0125] (6-1) Modification 1 Modification 1 will be described. Modification 1 is an example in which a pattern is used instead of a trained model to evaluate a symptom.

[0126] In the first modification, a pattern database is used in addition to or instead of the model database (FIG. 6). FIG. 11 is a diagram showing the data structure of the pattern database in the first modification.

[0127] The pattern database stores pattern information. The pattern information is information about patterns of movement information (e.g., skeletal information) that serve as a basis for evaluation of one or more disease symptoms. Such patterns can be determined, for example, by analyzing the characteristics of skeletal or other features that appear when a patient with a specific disease or a person showing symptoms thereof performs each exercise.

[0128] 6, the pattern database includes a "pattern ID" field, a "pattern details" field, and an "applicable conditions" field. Each field is associated with the other fields.

[0129] The "pattern ID" field stores a pattern ID, which is information for identifying the pattern corresponding to the record.

[0130] The "pattern details" field stores detailed pattern information. The detailed pattern information is information about the details of the pattern corresponding to the record. The detailed pattern information includes information that can identify the pattern (for example, information that defines the position, speed, or acceleration of a specific joint or the orientation of a specific joint, or conditions related to time-series changes of these, or information that indicates where such information is stored).

[0131] The "applicable condition" field stores applicable condition information. The applicable condition information is information about the conditions under which the pattern corresponding to the record in question is applicable. As an example, the applicable condition information may include information that can identify the type of exercise (e.g., the type ID mentioned above) and information that can identify the disease (i.e., the target disease). As information that can identify the disease, for example, a disease ID that uniquely identifies a single disease or a group of diseases consisting of multiple diseases may be defined.

[0132] As a first example of symptom evaluation (step S130 in FIG. 7 ), the server 30 references the pattern database ( FIG. 11 ) and extracts records that include an event ID value corresponding to the first event as an application condition. In this case, the patterns corresponding to each record constituting the extraction result correspond to characteristics related to symptoms of the target disease. The server 30 compares these patterns with the user movement information to evaluate the user for symptoms of the target disease.

[0133] As a second example of symptom evaluation (S130), when a target disease is specified, the server 30 references the pattern database ( FIG. 11 ) and extracts records including, as application conditions, an event ID corresponding to the first event and a disease ID value corresponding to the specified target disease. In this case, the patterns corresponding to each record constituting the extraction result correspond to characteristics related to the symptoms of the target disease. The server 30 compares the patterns with user movement information to evaluate the user for symptoms of the target disease. The target disease may be specified by the user, a related party of the user, or an algorithm. Information capable of identifying the specified target disease may be included in user data, transmitted from a related party's terminal to the server 30, or generated by the server 30.

[0134] As described above, the server 30 of Modification 1 may select a pattern corresponding to the first type of exercise and compare the pattern with the user movement information to evaluate the user for signs of a target disease. Using the pattern corresponding to the first type of exercise reduces the influence of differences depending on the type of exercise, such as the posture or movement of the skeleton or other feature points, thereby improving the accuracy of the evaluation.

[0135] The server 30 may select a pattern corresponding to the first sport and the target disease and compare the pattern with the user movement information to evaluate the user for signs of the target disease. Using a pattern corresponding to a combination of the first sport and the target disease can improve the accuracy of the evaluation because it is less affected by differences that depend on at least one of the sport or disease, such as the posture or movement of skeletal or other feature points.

[0136] (7) Other Modifications The storage device 11 may be connected to the client device 10 via the network NW. Each input device or output device may be built into the client device 10. The storage device 31 may be connected to the server 30 via the network NW.

[0137] In the above embodiment, the information processing system 1 is implemented as a client / server system. However, the information processing system 1 may also be implemented as a peer-to-peer system or a standalone computer. For example, the client device 10 may evaluate the symptoms.

[0138] Each step of the above information processing can be executed by either the client device 10 or the server 30. As an example, the server 30, instead of the client device 10, may acquire user movement information by analyzing sensing data.

[0139] In the above description, an example has been shown in which a user video is captured using the camera 16 of the client device 10. However, the user video may be captured using a camera other than the camera 16. In the above description, an example has been shown in which the user depth is measured using the depth sensor 17 of the client device 10. However, the user depth may be measured using a depth sensor other than the depth sensor 17.

[0140] The acceleration data may be used as part of the input data for the trained model described in this embodiment or the modified example. Alternatively, the acceleration data may be used to analyze the user's movements (e.g., skeleton). The acceleration data may be acquired, for example, by the acceleration sensor 19 when capturing a user video or by an acceleration sensor mounted on a wearable device (not shown) worn by the user.

[0141] The information processing system 1 of this embodiment and each modification can also be applied to a video game in which game progress is controlled according to the player's physical movements. The video game may be a mini-game that can be played while the aforementioned therapeutic app, rehabilitation app, or fitness app is being executed. As an example, the information processing system 1 estimates the user's movements (e.g., skeletal structure) based on the user's video during gameplay. The estimation of the user's movements may be based on at least one of the user's depth and user acceleration in addition to the user's video. Based on the results of the estimation of the user's movements, the information processing system 1 evaluates the degree to which the user's posture during exercise (e.g., gymnastics) matches an ideal posture (model). The information processing system 1 may determine one of the following depending on the results of this evaluation (e.g., a numerical value indicating the degree to which the user's posture matches the ideal posture). This can enhance the effect of the video game on improving the user's health. The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) given to the user. The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) given to the user. Game parameters related to the progress of the video game (e.g., score, damage).

[0142] In addition to or instead of microphone 18, a microphone (included in or connected to a wearable device (not shown) worn by a user) may receive sound waves emitted by the user when capturing a user video and generate sound data. The sound data may constitute input data for the trained model described in this embodiment or the modified example. The sound emitted by the user is, for example, sound generated by the user's breathing or vocalization.

[0143] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above-described embodiments. Furthermore, the above-described embodiments can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above-described embodiments and modifications can be combined.

[0144] 1: Information processing system 10: Client device 11: Storage device 12: Processor 13: Input / output interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 21: Display 30: Server 31: Storage device 32: Processor 33: Input / output interface 34: Communication interface

Claims

1. A program that causes a computer to function as: a means for acquiring event information relating to a first event, which is an event selected from a plurality of event types by a user, a related person, or an algorithm; a means for acquiring user movement information relating to the movements of the user obtained by analyzing images of the user performing the first event; and a means for evaluating the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first event, based at least on the user movement information and characteristics relating to the symptoms of the target disease.

2. The program according to claim 1, wherein the evaluating means evaluates the user for symptoms of the target disease by applying a trained model corresponding to the first type to input data based on the user movement information.

3. The program described in claim 2, wherein the evaluating means evaluates the user for symptoms of the target disease by applying a trained model corresponding to the first event and the target disease to input data based on the user movement information.

4. The program according to claim 1, wherein the evaluating means selects a pattern corresponding to the first event and evaluates the user for symptoms of the target disease by comparing the pattern with the user movement information.

5. The program according to claim 4, wherein the evaluating means selects a pattern corresponding to the first event and the target disease, and evaluates the user for symptoms of the target disease by comparing the pattern with the user movement information.

6. The program of claim 1, wherein the user motion information indicates at least one of a position, a velocity, or an acceleration of a joint of the user.

7. The program of claim 1, which causes the computer to function as a means for acquiring at least one of facial expression information regarding the user's facial expression, gaze information regarding the user's gaze, tremor information regarding the tremors of the user's limbs, or assistive device information regarding the user's use of a walking aid, obtained by analyzing the image, and wherein the evaluating means evaluates the user for signs of the target disease based further on at least one of the facial expression information, the gaze information, the gaze information, and the assistive device information.

8. The program according to claim 1, which causes the computer to function as a means for acquiring voice information regarding the user's voice while performing the first type of exercise, and the evaluating means further evaluates the user for symptoms of the target disease based on the voice information.

9. The program described in claim 1, which causes the computer to function as: a means for selecting a second type of exercise from among the plurality of exercise types that is associated with the target disease and different from the first type when the user's evaluation result satisfies a predetermined condition; and a means for presenting information about the second type to the user.

10. The program according to claim 1, causing the computer to function as a means for presenting to the user information on medical institutions associated with the target disease when the user's evaluation result satisfies a predetermined condition.

11. The program according to claim 1, causing the computer to function as a means for presenting to the user information relating to tests associated with the target disease when the user's evaluation result satisfies a predetermined condition.

12. An information processing device comprising: a means for acquiring event information relating to a first event, which is an event selected from a plurality of event types by a user, a related person, or an algorithm; a means for acquiring user movement information relating to the movement of the user obtained by analyzing images of the user performing the first event; and a means for evaluating the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first event, based at least on the user movement information and characteristics relating to the symptoms of the target disease.

13. A method in which a computer executes the steps of: acquiring event information relating to a first event, the first event being an event selected from a plurality of event types by a user, a related person, or an algorithm; acquiring user movement information relating to the movement of the user obtained by analyzing images of the user performing the first event; and evaluating the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease, associated with the first event, based at least on the user movement information and characteristics relating to symptoms of the target disease.

14. A system comprising a first information processing device and a second information processing device, wherein the first information processing device comprises: a means for acquiring event information relating to a first event, which is an event selected from a plurality of event types by a user, a related person, or an algorithm; a means for acquiring user movement information relating to the movement of the user obtained by analyzing images taken of the user performing the first event; a means for evaluating the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first event, based at least on the user movement information and characteristics relating to the symptoms of the target disease; and a means for transmitting information based on the results of the user's evaluation to the second information processing device.

Images