Information processing device, method, program, and system

Abstract

To provide a technique capable of evaluating a symptom of abnormality of a user while suppressing a burden applied to the user.SOLUTION: A program of one embodiment causes a computer to function as: means for acquiring item information on a first item which is an exercise item selected from a plurality of exercise items by a user, a related party, or an algorithm; means for acquiring user movement information on the user's movement obtained by analyzing images photographing the user who executes an exercise of the first item; and means for evaluating the user about a symptom of a target disorder which is at least one of a cranial nerve disease or a peripheral nerve / muscle disorder associated with the first item, on the basis of at least the user movement information and characteristics on the symptoms of the target disorder.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a method, a program, and a system. [Background technology]

[0002] For example, early detection and treatment of neurological diseases such as dementia may slow the progression of the disease and alleviate its symptoms. The symptoms of neurological diseases vary, but for example, the disease may affect the patient's walking, causing walking disorders.

[0003] Patent Document 1 describes a technical idea that focuses on posture, vibration, and gait among the patient's physical movements, and aims to simply and quantitatively measure movement disorders that appear in at least one of these, in order to automatically diagnose neurological disorders such as Parkinson's disease. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-184964 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the technical concept described in Patent Document 1, the patient is forced to wear more than a dozen acceleration sensors on his or her body to measure posture. Therefore, measuring the movement disorder of a patient on a daily basis using this technical concept places a large burden on the patient and is not realistic.

[0006] An object of the present disclosure is to provide a technique for evaluating signs of abnormality in a user while minimizing the burden on the user. [Means for solving the problem]

[0007] A program of one embodiment of the present disclosure causes a computer to function as: means for acquiring event information regarding a first event, which is an event selected from a plurality of events by a user, a related person, or an algorithm; means for acquiring user movement information regarding the user's movement obtained by analyzing images taken of the user performing the first event; and means for evaluating the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first event, based at least on the user movement information and characteristics related to the symptoms of the target disease. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing a configuration of a client device according to the present embodiment. [Diagram 3] FIG. 2 is a block diagram showing a configuration of a server according to the present embodiment. [Figure 4] FIG. 1 is an explanatory diagram of one aspect of the present embodiment. [Diagram 5] FIG. 2 is a diagram showing a data structure of an exercise event database according to the present embodiment. [Figure 6] FIG. 2 is a diagram illustrating a data structure of a model database according to the present embodiment. [Figure 7] 4 is a flowchart of a symptom evaluation process according to the present embodiment. [Figure 8] 11A and 11B are diagrams showing examples of screens displayed in the symptom evaluation process of the present embodiment. [Figure 9] 11A and 11B are diagrams showing examples of screens displayed in the symptom evaluation process of the present embodiment. [Figure 10] 11A and 11B are diagrams showing examples of screens displayed in the symptom evaluation process of the present embodiment. [Figure 11] FIG. 13 is a diagram showing a data structure of a pattern database in the first modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and the repeated description will be omitted.

[0010] (1) Information Processing System Configuration The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to the present embodiment.

[0011] As shown in FIG. 1, the information processing system 1 includes a client device 10 and a server 30.

[0012] Here, the number of client devices 10 varies depending on, for example, the number of users. Therefore, the number of client devices 10 may be two or more. Furthermore, terminals of people related to each user may also be included in the information processing system 1.

[0013] The interested parties may include, for example, at least one of the following: · User's family A person who plans / plans the user's exercise regimen (typically a medical professional (e.g., a doctor (which may include the user's personal physician), a nurse, a pharmacist, a physical therapist, an occupational therapist, a clinical laboratory technician), or a nutritionist) The person in charge of the user's exercise regimen (typically a medical professional, nutritionist, or trainer)

[0014] The client device 10 and the server 30 are connected via a network (eg, the Internet or an intranet) NW.

[0015] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

[0016] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a server computer.

[0017] (1-1) Client device configuration The configuration of the client device will now be described with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0018] 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

[0019] The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of a Read Only Memory (ROM), a Random Access Memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0020] The programs include, for example, the following programs: ·OS (Operating System) programs · Programs for applications that process information (e.g. web browsers, therapeutic apps, rehabilitation apps, or fitness apps) Here, the diseases that are the target of therapeutic or rehabilitation apps are diseases in which exercise may contribute to improving symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

[0021] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0022] The processor 12 is a computer that realizes the functions of the client device 10 by running a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Gate Array)

[0023] The input / output interface 13 is configured to obtain information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10.

[0024] The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof. The output device is, for example, a display 15, a speaker, or a combination thereof.

[0025] The communication interface 14 is configured to control communications between the client device 10 and an external device (eg, another client device 10 (eg, a terminal of a participant), a server 30). Specifically, the communication interface 14 may include a module for communication (eg, a WiFi module, a mobile communication module, a Bluetooth module, or a combination thereof).

[0026] The display 15 is configured to display an image (still image or moving image). The display 15 is, for example, a liquid crystal display or an organic EL display.

[0027] The camera 16 is configured to capture images and generate image signals.

[0028] The depth sensor 17 is, for example, a LIDAR (Light Detection And Ranging) sensor. The depth sensor 17 is configured to measure a distance (depth) from the depth sensor 17 to a surrounding object (for example, a user). Note that the depth sensor 17 is not essential and may be removed from the client device 10.

[0029] The microphone 18 is configured to receive sound waves and generate a sound signal, and is preferably placed in the vicinity of the user's body (particularly the respiratory tract), such as an earphone microphone.

[0030] The acceleration sensor 19 is configured to detect acceleration.

[0031] (1-2) Server configuration The configuration of the server will now be described with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0032] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0033] The storage device 31 is configured to store programs and data, and is, for example, a combination of a ROM, a RAM, and a storage.

[0034] The programs include, for example, the following programs: -OS programs Application programs that perform information processing

[0035] The data includes, for example, the following data: Databases referenced in information processing Results of information processing

[0036] The processor 32 is a computer that realizes the functions of the server 30 by starting a program stored in the storage device 31. The processor 32 is, for example, at least one of the following. ·CPU GPU ·ASIC FPGA

[0037] The input / output interface 33 is configured to obtain information (eg, a user's instruction) from an input device connected to the server 30 and to output information to an output device connected to the server 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0038] The communication interface 34 is configured to control communications between the server 30 and external devices (eg, the client device 10).

[0039] (2) One aspect of the embodiment An embodiment of the present invention will now be described with reference to Fig. 4, which is an explanatory diagram of an embodiment of the present invention.

[0040] As shown in Fig. 4, the client device 10 senses the user US1 during exercise. Although not shown, a wearable device equipped with, for example, an acceleration sensor may also sense the user US1 during exercise. The user US1 is typically a person who undergoes exercise therapy, for example, a participant in a (cardiac) rehabilitation program or an exercise instruction program, but is not limited to this.

[0041] The user US1 is performing an exercise type (hereinafter referred to as a "first type") selected from a plurality of available exercise types by the user US1, a related person, or an algorithm. Here, the available exercise types are exercise types associated with any disease in the storage device 31 of the server 30 or another storage device accessible by the server 30.

[0042] The disease associated with the athletic event may be a cranial nerve disease, a peripheral nerve / muscle disease, or a combination thereof. The cranial nerve disease refers to a cranial nerve disease, particularly an intracranial disease, and may include at least one of dementia (e.g., Alzheimer's type, vascular, or Lewy body type), normal pressure hydrocephalus, or cerebellar disease. The peripheral nerve / muscle disease may include at least one of Parkinson's disease, myositis, muscular dystrophy, or Charcot-Marie-Thu's disease.

[0043] As an example, the camera 16 captures an external view (e.g., the whole body) of the user US1 during exercise from the front or oblique front at a distance of, for example, about 2 m. The camera 16 may be installed at an appropriate height using a tripod or other height adjustment means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. It is also possible to generate three-dimensional video data by combining, for example, video data (two-dimensional) generated by the camera 16 and, for example, depth data generated by the depth sensor 17. The microphone 18 receives sounds (e.g., sounds generated by breathing or speaking) emitted by the user US1 during exercise and generates a sound signal. The acceleration sensor 19 measures the acceleration of the user US1 during exercise.

[0044] The client device 10 acquires various sensing data and performs analysis as necessary. As an example, the client device 10 may refer to video data acquired from the camera 16 and analyze the body movement of the user US1 during exercise (particularly, the movement of the skeleton or other feature points over multiple points of time, and the state of the skeleton or other feature points at a single point of time). The client device 10 may further refer to depth data acquired from the depth sensor 17 to analyze the body movement of the user US1 during exercise. The client device 10 transmits user data including at least one of the sensing data or the analysis result of the sensing data to the server 30. However, a part or all of such analysis may be performed by the server 30.

[0045] The server 30 evaluates the user US1 for at least one of cranial nerve disease or peripheral nerve / muscle disease (hereinafter referred to as "target disease") associated with the first category, based on the user data acquired from the client device 10. Specifically, the server 30 performs the evaluation based on the characteristics related to the symptoms of the target disease (e.g., a specific type of movement disorder) and the analysis results of the user's body movements.

[0046] The server 30 generates information based on the result of the evaluation and presents it to the user US1 via the client device 10. Alternatively, the server 30 may present it to the above-mentioned related person instead of the user US1. In this way, when the user US1 is evaluated to have a symptom of the target disease, it is possible to encourage the user US1 to visit a medical institution or undergo an examination, for example. In addition, according to this embodiment, the user US1 is not forced to wear a large number of sensors (for example, attaching acceleration sensors to various parts of the body) or to perform a very limited type of exercise (for example, only walking), so the burden on the user US1 is light. Therefore, such evaluation is easy to perform on a daily basis (for example, as part of exercise therapy), and can contribute to early detection of the target disease.

[0047] (3) Database The databases of this embodiment are stored in the storage device 31 or another storage device accessible by the server 30.

[0048] (3-1) Sports event database The exercise event database of this embodiment will be described below. Fig. 5 is a diagram showing the data structure of the exercise event database of this embodiment.

[0049] The exercise type database stores exercise type information. The exercise type information is information related to the exercise type (for example, the above-mentioned available exercise types). Here, in this embodiment, the exercise types include exercise types that can be performed without using a device capable of adjusting the exercise load, such as gymnastics, bodyweight training, dancing, walking, running, and treadmills. These exercise types include types performed in a standing position, and therefore are rich in variation. Furthermore, in this embodiment, the exercise load of these exercise types can be adjusted through the form (for example, the range of motion of a part, the degree of opening of the arms or legs, etc.), pace, number of repetitions, or the time or number of rest periods. However, the exercise types in this embodiment can also include exercise types performed using a device capable of adjusting the exercise load, such as an ergometer or strength training using training equipment.

[0050] 5, the exercise event database includes an "ID" field, a "name" field, and an "exercise load" field. Each field is associated with the others.

[0051] The "ID" field stores an exercise type ID. The exercise type ID is information for identifying the exercise type corresponding to the record.

[0052] The "name" field stores information about the name of the exercise type. The exercise type name information is information about the name of the exercise type corresponding to the record.

[0053] The "exercise load" field stores exercise load information. The exercise load information is information on the standard exercise load of the exercise type corresponding to the record. The standard load is, for example, information on the exercise load when a person with standard physical function performs the corresponding exercise type. As an example, the exercise load may be derived by actually measuring the exercise load when a plurality of people perform the corresponding exercise type and statistically processing (for example, averaging) the measurement results, or may be obtained by referring to the exercise load set for the exercise type by a third party. The exercise load information may be managed in units finer than the exercise type. As an example, the exercise load information may be managed for each variation such as the form, pace, number of reps, or time or number of rests for each exercise type. The "exercise load" field is not essential and may be removed from the exercise type database.

[0054] (3-2) Model Database The model database of this embodiment will now be described with reference to Fig. 6, which is a diagram showing the data structure of the model database of this embodiment.

[0055] The model database stores model information, which is information about a trained model that makes inferences about one or more disease symptoms based on input data related to at least body movements (e.g., skeletons).

[0056] 6, the model database includes a "model ID" field, a "model details" field, and an "applicable conditions" field. Each field is associated with the others.

[0057] The "Model ID" field stores a model ID. The model ID is information for identifying the trained model corresponding to the corresponding record.

[0058] The "model details" field stores model details. The model details are information about the details of the trained model corresponding to the record. The details include information that can identify the structure of the trained model (e.g., the layer structure in a neural network, the connection relationships between nodes, and the weights of each edge). The details may include, for example, information indicating the values ​​of each parameter that defines the structure of the trained model, or may include information indicating the location where the values ​​are stored.

[0059] The "applicable condition" field stores applicable condition information. The applicable condition information is information on the conditions under which the trained model corresponding to the record is applicable. As an example, the applicable condition information may include information capable of identifying an exercise type (e.g., the above-mentioned type ID) and information capable of identifying a disease (i.e., a target disease). As information capable of identifying a disease, for example, a disease ID that uniquely identifies a single disease or a group of diseases consisting of multiple diseases may be defined.

[0060] (3-3) Other databases Other databases may be constructed, and as an example, a user profile database is described.

[0061] The user profile database stores user profile information. The user profile information is information about a profile of a user (for example, a person receiving exercise therapy) of the information processing system 1.

[0062] The user profile database may store records that include, for example, at least one of the following information: User ID User name information ·Physical information - Information on the appropriate exercise load for the user Information indicating the user's affiliation

[0063] The user ID is information for identifying the user corresponding to the corresponding record. The user name information is information relating to the name of the user (eg, name, account name, etc.) corresponding to the corresponding record. The physical information is information about the body (functions) of the user corresponding to the corresponding record. As an example, the physical information may include information about the user's age, sex, weight, height, illnesses, and the like.

[0064] (4) Information processing The information processing of this embodiment will be described.

[0065] (4-1) Symptom evaluation processing The symptom evaluation process of this embodiment will be described. Fig. 7 is a flowchart of the symptom evaluation process of this embodiment. Fig. 8 is a diagram showing an example of a screen displayed in the symptom evaluation process of this embodiment. Fig. 9 is a diagram showing an example of a screen displayed in the symptom evaluation process of this embodiment. Fig. 10 is a diagram showing an example of a screen displayed in the symptom evaluation process of this embodiment.

[0066] The symptom evaluation process starts when, for example, any one of the following start conditions is met. The symptom evaluation process was invoked by another process. The user or a person associated with the user performed an operation to invoke the symptom evaluation process. The client device 10 enters a predetermined state (for example, a predetermined application is started). The appointed time has arrived. A certain amount of time has passed since a certain event.

[0067] As shown in FIG. 7, the client device 10 acquires sensing data (S110). Specifically, the client device 10 may enable the operation of the camera 16 to start taking a video of the user exercising (hereinafter referred to as "user video"). The client device 10 may also enable the operation of the depth sensor 17 to start measuring the distance from the depth sensor 17 to each part of the user exercising (hereinafter referred to as "user depth"). The client device 10 may enable the operation of the microphone 18 to start collecting sound (e.g., sound generated by the user's voice or breathing (hereinafter referred to as "user sound")). The client device 10 may enable the operation of the acceleration sensor 19 to start measuring acceleration.

[0068] Then, the client device 10 acquires the sensing data from each sensor. Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 may acquire user video data from the camera 16, user depth data from the depth sensor 17, user sound data from the microphone 18, and user acceleration data relating to the acceleration of the user (hereinafter, "user acceleration") from the acceleration sensor 19.

[0069] After step S110, the client device 10 executes generation of user data (S111). Specifically, the client device 10 generates user data based on the sensing data acquired in step S110. The user data may include at least one of the following: Data acquired in step S111 (e.g., user video data, user depth data, user sound data, or user acceleration data) Data obtained by processing the data acquired in step S111 Data acquired by analyzing the data acquired in step S111 (for example, movement information (one example of which is skeletal data), facial expression data, gaze data, tremor data, assistive device data, voice data, or a combination thereof) Information that can identify the type of exercise (first type) that the user was performing in step S110

[0070] After step S111, the client device 10 executes transmission of user data (S112). Specifically, the client device 10 transmits to the server 30 the user data generated in step S111.

[0071] After step S112, the server 30 performs symptom evaluation (S130). Specifically, the server 30 receives the user data transmitted by the client device 10 in step S112. The server 30 acquires user skeletal information based on the user data acquired from the client device 10. The user skeletal information is information about the user's skeleton obtained by analyzing the user video data. In addition to the user video data, user depth data may be referenced to generate the user skeletal information. The user skeletal information may be included in the user data, or may be generated by the server 30 analyzing data included in the user data. Note that the user skeletal information is an example of information about the movement of the user's body during exercise (hereinafter referred to as "user movement information"), and may be used together with information about the movement of other feature points, or may be replaced with this.

[0072] The user skeletal information is information (e.g., data such as feature amounts) about the user's skeleton during exercise. The user skeletal information includes, for example, information about the position, speed, or acceleration of each part of the user's body (which may include information about changes in the parts of the muscles used by the user, or about shaking of the user's trunk). The user skeletal information can be obtained by analyzing the skeleton of the user during exercise with reference to the user video data (or the user video data and the user depth data). As an example, Vision, which is an SDK for iOS (registered trademark) 14, or other skeleton detection algorithms (e.g., OpenPose, PoseNet, MediaPipe Pose) can be used to analyze the skeleton.

[0073] The results of motion detection, such as skeletal detection, can be used for quantitative evaluation, qualitative evaluation, or a combination of both of these of exercise. As a first example, the results of motion detection can be used to count the number of reps. As a second example, the results of motion detection can be used to evaluate the form of the exercise, or the appropriateness of the load applied by the exercise. For example, when the exercise is squats, the results of motion detection can be used to evaluate whether the knees are sticking out too far forward, resulting in a dangerous form, or whether the hips are lowered firmly and deeply to apply a sufficient load.

[0074] The server 30 also acquires information about the first event (for example, information that can identify the first event, such as an event ID corresponding to the first event). Such information may be included in the user data. Alternatively, the information may be transmitted from the terminal of the participant to the server 30 or generated by the server 30 during the process in which the first event is selected by the participant or an algorithm.

[0075] The server 30 evaluates the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first type, based on at least the user movement information and characteristics related to the symptoms of the target disease.

[0076] As a first example of symptom evaluation (S130), the server 30 refers to the model database (FIG. 6) and extracts records including a value of an event ID corresponding to the first event as an application condition. In this case, the trained model corresponding to each record constituting the extraction result corresponds to a feature related to a symptom of the target disease. The server 30 evaluates the user for symptoms of the target disease by applying the trained model to input data based on user movement information.

[0077] The trained model focuses on the presence or absence of characteristic skeletal or other feature postures or movements observed as symptoms of the corresponding disease, and makes inferences regarding the presence or absence of signs of the disease, or the degree of possibility of the presence of signs of the disease. The trained model accepts input data based on movement information (e.g., skeletal information) and outputs evaluation results. The evaluation results may be binary data indicating the presence or absence of a sign of a disease, or may be multi-valued data indicating the possibility of the presence of such a sign in multiple stages. The trained model corresponds to a trained model created by supervised learning using a training dataset, or a derived model or distilled model of the trained model.

[0078] As a second example of symptom evaluation (S130), when a target disease is specified, the server 30 refers to the model database (FIG. 6) and extracts records including an event ID corresponding to the first event and a disease ID value corresponding to the specified target disease as application conditions. In this case, the trained model corresponding to each record constituting the extraction result corresponds to a feature related to the symptom of the target disease. The server 30 evaluates the user for symptoms of the target disease by applying the trained model to input data based on user movement information. The target disease may be specified by the user, a related person of the user, or an algorithm. Information capable of identifying the specified target disease may be included in the user data, or may be transmitted from a related person's terminal to the server 30, or may be generated by the server 30.

[0079] After step S130, the server 30 executes information generation (S131). Specifically, the server 30 generates information to be presented to the user based on the result of the evaluation in step S130.

[0080] As a first example of information generation (S131), when an evaluation result is obtained indicating that there is a symptom of any of the target diseases or that the possibility of there being any of the target diseases exceeds a threshold (an example of "when the user's evaluation result satisfies a predetermined condition"), the server 30 may select an exercise type (hereinafter referred to as "second exercise type") to be recommended to the user next, and generate information (e.g., an exercise ID) that can identify the selected second exercise type. As an example, the server 30 may select an exercise type other than the first exercise type as the second exercise type among the exercise types associated with the target diseases evaluated in step S131 as having a symptom or the possibility of having a symptom exceeding a threshold. The server 30 may further select the second exercise type by taking into consideration the exercise load suitable for the user and the exercise load of each exercise type. In this way, by sensing the user performing the second exercise type recommended in this way again and performing the evaluation of the symptoms (S130) again, a more reliable evaluation result can be obtained. In other words, it can be confirmed whether the symptoms observed when performing the first exercise type are merely coincidental, or whether similar symptoms are observed regardless of the exercise type.

[0081] As a second example of generating information (S131), when an evaluation result is obtained indicating that there is a symptom of any of the target diseases, or that the possibility of there being a symptom of any of the target diseases exceeds a threshold, the server 30 may generate information capable of identifying a medical institution associated with the target disease. As an example, information on medical institutions suitable for diagnosing the target disease is associated with each target disease and stored in a database in advance, and the server 30 can generate information by referring to the database. The server 30 may narrow down the medical institutions by considering information on the user's place of residence or current location in addition to the target disease.

[0082] As a third example of generating information (S131), when an evaluation result is obtained indicating that there is a symptom of any of the target diseases, or that the possibility of there being a symptom of any of the target diseases exceeds a threshold, the server 30 may generate information recommending the implementation of a test associated with the target disease (e.g., a screening test such as the Hasegawa Dementia Scale).As an example, information on tests suitable for each target disease is associated with the target disease and stored in a database in advance, and the server 30 can generate information by referring to the database.

[0083] As a fourth example of generating information (S131), when an evaluation result indicating that there is a symptom of any of the target diseases or that the possibility of there being a symptom of any of the target diseases exceeds a threshold is obtained, the server 30 may combine two or more of the above first to third examples. For example, the server 30 may perform the above first example one or more times, and when the total number or percentage of times that an evaluation result indicating that there is a symptom of the target disease or that the possibility exceeds a threshold is obtained reaches a predetermined value, the server 30 may perform at least one of the above second or third examples.

[0084] In addition, if the evaluation result indicates that there are no symptoms of any of the target diseases, or that the possibility of the presence of symptoms of any of the target diseases does not exceed a threshold, the server 30 may omit the steps of generating information (S131) ​​to presenting the information (S132).

[0085] After step S131, the server 30 executes the presentation of information (S132). Specifically, the server 30 transmits the information generated in step S131 to the client device 10. The server 30 can also present information to a related person of the user in addition to the user or instead of the user. In this case, the server 30 can transmit the information generated in step S132 to the terminal of the related person.

[0086] After step S132, the client device 10 executes screen display (S113). Specifically, the client device 10 receives the information transmitted by the server in step S131. The client device 10 displays a screen based on the received information (not limited to an application screen, but may include a notification (application notification, or a message such as an e-mail or chat)) on the display 21.

[0087] As a first example of screen display (S113), the client device 10 displays the screen of Fig. 8 on the display 21. The screen of Fig. 8 includes objects J20 to J22.

[0088] The object J20 displays information about the second event. The object J20 also receives a user instruction to start each exercise event or to play a model video of each exercise event. When the object J20 is selected, the client device 10 may re-execute the sign evaluation process of this embodiment with the exercise event (second event) corresponding to the object J20 as the new first event, or may play a model video of the exercise event.

[0089] The object J21 accepts a user instruction to select an exercise type from among the second types. When the object J21 is selected, the client device 10 may display, for example, a list of exercise types other than the second type and accept a user instruction to select an exercise type. In response to the acceptance of such a user instruction, the client device 10 may play a model video of the exercise type selected by the user, or, if the exercise type is associated with a target disease, may re-execute the symptom evaluation process of this embodiment with the exercise type as a new first type.

[0090] The object J22 receives a user instruction to end the exercise. When the object J22 is selected, the client device 10 ends the symptom evaluation process of this embodiment.

[0091] As a second example of screen display (S113), the client device 10 displays the screen of Fig. 9 on the display 21. The screen of Fig. 9 includes objects J30 to J31.

[0092] Object J30 displays information recommending a visit to a medical institution. The object J31 accepts a user instruction to view information about medical institutions associated with the target disease. In response to the user instruction, the client device 10 may place an object that displays information about medical institutions associated with the target disease on the screen, or transition to a screen that displays such information.

[0093] As a third example of screen display (S113), the client device 10 displays the screen of Fig. 10 on the display 21. The screen of Fig. 10 includes objects J40 to J41.

[0094] Object J40 displays information recommending the performance of tests related to the target disease. The object J41 receives a user instruction to start a test associated with a target disease. In response to the reception of the user instruction, the client device 10 may place an object on the screen that displays information about the test associated with the target disease (e.g., questions included in the test, or a link to access a website where the test can be taken) or transition to a screen that displays such information.

[0095] After step S113, the client device 10 may end the symptom evaluation process (FIG. 7).

[0096] (5) Training Dataset We will explain the teacher dataset that can be used for supervised learning to construct the trained model of this embodiment.

[0097] The training data set includes multiple training data. The training data is used for training or evaluating a model to be learned (hereinafter, referred to as a "target model"). The training data includes a sample ID, input data, and correct answer data.

[0098] The sample ID is information for identifying the training data.

[0099] The input data is data that is input to the target model during training or evaluation. The input data corresponds to examples used during training or evaluation of the target model. As an example, the input data is data (i.e., relatively dynamic data) regarding the subject's body movements during exercise (i.e., performing the type of exercise corresponding to the target model) and data (i.e., relatively static data) regarding the subject's health condition. At least a portion of the data regarding the subject's body movements is obtained by analyzing the subject's body movements with reference to the subject video data (or the subject video data and subject depth data).

[0100] The subject video data is data on a subject video of a subject during exercise. The subject video data can be acquired, for example, by capturing an external view (e.g., the whole body) of a subject during an examination related to exhaled gas (e.g., a CPX examination) from the front or diagonally in front (e.g., 45 degrees forward) with a camera (e.g., a camera mounted on a smartphone).

[0101] The subject depth data is data on the distance (depth) from the depth sensor to each part of the subject during exercise. The subject depth data can be acquired by operating the depth sensor when taking a video of the subject.

[0102] The subjects typically include those diagnosed with a specific disease or a symptom thereof and those diagnosed with no such disease or a symptom thereof. The diagnosis is usually performed by a doctor, but may be substituted by a judgment using an algorithm (which may include a trained model). However, for the same subject, data collected at the time when the subject was diagnosed with a specific disease or a symptom thereof and data collected at the time when the subject was diagnosed with no such disease or a symptom thereof may be used as teacher data. In addition, one of the subjects may be the same person as a user who is evaluated for symptoms of a specific disease when the information processing system 1 is operated.

[0103] In this example, the input data includes at least motion data. The motion data may include elements similar to the above-mentioned user motion information (e.g., skeletal information), and may be obtained by analyzing the motion (e.g., skeletal) of a subject during exercise with reference to subject video data. In addition to the subject video data, at least one of subject depth data or acceleration data measured by a wearable device attached to the subject may be referenced to obtain the motion data.

[0104] The correct answer data is data that corresponds to the correct answer for the corresponding input data (example question). The target model is trained (supervised learning) to output the correct answer data as close as possible to the input data. As an example, the correct answer data represents the presence or absence of a specific disease or its symptoms.

[0105] The correct answer data indicates, for example, the presence or absence of a specific disease or its symptoms for the subject from whom the corresponding input data was obtained. That is, the correct answer data corresponding to the input data obtained from a subject diagnosed with a specific disease or its symptoms has a value indicating the presence of the disease or its symptoms. On the other hand, the correct answer data corresponding to the input data obtained from a subject diagnosed with no specific disease or its symptoms has a value indicating the absence of the disease or its symptoms.

[0106] In addition to the motion data, at least one of the following may be added to the input data together with the motion data, or a trained model may be constructed that performs similar inference on input data that includes at least one of the following as an element. In this case, the input data when the trained model is operated will also include a similar element based on the user's sensing results. Facial expression data (an example of "facial expression information") - Gaze data (an example of "gaze information") Tremor data (an example of "tremor information") Assistive device data (an example of "assistive device information") Voice data (an example of "voice information") Includes health status data (an example of “health status information”).

[0107] Facial expression data is data (e.g., features) about a subject's facial expressions while exercising. Facial expression data can be analyzed by applying an algorithm or trained model to the subject video data. Alternatively, facial expression data for a training dataset can be obtained, for example, by humans labeling the subject videos.

[0108] Gaze data is data (e.g., features) about the subject's gaze during exercise. Gaze data can be analyzed by applying an algorithm or trained model to the subject's video data.

[0109] Tremor data is data (e.g., features) about a subject's tremor during motion. Tremor data can be analyzed by applying an algorithm or trained model to subject video data. Alternatively, tremor data for a training dataset can be obtained, for example, by human labeling of subject videos.

[0110] Assistive device data is data (e.g., features) about how a subject uses an assistive device (e.g., a walking aid such as a cane, or other equipment used in exercise) while exercising. Assistive device data can be analyzed by applying an algorithm or trained model to subject video data. Alternatively, assistive device data for a training dataset can be obtained, for example, by human labeling of subject videos.

[0111] Voice data is data (e.g., features) about the subject's voice while exercising. Voice data can be analyzed by applying an algorithm or trained model to sound data collected by microphones placed around the subject. Alternatively, aid data for a training dataset can be obtained by, for example, human labeling of the subject's voice.

[0112] The health condition data is data related to the subject's health condition. The health condition data can be obtained in various ways. The subject's health condition data may be obtained before, during, or after the subject's exercise. The subject's health condition data may be obtained based on a declaration from the subject or the subject's doctor, may be obtained by extracting information linked to the subject in a medical information system, or may be obtained via the subject's app (e.g., a health care app).

[0113] The health condition includes at least one of the following: ·age ·sex ·height ·body weight ·Body fat percentage Muscle mass ·Bone density History of current illness ·Past history Medication history Surgical history Life history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty score, etc.) Family history Respiratory function test results Results of tests other than respiratory function tests (e.g., blood tests, urine tests, electrocardiograms (including Holter ECGs), cardiac ultrasounds, X-rays, CT scans (including cardiac morphological CT and coronary artery CT), MRI scans, nuclear medicine tests, PET scans, etc.) Data obtained during cardiac rehabilitation (including Borg index)

[0114] It is also possible to construct a trained model for each of a plurality of health condition categories based on (at least a part of) the subject's health condition. In this case, (at least a part of) the user's health condition may be referenced to select the trained model. In this modification, the input data for the trained model may be data that is not based on the user's health condition, or may be data that is based on the user's health condition and the user's video.

[0115] (5) Summary As described above, the server 30 of this embodiment acquires user motion information related to the user's motion obtained by analyzing an image of the user performing a first type of exercise, which is an exercise type selected from a plurality of exercise types by the user, a related person, or an algorithm. The server 30 evaluates the user for signs of a target disease, which is at least one of a cranial nerve disease or a peripheral nerve / muscle disease associated with the first type, based on at least the user motion information and the features related to the symptoms of the target disease (a trained model in this embodiment). This reduces the burden on the user, making it easier to evaluate the signs of the target disease on a daily basis, which can contribute to early detection of the target disease.

[0116] The server 30 may evaluate the user for symptoms of the target disease by applying the trained model corresponding to the first type to the input data based on the user movement information. This makes it possible to obtain a statistically valid evaluation result without creating an evaluation algorithm. In addition, by using the trained model corresponding to the first type, the evaluation is less susceptible to the influence of differences depending on the type of exercise, such as the posture or movement of the skeleton or other feature points, and therefore the accuracy of the evaluation can be improved.

[0117] The server 30 may evaluate the user for symptoms of the target disease by applying a trained model corresponding to the first sport and the target disease to input data based on the user movement information. By using a trained model corresponding to a combination of the first sport and the target disease, the evaluation accuracy can be improved since it is less affected by differences that depend on at least one of the sport or disease, such as the posture or movement of the skeleton or other feature points.

[0118] The user motion information may indicate at least one of the positions, velocities, and accelerations of the user's joints, thereby allowing the posture or motion of the user's skeleton to be quantitatively expressed and objectively evaluated.

[0119] The server 30 may acquire at least one of facial expression information on the user's facial expression, gaze information on the user's gaze, tremor information on the tremor of the user's limbs, or assistive device information on the user's use of a walking assistive device, which are obtained by analyzing the above-mentioned images. The server 30 may further evaluate the user for signs of the target disease based on at least one of the facial expression information, gaze information, gaze information, and assistive device information. This allows evaluation from perspectives other than movement, making it easier to prevent signs from being overlooked.

[0120] The server 30 may acquire voice information regarding the user's voice while performing the first type of exercise, and may evaluate the user for symptoms of the target disease based on the voice information. This allows the user to be evaluated from the perspectives of movement and voice, making it easier to prevent symptoms from being overlooked.

[0121] When the user's evaluation result satisfies a predetermined condition, the server 30 may select a second type of exercise that is associated with the target disease and different from the first type from among the multiple types of exercise, and present information about the second type to the user. Then, by similarly evaluating symptoms for the second type of exercise, a more reliable evaluation result can be obtained.

[0122] When the user's evaluation result satisfies a predetermined condition, the server 30 may present the user with information on medical institutions associated with the target disease, thereby encouraging the user to visit a medical institution that specializes in the target disease and thereby to detect the target disease early.

[0123] When the user's evaluation result satisfies a predetermined condition, the server 30 may present the user with information on tests associated with the target disease, thereby encouraging the user to carry out established tests for the target disease and thereby to detect the target disease early.

[0124] (6) Variations A modification of this embodiment will now be described.

[0125] (6-1) Variation 1 A description will be given of Modification 1. Modification 1 is an example in which a pattern is used instead of a trained model to evaluate a symptom.

[0126] In the first modification, a pattern database is used in addition to or instead of the model database (FIG. 6). FIG. 11 is a diagram showing the data structure of the pattern database in the first modification.

[0127] The pattern database stores pattern information. The pattern information is information on a pattern of movement information (e.g., skeletal information) that serves as a criterion for evaluating one or more symptoms of a disease. Such a pattern can be determined, for example, by analyzing the characteristics of the skeleton or other features that appear when a patient with a particular disease or a person showing symptoms of the disease performs each exercise.

[0128] 6, the pattern database includes a "pattern ID" field, a "pattern details" field, and an "applicable condition" field. Each field is associated with the other fields.

[0129] The "Pattern ID" field stores a pattern ID. The pattern ID is information for identifying the pattern corresponding to the record.

[0130] The "pattern details" field stores pattern details. The pattern details are information about the details of the pattern corresponding to the record. The detailed information includes information that can identify a pattern (for example, information that defines the position, speed, or acceleration of a specific joint or the orientation of a specific joint, or conditions related to time-series changes of these, or information that indicates the location where such information is stored).

[0131] The "applicable condition" field stores applicable condition information. The applicable condition information is information on the conditions under which the pattern corresponding to the record is applicable. As an example, the applicable condition information can include information capable of identifying an exercise type (e.g., the above-mentioned type ID) and information capable of identifying a disease (i.e., a target disease). As information capable of identifying a disease, for example, a disease ID that uniquely identifies a single disease or a group of diseases consisting of multiple diseases can be defined.

[0132] As a first example of symptom evaluation (step S130 in FIG. 7), the server 30 refers to the pattern database (FIG. 11) and extracts records including an event ID value corresponding to the first event as an application condition. In this case, the patterns corresponding to the records constituting the extraction result correspond to characteristics related to symptoms of the target disease. The server 30 compares the patterns with the user movement information to evaluate the user regarding symptoms of the target disease.

[0133] As a second example of symptom evaluation (S130), when a target disease is specified, the server 30 refers to the pattern database (FIG. 11) and extracts records including an event ID corresponding to the first event and a disease ID value corresponding to the specified target disease as application conditions. In this case, the pattern corresponding to each record constituting the extraction result corresponds to a feature related to the symptom of the target disease. The server 30 evaluates the user for symptoms of the target disease by comparing such a pattern with the user movement information. The target disease may be specified by the user, a related person of the user, or an algorithm. Information capable of identifying the specified target disease may be included in the user data, or may be transmitted from a related person's terminal to the server 30, or may be generated by the server 30.

[0134] As described above, the server 30 of the first modification may select a pattern corresponding to the first type of sport and compare the pattern with the user movement information to evaluate the user for symptoms of a target disease. By using the pattern corresponding to the first type of sport, the evaluation is less susceptible to the effects of differences depending on the type of sport, such as the posture or movement of the skeleton or other feature points, and therefore the accuracy of the evaluation can be improved.

[0135] The server 30 may select a pattern corresponding to the first sport and the target disease, and compare the pattern with the user movement information to evaluate the user for symptoms of the target disease. By using a pattern corresponding to a combination of the first sport and the target disease, the evaluation can be more accurate because it is less affected by differences that depend on at least one of the sport or disease, such as the posture or movement of the skeleton or other features.

[0136] (7) Other modifications The storage device 11 may be connected to the client device 10 via a network NW. Each input device or output device may be built into the client device 10. The storage device 31 may be connected to the server 30 via the network NW.

[0137] In the embodiment, the information processing system 1 is implemented by a client / server system. However, the information processing system 1 may be implemented by a peer-to-peer system or a standalone computer. As an example, the client device 10 may evaluate the symptoms.

[0138] Each step of the above information processing can be executed by either the client device 10 or the server 30. As an example, the server 30, instead of the client device 10, may acquire user movement information by analyzing sensing data.

[0139] In the above description, an example has been given in which a user video is captured using the camera 16 of the client device 10. However, the user video may be captured using a camera other than the camera 16. An example has been given in which a user depth is measured using the depth sensor 17 of the client device 10. However, the user depth may be measured using a depth sensor other than the depth sensor 17.

[0140] It is also possible to use the acceleration data as part of the input data for the trained model described in this embodiment or the modified example. Alternatively, the user's movement (e.g., skeleton) may be analyzed by referring to the acceleration data. The acceleration data may be acquired, for example, by the acceleration sensor 19 when shooting the user's video, or by an acceleration sensor mounted on a wearable device (not shown) worn by the user.

[0141] The information processing system 1 of the present embodiment and each modified example is also applicable to a video game in which the game progress is controlled according to the player's body movement. The video game may be a mini-game that can be played while the above-mentioned treatment app, rehabilitation app, or fitness app is being executed. As an example, the information processing system 1 performs estimation regarding the user's movement (e.g., skeleton) based on the user's video during game play. The estimation regarding the user's movement may be performed based on at least one of the user depth or the user acceleration in addition to the user's video. The information processing system 1 evaluates the degree to which the user's posture during exercise (e.g., gymnastics) matches an ideal posture (model) based on the result of the estimation regarding the user's movement. The information processing system 1 may determine any one of the following depending on the result of this evaluation (e.g., a numerical value indicating the degree to which the user's posture matches the ideal posture). This can enhance the effect of the video game on improving the user's health. The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) offered to the user The quality (e.g., type) or quantity of video game benefits (e.g., in-game currency, items, bonuses) provided to users Game parameters related to the progression of a video game (e.g. score, damage)

[0142] In addition to or instead of the microphone 18, a microphone (a microphone provided in or connected to the wearable device) of a wearable device (not shown) worn by the user may receive sound waves emitted by the user when capturing a user video and generate sound data. The sound data may constitute input data for the trained model described in this embodiment or the modified example. The sound emitted by the user is, for example, a sound generated by the user's breathing or speaking.

[0143] Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited to the above embodiment. Furthermore, the above embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above embodiment and the modified examples can be combined. [Explanation of symbols]

[0144] 1: Information processing system 10: Client device 11:Storage device 12: Processor 13: Input / Output Interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 21: Display 30: Server 31: Storage device 32 : Processor 33: Input / Output Interface 34: Communication interface

Claims

1. A program that causes a computer to function as: means for acquiring item information regarding a first item, which is an exercise item selected from a plurality of exercise items that are aerobic exercise or resistance training performed in a standing position by a user, a person concerned, or an algorithm; means for acquiring user movement information regarding the movement of the user obtained by analyzing an image of the user performing the exercise of the first item; means for evaluating the user regarding signs of a target disease, which is at least one of a cerebrovascular disease or a peripheral nerve / muscle disease associated with the first item, based on at least the user movement information and characteristics regarding symptoms of the target disease; a program.

2. The means for evaluating evaluates the user regarding signs of the target disease by applying a learned model corresponding to the first item to input data based on the user movement information. The program according to claim 1.

3. The means for evaluating evaluates the user regarding signs of the target disease by applying a learned model corresponding to the first item and the target disease to input data based on the user movement information. The program according to claim 2.

4. The means for evaluating evaluates the user regarding signs of the target disease by selecting a pattern corresponding to the first item and comparing the pattern with the user movement information. The program according to claim 1.

5. The means for evaluating evaluates the user regarding signs of the target disease by selecting a pattern corresponding to the first item and the target disease and comparing the pattern with the user movement information. The program according to claim 4.

6. The user movement information indicates at least one of the position, speed, or acceleration of the joints of the user. The program according to claim 1.

7. A program that causes the computer to function as means for acquiring at least one of expression information regarding the expression of the user, gaze information regarding the gaze of the user, tremor information regarding tremors of the hands and feet of the user, or auxiliary tool information regarding the usage of a walking aid for the user, obtained by analyzing the image, and the means for evaluating evaluates the user regarding signs of the target disease based on at least one of the expression information, the gaze information, the gaze information, and the auxiliary tool information. The program according to claim 1.

8. Function the computer as means for acquiring voice information regarding the user's voice during the execution of the first type of exercise, The means for evaluating evaluates the user regarding signs of the target disease further based on the voice information. The program according to claim 1.

9. The computer, When the evaluation result of the user satisfies a predetermined condition, as means for selecting a second type of exercise that is associated with the target disease and different from the first type among the plurality of exercise types, As means for presenting information on the second type of exercise to the user, Function as, the program according to claim 1.

10. Function the computer as means for presenting information on a medical institution associated with the target disease to the user when the evaluation result of the user satisfies a predetermined condition. The program according to claim 1.

11. Function the computer as means for presenting information on an examination associated with the target disease to the user when the evaluation result of the user satisfies a predetermined condition. The program according to claim 1.

12. The program according to claim 1, wherein the plurality of exercise types are composed of exercises that repeat the same movement in at least one of a specific form and a specific pace.

13. Means for acquiring item information regarding a first type of exercise selected from a plurality of exercise types that are aerobic exercise or resistance training performed in a standing position by a user, a related person, or an algorithm, Means for acquiring user movement information regarding the movement of the user obtained by analyzing an image of the user performing the first type of exercise, Means for evaluating the user regarding signs of a target disease that is at least one of a cerebrovascular disease or a peripheral nerve / muscle disease associated with the first type, based on at least the user movement information and characteristics regarding symptoms of the target disease An information processing apparatus comprising.

14. The computer, A step of acquiring item information regarding a first type of exercise selected from a plurality of exercise types that are aerobic exercise or resistance training performed in a standing position by a user, a related person, or an algorithm, A step of acquiring user movement information regarding the movement of the user obtained by analyzing an image of the user performing the first type of exercise, A step of evaluating the user based on at least the user movement information and characteristics related to the symptoms of a target disease, which is at least one of cerebrovascular diseases or peripheral nerve / muscle diseases associated with the first category. A method of executing the same. **Claim 15** A system including a first information processing device and a second information processing device, wherein the first information processing device has means for acquiring item information regarding a first item, which is an exercise item selected from a plurality of exercise items that are aerobic exercise or resistance training performed in a standing position by a user, a person related to the user, or an algorithm; means for acquiring user movement information regarding the movement of the user obtained by analyzing an image of the user performing the exercise of the first item; means for evaluating the user based on at least the user movement information and characteristics related to the symptoms of a target disease, which is at least one of cerebrovascular diseases or peripheral nerve / muscle diseases associated with the first category; and means for transmitting information based on the result of the evaluation of the user to the second information processing device. The system is provided with the above components. System.

Images