Self-assessment neurological health care system

Abstract

A system has a digital health avatar can substitute to a neurologist, nurse practitioner, physical assistant or research coordinator (in the context of a clinical trial) to run efficiently a routine neurological examination and document the report. This avatar supports the neurologist in his / her work, and will not replace the neurologist who still need to use his / her expertise to do the final diagnostic and manage the patient disease.

Description

[0001] SELF-ASSESSMENT NEUROLOGICAL HEALTH CARE SYSTEM

[0002] Related Application

[0003] This application claims the benefit of priority of US Provisional Application No. 63 / 531,501, filed on August 8, 2023, the entire content of which is relied upon and incorporated herein by reference in its entirety.

[0004] Background

[0005] The medical examination of a patient (e.g., with neurological disease) is done manually, such as by a neurologist nurse practitioner, physician assistant and in the context of a clinical trial, a research coordinator. However, such examinations are time-consuming and involve a wide range of tasks requiring different skill levels. For example, a medical examination can require basic tasks that require a low skill level (e.g., taking the patient’s name, age, temperature) that can be performed by an administrative staff or a medical technician; and also complicated tasks that require a high skill level (e.g., performing a physical or clinical examination and assessing patient conditions, and making a final determination of an illness or condition).

[0006] In the case of neurological diseases, such as Myasthenia Gravis (MG), examination is very nuanced since it involves detecting and assessing the patient’s physical state variables (e.g., physical, emotive, cognitive, and social states), environmental conditions (e.g., ambient temperature, humidity, airborne particle level), and physiological conditions (e.g., pulse, blood pressure, EKG, blood oxygen levels). And the standardized tests for MG require the detection and assessment of body movement (e.g., puffing the cheeks, tongue to cheek, sit to stand) and confirming that the patient properly performed the test.

[0007] As a result, there is great variation in examinations dependent on the examiner’s experience and time constraints of the examiner and the patient. Examinations may not even occur because of availability of an examiner. Further, even among expert clinicians there is variation in the performance of a standardized examination. Typically, +20% of the examination demonstrates inter-examiner variability. Studies of neurological examination correlating to objective radiological assessments show the sensitivity of the examination can be poor and highly subjective. Clinical trial outcome measures in myasthenia have correlations of 0.7 variation in scoring by trained, experienced examiners vary by 10% on certain scales. This impacts the number of patients required for clinical trials. Further, regardless of who is performing an examination, a research coordinator or a physician, the examinations are subjective.

[0008] In addition, imaging and laboratory testing cannot replace an in-person meeting with the direct evaluation of symptoms by the neurologist or trained examiner. [1, 2, 27, 28, 29, 30] Further, the neurological examination as performed in routine clinical practice is expensive, requiring a highly trained practitioner.

[0009] Attempts at medical examination systems have been made to use administer tests and perform evaluations on patients. For example, avatars and chatbots have been around for some time. But similar to the history of automatic speech recognition

[0031] , that bifurcated with deep learning algorithm, there is a before ChatGPT and after ChatGPT dramatic change of technology for conversational systems. The future development of applications, avatars and robots in mental health was discussed in

[0014] , Several attempts to design a medical triage system have been reported in conference papers [10, 11, 17], and in US Patent No. 10572626

[0018] to Balram. In addition, US Patent No. 10592503

[0019] to Kozloski et al. discloses a system that provides empathic answers in a question answering system.

[0010] ChatGPT does not appear to be a game changer in the field of conversational systems. It may be too early to determine all the consequences of that technology would have on the practice of medicine; But the limit and risk of Chat GPT has begun to be addressed in the literature

[0032] ,

[0011] According to Stewart Gandolf, CEO of Healthcare Success, prior to the opening of the ChatGPT4, Chapt GPT limitations and concerns for healthcare content include that its answers and information can be wrong or incomplete; it lacks empathy; it does not understand your target audience; it lacks experience; and it lacks specificity and clarity.

[0012] Summary

[0013] Those prior systems are not satisfactory for a number of reasons, including that they are not trusted by the patient. It is technically challenging to design a system that can be trusted by a patient. It is one object of the present disclosure to provide a system that obtains good examination data by checking that the patient answers the question and performs the physical test correctly. Another object is to compute the score from the video and speech with accuracy. Another object is to have a system that is accepted by the patient and the patient talks with the system (here an avatar) and the avatar deliver a human touch that help along these lines. The inventors have determined that the relationship between the medical examiner and patient in medical care is sensitive and must be addressed carefully

[0014] , In addition, those systems don’t account for the role of validated questionnaires in standard of care. Accordingly, there is a need to provide a medical examination system that is reliable, consistent, and trusted by the patient.

[0014] One benefit of the present system is that patients can get anytime anywhere an assessment of their neurological disease by going to the cloud to access the avatar application software. This can be used for triage by hospital systems to help the neurologist prioritize their patients, and spend more time with them. On the other hand, a patient would be able to frequently get his test and speed up diagnostic: today the shortage of neurologist in the USA is such that it may take month before getting a diagnostic.

[0015] Brief Description of the Drawings

[0016] FIG. 1 A shows an avatar simulation to perform a Myasthenia Gravis (MG) core examination.

[0017] FIG. IB shows the structure of a standard Myasthenia Gravis examination.

[0018] FIG. 1C is a snapshot of the video of the MG Core Examination, as captured by the patient’s processing device.

[0019] FIG. ID is an overall operating algorithm.

[0020] FIG. 2 shows management of core examination parts.

[0021] FIG. 3 shows the generation of patient instructions.

[0022] FIG. 4 is a patient report showing the visual control of patient eye and face examination. FIG. 5Ais a patient report showing visual control of sit to stand and hip flexion.

[0023] FIG. 5B is a patient reporting showing visual control of patient arm extension examination.

[0024] FIG. 6 shows avatar introducing itself to the patient in compliance with Al ethical principles edited by the ECU.

[0025] FIG. 7A shows management of the avatar, Patient Dialogue.

[0026] FIG. 7B shows management of the avatar, Answer Generation.

[0027] FIG. 8A shows a patient report. FIG. 8B shows the underlying principle of report construction.

[0028] FIG. 9Ais a diagram of a cyber-physical telehealth system, which includes a practitioner system and a patient system, according to exemplary embodiments;

[0029] FIG. 9B is a diagram of the patient system, which includes a patient computing system, a camera enclosure, and a hardware control box, according to exemplary embodiments;

[0030] FIG. 9C is a diagram of the patient system according to another exemplary embodiment;

[0031] FIG. 9D is a diagram of the patient computing system of FIG. 9A according to exemplary embodiments.

[0032] FIG. 10 is a diagram of the practitioner computing system of FIG. 9A.

[0033] FIG. 11 shows the digital twin data.

[0034] FIG. 12A shows an overview of patient data for MG examinations.

[0035] FIG. 12B shows practitioner classification of patient data for MG examinations.

[0036] FIG. 13A shows landmark points used to find the ROI of both eyes.

[0037] FIG. 13B shows the detection of eye gaze looking successively up, then left and right using a deep learning algorithm with the ROI.

[0038] Detailed Description

[0039] In describing the illustrative, non-limiting embodiments illustrated in the drawings, specific terminology will be resorted to for the sake of clarity. However, the disclosure is not intended to be limited to the specific terms so selected, and it is to be understood that each specific term includes all technical equivalents that operate in similar manner to accomplish a similar purpose. Several embodiments are described for illustrative purposes, it being understood that the description and claims are not limited to the illustrated embodiments and other embodiments not specifically shown in the drawings may also be within the scope of this disclosure.

[0040] FIGS. 9A-9D show a health examination system 100, for example a telehealth system. The examination system 100 includes a patient computing system 200 and a practitioner computing system 120. FIG. 9B shows details of the patient computing system 200, FIG. 9C shows the patient computing system 200 at a medical facility, FIG. 9D shows a patient computing system 500 having a patient processing device 502 (FIG. 9B), and FIG. 10 shows a practitioner computing system 120. FIGS. 9A-9D are diagrams of a remotely-controllable cyber- physical telehealth system 100 according to exemplary embodiments. The examination system 100 can be used in combination or integrated with, any suitable health examination system, such as the ones shown and described in PCT / US23 / 61783 (Publ. No. WO 2023 / 150575), or PCT / US2023 / 032070 (Publ. No. WO 2024 / 076441), which are hereby incorporated by reference in their entireties.

[0041] In the embodiment of FIG. 9A, the examination system 100 includes a practitioner computing system 120 (for use by a physician or other health practitioner 102) in communication, via one or more communications networks 170, with a patient system 200 (FIG. 9B) and a patient computing system 500 (FIG. 9B) located in a patient environment 110 of a patient 101. The practitioner system 120 includes a practitioner display 130, a practitioner camera 140, a practitioner microphone 150 (which can be part of the camera 140 or separate), a practitioner speaker 160, and a practitioner system controller 190 that includes a practitioner keyboard and a practitioner mouse. In some embodiments, the patient environment 110 includes a remotely-controllable lighting system 114, which enables the brightness of the patient environment 110 to be remotely adjusted by the practitioner at the practitioner computing system 120, by the practitioner processing device 122 itself, or by the central server 180. The communications network(s) 170 may include wide area networks 176 (e.g., the Internet), local area networks 178, etc. In some embodiments, the patient computing system 500 and the practitioner system 120 are in communication with a central server 180 having a processing device in communication with a database 182 to store the data from the analysis via the communications network(s) 170.

[0042] The central data processing device 180 (and / or the practitioner processing device 122 and / or the patient processing device 502) can access electronic health records (EHR) 185. Those include, for example, Demographic (Sex, Birth Date, Ethnicity, Race, . ..), Disease Data (Date of diagnosis, Age at Diagnosis, First Symptoms, Distribution of Symptoms, etc.), MGFA (MG Foundation of America) classification, i.e. category level of the disease, Antibody status, Treatment Information (drug, Thymectomy, etc.), Comorbidities (diabetes, cardiovascular condition, cancer, etc.), Previous Visit summary information, and MG-Q0L15r if available (quality of life questionnaire), which can be retrieved from (i) the patient digital record, (ii) possibly clinical trial or clinical study. The MG-CE score and ADL score are determined by the system 100 (e g., at the central data processing device 180 and / or the practitioner processing device 122) each time the patient has a medical visit.

[0043] The cyber-physical system 100 generates and stores patient status data indicative of the patient status (for example, physical, emotive, cognitive, and / or social state) of the patient 101, including data based on objective metrics and / or subjective assessments. Additionally, the cyberphysical system 100 may also provide functionality for the practitioner 102 to provide subjective assessments of the physical, emotive, cognitive, and / or social state of the patient 101. Together with the electronic health records 185 of the patient 101, those objective metrics and / or subjective assessments can be used by the practitioner processing device 122 and / or the central server processing device 180 to form a digital representation of the patient 101 referred to as a digital twin 800 that includes physical state variables 820 indicative of the physical state of the patient 101, emotive state variables 840 indicative of the emotive state of the patient 101, cognitive state variables 860 indicative of the cognitive state of the patient 101, and / or social state variables 880 indicative of the social state of the patient 101. The digital twin 800 is stored in the database 182, and provides a mathematical representation of the state of the patient 101 (e g., at each of a number of discrete points in time), which may be used by a heuristic computer reasoning engine 890 at the central server 180, that uses artificial intelligence to support clinical diagnosis and decision-making.

[0044] FIGS. 9B-9C are diagrams of the patient system 200 according to exemplary embodiments. In the embodiment of FIG. 9B, the patient system 200 includes a patient computing system 500, patient display 230, a patient camera 240, a patient thermal imaging camera 250, patient speakers 260, a patient eye tracker 270, a patient laser pointer 280, as well as patient input devices such as a patient microphone 262 (which can be part of the camera 240 or separate), patient keyboard and patient mouse. The patient camera 240 is a high definition, remotely-controllable pan-tilt-zoom (PTZ) camera with adjustable horizontal position (pan), vertical position (tilt), and focal length of the lens (zoom). In some embodiments, the patient display 230 may be mounted on a remotely-controllable rotating base 234, enabling the horizontal orientation of the patient display 230 to be remotely adjusted. Additionally, in some embodiments, the patient display 230 may also be mounted on a remotely-controllable vertically- adjustable mount (not shown), enabling the vertical orientation of the patient display 230 to be remotely adjusted. The presence of the patient camera 240 may distract the patient 101 and prevent the patient 101 from focusing on the interaction with the practitioner 102. In particular, in embodiments where the patient camera 240 is a remotely-controllable pan-tilt-zoom (PTZ) camera, any movement of the patient camera 240 may be particularly distracting. Therefore, in the embodiment of FIG. 9B, the patient camera 240 is optionally enclosed in the camera enclosure 600 that includes a one-way mirror. The mirror allows light to pass from the camera 240 to the patient 101, enabling the camera 240 to capture images of the patient 101. However, the mirror reflects light from the patient 101 (for example, toward an interior wall), preventing the patient 101 from seeing the patient camera 240 and any movement of the patient camera 240. Hiding the patient camera 240 in the camera enclosure 600 ensures that the patient camera 240 is as minimally invasive as possible, enabling the patient 101 to focus on the dialog with the practitioner 102 dialogue between the patient 101 and the practitioner 102.

[0045] As shown in FIG. 9C, the patient system 200 may be used in clinical settings, for example by a patient 101 in a hospital bed 201. As shown in FIGS. 9B, 9C, the patient system 200 may be used in conjunction with a patient computing system 500 that includes a patient processing device 502 such as, for example, a traditional desktop computer 202, for example having a display 204 and a keyboard 206. In those embodiments, for example, the patient system 200 may be realized as a compact system package that can be mounted on the display 204.

[0046] FIG. 9D is a block diagram of the patient computing system 500 according to exemplary embodiments. In the embodiment of FIG. 9D, the patient computing system 500 includes a processing device 502 such as a compact computer 510, a communications module 520, environmental sensors 540, and one or more universal serial bus (USB) ports 560. The environmental sensors 540 may include any sensor that measures information indicative of an environmental condition of the patient environment 110, such as a temperature sensor 542, a humidity sensor 546, an airborne particle sensor 548, etc. In some embodiments, the patient computing system 500 may include one or more physiological sensors 580. The physiological sensors 580 may include any sensor that measures a physiological condition of the patient 101, such as a pulse oximeter, a blood pressure monitor, an electrocardiogram, etc. The physiological sensors 580 may interface with the patient computing system 500 via the USB port(s) 560, which may also provide functionality to upload physiological data from an external health monitoring device (e.g., data indicative of the sleep and / or physical activity of the patient captured by a smartwatch or other wearable activity tracking device). Additionally or alternatively, one or more environmental sensors 540 and / or physiological sensors 580 (such as the breath sensor 340 described above) may be located in or on the control box 300.

[0047] FIG. 10 is a detailed block diagram of an example embodiment of the practitioner computing system 120, which can be utilized for the practitioner system 120 of FIG. 9A. The practitioner computing system 120 comprises a processing device 122, such as a server or personal computer. The practitioner system 120 includes, or is in communication with, a storage device 124 such as a memory or database, a language generator module 126 that implements a dialogue algorithm, data classification module 134, and an answer generator module 132 that implements an answer algorithm.

[0048] The practitioner processing device 122 and / or the central server processing device 180 control operation of the entire system 100, including displaying the avatar 232, implementation of the core examination (FIG. 2, 6), generation of patient instructions (FIG. 3), the avatar communication dialogue (FIGS. 7A), the avatar answer generation (FIG. 7B), and generation of the patient report (FIGS. 4, 5A, 5B, 8A, 8B). The practitioner processing device 122 and / or the central server processing device 180 can send instructions to the patient processing device 500 to control one or more of these operations, such as for example to control the avatar display and dialogue.

[0049] In addition, for example, the practitioner processing device 122 and / or the central server processing device 180 can send patient device control signals to, and receive patient device data from (via the patient processing device 500) the patient computing devices, e.g., the thermal imaging camera 250, display 230, rotating base 234, speakers 260, laser pointer 280, eye tracker 270, keyboard, mouse, via the communication module 520 (e.g., wired or wireless communication). For example, the patient device control signals can present the avatar on the display 230, and the patient device data can include receiving eye tracking data detected by the eye tracker 270, and patient input data received at the patient keyboard such as patient responses to medical questions, biographical information.

[0050] Further, the practitioner processing device 122 and / or the central server processing device 180 can send patient environmental control signals to, and receive patient environmental data from (via the patient processing device 500) the patient environmental sensors, e.g., the temperature sensor 542, humidity sensor 546, particle sensor 548, via the communication module 520 (e.g., wired or wireless communication). For example, the patient environmental control signals can control the environmental sensors to take the ambient room temperature, humidity level, and / or particle level. And, the patient environmental data can include the detected room temperature from the temperature sensor 542, detected humidity level from the humidity sensor 546, and detected particle level from the particle sensor 548.

[0051] The practitioner processing device 122 and / or the central server processing device 180 can send patient physiological control signals to, and receive patient physiological data from (via the patient processing device 500) the patient physiological sensors 580, e.g., pulse oximeter, blood pressure monitor, and EKG sensor, via the communication module 520 (e.g., wired or wireless communication). For example, the patient physiological control signals can control the physiological sensors 580 to take the patient’s pulse and oxygen levels, blood pressure, and EKG. And, the patient physiological data can include receiving the patients pulse and oxygen levels from the pulse oximeter, blood pressure from the blood pressure monitor, and EKG from the EKG device.

[0052] It is noted that the practitioner processing device 122 is shown as being separate and remote from central server processing device 180. However, it will be recognized that the practitioner processing device 122 and central server processing device 180 can be co-located at the practitioner location, and all operation can be implemented by the practitioner processing device 122 alone or by the central server processing device 180 alone (with the practitioner processing device 122 providing practitioner input).

[0053] The patient data can include, for example, patient status data (e.g., current and / or past physical, emotive, cognitive, and / or social state, such as stored as the digital twin 800), patient biographic data (e.g., demographics, disease data, disease classifications, EHR data, etc.), patient input data (e.g., from the patient keyboard, mouse, and microphone), patient environmental data (e.g., temperature), and patient physiological data (e.g., pulse, blood pressure). The patient data can be stored, for example, at the practitioner storage device 124 and / or the central storage 182, and / or at storage at or in communication with the patient system 500.

[0054] The system 100 administers a medical examination to a patient. It provides a user interface in the form of an avatar 232, that is very relatable and trustworthy, so that the patient is comfortable with responding to the avatar and following instructions by the avatar. This is accomplished, for example, by continuously monitoring the patient and determining the patient’s status data, and continuously determining if the patient is paying attention, well positioned, doing the exercise properly, relaxed (any emotion), or has questions, and then providing dialogue that is customized to the disease (e.g., MG), and that is responsive to the patient status. For example, the system continually monitors the patient and determines if the patient is stressed based, for example, on any of the patient data. If the patient is stressed, the avatar dialogue is modified to increase the level of sentiment or empathy. And, the avatar’s verbal and non-verbal features are adjusted as well. That is, once the avatar dialogue is generated, the avatar verbal and non-verbal features are generated to correspond with the avatar dialogue. So, if the dialogue has high sentiment, the avatar’s facial features and tone of voice also expresses a high sentiment. And if the patient is relaxed, or to help the patient relax, the avatar dialogue is relaxed or soothing and the avatar verbal and non-verbal features are also relaxed, for example, the avatar can be shown smiling. In some embodiments, the avatar verbal and non-verbal features can be based on the patient data and / or patient status data directly.

[0055] Thus, during the entire medical examination (e.g., introduction, explanation, instructions, exercise, conclusion), the avatar dialogue and verbal and non-verbal features are dynamically adjusted in real time, based on the current (and / or past) patient data and / or patient status data.

[0056] Digital Twin 800

[0057] As shown in FIG. 9A, the system 100 includes digital twin 800 data. A digital twin is “a virtual representation that serves as the real-time digital counterpart of a physical object or process.” As used here, the digital twin 800 can be any suitable data, such as shown and described in PCT / US2023 / 061783 (WO2023 / 150575), which is hereby incorporated by reference in its entirety.

[0058] As noted above, the digital twin 800 includes physical state variables 820 indicative of the physical state of the patient 101, emotive state variables 840 indicative of the emotive state of the patient 101, cognitive state variables 860 indicative of the cognitive state of the patient 101, and / or social state variables 880 indicative of the social state of the patient 101. The digital twin 800 is stored in the database 182, and provides a mathematical representation of the state of the patient 101 (e.g., at each of a number of discrete points in time), which may be used by a heuristic computer reasoning engine 890 at the central server 180, that uses artificial intelligence to support clinical diagnosis and decision-making. The digital twin is not just a data registry, but a functional virtual representation of the patient that integrate all these data in a model and is used to test hypotheses / answer patient questions / predict disease path / improve diagnosis, etc.

[0059] The state variables 810 are calculated by a sensor data classification module and form the digital twin 800 that may be the input of a heuristic computer reasoning engine 890. The sensor data and / or state variables 810 and recommendations from the digital twin 800 and / the heuristic reasoning engine 890 may be displayed to the practitioner 102 via the practitioner user interface 130.

[0060] The cyber-physical system 100 has access to the digital twin 800 of the patient 101, which includes a mathematical representation of biological characteristics of the patient 101 (e.g., eye color, height, weight, distances between body landmarks 701 and face landmarks 702, etc.). The digital twin 800 can, for example, be stored in the storage device may be provided to the practitioner processing device 122 and / or the central server processing device 180. Accordingly, the practitioner processing device 122 and / or the central server processing device 180 is able to use that specific knowledge of the patient 101 (together with general artificial intelligence and computer vision algorithms) to perform various operations relevant to the particular assessment being performed at every phase of examination and every operation shown in the figures. For example, data in the digital twin 800 can be utilized format dialogue, including questions asked to the patient, answers to patient questions, exercise instructions, step 302, monitoring exercises, step 306, determining if the patient is paying attention, step 310, if the patient is well positioned, step 312, etc.

[0061] FIG. 11 illustrates the construction of a digital twin 800 of the patient 101 according to an exemplary embodiment. The digital twin 800 of patient 101 may be far more complex and may require a strong ethical principal. In the cyber-physical system 100 described herein, the digital twin 800 is a model that can behave as a user of a telehealth system. The digital twin 800 of the patient 101 is an agent-based model with state variables 810 that represents the anatomic and metabolic variables of the patient 101 as well as a behavior state related to stress, cognitive load, memory components, etc. Statistical rules describe how the dynamical system transitions from one set of state variable values 810 to another in time. State variables 810 are preferably discrete numbers, and statistical rules are parametrized. Such an agent-based model has hundreds of unknown parameters that can be interpreted as the “gene set” of the user with respect to the model (similar to the inventors’ work in system biology). Those unknown parameters are calibrated using the training telehealth data. Meanwhile, the model is calibrated dynamically and keeps evolving with the patient observations accumulated at each telehealth session.

[0062] As shown in FIG. 11, all present and past input of the telehealth session (e.g., patient audio data 723 from the patient microphone 262, patient video data 724 from the patient camera 240 (FIG. 1C), patient responses from the input devices 206, thermal images from the thermal imaging camera 250, eyetracking data, environmental data from the environmental sensors 540, physiological data from the physiological sensors 580, notes from the practitioner 102) are input into the digital twin 800 of the patient 101.

[0063] The digital twin 800 of the patient 101 is specific to the disease management and describes the dynamic of the variables (e.g., physical state variables 820, emotive state variables 840, cognitive state variables 860, and / or social state variables 880) under the stimuli of the cognitive test or medical examination run by the practitioner 102.

[0064] The physical state variables 820 are models specific to the disease. In the Myasthenia Gravis (MG) example above, for instance, the physical state variables 820 may include eye motion, upper body motion, facial motion focusing on lip when talking, or cheek when the patient uses their tongue. As described above, the sensor data classification module 720 may directly or indirectly recover many physical state variables 820 through the sensor data 740 using computer vision and signal processing. The digital twin 800 operates on those physical state variables 820 with a stochastic operator and can damp the noise on measurement to some extent.

[0065] The emotive state variables 840 may include indicators on happiness, sadness, fear, surprise, disgust, anger or being neutral. As described above, the sensor data classification module 134 may identify many emotive state variables 840 through facial expression and eye tracking.

[0066] The cognitive state variables 860 include the ability to process a specific task and / or the ability to memorize specific data. Typically, the telehealth session concentrates on a specific cognitive variable to establish a diagnostic and monitor the progression of a disease.

[0067] As described above, the cyber-physical system 100 may be used to conduct a cognitive assessment and the sensor data classification module 134 may identify many cognitive state variables 860 through sensor data. As such, the cyber-physical system 100 can be used in controlled consultations (such as the consultations for Myasthenia Gravis patients 101 described above) and can also be beneficial on routine consultations for a patient 101 without chronic disease.

[0068] The social state variables 880 measure some aspect of aggressive behavior, mutualistic behavior, cooperative behavior, altruistic behavior, and parental behavior.

[0069] The emotive state variables 840, cognitive state variables 860, and social state variables 880 are correlated with noninvasive measurements during human computer interaction and may be identified using the sensor data captured by the sensor array of the cyber-physical system 100.

[0070] The digital twin 800 can be used also to support heuristic computer reasoning engine 890 to deliver telehealth assistance.

[0071] Based on the digital twin 800, the heuristic computer reasoning engine 890 of the central processing device 180 (or at the practitioner processing device 122) runs an artificial intelligence algorithm (on-site or in the cloud) that can reconciliate all these data sets and produce a high- quality analytical report that supports the provider’s diagnostic analysis and decisions. The heuristic computer reasoning engine 890 continually consults the digital twin 800 of the patient 101 to assist the practitioner 102 with the workflow process (starting with the medical history and updates and following with the analytical processing of cognitive and other test results in the context of all of the sensor data described above) to support a high quality clinical practice.

[0072] The heuristic computer reasoning engine 890 may be, for example, the HERALD system described in PCT Pub. No. 2022 / 217263, which describes a broad spectrum of applications that have been tested in clinical conditions to improve workflow efficiency and safety and is hereby incorporated by reference. The HERALD system 890 provides the architecture of a heuristic computer reasoning system that assists in the optimized efficiency of workflow by exercising the digital process analogue of a “thought experiment.” The heuristic computer reasoning engine 890 is comprised of a “learning” algorithm, developed as a system. For example, HERALD 890 can assist a system operator in charge of a medical facility with complex workflow efficiency issues and multiple human factor contingencies to gain a higher level of certainty in processes and increased efficiency in resources utilization - both staff and equipment. The same architecture can be a heuristic computer reasoning system 890 that coaches an individual to improve their behavior according to some specific objective such as work quality, quality of health or enhanced productivity. Once the heuristic computer reasoning system 890 is in place, it allows the construction of a sentient computer system that generates its own most relevant questions and provides optimum decision-making support to an operator based on continuous monitoring of working environments through multiple inputs and sensors. Not only does the present system allow for a more statistically based, informed decision-making tool, but also the system monitors collected data with a continually adaptive system in order to more accurately predict potential future outcomes and events. As more data is collected and as data sets expand, the inherent predictive value of the system increases through enhanced accuracy and reliability. This value is manifested in increasingly efficient workflow operations which may be best reflected through resultant changes in the principal’s behavior and overall satisfaction.

[0073] The cyber-physical system 100 uses the same three components as the HERALD system, including sensing (set of low-level artificial intelligence algorithms running on sensor output that acquire multimodalities input from the sensor array mounted on the display and patient input), communication (a medium level layer of artificial intelligence to communicate with the medical doctor running the telehealth session either by text messages, graphic user interface or voice recognition), and an evolving model of the problem domain for the patient consultation to support heuristic reasoning using a customized digital twin 800 of the medical doctor workflow that assimilates data from the sensors and communications to best describe the clinical practice.

[0074] Once this agent based model is in place, or any dynamical system that will mimic patients’ reactions to external stimuli such as environmental conditions, medical conditions including those induced by drugs, questions given by the providers, etc. one can exercise the heuristic computer reasoning schema 890 to support the telehealth session of the provider. As shown in FIG. 8, three categories of actions that the cyber-physical system 100 may provide are: (1) assisting the workflow of the telehealth session to make sure that all steps are covered and there is no gap in the data acquisition that will limit the quality of the diagnostic; (2) providing rational support and analytics on test results and quantified behavior indicators to document the report of the medical doctor as the telehealth session progresses; and (3) providing advanced warning on potential pitfalls based on observations exercising relevant modalities of the sensors to detect when patients start to behave outside “normal parameters” and / or run the HERALD system to constantly check on what might be overlooked with patient conditions. The digital twin 800 of the patient 101 continues to improve by comparing prediction with realization, using for example a mathematical genetic algorithm that optimizes the objective function defined by best clinical practices in the specific medical domain. The objective function might be vastly different in palliative care than in the management of a disease toward recovery. Further, as detailed in PCT Pub. No. 2022 / 217263, the system must operate within the parameter bounds defined by ethical principles.

[0075] It is noted that the digital twin 800 is utilized to provide patient data for the system 100. However, any suitable data management structure can be utilized within the spirit and scope of the disclosure, and a digital twin 800 is only one non-limiting example.

[0076] Examination Training Data

[0077] The system 100 is configured to work in conjunction with training data, which can be stored in the storage device 124 at the practitioner processing device 122 and / or the database 182 at the central server processing device 180. The training data is configured from sample transcript data (i.e., text data and audio / video data) of 106 sample telemedicine consultations provided by the MGnet network (see mgnet.rarediseasesnetwork.org / ). The sample transcript data was analyzed by natural language processing by extracting the dictionary of the most frequent (meaningful) words or sentences. And, various scores or conditions can be determines, such as for example the sentiment score statistic can be computed for each telemedicine visit transcript. This data set is used to train the examination algorithm and provide a baseline that guides the entire examination, including the dialogue.

[0078] The training data can also include, for example all patient data for the sample patients, including the sample patient digital twin 800 data (e.g., biological characteristics of the sample patient 101, the sample patient physical state variables 820 for the sample patient, the sample patient emotive state variables 840, sample patient cognitive state variables 860, sample patient social state variables 880), patient status data, as well as sample patient physiological data and sample patient environmental data, and EHR data 185. The training data also includes the transcript of the conversation between the doctor and the patient during previous telemedicine sessions, which is used to build a dictionary with all needed technical terms that is then stored in the practitioner storage device 124 and / or the central storage 182. FIGS. 12A, 12B give an overview of the training data, where the 106 samples are shown as the patient visits, and the eight (8) exercises are shown as the MGCD-score. FIG. 12A shows the patient score, where a light color indicates low score or no problem, and a dark color indicates high score or severe problem. FIG. 12B shows the classification of the patient by the medical practitioner, where a light color indicates that the practitioners agreed to the patient diagnosis, and a dark color indicates that the practitioners disagreed.

[0079] The distribution of scores for each subject and exam category at each visit is shown. With the exception of dysarthria, the population has a distributed representation of scores. Of 54 subjects, only four had confirmed signs of dysarthria, and five had suspicion, i.e. one positive score from either the observer or one reviewer without any confirmation by the adjudicator. Hence, we discarded the count-to-50 test from evaluation. We had previously concluded that the count-to-50 test is poorly reproducible and of limited clinical value.

[0080] The average severity of the disease in each category provides context to the ROS. The population of patients had relatively low severity of disease. The tongue-to- cheek examination (column 4) and sit-to-stand exercise (column 8) make the classification of score reproducibility into Reproducible or Non-Reproducible less confident than high average severity, such as ptosis (column 1), diplopia (column 2) and single breath count (column 7).

[0081] Avatar 232 (FIGS. 1A, 6)

[0082] The health examination system 100 is used to conduct a preliminary examination of a patient 101 and generate a report that a medical practitioner (e.g., nurse, physician, physician’s assistant) can use to base further examination. To do so, with respect to FIG. 1A, the system 100 employs a graphic 232 to establish a comfortable and relatable user-friendly interface between the patient 101 and the system 100 that the patient 101 can trust and follow, without requiring a human person to be involved. In the embodiment shown, the graphic 232 is an avatar 232 that is displayed in a display area of the patient display device 230. More specifically, the practitioner processing device 122 and / or the central server processing device 180 control operation fo the avatar 232 by sending control signals to the patient processing device 500 to display the avatar 232 on the patient display device 230.

[0083] Two key components make the avatar 232 trusted by the patient 101. First, the avatar is realistic in appearance, which can be achieved by the practitioner processing device 122 and / or the central server processing device 180 operating in combination with any suitable application to generate a realistic avatar. Second, as discussed below, the avatar 232 is provided with a conversational dialogue, which the practitioner processing device 122 and / or the central server processing device 180 achieves in coordination with any suitable conversational language generator, such as the model called ChatGPT which interacts in a conversational way.

[0084] The practitioner processing device 122 and / or the central server processing device 180 configures the avatar 232 (via the patient processing device 500), to conduct a core examination of a patient having Myasthenia Gravis, based on one or more of the patient data, and in particular the patient status data. The avatar 232 simulates the behavior of a medical practitioner, and for example can be dressed to look like a medical practitioner. In addition, the avatar 232 appears to sense, compute and act to run precisely and efficiently the neurological core examination of a patient, and automatically generate a documented medical report that the medical doctor can use to support diagnostic and medical recommendations. The avatar 232 is a humanoid chatbot that can perform the core examination precisely and efficiently without raising any patient concerns.

[0085] The practitioner processing device 122 and / or the central server processing device 180 provide the avatar 232 with certain avatar characteristics, such as for example, design, attributes, speech pattern, behavior, eye contact, facial expression or emotion (happy, serious, relaxed), and any suitable humanoid voice, appearance (dress), background, an animation, based on one or more of the patient data, and in particular the patient status data. Once the avatar characteristics are determined by the processing device 122 and / or 180, the processing device 122 and / or 180 can then generate the visual avatar for example by communication with any suitable libraries, using for example the Omiverse software from Nvidia, but provided with the speech text provided by the practitioner processing device 122 and / or the central server processing device 180, and possibly a time dependent set of parameter managing facial expression and eye gaze direction. Designing the avatar 232 for patient acceptance is challenging. One of the requirements of the avatar 232 is the ability to adapt the character to the patient, in order to increase acceptance and deal with racial stereotypes

[0037] , FIGS. 1A, 6 show a royalty-free Caucasian female (though other avatars are available, such as an African-American male character) from Reallusion (https: / / www.reallusion.com / character-creator / ). The practitioner processing device 122 and / or the central server processing device 180 can use, for example, the software Reallusion's iClone software to simulate the avatar 232 using as an input the transcripts stored at the practitioner database 124 and / or the central database of telemedicine consultation with Myasthenia Gravis patients.

[0086] In addition, the practitioner processing device 122 and / or the central server processing device 180 can give the avatar 232 avatar characteristics that are best suited to the specific patient, or that chosen by the patient, based on one or more of the patient data, and in particular the patient status data. For example, the avatar can be configured with a certain race and sex, which can have an impact on the patient approval and acceptance of the avatar 232. And, the avatar 232 can automatically speak in any language. That is, the practitioner processing device 122 and / or the central server processing device 180 can detects the patient’s speaking via the microphone, determine the language being spoke by the patient, and configure the avatar to speak that same language.

[0087] To generate the avatar characteristics, the practitioner processing device 122 and / or the central server processing device 180 can utilize all patient data, including the digital twin 800 data (e.g., biological characteristics of the patient 101, the physical state variables 820 for the patient, the emotive state variables 840, cognitive state variables 860, social state variables 880), patient status data, as well as physiological data from the physiological sensors 580 and environmental data from the environmental sensors 540, as well as the EHR data 185 including examination data such as the stage of examination and number of past visitations. It can also utilize all training data, including the training patient digital twin 800 data (e.g., training patient biological characteristics of the training patient 101, the training patient physical state variables 820 for the training patient, the training patient emotive state variables 840, training patient cognitive state variables 860, training patient social state variables 880), patient status data, as well as training physiological data, and training environmental data and EHR data and training dialogue data.

[0088] That configuration can be based on any suitable avatar configuration algorithm, for example, to weight different variables differently so that certain variables are more important for a given feature or attribute of the avatar. For example, that patient data can be used to generate an avatar having desirable attributes (sex, age, language, dress, speech pattern, voice, etc.) and / or simulate the behavior of the medical practitioner. Accordingly, if the patient data indicates that the patient is a young male (e.g., based on the patient’s age and an average age from the patient training data) from Boston who is nervous and has only had MG for a short time (e.g., based on the disease and length of time as compared to the training patient data), then the avatar can have a first set of avatar characteristics, such as the appearance and behavioral attributes of an older male with a Boston speech pattern. The avatar may have a second set of avatar characteristics that are different than the first set if the patient is an older female with a severe MG case that has persisted for many years. And, the age of the patient may have a low weight factor than the sex of the patient for one or more avatar characteristics, such as the age appearance of the avatar, and a higher weight factor for other avatar characteristics, such as the avatar speech pattern.

[0089] Chatbot Dialogue

[0090] In addition to providing a realistic looking and behaving avatar 232, the avatar must provide a conversational dialogue with the patient with certain dialogue characteristics, such as for example emotion (sentiment, empathy, encouragement, support) and delivery (tone, forcefulness, humor, understanding, volume, and speed), based on one or more of the patient data, and in particular the patient status data. The practitioner processing device 122 and / or the central server processing device 180 can have a dialogue generator 122 (a dialogue generator 126 is shown, for example, for the practitioner processing device 122, FIG. 10) that provides a conversational dialogue, such as for example a humanoid chatbot to handle (i) the dialogue with the patient to reach a specific goal, (ii) have a humanoid voice, and (iii) have a friendly face. The dialog generator 126 (of the practitioner processing device 122 and / or the central server processing device 180) can communicate, for example, with any suitable application to support realistic text to speech, such as IBM, Azure, Google, Amazon etc. And a friendly face on a speech can be done in communication with software such sadtalker.github.io / , reallusion.com / crazytalk / , and nvidia.com / en-us / omniverse / apps / audio2face / .

[0091] One object of the present disclosure is to provide a system and method that uses the ChatGPT technology and puts that ChatGPT technology under the supervision of the neurologist doctor in a synergistic way. The present system is designed to address that problem and improve delivery of health care. In one example embodiment, the practitioner processing device 122 and / or the central server processing device 180 sends a prompt to ChatGPT to query the library. Everything with Chat GPT is virtual and cloud base, so that no hardware is involved. The prompt can be a code that sends a question to Chat GPT in text format. In response to the prompt, the practitioner processing device 122 and / or the central server processing device 180 receives an answer from Chat GPT in text format. That question and answer pair can be stored at practitioner computer system 120, such as in the storage device 124, and / or at the central server processing device 180, such as at central database 182. A check list of what can go wrong is documented, and provides the frame to handle it.

[0092] As it accumulates experience, the system improves its performance and fills in any gap in the clinical discussion left. This can be done in accordance with any suitable machine learning techniques to build upon the training data with further data that the system 100 records. With the telemedicine consultation system 100, there is a dialogue, a patient feedback and a score. Those elements are used to optimize the next sessions. For example, the vocabulary dictionary gets bigger and bigger with annotation. The rate of success is reported and gives an opportunity to examine failures and improve the code. The training set is getting larger and the deep learning network increases its performance. In this way, the system 100 builds its own customized MG patient dialogue library of questions and answers that is specific to MG patients, and can send future questions and queries to the customized MG patient dialogue library instead of or in addition to ChatGPT.

[0093] The dialogue format makes it possible for the practitioner processing device 122 and / or the central server processing device 180 to use ChatGPT to answer follow up questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests” [22-26], While ChatGPT is used here, other suitable language generators or generative language libraries can be utilized in addition to or instead of ChatGPT, such as for example Geminis from Google, that benefits from a smaller training data set, but has some inherent advantage with its bidirectional encoder representation

[0036] ,

[0094] The present system 100 brings several new unique features that are essential to the success of the construction of a neurologist examination, including the avatar sense, compute and actions. The practitioner processing device 122 and / or the central server processing device 180 use the patient computer devices, patient environmental sensors 540, and patient physiological sensors 580, to adapt the avatar 232 questions and answers in response to verbal patient feedback and non-verbal patient feedback and trains accordingly in addition to the existing ChatGPT conversational capability, based on one or more of the patient data, and in particular the patient status data. For example, the practitioner processing device 122 and / or the central server processing device 180 receive non-verbal feedback from the patient computer vision (e.g., patient camera 240) and verbal feedback from the patient microphone 262 to perform speech analysis using Artificial Intelligence (Al).

[0095] The avatar 232 is configured, based on one or more of the patient data, and in particular the patient status data, to provide to the patient: Empathy, Explanation of the examination process, and Education, Encouragement, and support. The avatar 232 analyzes each of its tentative avatar answers to check on semantic and risk factors prior to provide any final avatar answers to the patient. It uses either recursively ChatGPT itself to analyze its own generated language or prior independent learning lessons accumulated from past examination in a data base. The avatar 232 can follow a rigorous and validated procedure to run the core examination of the patient and check autonomously the quality of the data acquisition to optimize the patient symptom examination. The avatar 232 automatically constructs a medical report on the examination including details and video clip on demand to assist the neurologist in his / her diagnostic, and give an opportunity to the doctor to validate quickly the medical examination.

[0096] The practitioner processing device 122 and / or the central server processing device 180 generates the dialogue with desired dialogue characteristics, based on one or more of the patient data, and in particular the patient status data. The dialogue is used throughout the entire medical examination, including exercise explanation, step 302, instructions, step 304, exercise monitoring, step 306, and conclusion, step 308, as well as patient attention, step 310, positioning, step 312, proper exercise, step 314, and question / answer, step 316. The practitioner processing device 122 and / or the central server processing device 180 can utilize all patient data, including the digital twin 800 data (e.g., biological characteristics of the patient 101, the physical state variables 820 for the patient, the emotive state variables 840, cognitive state variables 860, social state variables 880), patient status data, as well as physiological data from the physiological sensors 580 and environmental data from the environmental sensors 540, as well as EHR data 185 including the examination data such as the stage of examination and number of past visitations. It can also utilize all training data, including the training patient digital twin 800 data (e g., training patient biological characteristics of the training patient 101, the training patient physical state variables 820 for the training patient, the training patient emotive state variables 840, training patient cognitive state variables 860, training patient social state variables 880), patient status data, as well as training physiological data, and training environmental data and EHR data. That dialogue configuration can be based on any suitable dialogue algorithm, for example, to weight different variables differently so that certain variables are more important for a given dialogue. For example, the patient data can be used to generate dialogue having different dialogue characteristics depending on the patient data and the training patient data. Accordingly, if the patient data indicates that the patient is young and nervous (compared to the average person, including any training data) while performing a first exercise, then a first set of dialogue characteristics can be utilized, such as for example high empathy, supportive and soothing. The dialogue can have a second set of dialogue characteristics different from the first set if the patient is older and has a long history of examinations and noncompliance, such as for example more forceful. And, the age of the patient may have a low weight factor than the sex of the patient for one or more dialogue characteristics, such as the speed or pitch, and a higher weight factor for other avatar characteristics, such as empathy.

[0097] Overall Core Examination (FIGS. 2, 7A)

[0098] FIG. 2 shows the overall management of the medical examination 300 that is conducted by the avatar 232 (as operated by the practitioner processing device 122 and / or the central server processing device 180, via the patient processing device 500). The present disclosure describes a step-by-step method to generate the dialogue of the avatar 232 with the patient 101 during the neurology examination. We take the example of a teleconsultation for a patient suffering from myasthenia gravis (MG), but the process will be entirely similar for a number of neurological diseases, such as for example ALS (Amyotrophic Lateral Sclerosis), Multiple Sclerosis, Alzheimer, PTSD (Post-Traumatic Stress Disorder), post-stroke patients etc.

[0099] A standardized neurological examination 300 has been developed for patients with MG. At step 302, the avatar 232 starts the examination with simple questions on the patient condition first. Then the avatar 232 guides the patient to do several (eight) physical exercises, steps 304- 308, FIGS. IB, 1C, that starts with eye motion examination, asses various body motion such as arm extension, hip flexion, and end with speech exercises, without being limited to these exercises. At step 304, the avatar is configured to provide instructions to the patient and check compliance. At step 306, the avatar is configured to monitor the exercise and check timing. And at step 306, the avatar is configured to comfort the patient and conclude the exercise. As the examination is being administered, the practitioner processing device 122 and / or the central server processing device 180, via the patient processing device 500, continuously operates to confirm that the patient 101 is paying attention, step 310, that the patient is positioned properly, step 312, performing the exercise correctly, step 314, and is relaxed or has questions, step 316.

[0100] But even before the neurological examination starts, the avatar introduces itself and clearly state its mode of operation, as illustrated by the dialogue noted in FIG. 6. No matter the realism of the avatar 232 that can be almost perfect with software such as Nvidia provides, it is clearly stated to the patient that examination is run by an Al system, has its limit and will refer to the doctor for whatever questions or uncertainties is left out by the Al processing algorithm; as may be required by local laws on Al and transparency requirements.

[0101] The overall core examination is described as an overall operating algorithm, as in FIG. ID and may have variations in the order of task and number of tasks depending on patient conditions. The avatar 232 is configured to ask a number of questions (shown by the rectangles), which involves the use of natural language processing to control the dialogue of the avatar 232. The avatar 232 is configured to have the patient perform a number of exercises (shown by the circles). The exercises involve the use of natural language processing to configure the dialogue of the avatar 232 to explain the exercise and provide feedback to the patient. As further shown, computer vision is used by the practitioner processing device 122 and / or the central server processing device 180, via the patient processing device 500, to confirm that the user is properly performing the exercise. For example, the practitioner processing device 122 and / or the central server processing device 180, via the patient processing device 500, receive patient device data from one or more of the patient computing devices (e.g., the patient camera 240), as well as patient environmental data from the patient environmental sensors 540, and / or patient physiological data from the patient physiological sensors 580.

[0102] In addition, as shown in FIG. ID, certain exercises may also involve voice analysis. The practitioner processing device 122 and / or the central server processing device 180, via the patient processing device 500, receive verbal voice signals from the patient microphone 262, and analyze those voice signals to determine the patient status and condition of the patient.

[0103] Referring to FIG. 2, each physical exercise follows a step-by-step procedure. At step 302, the avatar is configured (i.e., by the practitioner processing device 122 and / or the central server processing device 180, via the patient processing device 500) to explain the exercise, why / how / hint on best execution prior to start the exercise, step 302 (FIG. 2). The avatar 232 is also configured to illustrate on the patient display, how to perform the exercise, and mimics how the exercise is to be performed. For example, for the cheek puff exercise (FIGS. IB, 1C), the avatar 232 mimics a cheek puff and explains how to orient the patient gaze as part of the explanation, so that the patient can follow the example illustrated by the avatar. Here, the practitioner processing device 122 and / or the central server processing device 180 can access any suitable library to perform the animation.

[0104] At step 304, the avatar 232 is configured to provide a quick dialogue with the patient 101 to support trial and error, and to correct the position of the patient until the patient is positioned correctly to start the exercise.

[0105] At step 306, the avatar is configured to next start the exercise and monitor the time, keep the patient on track by letting them know when it is mid-time, tell the patient when exercise elapse time has ended, and if it was successful, step 206.

[0106] At step 308, the avatar 232 is configured to provide a relaxing phase post exercise prior to going to the next exercise. The avatar 232 guides a gentle dialogue with the patient to keep the patient 101 engaged and in his / her comfort zone. Typical dialogue starts with how did you feel during this exercise, followed by any concerns and / or light discussions. At step 308 (FIG. 2), as the patient is conducting an exercise, the practitioner processing device 122 and / or the central server processing device 180 continuously checks to see that the patient is comfortable and concludes the exercise. This operation has the most complex interaction, and is essential to keep the patient in his / her comfort zone. It is the signature of how the doctor cares about the patient. Typically, the initiation of the dialogue in step 308 can be done with an algorithm that uses known conversation starters.

[0107] Generating Patient Instructions (FIGS. 2, 3; step 304)

[0108] During the operation, the practitioner processing device 122 and / or the central server processing device 180 will need to generate patient instructions, based on one or more of the patient data, and in particular the patient status data. For example, patient instructions are provided to the patient during explanation of exercise, step 302, exercise instructions and compliance, step 304, monitoring of the exercise, step 306, and exercise conclusion, step 308; as well as during monitoring patient attention, step 310, correcting patient positioning, step 312, correcting exercise performance, step 314, and patient relaxation, step 316.

[0109] FIG. 3 shows an example embodiment for the practitioner processing device 122 and / or the central server processing device 180 to generate patient instructions, step 400. Turning initially to step 408, the practitioner processing device 122 and / or the central server processing device 180 prepares video clip instructions based on parsed transcripts of MG clinical examinations, step 402, instructions generated by prompts sent to ChatGPT, step 404, and generated visual animation, step 406. It includes tasks that the avatar (i.e., as configured by the practitioner processing device 122 and / or the central server processing device 180 and implemented via the patient processing device 500) should manage along with the patient feedback that is necessary to optimize the implementation of the physical exercise and corresponding data acquisition. Namely, (1) instruction generated by ChatGPT, (2) video clip instruction, (3) annotated specialist review with classification, and (4) validation with patient feedback and compliance. In particular, FIG. 3 shows the schema used to generate the instructions in step 304, whereby the avatar 232 provides instructions to the patient and checks compliance.

[0110] Starting at step 402, the practitioner processing device 122 and / or the central server processing device 180 receives the examination transcript (text and / or video / audio) from the patient processing device 502 (i.e., as captured by the patient devices such as the patient camera 240 and patient microphone. It parses the transcript to clean up the text because the speech is not as good in text format, and some very basic editing can be done such as to correct spelling and punctuation. Instructions are generated by the language generator, step 404, such as by ChatGPT, and visual animation is generated, step 406. In certain embodiments, initial instructions or script can be retrieved from the practitioner storage device 124 and / or the central storage 182.

[0111] At step 408, the transcript (step 402), instructions (step 404), and visual animation (step 406) are used to generate the video clip instructions. Once the text script is prepared, it is used to build the video with the animation library stored in the practitioner storage device 124 and / or the central storage 182.

[0112] At step 410, the video clip instructions can optionally be manually reviewed by an annotated specialist, and classified. At step 412, it is validated with patient feedback and compliance. If the patient indicates that the instruction is understood, such as by text or nonverbal communication (e.g., performing the action requested by the instruction), then the patient instruction is saved, step 414, to the storage 124 and / or 182. If the instruction is not understood, then it is discarded, step 416.

[0113] To generate the patient instructions, the practitioner processing device 122 and / or the central server processing device 180 can utilize all patient data, including the digital twin 800 data (e.g., biological characteristics of the patient 101, the physical state variables 820 for the patient, the emotive state variables 840, cognitive state variables 860, social state variables 880), patient status data, as well as physiological data from the physiological sensors 580 and environmental data from the environmental sensors 540, as well as the EHR data 185 including examination data such as the stage of examination and number of past visitations. It can also utilize all training data, including the training patient digital twin 800 data (e.g., training patient biological characteristics of the training patient 101, the training patient physical state variables 820 for the training patient, the training patient emotive state variables 840, training patient cognitive state variables 860, training patient social state variables 880), patient status data, as well as training physiological data, and training environmental data and EHR data.

[0114] That configuration can be based on any suitable instruction configuration algorithm, for example, to weight different variables differently so that certain variables are more important for a given feature or attribute of the instruction. For example, that patient data can be used to generate an instruction having desirable dialogue characteristics. Accordingly, if the patient data indicates that the patient is a young male (e.g., based on the patient’s age and an average age from the patient training data) who is nervous and has only had MG for a short time (e.g., based on the disease and length of time as compared to the training patient data), then the instructions can have a first set of instruction with a first set of dialogue characteristics. The avatar may have a second set of instructions with a second set of dialogue characteristics that are different than the first set if the patient is an older female with a severe MG case that has persisted for many years. And, the age of the patient may have a low weight factor than the sex of the patient for one or more instruction characteristics.

[0115] Continuous Operations Referring to FIG. 3, throughout the examination, certain things are continuously monitored by practitioner processing device 122 and / or central server processing device 180, namely if the patient is paying attention, step 310, if the patient is well positioned, step 312, if the patient is performing the exercise correctly, step 314, and if the patient is relaxed or has questions, step 316, all based on one or more of the patient data, and in particular the patient status data.

[0116] For example, at step 310, an image detector or sensor 240 (e.g., a video camera) and computer vision algorithm is used to track the patient’s eye gaze and the eye motion, to check if the patient is paying attention to the avatar’s explanations. For example, the practitioner processing device 122 and / or the central server processing device 180 instructs the patient processing device 500 to have the avatar 232 initiate an exercise and activate the camera 240, step 306. During the exercise, the camera 240 provides (via the patient processing device 500) patient image data, such as shown in FIG. 1C, to the practitioner processing device 122 and / or the central server processing device 180. The image analysis module 128 of the practitioner processing device 122 and / or the central server processing device 180 analyzes that patient image data to determine if the patient is paying attention, 310. For example, the image analysis module 128 can implement on the patient image data, a gaze tracking algorithm that has been trained on MG patient data using deep learning.

[0117] If the patient is not paying attention, step 310, the avatar 232 is configured to implement a corrective patient dialogue, step 700, FIG. 7A, to start a dialogue with the patient, step 702, evaluate the patient and possibly remedy the situation. There are two types of avatar dialogues, step 702, namely self-generated 710, and addressing a patient question, step 730. A selfgenerated dialogue, step 710, is a dialogue that is automatically self-generated by the practitioner processing device 122 and / or the central server processing device 180 based on an analysis of the exercise, without any prompting by the patient. For most self-generated messages, step 710, a good library of text messages might be satisfactory, unless the patient sensing of the patient’s emotion, cognitive load or frustration says otherwise based on user input (including verbal input or in response to text questions and answers) and non-verbal input. MG patients can often have facial irregularities that make it more difficult for machine-language to determine emotion. Thus, to evaluate the patient emotion, the practitioner processing device 122 and / or the central server processing device 180 can use various input, such as natural language processing with sentiment analysis, voice analysis with change of pitch, change in facial expression etc.

[0118] Thus, at step 712, patient sensing is performed to determine the patient status dynamically and in real-time, categorize the patient status (for example, acceptable or low level, moderate level, or high level) dynamically and in real-time, and provide a response based on the patient status category dynamically and in real-time. Here, the practitioner processing device 122 and / or the central server processing device 180 can analyze the patient image data, and / or other patient data from the patient devices, and / or environmental data from the environmental sensors 540, and / or physiological data from the patient physiological sensors 580 or EHR data 185, and determine the patient’s emotion, cognitive load, and / or frustration, and provide a suitable dialogue for the avatar that encourages the patient to pay attention. Any suitable technique can be used to evaluate non-verbal feedback from the patient and determine the patient’s emotions, cognitive load or frustration, such as for example obtained from video with computer vision, speech analysis and artificial intelligence. Such systems are discussed, for example, in WO 2023 / 150575, and references [33, 34, 35], the entire contents of which are hereby incorporated by reference.

[0119] If it is determined that the patient’s emotion, cognitive load and frustration are at acceptable or low levels, the practitioner processing device 122 and / or the central server processing device 180 configures the avatar to provide a first response, for example to use a transition word or sentence, step 714. A transition sentence might be, for example, “I know this might be a little difficult.”

[0120] If it is determined that the patient is at a moderate level, such as that the patient is upset, has a moderate cognitive load, and / or is moderately frustrated, then the avatar 232 is configured to provide a second response, such as for example an encouraging message to support the patient, step 716.

[0121] If it is determined that the patient is at a high patient status level, such as that the patient is tired, has a high cognitive load or is highly frustrated, then the avatar is configured to provide a third response, such as for example a message to relax physical stress, step 718.

[0122] It is further noted that the responses, steps 714, 716, 718, can be dynamically adjusted depending on the determined patient status, rather than a patient status category. For example, the practitioner processing device 122 and / or the central server processing device 180 can generate a prompt and send the prompt to the generative language library such as Chat GPT to generate a customized response, or to a customized MG patient dialogue library stored at the practitioner storage device 124 and / or the central database 182 (FIG. 9A), and provide that response to the avatar to display to the patient.

[0123] In an illustrative example, let’s take the arm extension exercise, FIGS. 1C, 5B. The practitioner processing device 122 and / or the central server processing device 180 configures the avatar 232 to start the exercise, and tracks (using the patient camera 240) the patient arm position dynamically and in real-time. If the patient’s arms start to lower by a first threshold amount, the practitioner processing device 122 and / or the central server processing device 180 can notice it and configure the avatar 232 (e.g., using a prompt to ChatGPT or the patient library) to provide a second response, step 716, such as to gently ask the patient if he can hold up his arms, or if he is too tired to pursue the exercise. This is something that is not consistently managed manually, especially if the neurologist is busy entering information into EPIC (Electronic Patient Record Management) to notice and doesn’t have a permanent level of attention as in the system 100.

[0124] At step 719, once the avatar 232 administers the appropriate response, steps 714, 716, 718, the practitioner processing device 122 and / or the central server processing device 180 continues to determine whether or not the patient is paying attention, step 310 (FIG. 2). As long as the patient is not paying attention, step 310, the practitioner processing device 122 and / or the central server processing device 180 will continue to conduct patient sensing, step 712, and provide a proper response, steps 714, 716, 718. The longer the patient persists in not paying attention, the practitioner processing device 122 and / or the central server processing device 180 will generate an appropriate response based on the prompts provided to ChatGPT and / or the MP patient dialogue library.

[0125] At any time during the examination, the patient can ask a question, either verbally via the patient microphone or by tactile input at the patient mouse or keyboard (or touchscreen). The patient processing device 502 sends all patient input to the practitioner processing device 122 and / or the central server processing device 180, which detects the question and configures the avatar to address the question, step 730. At step 732, the practitioner processing device 122 and / or the central server processing device 180 determines if the question is similar to a previously-asked question. For example, the practitioner processing device 122 and / or the central server processing device 180 can retrieve the previously-asked questions and associated answers from the practitioner storage device 124 and / or the central database 182, and compare the current question with the previously-asked questions. If a threshold similarity is determined, then the practitioner processing device 122 and / or the central server processing device 180 loads the answer from the database, step 734, and configures the avatar 232 (e.g., using the avatar display algorithm and dialogue algorithm) to present the retrieved answer to the patient, step 738.

[0126] If the practitioner processing device 122 and / or the central server processing device 180 determines that the question is not similar to a previously-asked question, then the practitioner processing device 122 and / or the central server processing device 180 generates an answer, step 736, such as by sending a prompt to ChatGPT which responds with an answer that the practitioner processing device 122 and / or the central server processing device 180 sends to the patient processing device 500 to present as the avatar 232 to the patient on the patient display device 230, step 738.

[0127] At step 739, if the patient doesn’t have any further questions, and the patient is paying attention, step 310, the practitioner processing device 122 and / or the central server processing device 180 moves to the next operation, to determine if the patient is well positioned, step 312.

[0128] At step 312, the practitioner processing device 122 and / or the central server processing device 180 determines if the patient is well positioned to conduct the exercise. Here, for example, it checks to see if the patient is too close or far from the patient camera 240, or otherwise partially off center or not fully captured by the camera 240, and is determined based on the image data provided by the patient camera 240. The patient dialogue operation, step 700 (FIG. 7A), is the same as for the patient paying attention, step 310; except now the practitioner processing device 122 and / or the central server processing device 180 are sensing, at step 712) if the patient is properly positioned, and any questions, step 730, are based on the position of the patient, including prompts sent to ChatGPT. Accordingly, once the practitioner processing device 122 and / or the central server processing device 180 determine that the patient is well positioned, step 312, and has no questions, step 730, operation moves on to determine if the patient is performing the exercise correctly, step 314.

[0129] At step 314, the practitioner processing device 122 and / or the central server processing device 180 determines if the patient performing the exercise properly. FIGS. 4, 5 A, 5B show non-limiting illustrative example exercises, including eye and face examination, sit to stand and hip flexions, and arm extension, respectively. The patient dialogue operation, step 700 (FIG. 7A), is the same as for the patient paying attention, step 310, and if the patient is well positioned, step 312; except now the practitioner processing device 122 and / or the central server processing device 180 are sensing, at step 712) if the patient is properly performing the exercise, and any questions, step 730, are based on the position of the patient, including prompts sent to ChatGPT. For example, the practitioner processing device 122 and / or the central server processing device 180 tracks dynamically and in real-time the position of the patient’s arm. If during the arm extension exercise the patient arms are well positioned and stay up long enough (e.g., 2 minutes), then the practitioner processing device 122 and / or the central server processing device 180 determines that user has performed the exercise properly. Exact same story for each exercise. It is further noted that tracking if a patient properly performs an exercise can be conducted in any suitable manner, such as disclosed in PCT / US23 / 61783 (Publ. No. WO 2023 / 150575), or PCT / US2023 / 032070 (Publ. No. WO 2024 / 076441), which are hereby incorporated by reference in their entireties. Accordingly, once the practitioner processing device 122 and / or the central server processing device 180 determine that the patient is performing the exercise properly, step 314, and has no questions, step 730, operation moves on to determine if the patient is relaxed or has questions, step 316.

[0130] At step 316, the practitioner processing device 122 and / or the central server processing device 180 continuously determines if the patient is relaxed (or anxious) or has questions. The patient dialogue operation, step 700 (FIG. 7A), is the same as for the patient paying attention, step 310, the patient being well positioned, step 312, and the patient performing the exercise properly, step 314; except now the practitioner processing device 122 and / or the central server processing device 180 are sensing, at step 712) if the patient is relaxed, and has any questions, step 730. Accordingly, once the practitioner processing device 122 and / or the central server processing device 180 determine that the patient is relaxed, step 316, and has no questions, step 730, operation concludes for that exercise.

[0131] The practitioner processing device 122 and / or the central server processing device 180 continues the same operation of FIG. 2 for each exercise required by the examination, until all of the core examination exercises are done properly.

[0132] Within the identified ocular phase, we employ computer vision to segment individual exercises (ptosis, diplopia left, diplopia right) based on gaze direction. We use a Haar cascade classifier (OpenCV

[0010] ) to detect the patient's face and then fit a 68-point landmark detector (DLib

[0011] ) to locate key facial features, including the eyes. A convolutional neural network (CNN) classifies video frames into four categories: "up," "left," "right," and "neutral," indicating the patient's gaze direction. This network is trained on video frames from our dataset, where frames during ptosis exercises are classified as "up," diplopia exercises as "left" or "right," and gaps between exercises as "neutral" (refer to [6,12] for details).

[0133] In FIG. 13 A, Dlib 68 points landmark used to find the ROI of both eyes. In FIG. 13B, the right detection of eye gaze looking successively up, then left and right using a deep learning algorithm with the ROI. The curves represent the density for each gaze direction (up, left and right) over the ocular exercises time window.

[0134] The practitioner processing device 122 and / or the central server processing device 180 analyzes the CNN output over the entire ocular phase. A sliding window calculates the "density" of each gaze class (up, left, right) over a 5-second interval. The exercise segment is identified as the longest contiguous period where a specific gaze class density exceeds 60% of the maximum density. Segments shorter than 10 seconds are considered undetectable, while segments between 10 and 45 seconds are partially detectable. Only segments exceeding 45 seconds are classified as fully detected.”

[0135] * *

[0136] To conduct the continuous monitoring, the practitioner processing device 122 and / or the central server processing device 180 can utilize all patient data, including the digital twin 800 data (e g., biological characteristics of the patient 101, the physical state variables 820 for the patient, the emotive state variables 840, cognitive state variables 860, social state variables 880), patient status data, as well as physiological data from the physiological sensors 580 and environmental data from the environmental sensors 540, as well as the EHR data 185 including examination data such as the stage of examination and number of past visitations. It can also utilize all training data, including the training patient digital twin 800 data (e.g., training patient biological characteristics of the training patient 101, the training patient physical state variables 820 for the training patient, the training patient emotive state variables 840, training patient cognitive state variables 860, training patient social state variables 880), patient status data, as well as training physiological data, and training environmental data.

[0137] That configuration can be based on any suitable monitoring algorithm, for example, to weight different variables differently so that certain variables are more important for a given action. Accordingly, if the patient data indicates that the patient is a young male (e g., based on the patient’s age and an average age from the patient training data) who is nervous and has only had MG for a short time (e.g., based on the disease and length of time as compared to the training patient data), then the monitoring may give more time for the patient to pay attention.

[0138] Answer Generation (FIG. 7B)

[0139] Throughout operation, the practitioner processing device 122 and / or the central server processing device 180 will need to answer questions posed by the patient through the patient processing device 502, such as for example at step 736 (FIG. 7A), based on one or more of the patient data, and in particular the patient status data. The practitioner processing device 122 and / or the central server processing device 180 can be provided with an answer generator module 132 (e.g., FIG. 10 shows module 132 for the practitioner processing device 122) and controls operation of answer generation, including the steps shown in FIG. 7B. In that figure, the answer generator module utilizes a question algorithm that is used to generate an answer, step 750, by generating a prompt, transmitting the prompt to the Generative Language Software (GLS) (also called Generative Language Library), and receiving a response from the GLS.

[0140] Referring to FIG. 7B, every time the practitioner processing device 122 and / or the central server processing device 180 needs to generate a new answer, it sends a prompt to the GLS. Any suitable GLS can be utilized, such as Open Al, Google, Meta, Apple, Amazon, ChatGPT, or Geminis. One can ask 100 times the same question to a GLS and it will come back with slightly different answers. The key is to guide the GLS to provide an optimum answer that would satisfy the patient and avoid errors. First, the question of the patient are subject specifics, not general, so GLS are usually much better with those than general question. Second, the answers generated by GLS are based on a transcript of the patient questions. The transcript has been clean up and filtered to retain only meaningful questions with respect to the disease using Natural Language Processing tools. We automatically extract the main concern of the patient according to the dictionary of symptoms of the disease and background of the patient.

[0141] And third, the practitioner processing device 122 and / or the central server processing device 180 analyzes the answer with the GLS itself in a recursive process. That is, the practitioner processing device 122 and / or the central server processing device 180 asks the GLS about how many words, how many sentences in its answer, what are the technical words of that answer, we ask for a sentiment analysis of the answer to check the level of sentiment and / or empathy of the answer etc. If the answer does not comply with the topic dictionary, or is too long, or not empathic enough, we don’t use that answer and ask the GLS to generate a better answer according to our criteria.

[0142] The database 124 (either at the practitioner processing device 122 and / or the central server processing device 180) has answer trained data that has been trained based on sample telemedicine consultations transcripts between medical practitioners and patients. The practitioner processing device 122 and / or the central server processing device 180 generate trained answers based on that trained data, such as for example by determining text of the patient speaking, and text ending with a question mark.

[0143] Starting at step 751, the patient processing device 502 transmits all text from the patient input, video and / or audio data from the patient camera 240, and audio from the microphone 262, to the practitioner processing device 122 and / or the central server processing device 180. That determines if the patient 101 is asking a question. Once a question is asked, the practitioner processing device 122 and / or the central server processing device 180 encapsulates that question, step 752, into a first answer prompt. That is, if the patient asks “what is ptosis,” then that question is encapsulated into a first answer prompt. For example, the first answer prompt can be “what should I tell a patient who asks what is ptosis?”

[0144] For example, the practitioner processing device 122 and / or the central server processing device 180 can generate a transcript of the dialogue using a speech to text software. Using the transcript as an input (as well as the video view) we run the question algorithm. This can be performed with any suitable system, such as the AssemblyAI API, which allows for high quality transcriptions in many languages. It gives the system access to timestamps, speaker identification, and other features for every single word in the transcript. It then browses the text for questions or concerns, such as for example using a semantic analysis exiting tool.

[0145] At step 754, the practitioner processing device 122 and / or the central server processing device 180 sends the first answer prompt to the language generator, ChatGPT, and receives a first response answer. The prompt is sent to Chat-GPT to answer these questions or concerns with additional guidelines we provide that are specific to the core examination. The system does not directly ask ChatGPT the question of the patient: what the system asks to ChatGPT is more like: “what should I say to the patient who ask me X, in order to make sure that Y is fulfilled”. For a certain disease, such as MG, we want for example enforce empathy, or avoid specific risks known from the neurological examination.

[0146] At step 756, the practitioner processing device 122 and / or the central server processing device 180 analyze (e.g., using natural language processing) that first text response answer by first separating the first response answer into sentences, step 756, and testing the first response answer, step 758, for desired answer characteristics, such as a desired level of sentiment or empathy. Sentences can be identified based on the presence of a colon, semi-colon, period, etc. If the value it is not acceptable, step 760, it generates a second answer prompt, and sends it to the language generator, ChatGPT, until an acceptable response is received. The first response can also be analyzed for a quantitative indicator on length and semantic using NLP and / or Chat GPT itself, compute the empathy level of the text (=it is a call to a sentiment analysis library, such as stored in the storage device 124, 182), check if the answer has critical element from our dictionary, etc. It can also browse and filter the answers to check it is conformed to a format we provide. There are either standard library of NLP to do this or we can ask Chat GPT itself to give us the analysis, or both and compare the answers for robustness. The practitioner processing device 122 and / or the central server processing device 180 resends the prompt to generate a new answer if the format is not optimum, such as that the text is too long, too many sentences, more than one paragraph, numbered or bullet points, too vague (check with respect to our dictionary), sentiment level too low, etc.

[0147] At step 762, a disclaimer is added to the acceptable answer, such as “You should ask the medical practitioner, however, ... The practitioner processing device 122 and / or the central server processing device 180 sends the text generated by Chat GPT through the avatar (i.e., via the patient processing device 502), and asks the patient for quick feedback, such as if it answers the question and if it makes sense. It is also possible to analyze the facial expression of patient to detect failure in the dialogue and do damage control. But a simple yes / no answer can do.

[0148] The feedback generated by the patient’s response is used to train the question algorithm and Al dialogue and stored in the practitioner storage device 124 and / or the central storage 182. Once a sufficient number of dialogues are stored, the practitioner processing device 122 and / or the central server processing device 180 can send prompts to the practitioner storage device 124 and / or the central storage 182, and skip the GLS Chat GPT. The practitioner processing device 122 and / or the central server processing device 180 does not need to use another library to extract the structure of the sentence generated by ChatGPT, and run a semantic analysis. ChatGPT has the capability to extract a specific sentence in its answer according to one of our criteria on MD-patient interaction such as length of the answer, simplicity of the sentence structure etc., and check if that criteria is fully satisfied. ChatGPT, and BERT in particular, have a good capability also to run a sentiment analysis of the text, which can be useful in alerting the avatar when the dialogue is not productive for the patient. Let us emphasize that the conversational system of our avatar is by no means reduced to ChatGPT simply answering the patient: as warned by its authors, ChatGPT is not designed for this, and it would be dangerous to do so.

[0149] One or more metrics can be used to rank the quality of an answer to the patient, including but not limited to: (a) Semantic Similarity: Measures how similar two pieces of text are in meaning. This can be computed using cosine similarity between vector representations of the texts. In our case we can use that metric to check if the generated answer does correlate with the patient question, including technical terms from the dictionary stored the practitioner storage device 124 and / or the central storage 182. (b) Named Entity Recognition (NER): Identifies and classifies proper nouns in the text, such as names of people, organizations, locations, dates, etc. We like to be very careful with questions related to specific name of people and organization and may decline the answer explain the avatar is not competent to answer that question, (c) Sentiment Analysis: Determines the sentiment or emotional tone of a text, typically categorizing it as positive, negative, or neutral. We start from the standard Vader library, but add our own sentiment analysis using the manual annotation of the transcript of the telemedicine sessions we have archived. So the sentiment analysis metrics gets customized to the disease and indeed more accurate, (d) Topic Modeling: Identifies the main topics present in a collection of documents. Common techniques include Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF). We have to differentiate the conversation about the disease from the one that is external to this topic. The avatar should not answer generic question like a human will do. It is too risky to be wrong, (e) Keyword Extraction: Identifies the most important words or phrases within a text. It is important to check that the answer has enough meaning that way. (f) Word Sense Disambiguation (WSD): Determines which sense of a word is used in a given context, especially when the word has multiple meanings. If WSD needs to be used extensively, the answer is suspect and should be discarded, (g) Dependency Parsing: Analyzes the grammatical structure of a sentence and establishes relationships between "head" words and words that modify those heads. We like to generate answer that have the simplest grammatical structure as possible, (h) Textual Entailment: Determines if one piece of text logically follows from another piece of text (entailment), contradicts it, or is neutral. We like to discard answers that have fuzzy logic according to this analytic, (i) Paraphrase Detection: Identifies whether two texts are paraphrases of each other, i.e., convey the same meaning using different words. We may privilege paraphrase or discard it depending on any cognitive knowledge of the patient reported in the HER notes.

[0150] The practitioner processing device 122 and / or the central server processing device 180 can use those metrics and outputs individually or combined to provide a comprehensive understanding of the semantic content of text data, and consequently assess the quality of the answer, using, for example, a natural language processor and / or by accessing existing software libraries: NLTK [https: / / www.nltk.org / ], spaCy [https: / / spacy.io / ], Gensim [https: / / pypi.org / project / gensim / ], Stanford NLP [ https: / / nlp.stanford.edu / ].

[0151] The quality can be rated and / or scored. If the target quality is not reached, we generate again the answer with the generative language library adding additional instruction related to those quality assessment above that are the weakest. The same prompt is run (e.g., by the Answer Generator 132) multiple times (e.g., 10 times), and then the best answer is selected according to the criteria: dictionary score, length, semantic analysis, sentiment, etc.

[0152] To generate the prompt, the practitioner processing device 122 and / or the central server processing device 180 can utilize all patient data, including the digital twin 800 data (e.g., biological characteristics of the patient 101, the physical state variables 820 for the patient, the emotive state variables 840, cognitive state variables 860, social state variables 880), patient status data, as well as physiological data from the physiological sensors 580 and environmental data from the environmental sensors 540, as well as the EHR data 185 including examination data such as the stage of examination and number of past visitations. It can also utilize all training data, including the training patient digital twin 800 data (e.g., training patient biological characteristics of the training patient 101, the training patient physical state variables 820 for the training patient, the training patient emotive state variables 840, training patient cognitive state variables 860, training patient social state variables 880), patient status data, as well as training physiological data, and training environmental data.

[0153] That configuration can be based on any suitable answer algorithm, for example, to weight different variables differently so that certain variables are more important for a given feature or answer characteristic. For example, that patient data can be used to generate an answer prompt having desirable answer characteristic and dialogue characteristics. Accordingly, if the patient data indicates that the patient is a young male (e.g., based on the patient’s age and an average age from the patient training data) who is nervous and has only had MG for a short time (e.g., based on the disease and length of time as compared to the training patient data), then the answer prompt can have a first set of answer prompt characteristics, such as a first sentiment or empathy level. The answer prompt may have a second set of answer prompt characteristics that are different than the first set if the patient is an older female with a severe MG case that has persisted for many years, such as a second sentiment or empathy level greater than or less than the first sentiment or empathy level. And, the age of the patient may have a low weight factor than the sex of the patient for one or more answer prompt characteristics, and a higher weight factor for other avatar characteristics.

[0154] Example: Illustration with Patient Orange

[0155] We analyze a clinical examination for a Mysthenia Gravis patient coding each phase by letter defined below:

[0156] A. Step 302, explain the exercise, why / how / hint on best execution prior to start the exercise.

[0157] B. Step 304, Quick Dialogue with the patient to support Trial and Error to correct the position of the patient until it is OK to start.

[0158] C. Step 306, Start the exercise and monitor the time, tell the patient when it is midtime, tell the patient when exercise elapse time has ended

[0159] D. Step 308, Relaxing phase post exercise prior to go to next exercise: this is where the MD experience guides a gentle dialogue with the patient to keep him engaged and his / her comfort zone; typical dialogue starts with how did you feel during this exercise, followed by any concerned and or light discussions. Report on ptosis test:

[0160] A:

[0161] Okay, so I'll start the core examination.

[0162] Sir, we're going to just start with looking for eyelid troop.

[0163] So if you could just kind of maintain your head straight at the camera.

[0164] And then I'm going to time you here.

[0165] And I just want you to look up at the ceiling, find a mark and just look up there.

[0166] It's going to be about a minute.

[0167] B:

[0168] And just stay right there.

[0169] C:

[0170] Okay, you go ahead right now.

[0171] D:

[0172] I know this is a little difficult.

[0173] You can blink, but just keep in that position.

[0174] You're doing great.

[0175] C:

[0176] Just keep up there.

[0177] Now just look straight ahead.

[0178] D:

[0179] Okay, thank you.

[0180] Remark: we notice that C and D are interlaced during the exercise.

[0181] % diplopia left:

[0182] A:

[0183] So now we're going to just move to the next test.

[0184] So we're going to be looking for double vision.

[0185] If you start seeing double, you can just tell us, but we're going to have you look to the left horizontally again, pick out a point, and we're going to do this for a minute.

[0186] B: So keep your head just look. Yeah, that's correct. Okay.

[0187] And once you start, just stay in that position.

[0188] It doesn't have to be too far, but just try to keep your head straight and looking at that single point.

[0189] And if at any point you start to appreciate some double vision, you tell. Me.

[0190] But otherwise, just stay in that position.

[0191] And again, we're going to do this for about another 30 seconds.

[0192] C:

[0193] It. Okay. That's it.

[0194] D:

[0195] And you can just relax for a moment.

[0196] % diplopia right:

[0197] A:

[0198] And let's do it to the other side just to test it, but just give it a minute.

[0199] B:

[0200] Hold on. Why don't you look straight ahead, we'll start the test again.

[0201] Try to keep your head straight and pick out a point on the opposite side. And let's go ahead.

[0202] C:

[0203] Doing well. Just got another 15 seconds, we'll be done.

[0204] D:

[0205] All right, excellent.

[0206] % cheek puff:

[0207] A:

[0208] Okay, so the next piece is a little bit easier.

[0209] I'm going to ask you to puff your cheeks out. Yeah. Just relax.

[0210] And then we're just going to kind of keep it there for, whatever, 10 seconds.

[0211] And then I'm just going to ask you to kind of press gently your fingers against your cheeks once you start puffing your cheek out. Right now? Yeah, just keep it like that.

[0212] D:

[0213] That's perfect. All right.

[0214] A:

[0215] And gently with your fingers.

[0216] D:

[0217] Just excellent.

[0218] C:

[0219] Okay, good.

[0220] D:

[0221] We're good with that? Yeah. All right.

[0222] % cheek push left

[0223] A:

[0224] Okay, so a little bit more difficult here in one way. So let me just show you what we're going to do.

[0225] You're going to push your tongue into your cheek and then start pressing slightly on it.

[0226] But first, keep it deformed there for a moment. Let it sit there.

[0227] B:

[0228] I know you're having some problems with your tongue, so we're just going to let it stay there.

[0229] Just keep it there for a moment. Just keep it there. We're going to keep that up. Just again, we're doing this.

[0230] I know, because we talked about your tongue feeling a little bit weak.

[0231] It's going to keep it there for 30 seconds.

[0232] D:

[0233] Okay, good. Now just relax.

[0234] Hold on.

[0235] How did that feel to you?

[0236] Patient: It felt like at first I could press strongly, but then not as strongly. D:

[0237] Yeah.

[0238] Patient: Even though I had it in the position.

[0239] D:

[0240] Yeah.

[0241] % cheek push right:

[0242] B:

[0243] Then once again, puff your cheek out a little bit and once your tongue into the opposite side.

[0244] Good.

[0245] D:

[0246] You just have the same feeling there.

[0247] Patient: I felt slightly less ability to push on this side than that side.

[0248] D:

[0249] Okay.

[0250] Okay. We're just going to let it rest a little bit.

[0251] The dialogue is not as systematic as an algorithm would be. Dialogue D can be initiated at any point in time during the examination. We lose progressively the structure as the examination goes, probably because part of the dialogue becomes implicit. The amount of feedback from the patient may vary greatly in clinical condition. We may automatically push the transcript of some of the key feedback from the patients in the notes of the medical exam, by looking at anchor in the conversation such as “feeling”, “ability” etc.

[0252] Patient Report Generation (FIGS. 4, 5A, 5B, 8A, 8B)

[0253] Once the examination is completed, the practitioner processing device 122 and / or the central server processing device 180 generates a patient report that can be evaluated by the medical practitioner, such as a nurse, physician, or physician’s assistant. FIGS. 4, 5A, 5B show examples of report that is generated, including information on the patient’s control of eye and face (FIG. 4), sit to stand and hip flexion (FIG. 5A), and arm extension (FIG. 5B). The reports are formatted to be simple and easy to read, with information that is useful to the medical practitioner in a graphic format that displays the patient information and results needed by the medical practitioner.

[0254] FIG. 4 is a report on the results of the visual control of a patient eye and face examination based on the exercises administered by the practitioner processing device 122 and / or the central server processing device 180, via the patient processing device 502 and the avatar 232. Images of the patient performing the exercise related to that report, are included in the report, as well as markings from analysis of the patient images. The report in FIG. 4 is for eye opening, which is derived from one or more examinations including an upgaze, and the images shown are as the patient gazes up with annotated measurement markings as lines around the eye. In addition, the report includes metrics obtained from the exercises, that are relevant to this report, including eye opening, and fatigue level (or other patient data or patient status data). The report also shows that the patient tracked the exercise well, and there were no errors in performing the exercise. A slide bar is shown indicating the likelihood of an eye disease or symptom, such as ptosis.

[0255] FIG. 5A shows another example report of the same patient performing a different exercise, here the sit to stand to measure hip flexion. And FIG. 5B shows another example report, for visual control of patient arm extension examination, from the arm extension exercise. Again, images, metrics and markers are included in these report, as well as the progression for sit to stand or arm raise.

[0256] FIG. 8A further reports various body / facial measurement scores or ratings for static and dynamic results, including distance from the upper lid to the pupil score or rating, eye alignment score or rating, arm fatigue score or rating, sit to stand speech analysis score or rating, and cheek puff score or rating, as well as an overall MG score (here, the scores “1” and “2” are low signs of MG). For example, for the arm test, the system 100 determines a static measurement of how much (i.e., distance and / or number of times) the arms drop during the 2 minutes elapse time of the exercise. And, determines a dynamic measurement of how fast / slow, and how many oscillations. As used here, static is a measure independent of time, such as for example, the eye opening at the start or the end of the exercise. Dynamic means the time dependent variation of eye opening. In the graphs, the y coordinate of the graph are in pixels, and the x coordinate is time in seconds. Referring to FIG. 8B, the graph illustrates the dynamic metric of speed of decay, here the speed at which the patient opens and holds up the right and left eyes, to provide a dynamic metric of the exercise. The dynamic piece gives a better metric of the answer of the neurological systems and tells us for example if the patient is struggling to do the exercise. It underlies the reports of FIGS. 4, 5 A, 5B, 8 A, but need not be displayed to the practitioner. The top half-circle has four stages, each a different color, and the bottom arc shows where this patient falls within those four stages. For example, for arm fatigue, the patient is scored as 0.8, which is within the first stage, but trending upward. The gauge goes from 0 (no sign) to 4 worst sign. So the score 0.8 is on this scale, in the interval (0 1) that is the low symptom phase. Dynamic corresponds to the speed of eye lid drop for this ptosis test. Static metric is just the eye opening metric at a specific time. The report in FIG. 8A further shows the trend for that patient, such as based on the patient’s prior examinations.

[0257] As the sequence of questions and physical examination exercise is done, the practitioner processing device 122 and / or the central server processing device 180 automatically provides a report on the patient symptoms, as shown in FIGS. 4, 5A, 5B, 8A, using the underline principle exemplified in FIG. 8B. All computer vision algorithms provide error estimates and compare the patient condition to a control population, so the scale of the gauge is rigorously justified.

[0258] In FIG. 8B, the bar graph presents a visual graphic representation that compares the patient to a healthy patient (=benchmark). It provides an absolute metric of how bad the symptom is for that patient.

[0259] Conclusion

[0260] The system and method of the present invention include operation by one or more processing devices, including the practitioner processing device 122, central processing device 180, and the patient processing device 502. It is noted that the processing device can be any suitable device, such as a computer, server, mainframe, processor, microprocessor, PC, tablet, smartphone, or the like. The processing devices can be used in combination with other suitable components, such as a display device (monitor, LED screen, digital screen, etc.), memory or storage device, input device (touchscreen, keyboard, pointing device such as a mouse), wireless module (for RF, Bluetooth, infrared, WiFi, etc.). The information may be stored on a computer hard drive, on a CD ROM disk or on any other appropriate data storage device, which can be located at or in communication with the processing device. The entire process is conducted automatically by the processing device, and without any manual interaction. Accordingly, unless indicated otherwise the process can occur substantially dynamically and in real-time without any delays or manual action.

[0261] It is noted that certain operations of the disclosure have been described as being performed by the practitioner processing device 122 and / or the central processing device 180, which are accessed by the patient processing device 502 through the cloud WAN 176 (directly or via a LAN 178), including generation of the avatar and the avatar dialogue. However, it should also be noted that those operations can be performed by the patient processing device 502, such as on a standalone computer having storage. In other embodiments, the practitioner processing device 122, central processing device 180, and the patient processing device 502 can all be the same processing device, and not separate devices as shown in the figures and described above, and located either at the medical facility or the patient location.

[0262] References

[0263] The following references are hereby incorporated by reference in their entireties.

[0264] [1] Anderson NE, Mason DF, Fink JN, Bergin PS, Charleston AJ, Gamble GD. Detection of focal cerebral hemisphere lesions using the neurological examination. J Neurol Neurosurg Psychiatry. 2005 Apr;76(4):545-9. doi: 10.1136 / jnnp.2004.043679. PMID: 15774443; PMCID: PMC 1739581. https: / / jnnp.bmj.eom / content / 76 / 4 / 545.short.

[0265] [2] Schmidt, W.U., Lutz, M., Ploner, C.J. et al. The diagnostic value of the neurological examination in coma of unknown etiology. J Neurol 268, 3826-3834 (2021). https: / / doi.org / 10.1007 / sQ0415-021-10527-4.

[0266] [3] Mark Thaller and Thomas Hughes, Inter-rater agreement of observable and elicitable neurological signs, Clin Med June 2014, DOI: https: / / doi.org / 10.7861 / clinmedicine.14-3-264.

[0267] [4] Barnett C, Bril V, Kapral M, Kulkarni A, Davis AM. Development and validation of the Myasthenia Gravis Impairment Index. Neurology. 2016 Aug 30;87(9):879-86. doi: 10.1212 / WNL.000000000000297L Epub 2016 Jul 8. PMID: 27402891; PMCID: PMC5035154.

[0268] [5] M Benatar, A systematic review of diagnostic studies in myasthenia gravis, Review published: 2006 in Database of Abstracts of Reviews of Effects (DARE): Quality-assessed Reviews, York (UK): Centre for Reviews and Dissemination (UK). [6] S. Baee, M. Rucker, A. Baglione, M.K. Ameko, L. Barnes, A Framework for Addressing the Risks and Opportunities Al-Supported Virtual Health Coaches, Proceedings of the 14th EAI International Conference on Pervasive Computing Technologies for Healthcare (2020), pp. 251-254.

[0269] [7] E. Batbaatar, J. Doijdagva, A. Luvsannyam, M.M. Savino, P. Amenta, Determinants of Patient Satisfaction: A Systematic Review Perspectives in Public Health, 137 (2) (2017), pp. 89-101.

[0270] [8] C. Baxter, J. A. Carroll, B. Keogh, C. Vandelanotte, Assessment of Mobile Health Apps Using Built-in Smartphone Sensors for Diagnosis and Treatment: Systematic Survey of Apps Listed in International Curated Health App Libraries, JMIR mHealth and uHealth, 8 (2) (2020), p. el 6741.

[0271] [9] S. Chandra, M. Mohammadnezhad, P. Ward, Trust and Communication in a Doctor- Patient Relationship: A Literature Review, Journal of Healthcare Communications, 3 (3) (2018), pp. 1-6., 36.

[0272]

[0010] D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, K. Georgila, J. Gratch, A. Hartholt, M. Lhommet, G. Lucas, SimSensei Kiosk: A Virtual Human Interviewer for Healthcare Decision Support Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org) (2014), pp. 1061-1068.

[0273]

[0011] J.O. Egede, D. Price, D.B. Krishnan, S. Jaiswal, N. Elliott, R. Morriss, M.J.G. Trigo, N. Nixon, P. Liddle, C. Greenhalgh, M. Valstar, Design and Evaluation of Virtual Human Mediated Tasks for Assessment of Depression and Anxiety, Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents (2021), pp. 52-59.

[0274]

[0012] S. Jaiswal, M. Valstar, K. Kusumam, C. Greenhalgh, Virtual Human Questionnaire for Analysis Of Depression, Anxiety and Personality, Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (2019), pp. 81-87, July.

[0275]

[0013] M. Kocur, P Schauhuber, V. Schwind, C. Wolff, N. Henze, The Effects of Self-and External Perception of Avatars on Cognitive Task Performance in Virtual Reality, 26th ACM Symposium on Virtual Reality Software and Technology (2020), pp. 1-11, November.

[0276]

[0014] E. Miller, D. Polson, Apps, Avatars, and Robots: The future of Mental Healthcare, Issues in Mental Health Nursing, 40 (3) (2019), pp. 208-214.

[0015] L Qui, I. Benbasat, An Investigation into the Effects of Text to Speech Voice and 3D Avatars on the Perception of Presence and Flow of Live Help in Electronic Commerce, ACM Transactions on Computer Human Interaction, 12 (4) (2005), pp. 329-355.

[0277]

[0016] M. Rheu, Y. Jang, W. Peng, Enhancing Healthy Behaviors through Virtual Self: A Systematic Review of Health Interventions using Avatars, Games for Health Journal, 9 (2) (2020), pp. 85-94.

[0278]

[0017] A.A. Rizzo, B. Lange, J.G. Buckwaiter, E. Forbell, J. Kim, K. Sagae, J. Williams, B.O. Rothbaum, J. Difede, G. Reger, T. Parsons, et al. An Intelligent Virtual Human System for Providing Healthcare Information and Support, J.D. Westwood, et al. (Eds.), Medicine Meets Virtual Reality, IOS Press (2011), pp. 503-509.

[0279]

[0018] Nikhil Balram , Mountain View , CA (US ), Advanced telemedicine system with virtual doctor, US10572626B2 , United States.

[0280]

[0019] James R. KozloskiJames W. Murdock, IVClifford A. PickoverGeorge F. Walker, Empathy injection for question-answering systems, US10592503B2, United States.

[0281]

[0020] Clavel Chloe, Labeau Matthieu, Cassell Justine, Socio-conversational systems: Three challenges at the crossroads of fields, Frontiers in Robotics and Al, Vol 9, 2022, https: / / www.frontiersin.org / articles / 10.3389 / frobt.2022.937825.

[0282]

[0021] S. Fernandes, R. Gawas, P. Alvares, M. Fernandes, D. Kale and S. Aswale, "Survey on Various Conversational Systems," 2020 International Conference on Emerging Trends in Information Technology and Engineering (ic-ETITE), Vellore, India, 2020, pp. 1-8, doi: 10.1109 / ic-ETITE47903.2020.126.

[0283]

[0022] Alec Radford et al., Language Models are Unsupervised Multitask Learner, (2019).

[0284]

[0023] Alec Radford et al., Improving Language Understanding by Generative Pre-training, (2018).

[0285]

[0024] Jacob Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, (2018).

[0286]

[0025] Jacob Devlin et al., BERT: Bidirectional Encoder Representations from Transformers, (2018).

[0287]

[0026] Zhilin Yang et al., XLNet: Generalized Autoregressive Pretraining for Language Understanding, (2019).

[0027] Wasserman EB, Shah MN, Jones CM, Cushman JT. The Neurologic Examination in the Emergency Department. Emerg Med Clin North Am. 2016;34(4):915-930. doi: 10.1016 / j.emc.2016.06.013.

[0288]

[0028] Filippi M, Rocca MA, Barkhof F, et al. Association between pathological and MRI findings in multiple sclerosis. Lancet Neurol. 2012;l l(4):349-360. doi: 10.1016 / S1474- 4422(12)70003-0.

[0289]

[0031] Douglas O’Shaughnessy, Invited paper: Automatic speech recognition: History, methods and challenges, Pattern Recognition, Volume 41, Issue 10, 2008, Pages 2965-2979, ISSN 0031-3203.

[0290]

[0032] Peter Lee, Ph.D., Sebastien Bubeck, Ph.D., and Joseph Petro, M.S., M.Eng., Benefits, Limits, and Risks of GPT-4 as an Al Chatbot for Medicine, March 30, 2023, N Engl J Med 2023; 388: 1233-1239; DOI: 10.1056 / NEJMsr2214184.

[0291]

[0033] Ihme Kias, Unni Anirudh, Zhang Meng, Rieger Jochem W., Jipp Meike, Recognizing Frustration of Drivers From Face Video Recordings and Brain Activation Measurements With Functional Near-Infrared Spectroscopy, Frontiers in Human Neuroscience, Vol 12, 2018, DOI=10.3389 / fnhum.2018.00327.

[0292]

[0034] L. C. De Silva, T. Miyasato and R. Nakatsu, "Facial emotion recognition using multi-modal information," Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications, Cat., Singapore, 1997, pp. 397-401 vol.l, doi:

[0293] 10.1109 / ICICS.1997.647126.

[0294]

[0035] Chamishka, S., Madhavi, I., Nawaratne, R. et al. A voice-based real-time emotion detection technique using recurrent neural network empowered feature modelling. Multimed Tools Appl 81, 35173-35194 (2022). https : / / doi . org / 10.1007 / s 11042-022- 13363-4.

[0295]

[0036] Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina (11 October 2018). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". arXiv: 1810.04805v2.

[0296]

[0037] Nicole Davis, Nils Olsen, Vanessa G.Perry, Marcus M. Stewart, and Tiffany

[0297] B. White, I’m Only Human? The Role of Racial Stereotypes, Humanness, and Satisfaction in Transactions with Anthropomorphic Sales Bots, Journal of the Association for Consumer Research, volume 8, number 1, January 2023, https: / / doi.org / 10.1086 / 722703.

Claims

Claims1. A medical examination administration system for use with or at a patient computer having a patient camera, patient microphone, patient input device, and a patient display device accessible by a patient, the system comprising: a processing device configured to create an avatar and an avatar dialogue, and display the avatar and avatar dialogue to the patient display device to perform a medical examination on the patient including a patient exercise to be performed by the patient; said processing device configured to receive patient image data from the patient camera, patient input data from the patient input device, and / or patient verbal data from the patient microphone, and determine a patient status based on the patient image data, patient input data, and / or patient verbal data; said processing device further configured to receive patient question input from the patient microphone or patient input device, generate a question prompt based on the patient status and the patient question input, apply the question prompt to a generative language program, receive an automated answer from the generative language program, and display the answer as part of the avatar dialogue.

2. The system of claim 1, wherein the patient status includes a patient emotional status, cognitive load, and / or frustration.

3. The system of claim 1 or 2, said processing device further configured to determine an answer requirement based on the patient status, evaluate the automated answer for the answer requirement, and generate another question prompt if the automated answer does not satisfy the answer requirement.

4. The system of claim 4, wherein the answer requirement is a sentiment or empathy level requirement, the evaluating the automated answer for the answer requirement generates an evaluation level of the answer requirement for the automated answer, and the automated answer does not satisfy the answer requirement if the evaluation level is below the sentiment or empathy level requirement.

5. The system of claim 3 or 4, where the another question prompt is generated to improve the answer requirement.

6. The system of any one of claims 3-5, wherein the evaluating the automated answer for the answer requirement ranks the quality of the answer based on the answer requirement.

7. The system of claim 6, wherein the evaluating is based on sentiment analysis.

8. The system of claim 6, wherein the evaluating is based on semantic similarity, named entity recognition, sentiment analysis, topic modeling, keyword extraction, word sense disambiguation, dependency parsing, textual entailment, and paraphrase detection.

9. The system of any one of claims 3-8, wherein the answer requirement includes word count, word simplicity, empathy, and risk mitigation that avoids potentially upsetting information.

10. The system of any one of claims 1-9, said processing device further configured to analyze the patient image data to determine patient metrics and a patient medical condition, and generate a patient report that includes a patient image from the patient image data and graphical representations of patient static metrics, patient dynamic metrics, the patient medical condition.

11. The system of any one of claims 1-10, wherein the medical examination comprises a neurological examination.

12. The system of any one of claims 1-11, wherein the medical examination comprises a Myasthenia Gravis neurological medical examination.

13. The system of claim 12, wherein the patient exercises include standardized exercises to determine Myasthenia Gravis, including sit to stand, arm extension, cheek puff, tongue to cheek, and diplopia.

14. The system of any one of claims 1-13, wherein the patient image data includes the patient image of the patient performing the patient exercise.

15. The system of any one of claims 1-14, wherein said processing device configured to receive patient image data from the patient camera and determine if the patient is paying attention to the avatar, properly positioned to perform the patient exercise, and properly performing the patient exercise, and to generate the avatar dialogue in response to the determination to complete the medical examination.

16. The system of claim 15, wherein if the patient is determined to not be properly positioned, the patient dialogue instructs the patient to properly position.

17. The system of any one of claims 1-16, wherein the patient dialogue is based on the determined patient status.

18. The system of any one of claims 1-17, wherein the patient dialogue is provided dynamically and in real-time19. The system of any one of claims 1 -18, said processing device comprising: an image analysis module configured to perform image analysis on the patient image data from the patient camera; a dialogue generator module configured to generate the patient dialogue based on a patient dialogue algorithm; and an answer generator module configured to generate the question prompt based on an answer algorithm.

20. The system of any one of claims 1-19, said processing device further configured to receive patient data, including the digital twin data, physiological data from patient physiological sensors at the patient computer, environmental data from patient environmental sensors at the patient computer, patient electronic health records data.

21. The system of claim 20, wherein the patient electronic health records data includes examination data such as the stage of examination and number of past visitations22. The system of claim 20, wherein the digital twin data includes biological characteristics of the patient, physical state variables for the patient, emotive state variables, cognitive state variables, and social state variables.

Images