Systems and method for clinical assessment loops using machine learning models

Abstract

In one aspect a method for generating an objective clinical assessment of a subject using a machine learning (ML) model. The method may include obtaining an observational clinical assessment rubric comprising a plurality of codes. The method may include obtaining a digital representation comprising observational data of the subject in a clinical environment. The method may include segmenting, using the ML model, the digital representation into a plurality of segments. The method may include pairing, using the ML model, a first segment of the plurality of segments with a first code of the plurality of codes. The method may include assigning, based on the clinical assessment rubric, a first score for the pairing. The method may include generating a clinical assessment of the subject based on the first score.

Description

SYSTEMS AND METHOD FOR CLINICAL ASSESSMENT LOOPS USING MACHINELEARNING MODELSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 519,623, filed on August 15, 2023, which is incorporated herein by reference in its entirety.TECHNICAL FIELD

[0002] Aspects of the present disclosure relate to clinical assessment loops using machine learning models.BACKGROUND

[0003] Clinical assessment plays an essential role in the development of medical students. Typically, a clinical assessment involves a licensed physician accompanying the student as they conduct cases as if they were the physicians responsible for the patients. The student may gather data from history and physical examinations, communicate conclusions to the patient, and negotiate a management plan. The attending physician may observe and assess the student’s clinical performance. The assessment may include evaluating a student’s performance using data gathering techniques, clinical management, and professional documentation.

[0004] The assessment may additionally include evaluating the softer skills required of professional physicians. The attending physician may have to score the student’s performance for multiple categories of a rubric. The categories may include, for example, 1) demonstrating compassion, integrity, and respect for others, 2) demonstrating responsiveness to patient needs that supersedes self-interest, and 3) demonstrating respect for patient privacy and autonomy. As medical knowledge and physician roles evolve at a rapid pace, clinical assessment models leverage data and technology in an attempt to improve the efficiency of clinical assessments.SUMMARY

[0005] There are multiple challenges that commonly arise with existing clinical assessment models.

[0006] For example, clinical assessment models often yield cursory, fragmented, and biased data. A lack of adequate resources may yield inadequate data. Additionally, evaluating physicians may lack expertise in using reliable assessment strategies or adequate time to provide pertinent narrative feedback.

[0007] For another example, norm referenced assessment (e g., comparing the student to their peers) carries the potential for unconscious bias as current criterion referenced assessments (e.g., competency -based Likert scales) are often vague and difficult to interpret. An inconsistent standard and method of assessments across evaluators prevents consistent performance tracking and feedback.

[0008] According to one aspect, a method for generating an objective clinical assessment of a subject using a machine learning (ML) model. The method includes obtaining an observational clinical assessment rubric comprising a plurality of codes. The method includes obtaining a digital representation comprising observational data of the subject in a clinical environment. The method includes segmenting, using the ML model, the digital representation into a plurality of segments. The method includes pairing, using the ML model, a first segment of the plurality of segments with a first code of the plurality of codes. The method includes assigning, based on the clinical assessment rubric, a first score for the pairing. The method includes generating a clinical assessment of the subject based on the first score.

[0009] According to some embodiments, the digital representation is a narrative description comprising a string of text, and the method further includes: parsing the string of text into a plurality of tokens, and classifying one or more tokens of the plurality of tokens as being associated with the first code.

[0010] According to some embodiments, the ML model is a large language model (LLM).

[0011] According to some embodiments, obtaining the digital representation comprises: converting audio data into the string of text; converting video data into a string of text; and receiving the string of text from a computing device. The computing device comprises a second ML model.

[0012] According to some embodiments, the method further includes: for each code of the plurality of codes, parsing the code into a plurality of tokens. The code comprises a string of text.

[0013] According to some embodiments, the method further includes: modifying a code in the clinical assessment rubric based on the observational data.

[0014] According to some embodiments, each code of the plurality of codes is associated with a trait or behavior.

[0015] According to some embodiments, the observational data indicates a level of eye contact.

[0016] According to some embodiments, pairing comprises: determining a classification score based on the first segment and the first code; and determining the classification score is above a classification threshold.

[0017] According to some embodiments, the method further includes: applying a filter to the digital representation. The filter performs one or more of: distorting private health information, blurring sensitive information, or transforming a human subject in the digital representation into a race and gender neutral humanoid

[0018] In another aspect, an apparatus is provided. The apparatus includes one or more processors and a non-transitory computer-readable medium coupled to the processor. The one or more processors are configured to perform any one of the methods described above.

[0019] In another aspect, a computer program product is provided. The computer program product includes a non-transitory computer readable medium including computer readable instructions that, when executed by one or more processors, cause the one or more processors to perform any one of the methods described above.BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of embodiments of the invention.

[0021] FIG. 1 illustrates a block diagram, according to some embodiments.

[0022] FIG. 2 illustrates a block diagram, according to some embodiments.

[0023] FIG. 3 is a flow diagram illustrating a process, according to some embodiments.

[0024] FIG. 4 is a flow diagram illustrating a process, according to some embodiments.

[0025] FIG. 5 is a flow diagram illustrating a process, according to some embodiments.

[0026] FIG. 6 is a method, according to some embodiments.

[0027] FIG. 7 is a block diagram illustrating an apparatus, according to some embodiments.DETAILED DESCRIPTION

[0028] Aspects of the present disclosure relate to generating objective clinical assessment of a medical student using one or more machine learning (ML) models. The ML models may provide an ethical, equitable, science-based approach to optimizing and individualizing the learning environment, training, and career outcomes of medical professionals based on multiple data points, data science, and research. Providing a clinical assessment using a ML model engages and empowers the learner, provides reliable data free of bias, provides data that encourages change and tracks progress, instills a growth mindset, accountability, and life-long learning, and minimizes administrative burden on the faculty.

[0029] ML models trained on clinical assessment data may efficiently translate observational data into a clinical assessment in a consistent and non-biased manner. The ML models may translate vague and difficult to interpret narrative statements into a concrete observational coding rubric for improved clinical training.

[0030] FIG. 1 is a block diagram, according to some embodiments. In some embodiments, FIG. 1 illustrates Architecture 100 for generating an objective clinical assessment of a subject 101B (e.g., medical student) using a ML model. A user 101A may obtain observation data associated with a clinical performance of subject 101B. For example, the observation data may be a narrative description provided by a physician of subject lOlB’s performance during a surgery. Architecture 100 allows a user 101A to assess and score the clinical performance of subject 101B. User 101A may use Architecture 100 to transform the observational data associated with subject 10 IB into an objective clinical assessment.

[0031] User 101A of Architecture 100 may operate a user device 102. User device 102 may be an electronic computing device, such as a mobile device, laptop, computer, desktop, tablet, and the like, capable of communication with one or more other devices through a network such as the Internet. User device 102 may include hardware configured to run a clinical assessment loop (CAL) software system. In some embodiments, user device 102 may transmit the observational data towards server 104 over network 110 (e.g., the internet). Server 104 may be configured to run the CAL software system.

[0032] User device 102 may additionally include an image sensor (e.g., video camera) configured to capture visual observational data (e.g., images and video). The image sensor may be configured to track the motion of one or more persons with a pan-tilt-zoom feature. Userdevice 102 may additionally include an audio sensor (e.g., universal serial bus (USB) microphone) configured to capture audio observation data. The audio sensor may be configured to capture a wide range of frequencies with a high signal to noise ratio. In some embodiments, Architecture 100 may include one or more additional devices for capturing observational data. For example, a hospital room may include multiple cameras and microphones for monitoring activity. The one or more additional devices may transmit the captured observational data to user device 102 and / or server 104 for processing.

[0033] User 101 A may be associated with a type of user. For example, user 101 A may be a clinical assessment team (CAT) member, educator, faculty, staff, learner, or patient. In some embodiments, each user may have a radio frequency identification (RFID) card identifying the specific user and the type of user. User device 102 may additionally include a RFID scanner configured to read the RFID card of each user.

[0034] In some embodiments, user device 102 and / or server 104 may transmit data associated with the clinical assessment (e.g., observational data and rubric scoring) to data storage 106. The data may be stored under a comprehensive data security and governance framework. In some embodiments, data storage may comprise multiple cloud-computing devices and may store the encrypted data using block-chain and federated network technology. In some embodiments, a system administrator with access at a central level may have privileged control over the software platform and may anonymized the aggregate data. In some embodiments, the data may be segmented and isolated for individual institution data security with institutionspecific access control.

[0035] In some embodiments, user device 102 and / or server 104 may be configured to generate the objective clinical assessment using an ML model. User device 102 and / or server 104 may obtain an observational clinical assessment rubric comprising a plurality of codes. Each code in the rubric may be associated with a desired trait or behavior of a medical student (e.g., communication skills, empathy, integrity, etc.). User device 102 and / or server 104 may obtain a digital representation of the observational data of the student in a clinical environment. For example, the digital representation may be a string of text converted from an audio description of the student’s performance given by his attending physician. Then using a ML model, user device 102 and / or server 104 may segment the digital representation into a plurality of segments. Again using the ML model, user device 102 and / or server 104 may pair one ormore of the plurality of segments with a code of the plurality of codes. Based on the clinical assessment rubric, user device 102 and / or server 104 may assign a score to each of the pairings. User device 102 and / or server 104 may then generate the objective clinical assessment of the student based on the score.

[0036] FIG. 2 illustrates a block diagram of an overview example of CAL model 200 implemented by Architecture 100. CAL model 200 may standardize observational assessment with structured feedback to efficiently and reliably implement precision education. CAL model200 may include three components: a clinical assessment team (CAT) 202, clinical coaches 204, and a learner 206.

[0037] CATs 202 may be embodied as medical education assessment specialists trained to frequently obtain competency data using standardized behavioral anchors. CATs 202 may be embedded in a clinical setting to observe learner’s 206 behavior as they perform clinical activities 205 and gather additional targeted quantitative and narrative feedback from the clinical team, including patients. In some embodiments, CATs 202 may interview physicians minimizing workflow disruptions while maximizing the quality of collected data. Transparent criterion referenced behavioral anchors may provide a shared mental model of expectations to promote reliability across raters, experiences, and cohorts. Observational data may be documented and processed in real-time into standardized assessment data 201 for a user profde for each learner 206.

[0038] Clinical Coaches 204 may analyze and incorporate standardized assessment data201 into coaching techniques. The coaching techniques may include the four phases of a master adaptive learner (MAL) cycle. Coaching sessions may utilize the MAL coaching framework at the task level (e.g., history taking), the competency level (e.g., professionalism), and the metacognitive level (e.g., confidence). Clinical Coaches 204 may gauge learner’s 206 proficiency of the MAL cycle application and encourage graduated autonomy in reviewing external feedback, identifying target content, and creating action plans 203. Clinical Coaches 204 may support learner’s 206 development of individualized learning strategies through scaffolded techniques and focusing on each learner’s 206 personal strengths, experiences, and needs. By incorporating coaching sessions throughout the clerkship year, each learner 206 will have longitudinal support to efficiently maximize their potential across all competencies and to prepare for future challenges.

[0039] Learners 206 (e.g., medical students) may collaborate with their clerkship coach 204 to identify personal strengths and learning opportunities. They may participate in informed self-assessment using the assessment data. The embodiments described herein may leverage data and machine learning to improve the personalization of education and the efficiency of learning.

[0040] Referring back to FIG. 1, user device 102 may obtain observational data using process 300. FIG. 3 is a flow diagram, according to some embodiments. In some embodiments, FIG. 3 illustrates a process 300 for gathering observational data in the CAL system. User device 102 may perform the steps of process 300. In some embodiments, another computer device, such as server 104, may perform one or more steps of process 300. Process 300 may begin with block 302. In some embodiments, user device 102 is configured to render a graphical user interface (“GUI”) to present one or more displays to a user as described herein.

[0041] At 302, user device 102 may present a secure CAT login display to a user. In FIG. 3, the user may be embodied as a member of a CAT. User device 102, however, may present a custom interface for each type of user. For example, user device 102 may present a learner with a first type of secure login display and an educator with a different second type of secure login display. The user may access the secure login display using an application or website. After inputting their login credentials, the user may input observational data into user device 102.

[0042] At 304, user device 102 may display a CAT dashboard to the user. The CAT dashboard may display the learners assigned to the user. The CAT dashboard may include productivity features (e.g., message center, task list, and calendar). Additional productivity features in the CAT dashboard may be included to streamline workflow by automating dynamic workflow scheduling. A ML model may be configured to optimize task priority in the user’s schedule based on learner activities, schedules, and assessment needs. The ML model may continuously update the user’s schedule based on updated learner data. In some embodiments, the productivity features may be optimized by integrating with 3rd party scheduling and task applications. For each learner, the CAT dashboard may include a thumbnail profile including pending assessment needs and deadlines. The user may select one or more learner’s profile to move to an individual learner assessment package dashboard.

[0043] At 306, user device 102 may display an individualized learner assessment package dashboard to the user. The individualized learner assessment package dashboard may provide an overview of complete and pending individual learner assessments. In someembodiments, the overview may include group headings for a set of assessments under which the subset assessment topics are displayed. For each topic, productivity features may be customized and include icons for due dates, topic quotas, estimated times to complete each assessment, participants needed to complete each assessment, and difficulty in completing each assessment. For example, an assessment of a given topic may require 10-15 minutes of direct observation, an interview of a supervising educator, 10 minutes of documentation, and rank as “difficult” to complete because the trait is rarely encountered. The above information may be continuously updated using data analytics discussed below in step 324. The user’s assessment productivity metrics may be factored when determining the user’s task schedule at step 304. In some embodiments, the user may select a specific trait and / or topic to being the assessment.

[0044] At 308, user device 102 may display an assessment topic screen. The assessment topic screen may display one or more data entry fields. In some embodiments, a topic title may be displayed at the top of the screen with a brief summary containing defining behavioral features, productivity metrics, and special instructions for task completion (e.g., setting, learner’s interaction target, assessment pitfalls, previously created task notes, etc.). The topic being assessed may be identified with a unique serial number. In some embodiments, the user may select from session information, observational rubric assessment input, or narrative assessment input.

[0045] At 310, the user may input session information into user device 102. The session information may include standard information such as a date / time timestamp to start the session, the participants involved, and the course the learner is completing. Qualitative data regarding the context of the session may be entered such as unique session features, observed details before and after the session, patient types (new vs established), learner experience level with the encounter type, extrinsic distractors, etc.

[0046] At 312, the user may input an observational rubric assessment input into user device 102. In some embodiments, the rubric may be embodied as a grid with point values (row) assessment categories (column), and brief behavioral descriptors for each point value. For example, one category may be labelled as knowledge of practice with a point scoring system of 1-5. Score 1 may contain a description of “demonstrates a lack of basic medical knowledge required to reason effectively” and score 5 may contain a description of “demonstrates astute clinical reasoning through targeted, hypothesis-generated questions.” As the user (e.g., CAT)observes the learner’s behavior, they may select one or more behavioral descriptors for each category. The user may additionally add comments to specific selections.

[0047] At 314, a ML model may provide clarifying questions based on assessment codebook resources if the observed behavior is indeterminate for a specific point value. For example, if the attending physician cannot decide whether to rate the learner’s performance a 3 or a 4, the ML model may ask the physician one or more questions. Based on the responses, the ML model may suggest a score value. In some embodiments, the rubric may include hyperlinks to content in the assessment codebook resources. The assessment codebook may be embodied as source reference material used to train CATs and may be a training data set for ML models. The assessment codebook may include descriptive narratives to establish the context of the traits and topics being assessed in the learner’s field, defines observable behaviors at different performance levels, and exemplars of each.

[0048] At 316, user device 102 may process the rubric scores using an algorithm for score aggregation and mapping to assigned insight reports (e.g., performance curves by topic serial number). User device 102 may link the rubric scores with the session data, comments, and a user profile to allow bidirectional reference and validation of information. In some embodiments, user device 102 may prepare the data for advanced analytics by performing data preprocessing and feature engineering. The data preprocessing and feature engineering may include one or more of the following steps:

[0049] 1) Data cleaning to remove errors, inconsistencies, and inaccuracies in the data. In some embodiments, user device 102 may keep data outliers, as they may be useful in discovering new trends and insights. 2) Data transformation. For example, user device 102 may perform normalization and standardization techniques with the rubric. 3) Feature selection for one or more metrics. The feature engineering at this step may be included as part of a continuous improvement / machine learning as it finds new relevant variables for new trends. 4). Data splitting to divide what may be needed for the clinical assessment and what may be needed to develop and validate a ML model.

[0050] At 318, the user may input a narrative assessment into user device 102. The user may input the narrative assessment via written data entry or voice recognition with transcription. In some embodiments, user device 102 may be configured to transcribe foreign languages into English. The user may input free text in a comment box. Using the narrative assessment, userdevice 102 may populate multiple data fields depending on the assessment topic. In some embodiments, a ML model may analyze the narrative input and provide differentiating and clarifying questions. The ML model may include questions based on detected assessment gaps in in the learner’s profile. The user may use the question provided by the ML model to expand upon the narrative assessment.

[0051] At 320, user device 102 may compile and sort the narrative content by coding data using a ML model. The ML model may be embodied as a large language model. User device 102 may link the narrative data linked to session data input, comments, and user profile data to allow bidirectional reference and validation of information. In some embodiments, user device 102 may prepare the data for advanced analytics by performing data preprocessing and feature engineering. The data preprocessing and feature engineering may be similar to that described in step 316. In some embodiments, transforming narrative data may include encoding categorical data and adjusting for time series with rolling statistics.

[0052] At 322, each participant providing assessment input may review and verify the observational data. For example, the user (CAT) and attending physician may verify the observational data is correct. Once the data is verified, each participant may digitally sign. In some embodiments, user device 102 may begin processing the observational rubric input and narrative assessment input once the data is verified and the participating parties sign.

[0053] At 324, user device 102 may use a ML model to perform descriptive, diagnostic, predictive, and prescriptive multi-dimensional analysis across individual learner, cohort, individual program, and system levels. The narrative data may be coded into the context of the assessment program using the ML model. The ML model may be trained using the assessment codebook resources as the training data set. Analytics of the assessment topics and meta-analysis of the assessment process may be performed (efficiency, bias). Anomaly detection and deep learning provided by the ML model may continuously provide quality improvement of the assessment system to identify new themes and behaviors, redundant themes with poor trait differentiation, underutilized metrics, flawed metrics, and revision recommendations.

[0054] At 326, user device 102 may update a user profile based on the obtained and process data. Each user profile may contain demographic information. Users may be defined as CATs, faculty, staff, or learners. Either the patient or the patient’s bedside nurse upon intake mayenter patient profile information. Each user profile may be linked to a corresponding assessment encounter from which the data was obtained.

[0055] At 328, user device 102 may employ advanced deidentification methods to strip data of personally identifiable information for enhanced privacy and security measures. Deidentifcation methods include generalization and aggregation. Deidentified data may be aggregated into a central repository for aggregate analysis.

[0056] At 330, user device 102 may generate insight reports based on the user type. Reports may be generated in real time and may be customized through filtering. The insight reports may include progress curve analysis, behavioral insights (trend identification, anomaly detection), performance prediction, personalized prescriptive recommendations, comparative performance reports, automated feature engineering reports, privacy and security assurance reports, demographic trend reports, continuous enhancement reports, and data quality / integrity reports.

[0057] FIG. 4 is a flow diagram, according to some embodiments. In some embodiments, FIG. 4 illustrates process 400 for accessing the observational data in the CAL system. User device 102 may perform the steps of process 400. In some embodiments, another computer device, such as server 104, may perform one or more steps of process 400. Process 400 may begin with block 402.

[0058] At 402, user device 102 may provide the generated insight reports to one or more users. User device 102 may provide the insight reports to specific user types based on tasks required for their role.

[0059] At 404, user device 102 may present secure login and interface for users in an educator role. These users may receive insight reports about the assigned learners, comparative reports (cohort, system), and personal educator analysis reports.

[0060] At 406, user device 102 may display an educator dashboard to the user. The educator dashboard may display the assigned learner thumbnail profiles for selection. The educator dashboard may include productivity features similar to the CAT dashboard. An ML model may provide automated dynamic scheduling for the user based on learner activity, educator workflow, available schedules, and feedback needs. Feedback needs may be determined by a baseline feedback plan, progress curve analysis, behavioral insights, comparative analysis,personalized prescriptive recommendations, and anomaly detection described in the insight reports.

[0061] At 408, the user may access one or more educator session documents on user device 102. The educator session documents may include educator feedback documents, frameworks, and templates. The user may access to the learner’s performance reports and may access components of observational data such as progress curves with behavior comments or specific narrative feedback for a session. In some embodiments, an ML model may suggest personalized prescriptive recommendations based on learner performance and needs. An interactive personalized education plan template may be shared with the learner for collaboration.

[0062] At 410, a user may modify assessments for the learner. The user may add additional assessments for the learner using the educator session documents in both quantitative and narrative form. The user may document additional metrics and assessment of the feedback process. Data entry may be about a single learner or multiple learners. Data entry may be written or transcribed via voice recognition.

[0063] At 412, user device 102 may process the assessment data as described in steps 316 and 320. Metrics and data from the feedback session may be processed for program evaluation. Learner intervention and action plans may be documented as a time stamp on the progress curve to analyze deviations in behavior. Narrative data may be coded as described previously.

[0064] At 414, user device 102 may provide secure login and interface for users with the learner role. Learners (e.g., medical students) may have limited access to insight reports and observational data. Learners may independently view progress curves and narrative summary reports. Educators may have the option to review reports prior to releasing the report to the learner.

[0065] At 416, user device 102 may display a learner dashboard. The learner dashboard may allow access to real-time performance curves and insight reports. The dashboard may be customizable and displayed in thumbnail view. The user may be able to review assessment metrics on the learner dashboard. Learners may select buttons to access feedback session documents or evaluations to complete. In some embodiments, the learner dashboard may have limited productivity features available.

[0066] At 418, the user may access learner feedback session documents. The learner feedback session documents may include insight reports, templates, documents, and workbooks.

[0067] In some embodiments, a user may interact with reflective practice chats integrated with a ML model to explore interactive prompts. The chat feature may include questions and conversations related to specific reflective practice topics such as bias, medical errors, team dynamics, etc. The chat feature may be embodied as a chatbot. The chatbot may ask probing questions and paraphrase the learner’s comments. It may use different techniques to facilitate the learner to self-assess and self-explore but may not offer opinions or try to teach / change behavior. Once complete, the learners narrative comments may be analyzed via rubrics and methods similar to the other narrative data. The learner’s narrative comments may be used to evaluate competencies related to internal perspectives and attitudes.

[0068] At 420, user may access templates, rubrics, and narrative feedback evaluations which may be completed by the learner.

[0069] At 422, user device 102 may process data from the feedback session and student evaluations as described in steps 316 and 320. Learner intervention and action plans are documented as a time stamp on the progress curve to analyze deviation in behavior. Narrative data is coded as described previously.

[0070] At 424, user device 102 may perform feedback data analytics. Feedback data analytics may add quantitative and narrative assessment data for specific topics from feedback sessions similar to the process described in FIG. 3. The information may be incorporated into the assessment portfolio for each learner and at the cohort level. Specific analysis of feedback quality, techniques, and outcomes may be analyzed for educator and program evaluation. Feedback data analytics and insight reports may be generated similar to the process described in Step 330.

[0071] At 426, user device 102 may present secure login and interface for users with an administrator role. The user may have access to all insight reports specific to their institution. Comparative reports can be requested from the central system. In some embodiments, the administration interface may be integrated with 3rd party learning management systems.

[0072] At 428, deidentified data, analytics, and insight from across multiple programs may be accessed and analyzed at a central system level.

[0073] At 430, user device 102 may employ advanced deidentification methods to strip data of personally identifiable information for enhanced privacy and security measures. Deidentifcation methods include generalization and aggregation. Deidentified data may be aggregated into a central repository for aggregate analysis.

[0074] At 432, user device 102 may generate insight reports based on the user type. Reports may be generated in real time and may be customized through filtering. The insight reports may include progress curve analysis, behavioral insights (trend identification, anomaly detection), performance prediction, personalized prescriptive recommendations, comparative performance reports, automated feature engineering reports, privacy and security assurance reports, demographic trend reports, continuous enhancement reports, and data quality / integrity reports.

[0075] FIG. 5 is a flow diagram, according to some embodiments. In some embodiments, FIG. 5 illustrates process 500 for processing the observational data using a ML model in a CAT system. User device 102 may perform the steps of process 500. In some embodiments, another computer device, such as server 104, may perform one or more steps of process 500. Process 500 may begin with block 502.

[0076] At 502, a user may select a ML model referred to as “CATai.” The user may select CATai in an assessment settings option. In some embodiments, CATai may be embodied as a large language model. Users may have the option to opt out of participation. CATai may operate on user device 102. In some embodiments, CATai may operate on another computing device such as server 104.

[0077] At 504, user device 102 may identify one or more persons and their user profile. For example, a user may identify that a learner, physician, CAT, and patient are in a hospital room. In some embodiments, user device 102 may identify the one or more persons based on a badge (e.g., Badge ID, RFID) or using facial recognition. User device 102 may obtain data associated with the badge and facial identification from the user profiles. Patients may have system identification with the hospital bracelet to allow for tracking in relation to the learner. If user device 102 is unable to identify a person with a profile, the person may be deemed a visitor.

[0078] At 506, the CATai may obtain audio and video data input. The audio and video data input may include gross movement tracking, facial expression tracking, spacial relationships to other subjects, voice recognition with natural language processing, and speech patterntracking. The audio and video data input may be embodied as the observation data described in FIGs. 3 and 4. In some embodiments, the CATai platform may obtain multisource feedback via direct interviews. The CATai platform may conduct interviews by remote human video, Al avatar, written prompts, or audio prompts. Feedback from educators, peers, medical staff, patients, and visitors may be written or verbal and can provide translation assistance. A ML model may generate follow up probing and clarifying questions in the context of the assessment structure. In some embodiments, the audio and / or visual data may be received from a computing device comprises a second ML model (e.g., virtual nurse).

[0079] At 508, user device 102 may process the audio data for verbal content, nonverbal content, and / or relational data. Audio data processing may use voice recognition to document verbal content and the source of the spoken content based on the user. Nonverbal audio input related to verbal data may include vocal patterns of intonation, cadence, sounds, and volume. Relational data such as length of pauses, reactions, and simultaneous speech may be documented. In some embodiments, the native language and translated verbal content are both preserved. In some embodiments, the audio data may be converted into a string of text.

[0080] At 510, user device 102 may process the video data. The video data processing may include identifying gross motor movements, fine motor movements including facial expressions, and relational movement (touching, mirroring, reactions). Movements may be processed to include timestamps within the encounter. In some embodiments, the video data may be converted to a string of text.

[0081] At 512, user device 102 may perform audio and video data identity protection and bias mitigation in some embodiments. A ML model may detect and remove personally identifiable information from the audio data. The ML model may additionally perform voice modulation modifying the gender of a person and / or altering their accent. Video filtering may modify the appearance of the subjects using the same demographic features in the user profile, different demographic features, or neutral humanoid demographics. Filtering, however, may allow for the preservation of facial expression data.

[0082] At 514, user device 102 may perform data analytics following the methods described step 324. Using CATai may allow for the assessment of multiple rubrics simultaneously with every patient encounter. The volume of data at the individual, cohort, and aggregate level require enhanced data processing, analysis, storage, and security. CATai mayanalyze unique features such as the impact of verbal cadence or voice inflection on patient perceptions of the learner.

[0083] At 516, user device 102 may obtain data from and update the user profiles based on the CATai analytics.

[0084] At 518, user device 102 may obtain session information for performing data analytics. Session information may include standard information such as a date / time timestamp to start the session, the participants involved, and the course the learner is completing.Qualitative data regarding the context of the encounter can be entered such as unique encounter features, observed details before and after the session, patient types (new vs established), learner experience level with the encounter type, extrinsic distractors, etc. In some embodiments, user device 102 may identify a red flag behavior, which may result in immediate notification educators.

[0085] At 520, user device 102 may obtain an observational clinical assessment rubric. Rubrics may be fused and coded to allow continuous assessment of all topics and behaviors simultaneously. Audio input may be coded in a manner described previously. Computer vision may train Al models (e.g., CATai) to identify and interpret nuanced movement patterns. User device 102 may obtain the rubric by altering the assessment code book for computer interpretation of abstract behavioral data based on detailed audio and video input. In some embodiments, user device 102 may modifying a code in the clinical assessment rubric based on the observational data.

[0086] At 522, federated learning may improve data security through program analysis and improvement at a local and system level. Automated feature engineering continuously generates transformative insights and features through analysis of aggregate deidentified data.

[0087] At 524, user device 102, using CATai, may code the narrative content (Al audio and subject interview data) to the adaptive rubrics for inclusion with the data analysis and insight reports. Content that remains uncoded may be analyzed for themes and anomaly detection. Narrative data summaries may be developed to condense information into useful content.

[0088] In some embodiments, coding the narrative content may include segmenting, using CATai, the narrative content into a plurality of segments; pairing, using CATai, a first segment of the plurality of segments with a code of the clinical assessment rubric; and assigning,based on the clinical assessment rubric, a first score for the pairing. In some embodiments, paring may comprise determining a classification score based on the first segment and the code and determining the classification score is above a classification threshold.

[0089] Where the narrative content comprising a string of text, user device 102 may parse the string of text into a plurality of tokens, and classify one or more tokens of the plurality of tokens as being associated with the first code. Then for each code of the plurality of codes, parsing the code into a plurality of tokens, wherein the code comprises a string of text.

[0090] While step 524 describes coding narrative content, user device 102 may additionally code non-narrative content into the rubric. The non-narrative content may include audio data, video data, and other performance metrics. For example, how long the learner held eye contact with the patient.

[0091] At 526, user device 102 may generate CATai insight reports. The CATai insight reports may be embodied as an objective clinical assessment. The CATai insight reports may be consistent with the insight reports discussed in steps 330 and 432. The CATai insight reports may include analytics generated by CATai, including voice patterns, subject interaction patterns, correlation with patient feedback, etc. In some embodiments, user device 102 may identify bias at different relationship levels (learner to patient, learner to staff, staff to learner, etc) using the results of the filtering step and integration of user profile data. In some embodiments, user device 102 may generate a clinical assessment of the subject based on the first score.

[0092] At 528, user device 102 may provide the CATai insight reports to one or more users for review. For example, a CAT may review the CATai insight reports in FIG. 3 and educators may review the CATai insight reports in FIG. 2.

[0093] At 530, user device 102 may use an ambient natural language processing algorithm in some embodiments. The ambient natural language processing algorithm may be external to the assessment features described above and may be used for system development at the central level.

[0094] At 532, user device 102 may deidentify all CATai data and collect the data as an aggregate data set. In some embodiments, the user device 102 may encrypt the data.

[0095] FIG. 6 is a method, according to some embodiments. The method 600 involves a series of steps that may be performed by Architecture 100. In some embodiments, method 600 isfor generating an objective clinical assessment of a subject using a machine learning (ML) model.

[0096] Step 602 includes obtaining an observational clinical assessment rubric comprising a plurality of codes.

[0097] Step 604 includes obtaining a digital representation comprising observational data of the subject in a clinical environment.

[0098] Step 606 includes segmenting, using the ML model, the digital representation into a plurality of segments.

[0099] Step 608 includes pairing, using the ML model, a first segment of the plurality of segments with a first code of the plurality of codes.[000100] Step 610 includes assigning, based on the clinical assessment rubric, a first score for the pairing.[000101] Step 612 includes generating a clinical assessment of the subject based on the first score.[000102] In some embodiments, the digital representation is a narrative description comprising a string of text, and the method further comprises: parsing the string of text into a plurality of tokens; and classifying one or more tokens of the plurality of tokens as being associated with the first code.[000103] In some embodiments, the ML model is a large language model (LLM).[000104] In some embodiments, obtaining the digital representation comprises: converting audio data into the string of text; converting video data into a string of text; and receiving the string of text from a computing device, wherein the computing device comprises a second ML model (e.g., virtual nurse).[000105] In some embodiments, method 600 further includes, for each code of the plurality of codes, parsing the code into a plurality of tokens, wherein the code comprises a string of text.[000106] In some embodiments, method 600 further includes modifying a code in the clinical assessment rubric based on the observational data.[000107] In some embodiments, each code of the plurality of codes is associated with a trait or behavior (e.g., communication skills, interpersonal dynamics teamwork, empathy, integrity, etc.).[000108] In some embodiments, the observational data indicates a level of eye contact.[000109] In some embodiments, pairing comprises: determining a classification score based on the first segment and the first code; and determining the classification score is above a classification threshold.[000110] In some embodiments, method 600 further includes applying a filter to the digital representation, wherein the filter performs one or more of: distorting private health information, blurring sensitive information, or transforming a human subject in the digital representation into a race and gender neutral humanoid.[000111] FIG. 7 is a block diagram of an apparatus 700, according to some embodiments. An apparatus 700 may perform all embodiments described above. In some embodiments, the apparatus 700 may be the user device 102 and / or server 104 as described herein. As shown in FIG. 7, the apparatus may comprise: processing circuitry (PC) 702, which may include one or more processors (P) 755 (e.g., a general purpose microprocessor and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like); a network interface 748 for enabling the apparatus to transmit data to and receive data from other devices connected to a network 710 (e.g., an Internet Protocol (IP) network such as the Internet) to which network interface 748 is connected; and a local storage unit (a.k.a., “data storage system”) 708, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 702 includes a programmable processor, a computer program product (CPP) 741 may be provided. CPP 741 includes a computer readable medium (CRM) 742 storing a computer program (CP) 743 comprising computer readable instructions (CRI) 744. CRM 742 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 744 of computer program 743 is configured such that when executed by PC 702, the CRI causes the apparatus to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, the apparatus may be configured to perform steps described herein without the need for code. That is, for example, PC 702 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.[000112] The following are certain enumerated embodiments further illustrating various aspects of the disclosed subject matter.[000113] Al . A method for generating an objective clinical assessment of a subject using a machine learning (ML) model, the method comprising: obtaining an observational clinical assessment rubric comprising a plurality of codes; obtaining a digital representation comprising observational data of the subject in a clinical environment; segmenting, using the ML model, the digital representation into a plurality of segments; pairing, using the ML model, a first segment of the plurality of segments with a first code of the plurality of codes; assigning, based on the clinical assessment rubric, a first score for the pairing; and generating a clinical assessment of the subject based on the first score.[000114] A2. The method of embodiment Al, wherein the digital representation is a narrative description comprising a string of text, and the method further comprises: parsing the string of text into a plurality of tokens; and classifying one or more tokens of the plurality of tokens as being associated with the first code.[000115] A3. The method of embodiment A2, wherein the ML model is a large language model (LLM).[000116] A4. The method of any one of embodiments A1-A3, wherein obtaining the digital representation comprises: converting audio data into the string of text; converting video data into a string of text; and receiving the string of text from a computing device, wherein the computing device comprises a second ML model (e.g., virtual nurse).[000117] A5. The method of any of embodiments A1-A4, further comprising: for each code of the plurality of codes, parsing the code into a plurality of tokens, wherein the code comprises a string of text.[000118] A6. The method of any of embodiments A1-A5, further comprising: modifying a code in the clinical assessment rubric based on the observational data.[000119] A7. The method of any of embodiments A1-A6, wherein each code of the plurality of codes is associated with a trait or behavior (e.g., communication skills, interpersonal dynamics teamwork, empathy, integrity, etc ).[000120] A8. The method of any of embodiments A1-A7, wherein the observational data indicates a level of eye contact.[000121] A9. The method of any of embodiments A1-A8, wherein pairing comprises: determining a classification score based on the first segment and the first code; and determining the classification score is above a classification threshold.[000122] A10. The method of any one of embodiments A1-A9, further comprising: applying a filter to the digital representation, wherein the filter performs one or more of: distorting private health information, blurring sensitive information, or transforming a human subject in the digital representation into a race and gender neutral humanoid.[000123] Bl. A device adapted to perform any one of the methods in embodiments Al-A10 and Bl.[000124] Cl . A computer program comprising instructions which when executed by processing circuity of a device causes the device to perform the method of any one of embodiments A1-A10 and Bl.[000125] While various embodiments of the present disclosure are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the abovedescribed exemplary embodiments. Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. Any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.[000126] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel. That is, the steps of anymethods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step.

Claims

CLAIMS1. A method (600) for generating an objective clinical assessment of a subject ( 10 IB) using a machine learning (ML) model, the method comprising: obtaining (602) an observational clinical assessment rubric comprising a plurality of codes; obtaining (604) a digital representation comprising observational data of the subject in a clinical environment; segmenting (606), using the ML model, the digital representation into a plurality of segments; pairing (608), using the ML model, a first segment of the plurality of segments with a first code of the plurality of codes; assigning (610), based on the clinical assessment rubric, a first score for the pairing; and generating (612) a clinical assessment of the subject based on the first score.

2. The method of claim 1, wherein the digital representation is a narrative description comprising a string of text, and the method further comprises: parsing the string of text into a plurality of tokens; and classifying one or more tokens of the plurality of tokens as being associated with the first code.

3. The method of claim 2, wherein the ML model is a large language model (LLM).

4. The method of any one of claims 1-3, wherein obtaining the digital representation comprises: converting audio data into the string of text; converting video data into a string of text; and receiving the string of text from a computing device, wherein the computing device comprises a second ML model.

5. The method of any of claims 1-4, further comprising:for each code of the plurality of codes, parsing the code into a plurality of tokens, wherein the code comprises a string of text.

6. The method of any of embodiments 1-5, further comprising: modifying a code in the clinical assessment rubric based on the observational data.

7. The method of any of embodiments 1-6, wherein each code of the plurality of codes is associated with a trait or behavior.

8. The method of any of embodiments 1-7, wherein the observational data indicates a level of eye contact.

9. The method of any of embodiments 1-8, wherein pairing comprises: determining a classification score based on the first segment and the first code; and determining the classification score is above a classification threshold.

10. The method of any one of embodiments 1-9, further comprising: applying a filter to the digital representation, wherein the filter performs one or more of: distorting private health information, blurring sensitive information, or transforming a human subject in the digital representation into a race and gender neutral humanoid.

11. An apparatus (700) comprising: a memory (742); and processing circuitry (702) coupled to the memory, wherein the apparatus is configured to perform the methods of any one of claims 1-10.

12. A computer program (743) comprising instructions (744) which when executed by processing circuity (702) causes the processing circuitry to perform the method of any one of claims 1-10.

Images