Privacy-Preserving Interpretable Skill Learning for Healthcare Decision Making

Abstract

A method and system for training a healthcare treatment machine learning model includes aggregating local weights from a set of clients to update a global set of weights for an imitation-based skill learning model. A set of local prototype vectors is clustered from multiple clients to generate clusters. Representative vectors are selected for the clusters as a global prototype set. Client-specific prototype vectors are determined for the clients based on the representative vectors. The updated global set of weights and the client-specific prototype vectors are distributed to the clients.

Description

[Technical Field]

[0001] Related Application Information This application claims priority to U.S. Patent Application No. 63 / 442,475, filed February 1, 2023, U.S. Patent Application No. 63 / 526,702, filed July 14, 2023, and U.S. Patent Application No. 18 / 425,715, filed January 29, 2024, each of which is incorporated by reference in its entirety. [Background technology]

[0002] The present invention relates to machine learning systems, and more particularly to interpretable skill learning by imitation. 2. Description of Related Art

[0003] Imitation learning is a type of machine learning technique that replicates expert skills through expert demonstrations. While imitation learning is useful in decision-making tasks, challenges limit its applicability in real-world scenarios. For example, imitation learning systems may lack inherent interpretability—explicitly explaining the underlying rationale for learned skills—making it difficult for human operators to trust the learned policy. Furthermore, due to the scarcity of expert demonstrations for a particular context, policies may need to be learned based on various data silos. However, in privacy-sensitive applications such as finance and healthcare, sharing such data may be difficult or impossible due to regulatory or practical constraints. Summary of the Invention

[0004] A method for training a healthcare treatment machine learning model includes aggregating local weights from a set of clients to update a global set of weights for an imitation-based skill learning model. Sets of local prototype vectors from multiple clients are clustered to generate clusters. Representative vectors are selected for the clusters as a global set of prototypes. Client-specific prototype vectors are determined for the clients based on the representative vectors. The updated global set of weights and the client-specific prototype vectors are distributed to the clients.

[0005] A system for training a healthcare treatment machine learning model includes a hardware processor and a memory that stores a computer program. When executed by the hardware processor, the computer program causes the hardware processor to: aggregate local weights from multiple clients to update a global set of weights for an imitation-based skill learning model, cluster the sets of local prototype vectors from the multiple clients to generate multiple clusters, select representative vectors of the multiple clusters as a global set of prototypes, determine client-specific prototype vectors for the multiple clients based on the representative vectors, and distribute the updated global set of weights and the client-specific prototype vectors to the multiple clients.

[0006] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0007] The present disclosure provides details in the following description of preferred embodiments with reference to the following figures.

[0008] [Figure 1] FIG. 1 is a block diagram of a system for integrated training of skill learning models, according to one embodiment of the present invention.

[0009] [Figure 2] FIG. 2 is a block diagram of a client for local imitation learning, according to one embodiment of the present invention.

[0010] [Figure 3] 1 is a pseudocode of a method for training an interpretable skill-learning model according to one embodiment of the present invention.

[0011] [Figure 4] 1 is a pseudocode of a method for local prototype projection according to one embodiment of the present invention.

[0012] [Figure 5] FIG. 1 is a block diagram of a healthcare facility using a skill mimicry system to guide patient treatment, according to one embodiment of the present invention.

[0013] [Figure 6] FIG. 1 is a diagram of patient treatment guided by actions selected by a skill imitation system, according to one embodiment of the present invention.

[0014] [Figure 7] FIG. 1 is a block diagram of a method for integrated training of a skill learning model, according to one embodiment of the present invention.

[0015] [Figure 8] FIG. 1 is a block diagram of a computing device capable of performing joint training of a skill learning model, according to one embodiment of the present invention.

[0016] [Figure 9] 1 is an exemplary neural network architecture that can be used as part of an imitation learning layer, according to one embodiment of the present invention.

[0017] [Figure 10]1 is an exemplary deep neural network architecture that can be used as part of an imitation learning layer, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] Interpretable skill learning can be implemented using global policies that incorporate data from various sources and provide explainable interpretations for each local user without violating privacy or data sovereignty. Interpretable skill learning can capture various patterns in the trajectories of experts' demonstrations and extract prototype information as skills that provide implicit guidance for policy learning and explicit explanations for the inference process. An aggregation mechanism is coupled with the skill-based learning model to maintain global information utilization and local interpretability under a unified framework.

[0019] Referring now to FIG. 1, a diagram of the integrated interpretable learning framework is shown. A client set C includes |C| clients 102. Each client aims to learn an interpretable skill learning model that includes a client-specific prototype layer that generates segment representations and connects other learning components that perform imitation learning tasks. A set of parameterized vectors in the prototype layer learns from the segment representations to establish prototypes, construct skill embeddings, and provide contextual information for imitation learning.

[0020] At the server 108, client models are aggregated by integrating averaging 104 and knowledge alignment 106 to integrate similar prototype information. After global and local training, the parameterized vectors of each client 102 can be transformed into interpretable prototypes through associations in the representation space of the local training data. Each client can explain the skills it employs based on the most similar prototypes it uses in its inference process.

[0021] At the start of global training, the server 108 initializes and distributes a globally interpretable imitation-based skill learning model to the local clients 102. The model can be divided into two parameter sets: one set containing the parameterized vectors of the prototype layer and the other set containing the weights of the convolutional layer 202 and the imitation learning layer 204.

[0022] At each global epoch, joint training is performed by aggregating local models from clients 102 after local training. Joint averaging 104 can be used to aggregate local models to generate segment representations and final behaviors. Knowledge alignment 106 is performed to correct for different parameterized vector sets among clients 102 due to heterogeneity in expert demonstrations among various clients.

[0023] Knowledge alignment 106 may include clustering to identify group membership for each parameterized vector, for example, using K-means and a Gaussian mixture model with K groups. Vectors with similar prototype / expert representations are merged into the same cluster. Based on the identified membership, each local vector may be matched with a global centroid vector of the group and distributed to local clients. As a result, each client 102 owns a specific set of vectors that best represent the prototype information from its own data throughout the process. Expert knowledge is shared and integrated among various clients 102.

[0024] While the present embodiment is described with particular focus on applications in the field of medical decision making, it should be understood that the present embodiment may be applied to a wide variety of different situations. For example, the client 102 may represent an autonomous vehicle, and the behavioral policy dictates the steering actions that the vehicle should take based on information sensed from the vehicle's environment.

[0025] 2, the client 102 is shown in more detail. The input for the sequence is provided to a convolutional layer 202 and an imitation learning layer 204. The convolutional layer 202 processes the segments and generates a representation vector at each time step t.

number

number

[0026] The imitation learning layer 204 learns the current time step s in the sequence. t Therefore, the imitation learning layer 204 is provided with the current state s t Behavior that applies skills based on a sequence of previous states in response to a t Because behaviors are based on a set of prototypes, it is possible to determine which prototypes were most influential in the behavior. These prototypes may be based on predetermined skills or conditions, and the selected prototypes can be used to provide evidence for why a particular behavior was selected.

[0027] The input data represents a sequence of states for each client 102. The input trajectory can be divided into non-overlapping segments of length m. Each state can come with m-1 previous states from the same trajectory, encoding the temporal dynamics up to the current time step t. The skill transitions across successive segments are captured by the learning model. Padding of initial states can be done during training if necessary.

[0028] The convolutional layer 202 encodes the temporal dynamics from the input segments. It is envisioned that a convolutional, recursive, or transformer-based encoder may be used. A convolutional encoder is particularly useful for extracting salient information from short segments.

[0029] The contextual imitation learning layer 204 determines the actions to be taken. Contextual policies are built on behavioral cloning to learn state-to-action mappings in a supervised manner.

[0030] The prototype layer 206 generates skill embeddings. Parameterized vectors are used to reconstruct the skill information stored by the segment representations, enabling a flexible interpretation structure. During inference, all parameterized vectors are projected onto prototypes, which are segment representations of the local training data. The underlying skill of a segment is explained by similar prototypes with high weights.

[0031] The input segment at time step t is expressed as:

number

number

number

number

number

[0032] The prototype layer 206 includes:

number

number

number

number

number

number

number

[0033] Therefore, skill embedding

number

number

number

number

[0034] The imitation learning layer 204 learns the input state s in a supervised manner. t Action from a t To learn the mapping to , we construct a contextual policy based on behavioral cloning. The contextual policy is parameterized by θ and is shown as follows:

number

number

[0035] The learning objectives include a segment-level imitation learning objective and three objectives to enhance the interpretability of non-overlapping segment prototypes and skills. Each component is introduced below with a batch size n and segment length m based on a batch of segments from the training dataset.

number

[0036] A first goal may be to minimize the loss of imitation learning using behavioral cloning.

number

[0037] The second objective is to regularize the non-overlapping segment representations output from the convolutional layer 202 to be as close as possible to their nearest prototypes, thereby enforcing a clustering structure of segments in the representation space.

number

number

number

[0038] The third objective is to

number

number

number

[0039] The two goals are to impose a double regularization on the learning of the convolutional layer 202 and the prototype layer 206 towards a clearer representational structure for interpretation.

[0040] The fourth objective is to enforce a variety of structures for the learnable parameterized prototype vectors, avoiding redundancy and improving the generalization of the resulting prototypes, where the distance L2 between each pair of vectors is limited by a threshold d min Penalties are imposed using

number

[0041] Referring now to Figure 3, we present pseudocode for joint training of skill prediction models. Global prototype alignment is performed by clustering during training, and local prototype projection is performed at model deployment time to enhance interpretability in a privacy-preserving manner.

[0042] The global server 108 starts at the beginning of global training and distributes the global interpretable skill learning model to the local clients 102. The model is divided into two parameter sets: one containing the learnable vectors of the prototype layer 206

number

number

[0043] For each global epoch, the server 108 aggregates the local models from the clients 102 after local training.

number

[0044] For this purpose, prototype aggregation may be performed on the server 108 side. After receiving the K×|C| vectors (where K is the number of prototypes and C is the client set), the server 108 performs clustering to identify the membership of each vector. Clustering may be performed by any suitable clustering method, such as K-means or Gaussian mixture models. Vectors with similar skill expressions may be placed in the same cluster.

[0045] After clustering, a centroid vector representing the skill mode is obtained by averaging all vectors belonging to the same cluster. Each vector is matched by a centroid vector based on the identified membership, and the vector set is distributed to local clients. In this way, each client 102 owns a specific set of vectors that best represent the skill from its own data throughout the training process, and skill knowledge is shared among different clients. A matching function replaces each element of the local vector with the centroid of the respective cluster, as determined by the identified cluster membership.

[0046] Referring now to Figure 4, pseudocode for local prototype projection is shown. After training is complete and the local vector set containing the skill representation is sufficiently regularized, the returned skill learning model may not be easily interpretable. This is because the vectors are approximate and not associated with real data with explicit prototypes. To make each local skill learning model interpretable, local prototype projection may be performed for each vector by assigning it to the training segment with the smallest L2 distance in the representation space.

[0047] After this stage, the vectors are updated to explicit prototypes, and the local model can capture the changing patterns to build meaningful skill embeddings based on the prototypes. Because this stage is privacy-sensitive, it may be performed only once on local data during the inference and inference stages for each client 102, preserving the privacy of expert demonstrations and ensuring that data or data representations are not leaked to the global server 108.

[0048] Referring now to FIG. 5, a diagram of skill learning in the context of a healthcare facility 500 is shown. Imitation learning may be used to learn mappings between states and actions when formulating patient treatments in a healthcare context. Imitation learning aims to replicate expert actions, such as diagnostic and therapeutic actions performed by physicians, based on demonstrations from a series of recordings. To this end, an interpretable sequence modeling framework can be used to identify expert trajectories based on sequence data with temporal characteristics. Learning and inference are performed at the segment level, capturing the temporal variation of states and identifying skills that are transferable across different trajectories. Thus, an interpretable skill learning model is provided to learn treatment strategies using segment-level expert demonstrations, resulting in representative skills that are transferable across different trajectories. The prototypes described above represent well-defined skills in the form of therapeutic actions that can be performed in a healthcare context.

[0049] Imitation learning can be used to aid in the monitoring and treatment of multiple patients, for example, to accommodate changes in a patient's specific healthcare needs. The healthcare facility may include one or more medical professionals 502 that provide information related to events and measurements of system status to the skill imitation system 508. The treatment system 504 may further be designed to monitor the patient's condition to generate a medical record 506 and automatically implement and adjust treatment as needed.

[0050] The skill mimicking system 508 learns the skills applied by the medical professional 502 in response to a patient's clinical condition based on information drawn from at least the medical professional 502, the treatment system 504, and the medical record 506. For example, the medical record 506 may include the patient's past healthcare conditions (e.g., biometric information and symptom descriptions) and actions taken by the medical professional 502 in response to those conditions.

[0051] The different elements of the healthcare facility 500 can communicate with each other over the network 510, for example, using any suitable wired or wireless communication protocol and medium. Thus, the skill mimicking system 508 can access the remotely stored medical records 506, communicate with the treatment system 504, and receive instructions and send reports to the medical professional 502. In particular, the skill mimicking system 508 can automatically trigger treatment changes for the patient in response to new information obtained from the medical records 506 by sending instructions to the treatment system 504.

[0052] In some cases, the skill mimicry system 508 may generate specific treatment plans for the patient, such as a prescription plan including medications to aid in the patient's treatment, a meal plan to address the patient's dietary needs, a rehabilitation plan to provide physical therapy and other activities necessary for the patient's recovery, or a discharge plan indicating whether the patient may return home, should remain, or should be transferred to another health care facility. Thus, the output of the skill mimicry system 508 may include one or a combination of the above automated treatment and planning outputs. In some cases, the treatment plan may be used by a medical professional to assist in decision-making for patient management.

[0053] In some cases, the healthcare facility 500 represents a single client 102, whose records are used in-house for local training, and whose privacy-sanitized information is passed to a remote server 108 for integrated learning. In other cases, the healthcare facility 500 includes multiple clients 102, with personal information siloed within such individual systems, and the server 108 also operated by the healthcare facility 500.

[0054] Referring now to FIG. 6, a patient 602 is shown in the context of a health care system. For example, the patient 602 may be undergoing a hemodialysis (also simply referred to as "dialysis") session. During dialysis, a dialysis machine 604 automatically draws the patient's blood, processes and purifies the blood, and reintroduces the purified blood into the patient's body. Dialysis may take four hours and may occur every three days, although other durations and periods are contemplated. While dialysis is specifically intended, it should be understood that any suitable medical treatment or monitoring may be used instead.

[0055] Before, during, and after a dialysis session, the patient 602 may experience treatment-related health events. Such health events, which may be dangerous to the patient 602, can be predicted based on knowledge of previous health events and the patient's current health metrics. The recommendation 608 can further include information related to the type of predicted event and measurements of the patient's condition. It is specifically contemplated that the recommendation be made before the dialysis session begins so that treatment can be adjusted.

[0056] Recommendations can be made based on various input information. Some of the information includes a patient's static profile, including information such as age, gender, when dialysis was initiated, and previous health events. The information also includes dialysis measurement records, which may be taken for each dialysis session, as well as dynamic data such as blood pressure, weight, venous pressure, blood test measurements, and cardiothoracic ratio (CTR). Blood test measurements are taken periodically, such as twice a month, and can measure factors such as albumin, glucose, and platelet count. CTR can also be measured periodically, such as monthly. Dynamic information can also be recorded during a dialysis session, using sensors on the dialysis machine 604, for example. The dynamic information can be modeled as a time series over each frequency.

[0057] Additionally, the system itself may be monitored within a healthcare environment. For example, the operating parameters of a dialysis machine 604 or other system within a hospital or other healthcare facility are often monitored to predict events, as described below, along with the history of past events in the system.

[0058] During treatment, the condition of the patient 602 may be continuously monitored, such as by tracking the patient's heart rate and other vital signs. If the patient's vital signs indicate an impending or ongoing adverse health event, treatment may be modified accordingly. For example, the treatment system may automatically administer medication or stop treatment in response to a negative health event.

[0059] 7, a method for training a local skill prediction model is shown. Following the pseudocode of FIG. 3, block 702 begins by computing a set of clients C, a local dataset D, and c , initialize a global model including the number of prototypes K, the number of global epochs T, the number of local epochs E, and the learning rate η. Global Interpretable Skill Learning Model

number

number

number

number

[0060] Block 704 distributes the global weights and client-specific prototypes from the server 108 to the clients 102. This information may be transmitted by any suitable means. Block 706 then distributes the updated local parameters, as shown in FIG.

number

[0061] At the server 108, the local weights are aggregated to form an updated global set of weights at block 708. This may be done by averaging the updated local weights received from the clients. Block 710 performs clustering and selects centroids to update the prototypes, as described above.

[0062] If block 712 determines that the global epoch number T has not yet been reached, processing returns to block 704, where updated global weights and client-specific prototypes are distributed to clients 102. Otherwise, the final global weights and client-specific prototypes are distributed and processing ends.

[0063] 8, an exemplary computing device 800 is shown, in accordance with an embodiment of the present invention. The computing device 800 is configured to perform skill imitation.

[0064] Computing device 800 may be embodied as any type of computational or computing device capable of performing the functions described herein, including, but not limited to, a computer, a server, a rack-based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and / or a user electronic device. Additionally or alternatively, computing device 800 may be embodied as one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically separated computing device.

[0065] 8, computing device 800 illustratively includes a processor 810, an I / O subsystem 820, a memory 830, a data storage device 840, and a communications subsystem 850, and / or other components and devices typically found in a server or similar computing device. In other embodiments, computing device 800 may include other or additional components (e.g., various input / output devices) such as those typically found in a server computer. Furthermore, in some embodiments, one or more of the illustrative components may be incorporated into or otherwise form part of another component. For example, memory 830, or portions thereof, may be incorporated into processor 810 in some embodiments.

[0066] The processor 810 may be embodied as any type of processor capable of performing the functions described herein, including a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.

[0067] Memory 830 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, memory 830 may store various data and software used during operation of computing device 800, such as an operating system, applications, programs, libraries, and drivers. Memory 830 is communicatively coupled to processor 810 via I / O subsystem 820 and may be embodied as circuits and / or components for facilitating input / output operations with processor 810, memory 830, and other components of computing device 800. For example, I / O subsystem 820 may be embodied as or otherwise include a memory controller hub, an input / output control hub, a platform controller hub, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.), and / or other components and subsystems for facilitating input / output operations. In some embodiments, I / O subsystem 820 may form part of a system-on-chip (SOC) and be integrated with processor 810, memory 830, and other components of computing device 800 on a single integrated circuit chip.

[0068] Data storage 840 may be embodied as any type of device or apparatus configured for short-term or long-term storage of data, such as, for example, a memory device and circuit, a memory card, a hard disk drive, a solid-state drive, or other data storage device. Data storage 840 may store program code 840A for model training, program code 840B for predicting events, and / or program code 840C for executing corrective actions in response to predicted events. Communications subsystem 850 of computing device 800 may be embodied as any network interface controller or other communications circuitry, device, or collection thereof capable of enabling communications between computing device 800 and other remote devices over a network. Communications subsystem 850 may be configured to achieve such communications using any one or more communications technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand, Bluetooth, Wi-Fi, WiMAX, etc.).

[0069] As shown, computing device 800 may also include one or more peripheral devices 860. Peripheral devices 860 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 860 may include a display, a touch screen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0070] Of course, computing device 800 may include other elements (not shown) or omit certain elements, as would readily occur to one skilled in the art. For example, various other sensors, input devices, and / or output devices may be included in computing device 800, depending on the particular implementation of the same, as would be readily understood by one skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memory, etc. may be utilized in a variety of configurations. These and other variations of processing system 800 will be readily contemplated by one skilled in the art in view of the teachings of the present invention provided herein.

[0071] 9 and 10, exemplary neural network architectures are shown that can be used to implement portions of the present model, such as the imitation learning layer 204. Neural networks are generalized systems whose functionality and accuracy improve with exposure to additional empirical data. Neural networks learn by exposure to empirical data. During training, the neural network stores and adjusts weights that are applied to input empirical data. By applying the adjusted weights to the data, the neural network can identify that data belongs to a specific predefined class from a set of classes and output the probability that the input data belongs to each class.

[0072] Empirical data (also called training data) from a set of examples is formatted as a string of values ​​and fed to the input of a neural network. Each example is associated with a known outcome or output. Each column is represented as an (x,y) pair, where x represents the input data and y represents the known output. The input data can be of various data types and contain multiple different values. The network can have one input node for each value that makes up the example input data, and a separate weight can be applied to each input value. The input data can be formatted, for example, as a vector, array, or string, depending on the architecture of the neural network being built and trained.

[0073] Neural networks "learn" by comparing neural network outputs generated from input data with known values ​​from examples and adjusting stored weights to minimize the difference between the output value and the known value. Adjustments can be made to the stored weights through backpropagation, where the weight's influence on the output value is determined by calculating a mathematical gradient and adjusting the weight in a way that shifts the output toward the minimum difference. This optimization, called gradient descent, is a non-limiting example of how training occurs. A subset of examples with known values ​​that were not used in training can be used to test and validate the accuracy of the neural network.

[0074] During operation, a trained neural network can be used on new data not previously used for training or validation through generalization. The adjusted neural network weights can be applied to the new data, and the weights estimate the function developed from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0075] In a layered neural network, nodes are arranged in layers. An exemplary simple neural network has an input layer 920 of source nodes 922 and a single computational layer 930 with one or more computational nodes 932 that also function as output nodes, with a single computational node 932 for each possible category into which an input example may be classified. The input layer 920 may have a number of source nodes 922 equal to the number of data values ​​912 in the input data 910. The data values ​​912 in the input data 910 may be represented as a column vector. Each computational node 932 in the computational layer 930 generates a weighted linear combination of values ​​from the input data 910 provided to the input node 920 and applies a differentiable nonlinear activation function to the sum. An exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

[0076] A deep neural network, such as a multilayer perceptron, may have an input layer 920 of source nodes 922, one or more computational layers 930 with one or more computational nodes 932, and an output layer 940 with one output node 942 for each possible category into which an input example may fall. The input layer 920 may have a number of source nodes 922 equal to the number of data values ​​912 in the input data 910. The computational nodes 932 in the computational layer 930 are also referred to as hidden layers because they are located between the source nodes 922 and the output nodes 942 and are not directly observed. Each node 932, 942 in the computational layer generates a weighted linear combination of values ​​from the values ​​output by the nodes in the previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the values ​​from each previous node may be, for example, w1, w2, ... w n-1 ,w n The output layer provides the network's overall response to the input data. Deep neural networks can be fully connected, where each node in a computational layer is connected to every node in the previous layer, or the connections between layers can have other configurations. If links between nodes are missing, the network is said to be partially connected.

[0077] Training a deep neural network involves two phases: a forward phase, in which the weights of each node are fixed and inputs are propagated through the network, and a backward phase, in which error values ​​are propagated back through the network and the weight values ​​are updated.

[0078] Computational nodes 932 in one or more computational (hidden) layers 930 perform nonlinear transformations on the input data 912 to generate a feature space, where classes or categories may be more easily separated in the feature space than in the original data space.

[0079] The embodiments described herein may be entirely hardware, entirely software, or contain both hardware and software elements. In a preferred embodiment, the invention is implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0080] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. A computer-usable or computer-readable medium may include any apparatus that stores, communicates, propagates, or transports a program for use by or in connection with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device) or propagation medium. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0081] Each computer program can be tangibly stored on a machine-readable storage medium or device (e.g., program memory or magnetic disk) readable by a general-purpose or special-purpose programmable computer to configure and control the operation of the computer when the storage medium or device is read by the computer to perform the procedures described herein. The system of the present invention can also be considered to be embodied in a computer-readable storage medium configured with a computer program, where the configured storage medium causes the computer to operate in a particular, predetermined manner to perform the functions described herein.

[0082] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements via a system bus. The memory elements may include local memory employed during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0083] Network adapters may also be coupled to the system to enable the data processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0084] As employed herein, the terms “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.) that may be on-board or off-board or dedicated for use by the hardware processor subsystem.

[0085] In some embodiments, the hardware processor subsystem may include and execute one or more software elements, which may include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0086] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0087] These and other variations of the hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

[0088] References herein to "one embodiment" or "one embodiment" of the present invention, as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in one embodiment," as well as any other variations thereof, in various places throughout this specification do not necessarily all refer to the same embodiment. However, it should be understood that features of one or more embodiments may be combined given the teachings of the present invention provided herein.

[0089] For example, in the case of "A / B," the use of any of the following " / ," "and / or," "at least one," such as "A and / or B" or "at least one of A and B" will be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), the selection of only the first and third listed alternatives (A and C), the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as there are listed items.

[0090] The foregoing is understood in all respects to be illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description, but from the claims which are interpreted in accordance with the full breadth permitted by the patent laws. It will be understood that the embodiments shown and described herein are merely exemplary of the invention, and that those skilled in the art could make various modifications without departing from the scope and spirit of the invention. Various other feature combinations could be implemented by those skilled in the art without departing from the scope and spirit of the invention. Having thus described aspects of the invention with the detail and particularity required by the patent laws, what is desired to be claimed and protected by Letters Patent is set forth in the appended claims.

Claims

1. 1. A computer-implemented method for training a healthcare treatment machine learning model, comprising: aggregating 708 local weights from multiple clients to update a global set of weights for the imitation-based skill learning model; clustering 710 a set of local prototype vectors from the plurality of clients to generate a plurality of clusters; selecting (710) representative vectors of the plurality of clusters as a set of global prototypes; determining (710) client-specific prototype vectors for the plurality of clients based on the representative vectors; and distributing (714) the updated set of global weights and the client-specific prototype vectors to the plurality of clients.

2. 10. The method of claim 1, The method, wherein the global set of weights includes weights of convolutional layers and weights of imitation learning layers.

3. 3. The method of claim 2, The imitation learning layer implements a behavior selection policy based on behavior cloning.

4. 10. The method of claim 1, The method, wherein selecting the representative vectors includes determining a centroid for each of the plurality of clusters.

5. 10. The method of claim 1, The method further comprising, at the plurality of clients, learning the local weights and the local prototype vectors based on initial global weights and initial prototypes.

6. 6. The method of claim 5, The method, wherein the training comprises minimizing an objective function including an imitation loss and multiple regularization losses.

7. 7. The method of claim 6, The method, wherein the plurality of regularization losses include a loss that regularizes segment representations from an imitation-based skill learning model to neighbor the closest prototypes as possible, a loss that inversely regularizes prototype vectors to resemble the segment representations as closely as possible, and a loss that enforces a diverse structure of learnable parameterized prototype vectors to avoid redundancy and improve the generalizability of the resulting prototypes.

8. 10. The method of claim 1, The local prototype vectors correspond to actionable therapeutic actions in a medical context.

9. 9. The method of claim 8, Measuring patient status information; selecting a therapeutic action based on the skills predicted by an imitation-based skill learning model based on the measured state information; and informing a medical professional of said treatment action to assist said medical professional in making patient management decisions.

10. 10. The method of claim 9, The method, wherein the therapeutic action includes instructions to a therapeutic system to automatically administer the therapy to the patient.

11. 1. A system for training a healthcare treatment machine learning model, comprising: a hardware processor (810); and a memory (840) for storing a computer program, the computer program, when executed by the hardware processor, causing the hardware processor to: aggregating 708 local weights from multiple clients to update a global set of weights for the imitation-based skill learning model; clustering 710 a set of local prototype vectors from the plurality of clients to generate a plurality of clusters; selecting (710) representative vectors of the plurality of clusters as a set of global prototypes; determining (710) client-specific prototype vectors for the plurality of clients based on the representative vectors; and distributing (714) the updated set of global weights and the client-specific prototype vectors to the plurality of clients.

12. 12. The system of claim 11, The system, wherein the global set of weights includes weights of convolutional layers and weights of imitation learning layers.

13. 13. The system of claim 12, The imitation learning layer implements a behavior selection policy based on behavior cloning.

14. 12. The system of claim 11, The computer program further causes the hardware processor to execute the step of selecting the representative vector by determining a centroid for each of the plurality of clusters.

15. 12. The system of claim 11, The computer program further causes the hardware processor to trigger, at the plurality of clients, learning of the local weights and the local prototype vectors based on initial global weights and initial prototypes.

16. 16. The system of claim 15, The system in which the training involves minimizing an objective function that includes an imitation loss and multiple regularization losses.

17. 17. The system of claim 16, The system, wherein the plurality of regularization losses include a loss that regularizes segment representations from an imitation-based skill learning model to neighbor the closest prototypes as possible, a loss that inversely regularizes prototype vectors to resemble the segment representations as closely as possible, and a loss that enforces a diverse structure of learnable parameterized prototype vectors to avoid redundancy and improve the generalizability of the resulting prototypes.

18. 12. The system of claim 11, The local prototype vectors correspond to therapeutic actions that can be performed in a medical context.

19. 20. The system of claim 18, The computer program further causes the hardware processor to: Measuring patient status information; selecting a therapeutic action based on the skills predicted by an imitation-based skill learning model based on the measured state information; and informing a medical professional of said treatment action to assist said medical professional in making patient management decisions.

20. 20. The system of claim 19, The system wherein the therapeutic action includes instructions to a therapeutic system to automatically administer the therapy to the patient.

Images