Machine learning model validation and certification

Abstract

The present disclosure relates to methods and apparatuses for verifying and authenticating usage of machine learning models. For example, various techniques are described herein for limiting machine learning model attacks and / or exploiting models for malicious use, as well as for detecting when such attacks / exploits occur. Additionally, various embodiments described herein facilitate protection of sensitive and / or valuable data, for example by ensuring that only permitted usage is allowed. Furthermore, techniques are also described for version tracking, usage tracking, permission tracking, and machine learning model evolution.

Description

TECHNICAL FIELD

[0001] Various embodiments described herein generally relate to security and artificial intelligence. More specifically, but not exclusively, various methods and apparatus disclosed herein relate to verifying and authenticating usage of machine learning models. BACKGROUND

[0002] Artificial intelligence (“AI”) components, such as machine learning models (also referred to as “statistical models”), can be used to apply complex computations to complex inputs to generate various types of outputs. However, for many complex and / or autonomous tasks that employ machine learning models, security vulnerabilities pose a serious risk to data privacy and / or security. For example, machine learning models are becoming increasingly prevalent in the healthcare field, and they are widely used for various applications, e.g., clinical decision support (“CDS”), controlling medical devices, etc. Regulations such as the Health Insurance Portability and Accountability Act (“HIPAA”) place a high value on patient privacy. Moreover, security vulnerabilities in input data, changes in the integrity of mathematical operations performed based on parameters of the machine learning model and / or the model itself, can all lead to erroneous outputs, which can be especially disruptive in the medical field.

[0003] Determining whether a machine learning model is compromised is challenging. Attackers have a variety of methods to compromise machine learning models, causing them to operate in unexpected and / or malicious ways, e.g., modifying weights, exploiting vulnerabilities in inputs (which can lead to consistent malicious outputs), or bypassing algorithms through custom code. These malicious attacks can be implemented by modifying the machine learning model in non-volatile memory (e.g., “on disk”), or in real-time during execution using volatile memory (e.g., random access memory or “RAM”). SUMMARY

[0004] The present disclosure relates to methods and apparatus for verifying and authenticating usage of machine learning models. For example, various techniques are described herein for limiting machine learning model attacks and / or exploiting vulnerabilities of the model for malicious use, and for detecting when such attacks / exploits occur. Additionally, various embodiments described herein facilitate protection of sensitive and / or valuable data, e.g., by ensuring that only permitted usage is allowed. Moreover, techniques are described herein for version tracking, usage tracking, permission tracking, and machine learning model evolution.

[0005] Various examples related to healthcare are described herein. In particular, the example techniques depicted and / or illustrated by the accompanying figures are generally described in a healthcare context. However, this is not meant to be limiting. The techniques described herein are widely applicable outside of the healthcare context. For example, the techniques described herein can be used at security checkpoints where image data is used in conjunction with training a machine learning model (e.g., a convolutional neural network) to determine a threat / risk posed by an individual. They can also be applied to other situations where machine learning is generally used, e.g., financial analysis, risk analysis, etc.

[0006] The examples described herein can also be used to vary the level of detail and / or granularity provided to an end user. For example, the techniques described herein can cause a (e.g., last) output layer of a trained machine learning configured to classify to be blocked based on upstream input. However, the upstream input can still be useful for various purposes.

[0007] As an example, a last layer of a neural network can be trained to receive segmented image data (e.g., image data indicating regions of interest for potential cancer concerns) and classify one or more of the regions of interest as malignant or benign. However, the segmented image data provided by a previous layer can be useful even if not used for classification, e.g., for annotating regions of interest for potential cancer concerns in a digital image. Thus, if a nurse or other medically qualified or licensed medical personnel provides their credentials to the model, the model blocks the data from propagating forward through the last layer and only provides the segmented image data, which can then be used, e.g., to generate a heat map showing regions of general concern without providing a specific classification of whether the region is malignant or benign. However, if a radiologist provides his or her credentials, the last layer can be activated (or unlocked) and the radiologist can receive a classification for each region of interest.

[0008] Generally, in one aspect, a method can be implemented using one or more processors and can include providing an encryption key associated with a particular entity, where the particular entity has access to a machine learning model trained to generate one or more outputs based on data applied across a plurality of inputs; encrypting one or more parameters of the trained machine learning model using the encryption key; encrypting input data to be applied as input across the trained machine learning model; applying the encrypted input data as input across the encrypted trained machine learning model to generate an encrypted output; decrypting the encrypted output using a decryption key to generate a decrypted output; analyzing the decrypted output using the decryption key to determine that one or more of the parameters of the trained machine learning model have been compromised; and causing one or more computing devices to provide an output indicating that the one or more parameters of the trained machine learning model have been compromised.

[0009] In various embodiments, the encryption of the one or more parameters is performed using homomorphic encryption. In various versions, the encryption of the input data is performed using homomorphic encryption.

[0010] In various embodiments, the analyzing includes determining whether the decrypted output conforms to an expected output structure. In various embodiments, the determining includes determining whether one or more output values of the decrypted output fall within a particular range. In various embodiments, the determining includes determining whether a plurality of output values of the decrypted output collectively add up to a predetermined value.

[0011] In various embodiments, the particular entity accesses the trained machine learning model using a software application executing on a computing device operated by the particular entity. In various embodiments, the method further includes configuring the software application to encrypt the input data in response to one or more events. In various versions, the one or more events include a user associated with the particular entity operating the software application to perform a task including applying the input data across the trained machine learning model.

[0012] In various embodiments, the trained machine learning model includes a generative adversarial network, a convolutional neural network, or a recurrent neural network. In various embodiments, the digital key is unique to the particular entity and the trained machine learning model.

[0013] Additionally, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause performance of any of the methods described above. Some embodiments also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the methods described above.

[0014] In another aspect, at least one non-transitory computer-readable medium can be provided, the non-transitory computer-readable medium comprising data forming a trained machine learning model, wherein the trained machine learning model comprises: an input layer that accepts a plurality of input data points; at least one hidden layer to perform computations on the input data points; an output layer to provide at least one output value based on the computations; and at least one gate node to selectively allow or prevent, conditioned on at least one of the input data points, data received from an upstream layer of the trained machine learning model from passing unaltered to a downstream layer of the trained machine learning model.

[0015] "Homomorphic encryption" is a form of encryption that allows computations on encrypted data. Homomorphic encryption generates encrypted results that, when decrypted, match the results of operations performed on the original, unencrypted data, as if the operations were performed on the original, unencrypted data.

[0016] It should be appreciated that all combinations of the foregoing concepts (and additional concepts) described below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, with reference to claims that follow, it is contemplated that claims can be drafted to include additional steps or elements, in various combinations and / or permutations, even if such additional steps or elements are not expressly disclosed in the above paragraphs. It is intended that additional claims pertaining to additional features or aspects of the application can be drafted in such a manner and can be added to the present claims after further study or practice of the application. It should be understood that the use of "or" in the claims is meant to encompass both "and" and "or" unless specifically stated otherwise. It should also be understood that the use of "comprise" or "comprises" or "comprising" or "comprises" is meant to encompass the presence of stated features, integers, steps or components, but not the exclusion of others. It should also be understood that the use of "comprise" or "comprises" or "comprising" or "comprises" is meant to encompass the presence of stated features, integers, steps or components, but not the exclusion of others. BRIEF DESCRIPTION OF DRAWINGS

[0017] In the drawings, like reference numerals refer to like parts throughout the various views. Also, the drawings can not be to scale, with emphasis generally being placed upon illustrating the principles of the embodiments described herein.

[0018] Figure 1 FIGURE 1 illustrates an example environment in which selected aspects of the present disclosure can be implemented in accordance with various embodiments.

[0019] Figure 2 An example of how integrity of a machine learning model can be verified in accordance with various embodiments is depicted.

[0020] Figure 3 One example of how digital keys used to verify the integrity of a machine learning model according to various embodiments can also be used to authenticate use of the machine learning model is depicted.

[0021] Figure 4 Another example of how the integrity of a machine learning model can be verified and how the model can be protected from attack according to various embodiments is depicted.

[0022] Figure 5 An example of how the integrity of a machine learning model can be protected and at the same time authentication of access to the model can also be facilitated according to various embodiments is depicted.

[0023] Figure 6 Another example of how the integrity of a machine learning model can be protected and at the same time authentication of access to the model can also be facilitated according to various embodiments is depicted.

[0024] Figure 7 , Figure 8 and Figure 9 An example method for practicing selected aspects of the present disclosure according to various embodiments is depicted.

[0025] Figure 10A and Figure 10B One example of how embodiments of the present disclosure can facilitate the use of a machine learning model according to various embodiments is shown. Figure 5

[0026] Figure 11A and Figure 11B One example of how embodiments of the present disclosure can facilitate the use of a machine learning model according to various embodiments is shown. Figure 6

[0027] Figure 12A and Figure 12B A schematic overview in a generalized manner of how a gating layer and / or individual gate nodes can be inserted into a trained machine learning model is shown.

[0028] Figure 13A , Figure 13B and Figure 13C Examples of how the techniques can be applied to various types of machine learning models other than feedforward neural networks are depicted.

[0029] Figure 14A and Figure 14B Examples of how a gating layer and / or gate nodes can be implanted into an ensemble type model (e.g., a processing pipeline that includes multiple machine learning models) are shown.

[0030] Figure 15 An example computing system architecture is depicted. DETAILED DESCRIPTION ​​

[0031] Artificial intelligence (“AI”) components, such as machine learning models (also referred to as “statistical models”), can be subject to various security risks. Security vulnerabilities in input data, as well as changes in the integrity of mathematical operations performed based on the machine learning model and / or parameters of the model itself, can result in erroneous outputs. Machine learning models are often developed to perform complex and / or autonomous tasks for which security vulnerabilities pose a serious risk to data privacy and security. For example, machine learning models are becoming increasingly prevalent in the healthcare industry, where regulations such as the Health Insurance Portability and Accountability Act (“HIPAA”) place a high value on patient privacy. Thus, in addition to the risk that models can be maliciously manipulated, access to the models can also require strict security. In view of the foregoing, various embodiments and implementations of the present disclosure are directed to verifying and authenticating use of machine learning models.

[0032] Reference is made to Figure 1 FIG. 1 schematically depicts an example environment in which selected aspects of the present disclosure can be implemented, in accordance with various embodiments. An AI provider system 100 can include one or more computing devices and / or servers (e.g., blade servers) operated by an entity, such as a business, a government entity, an individual, a non-profit organization, etc., to provide AI services to others. In various embodiments, the AI provider system 100 can be in communication with one or more remote computing systems 102 1-N communicatively coupled. In some implementations, the AI provider system can be implemented across multiple computing devices, forming what can be referred to as a “cloud” infrastructure or “cloud.”

[0033] In various implementations, the AI provider system 100 can provide access to one or more machine learning (“ML”) models to one or more individuals (“users”). The users can use the ML models for various purposes, such as making predictions, classification, diagnosis (e.g., clinical decision support or “CDS”), operating instruments (e.g., changing parameters of medical instruments), performing object and / or entity recognition, handwriting recognition, etc. In some embodiments, depending on the credentials provided by and / or on behalf of the individuals and / or entities, the AI provider system 100 can provide various levels of access to the ML models to different individuals and / or entities.

[0034] In Figure 1In particular embodiments, the first remote computing system 1021 can take the form of a healthcare or hospital computing system / network, which includes one or more computing devices, servers, instruments, smart appliances, networked medical devices, and the like. The first remote computing system 1021 can also include various healthcare-related computing subsystems (not depicted), e.g., a hospital information system (“HIS”), an electronic health record (“EHR”) database, and the like. In various embodiments, the plurality of client devices 112 1-P operated by a plurality of users 114 1-M may be connected to the computing system 1021, e.g., via one or more networks (not depicted, possibly including the Internet). In addition to or instead of serving a system of multiple computing devices, the AI provider system 100 can serve individual computing devices, e.g., 1022 and 102 N .

[0035] In various embodiments, one or more ML models can be stored by the AI provider system 100 in an ML model database 104. These ML models can take various forms, e.g., deep learning neural networks, recurrent neural networks (“RNNs”), convolutional neural networks (“CNNs”), support vector machines, decision trees, reinforcement learning models, generative adversarial networks (“GANs”), and the like. The AI provider system 100 can make these ML models available to the remote computing systems 102 in various ways. In some embodiments, the remote computing systems 102 can download copies of the ML models from the database 104 and store them locally, e.g., in their own databases (e.g., 116). Additionally or alternatively, in some embodiments, the AI provider service 100 can keep the ML models exclusively in the database 104, and can apply data received from the remote computing systems 102 to the ML models on demand. In some such embodiments, the AI provider system 100 and / or the remote computing systems 102 can encrypt data exchanged (e.g., inputs to the ML models, outputs generated from the ML models, and the like).

[0036] In some embodiments, the AI provider system 100 can include a licensing engine 106, an application engine 107, and / or an integrity engine 108. These engines can be implemented using any combination of software and hardware, and can be implemented on a single computing device or across multiple computing devices (e.g., as “cloud-based components”). In other embodiments, one or more of the engines 106-108 can be combined and / or omitted, or implemented at least partially on another computing system (e.g., on a remote computing system 1021).

[0037] The permissions engine 106 can be configured to check or provide on behalf of individual users and / or entities credentials provided by the individual users and / or entities to determine which ML models the individual users / entities should have access to, what level of permissions the individual users / entities should have to access each ML model, the time and / or different number of uses each individual user / entity should have access to a particular ML model, etc. In other words, the permissions engine 106 can be configured to authenticate users and / or entities to use various ML models, whether the ML models are stored locally in the database 104 by the AI provider service 100 or remotely in one or more remote computing systems 102 (e.g., in the database 116).

[0038] The application engine 107 can be configured to apply input across one or more ML models stored in the database 104 to generate output. As previously mentioned, in some embodiments, the AI provider service 100 can retain ML models locally in the database 104 and can receive, through the application engine 107, input to be applied by the application engine 107 across those ML models from a remote computing system 102. For example, in some implementations, a remote computing system 102 can provide input data (e.g., a digital image, a waveform, text, etc.) to the AI provider service 100. The application engine 107 can apply the input data across one or more ML application models (e.g., for which the permissions, the entity / user operating the remote computing system 102 was selected, etc.) to generate output. Data indicative of the output and / or the output itself can be returned to the remote computing system 102. However, in other embodiments, one or more remote computing systems 102 (e.g., the entity computing system 1021) can download ML models it is going to use and store them locally, e.g., in the database 116. In some such embodiments, the remote computing system 102 can have its own application engine (not depicted) that applies data across the locally stored ML models.

[0039] The integrity engine 108 can be configured to check various aspects of ML models stored locally to the Al provider system 100 (e.g., stored in the database 104) and / or stored remotely (e.g., stored in the database 116) to determine whether and / or how those ML models can have been compromised. For example, a malicious party can gain access to an ML model stored in the database 116 and can alter one or more aspects of the ML model, such as altering or deleting one or more parameters or weights in various layers. Alternatively, a licensed entity can attempt to make such changes to a model that it stores locally without permission to do so. In either case, the integrity engine 108 can be configured to apply the various techniques described herein, or cause those techniques to be applied to one or more remote computing systems 102, in order to verify the integrity of the ML model and / or take appropriate remedial action when it determines that the ML model has been compromised. In some embodiments, the integrity engine 108 can verify the integrity of an ML model by applying a digital key as input to generate an output across the ML model, which is then verified by the integrity engine 108 as described herein.

[0040] In some implementations, ML models are stored remotely from the Al provider system 100, such as in the database 116, and one or more client devices 112 1-M And / or the entity computing system 1021 can host a software application that can be operated by an end user 114 to utilize the ML model. In some implementations, the software application can be provided (e.g., in an app store) and maintained by the Al provider system 100. In some such embodiments, the Al provider system 100 (e.g., through the integrity engine 108) can retain the right to cause the remotely executing software application to periodically perform the techniques described herein (e.g., the techniques employed by the integrity engine 108).

[0041] For example, in some embodiments, when a particular user 114 (e.g., a nurse) operates a client device 112 to interact with a software application, the nurse can log into the client device 112 with one or more credentials. These credentials can authenticate the nurse to apply data with the software application across one or more ML models. The nurse can not explicitly be aware that he or she will access an ML model, but rather the nurse can simply interact with a graphical user interface (“GUI”) or other input component to view some patient data generated by a CDS algorithm in response to various other data. The CDS algorithm itself can apply appropriate input data across an appropriate ML model to generate an output. The output itself or some other data generated from the output (e.g., a classification, a prediction, etc.) can then be presented to the nurse via the software application. In some embodiments, the nurse’s credentials can limit the nurse’s access to certain models while denying the nurse access to other models (if the nurse attempts to use a functionality that relies on a restricted model, an audible or visual output can be provided, such as “You do not have permission to perform this action”). Additionally or alternatively, in some embodiments, the nurse’s credentials can limit what is ultimately output from a model, which will be described in greater detail below.

[0042] In some embodiments, prior to applying clinical data across an ML model, the nurse’s credentials or some other piece of data retrieved based on the nurse’s credentials (e.g., a digital key) can be retrieved and applied as input across at least a portion of the ML model (e.g., immediately prior to the “actual” clinical data that is to be applied). The ML model can generate, based on the applied data, what will be referred to herein as one or more “validation outputs.” These validation outputs can be compared to known validation outputs, such as by the integrity engine 108 or at the entity computing system 1021 (or even locally at the client device 114 being operated by the nurse).

[0043] For example, these known validation outputs can have been generated based on prior application of the piece of data as input across at least a portion of the ML model. If the validation outputs match or otherwise correspond to the known validation outputs, the ML model can be deemed “uncompromised.” The clinical data under consideration by the nurse can then be applied as input across the ML model to generate the output previously described. In various implementations, the nurse can not even be aware that the integrity of the ML model has been tested or that the ML model has been used.

[0044] The integrity of ML models can be verified at other points in time in addition to or instead of immediately prior to application of clinical data across the ML model. For example, in some implementations, the integrity engine 108 can periodically (e.g., once an hour, once a day, overnight, on weekends, etc.) request that a remote execution software application that provides users with access to ML models perform the techniques described herein. In some embodiments, the remote execution software itself can be configured to periodically verify the integrity of the ML models stored locally thereto. Additionally or alternatively, in some implementations, the integrity engine 108 can employ the techniques described herein to verify the integrity of the ML model(s) stored in the database 104, either alone or in combination with the application engine 107 (periodically, on demand, randomly, etc.). In some embodiments, the digital keys or other pieces of data used to verify the integrity of the ML models can be applied across the model in conjunction with clinical inputs. Examples of which will be described herein.

[0045] For example, the integrity engine 108 can employ various techniques to verify the integrity of ML models. For example, referring now to Figure 2 , depicts an example MF model in the form of a feedforward neural network (“FFNN”) 220. As previously described, the FFNN 220 can be stored locally in the database 104 at the AI provider system 100 or remotely on one or more remote computing systems 102 (e.g., in the database 116). The FFNN 220 includes multiple layers, including an input layer 221, two hidden layers 222 1-2 , two sets of weights 223 1-2 between the layers, and an output layer 224. The FFNN 220 is provided for illustrative purposes only and is relatively small. It should be understood that the techniques described herein are applicable to networks having other arrangements, including networks having more or fewer layers, more or fewer nodes, etc. For example, many neural networks have many more inputs, many more layers, and can include other components, e.g., memory nodes (e.g., long short-term memory nodes, gated recurrent units, etc.). Moreover, the techniques herein are applicable to any machine learning model, including decision trees (e.g., random forests), support vector machines, and logistic regression, etc. The techniques apply to supervised and unsupervised models as well as regression and classification models.

[0046] In various embodiments, to verify the integrity of FFNN 220, a piece of data such as a unique identifier or digital key 226 can be applied as input across FFNN 220 to generate an output. Digital key 226 can take various forms. In some embodiments, digital key 226 can take the form of a secure private key generated using, for example, Data Encryption Standard ("DES"), Digital Signature Algorithm ("DSA"), Triple DES, Advanced Encryption Standard ("AES"), Twofish, Rivest-Shamir-Adleman ("RSA"), and / or Pretty Good Privacy ("PGP"), among others. However, other types of digital keys (or more generally, pieces of data) and / or techniques for generating them can also be employed. In some embodiments, the piece(s) of data applied across FFNN 220 to verify its integrity (e.g., digital key 226) can be unique to FFNN 220 and to one or more users or entities using FFNN 220. As will be briefly discussed, in some embodiments, digital key 226 can also be used as a reference to a (secure) lookup table that provides various information (e.g., metadata) about FFNN 220, such as its version, usage, user permissions, training history, and the like.

[0047] In various implementations, digital key 226 can be applied as input across all or portions of FFNN 220 in various ways. For example, digital key 226 can be applied to input layer 221 as input, as shown in Figure 2 In some such embodiments, the dimensionality of digital key 226 (e.g., how many different characters, numbers, symbols, and / or elements it contains) can match that of input layer 221, although this is not required. Additionally or alternatively, in some embodiments, digital key 226 can be applied at any other layer and / or individual node in the forward direction (right side of Figure 2 ) or even in the backward direction (left side of Figure 2 ). The various portions of FFNN 220 can then be compared to known verification outputs 230 based on the verification output(s) 228 generated by digital key 226, e.g., as shown by arrow 232. If verification output 228 matches or otherwise corresponds to known verification output 230, then FFNN 220 can be deemed un-compromised. Otherwise, FFNN 220 can be deemed compromised, and various remedial measures can be taken, such as alerting AI provider system 100 and / or remote computing system 102 locally, removing access to FFNN 220, and the like.

[0048] In Figure 2In particular embodiments, the digital key 226 is applied to the entire input layer 221. Then, the outputs from the two hidden layers 221 1-2 1-2 and the output data from the output layer 224 are captured and combined to generate a verification output 228. Various proprietary or other algorithms can be used to combine these data in various ways. In other embodiments, other combinations of outputs generated by the FFNN 220 can be captured to generate verification outputs different from those depicted in Figure 2

[0049] The known verification output 230 can be pre-computed in, for example, a secure environment, prior to verifying the FFNN 220. For example, whenever the FFNN 220 is trained or retrained, e.g., by the AI provider system 100, the digital key 226 can be applied across all or a portion(s) of the FFNN 220 to generate an output. The output of various nodes and / or layers can be captured to generate the known verification output 230. This known verification output 230 can then be saved in, for example, non-volatile memory (e.g., saved in the database 104 locally at the AI provider system 100 or saved remotely in the database 116). In some implementations, the known verification output 230 can be encrypted while stored in non-volatile memory, e.g., so that end users cannot access it. Additionally or alternatively, the known verification output 230 can be encrypted using a public key assigned to an entity that operates the remote computing system 102. The same entity can be provided with a digital private key (e.g., 226) that can both be applied across the FFNN 220 for verification purposes and used to decrypt the known verification data 230. Alternatively, a different private key than the digital key 226 can be used to decrypt the known verification data 230.

[0050] As long as the digital key 226 is applied across the same portion(s) of the FFNN 220 when generating the known verification output 230 as when generating the verification output 228 later and as long as the same outputs from the same combination of nodes / layers are captured, the verification output 228 and the known verification output 230 should be comparable (e.g., match). Thus, in some embodiments, if the subsequently generated verification output 228 does not match the previously generated known verification output 230, this can indicate that the FFNN 220 has been compromised.

[0051] The FFNN 220 can take various forms, e.g., a convolutional neural network, a “deep learning” network, a generative adversarial network (“GAN”), etc. While the FFNN 220 is depicted in​​Figure 2 The model is described as a feedforward neural network, but this is not intended to be limiting. In other embodiments, other types of ML models can also be used with the techniques described herein, such as recurrent neural networks, long short-term memory (“LSTM”) networks, ensemble networks, etc.

[0052] Figure 3 An example of a lookup table 334 that can store information, such as information that can be used by a licensed engine 106, to determine information about an ML model stored in database 104 (or a remotely stored ML model, for example, in database 116 or in remote computing systems 1022, 1023). N The information includes ML models, etc., stored in local volatile or non-volatile memory. In various embodiments, the licensing engine 106 may receive a digital key 226 as input. The licensing engine 106 may perform various types of functions to map the digital key 226 to one or more records in the lookup table 334. For example, in some embodiments, the licensing engine 106 may perform various hash functions to map the digital key 226 to one or more records in the lookup table 334.

[0053] In this example, the licensing engine 106 performs a computation (e.g., a hash function) to map the digital key 226 to a first entry in lookup table 334. This first entry specifies various information about an ML model stored, for example, in database 104 (or remotely within AI provider system 100). In this example, the entry has a "description" of "Lung Cancer Risk AI," indicating that it is an ML model trained to receive various clinical parameters associated with a patient (e.g., vital signs, digital images of the lungs, CT scans, magnetic resonance imaging ("MRI") data, demographic data, symptoms, medications, etc.) as input and generate an output indicating lung cancer risk. The entry has a "version" of "1.0.1," a "deployment date" of May 3, 2018, the licensee's name, a license expiry date of "retention date" (in this case, N / A, because the ML model is still in its original form), compatible hardware, and compatible software (e.g., software configured to apply input data across models).

[0054] In various implementations, when a user such as a physician or nurse operates the software to cause application of input data across the model, the user’s digital key (e.g., 226) can be checked against the lookup table 334 in addition to being applied across the FFNN 220. If the user has sufficient privileges, the license is not expired, and other criteria specified by the entry are met, the user can be allowed to apply the input across the ML model. In some implementations, the entry can be updated (logged) to reflect that the ML model has been used, e.g., by a particular user at a particular time. If the license specifies a limited number of applications of the ML model, the current use can apply to the limit.

[0055] In some embodiments, various data associated with use of the ML model can be encrypted at various stages in order to verify the integrity of the model and / or authenticate use of the ML model. For example, in Figure 4 , three stages of application of the ML model are illustratively described: an input stage 438, encryption of the ML model and weights in a non-volatile memory stage 440, and an output stage 442.

[0056] In the input stage 438, input data can be received / obtained / retrieved from various sources 444. These sources can include, but are not limited to, image data 4441 obtained from medical imaging devices such as X-rays, CT scans, MRIs, EKGs, etc., imaging protocol data 4442 (e.g., Digital Imaging and Communications in Medicine or “DICOM,” Picture Archiving and Communication System or “PACS,” etc.), demographic data 4443, and medical history data 4444 (e.g., obtained from an EHR). Other sources of input data are contemplated herein.

[0057] Prior to or during the input stage 438, an encryption key 446 can be provided, e.g., by the AI provider system 100, to one or more remote computing systems 102 (see Figure 1 ). This encryption key 446 can be used by one or more users (e.g., 114 in Figure 1 ) to generate encrypted data 448 from data provided by the sources 444. When the encrypted input data 448 is applied across one or more ML models (e.g., FFNN 420 (which can be similar to or different from 220 in Figure 2 ), various measures can be taken.

[0058] In some embodiments, the decrypted data 448 can be decrypted at block 450 using a unique digital private key 426 (which can be similar to the digital key 226), such that, for example, the decrypted data can be applied as input across an unencrypted version of the FFNN 420 (as shown at 451). However, until this point, the input data can remain in its encrypted form 448. In these embodiments, encrypting the input data until its use (e.g., until it is loaded into volatile memory) provides at least some security against unauthorized parties gaining access to potentially sensitive data. For example, some hackers can be opportunists who, when encountering encrypted input data (i.e., while waiting to be applied across the FFNN 420), can look elsewhere for data to exploit.

[0059] Additionally or alternatively, in some embodiments, in the output stage 442 (to the right of Figure 4 the encrypted input data 448 can remain encrypted and can be used as input across an encrypted version of the FFNN 420'. In various embodiments, the encrypted FFNN 220' (indicated with shading) can be generated (e.g., using the encryption key 446 or other encryption key) to accept only encrypted data; unencrypted input can result in an erroneous output. Various types of encryption can be used, for example, RSA or other forms mentioned earlier. In some embodiments, the encrypted FFNN 420' can be generated using homomorphic encryption, which, as mentioned earlier, is a mechanism that enables mathematical operations such as those associated with the encrypted FFNN 420' to be applied to encrypted data to generate an encrypted output 453. The encrypted FFNN 420' can remain in an encrypted state even when loaded into volatile memory for use. The encrypted output 453 can then be decrypted (e.g., using the digital key 426) to produce a decrypted output 454, which can constitute the actual valid result due to the homomorphic nature of the encryption. Additionally or alternatively, in some embodiments, a garbled circuit can be employed.

[0060] In some embodiments, in the non-volatile memory stage 440 (to the left of Figure 4When the ML model 420 and the weights in the FFNN 420 (and the intermediate) are encrypted, the parameters and weights associated with the FFNN 420 can be encrypted at 456 using the encryption key 446 to generate an encrypted model file 458 and an encrypted weights file 460. Various encryption techniques can be employed, e.g., RSA or other techniques mentioned earlier. Later, e.g., when decrypted or encrypted input data is to be applied across the FFNN 420, the digital key 426 (or another digital key) can be used at block 462 to decrypt the encrypted model file 458 and the encrypted weights file 460. This can occur, for example, when the FFNN 420 is loaded into volatile memory (e.g., of the AI provider system 100 or the remote computing system 102) for use. Thus, when the FFNN 420 is stored in non-volatile memory (e.g., a disk, solid state memory, etc.), it can remain in an encrypted state and thus be protected, at least to some extent, from malicious users. Only when the FFNN 420 is to be used and loaded into volatile memory (e.g., RAM) is it decrypted (e.g., using the digital key 426 at block 462).

[0061] Figure 4 One benefit of the various encryption schemes depicted in FIGS. 5-7 is that it can prevent unauthorized use of the FNN 420 or its encrypted version 420'. As mentioned earlier, in some embodiments, only encrypted input can be applied across the FFNN 420 / 420' - unencrypted data will result in an erroneous output. Thus, a user wishing to use the FFNN 420 / 420' must have access to one or more digital keys (e.g., 426) and / or decryption keys (e.g., 446). Without these keys, an unauthorized user can not be able to use the FFNN 420 / 420'.

[0062] Figure 5 An example of how the integrity of a trained machine learning model can be protected and at the same time authentication of access to the model can be facilitated according to various embodiments is depicted in FIG. 5. In Figure 5 In FIG. 5, the ML model again takes the form of a FFNN 520, but this is not meant to be limiting. In Figure 5 In FIG. 5, the FFNN 520 can only produce meaningful and / or coherent output when appropriate credentials or other pieces of data (e.g., a digital key 526 (which can be similar to 226, 426)) are provided (e.g., as input across at least a portion of the FFNN 520). Otherwise, applying input data across the FFNN 520 can generate an erroneous output, all zeros, an output indicating that the FFNN 520 is being used without authorization, and / or an output indicating whether the FFNN 520 has been compromised in some way.

[0063] exist Figure 5 In, it can be seen from the relationship with Figure 4 Various inputs are obtained from sources 444 that are the same or similar to those depicted. The input scrambler 560 can be configured to scramble the input data obtained from source 444, either alone or in combination with data obtained from digital key 526, such that the data output from input scrambler 560 includes data from digital key 526 that is shuffled or mixed with the input data obtained from source 444. For example, and as indicated by the dashed line below input scrambler 560, the input from source 444 and the data from digital key 526 are distributed among the nodes of input layer 521, such that input layer 521 includes scrambled input. Additionally or alternatively, the input data can be scrambled by modulating the amplitude of the input data, for example, by adding / subtracting a constant value from each individual input stream. In some such embodiments, the scrambled input sequence may only be known to the algorithm manufacturer, and the scrambled input sequence may have been configured when the algorithm was created. If the input scrambling module is provided to the user alone or in combination with the data obtained from digital key 526 that will be used to perform the scrambling operation, the user will only be able to use this algorithm.

[0064] In various embodiments, one or more "gating" layers 562 may be provided, for example, between other layers of the FFNN 520. These gating layers 562 are operable to allow or block data from propagating forward through various parts of the FFNN 520. Figure 5 Two gating layers 5621 and 5622 are depicted, but more or fewer gating layers may be deployed in various embodiments. These gating layers 562 are capable of transitioning between a "locked" state and an "unlocked" state, as described below. For example, in the first gating layer 5621, three gate nodes 561 1-3 It is coupled to three of the six outputs of the scrambled input layer 560. The other nodes of the input layer 521 are not connected to gate node 561, but this is not a limitation. A given gating layer 562 may include more or fewer gate nodes 561. Gate node 561 1-3 The output is coupled to the input of the first hidden layer 5221. Although Figure 5 Two hidden layers, 5221 and 5222, are depicted, but this is not a limitation. More or fewer hidden layers can be provided. In some embodiments, gating layers can be inserted into the model after training and used primarily during inference to control the flow of data through the model.

[0065] The gate node 561 of the gate layer 5621 can be opened or closed via the first key locking routine 5631. 1-3The first key-locking routine 5631, which can take various forms (e.g., software including one or more if / else statements, hash functions, etc.) can be coupled with the input nodes of the input layer 521 including data from the digital key 526. If a suitable and / or authorized digital key 526 is provided, the first key-locking routine 5631 can unlock or open the gate nodes 561 1-3 , e.g., by setting them to one (unlocked). Otherwise, the first key-locking routine 5631 can lock or close the gate nodes 561 1-3 , e.g., by setting them to zero. In some embodiments, the second gating layer 5622 can be provided with its own gate nodes 561 4-6 and a second key-locking routine 5632 that can or can not be activated by the output of the first key-locking routine 5631.

[0066] In some embodiments, a data de-scrambler 566 can also be provided. As shown by the dashed lines below, the data de-scrambler 566 can be configured to de-scramble (or “unshuffle”) the output of the second hidden layer 5222. For example, in Figure 5 , the outputs O2, O0, and O1 are de-scrambled to O0, O1, and O2 in the final output layer 524.

[0067] The gating layer 562 provides a variety of benefits related to, for example, authentication. If inserted as an entire layer as depicted in Figure 5 , the gating layer 562 can be easily inserted into a trained ML model, as demonstrated in Figure 10A -B. Moreover, the gating layer 562 can be customized for individual licensees.

[0068] Figure 10A -B demonstrates how the gating layer 562 can be relatively easily inserted into a trained FFNN 1020. The FFNN 1020 can be similar in many respects to the FFNN 520. Once the gating layer 562 is inserted, the data flow through the model during inference can be controlled primarily using the gating layer 562. In some implementations, when the gating layer 562 is inserted into the FFNN 1020, routines such as Figure 10B , depicted in

[0069] In Figure 10B , k represents a digital key, e.g., 526. h i specifies a particular hidden layer 1022 i . In Figure 10A , there are three sets of weights W 1-3The first set of weights W1 is between the input layer 1021 and the first hidden layer 10221. The second set of weights W2 is between the gating layer 562 and the second hidden layer 10222. The third set of weights W3 is between the second hidden layer 10221 and the output layer 1024. Returning to Figure 10B The activation function at the top, for example, can be a sigmoid function that introduces nonlinearity into the FFNN 1020. As explained in the notes (beginning with the symbol “#”) in Figure 10B If a valid numerical key k is applied, G(k) will equal 1, meaning that the output from the first hidden layer 10221 is propagated unchanged to the second hidden layer 10222, as explained in the notes (beginning with the symbol “#”) in

[0070] Figure 5 and FIG. 10 illustrate how a gating layer can be inserted into a pre-trained feedforward neural network. And as will be described below, Figure 6 and FIG. 11 illustrate how an individual gating node can be inserted into a pre-trained feedforward neural network. However, the techniques for inserting a gating layer or individual gating nodes are not limited to this. For example, the gating layer techniques described herein can be used to wrap any type of machine learning model by inserting a gating layer before or after the input layer and / or output layer of the model. Moreover, the gating layer (or individual gating nodes) can be embedded in a variety of types of machine learning models that include multi-step computations or processes, as will be described in FIGS. 12-14. For example, a gating layer can be inserted at any given level in a decision tree, or between the joint probability and conditional probability in a Naive Bayes computation. Both of these approaches are particularly well-suited for “ensemble” machine learning models, and can be used to limit or direct the flow of information through these types of modeling methods, for example as shown in FIG. 13. This technique is applicable to supervised and unsupervised models, as well as regression and classification models.

[0071] Figure 6 depicts an embodiment similar in many respects to Figure 5 but introduces additional advantages. In Figure 6In this example, the trained ML model again adopts the form of an FFNN 620, which includes an input layer 621, two or more hidden layers 6221-6222, and an output layer 624. As mentioned earlier, more or fewer layers, including hidden layers, can be provided, and the trained ML model can take other forms. Also reappearing are the scrambler 660 and the descrambler 666, which can be used with... Figure 5 Components 560 and 566 operate similarly.

[0072] In this example, it does not include the entire gating layer. Figure 5 (562 in the middle), individual gate node 663 1-4 These nodes are scattered throughout the FFNN 620. For example, a first gate node 6611 couples the output from an input node (“I1”) to the input of a node in the first hidden layer 6221 and is controlled by a first key-locking routine 6631. The first gate node 6611 is controlled by the first key-locking routine 6631, which itself is coupled to a portion of the input layer 621 that includes data from the digital key 626. Similarly, a second gate node 6612 couples the output from another input node (“I2”) to another input node in the first hidden layer 6221. The second gate node 6612 is controlled by a second key-locking routine 6632, which, like the first key-locking routine 6631, is coupled to a portion of the input layer 621 that includes data from the digital key 626.

[0073] The third gate node 6613 is coupled between the output of a node in the first hidden layer 6221 and the input of a node in the second hidden layer 6222. Like the first gate node 6611 and the second gate node 6612, the third gate node 6613 is controlled by the third gate lock routine 6633. The third gate lock routine 6633 is again coupled to and controlled by the portion of the input layer 621 that includes data from the digital key 626.

[0074] The fourth gate node 6614 and the fifth gate node 6615 are coupled on one side between two nodes in the second hidden layer 6222, and on the other side between nodes in the scrambled output layer 664. These two gate nodes 661... 4-5 All are controlled by a single fourth door lock routine 6634. (This is related to the preceding door lock routine 663.) 1-3 Similarly, the fourth door lock routine 6634 is coupled to and controlled by the portion of the input layer 621 that includes data from the digital key 626.

[0075] With the door-locking routines 663 and the door nodes 661 spread throughout various locations of the FFNN 620, multi-level authentication can be facilitated in accordance with the digital key 626 provided as input. For example, in Figure 6 the input from the source 444 is provided to the data scrambler 660 along with data from the digital key 626. The data scrambler 660 scrambles these data as previously described to generate the scrambled input layer 621. Even with the scrambling, the data from the digital key 626 is provided as input to the door-locking routines 663 1-4 . These door-locking routines 663 can be configured (e.g., trained) to unlock and / or lock the door nodes 661 1-5 such that selected portions of the actual input data (‘10’, ‘11’, and ‘12’ in the scrambled input layer 621) are accurately or inaccurately provided from the source 444 to downstream components of the FFNN 620. Thus, a first digital key 626 provided by a user with relatively limited access can unlock only one or two selected door nodes 661. In contrast, a second digital key provided by another user with relatively unlimited access can unlock all of the door nodes 661 1-5 or a majority of the door nodes 661 1-5 in all of the door nodes 661.

[0076] As a non-limiting example, assume that the FFNN 620 is a convolutional neural network that receives digital images of patients as input. As noted above, privacy is no less important for many healthcare applications. Thus, different digital keys can be provided to different users in accordance with their relative level of access, such that the convolutional neural network processes only those portions of the images to which the respective user is granted access. For example, a user with limited permissions, such as a nurse or researcher who uses the FFNN 620 to analyze image data in an anonymous manner, can provide a digital key 626 that unlocks only those portions of the convolutional neural network that do not depict the face of a patient (sometimes referred to as “image patches”). A higher-level user, e.g., a doctor who cares for the patient depicted in the input data, can provide a digital key 626 that unlocks other portions of the input image, e.g., portions that depict the face of the patient.

[0077] In some embodiments, a similar process can also be performed at the output level, where, for example, the digital key 626 can unlock the desired level of output. For example, a nurse, researcher, or doctor using the FFNN 620 to classify an image can receive an output that provides decision support appropriate to their level of expertise. The nurse or researcher can have a global output, for example, indicating that the patient has signs of a suspicious lung nodule. In contrast, the doctor treating the patient can receive the location(s) and risk(s) of the malignant condition of the individual nodules, i.e., a more fine-grained output.

[0078] In another similar application, suppose a convolutional neural network is designed to analyze images of passengers captured at security checkpoints at an airport. A digital key provided by a low-level security agent can only unlock the portions of the FFNN 620 that correspond to image patches corresponding to certain regions of the passenger’s body. In contrast, a digital key provided by a high-level security agent or investigator (e.g., as part of a post-security breach investigation) can unlock a large portion, if not all, of the FFNN 620, enabling them to perform additional functions that the low-level security agent cannot use, e.g., facial recognition.

[0079] The locations at which the gate nodes 661 are inserted in the FFNN 620 can be selected in various ways. In some embodiments, these locations can be selected randomly or quasi-randomly. In other embodiments, these locations can be selected based on the relative impact that other nodes at or near those locations can have on the final output. For example, certain node pairs of a hidden layer 622 and / or between hidden layers can have a greater impact on the final output than other node pairs. In some embodiments, these high-impact node pairs can be coupled with gate nodes such that locking the gate nodes will render the output of the model meaningless, useless, etc. Additionally or alternatively, and as described above, in some embodiments, gate nodes (or gating layers) can be inserted at locations within a convolutional neural network that control which regions of an input digital image are processed, classified, etc. Thus, these inserted gate nodes (or gating layers) can be operated on by a digital key that indicates what level of access a user should have.

[0080] Figure 11A -B illustrates how gate nodes 661 can be inserted into an FFNN 1120 after training. The FFNN 1120 is similar in many respects to the FFNN 620. Once the gate nodes 661 are inserted, the data flow through the model during inference can be controlled primarily using the gate nodes 661. In some implementations, when the gate nodes 661 are inserted into the FFNN 1120, a routine such as depicted in Figure 11B can be employed.

[0081] In Figure 11B (Analogous to Figure 10B ), k again denotes a numeric key, e.g., 626. h i A particular hidden layer 1122 i is specified. Figure 11A In , again there are three sets of weights W 1-3 . The first set of weights W1 is between the input layer 1121 and the first hidden layer 11221. The second set of weights W2 is between the two gate nodes 561 1-2 and the second hidden layer 11222. The two nodes X 11 and X 13 of the first hidden layer 11221 are not coupled with a gate node. The third set of weights W3 is between the second hidden layer 11221 and the output layer 1124.

[0082] Figure 11B Back to Figure 11B , the activation function at the top can be, for example, a sigmoid function that introduces nonlinearity into the FFNN 1120. As explained in the notes (starting with the symbol “#”) in , if a valid numeric key k is applied, then G(k) will equal 1 for all gate nodes (i.e., X 12 and X 14 ), which means that the output from the nodes X 12 and X 14 in the first hidden layer 11221 is propagated unchanged to the second hidden layer 11222. For the non-gate nodes X 11 and X 13 of the first hidden layer 11221, G(k) is always one. On the other hand, if an invalid (or no) numeric key k is applied, then G(k) will equal 0 for the gate nodes (X 12 and X 14 ), which means that at least some of the output from the first hidden layer 11221 is essentially replaced with zero in propagating to the second hidden layer 11222. Thus, the output provided by the output layer 1124 will be meaningless.

[0083] Figure 7 FIGURE illustrates a flow diagram of an example method 700 for practicing selected aspects of the present disclosure, including the aspects illustrated in Figure 2 The operations of Figure 7 may be performed by one or more processors (e.g., one or more processors of the AI provider system 100 and / or one or more remote computing systems 102). To Figure 7 facilitate, the operations of Figure 7Additional steps beyond those illustrated in the figure can be performed in different order and / or in parallel Figure 7 steps, and / or one or more of the steps can be omitted. Figure 7 steps, and / or one or more of the steps can be omitted.

[0084] At block 702, the system can provide a digital key associated with a particular entity (e.g., 226, 426, 526, 626). For example, the Al provider system 100 can provide one or more digital keys to its customers (e.g., users of the remote computing system 102), which are unique to each customer / employee and / or to one or more ML models to which the customer has access.

[0085] At block 704, the system can apply the digital key as input across at least a portion of the trained machine learning model to generate one or more validation outputs. In various embodiments, the digital key can be applied as input across at least a portion of the ML model itself, and / or the digital key can be applied in conjunction with other inputs (e.g., data that the customer wishes to reason over). In the former case, in some embodiments, the digital key can be applied immediately prior to any other use of the ML model, or at other times (e.g., periodically) as needed (e.g., when the ML model is suspected of being compromised), at randomly selected times, etc. In various implementations, the digital key can be applied at the input layer of the model (e.g., 221) and / or at other potential entry points of the model (e.g., at one or more hidden layers, output layer, softmax / sigmoid layer, etc.).

[0086] At block 706, the system can compare one or more of the validation outputs to one or more known validation outputs. In various embodiments, the one or more known validation outputs can have been generated based on prior application of the digital key as input across at least the same portion of the trained machine learning model. Intuitively, if the ML model remains unchanged, then applying the same data across the same portion of the ML model at different times should yield the same output. Thus, in some embodiments, the comparison of block 706 seeks an exact match between the validation output(s) generated at block 704 and the known validation output(s). However, this is not meant to be limiting. In other embodiments, other types of comparisons can also be performed. For example, the validation output generated at block 704 can be applied as input to a hash function, and the result can be compared to another hash or value that serves as a proxy for or is generated from the known validation output(s).

[0087] At block 708, the system can determine the result of the comparison at block 706. If the answer at block 708 is that there is a match, then at block 710, the system can determine that one or more parameters of the trained machine learning model have been compromised. For example, the validation output generated at block 704 can not precisely match the known validation output. One possible reason is that one or more parameters of the ML model were tampered with, resulting in a difference between the validation output generated at block 704 and the known validation output.

[0088] At block 712, the system can cause one or more computing devices to provide output indicating that one or more parameters of the trained machine learning model have been compromised. For example, the AI provider system 100 can issue one or more audible or visual alerts, e.g., to a local user who is an employee of the entity that provides the AI provider system 100. Additionally or alternatively, in some embodiments, one or more alerts can be issued at one or more remote computing systems 102, e.g., to notify end users (e.g., 114) that a model to which they have access can no longer be reliable or trustworthy. In some embodiments, access to the ML model can be restricted to elimination until the integrity of the model can be reestablished, e.g., by retraining the model and / or by providing a replacement model.

[0089] Returning to block 708, if the answer is no, then at block 714, the system can determine that the trained ML model remains uncompromised. In some embodiments, no further action can be taken. In other embodiments, a successful integrity check can be logged, e.g., so that future investigators can determine that at least at one point in time the ML model was not compromised. This can help them determine when the ML model will be compromised in the future, if at all.

[0090] Figure 8 A flowchart of an example method 800 for practicing selected aspects of the present disclosure, including the aspects illustrated in Figure 4 will be described as being performed by a system. Other implementations can include additional steps, perform the steps in different orders and / or concurrently, omit certain steps, and / or modify other steps Figure 8 The operations of the method 800 can be performed by one or more processors (e.g., one or more processors of the AI provider system 100 and / or one or more remote computing systems 102). To Figure 8 The operations of the method 800 will be described as being performed by a system. Other implementations can include additional steps, perform the steps in different orders and / or concurrently, omit certain steps, and / or modify other steps Figure 8 The operations of the method 800 will be described as being performed by a system. Other implementations can include additional steps, perform the steps in different orders and / or concurrently, omit certain steps, and / or modify other steps Figure 8 The operations of the method 800 will be described as being performed by a system. Other implementations can include additional steps, perform the steps in different orders and / or concurrently, omit certain steps, and / or modify other steps Figure 8 The operations of the method 800 will be described as being performed by a system. Other implementations can include additional steps, perform the steps in different orders and / or concurrently, omit certain steps, and / or modify other steps

[0091] At block 802, a system (e.g., the remote computing device 102 or the AI provider system 100 prior to sending the model to the remote computing device 102) can encrypt one or more parameters of the trained machine learning model (e.g., 420) using an encryption key (e.g., 446) to generate an at least partially (if not fully) encrypted model (e.g., 420’). This can be similar to operation 456 in Figure 4 Various types of encryption can be employed at block 802, e.g., homomorphic encryption, etc.

[0092] At block 804, the system can encrypt input data to be applied as input across the trained machine learning model (e.g., 420’) (e.g., using the same encryption key (446)) to generate encrypted input data similar to 448 in Figure 4 At block 806, the system can apply the encrypted input data (e.g., 448) as input across at least a portion of the encrypted trained machine learning model to generate an encrypted output (e.g., 453).

[0093] At block 808, the system can decrypt the encrypted output using a decryption key (e.g., 426) to generate decrypted output data (e.g., 454). At block 810, the system can analyze the decrypted output data to determine whether one or more of the parameters of the trained machine learning model have been compromised. This analysis can take a variety of forms. In some embodiments, the analysis can include determining whether the decrypted output data (e.g., 454) conforms to an expected output structure. For example, one or more output values can be expected to be within a particular range, or can be expected to collectively sum to one. If the decrypted data 454 does not satisfy these constraints, it can be an indication that the model has been compromised. Based on the analysis at block 810, at block 812, the system can cause one or more computing devices (e.g., 102) to provide an output indicating that one or more parameters of the trained machine learning model have been compromised, similar to block Figure 7 710.

[0094] Figure 9 FIGURE illustrates a flow diagram of an example method 900 for practicing selected aspects of the present disclosure, including aspects presented in Figure 5 and / or Figure 6 FIGURE 1. Figure 9 The operations of FIGURE 1 can be performed by one or more processors (e.g., one or more processors of the AI provider system 100 and / or one or more remote computing systems 102). For ease of presentation, the operations of FIGURE 1 will be described as being performed by a system. Other implementations can include one or more processors performing the operations of FIGURE 1 in addition to Figure 9 Figure 9 ​The additional steps beyond those shown in the diagram can be performed in a different order and / or in parallel. Figure 9 One or more steps, and / or steps that can be omitted. Figure 9 One or more steps in the process.

[0095] At block 902, the system can provide a digital key associated with a specific entity. In various embodiments, the specific entity may have access to a machine learning model, such as an FFNN 220, 420, 520, or 620, which is trained to generate one or more outputs based on data applied across multiple inputs.

[0096] At block 904, the system can apply a digital key as input across at least the first part of the trained machine learning model to transition one or more gate nodes (e.g., 561, 661) scattered throughout the trained machine learning model between locked and unlocked states. For example, in Figure 5 In this process, digital key 526 is applied at the second, fourth, and fifth inputs of input layer 521 (after scrambling). Figure 6 That's also true. Figure 5 In this context, the data of digital key 526 is used to control the first key locking routine 5631 (its control gate node 561). 1-3 ) and the second key locking routine 5632 (its control gate node 561) 4-6 ).exist Figure 6 In this process, the data of digital key 626 is used to control the first key locking routine 6631 (which controls the first gate node 6611), the second key locking routine 6632 (which controls the second gate node 6612), the third key locking routine 6633 (which controls the third gate node 6613), and the fourth key locking routine 6634 (which controls the key node 661). 4-5 ).

[0097] At block 906, the system allows additional input data to be applied across at least the second part of the trained machine learning model (e.g., Figure 5 and Figure 6 Input I 1-3 ( ) as input to generate one or more outputs. For example, in Figure 5 In the middle, input data point I is applied at the first, third, and sixth inputs of the input layer 521 (after scrambling). 1-3 . Figure 6 That's also true.

[0098] Such as about Figure 5-6As described, in various embodiments, when a gate node is in an unlocked state, it can allow data received from an upstream layer of a trained machine learning model to be passed unchanged to a downstream layer of the same model. For example, in the unlocked state, the gate node can multiply data received from an upstream layer of the trained machine learning model by 1. In contrast, in a locked state, the gate node can disallow (or prevent) data received from an upstream layer of the trained machine learning model from being passed unchanged to a downstream layer. For example, in the locked state, the gate node can multiply data received from an upstream layer of the trained machine learning model by zero.

[0099] As previously mentioned Figure 5 As described, in some embodiments, one or more gate nodes may form a gate control layer that is inserted between two layers of a trained machine learning model. In some such embodiments, the gate nodes of the gate control layer can transition between an unlocked and locked state using a single key locking routine that receives all or part of a digital key (e.g., 526, 626) as input. And the entire digital key may not be used to lock / unlock all gate nodes. For example, in some embodiments, a first gate node can transition between a locked and unlocked state using a first portion of the digital key. A second gate node can transition between a locked and unlocked state using a second portion of the digital key that differs from (or only partially intersects with) the first portion of the digital key, and so on.

[0100] Figure 12A -B illustrates in a generalized manner how gating layers and / or individual gate nodes can be inserted into a trained machine learning model, which includes models other than neural networks (e.g., decision trees, support vector machines, random forests, k-nearest neighbors (“kNN”), Naive Bayes, logistic regression, etc.). Figure 12A The "encapsulation" method is demonstrated, in which a gating layer 12621 is inserted between the input layer 1221 and the model 1220, and a second gating layer 12622 is inserted between the model 1220 and the output layer 1264, similar to... Figure 5 The arrangement described in the text. Although in Figure 12A The text describes the gate layer 1262. 1-2 However, this does not mean there are restrictions; it can be changed to insert individual gate nodes.

[0101] Figure 12B It describes different "implantation" methods, which are more similar to Figure 6The method of where the gating layer 1262 (or individual gate nodes) is implanted between layers of the model 1220 such that the model 1220 is split into two parts: 1220i and 1220^ which are before and after the gating layer 1262, respectively. In some cases, the implantation method can not be applicable to certain types of machine learning models, e.g., support vector machines or logistic regression models. Ensemble-based methods (e.g., Ada Boost, stacking, bagging, etc.) can also utilize Figure 12B the implantation method of.

[0102] Figure 13A -C depicts an example of how the techniques can be applied to various types of machine learning models other than the previously described feedforward neural networks. Figure 13A shows how the gating layer G 1-4 or individual gate nodes can be inserted between a layer L 1-2 and L 3-8 of a decision tree 1320A. The bottom layer includes the output O 1-6 and the top layer includes the input.

[0103] Figure 13B shows an example of how the gating layer G 1-4 and G 5-8 can be inserted or implanted between the input t 1-3 and the distance to centroid layer D 1-4 and between the distance to centroid layer D 1-4 and the ranking / majority vote layer of a kNN classifier 1320B. Figure 13C shows an example of how the gating layer G 1-4 and G 5-8 can be inserted or implanted between the input I 1-3 and the bias subtraction layer b 1-4 and between the bias subtraction layer b 1-4 and the final activation layer (sign(W*I-b)) of a support vector machine 1320C.

[0104] While the examples described in this paper focus on in-model insertion of gating layers and / or gate nodes, this is not intended to be limiting. Similar mechanisms can be employed between different component models used as part of a larger ensemble model to control which output data is generated and which is not (or at least, coherent results are not generated). For example, the output of a first trained model can be used as part of a first CDS algorithm for a diagnostic, or as input to a second model for a second CDS diagnostic. If a customer only pays for a license to the first CDS algorithm and not for a license to the second CDS algorithm, one or more gating layers and / or gate nodes can be inserted between models to ensure that licensees only receive the content for which they have paid royalties.

[0105] Figure 14A -B demonstrates an example of how gating layers and / or gate nodes can be embedded into ensemble-type models (e.g., processing pipelines that include multiple machine learning models). Figure 14A In this context, the integrated model 1499A includes a "bagging" integrated model or a "stacked" integrated model, which comprises multiple component models 1420 arranged in parallel with each other. 1-N The bagging integration model comprises multiple component models of the same model type. The stacking integration model comprises multiple component models of different types. In this non-limiting example, the first model 14201 is divided into two parts: 14201' and 14201", with a gating layer 14621 inserted between them. One or more additional models (including model 1420) N The models are split in a similar manner. The outputs of these models are provided for another gating layer 1462. N+1 The input is fed into the consensus function and ultimately into the output 1464. Other arrangements of the stacked integration model or bagged integration model are anticipated in this paper.

[0106] Figure 14B This demonstrates how the techniques described in this paper can be applied with ensemble models of the "boosting" type, such as AdaBoost. Figure 14B In the integrated model 1499B, there are multiple component models 1420 arranged in series with respect to each other. 1-N A gating layer 1462 is inserted in the middle of these models. 1-N Similar to Figure 14A In this case, the final gating layer 1462 is inserted again between the final model 1420N and the output layer 1464. N+1 This paper envisions other arrangements of the lifting model with gating layers and / or individual gate nodes.

[0107] Figure 15is a block diagram of an example computer system 1510. Computer system 1510 typically includes at least one processor 1514 which communicates with a number of peripheral devices via bus subsystem 1512. These peripheral devices can include a storage subsystem 1524, including, for example, a memory subsystem 1525 and a file storage subsystem 1526, user interface output devices 1520, user interface input devices 1522, and a network interface subsystem 1516. Input and output devices allow user interaction with computer system 1510. Network interface subsystem 1516 provides an interface to an external network, and is coupled to corresponding interface devices in other computer systems.

[0108] User interface input devices 1522 can include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information to computer system 1510 or to communicate that information to another computer system.

[0109] User interface output devices 1520 can include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. Display subsystems can include cathode ray tube (CRT), flat-panel devices such as liquid crystal displays (LCDs), projection devices, or some other mechanism for creating a visible image. Display subsystems can also provide non-visual display, such as via audio output devices. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from computer system 1510 to the user or to another machine or computer system.

[0110] Storage subsystem 1524 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 1524 can include the logic to perform selected aspects of the methods described herein and / or to implement one or more of the components depicted in the various figures.

[0111] These software modules are generally executed by processor 1514 alone or in combination with other processors. Memory 1525 used in the storage subsystem 1524 can include a number of memories including a main random access memory (RAM) 1530 for storage of instructions and data during program execution and a read only memory (ROM) 1532 in which fixed instructions are stored. A file storage subsystem 1526 can provide persistent (nonvolatile) storage for program and data files, and can include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations can be stored by file storage subsystem 1526 in the storage subsystem 1524, or in other

[0112] Bus subsystem 1512 provides a mechanism for letting the various components and subsystems of computer system 1510 communicate with each other as intended. Although bus subsystem 1512 is shown schematically as one bus, alternative implementations of the bus subsystem can use multiple buses.

[0113] Computer system 1510 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 15 The description of computer system 1510 depicted in Figure 15 The computer system depicted in

[0114] While several inventive embodiments have been described and illustrated, it is understood that various other embodiments can be made within the scope of the present invention in light of the foregoing description. More generally, a person skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings of the present application are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that other embodiments may

[0115] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the terms.

[0116] The words "comprise," "comprising," "include," "including," and "includes" as used herein, specify the presence of stated features, integers, steps, processes, actions, elements, or components but do not preclude the presence or addition of one or more other features, integers, steps, processes, actions, elements, components, or groups thereof.

[0117] The phrase "and / or," as used herein in the specification and in claims, should be understood to mean "either or both" of the elements so conjoined, i.e., elements that are conjunctively present (whether associated with one another or present in a same location as the conjunctively present elements). Multiple elements listed with "and / or" should be construed in the same manner, i.e., "one or more" elements so conjoined. Other elements can optionally be present other than the elements specifically identified by the "and / or" clause, whether related or unrelated to those elements specifically identified. Thus, as a non-limiting example, a reference to "A and / or B" can refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); in yet another embodiment, to both A and B (optionally including other elements); and in yet another embodiment, to both A and B and at least one other element (optionally including other elements not mentioned specifically above). As used herein in the specification and in claims, "or" as used in a list of items prefaced by "at least one of" indicates a disjunctive list such that, for example, a list of "at least one of A, B, or C" means A or B or C or any combination thereof (e.g., A and B or A and C or B and C or A and B and C or A alone or B alone or C alone).

[0118] The phrase “and / or,” as used herein in the specification and in claims, should be understood to mean either “and / or” unless expressly indicated to the contrary. For example, in a list of items joined by “and / or” or “and / or”, such as “A and / or B and / or C,” at least one of A, B, and C, or any combination thereof, can be the minimum comprising the subject set of elements. In other words, A or, or B or, or C or any combination thereof, can be the minimum comprising the subject set of elements. In other words, when A, B, and C represent actions, property, items, or functions, and so on, the phrase “A and / or B and / or C” means at least one of the actions, property, items, or functions are possible (and use of “or” in the items A, B, and C means the minimal set of values). The phrase “and / or” as used herein in the specification and in the claims does not imply that theapparatuses, methods, or processes necessarily have related or interactive relationship, or that the features necessarily have to be included in any particular implementation. Unless otherwise expressly stated, the use of “or” in the claims should not be construed as an “exclusive or” unless expressly stated to be such.

[0119] The phrase “at least one,” as used herein in the specification and in claims, should be understood to mean one or more of the listed items or steps, that is, at least one but not necessarily including more than one. The use of the term “at least one” followed by a list of one or more items or steps should be interpreted as including at least one, but also including the possibility of more than one and, further, that an additional item or step from the list can also be present. For example, “at least one of A or B” or “at least one of A or B or C” shall be interpreted to mean at least one, but also to allow for the possibility of more than one, and that the additional item or step can also be present. The term “about” as used herein in the specification and in the claims, when used in reference to a particular recited numerical value, means that the value can vary from the recited value by no more than 1%, 2%, 5%, 10%, 15%, or 20% of the recited value.

[0120] It will be further understood that any method disclosed herein that includes more than one step or act can be interpreted to mean that the steps or acts of the method can be performed in any order or simultaneously, unless a particular ordering is specified or otherwise clear from the context.

[0121] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “providing,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of’ and “consisting essentially of’ shall be closed or semi-closed transitional phrases, as set forth in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03. It must be understood that certain expressions used in the claims, such as certain expressions referring to the use of articles “a” and “an” and “the” in the claims are used to provide all of the claimed elements or options of the claims, as the use of these articles is dependent on the jurisdiction in which the application is being examined. Accordingly, such expressions are to be read to “include, but not to the exclusion,” of additional subject matter unless specifically stated otherwise.

Claims

1. A method implemented using one or more processors, comprising: providing an encryption key associated with a particular entity (446), wherein the particular entity has access to a machine learning model (220, 420, 520, 620, 1020, 1120, 1220, 1320, 1420) trained to generate one or more outputs based on data applied across a plurality of inputs; encrypting (802) one or more parameters of the trained machine learning model using the encryption key; encrypting (804) input data to be applied as input across the trained machine learning model; applying (806) the encrypted input data as input across the encrypted trained machine learning model to generate an encrypted output; decrypting (808) the encrypted output using a decryption key to generate a decrypted output; analyzing (810) the decrypted output using the decryption key to determine that one or more of the parameters of the trained machine learning model have been compromised; and causing (812) one or more computing devices to provide an output indicating that the one or more parameters of the trained machine learning model have been compromised.

2. The method of claim 1, wherein, The encryption of the one or more parameters is performed using homomorphic encryption.

3. The method of claim 2, wherein, The encryption of the input data is performed using homomorphic encryption.

4. The method of claim 1, wherein, The analyzing includes determining whether the decrypted output conforms to an expected output structure.

5. The method of claim 4, wherein, The determining includes determining whether one or more output values of the decrypted output fall within a particular range.

6. The method of claim 4, wherein, The determining includes determining whether a plurality of output values of the decrypted output collectively add up to a predetermined value.

7. The method of claim 1, wherein, The particular entity accesses the trained machine learning model using a software application executing on a computing device (102, 112) operated by the particular entity.

8. The method of claim 7, further comprising configuring the software application to encrypt the input data in response to one or more events.

9. The method of claim 8, wherein, The one or more events include a user (114) associated with the particular entity operating the software application to perform a task that includes applying the input data across the trained machine learning model.

10. The method of claim 1, wherein, The trained machine learning model includes a generative adversarial network, a convolutional neural network, or a recurrent neural network.

11. The method of claim 1, wherein, The digital key is unique to the particular entity and the trained machine learning model.

12. A system comprising one or more processors and a memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to perform the following operations: providing an encryption key (446) associated with a particular entity, wherein, The particular entity has access to a machine learning model (220, 420, 520, 620, 1020, 1120, 1220, 1320, 1420) trained to generate one or more outputs based on data applied across a plurality of inputs; encrypting (802) one or more parameters of the trained machine learning model using the encryption key; encrypting (804) input data to be applied as input across the trained machine learning model; applying (806) the encrypted input data as input across the encrypted trained machine learning model to generate an encrypted output; decrypting (808) the encrypted output using a decryption key to generate a decrypted output; analyzing (810) the decrypted output using the decryption key to determine that one or more parameters of the trained machine learning model have been compromised; and causing (812) one or more computing devices to provide output indicating that the one or more parameters of the trained machine learning model have been compromised.

13. The system of claim 12, wherein, The encryption of the one or more parameters is performed using homomorphic encryption.

14. The system of claim 13, wherein, The encryption of the input data is performed using homomorphic encryption.

15. The system of claim 12, wherein, The analyzing includes determining whether the decrypted output conforms to an expected output structure.

16. The system of claim 15, wherein, The determining includes determining whether one or more output values of the decrypted output fall within a particular range.

17. The system of claim 15, wherein, The determining includes determining whether a plurality of output values of the decrypted output collectively add up to a predetermined value.

18. The system of claim 12, wherein, The particular entity accesses the trained machine learning model using a software application executing on a computing device (102, 112) operated by the particular entity.

19. The system of claim 12, wherein, The trained machine learning model includes a generative adversarial network, a convolutional neural network, or a recurrent neural network.

20. At least one non-transitory computer-readable medium comprising instructions that, in response to being executed by one or more processors, cause the one or more processors to perform the following: providing an encryption key (446) associated with a particular entity, wherein, The particular entity has access to a machine learning model (220, 420, 520, 620, 1020, 1120, 1220, 1320, 1420) trained to generate one or more outputs based on data applied across a plurality of inputs; encrypting (802) one or more parameters of a trained machine learning model using the encryption key; encrypting (804) input data to be applied as input across the trained machine learning model; applying (806) the encrypted input data as input across the encrypted trained machine learning model to generate an encrypted output; decrypting (808) the encrypted output using a decryption key to generate a decrypted output; analyzing (810) the decrypted output using the decryption key to determine that one or more parameters of the trained machine learning model have been compromised; and causing (812) one or more computing devices to provide output indicating that the one or more parameters of the trained machine learning model have been compromised.

Images