Proof of nerve flow

Abstract

Mechanisms are provided for implementing a neural flow proof engine and performing integrity verification based on the execution of a computer model with neural flows. Input data is input into a trained computer model including multiple layers of neurons. The neural flow proof engine records the output classes generated by the trained computer model and the neural flows through the multiple layers of neurons for a set of input data instances in the input data, thereby generating the recorded neural flows. The trained computer model is deployed to a computing platform, and the neural flow proof engine verifies the execution integrity of the deployed trained computer model based on the runtime neural flows of the deployed trained computer model and the recorded neural flows.

Description

Technical Field

[0001] This application generally relates to improved data processing apparatus and methods, and more particularly to mechanisms for performing neural flow proofs. Background Art

[0002] Deep learning systems have been widely deployed as part of artificial intelligence (AI) services because they can approach human performance when performing cognitive tasks. Deep learning is a class of machine learning techniques that uses cascaded multi-layer non-linear processing units for feature extraction and transformation. Each successive layer uses the output of the previous layer of the input. Supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) learning mechanisms are used to train deep learning systems. Learning can be performed with respect to multiple representation levels corresponding to different levels of abstraction, where these levels form a hierarchy of concepts.

[0003] Most modern deep learning models are based on artificial neural networks, although they can also include propositional formulas or latent variables organized layer by layer in deep generative models, such as nodes in deep belief networks and deep Boltzmann machines. In deep learning, each level, e.g., a layer of a neural network, learns to transform its input data into a slightly more abstract and synthetic representation. In an image recognition application, for example, the original input can be a pixel matrix, where multiple abstract layers of pixels are the identified features. Importantly, the deep learning process can learn which features place themselves optimally at which level, but this does not completely avoid the need for manual tuning. For example, manual tuning can be used to change the number of layers and layer sizes to provide different degrees of abstraction.

[0004] The "depth" in "deep learning" refers to the number of layers through which data is transformed. More precisely, a deep learning system has a substantial credit assignment path (CAP) depth. The CAP is a chain of transformations from input to output. The CAP describes the potential causal connection between the input and the output. For a feedforward neural network, the depth of the CAP is the depth of the network and is the number of hidden layers plus one (since the output layer is also parameterized). For a recurrent neural network, where signals can propagate through layers more than once, the CAP depth can be infinite. There is no generally agreed-upon depth threshold that separates shallow learning from deep learning, but most researchers agree that deep learning involves a CAP depth greater than 2. A CAP of depth 2 has been proven to be a universal approximator in the sense that it can simulate any function. Beyond this, more layers do not increase the network's function approximator ability, but additional layers help with learning features.

[0005] However, it is at least necessary to verify the integrity of the deployed trained computer model. Therefore, there is a need in the art to solve the foregoing problems. Summary of the Invention

[0006] From another aspect, the present invention provides a method in a data processing system including at least one processor and at least one memory, the at least one memory including instructions that are executed by the at least one processor to configure the at least one processor to implement a neural flow proof engine. The method includes: inputting, by the neural flow proof engine, input data into a trained computer model, where the trained computer model includes multiple layers of neurons; recording, by the neural flow proof engine, for a set of input data instances in the input data, the output classes generated by the trained computer model and the neural flow through the multiple layers of neurons, thereby generating the recorded neural flow, where the output class is one of a plurality of possible output classes; deploying the trained computer model to a computing platform; and verifying, by the neural flow proof engine, the integrity of the deployed trained computer model based on the runtime neural flow of the deployed trained computer model and the recorded neural flow.

[0007] A computer program product includes a computer-readable storage medium having a computer-readable program stored therein. When the computer-readable program is executed on a data processing system, it causes the data processing system to implement a neural flow proof engine that performs the following operations: inputting input data into a trained computer model, where the trained computer model includes multiple layers of neurons; recording, for a set of input data instances in the input data, the output classes generated by the trained computer model and the neural flow through the multiple layers of neurons, thereby generating the recorded neural flow, where the output class is one of a plurality of possible output classes; deploying the trained computer model to a computing platform; and verifying the integrity of the deployed trained computer model based on the runtime neural flow of the deployed trained computer model and the recorded neural flow.

[0008] From another aspect, the present invention provides a system including: at least one processor; and at least one memory coupled to the at least one processor, where the at least one memory includes instructions that, when executed by the at least one processor, cause the at least one processor to implement a neural flow proof engine configured to: input data into a trained computer model, where the trained computer model includes multiple layers of neurons; record, for a set of input data instances in the input data, the output classes generated by the trained computer model and the neural flow through the multiple layers of neurons, thereby generating the recorded neural flow, where the output class is one of a plurality of possible output classes; deploy the trained computer model to a computing platform; and verify the execution integrity of the deployed trained computer model based on the runtime neural flow of the deployed trained computer model and the recorded neural flow.

[0009] From another aspect, the present invention provides a computer program product for implementing a neural flow proof engine. The computer program product includes a computer-readable storage medium that can be read by a processing circuit and stores instructions for performing a method for executing the steps of the present invention by the processing circuit.

[0010] From another aspect, the present invention provides a computer program stored on a computer-readable medium and loadable into the internal memory of a digital computer, including software code portions for performing the steps of the present invention when the program runs on the computer.

[0011] In an illustrative embodiment, a method in a data processing system including at least one processor and at least one memory, the at least one memory including instructions that are executed by the at least one processor to configure the at least one processor to implement a neural flow proof engine. The method includes inputting input data into a trained computer model by a neural flow authentication engine, where the trained computer model includes multiple layers of neurons. The method further includes recording, by the neural flow authentication engine, an output class generated by the trained computer model and a neural flow through the multiple layers of neurons for a set of input data instances in the input data, thereby generating the recorded neural flow. The output class is one of multiple possible output classes. The method also includes deploying the trained computer model to a computing platform and validating the integrity of the deployed trained computer model by the neural flow proof engine based on the runtime neural flow of the deployed trained computer model and the recorded neural flow.

[0012] In other illustrative embodiments, a computer program product is provided that includes a computer-usable or readable medium having a computer-readable program. When the computer-readable program is executed on a computing device, the computer-readable program causes the computing device to perform various operations and combinations of operations outlined above in connection with the method illustrative embodiments.

[0013] In yet another illustrative embodiment, a system / apparatus is provided. The system / apparatus may include one or more processors and a memory coupled to the one or more processors. The memory may include instructions that, when executed by the one or more processors, cause the one or more processors to perform various operations and combinations of operations outlined above in connection with the method illustrative embodiments.

[0014] These and other features and advantages of the present invention will be described in the following detailed description of the exemplary embodiments of the present invention, or will become apparent to those of ordinary skill in the art in view of the following detailed description of the exemplary embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] When read in conjunction with the accompanying drawings, the present invention, its preferred modes of use, and further objects and advantages will be best understood by reference to the following detailed description of illustrative embodiments, in which:

[0016] Figure 1 is an example diagram showing a preprocessing stage of an operation for collecting neural flow information for training data for training a computer model according to an illustrative embodiment;

[0017] Figure 2 is an example diagram showing a deployment stage of an operation for performing a runtime neural flow proof of a deployed computer model according to an illustrative embodiment;

[0018] Figure 3 depicts a graphical representation of an example distributed data processing system in which aspects of the illustrative embodiments can be implemented;

[0019] Figure 4 is a block diagram of an example computing device in which aspects of the illustrative embodiments can be implemented;

[0020] Figure 5 is a flowchart outlining example operations for performing the functions of the preprocessing stage according to an illustrative embodiment;

[0021] Figure 6 is a flowchart outlining example operations for performing the functions of the deployment stage according to an illustrative embodiment, the deployment stage functions including runtime neural flow proof of the neural flow;

[0022] Figure 7 depicts a cloud computing environment according to an embodiment of the present invention; and

[0023] Figure 8 depicts an abstract model layer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Artificial intelligence (AI) systems and AI services utilize machine learning (ML) models as part of their backend engines. For example, many AI systems and services use deep neural networks (DNNs) (also known as deep learning models) to perform the underlying intelligent operations that support system operations and services. The present invention specifically relates to an improved computer tool for helping to ensure the execution integrity of such machine learning models in various computer runtime environments and, in some illustrative embodiments, in cloud computing environments. It should be understood that the illustrative embodiments described herein will relate to computer models and are intended to relate to any type of machine learning computer model. A deep learning model is a neural network having two or more intermediate layers of neurons and is used as a primary example of a computer model that can be used with the illustrative embodiments. However, the implementation of the illustrative embodiments with deep learning models is not intended to limit the potential applications of the illustrative embodiments. Instead, any other known or later developed neural network models with various numbers or sizes of intermediate layers for which exemplary embodiments can be implemented are considered to be within the scope of the present invention.

[0025] Deep learning models are being more commonly used in public cloud computing environments (also simply referred to herein as "cloud" platforms, environments, etc.) to provide artificial intelligence cloud services such as image recognition and analysis services, decision support services, etc. For example, many cloud computing environments are multi-tenant cloud computing environments or cloud platforms where the architecture allows customers (or tenants) to share computing resources in a public or even private cloud, the computing resources including the hardware and software provided by the cloud provider. Each tenant's data is isolated and remains invisible to other tenants. Most public cloud providers use a multi-tenant architecture. With respect to DL models, each tenant can utilize a separate DL model instance. The DL model itself can be provided by a third party or even by the cloud provider, where a single instance of the DL model is specifically configured for a particular tenant and the particular cloud services they wish to provide.

[0026] Deploying deep learning (DL) models in a public cloud computing environment is an effective way to bootstrap artificial intelligence services. However, cloud computing tenants typically have concerns about the runtime integrity of the execution of the deployed DL models in such public cloud computing environments. These concerns are based on three main viewpoints of cloud computing tenants. First, attackers who are able to penetrate the cloud platform often inject backdoors into the DL models, where a backdoor is an undisclosed way of gaining access to a program, online service, or computing system by bypassing normal authentication mechanisms. Such a backdoor can be activated by maliciously crafted inputs having a special embedded pattern, such as a watermark in image data, a Trojan horse (or "Trojan"), etc. Thus, the attacker can deviate the DL model execution and influence the final prediction for their own benefit.

[0027] Second, cloud platform providers may have a financial incentive to replace a tenant's DL model with a pruned or "compressed" counterpart, which can reduce storage and computational costs at the expense of some loss of accuracy. That is, the DL model trainer can retrain the DL model by attempting to remove some components in the DL model, such as connections, neurons, or layers. However, the DL model trainer needs to ensure that the pruned model can achieve the same level of accuracy as the original DL model. Thus, the cloud provider, i.e., the entity that provides computing resources to host cloud services, can significantly reduce its operating costs for the online service model. However, tenants with high standards of accuracy may not accept such model transformations and replacements. This can also lead to liability issues, i.e., determining who is responsible for incorrect predictions at runtime, as such model transformations / replacements may be the cause of incorrect predictions, or the original model may already be the cause of incorrect predictions.

[0028] Third, existing DL models are known to be vulnerable to attacks. An attacker can exploit DL model vulnerabilities to craft adversarial samples as online prediction inputs. Such adversarial inputs will be misclassified into a category predetermined by the attacker. Cloud tenants tend to detect those misclassified samples and analyze why they are classified into a particular category different from the training data. In addition, cloud tenants can collect those misclassified samples for adversarial training to further enhance the robustness of their DL models. As will be described below, it will be understood that through the mechanisms of the illustrative embodiments and the implementation of the neural flow proof of the illustrative embodiments, adversarial examples can be detected at runtime, and these adversarial examples can be collected for further investigation.

[0029] Illustrative embodiments provide a mechanism for solving the problem of the execution integrity of computer models (or simply referred to as "models"), such as deep learning (DL) models, in various computing platforms such as public cloud platforms and data processing environments. Illustrative embodiments provide a runtime neural flow proof scheme for verifying the execution integrity of a computer model at the abstract level of neural flow. Neural flow is the execution order of a deep learning model and is represented by the cross-layer neuron activation states corresponding to a specific input.

[0030] For each input instance provided to a trained and deployed computer model such as a DL model, a Trusted Execution Environment (TEE) is utilized to faithfully record and log its associated neural flow for its execution to pass through and send a proof report to the tenant. On the tenant side, the user maintains a measurement database that stores the pre-recorded neural flows for all training data of different classes, i.e., the potential classification outputs of the DL model. Based on the prediction results, i.e., the classification outputs generated by the DL model in the cloud computing environment, the tenant can verify whether the neural flow recorded in the cloud deviates from the neural flow of the training data of the same class of their DL model. If the DL model in the cloud computing environment deviates from the neural flow of the training data, or deviates by more than a predetermined amount indicated by one or more predefined criteria, this can be considered an abnormal neural flow, which indicates that the DL model in the cloud computing environment may have been compromised and needs to be taken back for further investigation. Thus, the illustrative embodiments allow cloud platform providers to prove the runtime execution state to their tenants and allow tenants to verify the state of their online DL models with their locally resident measurement database.

[0031] For an illustrative embodiment, assume that a cloud computing environment user (i.e., an entity that owns a computer model such as a DL model) who wishes to deploy a computer model to a cloud computing environment has trained their model using their own training data (such as part of an offline preprocessing operation). The user (the entity that owns the model; also referred to as a "tenant" in the context of a cloud computing platform) intends to supply the model to a model of a service cloud platform provider to instantiate an online cloud service, such as an online classification or prediction service using a computer model trained with machine learning such as a DL model. For example, the user may want to instantiate an image recognition classification service, a healthcare decision support service, or any one of numerous other services that perform classification operations, prediction operations, etc. using an artificial intelligence-based computer model such as a DL model.

[0032] After training the model, the user passes the training data through the trained model again, where the neural flow extraction engine of the illustrative embodiment operates to extract neuron activation information at each layer of the trained model. The neuron activation information can have different contents based on the specific layer type of the computer model. For example, for a convolutional layer, each filter within the layer is considered a unit, and the average activation value of each filter is calculated as part of the neuron activation information. For a fully connected layer, each neuron is considered a unit, and its activation value is extracted. For each layer, a threshold can be predetermined, and only the identities and activation values of the filters / neurons that meet the criteria associated with the threshold are recorded, e.g., only the activation values of the filters / neurons that meet or exceed the threshold are recorded.

[0033] The neural flow extraction engine combines neuron activations across layers to generate a neural flow for each training data instance. The neural flow aggregation engine aggregates the neural flows for all training data in the same class and generates a neural flow model for each class separately. The neural flow models are stored in a measurement database at the tenant's local computing system for later comparison with neural flows generated at a remotely located cloud deployment of the computer model.

[0034] After training the model as described above and storing the neural flow models for each class of output generated by the model, the user can deploy the computer model to the cloud computing environment of the cloud platform provider and start a cloud service corresponding to the deployed computer model, e.g., a trained DL model. Additionally, a Trusted Execution Environment (TEE) is generated or launched on the cloud computing environment. The TEE is an isolated execution environment for the computing resources of the cloud computing platform that provides security features such as isolated execution, integrity of applications executed within the TEE, and confidentiality of computing assets maintained within the TEE. Using the TEE, code can be executed within the TEE at a high level of trust because the TEE can ignore threats from the rest of the cloud computing platform outside the TEE. The creation and use of the TEE are well known in the art, and thus, no more detailed explanation is provided here.

[0035] Within the TEE, one or more recording components are provided to intercept the execution of the deployed model and obtain neural flow information for each input to the executing model. That is, the model (e.g., a DL model) executes outside the TEE, however, a recording component such as another instance of the neural flow extraction engine discussed above executes within the TEE. When the model receives an input for classification, prediction, etc., the recording component within the TEE captures the neural flow for the input by again extracting neuron activation information at each layer of the deployed model and combining the neuron activations across these layers, thereby generating the neural flow of the deployed model when processing the input. It should be understood that the model itself does not need to be modified for the mechanism within the TEE to capture neural flow information. The recording mechanism is added to the deep learning online execution framework. Once the model starts executing on a particular input data instance, the execution at each layer is intercepted by the recording mechanism as the input passes through the model. At each interception, the recording mechanism contained within the TEE is called to extract the activation information. Thereafter, all the activation information collected across all layers is combined to construct the neural flow for that input data instance.

[0036] The captured neural stream information can be stored in the secure storage device of the TEE and provided back to the user's local computing system, e.g., the local computing system or computing device of the tenant. For example, when the user sends an input instance for the deployed model to process, the user can associate the input with a specific challenge C for freshness testing. This challenge is used to defend against "replay attacks". That is, challenge C helps ensure that the returned neural stream is not replaced by a commercially pre-made or pre-prepared version of the neural stream provided by the computing platform. Thus, for each input, the user or, in the case of a cloud computing platform, the cloud tenant sends a randomly or pseudo-randomly generated challenge C to the recording component. This challenge can only be used within the TEE, and the computing platform provider cannot access challenge C. This challenge is associated with the neural stream recorded and associated with the input. When the user (or cloud tenant) receives the neural stream, the user / cloud tenant can verify the integrity and freshness of the neural stream based on this challenge C.

[0037] For example, within the TEE, the neural stream for the input is captured together with information about the input and the output of the model, and hashed together with challenge C, and the certified neural stream or "proof" is encrypted using a security key. As part of the startup of the TEE, this security key can be exchanged between the user or tenant, the computing system / device, and the TEE. The exchanged security key is used to encrypt the data exchanged between the TEE and the user or between the tenant computing system / device (such as the packaged recorded neural stream, the input / output of the computer model, and the hash with challenge C), thereby generating a sealed proof.

[0038] When the sealed proof is received at the proof engine of the user's local computing system from the TEE of the cloud computing platform, the user can use challenge C to first verify the integrity and freshness of the proof and extract the input / output information of the computer model and the neural stream corresponding to the processing of the input. Based on the output classification or prediction generated by the model for the input, the proof engine retrieves the corresponding neural stream model for the specific class stored based on the training data during the previously described preprocessing operation from the local measurement database. The neural stream in the proof is compared with the stored neural stream from the measurement database generated by the training data.

[0039] If the deployed model operates as it should and no attacker, cloud computing provider, etc. performs modifications to the deployed model, then the neural stream from the proof should closely approximate the stored neural stream for the same class in the measurement database. There may be some relatively small differences within a given tolerance. However, overall, the neural streams should be the same. However, if there are significant deviations, e.g., deviations greater than the given tolerance, or deviations that meet the damage criteria, then the deployed model may have been compromised in some way, either innocently or maliciously.

[0040] Using illustrative embodiments, the deviation between neural streams can be evaluated, for example, by applying predefined computer-executed rules, comparison with one or more deviation thresholds indicating the acceptable degree of deviation, etc., to determine whether to trigger an alert notification, thereby notifying the user of the potential impairment of the deployed model. If the alert notification is triggered, an alert can be output to the user, allowing the user to select whether to automatically withdraw the deployed model. If the user selects to withdraw the deployed model, a computer command can be sent to the cloud computing provider to cause the deployed model to be removed from public access via the cloud computing platform.

[0041] Accordingly, illustrative embodiments provide a mechanism for performing runtime neural stream proof of the execution of a deployed computer model, particularly with respect to neural network and deep learning (DL) network computer models. Specifically, runtime neural stream proof is facilitated by capturing and comparing neural streams regarding the execution of the computer model during the preprocessing or training phase of operation and after the deployment of the trained computer model. Runtime neural stream proof allows determination of whether the deployed model has been impaired after deployment, such that the model owner can take appropriate actions to ensure that their model does not generate outputs that are not desired by the model user.

[0042] Before beginning the discussion of aspects of the exemplary embodiments, it should first be understood that throughout the specification, the term "mechanism" will be used to refer to the elements of the present invention that perform various operations, functions, etc. As used herein, the term "mechanism" can be an implementation of a function or aspect of an illustrative embodiment in the form of a device, a computer-executed process or method, or a computer program product. In the case of a process, the process is implemented by one or more devices, apparatuses, computers, data processing systems, etc. In the case of a computer program product, the logic represented by computer code or instructions included in or on the computer program product is executed by one or more hardware devices to implement the functions associated with a particular "mechanism" or to perform the operations associated with a particular "mechanism". Thus, the mechanisms described herein can be implemented as dedicated hardware, software executed on general-purpose hardware, software instructions stored on a medium such that the instructions can be readily executed by dedicated or general-purpose hardware, a process or method for performing functions, or any combination of the above.

[0043] This specification and the claims may use the terms "a", "at least one", and "one or more" with respect to specific features and elements of the illustrative embodiments. It should be understood that these terms and phrases are intended to indicate that there is at least one specific feature or element in a particular illustrative embodiment, but there may also be more than one. That is, these terms / phrases are not intended to limit the specification or claims to the presence of a single feature / element or require the presence of multiple such features / elements. Instead, these terms / phrases only require at least a single feature / element, where the possibility of multiple such features / elements is within the scope of the specification and claims.

[0044] Furthermore, it should be understood that if used herein with respect to describing embodiments and features of the present invention, the use of the term "engine" is not intended to limit any particular implementation for implementing and / or performing actions, steps, processes, etc. attributable to and / or performed by the engine. An engine can be, but is not limited to, software, hardware, and / or firmware or any combination thereof that performs a specified function, including but not limited to any combination of a general and / or special purpose processor and appropriate software loaded or stored in a machine-readable memory and executed by the processor. Additionally, unless otherwise specified, any name associated with a particular engine is for purposes of convenience of reference and is not intended to be limited to a particular implementation. Moreover, any functionality attributed to an engine can be equally performed by multiple engines, incorporated into and / or combined with the functionality of another engine of the same or different type, or distributed across one or more engines in various configurations.

[0045] Furthermore, it should be understood that the following description uses multiple various examples of various elements of the illustrative embodiments to further illustrate example implementations of the illustrative embodiments and to assist in understanding the mechanisms of the illustrative embodiments. These examples are intended to be non-limiting and are not an exhaustive list of the various possibilities for implementing the mechanisms of the illustrative embodiments. Given this specification, it will be apparent to those of ordinary skill in the art that, without departing from the scope of the present invention, many other alternative implementations of these various elements can be utilized in addition to or as an alternative to the examples provided herein.

[0046] The present invention may be a system, method, and / or computer program product. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0047] A computer-readable storage medium can be a tangible device that is capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium can be, by way of example and not limitation, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer-readable storage medium includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punched card or raised structures in a groove having instructions recorded thereon, and any appropriate combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse through an optical fiber cable), or an electrical signal transmitted through a wire.

[0048] The computer-readable program instructions described herein can be downloaded to a respective computing / processing device from a computer-readable storage medium or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include a copper transmission cable, an optical transmission fiber, a wireless transmission, a router, a firewall, a switch, a gateway computer, and / or an edge server. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0049] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Java, Smalltalk, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer through any type of network connection, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, in order to perform aspects of the present invention, an electronic circuit, including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), may execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit.

[0050] Aspects of the present invention are described herein with reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0051] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed via the processor of the computer or other programmable data processing apparatus create a means for implementing the functions / acts specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium, which may direct a computer, a programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored comprises an article of manufacture including instructions for implementing aspects of the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0052] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of the flowchart and / or block diagram.

[0053] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of an instruction that includes one or more executable instructions for implementing the specified logical function. In some alternative embodiments, the functions recited in the blocks may occur out of the order recited in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by a special purpose hardware-based system that performs the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0054] As described above, the illustrative embodiments provide a mechanism specifically for facilitating a runtime neural flow proof for the execution of a deployed machine learning computer model to ensure the integrity of the computer model, such as after deployment in a cloud computing environment or other distributed data processing system. The mechanism of the illustrative embodiments operates in a local computing system associated with a user (computer model owner) for the purpose of capturing the neural flow of a trained computer model prior to deployment, and operates in a remotely located computing system for the purpose of capturing the neural flow after the trained computer model is deployed for runtime execution, where the remotely located computing system is one or more computing systems such as those providing a cloud computing platform. The mechanism allows the use of a trusted execution environment to capture the neural flow of the trained and deployed computer model and report it back to the user to verify the neural flow against the stored neural flow captured during a preprocessing or training phase prior to the deployment of the trained computer model. Based on this comparison, it can be determined whether the trained computer model deployed after training has been compromised, and appropriate actions can be taken.

[0055] Figure 1 is an example diagram of a preprocessing or training phase showing operations for collecting neural flow information for training data for a computer model. Figure 1The operations outlined assume that a deep neural network (DNN) or deep learning (DL) model 110 (hereinafter referred to as the DL model) has been trained using a training data set 120 and a machine learning training process. Such machine learning training processes are well known in the art, and thus, a detailed explanation of the process of training the DNN or DL model is not provided here. For example, as long as the machine learning training process adjusts the operating parameters, such as weights, of the respective layers L1-Ln of the DL model 110 so that the DL model 110 appropriately classifies the input into one of a plurality of recognizable classes, or generates a predicted output, that is sufficient.

[0056] After training the DL model 110, the training data is input into the now-trained DL model 110 again, as Figure 1 shown. The neural flow (NF) proof engine 130 captures and stores the neural flow for the training data and generates a neural flow model for each class in the set of recognizable classes or for the predictions in the set of potential predictions based on the captured neural flow. For purposes of description, it will be assumed that the DL model 130 is configured to classify the input into a class in a predetermined set of classes. In some illustrative embodiments, this classification involves calculating a probability value for each of a predetermined number of recognized classes such that the class with the highest probability value or score is selected as the output class for the input. Thus, the output of the DL model 110 can be a vector output, where each vector slot corresponds to a different class, and the value in the vector slot represents a probability or score indicating that the corresponding class is the correct classification for the input to the DL model 110. The probability value or score itself is calculated through the operations of the various neurons of the layers L1-LN of the DL model 110.

[0057] The neural flow extraction engine 132 of the neural flow (NF) proof engine 130 extracts the neuron activation information at each layer L1-LN of the trained DL model 110. The neural flow extraction engine 132 combines the neuron activations across L1-LN (e.g., the vector slots shown in bold in the Figure 1 neural flow) to thereby generate a neural flow 134 for each training data instance in the training data 120. Thus, when the trained DL model 110 processes a training data instance as an input to the DL model 110, the DL model 110 generates neuron activation information that is captured by the NF extraction engine 132 and is related to the output classification generated by the DL model 110. Thus, when each training data instance is processed, a corresponding neural flow 134 and output classification are generated for the training data instance and stored by the NF proof engine 130 for processing by the NF aggregation engine 136.

[0058] The NF aggregation engine 136 aggregates the neural streams 134 of all the training data classified in the same class and generates a neural stream model 140 for each class respectively. For example, if the DL model 110 is trained to classify an input into one of 100 different recognizable classes, then for each of these 100 classes, a corresponding neural stream model 140 is generated by aggregating the neural streams 134 of the training data instances classified into the corresponding class. That is, in this example, 100 neural stream models 140 will be generated. The aggregation of the neural streams 134 can be performed in any of a variety of different ways without departing from the scope of the present invention. For example, in an illustrative embodiment, a "set" of activated neurons and filters of all the training data in the same class can be identified (depending on whether the layer is a convolutional layer or a fully connected layer). In other illustrative embodiments, a machine learning model, such as a Bayesian model, an SVM model, or a graph neural network model, can be trained based on the activated neurons and filters. The machine learning model can consider the order of activation and the frequency at which the activation occurs. As a result, a similarity match for approximating unseen test data at runtime can be performed.

[0059] The neural flow model 140 is stored in the measurement database 150 in association with an indicator of the corresponding class to which the neural flow model 140 corresponds. Thus, when using the neural stream to verify the execution of the deployed model, the corresponding neural stream model 140 for a specific class can be retrieved from the measurement database 150 and used to perform model integrity verification, such as by performing model integrity verification through the neural stream verification engine 138 described below.

[0060] After training the DL model 110 as described above and storing the neural process model 140 of each class of the output generated by the DL model 110 in the measurement database 150, the user can deploy the trained DL model 110 to the cloud computing environment of a cloud platform provider and start a cloud service corresponding to the deployed DL model. Figure 2 is an example diagram showing a deployment phase of an operation for performing runtime neural stream attestation of an operation of the deployed DL model 210 according to an illustrative embodiment. In Figure 2 the right hand side of the figure, labeled "Cloud", refers to the cloud computing platform and cloud computing environment in which the DL model 210 is deployed and executed with respect to new inputs 220 received from users of the cloud service supported by the DL model 210. For example, the "Cloud" side of the figure can be a specific portion of the cloud computing resources (hardware and / or software) allocated to the owner of the DL model 210, such as a part of a multi-tenant cloud computing environment.

[0061] As Figure 2As shown, the cloud computing platform provides an NF runtime neural flow proof engine 230 and a trusted execution environment (TEE) 236 for the deployment of the DL model 210. In some illustrative embodiments, the NF runtime neural flow proof engine 230 or a subset of the components of the NF runtime neural flow proof engine 230 may be executed within the TEE 236, or the TEE 236 may be provided as part of the NF runtime neural flow proof engine 230. However, for ease of description, the runtime neural flow proof engine 230 is shown outside the TEE 236, which is not a requirement of the present invention. However, it should be noted in particular that the deployed trained DL model 210 executes outside the TEE 236, while the capture, storage, and reporting of the neural flow of the deployed trained DL model 210 are executed within the TEE 236 to ensure that the recorded neural flow is not tampered with.

[0062] As Figure 2 shown, the NF runtime neural flow proof engine 230 includes an NF extraction engine 232 that operates to record the neural flow (NF) of the DL model 210 operating on the input 220. The NF extraction engine 232 may operate in a manner similar to the Figure 1 NF extraction engine 132, but is executed in the cloud computing platform with respect to the deployed trained DL model 210. In some illustrative embodiments, the NF extraction engine 232 is executed within the TEE 236 to collect neural flow information, which is referred to as proven neural flow information because it is used to prove the correct execution of the deployed trained DL model 210.

[0063] The NF runtime neural flow proof engine 230 also includes a proof generation engine 234 that generates a proof output that is sent to the user's local computing system (such as Figure 1 shown) for verification by the NF proof engine 130. Data communication between the NF runtime neural flow proof engine 230 in Figure 2 and the NF proof engine 130 in Figure 1 is facilitated via the runtime neural flow proof interfaces 137 and 238 of the respective computing systems or platforms.

[0064] As Figure 2As shown, when the deployed trained DL model 210 receives an input 220 for processing such as classification, prediction, etc., the NF extraction engine 232 that can be executed within the TEE 236 captures and stores the neural flow 240 for the input 220 by re-extracting the neuron activation information at each layer L1-LN of the deployed trained DL model 210 and combining the neuron activations across L1-LN, thereby generating the neural flow 240 of the deployed trained DL model 210. The captured neural flow information 240 can be stored in the secure storage device of the TEE 236 and provided back to the NF proof engine 130 executing on the user's local computing system.

[0065] For example, when the user computing system 250 sends the input 220 for processing by the deployed trained DL model 210, the user computing system 250 can associate the input with a specific challenge C for determining the freshness of the proof received from the NF runtime neural flow proof engine 230. Within the TEE 236, the neural flow 240 for the input 220 is captured by the NF extraction engine 232 along with information about the output of the model. The extracted NF and the model input (i.e., the class to which the input is classified) along with the challenge C are provided to the proof generation engine 234. The proof generation engine 234 hashes the NF and the model output along with the challenge C and can encrypt the result using the secure key exchanged between the user / tenant computing device and the TEE as part of the TEE startup, thereby sealing the data and generating a proof, which is then sent back to the user computing device 250 implementing the NF proof engine 130. Alternatively, in some illustrative embodiments, the NF proof engine 130 can be provided on a computing device separate from the user computing device 250, provided that the separate computing device is also provided with the challenge such that the sealed proof can be accessed and verified.

[0066] When the sealed proof is received at the NF proof engine 130 of the user's local computing system (e.g., the user computing device 250) via the interfaces 137 and 238 from the TEE 236, the neural flow verification engine 138 of the NF proof engine 130 executing on the user computing device 250 (see Figure 1 ) can use the challenge C to first verify the integrity and freshness of the proof and extract the output information of the DL model and the neural flow corresponding to the processing of the input 220. Based on the output classification or prediction generated by the DL model 210 for the input 220, the neural flow verification engine 138 retrieves from the local measurement database 150 the previous Figure 1The corresponding neural flow model 140 for a specific class stored based on training data during the described preprocessing operation. The neural flow 240 in the proof is compared with the stored neural flows of the neural flow models 140 from the measurement database 150 generated by the training data.

[0067] As previously described, if the deployed trained DL model 210 operates as it should and no modifications to the deployed trained DL model 210 are performed by an attacker, cloud computing provider, etc., then the neural flow 240 from the proof should closely approximate the stored neural flow 140 in the measurement database 150 for the same class. However, if there is a significant deviation, e.g., a deviation greater than a given tolerance, or a deviation that meets a representation damage criterion specified, for example, in one or more computer application rules, then after the training and deployment of the DL model 110, the deployed trained DL model 210 may have been compromised in some way, either innocently or maliciously.

[0068] Using an illustrative embodiment, the deviation between the neural flows 140, 240 can be evaluated by a neural flow verification engine 138, e.g., by applying predefined computer-executed rules, comparison with one or more deviation thresholds indicating the degree of acceptable deviation, etc., to determine whether criteria or thresholds indicating potential compromise of the deployed trained DL model 210 are met, and an alert notification will be triggered to thereby notify the user of the potential compromise of the deployed model. If such a criterion or threshold is met, an alert notification is triggered and a notification and response action engine 139 is called to compose an alert and transmit the alert to the user computing device 250 or otherwise output the alert on the user computing device 250, thereby allowing the user to select whether to automatically revoke the deployed trained DL model 210. If the user selects to revoke the deployed trained DL model 210, a computer command can be transmitted to the cloud computing provider to cause the deployed trained DL model 210 to be removed from public access, e.g., to de-register from the available cloud services on the cloud computing platform.

[0069] Accordingly, mechanisms are provided for performing a runtime neural flow proof of the execution of a deployed computer model. A runtime neural flow proof involves capturing and comparing the neural flow of a computer model during a preprocessing or training phase of an operation and after deployment of the trained computer model. A runtime neural flow proof allows determination of whether a deployed model has been compromised after deployment, such that a model owner can take appropriate actions to ensure that their model does not generate outputs that are not desired by model users. Additionally, a runtime neural flow proof addresses the three problems that tenants have with respect to the DL models they deploy as described above. For example, for an attacker who injects a backdoor into a deployed trained DL model, if the attacker causes the DL model to misclassify an input, the neural flow will be modified to cause such misclassification, and this modification of the neural flow will be detected by the mechanisms of the illustrative embodiments. If a cloud computing platform provider prunes or "squeezes" a DL model to reduce storage and computing costs, the neural flow will again be modified, and this deviation from the trained DL model prior to deployment will be detected by the mechanisms of the illustrative embodiments. Additionally, if a DL model is subject to adversarial attacks, this will also result in a deviation of the neural flow, which is detected by the mechanisms of the illustrative embodiments. Thus, the claimed invention provides a mechanism for validating the integrity of a deployed trained DL model and for detection of potential compromise from an attacker or a cloud computing platform provider using a neural flow-based runtime neural flow proof model.

[0070] From the foregoing description, it is apparent that the present invention provides a computer tool for enhancing the integrity of a deployed computer model, particularly in a cloud computing or other distributed data processing system environment, and for detecting compromise of such a deployed computer model, such that appropriate notification and response actions can be taken. Thus, the illustrative embodiments can be utilized in many different types of data processing environments. To provide context for the specific elements and functions described in the illustrative embodiments, the following provides Figure 3 and Figure 4 as example environments in which aspects of the illustrative embodiments can be implemented. It should be understood that Figure 3 and Figure 4 are merely examples and are not intended to assert or imply any limitations on the environments in which aspects or embodiments of the present invention can be implemented. Many modifications can be made to the described environments without departing from the scope of the present invention.

[0071] Figure 3Depicts a graphical representation of an example distributed data processing system in which aspects of an exemplary embodiment can be implemented. The distributed data processing system 300 can include a computer network in which aspects of an exemplary embodiment can be implemented. The distributed data processing system 300 includes at least one network 302, which is a medium for providing a communication link between various devices and computers connected together within the distributed data processing system 300. The network 302 can include connections such as wired, wireless communication links, satellite communication links, fiber optic cables, and the like.

[0072] In the described example, servers 304A - 304C are connected to the network 302 along with a storage unit 308. Additionally, clients 310 and 312 are also connected to the network 302. These clients 310 and 312 can be, for example, personal computers, network computers, and the like. In the described example, the servers 304A - 304C provide data to the clients 310 - 312, such as boot files, operating system images, and applications. In the described example, the clients 310 - 312 are clients of a cloud computing system that includes the server 304A and possibly one or more other server computing devices 304B - 304C. The distributed data processing system 300 can include additional servers, clients, and other computing, data storage, and communication devices not shown.

[0073] In the described example, the distributed data processing system 300 is the Internet, where the network 302 represents a worldwide collection of networks and gateways that communicate with each other using the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol suite. The core of the Internet is a backbone of high - speed data communication lines between main nodes or main computers, which consists of thousands of commercial, government, educational, and other computer systems that route data and messages. Of course, the distributed data processing system 300 can also be implemented to include many different types of networks, such as intranets, local area networks (LANs), wide area networks (WANs), and the like. As described above, Figure 3 is intended to be an example and not an architectural limitation of different embodiments of the present invention. Therefore, Figure 3 the specific elements shown should not be considered as limitations on the environment in which the illustrative embodiments of the present invention can be implemented.

[0074] As Figure 3 shown, according to one illustrative embodiment, one or more of the computing devices, such as the server 304A, can be specifically configured to implement a deep learning cloud service platform 300, which is further configured with, for example, such as those that operate with the DL cloud service platform 300 Figure 2The runtime neural flow proof engine 230 shown. The configuration of the computing device may include providing dedicated hardware, firmware, etc. to facilitate the execution of the operations described herein with respect to the illustrative embodiments and the generation of the output. The configuration of the computing device may also or alternatively include providing a software application stored in one or more storage devices and loaded into the memory of a computing device such as server 304A for causing one or more hardware processors of the computing device to execute the software application, which should configure the processor to perform the operations and generate the output described with respect to the illustrative embodiments. Additionally, any combination of dedicated hardware, firmware, software applications executed on the hardware, etc. may be used without departing from the scope of the illustrative embodiments.

[0075] It should be understood that once the computing device is configured in one of these ways, the computing device becomes a dedicated computing device specifically configured to implement the mechanisms of the illustrative embodiments rather than a general-purpose computing device. Additionally, as described herein, the implementation of the mechanisms of the illustrative embodiments improves the functionality of the computing device and provides useful and specific results of runtime neural flow proof based on computer models of neural flows to facilitate the integrity of the deployed trained computer models.

[0076] As Figure 3 shown, one or more of servers 304A - 304C are configured to implement the deep learning cloud service 300 and the runtime neural flow proof engine 230. Although Figure 3 elements 300 and 230 are shown associated with a single server (i.e., server 304A), it should be understood that multiple servers (e.g., 304A - 304C) may together form a cloud computing system and be configured to provide the deep learning cloud service 300 and the runtime neural flow proof engine 230 such that the mechanisms of the deep learning cloud service 300 and the runtime neural flow proof engine 230 or portions thereof and the DL computing model 305 or portions thereof deployed to the DL cloud service 300 may be distributed across multiple server computing devices 304A - 304C. In some illustrative embodiments, multiple instances of the deep learning cloud service 300, the DL model 305, and the runtime neural flow proof engine 230 may be provided on multiple different servers 304A - 304C of the cloud computing system. The deep learning cloud service 300 may provide any deep learning or AI-based functionality of the deep learning system.

[0077] In some illustrative embodiments, the deep learning cloud service 300 may implement a cognitive computing system (cognitive system). As an overview, a cognitive system is a dedicated computer system or a group of computer systems that are configured with hardware and / or software logic (combined with the hardware logic on which the software executes) to emulate human cognitive functions. These cognitive systems apply human-like characteristics to convey and manipulate ideas, which, when combined with the inherent strength of digital computing, can solve problems with high accuracy and large-scale resilience. Cognitive systems perform one or more computer-implemented cognitive operations that approximate human thought processes and enable humans and machines to interact in a more natural way to extend and amplify human expertise and cognition. Cognitive systems include artificial intelligence logic, such as natural language processing (NLP)-based logic, image analysis and classification logic, electronic medical record analysis logic, etc., for example, as well as machine learning logic, which may be provided as dedicated hardware, software executed on hardware, or any combination of dedicated hardware and software executed on hardware. The logic of the cognitive system implements one or more cognitive operations, examples of which include but are not limited to question answering, identification of related concepts within different parts of the content in a corpus, image analysis and classification operations, intelligent search algorithms such as Internet web page searches (e.g., medical diagnosis and treatment recommendations), and other types of recommendation generation, such as items of interest to a particular user, potential new contact recommendations, etc. In some illustrative embodiments, the deep learning (DL) cloud service 300 may provide functionality for performing other cognitive operations such as image recognition and analysis based on the operation of the deployed trained DL model 305. Generally, the DL model 305 deployed to the DL cloud service 300 may provide classification and / or prediction functionality, and for example, the DL cloud service 300 performs further evaluation on this classification and / or prediction functionality.

[0078] IBM This is an example of such a cognitive system that can process human-readable language and identify inferences between text segments with human-like high precision at a much faster speed and larger scale than humans. Generally, such a cognitive system can perform the following functions: navigate the complexity of human language and understanding; ingest and process large amounts of structured and unstructured data; generate and evaluate hypotheses; weight and evaluate responses based only on relevant evidence; provide situation-specific advice, insights, and guidance; improve knowledge and learning through a machine learning process, leveraging each iteration and interaction; make decisions at the point of impact (contextual guidance); scale proportionally to the task; extend and amplify human expertise and cognition; identify harmonious, human-like attributes and characteristics from natural language; infer various language-specific or agnostic attributes from natural language; highly relevant recall (memory and recollection) from data points (images, text, speech); predict and sense based on experience using situation awareness that mimics human cognition; and answer questions based on natural language and specific evidence.

[0079] In an illustrative embodiment, a cognitive system that can be implemented as a deep learning cloud service 300 provides a mechanism for answering questions or processing requests from client computing devices such as client computing device 310 via one or more processing pipelines that implement one or more DL models 305. In other illustrative embodiments, the cognitive system provides image analysis and recognition capabilities based on one or more processing pipelines that implement one or more DL models 305. In either case, one or more DL models 305 perform classification or prediction operations, and the neural flow of the DL models 305 is monitored, captured, and reported to the model owner (user) to demonstrate the integrity of the deployed and trained DL models 305.

[0080] One or more processing pipelines in which the DL models 305 operate are artificial intelligence applications executed on data processing hardware that utilize results generated based on a cognitive evaluation of the input and other evidence data based on the trained DL models 305 to respond to requests, e.g., answer questions about a given subject area presented in natural language, or process requests to perform cognitive operations on input data that can be presented in natural language or as a structured request / query, analyze a given image to classify objects in the image and provide classification results, etc.

[0081] As Figure 3 shown, a client computing system, such as client computing system 310, is configured to implement such as Figure 1The illustrated neural flow proof engine 130 operates as previously described to create a neural flow model for a trained DL model 306 deployed to the DL cloud service 300, as a deployed and trained DL model 305 operating within the DL cloud service 300 to serve user input, such as input 308 from the computing device 312. The NF proof engine 130 further operates as previously described to combine the DL cloud service 300 and the deployed and trained DL model 305 based on the proof 307 received from the runtime neural flow proof engine 230 operating on the server 304A, using the created and stored NF model 306 for the trained DL model 306 to verify the integrity of the deployed and trained DL model 305. That is, the runtime neural flow proof engine 230 operates as previously described to capture the neural flow of the deployed and trained DL model 305 within a trusted execution environment and generate a proof 307 based on the captured neural flow, and the captured neural flow is transmitted to the deployed and trained DL model 305 owner computing system 310 for runtime neural flow proof of the integrity of the deployed and trained DL model 305.

[0082] In addition, the NF proof engine 130 is operable to retrieve the stored NF model (such as the NF model stored in the Figure 1 measurement database 150) for the trained DL model 306 and the classification output generated by the deployed and trained DL model 305 for the input 308 as specified in the proof 307. The NF proof engine 130 also operates to determine whether the difference between the NF in the proof 307 and the retrieved NF model indicates potential compromise of the deployed and trained DL model 305. In addition, the NF proof engine 130 may generate an alert and transmit / output the alert to authorized users to notify them of the potential compromise. In addition, the NF proof engine 130 may automatically send a command to the cloud computing service platform provider (e.g., the server 304A) in response to determining that the difference indicates potential compromise to cause the deployed and trained DL model 305 to be withdrawn or to revoke the registration of the deployed and trained DL model 305 as being accessible via the DL cloud service 300. This may be done in response to a user input requesting such withdrawal of the deployed and trained DL model 305.

[0083] As described above, the mechanisms of the illustrative embodiments utilize specially configured computing devices or data processing systems to perform operations for performing runtime neural flow proof of the integrity of deployed and trained computer models. These computing devices or data processing systems may include various hardware elements that are specially configured, by hardware configuration, software configuration, or a combination of hardware and software configurations, to implement one or more of the systems and / or subsystems described herein. Figure 4is a block diagram of only one example data processing system in which aspects of the illustrative embodiments can be implemented. Data processing system 400 is an example of a computer such as server 304A or client computing device 310 in Figure 3 wherein computer usable code or instructions for implementing the processes and aspects of the illustrative embodiments of the present invention can be located and / or executed to implement the operations, outputs, and external effects of the illustrative embodiments described herein.

[0084] In the depicted example, data processing system 400 employs a hub architecture including a north bridge and memory controller hub (NB / MCH) 402 and a south bridge and input / output (I / O) controller hub (SB / ICH) 404. Processing unit 406, main memory 408, and graphics processor 410 are connected to NB / MCH 402. Graphics processor 410 can be connected to NB / MCH 402 via an Accelerated Graphics Port (AGP).

[0085] In the depicted example, a local area network (LAN) adapter 412 is connected to SB / ICH 404. Audio adapter 416, keyboard and mouse adapter 420, modem 422, read only memory (ROM) 424, hard disk drive (HDD) 426, CD-ROM drive 430, universal serial bus (USB) ports and other communication ports 432, and PCI / PCIe devices 434 are connected to SB / ICH 404 via bus 438 and bus 440. PCI / PCIe devices can include, for example, an Ethernet adapter, add-in cards, and PC cards for notebook computers. PCI uses a card bus controller while PCIe does not. ROM 424 can be, for example, a flash basic input / output system (BIOS).

[0086] HDD 426 and CD-ROM drive 430 are connected to SB / ICH 404 via bus 440. HDD 426 and CD-ROM drive 430 can use, for example, an Integrated Drive Electronics (IDE) or Serial Advanced Technology Attachment (SATA) interface. Super I / O (SIO) device 436 can be connected to SB / ICH 404.

[0087] An operating system runs on processing unit 406. The operating system coordinates and provides control of the various components within Figure 4 data processing system 400. As a client, the operating system can be a commercially available operating system such as Windows 10. An object-oriented programming system, such as Java TM programming system, can run with the operating system and provide from Java executed on data processing system 400TM A call from a program or application to the operating system.

[0088] As a server, data processing system 400 may be, for example, a computer running Advanced Interactive Executive ) operating system or Operating System eServer TM System Computing systems, based on Data processing system 400 may be a symmetric multiprocessor (SMP) system including multiple processors in processing unit 406. Alternatively, a single processor system may be employed. IBM, IBM Watson, eServer, System p5, Power, and AIX are trademarks of International Business Machines Corporation, registered in many jurisdictions throughout the world. Registered Trademarks is used pursuant to a sublicense from The Linux Foundation, the exclusive licensee of Linus Torvalds, which owns the mark worldwide. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and / or its affiliates. Microsoft and Windows are trademarks of Microsoft Corporation in the U.S. and / or other countries.

[0089] Instructions for the operating system, object-oriented programming system, and applications or programs are located on storage devices such as HDD 426 and may be loaded into main memory 408 for execution by processing unit 406. The processes of the illustrative embodiments of the present invention may be performed by processing unit 406 using computer usable program code that may be located in a memory such as main memory 408, ROM 424, or in one or more peripheral devices 426 and 430, for example.

[0090] Such as Figure 4 A bus system such as bus 438 or bus 440 shown may include one or more buses. Of course, the bus system may be implemented using any type of communication structure or architecture that provides for data transmission between different components or devices attached to the structure or architecture. Communication units, such as Figure 4 The modem 422 or network adapter 412 may include one or more devices for transmitting and receiving data. The memory may be, for example, the main memory 408, the ROM 424, or a memory such as a Figure 4 The cache found in NB / MCH 402 in.

[0091] As described above, in some illustrative embodiments, the mechanisms of the illustrative embodiments may be implemented as dedicated hardware, firmware, etc., stored in a storage device such as HDD 426 and loaded into a memory such as main memory 408 for execution by one or more hardware processors such as processing unit 406. Thus, Figure 4 the computing device shown in becomes specifically configured to implement the mechanisms of the illustrative embodiments and is specifically configured to perform operations and generate the outputs described herein with respect to implementing a privacy-enhanced deep learning cloud service framework and a deep learning cloud service of one or more processing pipelines.

[0092] Those of ordinary skill in the art will understand that Figure 3 and Figure 4 the hardware in may vary depending on the implementation. In addition to the hardware described in Figure 3 and Figure 4 or as an alternative thereto, other internal hardware or peripheral devices may be used, such as flash memory, equivalent non-volatile memory, or optical disk drives, etc. Further, without departing from the scope of the present invention, the processing of the exemplary embodiments may be applied to a multiprocessor data processing system in addition to the SMP system mentioned above.

[0093] In addition, data processing system 400 may take the form of any of a variety of different data processing systems, including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants (PDAs), etc. In some illustrative examples, data processing system 400 may be a portable computing device configured with flash memory to provide non-volatile memory for storing, for example, operating system files and / or user-generated data. Substantially, data processing system 400 may be any known or later developed data processing system without architectural limitations.

[0094] Figure 5 is a flowchart outlining example operations for performing the functions of the preprocessing phase. As Figure 5 shown, the operation begins with training a DL model using a training data set (step 510). After the DL model is trained, the trained data is input again into the trained DL model (step 515) so that the neural flow of the trained DL model for each training data instance can be captured (step 520). Then, the captured neural flows are aggregated for each output class of the trained DL model such that the aggregation of the classes is a neural flow model for the class (step 525). Then, the neural flow model is stored in a measurement database for later use in verifying the integrity of the deployed trained DL model (step 530).

[0095] After a DL model has been trained, the neural flow of the trained DL model has been captured, and neural flow models for various classes have been generated and stored, the trained DL model is deployed to a cloud computing platform (step 535). At a later time, a user can send a request via a computing system to the cloud computing platform to verify the integrity of the deployed DL model (step 540). The request can include a challenge C that is used to seal or protect the proof returned from a runtime neural flow proof engine operating in the cloud computing platform.

[0096] In response to the request, a runtime neural flow proof engine at the cloud computing platform generates a proof, which is then received by the NF proof engine (step 545). The proof includes the neural flow captured by the runtime neural flow proof engine along with the output class generated by the deployed trained DL model for the input submitted with the request. The NF proof engine retrieves the corresponding stored neural flow model for the output class and compares the neural flow in the proof with the retrieved neural flow model for the output class (step 550). It is determined whether the comparison meets criteria indicating potential compromise of the deployed trained DL model (step 555). For example, computer-executable rules, thresholds, etc. can be applied to the differences between the neural flow in the proof and the neural flow model to determine whether the differences indicate compromise.

[0097] If the comparison yields a result of no compromise, the operation terminates. Otherwise, if the comparison yields a result of potential compromise, an alert is generated and output to an authorized user (step 560). Optionally, the deployed trained DL model can be recalled (step 565). After alerting the authorized user and requesting the authorized user to confirm the need to recall the deployed trained DL model, the revocation of the deployed trained DL model is performed. Then the operation terminates.

[0098] Figure 6 is a flowchart outlining example operations for performing deployment phase functions according to one illustrative embodiment, including runtime neural flow proof of neural flow. Figure 6 The operations outlined in assume that a trained DL model has been provided to and deployed for use on a cloud computing platform.

[0099] As Figure 6As shown, the operation begins with receiving a request to verify the execution integrity of the deployed DL model (step 610). As described above, the request may be accompanied by input data to be processed by the deployed DL model to generate an output classification, and may also include a challenge C for integrity verification and freshness testing. In some embodiments, as described above, a security key may also be exchanged for encrypting or sealing the proof generated by the trusted execution environment. In response to the request, in the trusted execution environment of the cloud computing platform, the neural flow of the deployed DL model is captured based on the given input (step 620). Based on the captured neural flow, output classification, and challenge C, a proof is generated in response to the request (step 630). Then, the proof is transmitted to the NF proof engine at the requester computing system to verify the runtime execution integrity of the deployed DL model based on the neural flow (step 640). Then the operation terminates.

[0100] Accordingly, the illustrative embodiments provide a mechanism for ensuring the integrity of a deployed trained computer model by providing a runtime neural flow proofing capability based on the neural flow within the computer model. The illustrative embodiments record the neural flow model prior to deployment of the computer model and use this as a basis for verifying the proof from the cloud computing platform after the trained computer model has been deployed as part of a cloud computing service. Thus, after a user provides a trained computer model for deployment on a cloud computing platform, the user (owner of the trained computer model) can determine whether their trained computer model has been compromised, either innocently or maliciously. As a result, the user is able to take responsive actions such as withdrawing the deployed computer model to ensure that the user does not rely on a compromised computer model.

[0101] Embodiments of the present invention can be implemented in conjunction with any type of computing environment now known or later developed. In some illustrative embodiments, the mechanisms of the illustrative embodiments are implemented on a cloud computing system; however, the implementation of the teachings recited herein is not limited to a cloud computing environment. A variety of types of distributed data processing system environments can be utilized to implement the mechanisms of the exemplary embodiments.

[0102] Assuming a cloud computing embodiment is utilized, it should be understood that cloud computing is a service delivery model for enabling convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0103] The characteristics of the cloud model are as follows:

[0104] (1) On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities, such as server time and network storage, as needed, without the need for human interaction with the service provider.

[0105] (2) Wide area network access: The capabilities are available over a network and accessed through standard mechanisms that facilitate use by heterogeneous thin and thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0106] (3) Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned according to demand. There is a location-independent aspect, as consumers generally do not control or know the exact location of the provided resources but can specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0107] (4) Rapid elasticity: In some cases, the ability to rapidly scale out and rapidly scale in can be provided quickly and elastically. To the consumer, the available capacity for provisioning generally appears unlimited and can be purchased in any quantity at any time.

[0108] (5) Measured service: The cloud system automatically controls and optimizes resource use by leveraging metering capabilities at an appropriate level of abstraction for the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource use can be monitored, controlled, and reported, providing transparency for both the provider and the consumer of the utilized service.

[0109] The service models are as follows:

[0110] (1) Software as a Service (SaaS): The capabilities provided to the consumer are to use the provider's applications running on the cloud infrastructure. The applications can be accessed from various client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even the individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0111] (2) Platform as a Service (PaaS): The capabilities provided to the consumer are to deploy the applications created or acquired by the consumer onto the cloud infrastructure, where the applications are created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage devices, but has control over the deployed applications and possibly the application hosting environment configuration.

[0112] (3) Infrastructure as a Service (IaaS): The ability provided to the consumer is to offer processing, storage devices, networks, and other basic computing resources where the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating system, storage devices, deployed applications, and possibly limited control over selected networking components (e.g., host firewall).

[0113] The deployment models are as follows:

[0114] (1) Private cloud: The cloud infrastructure is operated only for an organization. It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0115] (2) Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., missions, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0116] (3) Public cloud: The cloud infrastructure is available for the general public or large industrial groups and is owned by the organization selling the cloud services.

[0117] (4) Hybrid cloud: The cloud infrastructure is a combination of two or more clouds (private, community, or public), where the clouds remain distinct entities but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0118] The cloud computing environment is service-oriented, with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure of networks that includes interconnected nodes.

[0119] Now referring to Figure 7 , an illustrative cloud computing environment 750 is depicted. As shown, the cloud computing environment 750 includes one or more cloud computing nodes 710 with which local computing devices used by cloud consumers can communicate, such as personal digital assistants (PDAs) or cellular phones 754A, desktop computers 754B, laptop computers 754C, and / or in-vehicle computer systems 754N. The nodes 710 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud or a combination thereof as described above. This allows the cloud computing environment 750 to provide infrastructure, platform, and / or software as a service, for which cloud consumers do not need to maintain resources on local computing devices. It should be understood that Figure 7The type of computing device 754A-N shown is merely illustrative, and the computing nodes 710 and the cloud computing environment 750 can communicate with any type of computerized device via any type of network and / or network addressable connection (e.g., using a web browser).

[0120] Now referring to Figure 8 , a set of functional abstraction layers provided by the cloud computing environment 750 ( Figure 7 ) is shown. It should be understood in advance that Figure 8 the components, layers, and functions shown in

[0121] are only illustrative, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0122] (1) The hardware and software layer 860 includes hardware and software components. Examples of hardware components include: host 861; servers 862 based on RISC (Reduced Instruction Set Computer) architecture; server 863; blade server 864; storage device 865; and network and network components 866. In some embodiments, the software components include network application server software 867 and database software 868.

[0123] (2) The virtualization layer 870 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 871; virtual storage device 872; virtual network 873, including virtual private networks; virtual applications and operating systems 874; and virtual clients 875.

[0124] The workload layer 890 provides examples of workloads that can utilize the capabilities of a cloud computing environment. Examples of workloads and capabilities that can be provided from this layer include: mapping and navigation 891; software development and lifecycle management 892; virtual classroom education delivery 893; data analysis processing 894; transaction processing 895; and deep learning cloud service processing 896. Deep learning cloud service processing 896 can include a pipeline, computer models (e.g., DL models), and a runtime neural flow proof engine mechanism, such as the runtime neural flow proof engine 230 described above in one or more of the illustrative embodiments described Figure 2 above in the runtime neural flow proof engine 230 of the

[0125] As described above, it should be understood that the illustrative embodiments can take the form of a full hardware embodiment, a full software embodiment, or an embodiment that includes both hardware and software elements. In one example embodiment, the mechanisms of the illustrative embodiments are implemented in software or program code, which includes but is not limited to firmware, resident software, microcode, etc.

[0126] A data processing system suitable for storing and / or executing program code will include at least one processor that is directly or indirectly coupled to a memory element via a communication bus such as a system bus. The memory elements can include local memory, mass storage devices, and cache memory employed during the actual execution of the program code, and the cache memory provides at least some temporary storage of the program code to reduce the number of times the code must be retrieved from the mass storage device during execution. The memory can be of various types, including but not limited to ROM, PROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid state memory, etc.

[0127] Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system directly or via an intervening wired or wireless I / O interface and / or controller, etc. The I / O devices can take many different forms other than conventional keyboards, displays, pointing devices, etc., such as communication devices coupled via a wired or wireless connection, including but not limited to smart phones, tablet computers, touch screen devices, voice recognition devices, etc. Any known or later developed I / O device is intended to be within the scope of the illustrative embodiments.

[0128] A network adapter can also be coupled to the system to enable the data processing system to be coupled to other data processing systems or remote printers or storage devices via an intermediate private or public network. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters for wired communication. Network adapters based on wireless communication can also be utilized, including but not limited to 802.11a / b / g / n wireless communication adapters, Bluetooth wireless adapters, and the like. Any known or later-developed network adapter is intended to be within the scope of the present invention.

[0129] The description of the present invention has been presented for purposes of illustration and description, and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The embodiments were chosen and described in order to best explain the principles of the invention, the practical application, and to enable others of ordinary skill in the art to understand the invention with various modifications suited to the particular use contemplated. The terms used herein were chosen to best explain the principles of the embodiments, the practical application, or the technical improvement present in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method in a data processing system including at least one processor and at least one memory, the at least one memory including instructions that are executed by the at least one processor to configure the at least one processor to implement a neural flow proof engine, the method including: The input data is input into a trained computer model by the neural flow proof engine, where the trained computer model includes multiple layers of neurons; A trusted execution environment TEE is generated in the data processing system, and the TEE includes a neural flow recording component and a proof generation engine; The neural flow recording component records, for an input data instance set in the input data, the output class generated by the trained computer model and the neural flow through the multiple layers of neurons, thereby generating the recorded neural flow, where the output class is one of multiple possible output classes; The proof generation engine generates a proof, and the proof includes the packaged recorded neural flow; The trained computer model is deployed to a computing platform; and The neural flow proof engine verifies the execution integrity of the deployed trained computer model based on the runtime neural flow of the deployed trained computer model and the packaged recorded neural flow in the proof, where the verification includes: Determining a deviation between the runtime neural flow and one or more recorded neural flows, where the one or more recorded neural flows correspond to the same output class as that generated by the deployed trained computing model for runtime input data; and In response to the deviation satisfying a predetermined criterion, determining that the execution integrity of the deployed trained computer model has been compromised.

2. The method according to claim 1, further including: The neural flow proof engine generates a neural flow model for each of the multiple possible output classes based on the recorded neural flows; and The neural flow proof engine stores each neural flow model in a measurement database, where verifying the execution integrity of the deployed trained computer model further includes comparing the runtime neural flow with one or more of the stored neural flow models in the measurement database.

3. The method according to claim 2, wherein, Generating a neural flow model for each of the multiple possible output classes based on the recorded neural flows includes, for each class, aggregating the recorded neural flows associated with that class by at least one of the following: identifying a set of activated neurons and filters, or training a machine learning model based on the activated neurons and filters considering activation and the frequency of activation occurrence.

4. The method according to any one of claims 2 or 3, wherein, The measurement database is stored on a tenant computing device associated with the provider of the trained computer model, and the data processing system is a separate computing system of a cloud computing platform.

5. The method according to claim 4, wherein verifying the execution integrity of the deployed trained computer model further includes: The runtime neural flow of the deployed trained computer model is recorded based on the runtime input data processed by the deployed trained computer model; The recorded runtime neural flow is transmitted from the data processing system to the tenant computing device; and The recorded runtime neural flow is compared with one or more of the stored neural flow models in the measurement database, where the one or more stored neural flow models correspond to the same class as the runtime output class generated by the deployed trained computer model for the runtime input data.

6. The method according to claim 1, further including: In response to determining that the execution integrity of the deployed trained computer model has been compromised, the deployed trained computer model is withdrawn from further access by the users of the deployed trained computer model.

7. The method according to claim 1, wherein, For the set of input data instances in the input data, recording the output classes generated by the trained computer model and the neural flow through the multi-layer neurons includes: recording, by a neural flow recording component executing within the trusted execution environment, the neural flow through the multi-layer neurons for the input data instance; and generating, by a proof generation engine executing within the trusted execution environment, the proof, the proof including the neural flow through the multi-layer neurons recorded for the input data instance and the output classes generated by the deployed trained computer model for the input data instance.

8. The method according to claim 7, wherein, Generating the proof by the proof generation engine further includes: generating, by the proof generation engine, the proof further based on a challenge provided by a computing device of a provider of the computer model; encrypting, by the proof generation engine, the proof based on a security key exchanged between the computing device of the provider of the computer model and the TEE, thereby generating an encrypted proof; and transmitting, by the data processing system, the encrypted proof to the computing device of the provider of the computer model.

9. The method according to claim 1, wherein, The computing platform is a cloud computing platform having one or more tenants.

10. A data processing system, comprising: At least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory includes instructions that, when executed by the at least one processor, cause the at least one processor to implement a neural flow proof engine, the neural flow proof engine being configured to: input input data into a trained computer model, wherein the trained computer model includes multi-layer neurons; generate, in the data processing system, a trusted execution environment (TEE), the TEE including a neural flow recording component and a proof generation engine; recording, by the neural flow recording component, for a set of input data instances in the input data, the output classes generated by the trained computer model and the neural flow through the multi-layer neurons, thereby generating the recorded neural flow, wherein the output class is one of a plurality of possible output classes; generating, by the proof generation engine, a proof, the proof including the recorded neural flow that is packaged; deploying the trained computer model to a computing platform; and verifying the execution integrity of the deployed trained computer model based on the runtime neural flow of the deployed trained computer model and the recorded neural flow that is packaged in the proof, wherein the verification includes: determining a deviation between the runtime neural flow and one or more recorded neural flows, the one or more recorded neural flows corresponding to the same output class generated by the deployed trained computing model for runtime input data; and in response to the deviation satisfying a predetermined criterion, determining that the execution integrity of the deployed trained computer model has been compromised.

11. The data processing system according to claim 10, wherein, The neural flow proof engine is configured to: For each of the multiple possible output classes, generate a neural flow model based on the recorded neural flows; and Store each neural flow model in a measurement database, wherein verifying the execution integrity of the deployed trained computer model further includes comparing the runtime neural flow with one or more of the stored neural flow models in the measurement database.

12. The data processing system according to any one of claims 10 or 11, wherein, The neural flow proof engine being configured to generate a neural flow model for each of the multiple possible output classes based on the recorded neural flows includes, for each class, aggregating the recorded neural flows associated with that class by at least one of: identifying a set of activated neurons and filters, or training a machine learning model based on the activated neurons and filters considering the activation and the frequency of activation occurrence.

13. The data processing system according to claim 11, wherein the measurement database is stored on a tenant computing device associated with a provider of the trained computer model, and wherein the data processing system is a separate computing system of a cloud computing platform.

14. The data processing system according to claim 13, wherein, The neural flow proof engine is further configured to verify the execution integrity of the deployed trained computer model by: Recording the runtime neural flow of the deployed trained computer model based on the runtime input data processed by the deployed trained computer model; Transmitting the recorded runtime neural flow from the data processing system to the tenant computing device; And Comparing the recorded runtime neural flow with one or more stored neural flow models in the measurement database, the one or more stored neural flow models corresponding to the same class as the runtime output class generated by the deployed trained computer model for the runtime input data.

15. The data processing system according to claim 10, wherein, The neural flow proof engine is operable to further perform the following operation: in response to determining that the execution integrity of the deployed trained computer model has been compromised, revoke the deployed trained computer model from further access by the user of the deployed trained computer model.

16. The data processing system according to any one of claims 10 or 11, wherein: Recording the output class generated by the trained computer model and the neural flow through the multi-layer neurons for the set of input data instances in the input data includes: recording, by a neural flow recording component executing within the trusted execution environment, the neural flow through the multi-layer neurons for the input data instance; and Generating, by a proof generation engine executing within the trusted execution environment, the proof, the proof including the neural flow through the multi-layer neurons recorded for the input data instance and the output class generated by the deployed trained computer model for the input data instance.

17. The data processing system according to claim 16, wherein, Generating the proof by the proof generation engine further includes: The proof generation engine further generating the proof based on a challenge provided by a computing device of the provider of the computer model; The proof generation engine encrypting the proof based on a security key exchanged between the computing device of the provider of the computer model and the TEE, thereby generating an encrypted proof; and The data processing system transmitting the encrypted proof to the computing device of the provider of the computer model.

18. A computer program product for implementing a neural flow proof engine, the computer program product comprising: A computer-readable storage medium that can be read by a processing circuit and stores instructions for execution by the processing circuit to perform the method according to any one of claims 1 to 9.

19. A computer-readable medium comprising a computer program which, when run on a computer, is used to execute the method according to any one of claims 1 to 9.

Images