Secure artificial intelligence model training and registration system

Abstract

In general, the present disclosure describes a system for securely training an artificial intelligence (AI) model. The system can include a communication circuitry to receive a dataset, a machine learning algorithm, and an AI model from a provider. The dataset, the machine learning algorithm, and the AI model can be placed in a secure sandbox for user access. A user can securely train the AI model using the dataset in the secure sandbox. The secure sandbox can record AI model metadata associated with transactions involving the AI model and send the AI model metadata to a model registry. The model registry can attest to a lineage of the AI model.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 936,023, filed on November 15, 2019, which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure relates to computer networks, and more particularly to a secure artificial intelligence model training and registration system. Background Art

[0003] Cloud computing refers to the use of dynamically scalable computing resources accessible via a network such as the Internet. The computing resources (often referred to as the "cloud") provide one or more services to users. These services can be categorized by service type, which can include, for example, applications / software, platforms, infrastructure, virtualization, and servers and data storage. The name of the service type is often preceded by the phrase "as a service," so that, for example, the delivery of applications / software and infrastructure may be referred to as software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS), respectively.

[0004] The term "cloud-based services," or simply "cloud services," refers not only to services provided by the cloud, but also to a form of service provision in which a cloud customer contracts with a cloud service provider for the online delivery of cloud-provided services. The cloud service provider manages a public, private, or hybrid cloud to facilitate the online delivery of cloud services to one or more cloud customers.

[0005] Artificial intelligence (AI) is a field of computer science focused on computer systems capable of performing tasks that typically require human thought. Various machine learning computer algorithms are used to process training datasets to train AI models, which consist of parameters that determine the AI ​​algorithm's operation in inference mode. Further training on different training datasets allows the AI ​​model to be further refined or customized. Summary of the Invention

[0006] In general, the present disclosure describes a neutral artificial intelligence (AI) exchange system for securely creating, training, tracking, and providing AI models in a secure environment. The exchange system can be a private or standalone system available for use by coalition members seeking access to AI algorithms or others’ data to develop better AI models or to utilize AI models developed by others. The exchange system can provide a secure environment to coalition members, such as data scientists, for creating and training AI models in a secure manner, enabling future potential users of the trained AI models to verify the provenance information and characteristics of the trained AI models in the secure environment, e.g., without having to train the AI models themselves. In some instances, the exchange system can also permit coalition members to import existing AI models. The exchange system can track and securely store information related to the creation, training, or import of AI models therein, such as AI metadata. For example, the exchange system can track and store information about where an AI model was built, who built the AI model, one or more data sources used to train the AI model, and other information. The exchange system can create a hash of the AI model and can package the hash, the AI model, and the AI metadata in a container. The coalition members can use the hash to verify and prove the lineage or provenance information of a particular AI model contained in the container. If the AI model is further trained, the exchange system can continue to track and securely store AI metadata related to the further training. The exchange system can create a new hash and package the new hash, the further trained AI model, and the updated AI metadata in a new container. The exchange system can provide a library of containers of AI models created, trained, or imported in the exchange system to the coalition members.

[0007] The various techniques disclosed herein provide one or more technical advantages that can enable at least one practical application. For example, by tracking and storing AI metadata related to the creation, training, and importation of AI models, an exchange system allows users to make informed decisions about whether AI models registered with the exchange system can provide better results. For example, if a user wants to use an AI model to predict traffic patterns in an urban environment, an AI model trained on traffic data from a rural environment may not be as useful as an AI model trained on traffic data from another urban environment. By identifying the data used to train the AI ​​model, the system can provide users with important information that enables them to select a better AI model for their purpose. In another example, a user may prefer to use an AI model built by a company or individual with a track record of creating useful AI models, rather than using an unknown company or individual, or an imported company or individual with unknown pedigree. Therefore, by identifying the builder of an AI model, the system can help users select a preferred AI model. Additionally, by packaging the hash, AI model, and AI metadata in a container, the exchange system can verify the AI ​​model for the intended user by confirming that the AI ​​model was trained as indicated by the AI ​​metadata.

[0008] In one example, a system includes a memory; and a processing circuit system coupled to the memory, the processing circuit system being operable to: build an artificial intelligence (AI) model based on input from a user; train the AI ​​model based on input from the user; create first AI model metadata based on a first transaction associated with building and training the trained AI model; calculate a first hash based at least in part on the trained AI model; package the trained AI model, the first AI model metadata, and the first hash in a first container; register the first container; provide the user with secure access to the first container; and verify the trained AI model for the user based on the first hash.

[0009] In one example, a system includes a communication circuit system operable to receive a data set, a machine learning algorithm, and an AI model from a provider; a security sandbox coupled to the communication circuit system, the security sandbox operable to train the AI ​​model and record AI model metadata associated with the trained AI model; and a memory operable to store the data set, the machine learning algorithm, the AI ​​model, and the AI ​​model metadata.

[0010] In one example, a method includes: building, by a system, an AI model based on input from a user; training, by the system, the AI ​​model based on input from the user; creating, by the system, first AI model metadata based on a first transaction associated with building and training the trained AI model; calculating, by the system, a first hash based at least in part on the trained AI model; packaging the trained AI model, the first AI model metadata, and the first hash in a first container; registering, by the system, the first container; providing, by the system, secure access to the first container to the user; and verifying, by the system, the trained AI model for the user based on the first hash.

[0011] In one example, a method includes: receiving, by a system, a dataset, a machine learning algorithm, and an AI model from a provider; training, by the system, at least one of the AI ​​models based on at least one of the datasets; recording, by the system, AI model metadata associated with training the at least one AI model; and storing, by the system, the dataset, the machine learning algorithm, the AI ​​model, and the AI ​​model metadata.

[0012] In one example, a non-transitory computer-readable medium has instructions stored thereon that, when executed, cause one or more processors to: build an AI model based on input from a user; train the AI ​​model based on input from the user; create first AI model metadata based on a first transaction associated with building and training the trained AI model; calculate a first hash based at least in part on the trained AI model; package the trained AI model, the first AI model metadata, and the first hash in a first container; register the first container; provide the user with secure access to the first container; and verify the trained AI model for the user based on the first hash.

[0013] In one example, a non-transitory computer-readable medium stores instructions thereon that, when executed, cause one or more processors to receive a dataset, a machine learning algorithm, and an AI model from a provider; train at least one of the AI ​​models based on at least one of the datasets; record AI model metadata associated with training the at least one AI model; and store the dataset, the machine learning algorithm, the AI ​​model, and the AI ​​model metadata.

[0014] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram illustrating a conceptual diagram of a network system with a metro-based cloud exchange providing multiple cloud exchange points according to the techniques described herein.

[0016] Figure 2 is a block diagram illustrating a high-level view of a data center that provides an operating environment for cloud-based service exchange according to the techniques described herein.

[0017] Figures 3A to 3B is a block diagram illustrating an example network infrastructure and services provided by a programmable network platform for a cloud exchange that aggregates cloud services of multiple cloud service providers to provide to customers of the cloud exchange provider and aggregates access of multiple customers to one or more cloud service providers in accordance with the techniques described in this disclosure.

[0018] Figure 4 is a block diagram illustrating additional details of one example of a computing system operating in accordance with one or more techniques of this disclosure.

[0019] Figure 5 is a block diagram illustrating an example switching system in accordance with techniques of this disclosure.

[0020] Figure 6 is a block diagram depicting an example pipeline update process according to techniques of this disclosure.

[0021] Figure 7 is a block diagram illustrating an example AI model container according to techniques of this disclosure.

[0022] Figure 8 is a block diagram illustrating security aspects of a security sandbox according to techniques of this disclosure.

[0023] Figure 9 is a conceptual diagram of an example model registry update and example model usage process according to the techniques of this disclosure.

[0024] Figure 10 is a block diagram of an example model registry in accordance with the techniques of this disclosure.

[0025] Figure 11 is a conceptual diagram illustrating a functional overview of an example system in accordance with the techniques of this disclosure.

[0026] Figure 12 is a conceptual diagram illustrating the control flow of an example system in accordance with the techniques of this disclosure.

[0027] Figure 13 is a block diagram illustrating the architecture of an example system in accordance with the techniques of this disclosure.

[0028] Figure 14 is a flow chart illustrating an example AI exchange technique according to the present disclosure.

[0029] Figure 15 is a flow diagram illustrating an example security sandbox technique according to the present disclosure.

[0030] Like reference numerals refer to like elements throughout the drawings and text. DETAILED DESCRIPTION

[0031] The usefulness of an AI model may be readily available to users, but it can be difficult for them to assess. For example, the AI ​​model may have been trained on a dataset unknown to the user. The user may also be unaware of the creator of the AI ​​model. This can make it difficult for users to select the appropriate AI model for their purposes. Using an inappropriate AI model can lead to suboptimal results.

[0032] This disclosure describes a system and method for securely exchanging datasets, machine learning algorithms, and AI models. AI model metadata associated with transactions involving AI models can be recorded. The AI ​​model metadata can be provided to a model registry. The model registry can be used to prove the lineage of AI models within the exchange system. The disclosed techniques can provide users with information that may be helpful in selecting appropriate AI models.

[0033] Figure 1 A conceptual diagram of a network system with a metro-based cloud exchange providing multiple cloud exchange points, according to the techniques described herein, is illustrated. Each cloud exchange cloud exchange cloud exchange point 128A-128C (hereinafter described as a "cloud exchange point" and collectively referred to as "cloud exchange points 128") of a cloud-based service exchange 100 ("cloud exchange 100") can represent a different data center geographically located within the same metropolitan area ("metro-based," e.g., New York City, New York; Silicon Valley, California; Seattle-Tacoma, Washington; Minneapolis-St. Paul, Minnesota; London, United Kingdom, etc.) to provide a resilient and independent cloud-based service exchange through which cloud-based service customers ("cloud customers") and cloud-based service providers ("cloud providers") connect to receive and provide cloud services, respectively. In various examples, cloud exchange 100 can include more or fewer cloud exchange points 128. In some instances, cloud exchange 100 includes only one cloud exchange point 128. As used herein, references to a "cloud exchange" or "cloud-based service exchange" may refer to a cloud exchange point. A cloud exchange provider may deploy instances of cloud exchange 100 in multiple different metropolitan areas, each with one or more cloud exchange points 128.

[0034] Each cloud exchange point 128 comprises a network infrastructure and operating environment through which cloud customers 108A-108C (collectively, "cloud customers 108") receive cloud services 110 from multiple cloud service providers 110A-110N (collectively, "cloud service providers"). Cloud exchange 100 provides customers of the exchange (e.g., enterprises, network operators, network service providers, and SaaS customers) with secure, private virtual connections to multiple cloud service providers worldwide. Multiple cloud service providers participate in the cloud exchange by virtue of having at least one accessible port in the cloud exchange, through which customers can each connect to one or more cloud services offered by the cloud service provider. Cloud exchange 100 allows any customer's private network to be directly cross-connected to any other customer at a common point, thereby enabling the direct exchange of network traffic between customers' networks.

[0035] Cloud customers 108 may receive cloud-based services directly via layer 3 peering and physical connectivity with one of the cloud exchange points 128 or indirectly via one of the network service providers 106A-106B (collectively, “NSPs 106” or alternatively, “operators 106”). NSPs 106 provide “cloud transit” by maintaining a physical presence within one or more of the cloud exchange points 128 or aggregating layer 3 access from one or more customers 108. NSPs 106 may peer directly with one or more cloud exchange points 128 at layer 3 and, in doing so, provide one or more customers 108 with indirect layer 3 connectivity and peering through which customers 108 may obtain cloud services from cloud exchange 100. Figure 1In the example of FIG. 1 , each of the cloud exchange points 128 is assigned a different autonomous system number (ASN). For example, cloud exchange point 128A is assigned ASN1, cloud exchange point 128B is assigned ASN2, and so on. Thus, each cloud exchange point 128 is the next hop in a vector routing protocol (e.g., BGP) path from cloud service provider 110 to customer 108. Consequently, while each cloud exchange point 128 is not a transit network with one or more WAN links and accompanying internet access and transport policies, it peers with multiple different autonomous systems via external BGP (eBGP) or other exterior gateway routing protocols to exchange, aggregate, and route service traffic from one or more cloud service providers 110 to customers. In other words, cloud exchange points 128 can internalize the eBGP peering relationships that cloud service providers 110 and customers 108 may maintain on a pairwise basis. Instead, the customer 108 can configure a single eBGP peering relationship with the cloud exchange point 128 and receive multiple cloud services via the cloud exchange from one or more cloud service providers 110. Although primarily described herein with respect to eBGP or other layer 3 routing protocol peering between a cloud exchange point and customer, NSP, or cloud service provider networks, the cloud exchange point can learn routes from these networks through other means, such as through static configuration or via Routing Information Protocol (RIP), Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), or other route distribution protocols.

[0036] As an example above, customer 108C is illustrated as having contracted with the cloud exchange provider for cloud exchange 100 to directly access layer 3 cloud services via cloud exchange point 128C. Thus, for example, customer 108C receives redundant layer 3 connectivity with cloud service provider 110A. Customer 108B is illustrated as having contracted with multiple NSPs 106A, 106B to have redundant cloud access to cloud exchange points 128A, 128B via the respective transit networks of NSPs 106A, 106B. The contracts described above are instantiated in the network infrastructure of cloud exchange point 128 through L3 peering configuration within the switching equipment of NSP 106 and cloud exchange point 128 and L3 connectivity (e.g., layer 3 virtual circuits established within cloud exchange point 128 to interconnect the cloud service provider 110 network to the NSP 106 network and the customer 108 network), with all contracts having at least one port that provides connectivity within one or more of cloud exchange points 128.

[0037] In some examples, cloud exchange 100 allows customers 108A, 108B of any network service provider (NSP) or "carrier" 106A-106B (collectively, "carrier 106"), or corresponding ones of other cloud customers including customer 108C, to connect directly to any other customer network and / or any of the cloud service providers 110 via virtual layer 2 (L2) or layer 3 (L3) connections, thereby allowing network traffic to be exchanged directly between the customer networks and the cloud service providers 110. The virtual L2 or L3 connections may be referred to as "virtual circuits."

[0038] Operators 106 may each represent a network service provider associated with a transit network, and network subscribers of operator 106 may access cloud services provided by network service providers 110 via cloud exchange 100. Generally speaking, customers of cloud service providers 110 may include network operators, large enterprises, managed service providers (MSPs), and software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), virtualization as a service (VaaS), and data storage as a service (dSaaS) customers of such cloud-based services provided by cloud service providers 110 via cloud exchange 100.

[0039] In this way, cloud exchange 100 streamlines and simplifies the process of enabling cloud service providers 110 and customers (either via operators 106 or directly) to collaborate in a transparent and neutral manner. One example application of cloud exchange 100 is co-locating and interconnecting data centers, where cloud service providers 110 and operators 106 and / or customers 108 may already have a network presence, such as by having one or more accessible ports available for interconnection within the data center, which may represent any of cloud exchange points 128. This allows participating operators, customers, and cloud service providers to have a wide range of interconnection options within the same facility. Thus, operators / customers can have the option of establishing many-to-many interconnections, connecting only once to one or more cloud exchange points 128. In other words, instead of establishing separate connections across transit networks to access different cloud service providers or different cloud services of one or more cloud service providers, cloud exchange 100 allows customers to interconnect to multiple cloud service providers and cloud services.

[0040] The cloud exchange 100 can include an AI exchange 500 for enabling secure exchange of data sets, machine learning applications, and AI models between data set, machine learning algorithm, and AI model providers, such as the cloud service providers 110A-110N, and customers, such as the customers 108A-108C. The AI exchange 500 can also enable data set, machine learning algorithm, and AI model providers, such as the cloud service providers 110A-110N, and customers, such as the customers 108A-108C, to securely build, train, and import AI models. The cloud exchange 100 can also include a model registry 508. Although illustrated as directly coupled to the AI exchange 500, the model registry 508 can be indirectly coupled to the AI exchange 500. For example, the model registry 508 can be coupled to the AI exchange 500 through the cloud exchange point 128C. The model registry 508 can register AI models built, trained, or imported into the AI exchange 500 and enable the lineage of the AI models to be proven, as discussed in detail below. In the example of an AI model being imported into the AI exchange 500, the model registry 508 can only enable the enhancements, such as training, that occur within the AI exchange 500 to be proven. In some examples, the model registry 508 can be located in a different cloud exchange than the AI exchange 500.

[0041] The AI exchange 500 and the model registry 508 can represent one or more applications executing within one or more data centers of the cloud exchange 100 or alternatively behind the scenes or branches of a cloud provider, enterprise, consortium, or other computing infrastructure, such as Amazon®. The AI exchange 500 and the model registry 508 can be distributed in whole or in part across data centers, each data center associated with a different cloud exchange point 128 to make up the cloud exchange 100. Although illustrated as residing in a single cloud exchange 100, the AI exchange 500 and the model registry 508 can reside in multiple cloud exchanges.

[0042] The cloud exchange 100 can also include a programmable network platform 120 for dynamically programming the cloud exchange 100 to respond and reliably meet service requests encapsulating business demands for services provided by the cloud exchange 100 and / or cloud service providers 110 coupled to the cloud exchange. As a result, the programmable network platform 120 can orchestrate business-level services across heterogeneous cloud service providers according to well-defined service policies, quality-of-service policies, service-level agreements, and costs and also according to service topologies of the business-level services.

[0043] The programmable network platform 120 enables a cloud service provider implementing the cloud exchange 100 to dynamically configure and manage the cloud exchange 100 to, for example, facilitate virtual connections for cloud-based service delivery from multiple cloud service providers 110 to one or more cloud customers 108. The cloud exchange 100 can enable cloud customers 108 to connect directly to cloud service providers 110, bypassing the public Internet, in order to improve performance, reduce costs, increase security and privacy of connections, and leverage cloud computing for other applications. Thus, for example, in some aspects, enterprises, network operators, and SaaS customers can integrate cloud services with their internal applications as if these services were part of or directly coupled to their own data center networks.

[0044] In other examples, the programmable network platform 120 enables a cloud service provider to configure the cloud exchange 100 using an L3 instance requested by a cloud customer 108, as described herein. For example, a customer 108 may request an L3 instance through which to link multiple cloud service providers (e.g., to transfer the customer's data between two cloud service providers or to obtain a service mesh from multiple cloud service providers). In some examples, the programmable network platform 120 may implement the AI ​​exchange 500, the model registry 508, or both.

[0045] Programmable network platform 120 can represent applications that execute within one or more data centers of cloud exchange 100, or alternatively, externally at a backend or branch office of a cloud service provider (e.g., a cloud service provider). Programmable network platform 120 can be distributed in whole or in part among data centers, with each data center associated with a different cloud exchange point 128 to constitute cloud exchange 100. Although shown as implementing a single cloud exchange 100, programmable network platform 120 can control service provisioning for multiple different cloud exchanges. Alternatively or additionally, multiple separate instances of programmable network platform 120 can control service provisioning for a corresponding multiple different cloud exchanges.

[0046] In the illustrated example, programmable network platform 120 includes a service interface (or "service API") 114 that defines methods, fields, and / or other software primitives through which applications 130 (such as a customer portal) can invoke programmable network platform 120. Service interface 114 can allow operators 106, customers 108, cloud service providers 110, and / or cloud exchange providers to programmatically access the capabilities and assets of cloud exchange 100 in accordance with the techniques described herein.

[0047] For example, the service interface 114 can facilitate machine-to-machine communication to enable dynamic provisioning of virtual circuits in a cloud exchange for interconnecting customer and / or cloud service provider networks. Thus, the programmable network platform 120 enables automation of various aspects of cloud service provisioning. For example, the service interface 114 can provide a customer with an automated and seamless way to establish, tear down, and manage interconnections between multiple different cloud providers participating in a cloud exchange.

[0048] More example details of cloud-based service exchanges can be found in: U.S. patent application Ser. No. 15 / 099,407, filed on April 14, 2016, entitled “CLOUD-BASED SERVICES EXCHANGE”; U.S. patent application Ser. No. 14 / 927,451, filed on October 29, 2015, entitled “INTERCONNECTION PLATFORM FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF A CLOUD-BASED SERVICES EXCHANGE”; and U.S. patent application Ser. No. 14 / 927,306, filed on October 29, 2015, entitled “ORCHESTRATION ENGINE FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF INTERCONNECTIONS WITHIN A CLOUD-BASED SERVICES EXCHANGE,” the entire contents of each of which are incorporated herein by reference.

[0049] Figure 2 1 is a block diagram illustrating a high-level view of a data center 201 that provides an operating environment for a cloud-based services exchange 200 in accordance with the techniques described herein. Cloud-based services exchange 200 ("cloud exchange 200") allows customer networks 204D, 204E of any NSP 106A-106C and corresponding networks in NSP networks 204A-204C (collectively, "'private' or 'carrier' networks 204"), or other cloud customers including customers 108A, 108B, to connect directly to any other customer network and / or any cloud service provider 110A-110N via layer 3 (L3) or layer 2 (L2) connections, thereby enabling the exchange of cloud service traffic between customer networks and / or cloud service providers 110. Data center 201 may be located entirely within a centralized area, such as a warehouse or local data center complex, and provide power, cabling, security, and other services to NSPs, customers, and cloud service providers that locate their respective networks within data center 201 (e.g., for co-location) and / or connect to data center 201 via one or more external links.

[0050] The network service providers 106 can each represent a network service provider associated with a transit network through which network subscribers of the NSP 106 can access cloud services provided by the cloud service providers 110 via the cloud exchange 200. The cloud service providers 110 can include network operators, large enterprises, managed service providers (MSPs), and software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), virtualization as a service (VaaS), and data storage as a service (dSaaS) customers for such cloud-based services as provided by the cloud service providers 110 via the cloud exchange 200.

[0051] As such, the cloud exchange 200 streamlines and simplifies the process for cooperating cloud service providers 110 and customers 108 (either indirectly via the NSP 106 or directly) in a transparent and neutral manner. One example application of the cloud exchange 200 is in co-located and interconnected data centers in which the cloud service providers 110, NSPs 106, and / or customers 108 can already have a network presence, such as by having one or more accessible ports available for interconnection within the data center. This allows participating carriers, customers, and cloud service providers to have a wide range of interconnection options in the same facility.

[0052] The cloud exchange 200 of the data center 201 includes a network infrastructure 222 providing an L2 / L3 switching fabric through which the cloud service providers 110 and customers and / or NSPs interconnect. This enables customers and / or NSPs to have the option of creating a many-to-many interconnection to the exchange network and presenting the underlying network infrastructure 222 for the interconnection platform of the cloud exchange 200 only once. In other words, instead of having to establish separate connections across a transit network to access different cloud service providers or different cloud services of one or more cloud service providers 110, the cloud exchange 200 allows customers to interconnect to multiple cloud service providers and cloud services using the network infrastructure 222 within the data center, which can represent at least in part any of the edge networks described in this disclosure.

[0053] By using the cloud exchange 200, customers can purchase services and reach many end users in many different geographic regions without incurring the same fees typically associated with installing and maintaining multiple virtual connections to multiple cloud service providers 110. For example, the NSP 106A can extend services using the network 204B of the NSP 106B. By connecting to the cloud exchange 200, the NSP 106 can be able to generate additional revenue by offering other carriers the opportunity to sell network services using the NSP network 204C. For example, the NSP 106C can offer other NSPs the opportunity to use the NSP network 204C.

[0054] Cloud exchange 200 may include an AI exchange 500 and a model registry 508. In this example, AI exchange 500 is indirectly coupled to model registry 508 via network infrastructure 222. Cloud exchange 200 may also include a programmable network platform (not shown for simplicity), such as Figure 1 Programmable network platform 120.

[0055] Providers of data sets, machine learning algorithms, and AI models (collectively, “data”), such as cloud service providers 110A through 110N, can access the AI ​​exchange 500 in the cloud exchange 200 through the network infrastructure 222 to place their data on the AI ​​exchange 500. Customers, such as customers 108A and 108B, can access the AI ​​exchange through the network infrastructure 222 to purchase access to the data.

[0056] exist Figure 2 In the example shown in FIG, network infrastructure 222 represents a cloud exchange fabric and includes a plurality of ports that can be dynamically interconnected with virtual circuits by, for example, invoking a service interface of a programmable network platform (not shown). Each of the ports can be associated with one of: an operator 106, a customer 108, and a cloud service provider 110.

[0057] In some examples, a cloud exchange vendor (e.g., an enterprise or a cloud service provider nested within a cloud service provider) can request and obtain an L3 instance, can then create a vendor profile associated with the L3 instance, and subsequently operate as a vendor on the cloud exchange. The techniques of this disclosure enable multiple cloud service providers to participate in an enterprise's L3 instance (e.g., an L3 "routing instance" or an L2 "bridging instance"), while each cloud service provider flow is anchored using enterprise equipment.

[0058] Figures 3A to 3Bis a block diagram illustrating an example network infrastructure for a cloud exchange that includes an AI exchange 500 and provides access to the AI ​​exchange 500 to multiple providers of datasets, machine learning algorithms, and AI models, as well as customers of the datasets, machine learning algorithms, and AI models, in accordance with the techniques described in this disclosure. In this example, customer networks 308A-308C (collectively, "customer networks 308"), each associated with a different customer, access a cloud exchange point within a data center 300 to access the AI ​​exchange 500 and securely access the datasets, machine learning algorithms, and AI models contained therein. Customer networks 308 can each include an endpoint device (not shown for simplicity) that can access the AI ​​exchange 500. Example endpoint devices include servers, smartphones, workstations, laptops / tablets, and the like. Cloud service providers 320A-320C (collectively, “cloud service providers 320”) can access the cloud exchange point 303 within the data center 300 in order to access the AI ​​exchange 500, for example, to provide data sets, machine learning algorithms, and AI models to the AI ​​exchange 500 or to train AI models in the AI ​​exchange 500.

[0059] Customer networks 308A-308B include respective provider edge / autonomous system boundary routers (PE / ASBRs) 310A-310B. Each of PE / ASBRs 310A, 310B can execute an exterior gateway routing protocol to peer with one of PE routers 302A-302B ("PE routers 302" or more simply "PE 302") via one of access links 316A-316B (access links 316A-316C may be collectively referred to as "access links 316"). In the illustrated example, each of access links 316 represents a transit link between an edge router of customer network 308 and an edge router (or autonomous system boundary router) of cloud exchange point 303. For example, PE / ASBR 310A and PE router 302A can directly peer via an external gateway protocol (e.g., external BGP) to exchange L3 routes and L3 data traffic between customer network 308A and cloud service provider network 320 over access link 316A. In some cases, access link 316 can represent and alternatively be referred to as a connection circuit for an Internet Protocol-Virtual Private Network (IP-VPN) configured in Internet Protocol / Multiprotocol Label Switching (IP / MPLS) fabric 301, as described in further detail below. In some cases, access links 316 can each comprise a direct physical connection between at least one port of customer network 308 and at least one port of cloud exchange point 303, without an intervening transit network. Access links 316 can operate over a virtual local area network (VLAN) or stacked VLANs (e.g., QinQ tunneling), a virtual extended local area network (VxLAN), a label switched path (LSP), a generic routing encapsulation (GRE) tunnel, or other types of tunnels.

[0060] Although illustrated and primarily described with respect to L3 connectivity, PE router 302 can additionally provide L2 connectivity between customer network 308 and cloud service provider network 320 via access link 316. For example, a port of PE router 302A can be configured with an L2 interface that provides L2 connectivity to customer network 308A and cloud service provider 320A via access link 316A, where PE router 312A is coupled to a port of PE router 304A, which is also configured with an L2 interface. A port of PE router 302A can additionally be configured with an L3 interface that provides L3 connectivity to customer network 308A and cloud service provider 320B via access link 316A. PE router 302A can be configured with multiple L2 and / or L3 sub-interfaces.

[0061] Each of access links 316 and aggregation links 322A through 322D (collectively, aggregation links 322) may include a network interface device (NID) that connects customer network 308 or cloud service provider 320 to a network link between the NID and PE router 302 or one of PE routers 304A through 304D (collectively, "PE routers 304" or simply "PE 304"). Each of access links 316 and aggregation links 322 may represent or include any of several different types of links that provide L2 and / or L3 connectivity.

[0062] In this example, customer network 308C is not an autonomous system with an autonomous system number. Customer network 308C can represent an enterprise, network service provider, or other customer network located within the routing footprint of cloud exchange point 303. Customer network includes customer edge (CE) device 311, which can execute an exterior gateway routing protocol to peer with PE router 302B via access link 316C. In various examples, any of PE / ASBRs 310A to 310B can alternatively be or otherwise represent a CE device.

[0063] Access link 316 comprises a physical link. PE / ASBRs 310A and 310B, CE device 311, and PE routers 302A and 302B exchange L2 and L3 packets via access link 316. In this regard, access link 316 constitutes a transport link for AI exchange access via cloud exchange point 303. Cloud exchange point 303 may represent an example of any of cloud exchange points 128. Data center 300 may represent an example of data center 201.

[0064] In some examples, the cloud exchange point 303 aggregates customer 308 access to the cloud exchange point 303 and the AI ​​exchange 500, as well as data sets, machine learning algorithms, and AI models from any one or more cloud service providers 320. For example, Figures 3A to 3BThe diagram illustrates access links 316A and 316B connecting respective customer networks 308A and 308B to PE router 302A of cloud exchange point 303, and access link 316C connecting customer network 308C to PE router 302B. Any one or more of PE routers 302 and 304 may comprise an ASBR. PE routers 302 and 304 and IP / MPLS fabric 301 may be configured to interconnect any of access links 316 and any of aggregation links 322 to AI exchange 500. As a result, cloud service provider network 320A, for example, only needs to configure a single cloud link (here, aggregation link 322A) to provide data sets, machine learning algorithms, and AI models to multiple customer networks 308 via AI exchange 500.

[0065] Additionally, a single customer network (e.g., customer network 308A) only needs to configure a single cloud access link (herein, access link 316A) to the cloud exchange point 303 within the data center 300 in order to access the AI ​​exchange 500 and data sets, machine learning algorithms, and AI models from multiple cloud service provider networks 320 that provide cloud services via the cloud exchange point 303.

[0066] Each cloud provider network 320 includes a server configured to provide one or more cloud services to users, such as providing data sets, machine learning algorithms, and AI models. The cloud provider network 320 includes PE routers 312A to 312D (collectively referred to as "PE routers 312"), each of which executes an exterior gateway routing protocol, such as eBGP, to exchange routes with PE routers 304A to 304B (collectively referred to as "PE routers 304") of the cloud exchange point 303. Each cloud provider network 320 in the cloud provider network 320 can represent a public, private, or hybrid cloud. Each cloud provider network 320 in the cloud provider network 320 can have an assigned autonomous system number or be part of the autonomous system footprint of the cloud exchange point 303.

[0067] In the illustrated example, IP / MPLS fabric 301 interconnects PE 302 and PE 304 to AI switch 500. IP / MPLS fabric 301 includes one or more switching and routing devices, including PE 302, 304, that provide IP / MPLS switching and routing of IP packets to form an IP backbone. In some examples, IP / MPLS fabric 301 can implement one or more different tunneling protocols (i.e., in addition to MPLS) to route traffic between PE routers and / or associate traffic with different IP-VPNs. According to the techniques described herein, IP / MPLS fabric 301 implements an IP virtual private network (IP-VPN) to connect any of the customers 308 to AI switch 500 or multiple cloud service provider networks 320 to provide data center-based 'transport' and layer 3 connectivity.

[0068] Given that service provider-based IP backbone networks require wide area network (WAN) connections with limited bandwidth to transport service traffic from layer 3 service providers to customers, cloud exchange point 303 as described herein 'transports' service traffic and connects cloud service providers 320 to the AI ​​exchange 500 and to customers 308 within the high-bandwidth local environment of the data center 300 provided by the data center-based IP / MPLS fabric 301. In some example configurations, customer network 308 and cloud service provider network 320 can be connected to the same PE router of the IP / MPLS fabric 301 via corresponding links.

[0069] Access links 316 and aggregation links 322 may include attachment circuits that associate traffic exchanged with connected customer networks 308 or cloud service provider networks 320 with virtual routing and forwarding instances (VRFs) configured in PEs 302 and 304 and corresponding to IP-VPNs operating on IP / MPLS fabric 301. For example, PE router 302A may exchange IP packets with PE / ASBR 310A over a bidirectional label-switched path (LSP) operating on access link 316A, which is an attachment circuit for a VRF configured in PE router 302A. As another example, PE router 304A may exchange IP packets with PE router 312A over a bidirectional label-switched path (LSP) operating on aggregation link 322A, which is an attachment circuit for a VRF configured in PE router 304A. Each VRF may include or represent a different routing and forwarding table with different routes.

[0070] The PE routers 302, 304 of the IP / MPLS fabric 301 can be configured in a corresponding hub-and-spoke arrangement for cloud services, where PE 304 implements the cloud service hub and PE 302 is configured as a hub spoke (for various hub-and-spoke instances / arrangements). The hub-and-spoke arrangement ensures that service traffic can flow between the hub PE and any of the spoke PEs, but not directly between different spoke PEs. As further described below, in a hub-and-spoke arrangement based on the data center IP / MPLS fabric 301 and southbound service traffic (i.e., from the cloud service provider to the customer), PE 302 advertises routes received from PE / ASBRs 310A to 310B to PE 304, which advertises the routes to PE 312. For northbound service traffic (ie, from the customer to the cloud service provider), PE 304 advertises the routes received from PE 312 to PE 302, which advertises the routes to PE / ASBRs 310A-310B.

[0071] For some customers of cloud exchange point 303, the cloud exchange point 303 provider may configure a fully meshed arrangement, whereby each set of PEs 302 and 304 is coupled to a different customer site network for that customer. In this case, IP / MPLS fabric 301 implements a Layer 3 VPN (L3VPN) for cage-to-cage or redundant traffic (also known as east-west or horizontal traffic). L3VPN can enforce closed user groups, whereby each customer site network can send traffic to each other but cannot send or receive traffic outside the L3VPN.

[0072] PE routers may be coupled to each other based on a peer-to-peer model without the use of an overlay network. That is, PE / ASBRs 310A to 310B and PE 312 may not directly peer with each other to exchange routes, but may exchange routes indirectly via IP / MPLS fabric 301. Figure 3BIn the example shown, cloud exchange point 303 is configured to implement multiple Layer 3 virtual circuits 330A through 330C (collectively, "virtual circuits 330") to interconnect customer network 308 and cloud service provider network 320 with an end-to-end IP path. Cloud service provider 320 and customer 308 can each be an endpoint of multiple virtual circuits 330, where multiple virtual circuits 330 traverse one or more connection circuits between IP / MPLS fabric 301 and the cloud service provider / customer PE / PE or PE / CE pairs. Virtual circuits 330 represent Layer 3 paths through IP / MPLS fabric 301 between attachment circuits connecting the customer network to fabric 301 and attachment circuits connecting the cloud service provider network to fabric 301. Each virtual circuit 330 can include at least one tunnel (e.g., an LSP and / or Generic Routing Encapsulation (GRE) tunnel) having endpoints at PEs 302 and 304. PEs 302 and 304 can establish a complete mesh of interconnected tunnels.

[0073] Each virtual circuit 330 may include a different hub-and-spoke network configured within IP / MPLS network 301, with PE routers 302, 304 exchanging routes using a full or partial mesh of Border Gateway Protocol peering sessions (in this example, a full mesh of Multiprotocol Interior Border Gateway Protocol (MP-iBGP) peering sessions). MP-iBGP, or simply MP-BGP, is an example of a protocol for routers to exchange labeled routes to implement MPLS-based VPNs. However, PEs 302, 304 may exchange routes to implement IP-VPNs using other technologies and / or protocols.

[0074] In the example of virtual circuit 330A, PE router 312A of cloud service provider network 320A can send routes for cloud service provider network 320A to PE router 304A via a routing protocol (e.g., eBGP) peering connection with PE router 304A. PE router 304A associates the routes with a hub-and-spoke network, which may have an associated VRF that includes spoke PE router 302A. PE router 304A then exports the routes to PE router 302A; PE router 304A can export the routes specifying PE router 304A as the next-hop router along with a label identifying the hub-and-spoke network. PE router 302A sends the routes to PE / ASBR 310B via a routing protocol connection with PE / ASBR 310B. PE router 302A may send the route after adding the autonomous system number of cloud exchange point 303 (e.g., to the BGP autonomous system path (AS_PATH) attribute) and specifying PE router 302A as the next-hop router. Thus, even though cloud exchange point 303 may be based within a data center, cloud exchange point 303 is an autonomous system "hop" in the path of autonomous systems from customer 308 to cloud service provider 320 (or vice versa). PE / ASBR 310B installs the route into a routing database, such as a BGP routing information base (RIB), to provide layer 3 reachability to cloud service provider network 320A. In this way, cloud exchange point 303 "leaks" routes from cloud service provider network 320 to customer network 308 without requiring a direct layer 3 peering connection from cloud service provider network 320 to customer network 308.

[0075] PE / ASBR 310B and PE routers 302A, 304A, and 312A can perform similar operations in the reverse direction to forward routes originating from customer network 308B to PE router 312A, thereby providing connectivity from cloud service provider network 320A to customer network 308B. In the example of virtual circuit 330B, PE routers 312B, 304A, 302A, and PE / ASBR 310B exchange routes for customer network 308B and cloud service provider 320B in a manner similar to that described above for establishing virtual circuit 330B. As a result, cloud exchange point 303 within data center 300 internalizes the peering connection established between PE / ASBR 310B and each of PE routers 312A, 312B to perform cloud aggregation for multiple layer 3 cloud services provided by different cloud service provider networks 320A, 320B and deliver the multiple aggregated layer 3 cloud services to customer network 308B, which has a single access link 316B to cloud exchange point 303.

[0076] Figure 4is a block diagram illustrating further details of one example of a computing device operating in accordance with one or more techniques of this disclosure. Figure 4 A specific example of a server or other computing device 13500 may be illustrated that includes processing circuitry 13502 that may include one or more processors that may execute any one or more of the programmable network platform components, or any other systems, applications, or modules described herein. In other instances, other examples of computing devices 13500 may be used. Although for purposes of illustration, Figure 4 13500, but the computing device may be any component or system including one or more processors or other suitable computing environment for executing software instructions, and need not include, for example, Figure 4 Computing device 13500 is an example device that can implement AI exchange 500, model registry 508, or any portion of AI exchange 500 or model registry 508 according to the techniques of this disclosure.

[0077] like Figure 4As shown in the specific example of , computing device 13500 includes processing circuit system 13502, one or more input devices 13504, one or more communication circuit systems 13506, one or more output devices 13512, one or more storage devices 13508, and one or more user interface (UI) devices 13510. In some examples, computing device 13500 also includes a data set 13530 and a machine learning algorithm 13532 that can be provided to computing system 13500 by a cloud service provider such as cloud service providers 320A to 320C. Computing device 13500 may also include a container 13534. Container 13534 may include multiple containers, each container containing an AI model, AI model metadata associated with the AI ​​model, and a hash unique to the specific AI model trained. Where computing device 13500 implements AI exchange 500, computing device 13500 may also include a model registry controller 13536, which AI exchange 500 may use to interface with model registry 508; AI training application(s) 13538, which may be used to train AI models; a secure sandbox application 13542, which may be used to implement a secure sandbox for AI model transactions; and a marketplace application 13544, which may be used to implement a marketplace for datasets 13530, machine learning algorithms 13532, and AI models within containers 13534. Where computing device 13500 implements model registry 508, computing device 13500 may include a model registry application 13540, such as a model registry API gateway, blockchain application, and blockchain ledger(s) discussed later herein.

[0078] The computing device 13500 may also include one or more applications 13522 for performing other functions, a programmable network platform application (PNPAS) 13524, and an operating system 13516. Each of the one or more storage devices 13508 and the applications 13516 in the operating system can be executed by the computing device 13500. Each of the components of the computing system 13500 (e.g., processing circuitry 13502, one or more input devices 13504, communication circuitry 13506, one or more storage devices 13508, one or more UI devices 13510, and one or more output devices 13512) is coupled (physically, communicatively, and / or operationally) for inter-component communication. In some examples, the one or more communication channels 13514 may include a system bus, a network connection, an inter-process communication data structure, or any other method for conveying data between components. As an example, each of the components of the computing system 13500 (e.g., the circuit system 13502, one or more input devices 13504, the communication circuit system 13506, one or more storage devices 13508, one or more UI devices 13510, and one or more output devices 13512) can be coupled via one or more communication channels 13514.

[0079] In one example, processing circuitry 13502 is configured to implement functions and / or process instructions for execution within computing device 13500. For example, processing circuitry 13502 can process instructions stored in one or more storage devices 13508. Examples of processing circuitry 13502 may include any one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry.

[0080] One or more storage devices 13508 can be configured to store information within computing device 13500 during operation. In some examples, one or more storage devices 13508 are described as computer-readable storage media. In some examples, one or more storage devices 13508 are transient memory, meaning that the primary purpose of one or more storage devices 13508 is not long-term storage. In some examples, one or more storage devices 13508 are described as volatile memory, meaning that one or more storage devices 13508 do not retain their stored contents when the computer is shut down. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, one or more storage devices 13508 are used to store program instructions executed by processing circuitry 13502. In one example, one or more storage devices 13508 are used by software or applications running on computing device 13500 to temporarily store information during program execution.

[0081] In some examples, one or more storage devices 13508 also include one or more computer-readable storage media. One or more storage devices 13508 can be configured to store larger amounts of information than volatile memory. One or more storage devices 13508 can also be configured for long-term storage of information. In some examples, one or more storage devices 13508 include non-volatile storage elements. Examples of such non-volatile storage elements include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable (EEPROM) memory.

[0082] In some examples, computing device 13500 also includes one or more communication circuitry 13506. In one example, computing device 13500 utilizes communication circuitry 13506 to communicate with external devices via one or more networks, such as one or more wired / wireless / mobile networks. Communication circuitry 13506 can include a network interface card, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device that can send and receive information. In some examples, computing device 13500 uses communication circuitry 13506 to communicate with external devices.

[0083] In one example, computing device 13500 also includes one or more UI devices 13510. In some examples, one or more UI devices 13510 are configured to receive input from a user through tactile, audio, or video feedback. Examples of one or more UI devices 13510 include a presence-sensitive display, a mouse, a keyboard, a voice response system, a video camera, a microphone, or any other type of device for detecting commands from a user. In some examples, the presence-sensitive display includes a touch-sensitive screen.

[0084] One or more output devices 13512 may also be included in the computing device 13500. In some examples, the one or more output devices 13512 are configured to provide output to the user using tactile, audio, or video stimulation. In one example, the one or more output devices 13512 include a presence-sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting signals into an appropriate form understandable to a human or machine. Additional examples of the one or more output devices 13512 include speakers, cathode ray tube (CRT) monitors, liquid crystal displays (LCDs), or any other type of device that can generate understandable output to a user.

[0085] Computing device 13500 may include an operating system 13516. In some examples, operating system 13516 controls the operation of components of computing device 13500. For instance, in one example, operating system 13516 facilitates communication of one or more applications 13522 and programmable network platform application(s) 13524 with processing circuitry 13502, communication circuitry 13506, one or more storage devices 13508, one or more input devices 13504, one or more UI devices 13510, and one or more output devices 13512.

[0086] Applications 13522 and programmable network platform applications 13524 may also include program instructions and / or data that may be executed by computing device 13500. Example programmable network platform application(s) 13524 that may be executed by computing device 13500 may include L3 instances such as service module 13550 and virtual performance center module 13551.

[0087] Figure 5is a block diagram illustrating an example AI exchange system according to techniques of this disclosure. AI exchange 500 may include a secure sandbox 502, a marketplace 504, and a platform 506. AI exchange 500 may be a private or standalone system that can be used by consortium members seeking access to others' AI algorithms, datasets, or AI models to develop existing AI models to better meet their needs or leverage AI models developed by others. AI exchange 500 can provide consortium members, such as data scientists, with a secure environment for creating and training AI models in a secure manner that enables potential future users of trained AI models to verify the provenance and characteristics of the trained AI models (e.g., without requiring these potential users to train the AI ​​models themselves). Data providers 524A through 524C are shown coupled to secure sandbox 502. Data providers 524A through 524C may provide datasets 501A through 501C, respectively, to secure sandbox 502. Algorithm providers (Algo providers) 526A through 526C are also shown coupled to secure sandbox 502. Algorithm providers 526A to 526C may provide machine learning algorithms to the secure sandbox 502. The secure sandbox 502 may be a container where clients (such as Figures 1 to 2 Clients 108A to 108C) can access areas of machine learning algorithms and data sets (e.g., machine learning algorithms 503A to 503C and data sets 501A to 501C). Clients can also access data stored in secure sandbox 502 ( Figure 5 The data providers 524A to 524C and the algorithm providers 526A to 526C may be cloud service providers, such as Figure 1 In some examples, one or more of the data providers 524A to 524C or the algorithm providers 526A to 526C may also be a customer, such as Figure 1 Customers 108A to 108C.

[0088] In some examples, marketplace 504 can be provided by a cloud service provider, such as cloud service providers 110A through 110N. In some examples, AI exchange 500 can include multiple marketplaces 504. In some examples, marketplace 504 can target a specific industry segment, such as the aviation market, the healthcare market, the oil and gas market, the autonomous vehicle market, etc. Marketplace 504 can be operated by companies, universities, individuals, and / or other entities interested in AI models and / or datasets within the industry segment served by marketplace 504 (e.g., the aviation industry). For example, AI exchange 500 can be provided by a neutral, independent entity as a PaaS.

[0089] The AI ​​exchange 500 can be coupled to a model registry 508. In some examples, the model registry 508 can be implemented within the secure sandbox 502. The model registry 508 can be a blockchain-based registry that registers AI models and verifies AI model metadata. The model registry 508 can maintain key AI model metadata for operations performed using various AI models in the AI ​​exchange 500. The model registry 508 can include a model registry application programming interface (API) gateway 510, a key-value pair registry database (KVPR) 522, and blockchain records 512 (e.g., a blockchain ledger). In some examples, the blockchain records 512 can include records containing AI model metadata, such as model attributes 514, training transactions 516, training data usage 518, and training infrastructure 520. In other examples, the blockchain records can include records of containers 521, each containing an AI model, along with related AI model metadata and related hashes. In some examples, blockchain record 512 may include model attributes 514 , training transactions 516 , training data usage 518 , and training infrastructure 520 and container 521 .

[0090] The AI ​​model in the security sandbox 502 can be packaged together with its related AI model metadata and hash inside a container, which can be used to verify the AI ​​model and AI model metadata. The containerization of AI models will be discussed below. Figure 7 for a more detailed discussion.

[0091] AI models can be developed and enhanced in multiple stages. For example, a data scientist might build an initial AI model using an AI algorithm and a small dataset. Subsequently, the AI ​​model can be deployed in a production or industry environment and refined using larger datasets, for example, by an operations team. Therefore, there are many transactions that may occur involving a specific AI model within the secure sandbox 502. Transactions within the secure sandbox 502 (such as training a model on a specific dataset) can be monitored by the AI ​​exchange 500 and recorded as model metadata in the AI ​​container and model registry 508. The secure sandbox 502 can provide AI model metadata to the model registry 508 via the model registry API gateway 510. New metadata can be added to the existing metadata of, for example, a further trained AI model, and the resulting AI model, the combined metadata, and a new hash based on the resulting AI model can be containerized. The data scientist workflow and production deployment of the AI ​​model workflow within the secure sandbox 502 can be implemented as a pipeline using an orchestrator such as Kubeflow. Additionally, the containerization process can be implemented using such an orchestrator. The secure sandbox 502 can restrict unauthorized data sets, AI algorithms, AI models, and AI models built or refined from the secure sandbox 502, and attempts to ensure that every transaction involving the AI ​​model is logged. The AI ​​model built and trained in the secure sandbox 502 can be referred to herein as a secure AI model, and the AI ​​metadata associated with the secure AI model can be referred to herein as secure AI model metadata.

[0092] Figure 6 is a block diagram depicting an example pipeline update process according to techniques of this disclosure. Figure 6 Two separate pipelines are depicted: the Experimentation Pipeline 531 and the Industrialization Pipeline 533. For example, a data scientist from a consortium member can access the Experimentation Pipeline 531 through the Marketplace 504 by, for example, selecting the Experimentation option 530. Figure 6 As shown, the experiment pipeline 531 can be located in the secure sandbox 502. The data scientist can begin by defining a hypothesis (DH) 534. A hypothesis can be a problem statement or a use case. For example, the data scientist might define a hypothesis, namely, how to increase traffic flow during rush hour at a specific location. Then, if the data scientist is authorized to import that specific data, the data scientist can collect data (CD) 536. Collecting data 536 can involve ingesting data, such as dataset 501A, or can involve the data scientist importing data external to the AI ​​exchange 500 into the secure sandbox 502.

[0093] The data scientist may explore (EXP) 538. Exploration 538 may involve data analysis and / or cleaning. For example, the data scientist may analyze the data being used and recognize that there are anomalies in the data set and remove them. The data scientist may perform modeling (MDL) 540. Modeling 540 may involve feature identification. For example, the data scientist may identify that a traffic circle may affect traffic flow differently than a traffic light. The data scientist may also validate (VAL) 542. Validation 542 may involve testing the model and evaluating the results. If the data scientist believes the results are not good, the data scientist may iterate and improve the assumptions by returning to define hypotheses 534. If the data scientist believes the results are acceptable, the data scientist may choose to keep the model (MOD) 544.

[0094] As about Figure 5 As discussed, each transaction performed by a data scientist in an experiment pipeline 531 can be recorded. The experiment pipeline 531 can use a model registry controller 528 to record these transactions in the model registry 508 via, for example, a model registry API gateway 510. In some examples, transactions can be recorded in a container 521, such as Figure 5 In some examples, transactions can be recorded in model attributes 514 , training transactions 516 , training data usage 518 , and training infrastructure 520 .

[0095] Data scientists or operators ("users") from alliance members can access the industrialized pipeline 533 in a similar manner through the marketplace 504, for example, by selecting the industrialized option 532. Figure 6 502 . The user may select an AI model available in the secure sandbox and collect data (CD) 546 . Collecting data 546 may involve ingesting one or more production datasets. In some examples, the production dataset may be a larger dataset than the dataset used to collect data 536 in the experimental pipeline 531 . The production dataset may be a dataset such as datasets 501A to 501C or may be a dataset imported by the user if the user has permission to import a dataset into the secure sandbox 502 . The user may train a model (TM) 548 . Training a model 548 may involve running an AI model on data from collected data 546 . For example, a user may use AI training application 13538 to train an AI model. The user may continue integration (CI) 550 . Continuing integration 550 may involve operationalizing the AI ​​model, such as creating a representational state transfer (REST) ​​API or automatically running features.

[0096] The user may determine feedback (FB) 552. Feedback 552 may involve evaluating the model and may involve making collaborative and informed decisions about the efficacy of the AI ​​model. For example, the user may consult colleagues or other coalition members about the efficacy of the AI ​​model. The user may also determine monitoring (MON) 554. Monitoring 554 may involve monitoring the performance of the AI ​​model and testing the AI ​​model. The user may determine to maintain and improve the model, in which case the user may return to collect data 546. In some examples, the user may be completely satisfied with the AI ​​model and may wish to deploy the model (MOD) 556 to the field. For example, the user may export the AI ​​model 556 from the AI ​​exchange 500 to a server on an enterprise network residing at the user's workplace. In some examples, the user may export the AI ​​model 556 to multiple servers, which may be located in multiple locations. If desired, the user may later contact the model registry 508 to verify the metadata associated with the AI ​​model 556.

[0097] As with the experimental pipeline 531, each transaction in the industrialized pipeline 533 can be recorded. The industrialized pipeline 533 can use the model registry controller 528 to record these transactions in the model registry 508 through, for example, the model registry API gateway 510. In some examples, the transaction can be recorded in the container 521, such as Figure 5 In some examples, transactions can be recorded in model attributes 514 , training transactions 516 , training data usage 518 , and training infrastructure 520 .

[0098] Figure 7 is a block diagram illustrating an example AI model container according to the techniques of the present invention. A user can train an AI model (560) in a pipeline such as the experimental pipeline 530 or the industrialized pipeline 532. The user can build an AI model and the secure sandbox 502 can serialize the AI ​​model (562). The secure sandbox 502 can then containerize the AI ​​model (564). The container 566 can contain a serialized AI model object file 750 and AI model lineage information 762. In some examples, the serialized AI model object file 750 can be one or more PKL files. In some examples, the AI ​​model lineage information 762 can be stored in HD5 format.

[0099] The AI model lineage information 762 can include AI model attribute (AI model attribute) 752 and AI model transaction data attributes 760. In some examples, the AI model attribute 752 can be stored in a JSON file format. The AI model attribute 752 can include information related to the serialized AI model object file 750 contained in the container 566, such as a unique ID that identifies the serialized AI model object file 750, a version of the serialized AI model object file 750 (e.g., each time the serialized AI model object file 750 is de-serialized and trained, a new version of the serialized AI model object file 750 can be created and given a new version number), an identification of the AI model that is created by the serialized AI model object file 750, a class of the AI model (e.g., what type of AI model is the AI model), an algorithm type used to create the AI model represented by the serialized AI model object file 750, a framework 750 that the AI model represented by the serialized AI model object file is targeted for, one or more use cases of the AI model represented by the serialized AI model object file 750, a URL that can link to more information about the AI model represented by the serialized AI model object file 750, support details that can be helpful to a user of the AI model represented by the serialized AI model object file 750, and usage and rating information of the AI model represented by the serialized AI model object file 750, etc. The AI model attribute 752 can also include a model hash 764 that can be used to verify the serialized AI model object file 750 and the AI model lineage information 762.

[0100] The AI model training data attributes 760 can contain information related to the training of the AI model represented by the serialized AI model object file 750. In some examples, the AI model training data attributes 760 contain information related to each training transaction that occurs within the secure sandbox 502 (e.g., the experimental pipeline 530 and the industry pipeline 532). For example, the AI model training data attributes 760 can contain information about the training transaction of the AI model represented by the serialized AI model object file 750, such as a training date and time, a training duration, a training geography (such as where the training physically occurred or a source of data used for the training), a description of the training, an industry sub-market that the training is to benefit, a configuration of any hyperparameters used, a benchmark result, one or more training use cases (e.g., what use case was used for the trainer of the AI model), a training dataset size, an underlying structure used during the training, a cloud or natural flag (e.g., whether the training was performed on a cloud resource), and a flag that can identify whether the training was performed on the experimental pipeline 531 or the industry pipeline 533, etc.

[0101] The AI model properties 752 and / or the AI model transaction data properties 760 can be stored in folders such as folders 754A-754B and 756A-756D. The folder 754A can contain metadata associated with transactions of a provider of the AI model. The folder 754B can contain metadata associated with transactions of a customer of the AI model. The folders 756A-756B can be subfolders of the folder 754A, and the folders 756C-756D can be subfolders of the folder 754B. The folder 756A can contain metadata associated with internal training transactions, such as metadata of a provider of the AI model that is within the consortium, e.g., AI model training metadata 758A. The folder 756B can contain metadata associated with external training transactions, such as metadata of a provider that is outside the consortium, e.g., AI model training metadata 758B. The folder 756C can contain training metadata associated with internal training transactions of a customer, such as a customer that is a member of the consortium, e.g., AI model training metadata 758C. The folder 756D can contain training metadata associated with external training transactions of a customer, such as a customer that is not a member of the consortium, e.g., AI model training metadata 758D. Alternatively, the folders 756A and 756C can contain metadata related to training that occurs within the secure sandbox 502, and the folders 756B and 756D can contain metadata related to training that occurs outside the secure sandbox 502.

[0102] The container 566 can be published to the model registry 508, e.g., through the model registration controller 528 and the model registry API gateway 510 (neither of which are shown in Figure 7 the key-value registry database 522 and can store the container 566 in the blockchain record 512. The secure sandbox 502 can also store the container 566 locally.

[0103] Figure 85 is a block diagram illustrating security aspects of a security sandbox according to the technology of the present disclosure. The security sandbox 502 can contain a dataset (such as dataset (data) 501A) and a machine learning algorithm (such as machine learning algorithm (algo) 503A). The security sandbox 502 can also contain policies 570. Container security 572 can enforce container-level policies 570. Network security 574 can enforce network-level policies 570. Policies 570 can restrict datasets from entering and / or leaving the security sandbox 502. Policies 570 can also restrict machine learning algorithms from entering and / or leaving the security sandbox 502. Policies 570 can also restrict trained IA models from entering and / or leaving the security sandbox 502. Policies 570 can also restrict the identities of dataset providers, machine learning algorithm providers, and AI model providers. For example, network security 574 can enforce policies 570 to only grant previously authorized dataset providers, machine learning algorithm providers, and AI model providers access to the security sandbox 502. Policies 570 can also restrict model changes at the pipeline level or at the individual user level. For example, network security 574 may enforce policy 570 to prevent changes to existing AI models in a given pipeline, such as experimentation pipeline 531 , or to prevent specific users or classes of users from changing existing AI models.

[0104] As previously discussed, all operations or transactions performed on the AI ​​model in the secure sandbox 502 can be tracked and stored in a container such as container 566. Container 566 can be stored in the model registry 508 (and / or the secure sandbox 502) and includes AI model metadata associated with the training of the AI ​​model and any changes made to the AI ​​model in the experimentation pipeline 531 or the industrialization pipeline 533. In some examples, the model registry 508 can be implemented in the secure sandbox 502. In some examples, a flag is also stored indicating in which pipeline the transaction occurred. By tracking and storing all transactions that occur to the AI ​​model in the secure sandbox 502, the secure sandbox 502 can accurately audit the lineage of a given AI model, and the AI ​​exchange 500 can prove the AI ​​model lineage to clients of the AI ​​model by using the model registry 508.

[0105] Figure 9 is a conceptual diagram of an example model registry update and example model usage process according to the technology of this disclosure. If a user wants to add an AI model, the user can enter the marketplace 504. In some examples, the marketplace 504 can attempt to authorize AI model access (auth model access) (580) by sending an authorization request 580A to the model registry 508 through the model registry API gateway 510. In some examples, the model registry 508 can be in a Figure 5The model registry 508 may be implemented within the security sandbox of the security sandbox 502. The model registry 508 may respond by sending an authorization identity token 580B back to the marketplace 504. The user may then attempt to set a model registry record (set model registry) (582) through the marketplace 504. The marketplace 504 may send a model set request 582A to the model registry 508 through the model registry API gateway 510. The model set request 582A may include the authorization identity token 580B.

[0106] In response to model setup request 582A, model registry 508 may generate an AI model unique ID and hash. The hash may be based at least in part on the AI ​​model. Model registry 508 may use any known technique to generate the hash. Model registry 508 may send the AI ​​model unique ID and hash 582B to marketplace 504. Marketplace 504 may then use the AI ​​model unique ID and hash 582B to update the local AI model. The local AI model may be serialized and containerized using the associated AI model metadata and the associated hash and stored in model registry 508. The local model may also be moved or copied to the secure sandbox 502.

[0107] When a user wishes to perform an AI transaction such as training an existing AI model, the user may access the security sandbox 502 to authorize AI model access (586). The security sandbox 502 may send an authorization request (auth model access) 586A to the model registry 508 through the model registry API gateway 510. The model registry 508 may send an authorization identity token 586B in response to the authorization request 586A. The security sandbox 502 may then attempt to verify the checksum hash (valid cs hash) (588). The security sandbox 502 may send a message 588A to the model registry 508 through the model registry API gateway 510. The message 588A may include a local model hash and an authorization identity token 586B. In response to the message 588A, the model registry 508 may attempt to verify the local model hash. If the model registry 508 verifies the local model hash, the model registry 508 may send a hash verification response 588B to the security sandbox 502. Based on receiving the hash verification response 588B, the security sandbox 502 may permit the user to perform the AI ​​transaction (perfAI trans) (590). The security sandbox 502 may log the AI ​​transaction (log AI trans) in the model registry 508 (592). For example, the security sandbox 502 may send a message 592A to the model registry 508 via the model registry API gateway 510. The message 592A may include the authorization identity token 586B and a registry update request containing AI model metadata associated with the transaction, such as described above with respect to Figure 7In some examples, the security sandbox 502 can containerize the AI ​​model metadata associated with the transaction using the AI ​​model and the updated hash before sending message 592A, and message 592A can include the authorization identity token 586B and the container. In some examples, the model registry 508 can update the hash. In some examples, the security sandbox 502 can update the hash.

[0108] Figure 10 6 is a block diagram of an example model registry according to the techniques of this disclosure. The model registry 508 may include an admin app 600, a model reporter 602, a model registry API gateway 510, a model request validator 612, a blockchain adopter 614, a model publisher 616, a visualization manager 618, a ledger 624, and a blockchain infrastructure 622. The admin app 600 may enable an administrator to program, monitor, or otherwise interact with the model registry 508. The model reporter 602 may provide information about AI models registered in the model registry 508 to the security sandbox 502 and / or the marketplace 504.

[0109] As discussed above, the model registry API gateway 510 may provide an API that allows the security sandbox 502 and the marketplace 504 to communicate with the model registry 502. The model registry API gateway 510 may include a model update API (MU API) 604, a model setup API (MS API) 606, an authorization API (auth API) 608, and a model history API (MH API) 610. The security sandbox 502 and / or the marketplace 504 may use the model update API 604 to communicate updates to the AI ​​model. The model setup API 606 may be used by the marketplace 504 when installing a model in an onboard manner, such as described above with respect to Figure 9 The security sandbox 502 and the marketplace 504 may use the authorization API 608 to request and receive authorization, such as described above with respect to Figure 9 As discussed, the model history API 610 can be used by the security sandbox 502 and / or the marketplace 504 to obtain information about the history or lineage of an AI model.

[0110] The model request validator 612 may receive a request to validate the AI ​​model. The model request validator 612 may access the ledger 624 (via, for example, the blockchain adapter 614) and locate the hash associated with the AI ​​model (in some examples, within a container), check the hash against the hash associated with the request, and report the validation back to, for example, the security sandbox 502.

[0111] The blockchain adapter 614 can receive AI model metadata. In some examples, the AI model metadata can be in a container, such as the container 566, that includes the AI model, the AI model metadata (e.g., the model properties 752 and the model transaction data properties 760), and the hash (e.g., the hash 764) and can interact with the blockchain substrate 622 to store the container in the ledger 624. In some examples, the AI model metadata can not be in a container. The model publisher 616 can publish the models in the containers stored in the ledger 624 to the secure sandbox 502 and / or the marketplace 504. For example, the model publisher 616 can send the serialized and containerized AI models to the secure sandbox 502. The secure sandbox 502 can remove the serialized AI models from the container and can deserialize the serialized AI models, making the AI models available in the secure sandbox 502. The model publisher 616 can also provide information, such as the AI model metadata associated with the AI models, to the marketplace 504 to permit users to view information about AI models that can be present in the secure sandbox 504. The visualization manager 618 can publish various analytics reports to customers, such as available AI models, which AI models have been updated, available datasets, etc.

[0112] By utilizing a blockchain-based model registry, the model registry 508 can maintain an immutable record of AI models and related AI model metadata contained within the secure sandbox 502. The model registry 508 can utilize a hash-based verification process to restrict invalid AI model updates. As such, the model registry 508 can be provided as a PaaS, the model registry API gateway 510 can be provided as a SaaS, and verification (attestation) of lineage or provenance information for AI models can be provided as a service. The model registry 508 can support multiple marketplaces and multiple secure sandboxes, including marketplaces and secure sandboxes for different market segments, different datasets, different machine learning algorithms, and different AI models. In some examples, the model registry 508 can be built on top of a blockchain ledger, such that each member of a consortium can have an immutable copy of the model registry 508.

[0113] Figure 11 FIG. 1 is a conceptual diagram illustrating an example AI exchange system, such as the AI exchange 500, in accordance with the techniques of this disclosure. As shown, the AI exchange 500 includes a model registry 508, a model registry API gateway 510, a marketplace 504, a secure sandbox 502, a blockchain substrate 622, a blockchain adapter 614, a model publisher 616, and a visualization manager 618.

[0114] The exchange system may be operated by a consortium 630. The consortium 630 may be a group of companies, universities, individuals, or other entities that wish to use or provide datasets, machine learning algorithms, or AI models of other members of the consortium 630. The consortium 630 may provide the legal, financial, and regulatory framework for the example system.

[0115] Data marketplace 632 (which may be an example of marketplace 504) may be responsible for member registration, asset registration and cataloging (e.g., which data sets, machine learning algorithms, and AI models are available in the example system), trade agreements, payment clearing, auditing, regulation, dispute resolution, and third-party tool orchestration.

[0116] Secure data exchange 634 (which can be an example of secure sandbox 502) can include AI center 636. AI center 636 can be responsible for data ingestion, AI and machine learning, data warehousing, anonymization (for consortia whose members want their data sets to be anonymized), visualization, and distribution services. Secure data exchange 634 can also include computer / container platform 638, storage 640, network 642, and data center 644 (or portions thereof). Computer / container platform 638 can be responsible for compute services and container management. Storage 640 can be responsible for storage and key management. Network 642 can be responsible for network fabric, network function virtualization, and security services. Data center 644 can be responsible for data center services.

[0117] Figure 12 is a conceptual diagram illustrating the control flow of an example system according to the techniques of the present invention. TM The data marketplace 632 can communicate with the security sandbox 502 via a data marketplace / security sandbox API 652. The data marketplace / security sandbox API 652 can provide orchestration, security, refund, and logging services to the example exchange system. In some examples, the data marketplace / security sandbox API 652 can reside in the data marketplace 632. In other examples, the data marketplace / security sandbox API 652 can reside in the security sandbox 502. In other examples, the data marketplace / security sandbox API 652 can be external to both the data marketplace 652 and the security sandbox 502. A provider 672 , such as data providers 524A- 524C or algorithm providers 526A- 526C, may provide a data set, a machine learning algorithm, or an AI model to the security sandbox 502 .

[0118] The secure sandbox 502 may also be restricted by federation policies 570. The data exchange sandbox 654 may be accessed by clients 660 (such as clients 108A to 108C), which may use the data exchange sandbox 654 to access data sets, machine learning algorithms, and AI models, for example, placed there by data providers 672. A data analyzer 656 may interact with the data exchange sandbox 654 to provide analytical reports on the data within the data exchange sandbox 654 (such as data sets, machine learning algorithms, and AI models) and actions taken by data providers 672 and clients 660. A secure server 658 may provide a secure environment for the data exchange sandbox 654 and the data analyzer 656 to reside.

[0119] Figure 13 is a block diagram illustrating the architecture of an example system according to the present invention. Marketplace 504 may include member manager 700, asset manager 702, protocol 704, governor 706, billing 708, audit 710, and data scientist pipeline manager 712. For example, member manager 700 may maintain a list of authorized providers, such as data providers 672, and authorized customers, such as customer 660. Marketplace 504 and / or security sandbox 502 may utilize member manager 700 and federation policy 570 to restrict access to authorized providers and authorized customers.

[0120] The asset manager 702 can track assets, such as data sets, machine learning algorithms, and AI models, within the AI ​​exchange 500. The asset manager 702 can be used to provide an asset catalog to the clients 660.

[0121] Agreements 704 may include legal contracts and other agreements associated with the alliance and / or AI exchange 500. Governance 706 may include governance rules such as alliance policy 570 and may optionally include data governor 650. Governance 706 may include algorithm(s) to enforce alliance policy 570. Billing 708 may include a billing system that can track accounts payable and receivable for data providers 672 and customers 660. Billing 708 may provide periodic billing statements or billing reports based on user needs. Billing 708 may also track revenue generated or spent by specific member organizations of the alliance or the alliance as a whole.

[0122] Audit 710 can provide auditing functionality for AI exchange 500. For example, audit 710 can audit financial transactions, AI model transactions, or the movement of items into or out of secure sandbox 502 or marketplace 504. Data scientist pipeline manager 712 can manage the experimentation pipeline 531 and / or industrialization pipeline 533 discussed above.

[0123] The security sandbox 502 may include a secure exchange API gateway 714 , a federated analytics computing cluster manager 716 , a multi-tenant orchestration manager 718 , a security sandbox manager 720 , a logging service 722 , and a chargeback 724 .

[0124] The secure exchange API gateway 714 can provide an API for data providers 672 and clients 660 to access the secure sandbox 502. The federated analytics compute cluster manager 716 can provide users with analytical insights into the compute clusters involved in the secure sandbox 502 and / or AI exchange 500. For example, the federated analytics compute cluster manager 716 can report the total usage of all system compute cluster assets as a whole.

[0125] Multi-tenant orchestration manager 718 can operate to keep data (such as datasets, machine learning algorithms, and AI models) separate for each member of the federation or each user. In some examples, multi-tenant orchestration manager 718 can keep each user's data invisible to other users unless otherwise authorized. Security sandbox manager 720 can manage the resources of security sandbox 502. Logging service 722 can log access to security sandbox 502 or to specific datasets, machine learning algorithms, or AI models. For example, the logging service can record who accessed, when the access was made, what content was accessed, etc. Refund 724 can operate to charge users of security sandbox 502.

[0126] The platform 506 may include a security server 658, a key management service 728, a network function virtualizer (NFV) 730, an edge computing service 732, and an interconnect fabric 734. The security server 726 may provide a secure execution environment 504 for the security sandbox 502 and / or the marketplace. The key management service 728 may manage cryptographic keys used to maintain the security of the security sandbox and cryptographic keys used to interact with the model registry 508. For example, the key management service 728 may issue keys to new users and may compare keys received during access requests with locally stored keys. The network function virtualizer 730 may virtualize network node functions to create services associated with the AI ​​exchange 500. The edge computing service 732 may provide edge computing services to users of the AI ​​exchange 500. For example, the edge computing service 732 may move data used by users of the AI ​​exchange 500 closer to the users to reduce the AI ​​exchange response time experienced by the users. The interconnect fabric 734 (such as the IP / MPLS fabric 301) can provide interconnections between users such as the provider 672 and the customer 660, the security server 658 (on which the security sandbox 502 may reside), and the marketplace 504 (if it does not reside on the security provider 658). For example, additional description of an example platform 506 that provides key management and network function virtualization is found in U.S. patent application Ser. No. 16 / 006,458, filed Jun. 12, 2018, and U.S. Provisional Patent Application Ser. No. 62 / 908,976, filed Oct. 1, 2019, the entire contents of which are incorporated herein by reference.

[0127] Figure 14 is a flow chart illustrating an example AI exchange technique according to the present disclosure. The security sandbox 502 can build an AI model based on input from the user (802). For example, the user can utilize the experimental pipeline 531 or the industrialized pipeline 533 to provide input to the security sandbox 502, and the security sandbox 502 can, for example, use a machine learning algorithm (e.g., Figure 5 machine learning algorithm 503A) and at least one data set (e.g., Figure 5 The secure sandbox 502 may train the AI ​​model based on the input from the user (804). For example, the user may utilize the experimental pipeline 531 or the industrialized pipeline 533 to provide input to the secure sandbox 502, and the secure sandbox 502 may train the AI ​​model using another dataset (e.g., dataset 501B).

[0128] The security sandbox 502 may create first AI model metadata based on a first transaction associated with building and training the trained AI model (806). For example, the security sandbox 502 may monitor transactions involving the AI ​​model and create AI model metadata based on these transactions. For example, the security sandbox 502 may create AI model attributes 752 ( Figure 7 ) and / or AI model transaction data attributes 760 ( Figure 7 ).

[0129] The security sandbox 502 may calculate a first hash based at least in part on the trained AI model (808).For example, the security sandbox 502 may calculate a model hash 764 based at least in part on the trained AI model.

[0130] The secure sandbox 502 may package the trained AI model, the first AI model metadata, and the first hash in a first container (810). For example, the secure sandbox may serialize the trained AI model and package the serialized model object file 750 (which may be a serialized version of the trained AI model), AI model attributes 752, model transaction data attributes 760, and the model hash 764.

[0131] The model registry 508 may register the first container (812). For example, the security sandbox 502 may publish the container 566 to the model registry 508. The model registry 508 may store the container 566 and / or may store AI model metadata related to the container 566, such as model attributes 514, training transactions 516, training data 518, and / or training infrastructure 520 in the blockchain record 512.

[0132] The security sandbox 502 can provide the user with secure access to the first container (814). For example, the security sandbox 502 can provide secure access to the container 566 through the data exchange sandbox 654 or through the secure exchange API gateway 714.

[0133] The model registry 508 can verify the trained AI model for the user based on the first hash (816). For example, the model request validator 612 can receive a request to verify the AI ​​model. The model request validator 612 can access the ledger 624 (e.g., through the blockchain adapter 614) and locate the hash associated with the AI ​​model (in some examples, within a container), check the hash against the hash associated with the request, and report the verification back to the user.

[0134] In some examples, the security sandbox 502 may determine whether to further train the trained AI model after creating the first AI model metadata. For example, the security sandbox may monitor the AI ​​model to determine whether the trained AI model is further trained. The security sandbox 502 may create second AI model metadata based on the trained AI model being further trained, the AI ​​model metadata being based on the first transaction and the second transaction associated with further training the trained AI model. The model registry 508 (or alternatively, the security sandbox 502) may create a second hash based at least in part on the further trained AI model using any known technique. The security sandbox 502 may package the further trained AI model, the second AI model metadata, and the second hash in a second container. The model registry 508 may register the second container.

[0135] In some examples, the first AI model metadata includes one or more of the following: AI model attributes, AI model training transaction information, AI model training data usage information, or AI model training underlying structure information. In some examples, the model registry 508 registers the first container in a blockchain-based registry.

[0136] In some examples, the security sandbox 502 may create a secure environment. The security sandbox 502 may receive a machine learning algorithm from a first provider. The security sandbox 502 may place the machine learning algorithm in the secure environment. The security sandbox 502 may receive a dataset from a second provider. The security sandbox 502 may place the dataset in the secure environment. The security sandbox 502 may build a secure AI model in the secure environment based on the machine learning algorithm. The security sandbox 502 may train the secure AI model in the secure environment based on the dataset. The security sandbox 502 may create secure AI model metadata based on a transaction associated with the secure AI model, wherein the transaction includes building and training the secure AI model, and wherein the secure AI model metadata indicates provenance information of the secure AI model.

[0137] In some examples, the security sandbox 502 can create a secure environment by restricting one or more of: the ingress and egress of data, the ingress and egress of trained AI models, data provider identities, model provider identities, or model changes.

[0138] Figure 15is a flow diagram illustrating an example security sandbox technique according to the present disclosure. The security sandbox 502 may receive a data set, a machine learning algorithm, and an AI model from a provider (822). For example, the security sandbox 502 may receive data sets 501A to 501C from data providers 524A to 524C. The security sandbox 502 may receive machine learning algorithms 503A to 503C from algorithm providers 526A to 526C. The security sandbox 502 may receive an AI model from a provider or user (which may be considered a provider in some examples), who may create an AI model, for example, in an experimentation pipeline 531 or an industrialization pipeline 533.

[0139] The security sandbox 502 may train at least one of the AI ​​models based on at least one of the datasets (824). For example, the security sandbox 502 may train the AI ​​model on at least one dataset, such as dataset 501B, based on user input. The security sandbox 502 may record AI model metadata associated with training the at least one AI model (826). For example, the security sandbox may monitor transactions associated with the AI ​​model and record AI model parameters 752 and / or model transaction data attributes 760 or other metadata associated with the transactions. The security sandbox 502 may store datasets, machine learning algorithms, AI models, and AI model metadata (828). For example, when the computing system 13500 represents the security sandbox 502, the computing system 13500 may store the datasets, machine learning algorithms, AI models, and AI model metadata in storage device(s) 13508.

[0140] In some examples, secure sandbox 502 includes an experimentation pipeline 531 and an industrialization pipeline 533, and training and logging are performed by at least one of the experimentation pipeline 531 or the industrialization pipeline 533. In some examples, secure sandbox 502 can serialize AI models and package each serialized AI model into a separate container. In some examples, each separate container includes a serialized AI model, related AI model metadata, and a related hash.

[0141] In some examples, the secure sandbox 502 can transmit at least one of the AI ​​model metadata or the individual container to the model registry 508. In some examples, the secure sandbox 502 can receive proof of the AI ​​model's lineage from the model registry 508.

[0142] By providing an AI exchange that manages AI model metadata in an integrated manner with experimentation pipelines and industrialization pipelines, the AI ​​exchange can track the AI ​​model metadata associated with each transaction that affects an AI model within the AI ​​exchange. AI models and AI model metadata can be containerized for access and verification purposes. The model registry can provide verification services to the AI ​​exchange. For example, the model registry can prove the lineage of an AI model and the AI ​​model metadata associated with it. The model registry and AI exchange (or parts of it) may be run by a third party that is otherwise unrelated to the consortium using the AI ​​exchange, thereby providing additional assurance to consortium members that the datasets, machine learning algorithms, and AI models within the AI ​​exchange are the datasets, machine learning algorithms, and AI models they claim to be. Multiple marketplaces can use a single security sandbox. Similarly, a single marketplace can use multiple security sandboxes. In addition, a single model registry can serve multiple AI exchanges.

[0143] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices or other hardware devices. In some cases, various features of an electronic circuit system may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0144] If implemented in hardware, the present disclosure may be directed to an apparatus such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause a processor to perform one or more of the methods described above. For example, a computer-readable data storage medium may store such instructions for execution by a processor.

[0145] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, an article of manufacture may include one or more computer-readable storage media.

[0146] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM or cache).

[0147] Code or instructions can be software and / or firmware executed by processing circuitry, including one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described in the disclosure can be provided within software modules or hardware modules.

Claims

1. A system comprising: Memory; as well as processing circuitry coupled to the memory, the processing circuitry being operable to: Building a first artificial intelligence (AI) model based on input from the user; Training the first AI model based on input from the user; creating first AI model metadata based on a first transaction associated with building and training the trained first AI model; Calculating a first hash of the trained first AI model; Packaging the trained first AI model, the first AI model metadata, and the first hash in a first container among a plurality of containers, each container among the plurality of containers including a corresponding single trained AI model, metadata corresponding to the same single trained AI model, and a hash of the same single trained AI model; registering the first container; providing the user with secure access to the first container; as well as Based on the first hash, the trained first AI model is validated for the user.

2. The system of claim 1 , wherein the processing circuit is further operable to: determining that the trained first AI model is further trained after creating the first AI model metadata; Creating second AI model metadata based on further training of the trained first AI model, where the second AI model metadata is based on the first transaction and a second transaction, the second transaction being associated with the further training of the trained first AI model; Creating a second hash for further training of the AI ​​model; as well as packaging the further trained AI model, the second AI model metadata, and the second hash in a second container; and Register the second container.

3. The system according to claim 1, wherein the first AI model metadata includes one or more of the following: AI model attributes, AI model training transaction information, AI model training data usage information, or AI model training underlying structure information.

4. The system of claim 1 , wherein the processing circuit is operable to: The first container is registered in a blockchain-based registry.

5. The system of claim 1 , wherein the processing circuit is further operable to: Creating a secure environment by restricting one or more of: egress and ingress of data to and from the environment, egress and ingress of the trained first AI model to and from the environment, data provider identity, model provider identity, or model changes; receiving a machine learning algorithm from a first provider; placing the machine learning algorithm in the secure environment; receiving a data set from a second provider; placing the dataset in the secure environment; Building a secure AI model in the secure environment based on the machine learning algorithm; Training the secure AI model in the secure environment based on the data set; and Creating secure AI model metadata based on a transaction associated with the secure AI model, wherein the transaction includes the building of the secure AI model and the training of the secure AI model, and wherein the secure AI model metadata indicates provenance information of the secure AI model.

6. A system comprising: communication circuitry operable to receive a data set, a machine learning algorithm, and an AI model from a provider; a security sandbox coupled to the communication circuit, the security sandbox operable to: Training multiple AI models and recording AI model metadata associated with training the multiple AI models; Serializing the plurality of trained AI models; For each serialized trained AI model in the plurality of serialized trained AI models, packaging the serialized trained AI model, associated AI model metadata, and a hash of the serialized trained AI model into a single corresponding container in a plurality of containers, each corresponding container including a corresponding single serialized trained AI model, metadata corresponding to the corresponding single serialized trained AI model, and a hash of the corresponding single serialized trained AI model; as well as A memory operable to store the dataset, the machine learning algorithm, the AI ​​model, the AI ​​model metadata, and a corresponding container.

7. The system of claim 6, wherein the secure sandbox comprises an experimental pipeline and an industrialized pipeline, wherein the experimental pipeline and the industrialized pipeline are operable to train the AI ​​model and record the AI ​​model metadata associated with training the AI ​​model.

8. The system of claim 6, wherein the communication circuitry is further operable to transmit at least one of the separate container or the AI ​​model metadata to a model registry.

9. The system of claim 8, wherein the communication circuit is further operable to: A proof of the lineage of the AI ​​model is received from the model registry.

10. A calculation method comprising: constructing, by the system and based on input from the user, a first AI model; training, by the system and based on input from the user, the first AI model; creating, by the system, first AI model metadata based on a first transaction associated with building and training the trained first AI model; calculating, by the system, a first hash of the trained first AI model; packaging, by the system, the trained first AI model, the first AI model metadata, and the first hash in a first container among a plurality of containers, each container among the plurality of containers including a corresponding single trained AI model, metadata corresponding to the same single trained AI model, and a hash of the same single trained AI model; registering the first container by the system; providing, by the system, the user with secure access to the first container; as well as Based on the first hash, the trained first AI model is verified by the system for the user.

11. The calculation method according to claim 10, further comprising: After creating the first AI model metadata, determining, by the system, that the trained first AI model is further trained; Creating, by the system and based on the trained first AI model being further trained, second AI model metadata based on the first transaction and a second transaction, the second transaction being associated with the further training of the trained AI model; creating, by the system, a second hash of the further trained AI model; as well as packaging, by the system, the further trained AI model, the second AI model metadata, and the second hash in a second container; and The second container is registered by the system.

12. The computing method according to claim 10, wherein the first AI model metadata includes one or more of the following: AI model attributes, AI model training transaction information, AI model training data usage information, or AI model training underlying structure information.

13. The computing method of claim 10, wherein registering the first container comprises: The first container is registered in a blockchain-based registry.

14. The method according to claim 10, further comprising: Creating a secure environment by the system by restricting one or more of: egress and ingress of data to and from the environment, egress and ingress of the trained first AI model to and from the environment, data provider identities, model provider identities, or model changes; receiving, by the system, a machine learning algorithm from a first provider; placing, by the system, the machine learning algorithm in the secure environment; receiving, by the system, a data set from a second provider; placing, by the system, the data set in the secure environment; Based on the machine learning algorithm, the system builds a secure AI model in the secure environment; Based on the data set, the system trains the secure AI model in the secure environment; and Based on a transaction associated with the secure AI model, secure AI model metadata is created by the system, wherein the transaction includes the building of the secure AI model and the training of the secure AI model, and wherein the secure AI model metadata indicates provenance information of the secure AI model.

15. A calculation method: The system receives data sets, machine learning algorithms, and AI models from providers; training, by the system and based on at least one of the datasets, at least one of the AI ​​models; recording, by the system, AI model metadata associated with training the at least one AI model; Serializing, by the system, a plurality of trained AI models; For each serialized trained AI model in the plurality of serialized trained AI models, the system packages the serialized AI model, associated AI model metadata, and a hash of the serialized trained AI model into a single corresponding container in a plurality of containers, each corresponding container including a corresponding single serialized trained AI model, metadata corresponding to the corresponding single serialized trained AI model, and a hash of the corresponding single serialized trained AI model; as well as The system stores the data set, the machine learning algorithm, the AI ​​model, the AI ​​model metadata and the corresponding container.

16. The computing method according to claim 15, wherein the system comprises an experimental pipeline and an industrialized pipeline, and the training and the recording are performed by at least one of the experimental pipeline or the industrialized pipeline.

17. The calculation method according to claim 15, further comprising: At least one of the separate container or AI model metadata is transmitted to a model registry.

18. The calculation method according to claim 17, further comprising: A proof of the lineage of the AI ​​model is received from the model registry.

Images