Orchestration of workloads with AI models

Abstract

This disclosure relates to a method comprising the step of receiving a request to run a workload using an artificial intelligence model. The current resource utilization status in a distributed system may be determined. The current resource utilization status may be used to define a deployment configuration of the artificial intelligence model, which is defined by: the number and structure of input blocks of the artificial intelligence model, the number and structure of output blocks, and intermediate blocks, a second computer system running the intermediate blocks, and one or more first computer systems running the input and output blocks. The artificial intelligence model may be deployed according to the defined deployment configuration and the workload may be run.

Description

[Technical Field]

[0001] The present invention relates to the field of digital computer systems, and more specifically to a method for executing workloads in a distributed system.

[0002] A radio access network (RAN) may provide access to resources across sites of a mobile telecommunication system and coordinate management of these resources in accordance with a protocol stack. A radio access network may provide processing resources that can be used, for example, for inferencing artificial intelligence (AI) models. However, there is a need to introduce security for inferencing using AI models that need to be processed efficiently on constrained computing environments. [Summary of the Invention]

[0003] Various embodiments provide a method, a computer program product and a system for executing a workload in a distributed system, as defined by the subject matter of the independent claims. Advantageous embodiments are defined in the dependent claims. Embodiments of the present invention can be freely combined with each other if they are not mutually exclusive.

[0004] In one aspect, the present invention provides a method for executing a workload in a distributed system using an artificial intelligence model, the distributed system comprising a set of first computer systems configured to connect to at least one second computer system of the distributed system, the artificial intelligence model being configured to receive a specific input, process the specific input, and provide a specific output, the artificial intelligence model being configured to be divided into a set of one or more input blocks, an intermediate block, and a set of one or more output blocks, whereby the set of one or more input blocks receives the specific input and provides an intermediate output, the intermediate block receives the intermediate output as input and provides another intermediate output, and the set of one or more output blocks receives the another intermediate output as input and provides the specific output. The method comprises an orchestration method, the orchestration method comprising: receiving a request to execute a workload using the artificial intelligence model, the workload comprising receiving the specific input; determining a current resource utilization status in the distributed system; defining a deployment configuration of the artificial intelligence model using the current resource utilization status, wherein the deployment configuration is defined by: the number and structure of input blocks of the artificial intelligence model, the number and structure of output blocks of the artificial intelligence model, and the intermediate block, the second computer system that executes the intermediate block, and the one or more first computer systems that execute the input and output blocks; and deploying the artificial intelligence model according to the defined deployment configuration and executing the workload.

[0005] In one aspect, the present invention relates to a computer program product comprising a computer-readable storage medium having computer-readable program code embodied therein, the computer-readable program code being configured to implement the method of the above embodiments.

[0006] In one aspect, the present invention relates to a computer system for executing a workload in a distributed system using an artificial intelligence model, wherein the distributed system comprises a set of first computer systems configured to connect to at least one second computer system of the distributed system, the artificial intelligence model configured to receive a specific input, process the specific input, and provide a specific output, and the artificial intelligence model is configured to be divided into one or more sets of input blocks, intermediate blocks, and one or more sets of output blocks, thereby the set of one or more input blocks receiving the specific input and providing an intermediate output, the intermediate block receiving the intermediate output as input and providing another intermediate output, and the set of one or more output blocks receiving the other intermediate output as input and providing the specific output. The computer system is configured to: receive requests to run a workload using the artificial intelligence model, the workload including receiving the specific input; determine the current resource utilization status in the distributed system; use the current resource utilization status to define a deployment configuration of the artificial intelligence model, where the deployment configuration is defined by: the number and structure of input blocks, the number and structure of output blocks, and the intermediate blocks, a second computer system running the intermediate blocks, and one or more first computer systems running the input and output blocks; and deploy the artificial intelligence model and run the workload according to the defined deployment configuration. [Brief explanation of the drawing]

[0007] Hereinafter, embodiments of the present invention will be described in more detail with reference to the following drawings, merely as examples.

[0008] [Figure 1] This is a block diagram of a wireless communication system, which is an example related to this subject.

[0009] [Figure 2] This is a flowchart illustrating an example of a method for executing a workload in a distributed system, related to this subject.

[0010] [Figure 3] This is a flowchart illustrating an example of a method for executing a workload in a distributed system, related to this subject.

[0011] [Figure 4] This is a signaling diagram illustrating a method for defining an expanded configuration related to an example of this subject.

[0012] [Figure 5] This figure shows an example of AI model partitioning related to this topic.

[0013] [Figure 6A] This figure shows a method for executing a workload in a distributed system, which is an example related to this subject.

[0014] [Figure 6B] This is a diagram of a distributed system that is an example of this subject.

[0015] [Figure 7] This is a diagram of a computing environment related to an example of this subject.

[0016] [Figure 8] This figure shows a cloud computing environment related to one embodiment of the present invention.

[0017] [Figure 9] This figure shows an abstraction model layer according to one embodiment of the present invention. [Modes for carrying out the invention]

[0018] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best describe the principles, practical applications, or technological improvements over commercially available technologies of the embodiments, or to enable other those skilled in the art to understand the embodiments disclosed herein.

[0019] An artificial intelligence model may be configured to perform a task. A task may refer to the type of prediction or inference being made. A task may be based on a problem or question being asked and the available data. A task may be, for example, a classification task, a clustering task, or a prediction task. For example, a classification task assigns data to categories, and a clustering task groups data according to similarity. An artificial intelligence model may perform a task by receiving input data, processing the input data using a set of learnable parameters, and providing an output representing the outcome of the task. An artificial intelligence model may be provided as a deep neural network, a transformer, or another artificial intelligence model that can be divided into multiple blocks as described herein.

[0020] An artificial intelligence model can be configured to receive an input X and provide an output Y. Assuming that the same output Y can be obtained if the same input X is used as input to a partitioned model, the partitioning of the artificial intelligence model can be performed to make up any number of blocks.

[0021] The division of an artificial intelligence model may be performed to obtain at least three blocks S1, S2, and S3, where S1 is the input block, S2 is the intermediate block, and S3 is the output block. In one example, the number of blocks may be increased by dividing the input block S1 into n1 input subblocks S1(a), S1(b)...S1(n1) and the output block S3 into n2 output subblocks S3(a), S3(b)...S3(n2), where n1 and n2 are greater than or equal to 2. This may result in a number of blocks equal to n1 + n2 + 1. In the following, for the sake of simplicity, each of the input subblocks S1(a), S1(b)...S1(n1) may be referred to as an input block. Similarly, each of the output subblocks S1(a), S1(b)...S1(n1) may be referred to as an output block. In other words, an artificial intelligence model can be divided into one or more input blocks, one or more output blocks, and one intermediate block.

[0022] The division of an artificial intelligence model can result in multiple blocks, each with its own structure. The structure of a block may refer, for example, to the types of inputs and outputs of the block, and the number and types of operations performed by the block. For example, if the artificial intelligence model is a deep neural network, the structure of a block may be defined by the number of layers belonging to the block, and the number of layers may represent a specific number and types of operations performed by the model.

[0023] The execution of an artificial intelligence model may be carried out according to an execution pipeline. An execution pipeline may consist of three or more consecutive execution stages, each configured to receive an input, process the input using a subset of learnable parameters, and provide an output. The input of one execution stage other than the first execution stage may be the output of a preceding execution stage. For example, an input block S1 may represent one or more first execution stages in the pipeline, an output block S3 may represent one or more last execution stages in the pipeline, and an intermediate block S2 may represent the remaining execution stages. For example, in the case of a deep neural network, an execution stage may represent the processing of one or more layers of the deep neural network. The processing performed for one network layer may include, for example, weighting operations, convolution operations, or activation operations. The AI ​​model may be divided into two cut layers, and the intermediate output may include, for example, cut layer activation values. In general, this model partitioning can be applied to various AI architectures such as CNNs, or other AI architectures such as Transformers, ResNet, LSTM, or AI models that can be executed according to the execution pipeline as described above.

[0024] Different workloads may use an artificial intelligence (AI) model to perform tasks assigned to it. A workload can refer to one or more software applications and the data accessed by those applications. For example, a workload may include multiple stages, one or more of which may include feeding input data into an AI model to obtain outputs associated with the input data. The outputs may be further used by other stages of the workflow; for example, if an AI model is trained to predict whether a communication channel is trustworthy, a workflow for data scheduling may use the AI ​​model to discover them in order to schedule communications over trusted channels. In another example, an AI model may be trained for user facial recognition, and the results of the recognition may be used to enable service access for the user.

[0025] These workloads can be efficiently performed in accordance with this subject using a distributed system. A distributed system includes multiple first computer systems remotely connected to one or more second computer systems. The first computer systems may be, for example, local computer systems accessible to users. The second computer systems do not have to be part of the first computer systems. The second computer systems are remote from the first computer systems. The first computer systems may be configured to connect to the second computer systems by wired and / or wireless digital data communications of any form or medium, such as a communications network. Examples of communications networks include local area networks (LANs), wireless access networks (RANs), metropolitan area networks (MANs), wide area networks (WANs), Worldwide Interoperability for Microwave Access (WiMAX), wireless local area networks (WLANs), all or part of the Internet, any other one or more communications systems in one or more locations, or a combination thereof.

[0026] An artificial intelligence model can be deployed using a given deployment configuration. The deployment configuration can be defined by: the number and structure of blocks in the artificial intelligence model, a second computer system that executes intermediate blocks, and at least one of one or more first computer systems that execute the input and output blocks of the artificial intelligence model. For example, in the case where the artificial intelligence model is divided into three blocks S1, S2, and S3, one or more first computer systems can process the input and output blocks S1 and S3. In the case where the artificial intelligence model is divided into a higher number of blocks (e.g., n1+n2+1 blocks) S1(a), S1(b)...S1(n1), S2, S3(a), S3(b)... and S3(n2), one or more first computer systems can process n1+n2 input and output blocks S1(a), S1(b)...S1(n1), S3(a), S3(b)... and S3(n2).

[0027] The deployment configuration is, for example, a set of three entries [{Blocks} i ,{CS1} i ,{CS2} i ] can be defined by, where the first entry {Blocks} i This refers to the blocks that make up the artificial intelligence model, and the second entry {CS1} i This refers to one or more first computer systems and each set of blocks that are deployed (or will be deployed) on them, and the third entry {CS2} i `{Blocks}0,{CS1}0,{CS2}0}` refers to the second computer system in which the intermediate block is being (or will be) unfolded. The subscript `i` indicates the point in time when the unfolded configuration is defined and used. For example, the initial unfolded configuration may be defined by the set of entries [{Blocks}0,{CS1}0,{CS2}0], the immediate subsequent unfolded configuration may be defined by the set of entries [{Blocks}1,{CS1}1,{CS2}1], and so on.

[0028] Defined deployment configuration [{Blocks} i ,{CS1} i,{CS2} i Deploying an artificial intelligence model in accordance with ] comprises splitting the artificial intelligence model (if it has not already been split accordingly) to obtain entry {Blocks} i blocks as defined by, deploying an intermediate block at a second computer system {CS2} i and deploying input and output blocks in a first set of computer systems as described at entry {CS1} i . The method may comprise the foregoing steps.

[0029] Therefore, the present subject matter may provide an accurate method for optimal execution of workflows using available resources. In contrast to existing techniques, the present subject matter not only can search for resources for executing a program, but also adapts the structure of the program itself to find an optimal combination of resources. The present example can find a trade-off between secure execution of an artificial intelligence model and available processing resources. This execution can be secure because the input and output of the AI model are processed locally at the first computer system, which can prevent problems such as model inversion attacks by malicious parties and reverse engineering attempts on confidential input data and / or output labels by either malicious parties and / or honest-but-curious servers, for example in the cloud.

[0030] The present subject matter may provide an advantageous technique for defining a deployment configuration using a current resource utilization status. The definition of the deployment configuration may be further based on an estimate of resources required by each processing stage or processing step of the AI model. The resources required by each block of the AI model may be referred to as required block resources.

[0031] In one configuration definition example, the defined deployment configuration could be the initial deployment configuration, which means that the AI ​​model's deployment configuration has never been defined or used before. In this case, the set of entries [{Blocks} i ,{CS1} i ,{CS2} i All entries in ] may need to be defined. That is, the AI ​​model may be divided into multiple blocks, and first and second computer systems may be selected to deploy them.

[0032] In one example of a configuration definition, the defined deployment configuration is not the initial deployment configuration because the AI ​​model has been previously deployed and used according to an existing deployment configuration. In this case, there are alternative forms for defining the deployment configuration. In one example, the deployment configuration can be defined using resource utilization status and without using an existing deployment configuration. That is, the AI ​​model may be divided into multiple blocks, and first and second computer systems may be selected to deploy them. Alternatively, the deployment configuration can be defined using resource utilization status and further using an existing deployment configuration. For example, deployment configuration [{Blocks} i ,{CS1} i ,{CS2} i ] represents the current resource utilization status of the distributed system and the existing deployment configuration [{Blocks} i-1 ,{CS1} i-1 ,{CS2} i-1 It can be defined using ]. For the definition of a deployment configuration, an existing deployment configuration [{Blocks} i-1 ,{CS1} i-1 ,{CS2} i-1 One or more entries in ] may be changed, for example, a new deployment configuration may have the same number and structure of blocks as an existing deployment configuration, but with a different deployment system: [{Blocks} i ={Blocks} i-1 ,{CS1} i ≠{CS1} i-1 ,{CS2} i ≠{CS2} i-1] may have, where the sign ≠ indicates a difference. Alternatively, a new expansion construct can be provided as a function of an existing expansion construct, as follows: [{Blocks} i ={Blocks} i-1 ,{CS1} i ={CS1} i-1 ,{CS2} i ≠{CS2} i-1 ], [{Blocks} i ={Blocks} i-1 ,{CS1} i ≠{CS1} i-1 ,{CS2} i ={CS2} i-1 ], [{Blocks} i ≠{Blocks} i-1 ,{CS1} i ≠{CS1} i-1 ,{CS2} i ≠{CS2} i-1 ], [{Blocks} i ≠{Blocks} i-1 ,{CS1} i ={CS1} i-1 ,{CS2} i ≠{CS2} i-1 ], [{Blocks} i ≠{Blocks} i-1 ,{CS1} i ≠{CS1} i-1 ,{CS2} i ={CS2} i-1 ], or [{Blocks} i ={Blocks} i-1 ,{CS1} i ={CS1} i-1 ,{CS2} i ={CS2} i-1 ]. The last defined deployment configuration option indicates that the existing deployment configuration will be maintained. This may be the case if the current resource utilization status indicates that the existing deployment configuration is still valid and can be used further.

[0033] In one example of a configuration definition, the definition of an expanded configuration may include a first step of initializing a set of blocks into an initial set of blocks. The initial set of blocks may be the set of blocks last used in the case of updating an existing expanded configuration. In the case of an expanded configuration being a first configuration, the initial set of blocks may be (roughly) defined; for example, the initial set of blocks may be defined randomly. Following the notation above, the expanded configuration [{Blocks} i ,{CS1} i ,{CS2} i The definition of ] is a set of blocks {Blocks} if i > 0. i The set of blocks immediately preceding {Blocks} i-1 It can be started by initializing to {Blocks}; otherwise, a set of blocks {Blocks} i This is a roughly determined set of blocks {Blocks} -1 It can be initialized to [{Blocks}]. After this initialization, the expanded configuration [{Blocks} i ,{CS1} i ,{CS2} i Alternative techniques may be used to discover [the block]. In one example, a first and second computer system may be selected that can execute an initial set of blocks. Alternatively, the first and second computer systems may be defined first, and the set of blocks may be changed if the initial set of blocks cannot be deployed on the defined systems. Alternatively, the initial set of blocks may be updated, and the existing first and second computer systems may be maintained if they can execute the updated set of blocks; otherwise, a new configuration of the computer systems may be determined. Alternatively, the initial set of blocks may be updated, and a new configuration of the first and second computer systems may be determined.

[0034] In one configuration definition example, the current resource utilization status may be determined by evaluating a set of resource parameters. The set of resource parameters may include, for example, the hardware specifications of each of the first computer systems, for example, the resource parameters may be provided as a data structure, e.g., a vector, where each entry includes the hardware specifications of each of the first computer systems. The set of resource parameters may include, for example, the computational status of each computer system. The set of resource parameters may include, for example, link reliability in a distributed system. The set of resource parameters may include, for example, latency in a distributed system. The set of resource parameters may include, for example, data transfer rate in a distributed system. A deployment configuration may be defined using one or more rules, each of which may map a specific set of resource parameters to the deployment configuration. For example, if the utilization level of the first computer system CS1_0 is less than 10% and the second computer system belongs to a private cloud, the deployment configuration may be defined as a set of three blocks S1, S2, and S3, where the required block resources for the input and output blocks S1 and S3 can be provided by the first computer system CS1_0, and the required block resources for the main block S2 can be provided by the second computer system. In one example, the rules may be provided using reinforcement learning. The reinforcement learning action may determine the deployment configuration based on a set of resource parameters and the resources required by the AI ​​model.

[0035] Each of the configuration definition examples described above may result in one or more candidate deployment configurations that satisfy the current resource utilization status. In one example, the deployment configuration may be defined by randomly selecting one of the candidate deployment configurations. Alternatively, a performance analysis of the candidate deployment configurations may be performed, the candidate configurations may be ranked based on their performance, and the deployment configuration may be selected based on the ranking; for example, the highest-ranked candidate deployment configuration may be the one used to deploy the AI ​​model. Alternatively, a voting system and / or A / B testing may be used to select one of the candidate deployment configurations.

[0036] In one example, the processing resources required by each block in an artificial intelligence model can be estimated. The current resource utilization status of a distributed system can be used to identify first and second computer systems that can execute the blocks, given the estimated processing resources.

[0037] According to one example, before executing the orchestration method, the method includes deploying an artificial intelligence model according to an initial deployment configuration, and defining the deployment configuration includes updating the initial deployment configuration. The update includes: subdividing the artificial intelligence model; adding one or more first computer systems to execute input and output blocks; removing one or more of the initial first computer systems that execute input and output blocks; or selecting another second computer system to execute intermediate blocks.

[0038] According to one example, defining a deployment configuration using the current resource utilization status includes: performing capacity profiling of the first and second computer systems to determine whether each of the first and second computer systems can run one or more blocks of the artificial intelligence model; and defining the deployment configuration based on the capacity profiling.

[0039] According to one example, the current resource utilization status is defined by: the utilization level of network resources in a distributed system; the resource utilization level of a first computer system; the resource utilization level of a second computer system; or at least one of the resources in each of the first and second computer systems.

[0040] In one example, the deployment configuration is further defined using rules. The rules require at least one of the following: the maximum number of first computer systems that will be used to deploy the input and output blocks of the AI ​​model; or the maximum number of blocks of the artificial intelligence model. Each of the configuration definition examples described above may use rules for defining the deployment configuration.

[0041] In one example, the distributed system is a wireless communication system, the first computer system is a multi-access edge computing (MEC) node, and the second computer system is a cloud system. The second computer system may be part of a public cloud, or a private cloud or a hybrid cloud. For example, multiple public clouds, private clouds, and hybrid clouds may be provided, and each of these clouds may provide resources for the second computer system (for example, a cloud may provide the second computer system as a virtual machine). For example, a specific cloud may be selected first for the definition of a deployment configuration, and then the second computer system provided by the selected cloud may be used for the deployment configuration. Multiple clouds may be provided by the same cloud service provider or by different cloud service providers.

[0042] According to one example, the method involves: deploying an artificial intelligence model via a broadcaster in a distributed system; further including the broadcaster sending information to an orchestrator in the distributed system, so that when the orchestrator receives the information, it triggers the execution of the workload.

[0043] According to one example, an artificial intelligence model may be provided as a neural network (e.g., a deep neural network), a transformer, or any other artificial intelligence model that can be divided into multiple blocks as described in the methods herein, and as a corresponding architecture (e.g., in terms of parallelization, distribution). An input block may represent a first network layer, an intermediate block may represent an intermediate network layer, and an output block may represent a final network layer.

[0044] One example suggests that the workload involves at least one of the following: data analytics, sensor measurement fusion from different sources, image analysis, or processing of data streams heading to the cloud.

[0045] Artificial intelligence models can be partitioned using different techniques. In one first partitioning example, the partitioning of the AI ​​model may include a step of determining or identifying the execution pipeline. In one example, the execution pipeline may be provided in relation to the AI ​​model, for example, it may be predefined in a metadata file in relation to the AI ​​model. In this case, the metadata file may be read to extract the execution pipeline. Alternatively, the execution pipeline may be determined automatically using (e.g., by parsing and interpreting) the code that implements the AI ​​model. In this first partitioning example, the partitioning may further include a step of allocating the execution stages of the determined execution pipeline to a desired number of blocks of the AI ​​model (e.g., three blocks S1, S2, and S3 or more). In one example, the allocation may be done by randomly assigning the execution stages to each block. This may be advantageous in cases where the first and second computer systems of the deployment configuration have not yet been selected. In another example, the allocation may be performed based on the available resources in the first and second computer systems of the deployment configuration and based on the resources required for the execution stages of the pipeline. For example, the metadata file may further include estimates of the processing resources required by each execution stage of the pipeline. Alternatively, the processing resources required by each execution stage may be estimated using, for example, the number of lines of code and the type of command used in each execution stage.

[0046] In one second example of splitting, the splitting of an artificial intelligence model may involve dividing the code that runs the AI ​​model into three or more desired blocks, based on the programming language being used. This can be done by parsing and interpreting the code. The number of lines of code and the type of command may indicate the resources required by each block.

[0047] One third partitioning example is reinforcement learning, where a set of states, actions, and rewards is defined and learned in a (deep) neural network to achieve rules for discovering the best partitioning policy for the AI ​​model concerning the current first and second computer systems.

[0048] Executing a workflow involves running an AI model once or multiple times. Running an AI model may involve providing inputs to the AI ​​model and receiving outputs to the AI ​​model. According to one example, running an AI model involves: for each of two consecutive blocks of an AI model deployed on different systems: using an encoding protocol to encode the output of the first of the two blocks, and sending the encoded output to a first or second computer system to be used as input for the second of the two blocks. In one example where the AI ​​model is divided into more than three blocks, if input blocks S1(a) and S1(b) are each deployed on two first computer systems, the output of input block S1(a) may be encoded by the encoding protocol, and the encoded output may be sent to the first computer system where S1(b) is deployed. The output may be decoded at the receiving first computer system using the encoding protocol and then used as input for block S1(b). In another example where the AI ​​model is divided into three blocks, the output of input block S1 may be encoded by an encoding protocol, and the encoded output may be sent to a second computer system where S2 is being deployed. The output may be decoded in the second computer system using the encoding protocol and then used as input for block S2. Similarly, the output of main block S2 may be encoded by an encoding protocol, and the encoded output may be sent to a first computer system where input block S3 is being deployed. The output may be decoded in the first computer system using the encoding protocol and then used as input for block S3.

[0049] According to one example, the encoding protocol includes at least one of compression or encryption, and the second encoding protocol includes at least one of compression or encryption.

[0050] An encoding protocol may define how to encode the original data, how to obtain the encoded data, and how to decode the corresponding data, which allows the original data to be restored from the encoded data. Data encoding may include any one of the following: encryption, compression, ciphering, formatting, or assigning or interpreting a specific bit pattern to that data. This can make the transmission of data secure. Alternatively or in addition, this can ensure the efficient use of network resources, for example, because compression can reduce the data size. For example, in cases where encoding is performed by compressing the output, decoding of the compressed output is performed by decompressing the compressed output. In cases where encoding is performed by encrypting the output, decoding of the encrypted output is performed by decrypting the encrypted output.

[0051] This topic may offer the following advantages: It may introduce data retention behaviors applicable to enable artificial intelligence models to run efficiently on constrained computing devices. It may ensure an optimal data transmission size to ensure efficient use of network resources. It may adjust data retention behaviors to accommodate changes in computing resource availability on constrained computing devices. It may dynamically schedule model partitioning ratios for artificial intelligence models to distribute inference tasks across a constrained first computer system (e.g., edge computing devices) and a second computer system (e.g., cloud server instances). It may introduce security for distributed inference using large and complex artificial intelligence models (e.g., foundation models) that can be efficiently processed on constrained computing environments such as edge computing and Internet of Things (IoT) devices.

[0052] In one example, one or more output blocks may be removed from the first computer system before one or more input blocks are executed. For example, a management server may distribute input blocks, intermediate blocks, and output blocks between the first and second computer systems, respectively, ensuring that the first computer system can process either input blocks or output blocks at any given time instance to ensure maximum utilization of hardware resources. For example, an input block configuration may be removed after execution, encoding, and transmission to free up computing power for output blocks. The management server may be configured to connect to the first and second computer systems and to control the operation of the first and second computer systems.

[0053] In one example, after one or more input blocks have been executed, one or more output blocks may be deployed to a first computer system. For example, one or more output blocks may be downloaded from a management server after the execution of one or more input blocks. This can further improve resource utilization of the first computer system by saving processing resources that would otherwise be used to maintain the output blocks when they are not in use.

[0054] In one example, after executing one or more input blocks, the input blocks may be deleted, and one or more output blocks may be deployed to the first computer system. For example, one or more output blocks may be downloaded from a management server after the execution of one or more input blocks. This can further improve the resource utilization of the first computer system, as only one block type can be processed and managed by the first computer system at a time.

[0055] In one example, the execution of an artificial intelligence model involves the execution of a series of processing stages, and the division of the artificial intelligence model is performed such that the input blocks are configured to execute a first consecutive processing stage of a first number (N1), and the output blocks are configured to execute the last consecutive processing stage of a third number (N3), and the intermediate blocks are configured to execute a second consecutive processing stage of a second number (N2) following the first processing stage of the input blocks, and the sum of the first number, the second number, and the third number is the total number of processing stages in the artificial intelligence model, i.e., N1 + N2 + N3 is the number of processing stages in the artificial intelligence model. The execution stages of a previously defined model may include one or more processing stages of the model. In the case where the input blocks are further divided into multiple input blocks, N1 refers to the first consecutive processing stage executed by all input blocks. Similarly, in the case where the output blocks are further divided into multiple output blocks, N3 refers to the last consecutive processing stage executed by all output blocks.

[0056] In one example, the first number N1 is less than the second number N2 by the first delta value, where the third number N3 is less than the second number N2 by the second delta value. For example, N2-N1<Δ1 and N2-N3<Δ2, where Δ1 is the first delta value and Δ2 is the second delta value. The first and second delta values ​​are positive integers, and Δ1>0 and Δ2>0. In one example, the first and second delta values ​​may be user-defined values. This example can enable systematic and uniform processing of the artificial intelligence model regardless of the available resources. This can be particularly advantageous in cases where the first computer system has sufficient resources to process even the entire artificial intelligence model locally.

[0057] The intermediate block may be referred to as the main block because it may contain the majority of the processing stages of the artificial intelligence model.

[0058] In one example, the method further includes determining a first number N1 and a third number N3 based on the available resources in a first computer system. Alternatively, instead of determining N1 and N3, delta values ​​Δ1 and Δ2 may be determined in this example. The determination may be performed, for example, such that numbers N1 and N3 are as high as possible given the available resources.

[0059] For example, given a set of hardware specifications for a first computer system, such as RAM, CPU clock, and cache, and given the expected use of these specifications regarding the computational complexity of an artificial intelligence model, the numbers N1 and N3 (or Δ1 and Δ2) can be accurately estimated. This example may allow for the highest possible number of processing steps performed by input and output blocks, where higher numbers N1 and N3 make the execution of the artificial intelligence model more secure. This is because processing is performed locally and the output of the input block may be unpredictable.

[0060] In one example, the artificial intelligence model is a trained model. In this case, the execution of the artificial intelligence model using this method is the inference of the artificial intelligence model.

[0061] In one example, the first computer system has a smaller amount of processing resources than the second computer system. The second computer system could be any computer system that has processing resources to execute any defined intermediate block of the artificial intelligence model. For example, the second computer system could be any computer system that has processing resources to execute the entire artificial intelligence model.

[0062] In one example, the second computer system is offered as a service in a cloud computing environment. In another example, the second computer system may be offered as a cloud instance in a cloud computing environment. A cloud instance can be a server resource provided by a cloud service. In yet another example, the second computer system may be implemented using one or more functional abstraction layers provided by the cloud computing environment; for example, the hardware and software resources of the second computer system may be provided by the hardware and software layers of the cloud computing environment. The workload layer of the cloud computing environment may be used, for example, to implement the stages that will be performed by the second computer system. The cloud computing environment may remain unaware of any data and output labels because it does not possess a complete artificial intelligence model, and external model inversion and / or reverse engineering attacks can be mitigated by secure model coding, thus preserving the data. Inference can thus be performed exclusively and securely on edge devices, and the cloud environment is used as a pure computing and processing instance that has no knowledge of specific use cases and inference results on edge devices.

[0063] One example of an artificial intelligence model is a foundation model. A foundation model can be a large-scale artificial intelligence model that is trained on a massive amount of data, resulting in a model that can adapt to a wide range of downstream tasks. Examples of foundation models include Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer n series (GPT-n series). The first and last foundation model (FM) layers are processed on-device, and their intermediate cut layer activation values ​​are securely transmitted (received) by applying compression (decompression) while ensuring communication-efficient low-bandwidth transmission.

[0064] In one example, the artificial intelligence model is a deep neural network, where the input block represents the first network layer, the intermediate block represents the intermediate network layer, and the output block represents the final network layer. Following this example, each processing stage of the artificial intelligence model can represent the processing performed for each layer of the deep neural network. That is, the input block contains N1 first layers of the deep neural network, the intermediate block contains N2 layers of the deep neural network, and the output block contains N3 final layers of the deep neural network, where the total number of layers in the deep neural network is N1 + N2 + N3.

[0065] In one example, the first computer system is any one of the following: an edge device (e.g., a MEC node), user equipment (UE), or an Internet of Things (IoT) device. This example can be seamlessly integrated into a wireless or mobile communication system. The mobile communication system provides wireless connectivity to users. Users may include, for example, mobile devices, tablets, laptops, or individuals. The mobile communication system may include a radio access network (RAN) and a core network. The core network may provide Internet Protocol (IP) connectivity to the RAN. The RAN may manage the radio spectrum of users using radio devices such as base stations. The RAN may enable the processing of packets according to a processing pipeline. The processing pipeline has different layers. The layers include a baseband processing layer and a radio frequency (RF) processing layer. The baseband processing layer may be defined according to a protocol stack and may be executed by a baseband unit, which is included in the edge device.

[0066] A baseband unit may be associated with one or more base stations. For example, each of one or more base stations may serve users located within the geographical area or cell of the base station's service. A baseband unit may process baseband signals for users served by one or more base stations. Therefore, a baseband unit is said to be serving those users. A baseband unit may implement layers of the protocol stack, such as the Packet Data Convergence Protocol (PDCP) layer, the Radio Link Control (RLC) layer, the Medium Access Control (MAC) layer, and the Physical (PHY) layer. In one example, a baseband unit may be divided into functional entities, each configured to perform its own function, for example, one function may implement one or more layers of the stack protocol. For example, a baseband unit may be divided into two functional entities named a Centralized Unit (CU) and a Distributed Unit (DU). CU may provide support for higher layers of the protocol stack, such as the PDCP layer, while DU may provide support for lower layers of the protocol stack, such as the RLC, MAC, and physical layers.

[0067] The implementation of the baseband unit can be achieved using a specific hardware and software configuration of the edge device. The software configuration of the baseband unit may include an operating system and software modules for performing the functions of the baseband unit. In addition, the software configuration may indicate one or more vendors providing that software configuration. For example, the operating system and software modules may be provided by one or more vendors. The hardware configuration may include storage resources, data communication resources, and processing resources. In addition, the hardware configuration may indicate one or more vendors providing that hardware configuration. These resources may be provided by one or more vendors.

[0068] Figure 1 shows a diagram of a wireless communication system related to an example of this subject.

[0069] The wireless communication system 100 includes a core network 101 and a radio access network 102. The radio access network 102 may include, but is not limited to, remote radio components 107 equipped by base stations 109 and 111. Each base station 109 or 111 may include a remote radio unit (RRU) having an antenna, which may serve the UE 120 in the respective cells 121 and 122. The radio access network 102 may further include a first computer system 103. For the sake of simplicity, only three first computer systems are shown, but is not limited to them. Also, for the sake of simplicity in the drawings, only the components of one first computer system are described.

[0070] The first computer system 103 may include, for example, one or more sets of baseband units (BBUs) 105.1~n. The baseband units may be connected to their respective RRUs in the remote radio component 107 via fiber or cable 113. The first computer system 103 may be configured to connect to the core network 101 via a backhaul link 115. The first computer system 103 may include a central unit 117 configured to control the operation and deployment of the baseband units 105.1~n. Each of the first computer systems 103 may be provided as, for example, a MEC node. The MEC node can improve user services (e.g., with low latency). The first computer system 103 may process data provided by the BBUs, for example, using advanced techniques for image analysis. The radio access network 102 may include a control unit 110 for managing workloads in the wireless communication system 100.

[0071] The remote wireless component 107 and the first computer system 103 may be configured to connect to a cloud computing environment 130. The cloud computing environment 130 may include a second computer system 131. In one example, the second computer system 131 may be provided as a cloud instance in the cloud computing environment 130.

[0072] In one exemplary implementation, the cloud computing environment 130 may be provided as described, for example, with reference to Figures 8 and 9. For example, a second computer system 131 may be implemented using one or more functional abstraction layers provided by the cloud computing environment 130, for example, the hardware and software resources of the second computer system 131 may be provided by the hardware and software layers of the cloud computing environment 130. The workload layer of the cloud computing environment 130 may be used, for example, to implement the execution of the main block of the AI ​​model by the second computer system 131.

[0073] In one exemplary implementation, system 100 may be provided as an Open Radio Access Network (O-RAN), where the first computer system 103 may be located at one or more edge sites, and the remote radio component 107 may be located at one or more cell sites.

[0074] Figure 2 is a flowchart of a method for executing an artificial intelligence model, as an example of the subject matter. For illustrative purposes, the method in Figure 2 may be implemented in the system shown in Figure 1, but is not limited thereto. The method may be executed, for example, by the control unit 110.

[0075] In step 201, a request may be received to run a workload using the artificial intelligence model. The workload may include a step in which specific inputs will be used as inputs to the artificial intelligence model. In step 203, the current resource utilization status in the distributed system may be determined. The current resource utilization status may be used in step 205 to define the deployment configuration of the artificial intelligence model. The deployment configuration may be defined by: the number and structure of input blocks of the artificial intelligence model, the number and structure of output blocks, and intermediate blocks, a second computer system that runs the intermediate blocks, and one or more first computer systems that run the input and output blocks. The artificial intelligence model may be deployed in step 207 according to the defined deployment configuration. The workload may be run in step 209.

[0076] Figure 3 is a flowchart of a method for executing an artificial intelligence model, as an example of the subject matter. For illustrative purposes, the method in Figure 3 may be implemented in the system shown in Figure 1, but is not limited thereto. The method may be executed, for example, by the control unit 110.

[0077] In step 301, a request may be received to run a workload using the artificial intelligence model. The workload may include a step in which specific inputs will be used as inputs to the artificial intelligence model. In step 303, the current resource utilization status in the distributed system may be determined. In step 304, it may be determined whether the artificial intelligence model has already been deployed according to an existing deployment configuration. If the artificial intelligence model has not yet been deployed, the current resource utilization status may be used in step 305 to define the deployment configuration of the artificial intelligence model. If the artificial intelligence model has already been deployed, the current resource utilization status may be used in step 306 to update the existing deployment configuration of the artificial intelligence model. The deployment configuration may be defined by: the number and structure of input blocks of the artificial intelligence model, the number and structure of output blocks, and intermediate blocks, a second computer system that runs the intermediate blocks, and one or more first computer systems that run the input and output blocks. The artificial intelligence model may be deployed in step 307 according to the defined deployment configuration. The workload may be run in step 309.

[0078] Figure 4 is a signaling diagram illustrating a method for defining the deployment configuration of an artificial intelligence model, as an example of this subject. The method in Figure 4 may provide an illustrative implementation of step 205 in Figure 2 and step 305 in Figure 3. The method may be performed, for example, by a partitioning orchestrator and a partitioning modification unit. The partitioning orchestrator and the partitioning modification unit may be, for example, part of a control unit 110.

[0079] In step 401, the partition orchestrator may receive a request to define a deployment configuration for an AI model. In step 403, the partition orchestrator may collect a set of resource information regarding the current resource utilization status of the distributed system. The resource information may include hardware specifications such as RAM, CPU, memory, and cache for the first and second computer systems. The resource information may further include computational status in the distributed system, for example, in terms of hardware utilization and node occupancy. The resource information may further include network conditions such as link reliability, latency, and data transfer rate between the first computer systems and between the first and second computer systems. In step 405, the partition orchestrator may perform a capacity profiling method to evaluate the capacity of each first computer system to run several existing AI model blocks given the current context. Using the collected resource information and the results of the capacity profiling, the partition orchestrator may, in step 407, determine whether or not to update the existing deployment configuration. In cases where it is determined that an existing deployment configuration should be updated, the partition orchestrator may, in step 409, send a request to the partition modification unit to modify the partition of the AI ​​model. The request may be sent with information (named partition information), which may be prepared by the partition orchestrator. The partition information may include, for example, at least a portion of resource information. In one example, the partition information may include a set of computation and network conditions for all first computer systems, issuing node ratings for current usage and computation / operational environments to determine a preferred list of the best first computer systems that qualify for further considerations.

[0080] The partition modification unit may use the partition information to discover a new deployment configuration in step 411. In particular, the partition modification unit may discover a new set of AI model blocks. The partition modification unit may further use one or more rules to discover the deployment configuration. The rules may include, for example, specific Quality of Service (QoS) requirements that will be considered for optimal model repartitioning. The rules may further require a desired trade-off between energy efficiency (EE) and the number of cut layers CL per model (i.e., the number of blocks). The partition modification unit may send the new deployment configuration to the partition orchestrator in step 413. The partition orchestrator may invoke a model deployment method in step 415 to distribute the new partition model across the first and second collaborating computer systems.

[0081] Figure 5 shows a method for dividing a foundation model. The foundation model may be divided into three blocks. As shown, the first computer system 601 may include an input block 604 and an output block 606, while the remote second computer system 602 includes a main block 605. Input data 603 may be received in the first computer system 601 and processed by the input block 604. The output of the input block 604 may be processed by the main block 605 in the second computer system 602. The output of the main block 605 may then be processed by the output block 606 to obtain the inference result 607 of the input data 603.

[0082] Figure 6A shows an example of a method for workload orchestration in a distributed system, as shown in Figure 6B, relating to this subject.

[0083] The distributed system 700 may include a wireless communication system including nodes 701.1, 701.2, and 701.3, such as 5G base stations. Nodes 701.1, 701.2, and 701.3 may serve user equipment such as sensors in the geographical area of ​​their respective services. Nodes 701.1, 701.2, and 701.3 may be associated with MEC nodes 703.1, 703.2, and 703.3, respectively. MEC nodes 703.1, 703.2, and 703.3 may be enabled to process data provided by nodes 701.1, 701.2, and 701.3, respectively. MEC nodes 703.1, 703.2, and 703.3 may be configured to process data using resource virtualization. The distributed system 700 may further include a cloud system 704. Cloud system 704 may enable cloud services for MEC nodes 703.1-703 and may provide resources using virtualization techniques.

[0084] A distributed system 700 may be used to run a workload, which may involve running an artificial intelligence model 705. The artificial intelligence model 705 may be configured to be divided into several blocks, including at least three blocks: an input block, a main block, and an output block. In this example, the artificial intelligence model 705 may be initially divided and deployed into three blocks S1, S2, and S3. For example, the input and output blocks S1 and S3 are deployed on the MEC node 703.3, and the main block S2 is deployed on the cloud system 704. Each of these three blocks may be referred to as a partitioned model.

[0085] Figure 6A shows method 706 for workload orchestration in a distributed system. Upon receiving a workload 707, method 706 begins in step 708 by collecting resource information about the computing and operating environment from all MEC nodes 703.1, 703.2, and 703.3, and optionally from a cloud system 704. This may be done using stored information 709 on the distributed system. In step 710, the workload 707 may be classified by determining relevant characteristics of the workload, such as the CPU required to run the workload, whether the workload is critical, etc. In step 711, a deployment configuration may be determined for running the workload. For example, the determined deployment configuration may be an existing deployment configuration (i.e., an existing one is maintained) or a new deployment configuration. The entire workload may be deployed in step 712 using the deployment configuration and then run, or only a portion of the workload may be deployed in step 713 using the deployment configuration and then run.

[0086] Step 711 may be performed using optimization method 720. Optimization method 720 may include split orchestration (SO) method 721, split revision (SR) method 725, and reconfiguration broadcast (RB) method 730.

[0087] The partition orchestration method 721 may organize workload distribution based on the placement of partition models across MEC nodes 703.1 to 703.3. The partition orchestration method 721 may be executed by a partition orchestrator. The partition orchestrator may determine whether the inference of blocks S1 and / or S3 should be performed entirely on the MEC edge node 703.3, or whether it needs to be collaboratively distributed among node 703.3 and other candidate nodes, taking into account a given evaluation determined by the capacity profiling (CP) method 723 and several additional constraints such as a maximum MEC node utilization threshold. The capacity profiling method 723 may evaluate the capacity of Node-X to run some partition model partition S_X* given the current context. Given information regarding the computation and network conditions across connected MEC nodes, CP Method 723 may evaluate whether Node-X can process a partitioned model partition S_X* for a given non-inference-related Workload-Z under the current computing and operating environment. In practice, the workload may consist of existing AI workloads from other services / applications and non-AI workloads such as BBU processing or some analytics for a mobile network operator. In this case, the method aims to collect both types of workloads and determine whether the AI ​​workload (partition S_X*) is suitable for this Node-X.The determination of a partition orchestrator may be made given at least one of the following collected resource information i1) to i6): i1) a set of hardware specifications of the MEC node, such as RAM, CPU, memory, and cache; i2) information on computation status (e.g., hardware utilization, node occupancy); i3) information on network status (e.g., link reliability, latency, data transfer rate); i4) an evaluation of whether the inference of existing partition models S1 and / or S3 can be processed on the MEC node, as provided by capacity profiling (CP) method 723; i5) an equivalent set of information on several candidate nodes in the connected MEC domain; i6) further information on existing application load orchestration mechanisms such as Kubernetes clusters and / or MEC-related services, and their respective API endpoints. Depending on its decision, the split orchestrator may invoke the continuation of inference on the MEC edge node 703.3 according to the previous model split S=(S1,S2,S3), or invoke the split revision method 725 to initiate a Model Split Revision Process (MSRP) to determine a better (or best) model split S*=(S1*,S2*,S3*,...). If the split revision method 725 needs to be invoked, the split orchestrator provides the above set of compute and network conditions for all candidate nodes and issues node ratings regarding current usage and compute / operational environments to determine a preferred list of the best MEC nodes that qualify for further considerations. After the MSRP has been processed, the split orchestrator invokes the model deployment (MD) method 731 to distribute the new split inference workload across the collaborating MEC nodes and potentially within the existing application workload orchestration infrastructure. This is shown in Figure 6B, where the new partitioning model includes input blocks S1(a) and S1(b) and output blocks S3(a) and S3(b), in addition to the main block S2.Blocks S1(a) and S3(a) are deployed on MEC node 703.2, and blocks S1(b) and S3(b) are deployed on MEC node 703.1.

[0088] The partitioning of the AI ​​model may be performed, for example, as follows: The partition correction method 725 may invoke the Model Partition Correction Process (MSRP) to repartition and redistribute the model partition across the MEC nodes. The best model partition S*=(S1*,S2*,S3*) may be selected according to rule-based decisions such as the computational and operational conditions specific to the MEC node, a threshold for the maximum number of collaborating MEC nodes, or potentially rules within the existing workload orchestration infrastructure. The best model partition may be selected given at least one of the following partition information j1) to j5) that can be stored in storage 722: j1) the original FM model partition S=(S1,S2,S3) and the computational complexity of each model partition; j2) a preferred list of the best MEC nodes in terms of computation and operational status, provided by the partition orchestrator; j3) additional constraints provided by the partition orchestrator and MEC node-dependent QoS requirements that will be considered for optimal model repartitioning; j4) a desired trade-off between energy efficiency (EE), the number of cut layers CL per model partition and the total number of partitions (TP) (e.g., S1, S2, S3, ..., S_TP); j5) a set of available online and offline algorithms for achieving model partitioning with respect to the EE, CL, and TP trade-offs, provided by the model repartitioning (MR) method 726.This MR method may be implemented, for example, as follows: a rule-based engine is implemented with rules loaded that correlate certain specific parameters (e.g., memory / CPU capacity and load / idle state, processing power, context, edge profiling, edge load prediction, etc.) with actions, and may be extended by a) a learning scheme (e.g., reinforcement learning) to add to the rules mentioned above, along with a method for evaluating its performance; b) a set of functions for preprocessing incoming information and postprocessing the obtained results for further considerations in the optimization process; c) a voting system that allows multiple rule-based processes to compete for the best action to take, where the action is to discover an AI model deployment configuration; and d) a speculative evaluation system (e.g., a / b testing) that allows experimentation and evaluation of new rule configurations. Using steps a) and b) may result in multiple actions or multiple candidate deployment configurations for the AI ​​model. Steps c) and / or d) may be used to select one of the candidate deployment configurations. In stage a), reinforcement learning may enable a set of states, actions, and rewards defined and learned in a (deep) neural network to achieve rules for discovering the best partitioning policy for the current system. In stage b), the set of features may be a set of deterministic features based on expert knowledge, experience, and infrastructure / architectural knowledge of the deployed system. Alternatively, certain heuristics, such as those used for task scheduling and load distribution (e.g., simulated annealing), may be used instead of a rule-based engine.

[0089] The reconfiguration broadcast method 730 may broadcast the redistributed partition and sub-partition models across MEC nodes. The method may redistribute the new partition S* to selected MEC nodes in the MEC region, unfold the model partitions via the model deployment (MD) method 731, synchronize information about the current partition ratio and the environment of the participating MEC nodes, signal this to the partition orchestrator, and then start the inference process. This may be performed using the adapted FM model partitions S*=(S1*,S2*,S3*,...) provided by the MSRP and selected MEC edge nodes (X1,X2,X3,...) to perform the partition inference described above.

[0090] This topic may include the following items:

[0091] Item 1. A method for executing a workload in a distributed system using an artificial intelligence model, wherein the distributed system comprises a set of first computer systems configured to connect to at least one second computer system of the distributed system, the artificial intelligence model is configured to receive a specific input, process the specific input, and provide a specific output, the artificial intelligence model is configured to be divided into one or more sets of input blocks, intermediate blocks, and one or more sets of output blocks, thereby the set of one or more input blocks receiving the specific input and providing an intermediate output, the intermediate block receiving the intermediate output as input and providing another intermediate output, and the set of one or more output blocks receiving the other intermediate output as input and providing the specific The method provides an output, and the method comprises an orchestration method, the orchestration method comprising: receiving a request to run a workload using the artificial intelligence model, the workload including receiving the specific input; determining the current resource utilization status in the distributed system; defining a deployment configuration of the artificial intelligence model using the current resource utilization status, wherein the deployment configuration is defined by: a number and structure of input blocks of the artificial intelligence model, a number and structure of output blocks, and the intermediate blocks, a second computer system running the intermediate blocks, and one or more first computer systems running the input and output blocks; and deploying the artificial intelligence model according to the defined deployment configuration and running the workload.

[0092] Item 2. The method according to Item 1, wherein, before executing the blocks, the method comprises a step of deploying the artificial intelligence model according to an initial deployment configuration, the step of defining the deployment configuration includes a step of updating the initial deployment configuration, the updating step includes at least one of the following: a step of subdividing the artificial intelligence model; a step of adding one or more first computer systems to execute the input and output blocks; a step of removing one or more of the initial first computer systems to execute the input and output blocks; or a step of selecting another second computer system to execute the intermediate blocks.

[0093] Item 3. The method according to either of the preceding items 1 or 2, wherein the step of defining the deployment configuration using the current resource utilization status includes: performing capacity profiling of the first and second computer systems to determine whether each of the first and second computer systems can execute one or more blocks of the artificial intelligence model; and defining the deployment configuration based on the capacity profiling.

[0094] Item 4. The current resource utilization status is defined by: the utilization level of the network resources of the distributed system; the resource utilization level of the first computer system; or the resource utilization level of the second computer system, as described in any one of the preceding items 1 to 3.

[0095] Item 5. The deployment configuration is further defined using rules, the rules of which require: a maximum number of first computer systems to be used to deploy the input and output blocks; a maximum number of blocks of the artificial intelligence model; satisfaction of specific quality of service (QoS) requirements; or at least one of the desired energy efficiency (EE), as described in any one of the preceding items 1 to 4.

[0096] Item 6. The method according to any one of the preceding items 1 to 5, wherein the distributed system is a wireless communication system, the first computer system is a multi-access edge computing (MEC) node, and the second computer system is a cloud system.

[0097] Item 7. The method according to any one of the preceding items 1 to 6, further comprising the step of having a broadcaster of the distributed system perform the deployment of the artificial intelligence model; and the step of having the broadcaster transmit information to an orchestrator of the distributed system, such that when the orchestrator receives the information, it triggers the execution of the workload.

[0098] Item 8. The artificial intelligence model is a foundation model, as described in any one of the preceding items 1 to 7.

[0099] Item 9. The method according to any one of the preceding items 1 to 8, wherein the artificial intelligence model is a deep neural network, the input block represents a first network layer, the intermediate block represents an intermediate network layer, and the output block represents a final network layer.

[0100] Item 10. The method according to any one of the preceding items 1 to 9, wherein the first computer system has a smaller amount of processing resources than the second computer system.

[0101] Item 11. The workload is: the method described in any one of the preceding items 1-10, involving at least one of the following: data analytics, sensor measurement fusion from different sources, or processing of data streams heading to the cloud.

[0102] Item 12. The method according to any one of the preceding items 1 to 11, wherein the step of executing the artificial intelligence model includes, for every two consecutive blocks of the artificial intelligence model deployed on different computer systems: the step of encoding the output of the first block of the two blocks using an encoding protocol; and the step of transmitting the encoded output to the first computer system or the second computer system for use as input for the second block of the two blocks.

[0103] Item 13. The method according to Item 12, wherein the encoding protocol includes at least one of compression or encryption, and the second encoding protocol includes at least one of compression or encryption.

[0104] The computing environment 800 includes an example of an environment for executing at least a portion of the computer code involved in performing the method of the present invention, such as code 900 for workload orchestration. In addition to block 900, the computing environment 800 includes, for example, a computer 801, a wide area network (WAN) 802, an end user device (EUD) 803, a remote server 804, a public cloud 805, and a private cloud 806. In this embodiment, computer 801 includes a processor set 810 (including processing circuits 820 and a cache 821), a communication fabric 811, volatile memory 812, persistent storage 813 (including an operating system 822 and block 900 as identified above), a peripheral device set 814 (including a user interface (UI) device set 823, storage 824, and an Internet of Things (IoT) sensor set 825), and a network module 815. The remote server 804 includes a remote database 830. Public Cloud 805 includes Gateway 840, Cloud Orchestration Module 841, Host Physical Machine Set 842, Virtual Machine Set 843, and Container Set 844.

[0105] Computer 801 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device currently known or to be developed in the future that is capable of executing programs, accessing networks, or querying databases such as remote database 830. As is well understood in the field of computer technology, and depending on the technology, the execution of the computer implementation method may be distributed among multiple computers and / or across multiple locations. On the other hand, in this presentation of the computing environment 800, in order to keep the presentation as concise as possible, the detailed discussion focuses on a single computer, specifically computer 801. Computer 801 may be located in the cloud, although it is not shown in the cloud in Figure 7. On the other hand, computer 801 is not required to be located in the cloud, except to any extent that can be definitively shown.

[0106] The processor set 810 includes one or more computer processors of any type currently known or to be developed in the future. The processing circuitry 820 may be distributed across multiple packages, e.g., multiple coordinated integrated circuit chips. The processing circuitry 820 may implement multiple processor threads and / or multiple processor cores. The cache 821 is memory located within the processor chip package and is typically used for data or code that should be available for rapid access by threads or cores running on the processor set 810. The cache memory is typically organized into multiple levels, depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 810 may operate using qubits and be designed to perform quantum computing.

[0107] Computer-readable program instructions typically cause the processor set 810 of computer 801 to execute a series of operational steps, thereby loading them onto computer 801 to implement a computer implementation method, and the instructions thus executed instantiate the methods specified in the flowcharts and / or descriptions of the computer implementation methods contained herein (collectively referred to as the "Methods of the Invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as the cache 821 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 810 to control and direct the execution of the Methods of the Invention. In the computing environment 800, at least some of the instructions for executing the Methods of the Invention may be stored in block 900 in persistent storage 813.

[0108] The communication fabric 811 is a signal conduction path that enables various components of the computer 801 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as buses, bridges, physical input / output ports, and similar switches and conductive paths. Other types of signal communication paths, such as fiber optic communication paths and / or wireless communication paths, may be used.

[0109] Volatile memory 812 is any type of volatile memory currently known or to be developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, volatile memory 812 is characterized by random access, but this is not required unless explicitly stated. In computer 801, volatile memory 812 is located in a single package and resides inside computer 801, but alternatively or in addition, volatile memory may be distributed across multiple packages and / or located externally to computer 801.

[0110] Persistent storage 813 is any form of non-volatile storage for a computer, currently known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained whether or not power is directly supplied to computer 801 and / or persistent storage 813. Persistent storage 813 may be read-only memory (ROM), but typically at least a portion of the persistent storage allows for writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 822 may take several forms, such as various known proprietary operating systems or open-source portable operating system interface (OPV) type operating systems employing a kernel. The code contained in block 900 typically includes at least some of the computer code involved in performing the method of the present invention.

[0111] The peripheral device set 814 includes a set of peripheral devices for the computer 801. Data communication connections between the computer 801's peripheral devices and other components may be implemented in various ways, such as Bluetooth connections, near-field communication (NFC) connections, cable connections (such as universal serial bus (USB) type cables), insert-type connections (e.g., secure digital (SD) cards), connections via local area communication networks, and even connections via wide area networks such as the internet. In various embodiments, the UI device set 823 may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smartwatches), keyboard, mouse, printer, touchpad, game controller, and haptic device. Storage 824 is external storage such as an external hard drive, or insertable storage such as an SD card. Storage 824 may be persistent and / or volatile. In some embodiments, storage 824 may take the form of a quantum computing memory device for storing data in the form of qubits. In embodiments where computer 801 is required to have a large amount of storage (for example, computer 801 locally stores and manages a large database), this storage may be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 825 consists of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another may be a motion detector.

[0112] The network module 815 is a collection of computer software, hardware, and firmware that enables computer 801 to communicate with other computers via the WAN 802. The network module 815 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for transmission over a communication network, and / or web browser software for communicating data over the internet. In some embodiments, the network control and network forwarding functions of the network module 815 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN)), the control and forwarding functions of the network module 815 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for performing the method of the present invention can typically be downloaded from an external computer or external storage device to computer 801 through a network adapter card or network interface included in the network module 815.

[0113] WAN802 is any wide area network (e.g., the Internet) capable of transmitting computer data over non-local distances by any currently known or future-developed technology for transmitting computer data. In some embodiments, WAN802 may be replaced and / or supplemented by a local area network (LAN), such as a Wi-Fi network, designed to transmit data between devices located in a local area. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0114] The end-user device (EUD) 803 is any computer system used and controlled by an end-user (e.g., a customer of the company operating computer 801) and can take any of the forms discussed above in relation to computer 801. Typically, EUD 803 receives useful and valuable data from the operation of computer 801. For example, in a hypothetical case where computer 801 is designed to provide recommendations to the end-user, these recommendations would typically be communicated from the network module 815 of computer 801 to EUD 803 via WAN 802. In this way, EUD 803 can display or otherwise present the recommendations to the end-user. In some embodiments, EUD 803 may be a client device such as a thin client, heavy client, mainframe computer, or desktop computer.

[0115] The remote server 804 is any computer system that provides at least some data and / or functionality to computer 801. The remote server 804 may be controlled and used by the same entity that operates computer 801. The remote server 804 represents a machine that collects and stores useful and valuable data for use by other computers, such as computer 801. For example, in the hypothetical case where computer 801 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 801 from the remote database 830 of the remote server 804.

[0116] Public Cloud 805 is any computer system available for use by multiple entities, providing on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct active management of the computing resources of Public Cloud 805 is performed by the computer hardware and / or software of the Cloud Orchestration Module 841. The computing resources provided by Public Cloud 805 are typically implemented by virtual computing environments running on various computers that make up the computers of the host physical machine set 842, which is the area of ​​physical computers available in and / or to Public Cloud 805. Virtual computing environments (VCEs) typically take the form of virtual machines from the virtual machine set 843 and / or containers from the container set 844. These VCEs may be stored as images and are understood to be transferable either as images or after instantiation of VCEs, among and between various physical machine hosts. The cloud orchestration module 841 manages image transfer and storage, deploys new VCE instances, and manages active instanceizations of VCE deployments. The gateway 840 is a collection of computer software, hardware, and firmware that enables the public cloud 805 to communicate over the WAN 802.

[0117] Here, some further explanation of virtualized computing environments (VCEs) is provided. A VCE can be stored as an "image." A new active instance of a VCE can be instantiated from an image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature in the kernel that allows for the existence of multiple isolated user-space instances called containers. These isolated user-space instances typically behave as actual computers from the perspective of the programs running in them. Computer programs running on a normal operating system can utilize all of that computer's resources, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running inside a container can only use the contents of the container and the devices allocated to the container; this feature is known as containerization.

[0118] Private Cloud 806 is similar to Public Cloud 805, except that its computing resources are available only for use by a single enterprise. While Private Cloud 806 is shown communicating with WAN 802, in other embodiments, the private cloud may be completely isolated from the internet and accessible only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a distinct and isolated entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple configured clouds. In this embodiment, both Public Cloud 805 and Private Cloud 806 are part of a larger hybrid cloud.

[0119] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment that is currently known or may be developed in the future.

[0120] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal administrative effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0121] The characteristics are as follows:

[0122] On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0123] Broad network access: This capability is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0124] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. While consumers generally have no control or knowledge of the exact location of the resources provided, there is location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0125] Rapid elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, to scale out rapidly and release quickly to scale in rapidly. To consumers, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0126] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metric capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, thereby providing transparency to both service providers and consumers.

[0127] The service model is as follows:

[0128] Software as a Service (SaaS): The ability provided to consumers is the use of a provider's applications running on cloud infrastructure. These applications are accessible from various client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited, user-specific application configuration settings.

[0129] Platform as a Service (PaaS): The ability provided to consumers is to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, on a cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they do control the deployed applications and, in some cases, the configuration of the application hosting environment.

[0130] Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources, where consumers can deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, and deployed applications, and in some cases have limited control over selected networking components (e.g., host firewalls).

[0131] The deployment model is as follows:

[0132] Private Cloud: This cloud infrastructure operates solely for a specific organization. It may be managed by that organization or a third party, and may reside on-premises or off-premises.

[0133] Community Cloud: This cloud infrastructure is shared by several organizations and supports a specific community that shares common interests (e.g., mission, security requirements, policies, and compliance considerations). It may be managed by those organizations or a third party and may reside on-premises or off-premises.

[0134] Public Cloud: This cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0135] Hybrid Cloud: This cloud infrastructure is a complex of two or more clouds (private, community, or public) that remain separate entities but are joined together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load distribution across clouds).

[0136] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0137] Referring here to Figure 8, an exemplary cloud computing environment 1050 is shown. As shown, the cloud computing environment 1050 includes one or more cloud computing nodes 1010 to which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1054A, a desktop computer 1054B, a laptop computer 1054C, and / or an automotive computer system 54N, can communicate. The nodes 1010 may communicate with each other. They may be physically or virtually grouped within one or more networks, such as private, community, public, or hybrid clouds, or a combination thereof, as described above in this specification (not shown). This enables the cloud computing environment 1050 to provide infrastructure, platform, and / or software as a service, which does not require cloud consumers to maintain resources on their local computing devices for that purpose. The types of computing devices 1054A-N shown in Figure 8 are intended for illustrative purposes only, and it should be understood that the computing node 1010 and the cloud computing environment 1050 can communicate with any type of computerized device via any type of network and / or network addressable connection (e.g., using a web browser).

[0138] Referring now to Figure 9, a set of functional abstraction layers provided by the cloud computing environment 1050 (Figure 8) is shown. It should be understood in advance that the components, layers, and functionalities shown in Figure 9 are intended for illustrative purposes only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functionalities are provided:

[0139] The hardware and software layer 1060 includes hardware and software components. Examples of hardware components include: a mainframe 1061; a RISC (Reduced Instruction Set Computer) architecture-based server 1062; a server 1063; a blade server 1064; a storage device 1065; and network and networking components 1066. In some embodiments, the software components include network application server software 1067 and database software 1068.

[0140] The virtualization layer 1070 provides an abstraction layer where examples of virtual entities may be provided, including: virtual servers 1071; virtual storage 1072; virtual networks 1073 including virtual private networks; virtual applications and operating systems 1074; and virtual clients 1075.

[0141] In one example, the management layer 1080 may provide the functions described below. Resource provisioning 1081 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 1082 provides cost tracking as resources are used within the cloud computing environment and billing or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks, as well as protection of data and other resources. User portal 1083 provides consumers and system administrators with access to the cloud computing environment. Service level management 1084 provides cloud computing resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) planning and execution 1085 provides pre-arrangements and procurement of cloud computing resources where future requirements are expected to conform to the SLA.

[0142] The workload layer 1090 provides examples of functionality that can be utilized in a cloud computing environment. Examples of workloads and functions that can be provided from this layer include: mapping and navigation 1091; software development and lifecycle management 1092; virtual classroom education delivery 1093; data analytics processing 1094; transaction processing 1095; and an AI model inference engine (AIIE) 1096 that executes the main blocks of an artificial intelligence model in accordance with this subject.

[0143] Various aspects of this disclosure are described by explanatory text, flowcharts, block diagrams of computer systems and / or block diagrams of machine logic included in computer program product (CPP) embodiments. With respect to any flowchart, operations may be performed in a different order than those shown in a given flowchart, depending on the technology involved. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated stage, simultaneously, or with at least partial time overlap.

[0144] Computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media ("mediums") that collectively comprise a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. Computer-readable storage media may be, but are not limited to, electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination of those described above. Some known types of storage devices, including these media, include: diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices (such as pits / lands formed on the main surface of a punch card or disk), or any suitable combination of the foregoing. When the term "computer-readable storage medium" is used in this disclosure, it shall not be construed as storage in the form of transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses passing through optical fiber cables, electrical signals communicated through wires, and / or other transmitting media.As those skilled in the art will understand, data moves at several intermittent points in the normal operation of a storage device, typically during access, defragmentation, or garbage collection, but this does not make the storage device transient, because data is not transient while it is stored.

Claims

1. A method for running a workload in a distributed system using an artificial intelligence model, The distributed system comprises a set of first computer systems configured to connect to at least one second computer system of the distributed system, The artificial intelligence model is configured to receive a specific input, process the specific input, and provide a specific output, and the artificial intelligence model is configured to be divided into one or more sets of input blocks, intermediate blocks, and one or more sets of output blocks, so that the one or more sets of input blocks receive the specific input and provide an intermediate output, the intermediate block receives the intermediate output as input and provides another intermediate output, and the one or more sets of output blocks receives the other intermediate output as input and provides the specific output; The method comprises an orchestration method, the orchestration method being: The step of receiving a request to run a workload using the artificial intelligence model, the workload includes receiving the specific input; A step of determining the current resource utilization status in the distributed system; Using the current resource utilization status, the next step is to define the deployment configuration of the artificial intelligence model, where the deployment configuration is: The number and structure of input blocks, the number and structure of output blocks, and the intermediate blocks of the artificial intelligence model. A second computer system that executes the aforementioned intermediate block, and One or more first computer systems that execute the aforementioned input and output blocks Defined by; The step of deploying the artificial intelligence model and executing the workload according to the defined deployment configuration. A method having

2. Before executing the orchestration method, the method includes a step of deploying the artificial intelligence model according to an initial deployment configuration, the step of defining the deployment configuration includes a step of updating the initial deployment configuration, the updating step is: The step of further subdividing the aforementioned artificial intelligence model; A step of adding one or more first computer systems that execute the aforementioned input and output blocks; A step of removing one or more of the first computer systems of the initial deployment configuration that execute the input and output blocks; or The step of selecting another second computer system to execute the aforementioned intermediate block. The method of the preceding claim, comprising at least one of the following:

3. Using the current resource utilization status, the step of defining the deployment configuration is: A step of performing capacity profiling of the first and second computer systems to determine whether each of the first and second computer systems can execute one or more blocks of the artificial intelligence model; and The step of defining the deployment configuration based on the capacity profiling. The method according to any one of the prior claims, including the method described in any one of the prior claims.

4. The current resource utilization status is: The utilization level of the network resources of the aforementioned distributed system; The resource utilization level of the first computer system; or Resource utilization level of the second computer system A method according to any one of the prior claims, as defined by at least one of the following.

5. The aforementioned deployment configuration is further defined using rules, the rules being: The maximum number of first computer systems that will be used to unpack the aforementioned input and output blocks; Q: Maximum number of blocks in an artificial intelligence model; Fulfillment of specific quality of service (QoS) requirements; or Desirable energy efficiency (EE) A method according to any one of the prior claims, which requires at least one of the following.

6. The method according to any one of the preceding claims, wherein the distributed system is a wireless communication system, the first computer system is a multi-access edge computing (MEC) node, and the second computer system is a cloud system.

7. The step of performing the deployment of the artificial intelligence model by a broadcaster of the distributed system; The broadcaster transmits information to the orchestrator of the distributed system, so that when the orchestrator receives the information, it triggers the execution of the workload. The method according to any one of the prior claims, further comprising:

8. The method according to any one of the preceding claims, wherein the artificial intelligence model is a foundation model.

9. The method according to any one of the preceding claims, wherein the artificial intelligence model is a deep neural network or a transformer, the input block represents a first network layer, the intermediate block represents an intermediate network layer, and the output block represents a final network layer.

10. The method according to any one of the preceding claims, wherein the first computer system has a smaller amount of processing resources than the processing resources of the second computer system.

11. The aforementioned workload is: Data analytics, Sensor measurement fusion from different sources, image analysis, or processing of data streams heading to the cloud. The method according to any one of the prior claims, comprising at least one of the above.

12. The method according to any one of the preceding claims, wherein the step of executing the artificial intelligence model includes, for each of two consecutive blocks of the artificial intelligence model deployed on different computer systems: the step of encoding the output of a first block of the two blocks using an encoding protocol; and the step of transmitting the encoded output to the first computer system or the second computer system for use as input for a second block of the two blocks.

13. The method according to the preceding claim, wherein the encoding protocol includes at least one of compression or encryption.

14. A computer program product comprising a computer-readable storage medium on which computer-readable program code is embodied, wherein the computer-readable program code is configured to implement the method described in claim 1.

15. A computer system for running workloads in a distributed system using an artificial intelligence model, The distributed system comprises a set of first computer systems configured to connect to at least one second computer system of the distributed system, The artificial intelligence model is configured to receive a specific input, process the specific input, and provide a specific output, and the artificial intelligence model is configured to be divided into one or more sets of input blocks, intermediate blocks, and one or more sets of output blocks, so that the one or more sets of input blocks receive the specific input and provide an intermediate output, the intermediate block receives the intermediate output as input and provides another intermediate output, and the one or more sets of output blocks receives the other intermediate output as input and provides the specific output; The aforementioned computer system: Receiving a request to run a workload using the artificial intelligence model, the workload including receiving the specific input; To determine the current resource utilization status in the aforementioned distributed system; Using the current resource utilization status, define the deployment configuration of the artificial intelligence model, where the deployment configuration is: The number and structure of input blocks, the number and structure of output blocks, and the intermediate blocks of the artificial intelligence model. A second computer system that executes the aforementioned intermediate block, and One or more first computer systems that execute the aforementioned input and output blocks Defined by; Deploy the artificial intelligence model and execute the workload according to the defined deployment configuration. A computer system configured to perform the following actions.

Images