Dynamic artificial intelligence (AI) workflow execution

Abstract

This disclosure describes a dynamic workflow system that provides a framework for operating dynamic AI workflow tables on client devices. For example, the dynamic workflow system facilitates the addition, removal, or modification of AI workflows from a dynamic AI workflow table without requiring an operating system (OS) update. Providing dynamic AI workflows also enables the dynamic workflow system to provide additional benefits, such as instant AI workflow updates, the use of multiple AI table versions, AI workflow testing and evaluation, and the removal of defective or inefficient AI workflows without affecting other workflows. These benefits were not feasible with existing systems. The dynamic workflow system also provides a separate API that allows for customized access to AI workflows in AI workflow tables.

Description

DYNAMIC ARTIFICIAL INTELLIGENCE (Al) WORKFLOW EXECUTIONBACKGROUND

[0001] The landscape of computational devices has witnessed significant advancements in both hardware and software domains, particularly in the area of artificial intelligence (Al) models and workflows on client devices. For instance, Al workflows, which are essentially a series of operations that commonly use Al models to execute tasks such as image-to-text conversion, have become a vital component in computational processing. When an application needs to carry out a variety of tasks, it obtains support from the operating system (OS) via an operation call. In particular, the OS uses an application programming interface (API) to access an Al workflow table, selects an appropriate Al workflow, executes the Al workflow using one or more corresponding Al models to complete the task, and returns the result to the application. However, current systems and methodologies are limited to static workflows and workflow tables, which only change via OS system updates. As a result, the static nature of Al workflows leads to several challenges in managing and using Al workflows, which are described further below.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The following detailed description provides specific and detailed implementations accompanied by drawings. Additionally, each of the figures listed below corresponds to one or more implementations discussed in this disclosure.

[0003] FIGS. 1 A-1B illustrate overview examples of the dynamic workflow system providing various functions using Al workflows, including dynamically updating an Al workflow table and evaluating Al workflows.

[0004] FIG. 2 illustrates an example computing environment where the dynamic workflow system is implemented.

[0005] FIG. 3 illustrates an example sequence flow diagram of using a dynamic Al workflow table that includes new Al workflows to execute tasks.

[0006] FIGS. 4A-4B illustrate example sequence flow diagrams of testing new Al workflows to add to a dynamic Al workflow table.

[0007] FIG. 5 illustrates an example sequence flow diagram of evaluating and updating Al workflows in the dynamic Al workflow table.

[0008] FIG. 6 illustrates an example sequence flow diagram of the dynamic workflow system providing a separate API to access Al workflows.

[0009] FIGS. 7-8 each illustrate an example series of acts in a computer-implemented method for utilizing one or more Al workflows on a client device.

[0010] FIG. 9 illustrates example components included within a computer system used toimplement the dynamic workflow system.DETAILED DESCRIPTION

[0011] This disclosure describes a dynamic workflow system that provides a framework for operating dynamic Al workflow tables on client devices. For example, the dynamic workflow system facilitates the addition, removal, or modification of Al workflows from a dynamic Al workflow table without requiring an operating system (OS) update. Providing dynamic Al workflows also enables the dynamic workflow system to provide additional benefits, such as instant Al workflow updates, the use of multiple Al table versions, Al workflow testing and evaluation, and the removal of defective or inefficient Al workflows without affecting other workflows. These benefits were not feasible with existing systems. The dynamic workflow system also provides a separate API that allows for customized access to Al workflows in Al workflow tables.

[0012] Implementations of the present disclosure provide benefits and solve problems in the art with systems, computer-readable media, and computer-implemented methods by using a dynamic workflow system to allow Al workflows to operate dynamically within the operating system of a client device. As described below, the dynamic workflow system modifies the current configuration and management of Al workflows to enable the addition, removal, updating, and modification of Al workflows within an Al workflow table without replacing the entire table through a software update. Additionally, the dynamic workflow system provides additional flexible access to Al workflows that were previously not enabled or available under existing systems

[0013] To elaborate, in various implementations, the dynamic workflow system facilitates a dynamic Al workflow table for runtime operations. For example, the OS receives an operation call from a caller to perform a first task and, in response, obtains a dynamic Al workflow table with Al workflows. From the Al workflow table, a new Al workflow is selected where the new Al workflow was not included in the Al workflow table at the time of a previous operation call from the caller to perform the first task. Based on selecting the new Al workflow, it is executed on the client device to generate a first output that is provided to the caller in response to the operation call.

[0014] As another example, in various implementations, the dynamic workflow system enables testing and evaluations of Al workflows without affecting runtime operations. For example, the OS again receives an operation call from a caller to perform a first task and, in response, obtains a dynamic Al workflow table with Al workflows. In some instances, the dynamic workflow system obtains a testing Al workflow table. The first task is performed by executing a first Al workflow from the dynamic Al workflow table to generate a first output. Inaddition, the dynamic workflow system executes a second Al workflow from the testing Al workflow table to generate a second output. In these implementations, the first output is provided to the caller in response to the operation call while the second output of executing the first task is stored and not provided to the caller. Indeed, the caller is unaware that the dynamic workflow system executed the second Al workflow or generated the second output.

[0015] Further, in one or more implementations, the dynamic workflow system provides a separate API (e.g., separate from the Al workflow API implemented by the OS). The separate API allows scripts, applications, and users to access and utilize Al workflows in several ways, including some not enabled or made available by the Al workflow API. This way, the dynamic workflow system enables scripts to provide customized operation calls, which allows for direct interfacing with Al workflows, specific and complex tasks to be accomplished, and Al workflows to be tested and evaluated in new ways.

[0016] As described in this disclosure, the dynamic workflow system delivers several significant technical benefits in terms of improved flexibility, efficiency, and accuracy compared to existing systems. Moreover, the dynamic workflow system provides several practical applications that address problems related to providing improved outcomes for executing Al workflows on client devices.

[0017] To illustrate, the dynamic workflow system improves flexibility by implementing dynamic Al workflow tables on a client device. Existing systems provide static workflows that cannot be changed without significant hassle. In some instances, a workflow includes a chain of operations that uses various Al models, and each segment and call in the chain is fixed. For example, in existing systems, workflows are incorporated into operating systems and are only updateable through an operating system patch or update. By enabling dynamic Al workflow tables that can update Al workflows on the fly, the dynamic workflow system provides several benefits over existing systems.

[0018] For example, dynamic Al workflow tables enable Al workflows to be added, removed, or updated at any time without waiting for a system patch or update. This provides the technical benefit of new and / or updated Al workflows being quickly integrated into the OS of a client device. This also provides the technical benefit of allowing problematic Al workflows to be detected, disabled, and / or removed without affecting other Al workflows in the table.

[0019] Implementing dynamic Al workflow tables fiirther provides the technical benefit of performing testing and evaluations on Al workflows, which improves the accuracy and efficiency of a client device. For example, Al models are difficult to measure for accuracy at the time of training. Through testing and implementation, errors are found, and new models are generated, which can be included in new or updated Al workflows. Further, testing can determine that oneAl workflow is more favorable than another Al workflow (e.g., A / B testing), and the more favorable Al workflow can be added to the dynamic Al workflow table and / or the less favorable Al workflow removed. Indeed, because Al workflows are a component of a client device’s OS, determining the most efficient and accurate Al workflows used by the OS, and being able to add them to a dynamic Al workflow table used by the OS while also removing inferior workflows, improves client device performance.

[0020] As illustrated in the foregoing discussion, this disclosure utilizes a variety of terms to describe the features and advantages of one or more implementations described. To illustrate, this disclosure describes the dynamic workflow system in the context of a cloud computing system.

[0021] As an example, the term “Al workflow” refers to a structured piece of data that provides a systematic process of using automation and advanced algorithms to efficiently execute complex tasks. Many Al workflows include an Al model on the client device that performs a specific fimction and input parameters needed by the Al model. In some instances, one or more of the Al models are located remotely from the client device (e.g., a hybrid case). An Al workflow commonly involves using Al technologies such as machine learning, natural language processing, and robotic process automation to analyze, interpret, and execute tasks. Examples of Al workflows include document processing, image recognition, image enhancement, optical character recognition, chatbots, data intake, diagnostics, voice assistants, and anomaly detection. A set of Al workflows is stored in an Al workflow table, often stored in a data store (e.g., database).

[0022] As an example, the term “Al workflow tables” refers to a data structure that includes a set, collection, or group of Al workflows. In some instances, an Al workflow table refers to a database table of Al workflows. Al workflow tables are stored on client devices and queried by an OS (e.g., an Al workflow API of the OS) for access. Often, an Al workflow table includes multiple Al workflows. In a few implementations, an Al workflow table includes only one Al workflow.

[0023] Furthermore, the term “dynamic Al workflow table” refers to an Al workflow table that can be changed or modified at any time by adding, removing, updating, changing, and / or modifying Al workflows within the table. In some implementations, a dynamic Al workflow table is modifiable at times other than an OS patch or update. In most implementations, a dynamic Al workflow table refers to the Al workflow table used at runtime by an OS, while testing or experimental Al workflow table refers to other dynamic versions of Al workflow tables that are used by the dynamic workflow system for testing and experimentation.

[0024] As an example, the term “caller” refers to a computer application, script, user, program, or system that sends an operation call to the operating system to perform a task indicated in the operation call. An “operation call” (or “call” for short) refers to a request to perform a specifiedtask made by a caller. In some instances, an operation call is called a system call when the call is made to an operating system. A call often includes input arguments needed to perform the requested task.

[0025] As an example, the term “Al model” refers to a computational system that learns from data and makes predictions or decisions. Al models are commonly trained using algorithms to autonomously process input data to achieve specific tasks, such as image recognition or natural language understanding. Al models serve as the core component within Al workflows, enabling efficient and intelligent processing of tasks on client devices (or cloud platforms in some instances). Al models can include heuristic models, machine-learning models, and / or neural networks. In some implementations, an Al workflow includes a small generative model (SGM). In some instances, an Al model is implemented by the operating system. In some instances, an Al model is implemented by a third-party application or service included on a client device.

[0026] Also, the term “machine-learning model” refers to a computer model or computer representation that can be trained (e.g., optimized) based on inputs to approximate unknown functions. For instance, a machine-learning model can include (but is not limited to) an autoencoder model, a distortion classification model, a neural network (e.g., a convolutional neural network or deep learning model), a decision tree (e.g., a gradient-boosted decision tree), a linear regression model, a logistic regression model, or a combination of these models.

[0027] As another example, the term “neural network” refers to a machine learning model comprising interconnected artificial neurons that communicate and learn to approximate complex functions, generating outputs based on multiple inputs provided to the model. For instance, a neural network includes an algorithm (or set of algorithms) that employs deep learning techniques and utilizes training data to adjust the parameters of the network and model high-level abstractions in data. Various types of neural networks exist, such as convolutional neural networks (CNNs), residual learning neural networks, recurrent neural networks (RNNs), generative neural networks, generative adversarial neural networks (GANs), and single-shot detection (SSD) networks.

[0028] As an example, a “large generative model” (LGM) is a large artificial intelligence system that uses deep learning and a large number of parameters (e.g., in the billions or trillions), trained on one or more vast datasets to produce fluent, coherent, and topic-specific outputs (e.g., text and / or images). In many instances, a generative model refers to an advanced computational system that uses natural language processing, machine learning, and / or image processing to generate coherent and contextually relevant human-like responses.

[0029] Similarly, a “small generative model” (SGM) is a lightweight, smaller generative model with fewer parameters. Unlike their larger counterparts, SGMs operate efficiently within resource constraints and are designed for scenarios where computational resources, memory, ormodel size are limited. Despite their reduced complexity, SGMs still exhibit the ability to generate coherent and contextually relevant outputs, albeit on a smaller scale. Examples of SGMs include text summarization models, image captioning models, creative writing assistance models, chatbots, code generation models, and personalized recommendation systems.

[0030] Implementation examples and details of the dynamic workflow system are discussed in connection with the accompanying figures, which are described next. For example, FIGS. 1 A- 1B illustrate overview examples of the dynamic workflow system providing various functions using Al workflows, including dynamic updating of an Al workflow table and evaluating Al workflows according to some implementations. In particular, FIG. 1 A shows an overview of the features provided by the dynamic workflow system, while FIG. IB shows an overview of the dynamic workflow system implementing dynamic Al workflow tables and Al workflow testing according to some implementations.

[0031] As shown, FIG. 1A includes a client device 102 with the dynamic workflow system 104. For example, the dynamic workflow system is part of an OS on the client device or works in connection with the OS. The dynamic workflow system 104 includes a dynamic Al workflow table 106 with Al workflows 108. In some implementations, the dynamic Al workflow table 106 is stored elsewhere in the client device 102 but is shown to symbolically belong to the dynamic workflow system 104, which manages them.

[0032] As shown in FIG. 1 A, the dynamic workflow system 104 implements various technical improvements and features on the client device 102. For example, the dynamic workflow system 104 implements (box 110) performing runtime Al workflow executions using the dynamic Al workflow table 106. This concept is farther described in connection with FIG. 3 below. The dynamic workflow system 104 also implements (box 112) testing and evaluating the Al workflows 108, which is farther described in connection with FIGS. 4A-4B and FIG. 5 below. Furthermore, the dynamic workflow system 104 implements (box 114) providing a separate API to perform customized scripts with the Al workflows 108, which is further described in connection with FIG. 6 below.

[0033] FIG. IB also provides additional details regarding the dynamic workflow system 104 performing runtime Al workflow executions using the dynamic Al workflow table as well as evaluating the Al workflows. As shown, FIG. IB includes a series of acts 100 performed by or with the dynamic workflow system.

[0034] The series of acts 100 includes act 121 of adding a newly received Al workflow to a dynamic Al workflow table with a set of Al workflows. For example, the dynamic workflow system 104 manages Al workflows 138 that belong to a dynamic Al workflow table 136. This includes adding, removing, and modifying the Al workflows 138. For instance, the dynamicworkflow system 104 receives and adds a new Al workflow 140 to the dynamic Al workflow table 136 for the OS to use to execute future operation calls.

[0035] Act 122 includes obtaining the dynamic Al workflow table upon receiving an operation call from a caller to perform a first task. For instance, based on generating and / or maintaining the dynamic Al workflow table 136, the dynamic workflow system 104 implements the dynamic Al workflow table 136 for the OS of a client device to use. Accordingly, when a caller 130 (e.g., an application) needs assistance from the OS to perform a first task 134, it provides an operation call 132 to the OS, which obtains the dynamic Al workflow table 136 from a database or data store. In some implementations, the dynamic workflow system 104 provides a separate API that provides access to the dynamic Al workflow table 136.

[0036] Act 123 includes executing the first Al workflow on the client device to generate a first output. For instance, obtaining the dynamic Al workflow table 136 includes accessing a set of the Al workflows 138. In some instances, the Al workflows 138 include a first Al workflow 142. If appropriate to accomplish the first task 134, the first Al workflow 142 is selected, executed, and uses one or more Al models to generate the first output 144. For example, if the first task 134 is to recognize text in an image and the first Al workflow 142 uses a first image-text recognition machine-learning model to recognize text, executing the first Al workflow 142 includes sending the image to the image-text recognition machine-learning model and receiving recognized text back as the first output 144.

[0037] Act 124 includes performing testing by executing a second Al workflow on the client device to generate a second output. In some instances, a second Al workflow 146 is included in the dynamic Al workflow table 136 but in other instances, the second Al workflow 146 is obtained from another Al workflow table, such as a testing Al workflow table not accessible by the OS. In act 124, the second Al workflow 146 is executed to generate the second output 148. Continuing the above example, executing the second Al workflow 146 includes sending the image to an image-text recognition neural network and receiving recognized text back as the second output 148.

[0038] Act 125 includes providing the first output to the caller in response to the operational call. For example, the OS returns the first output 144 to the caller 130 to satisfy the operation call 132. Act 126 includes storing the second output for testing against the first output without providing the second output to the caller. For example, because the second Al workflow 146 was executed for testing purposes, the second output 148 is stored rather than provided to the caller 130. The caller 130 does not detect that additional Al workflows were executed based on the operation call 132.

[0039] Additionally, the dynamic workflow system 104 can use the outputs to improve the Alworkflows 138 in the dynamic Al workflow table 136. As described further below, in some instances, the dynamic workflow system 104 determines that the second output 148 is more favorable than the first output 144 and based in part on this determination, replaces the first Al workflow 142 in the dynamic Al workflow table 136 with the second Al workflow 146 so that when the OS needs to accomplish the first task, it executes the second Al workflow 146, which yields a more favorable (e.g., more accurate) output.

[0040] With a general overview in place, additional details are provided regarding the components, features, and elements of the dynamic workflow system. To illustrate, FIG. 2 shows an example of a client device where the dynamic workflow system is implemented according to some implementations. In particular, FIG. 2 illustrates a client device 200 with an operating system 202, a data store 210, and an application 206. While a client device 200 is shown, other computing devices may be used, such as a server device or a cloud computing device.

[0041] In various implementations, the client device 200 is associated with a user such as an administrator who interacts with the application 206, the dynamic workflow system 104, and / or the operating system 202. For example, the user requests the application 206 to perform a task, which is passed on to the operating system 202 and accomplished with an Al workflow managed by the dynamic workflow system 104.

[0042] In one or more implementations, the application 206 represents a software application located on the client device 200 that works with the operating system 202 to provide a service and / or a set of features to a user. The application 206 may be an executable program, a temporary service, a client-side web service, or another agent that runs on the client device 200 and makes operation calls to the operating system 202 for Al workflow tasks.

[0043] As shown, the data store 210 includes dynamic Al workflow tables 212 and Al models 214. As described above, the Al models 214 serve as one of the core components within Al workflows and enable the efficient and intelligent processing of tasks on client devices. In some instances, an Al model is provided by a third-party application or service included on the client device 200. In some cases, some or all of the data store 210 is on the operating system 202.

[0044] In many implementations, the dynamic Al workflow tables 212 include Al workflows managed by the dynamic workflow system 104. In various implementations, the dynamic Al workflow tables 212 include tables accessible by the operating system 202. In some instances, the dynamic Al workflow tables 212 also include tables not accessible by the operating system 202 (e.g., testing or experimental Al workflow tables) or tables accessible by the operating system 202 when an application (e.g., a caller) has sufficient permissions (e.g., has a license or subscription).

[0045] As shown, the operating system 202 includes an Al workflow API 208 and the dynamic workflow system 104. In various instances, the Al workflow API 208 implements Al workflowexecutions in response to operation calls to perform tasks. In some implementations, the dynamic workflow system is located outside of the operating system 202.

[0046] As mentioned earlier, the dynamic workflow system 104 implements the dynamic Al workflow tables 212 on the client device 200. As shown, the dynamic workflow system 104 includes various components and elements that are implemented in hardware and / or software. For example, the dynamic workflow system 104 includes an Al workflow manager 220, a testing and evaluation manager 222, a separate API manager 224, and a storage manager 226. The storage manager 228 includes stored outputs and a separate API 230. In some instances, the storage manager 226 includes Al workflows and / or dynamic Al workflow tables that are not accessible to the operating system 202.

[0047] As mentioned above, the dynamic workflow system 104 includes the Al workflow manager 220, which manages Al workflows. For example, the Al workflow manager 220 adds, removes, updates, and / or modifies Al workflows from the dynamic Al workflow tables 212. In some instances, the Al workflow manager 220 manages which Al workflows are included in which of the dynamic Al workflow tables 212. Additional details regarding managing Al workflows are provided below.

[0048] The dynamic workflow system 104 includes the testing and evaluation manager 222, which implements experiments, tests, and evaluations with Al workflows. For example, the testing and evaluation manager 222 monitors Al workflow executions and determines positive and negative patterns for one or more workflows. Based on the patterns observed by the storage manager 226, the testing and evaluation manager 222 determines whether to disable, remove, update, or flag a given Al workflow. The testing and evaluation manager 222 performs A / B testing and other experiments using, in part, the storage manager 226 of executed Al workflows, as further described below in the context of the dynamic workflow system 104.

[0049] The dynamic workflow system 104 also includes the separate API manager 224, which implements an API other than the Al workflow API 208. The separate API 230 allows scripts, users, programs, and other callers to directly access the dynamic Al workflow tables 212 without utilizing the operating system 202. In some implementations, the separate API manager 224 permits the testing and evaluation manager 222 to conduct experiments, tests, and evaluations without impacting the runtime execution of the operating system 202 for operation calls. Further details regarding the separate API are provided below.

[0050] Turning to the next figures, FIG. 3 illustrates an example sequence flow diagram of using a dynamic Al workflow table that includes new Al workflows to execute tasks according to some implementations. As shown, FIG. 3 includes various components in communication with each other, including the application 206, the operating system 202 (with the dynamic workflowsystem 104 and the Al workflow API 208), the data store 210, and the Al models 214. Each of these components was introduced above. FIG. 3 also includes a series of acts 300 performed by the operating system 202 and / or dynamic workflow system 104 to accomplish tasks using a dynamic Al workflow table.

[0051] The series of acts 300 includes act 302 of receiving a new Al workflow that performs a task. In particular, act 302 includes the dynamic workflow system 104 receiving a new Al workflow. For example, as new Al workflows are created by a program, user (e.g., administrator or programmer), or system, they are provided to the dynamic workflow system 104. For instance, the application 206 or another application provides a new Al workflow to the dynamic workflow system 104 to be added to the client device 202. In some instances, the dynamic workflow system 104 adds the new Al workflow to the dynamic Al workflow table, as shown in act 304.

[0052] In some instances, the dynamic workflow system 104 adds the new Al workflow to a secondary table that is not used for runtime operations. For example, the new Al workflow is added to a testing or experimental Al workflow table. In some implementations, the Al workflow is added to a premium version of a dynamic Al workflow table that includes Al workflows only available to authorized users.

[0053] Act 306 includes the application 206 receiving a request to perform a task. In various implementations, an application provides various features and functions to users, systems, and other applications. Accordingly, the application 206 may receive a request to perform the task.

[0054] In some instances, the application 206 relies on the services of the operating system 202 to perform some or all of the tasks. For example, for tasks such as image recognition or natural language understanding, the application 206 uses system processes to perform these tasks, often using specialty hardware on the client device geared towards using Al models. To illustrate, act 304 includes the operating system 202 receiving an operation call to perform the task from the application 206.

[0055] In particular, act 308 includes the Al workflow API 208 of the operating system 202 receiving the operational call. In various implementations, the operational call includes a request to perform the task (e.g., a requested output result) and one or more inputs (e.g., input parameters) needed to perform the task. Often, the input is a type of media to be processed to determine a result, such as an image, document, audio, or video file.

[0056] Act 310 includes querying and receiving a dynamic Al workflow table. In particular, the Al workflow API 208 sends a query to the data store 210 from the dynamic Al workflow table. The data store 210 processes the query, identifies the dynamic Al workflow table, and returns it to the Al workflow API 208. As mentioned, the dynamic Al workflow table includes a set of Al workflows that use Al models and / or other tools to accomplish various tasks.

[0057] Act 312 includes the Al workflow API 208 selecting the new Al workflow from the dynamic Al workflow table. For example, the Al workflow API 208 compares the parameters of the operation call (e.g., the requested task, the requested output result, and / or the inputs) to the Al workflows to determine an Al workflow to use. Often, the Al workflow API 208 will determine one Al workflow that matches the parameters of the operation call. If multiple Al workflows match the parameters, the Al workflow API 208 may select one or each of the Al workflows (and return outcomes for each selected workflow).

[0058] In this case, the Al workflow API 208 selects the new Al workflow. For example, the Al workflow API 208 selects an Al workflow that was recently added by the dynamic workflow system 104 to the dynamic Al workflow table. In various instances, the new Al workflow was not included in the dynamic Al workflow table at the time of the last or previous operation call to perform the task. In some instances, the new Al workflow is an updated or modified version of a workflow that was previously in the Al workflow table. Indeed, the dynamic workflow system 104 may add, remove, replace, update, and / or modify Al workflows within the dynamic Al workflow table at any time.

[0059] Act 314 includes the Al workflow API 208 providing the input including the operation call to an Al model included in the selected Al workflow. For example, the new Al workflow indicates one or more of the Al models 214 on the client device that is used to accomplish the requested task. In some instances, an Al workflow includes a chain of operations and / or Al models used to perform a task. Accordingly, the Al workflow API 208 executes the selected Al workflow and follows its instructions to provide the appropriate input or inputs to the designated Al model or models.

[0060] Act 316 includes one or more of the Al models 214 generating and returning an output for the task to the Al workflow API 208. For example, an Al model processes the input to generate the requested output. In some instances, multiple Al models process a chain of inputs and outputs to generate the requested output. The output is then returned to the Al workflow API 208.

[0061] Act 318 includes the Al workflow API 208 returning the output to the Al workflow API 208 in response to the operation call. For example, upon receiving the output generated by the Al models 214, the Al workflow API 208 provides it to the application 206, which requested it in the operation call. In some instances, the Al workflow API 208 performs some processing (e.g., filtering, refining, format conversion) on the output before providing it back to the application 206.

[0062] As mentioned above, FIGS. 4A-4B and FIG. 5 provide additional details regarding the dynamic workflow system 104 that implements testing and evaluating Al workflows. For instance, FIGS. 4A-4B illustrate example sequence flow diagrams of testing new Al workflows to add to adynamic Al workflow table according to some implementations. FIGS. 4A-4B include the same components shown in FIG. 3. FIGS. 4A-4B also include a series of acts 400 performed by the operating system 202 and / or dynamic workflow system 104 to perform Al workflow evaluations.

[0063] In FIG. 4A, the series of acts 400 includes act 402 of receiving an operation call to perform a task. In particular, the Al workflow API 208 receives an operation call from the Al workflow API 208 to have the operating system 202 perform a task using Al models and / or Al workflows as a runtime operation.

[0064] Act 404 includes the dynamic workflow system 104 detecting the operation call for the task. In these implementations, the dynamic workflow system 104, as an outside observer, detects when operation calls arrive at the operating system 202. This way, the dynamic workflow system 104 can run experiments and tests based on real-world operation calls. As shown, the dynamic workflow system 104 can run tests and experiments alongside runtime operations, without interfering with the runtime operations, to gain valuable data regarding Al workflows. Having a dynamic Al workflow table makes this type of testing and experimentation possible.

[0065] Act 406 includes the dynamic workflow system 104 determining to perform testing for the task by running a second implementation instance. As shown, based on detecting the operation call for the task, the dynamic workflow system 104 determines to run a test scenario to evaluate the accuracy and efficiency of a second Al workflow performing the task alongside a first Al workflow. In one or more implementations, the dynamic workflow system 104 provides instructions for obtaining and executing a second Al workflow. In some instances, the dynamic workflow system 104 signals to the Al workflow API 208 that it will perform an additional execution with a second Al workflow, as described below.

[0066] Act 408 includes the Al workflow API 208 querying for and receiving a dynamic Al workflow table from the data store 210. As described previously, the dynamic Al workflow table includes a set of Al workflow that can be executed to accomplish various tasks using Al models and / or other tools. In many cases, the dynamic Al workflow table represents runtime Al workflows accessible by the operating system 202.

[0067] Act 410 includes the dynamic workflow system 104 querying for and receiving a testing Al workflow table from the data store 210. For example, the dynamic workflow system 104 obtains a testing Al workflow that includes Al workflows to be tested and / or evaluated. In various implementations, the testing Al workflow table is not accessible by the operating system 202. In various implementations, act 408 and 410 occur at or near the same time.

[0068] In various implementations, the dynamic workflow system 104 queries and receives the dynamic Al workflow table (or causes the Al workflow API 208 to obtain a second instance of the dynamic Al workflow table). For example, the dynamic workflow system 104 tests an Alworkflow included in the dynamic Al workflow table. In some instances, the dynamic Al workflow table includes experimental Al workflows. In these instances, an experimental Al workflow has an indicator that prevents it from being used for normal execution.

[0069] Act 412 includes the Al workflow API 208 selecting a first Al workflow from the dynamic Al workflow table, which may occur as described above. Act 414 includes the dynamic workflow system 104 selecting a second Al workflow from the testing Al workflow table. In various implementations, the testing Al workflow table includes only the second Al workflow. In some implementations, the testing Al workflow table includes multiple Al workflows and the dynamic workflow system 104 selects one or more Al workflows to be tested. For example, while only one Al workflow is shown as being selected, the dynamic workflow system 104 may select multiple Al workflows to be tested. In some implementations, the dynamic workflow system 104 instructs the Al workflow API 208 to select and execute the second Al workflow from the testing Al workflow table.

[0070] Act 416 includes the Al workflow API 208 providing an input to a first Al model included in the first Al workflow. For example, the Al workflow API 208 executes the first Al workflow by providing the required input to the indicated Al model, as described above. Act 418 includes the first Al model of the Al models 214 generating and returning a first output for the task associated with the first Al workflow to the Al workflow API 208.

[0071] Act 420 includes the dynamic workflow system 104 providing an input to a second Al model included in the second Al workflow. For example, the dynamic workflow system 104 executes the second Al workflow and provides the required input to the second Al model of the Al models 214. In various implementations, the first Al model associated with the first Al workflow is different from the second Al model associated with the second Al workflow. In some instances, the first Al model and the second Al model are the same model or different versions of the same model. Similarly, the inputs to each Al model may be the same or different, depending on the model requirements.

[0072] In FIG, 4 A, act 422 includes the second Al model of the Al models 214 generating and returning a second output for the task associated with the second Al workflow to the dynamic workflow system 104. For example, the second Al model processes the task and generates a second output, which it returns to the dynamic workflow system 104. In some instances, the second Al model returns the second output to the Al workflow API 208, if instructed.

[0073] In FIG 4B, the series of acts 400 includes act 424 of the Al workflow API 208 returning the first output to the application 206 in response to the operation call. As described above, the Al workflow API 208 completes the operation call by returning the requested output (e.g., the first output) to the application 206 (e.g., the caller), which made the operation call. In variousimplementations, the Al workflow API 208 also provides the first output for the dynamic workflow system 104.

[0074] Act 426 includes the dynamic workflow system 104 storing the second output. Additionally, the dynamic workflow system 104 does not provide the second output to the application 206. In particular, the dynamic workflow system 104 and / or the Al workflow API 208 executes the second Al workflow and other test or experimental Al workflows transparently such that the Al workflow API 208 is unaware of these additional operations. Indeed, the dynamic workflow system 104 performs various testing and experimental steps, and swaps Al workflows in some instances, without affecting or disturbing the runtime operations of the operating system 202 completing the task for the application 206.

[0075] Act 428 includes the dynamic workflow system 104 determining telemetry metrics based on the first output and the second output. For instance, the dynamic workflow system 104 analyzes the outputs to determine one or more values, scores, or other metrics. For example, the dynamic workflow system 104 determines the accuracy and efficiency metrics of executing the task for the first output and the second output.

[0076] In various implementations, the dynamic workflow system 104 collects multiple outputs from executing the first Al workflow and the second Al workflow. The dynamic workflow system 104 may group these respective outputs when generating the telemetry metric. This way, the dynamic workflow system 104 determines telemetry metrics of the first Al workflow and / or the second Al workflow based on several samples. In some implementations, the outputs are used to generate better Al workflows and / or Al models.

[0077] Act 430 includes the dynamic workflow system 104 determining that the second output is more favorable than the first output. For instance, the dynamic workflow system 104 compares telemetry metrics corresponding to the first output, first Al workflow, and / or first Al model to telemetry metrics corresponding to the second output, second Al workflow, and / or second Al model. Based on the comparison, the dynamic workflow system 104 determines that the second output has a more favorable score (e.g., it is higher, better, more accurate, and / or more efficient) than the first output. For example, in the case of text recognition, the second output identifies more correct words than the first output. As another example, the second Al workflow generates the second output twice as fast as the first Al workflow takes to generate the first output given the same input.

[0078] Act 432 includes the dynamic workflow system 104 adding the second Al workflow to the dynamic Al workflow table. Based on determining that the second output is more favorable (e.g., through A / B testing), the dynamic workflow system 104 can add the second Al workflow to the dynamic Al workflow table. In particular, upon completing testing of the second Al anddetermining that it is acceptable and / or ready for runtime execution, the dynamic workflow system 104 may add it to the dynamic Al workflow table. For instance, the dynamic workflow system 104 moves the second Al workflow from the testing Al workflow table to the dynamic Al workflow table. Indeed, the dynamic workflow system 104 can dynamically add and remove Al workflows from the dynamic Al workflow table at any time. By adding the second Al workflow to the dynamic Al workflow table, the operating system 202 is now able to use it for runtime executions to perform the task.

[0079] Act 434 includes removing the first Al workflow from the dynamic Al workflow table. For example, the dynamic workflow system 104 may replace the first Al workflow with the second Al workflow by removing the first Al workflow. In some instances, the second Al workflow is an updated version of the first Al workflow. In various implementations, the dynamic workflow system 104 adds the second Al workflow to the dynamic Al workflow table without removing the first Al workflow.

[0080] While FIGS. 4A-4B show one implementation of concurrent A / B testing of Al workflows that is transparent to a caller, in some implementations, the dynamic workflow system 104 causes the operating system 202 on one or more client devices to execute the first Al workflow to complete the task in some instances and to execute the second Al workflow to complete the task in other instances. These implementations demonstrate another way the dynamic workflow system 104 allows two versions of an Al workflow to be tested and evaluated.

[0081] As shown, the dynamic workflow system 104 may perform testing and evaluating Al workflow in connection with the operating system 202 performing runtime execution of tasks. The dynamic workflow system 104 adds new and / or updated workflows to the dynamic Al workflow table to be used by the operating system 202. Similarly, in various implementations, the dynamic workflow system 104 evaluates Al workflows within the dynamic Al workflow table and determines when a workflow is causing issues and should be removed or disabled, as discussed in connection with the next figure.

[0082] As mentioned above, FIG. 5 also provides additional details regarding the dynamic workflow system 104 implementing testing and evaluating Al workflows. For instance, FIG. 5 illustrates an example sequence flow diagram of evaluating and updating Al workflows in the dynamic Al workflow table according to some implementations. FIG. 5 includes the same components shown in FIG. 3. FIG. 5 also includes a series of acts 500 performed by the operating system 202 and / or dynamic workflow system 104 to perform Al workflow evaluations.

[0083] In FIG. 5, the series of acts 500 includes act 502 of the operating system 202 communicating with the application 206 (e.g., the caller), the data store 210, and the Al models 214 to receive operation calls and execute them using Al workflows, using the approachesdescribed above. In particular, the Al workflow API 208 of the operating system 202 executes Al workflows from a dynamic Al workflow table to obtain output results from the Al models 214, and the outputs are returned to the application 206.

[0084] Act 504 includes the dynamic workflow system 104 generating patterns for one or more Al workflows. For example, the dynamic workflow system 104 determines telemetry metrics for Al workflows based on their usage, outputs, execution times, production, and / or errors. For instance, the dynamic workflow system 104 tracks how frequently an Al workflow is called, when it is called, and / or the number of different callers. Based on analyzing collected information for an Al workflow, the dynamic workflow system 104 generates patterns, statistics, and other telemetry metrics. Based on the patterns, the dynamic workflow system 104 may perform various actions, as provided below.

[0085] As shown, FIG. 5 includes various dashed horizontal lines. Each set corresponds to an example implementation regarding evaluating Al workflows. Each implementation may follow act 504, which is depicted by the dashed arrows originating from the top dashed horizontal line following act 504.

[0086] Act 506 corresponds to one implementation and includes the dynamic workflow system 104 determining that an Al workflow is faulty. For example, based on the patterns or other data, the dynamic workflow system 104 determines that an Al workflow includes a bug, a flaw, or is otherwise underperforming. In some instances, the dynamic workflow system 104 determines that the Al model in the Al workflow is faulty. Accordingly, the dynamic workflow system 104 may disable or remove the faulty Al workflow, as shown in act 508. This way, the dynamic workflow system 104 may remotely disable a workflow or model without affecting other workflows in the dynamic Al workflow table and / or without taking down an entire chain of Al workflows.

[0087] Act 510 corresponds to another implementation and includes the dynamic workflow system 104 determining that an Al workflow includes extra steps. For example, the dynamic workflow system 104 determines that a given Al workflow takes longer than the average to complete and / or that similar workflows include fewer steps or actions. In some instances, the dynamic workflow system 104 determines that a given Al workflow uses an older or outdated Al model.

[0088] In some implementations, the dynamic workflow system 104 determines that an Al workflow is not operating efficiently. For example, the dynamic workflow system 104 determines that a given Al workflow takes longer than the average to complete or that a given Al workflow is infrequently called while other workflows that accomplish similar tasks are frequently called. In some implementations, based on the patterns and / or telemetry metrics, the dynamic workflowsystem 104 determines that a given Al workflow is executing below an efficiency and accuracy threshold.

[0089] Act 512 includes the dynamic workflow system 104 generating a modified Al workflow with fewer steps (or that is more efficient or accurate). For example, the new Al workflow removes one or more unnecessary steps. In various implementations, the dynamic workflow system 104 may also perform testing of the new Al workflow, as described above, to ensure that the new Al workflow operates correctly and is more efficient than its previous version.

[0090] Act 514 includes the dynamic workflow system 104 providing the modified version of the Al workflow to the data store 210 to replace the previous version. For example, the dynamic workflow system 104 replaces the previous version of the Al workflow in the dynamic Al workflow table with the updated Al workflow version. This way, the dynamic workflow system 104 dynamically and transparently updates and swaps Al workflows with modified versions without the caller being aware of the change.

[0091] Act 516 corresponds to yet another implementation and includes the dynamic workflow system 104 determining that a third-party Al model is more accurate or efficient. For example, the dynamic workflow system 104 determines that a given Al workflow uses an older or outdated Al model. In some instances, the dynamic workflow system 104 determines that other Al workflows use more recent versions of an Al model or newer Al models to achieve the same task.

[0092] In one or more implementations, the dynamic workflow system 104 determines that a new application is installed or activated on the client device, which provides one or more updated Al models. For example, when a photo editing software application is added to a client device, the dynamic workflow system 104 determines that the new application provides one or more state- of-the-art image processing Al models. While newer Al models may be available on remote devices (e.g., network servers), this disclosure relates to locally stored Al models where no network connection is required during operation calls. However, in some instances, the dynamic workflow system 104 may determine that a new or updated Al model has been downloaded to the client device.

[0093] Act 518 includes the dynamic workflow system 104 providing the modified version of the Al workflow to the data store 210 to replace the previous version. For example, the dynamic workflow system 104 replaces the previous version of the Al workflow with the modified version within the dynamic Al workflow table stored in the data store 210, as described above.

[0094] Act 520 includes the dynamic workflow system 104 providing the updated Al workflow to replace the previous version. For example, the dynamic workflow system 104 replaces the previous version of the Al workflow with the updated Al workflow version thatutilizes the updated Al models, including third-party Al models.

[0095] As mentioned above, FIG. 6 provides additional details regarding the dynamic workflow system providing a separate API for permitting customized scripts with Al workflows. In particular, FIG. 6 illustrates an example sequence flow diagram of the dynamic workflow system providing a separate API for accessing Al workflows.

[0096] As shown, FIG. 6 includes various components in communication with each other, including the operating system 202 (with the dynamic workflow system 104 and the Al workflow API 208), the data store 210, and the Al models 214, as introduced earlier. Additionally, FIG. 6 includes a script 602. The script 602 represents a script, user, application, PowerShell, or system that directly interacts with the dynamic workflow system 104 to access the dynamic Al workflow table and / or other Al workflow tables.

[0097] FIG. 6 also includes a series of acts 600 performed by the dynamic workflow system 104 to accomplish tasks using a dynamic Al workflow table. The series of acts 600 includes act 604 of providing a separate API to access Al workflow tables. In various implementations, the separate API is separate and distinct from the Al workflow API implemented by the OS. For example, the separate API allows scripts, applications, and users to access and utilize Al workflows in several ways, including ways not permitted by the Al workflow API.

[0098] To elaborate, by providing a separate API, the dynamic workflow system enables scripts to make customized operation calls that may directly interface with Al workflows to accomplish specific and complex tasks. By using scripts, the dynamic workflow system 104 enables tests, experiments, and evaluations to be conducted without affecting the system operations or runtime operation calls. Indeed, the separate API permits scripts to use the Al workflows in a customized manner.

[0099] To illustrate, act 606 includes the dynamic workflow system 104 receiving an operation call to perform a task from the script 602. In some implementations, the dynamic workflow system 104 receives commands or instructions for executing Al workflows or portions of Al workflows. In response, the dynamic workflow system 104 obtains one or more Al workflow tables from the data store 210, as shown in act 608. For example, the dynamic workflow system 104 obtains the dynamic Al workflow table, testing Al workflow table, and / or other versions of Al workflow tables. In some implementations, the dynamic workflow system 104 obtains an Al workflow not stored in an Al workflow table.

[0100] Act 610 includes the dynamic workflow system 104 executing one or more Al workflows using the Al models 214. For example, the dynamic workflow system 104 follows the commands or instructions in the script 602 to select and execute one or more Al workflows. Additionally, the separate API includes interfaces that are not included in the Al workflow API208 and allows the script 602 direct interaction with an Al workflow or multiple Al workflows in one or more Al workflow tables.

[0101] In some implementations, based on the script 602, the dynamic workflow system 104 performs portions of an Al workflow or deviates from the steps in an Al workflow. For instance, the script 602 causes the dynamic workflow system 104 to execute a specific Al workflow with a deviation of sending the input to a different Al model than included in the Al workflow.

[0102] Act 612 includes the dynamic workflow system 104 storing, analyzing, and / or returning results to the script 602. For example, upon obtaining one or more outputs, the dynamic workflow system 104 provides them to the script 602. In some instances, the dynamic workflow system 104 stores the results of the operation calls and / or script instructions. Furthermore, following the script 602, the dynamic workflow system 104 may analyze and perform one or more actions to add, remove, update, or modify Al workflows from the dynamic Al workflow table, as described above.

[0103] Turning now to FIG. 7 and FIG. 8, each of these figures illustrates an example series of acts in a computer-implemented method for utilizing one or more Al workflows on a client device according to some implementations. While FIG. 7 and FIG. 8 both illustrate acts according to one or more implementations, alternative implementations may omit, add, reorder, and / or modify any of the acts shown.

[0104] The acts in FIG. 7 and FIG. 8 can be performed as part of a method (e.g., a computer- implemented method). Alternatively, a computer-readable medium can include instructions that, when executed by a processing system with a processor, cause a computing device to perform the acts in FIG. 7 or FIG. 8. In some implementations, a system (e.g., a processing system comprising a processor) can perform the acts in FIG. 7 or FIG. 8. For example, the system includes a processing system and a computer memory including instructions that, when executed by the processing system, cause the system to perform various actions, operations, or steps.

[0105] To illustrate, in FIG. 7, the series of acts 700 includes act 710 of obtaining a dynamic Al workflow table upon receiving an operation call from a caller to perform a first task. For instance, in example implementations, act 710 involves obtaining, upon receiving an operation call from a caller to perform a first task, a dynamic Al workflow table that includes a set of dynamic Al workflows, wherein at least one Al workflow in the dynamic Al workflow table uses an Al model on the client device to perform operations.

[0106] In various implementations, in connection with act 710, obtaining the set of dynamic Al workflows includes querying a database that includes a dynamic Al workflow table that includes the set of dynamic Al workflows. In some implementations, an application programming interface (API) on the operating system of the client device receives the operation call to performthe first task from the caller and / or selects the new Al workflow from the dynamic Al workflow table to perform the first task. In some implementations, the caller is an application on the client device.

[0107] As fiirther shown, the series of acts 700 includes act 720 of selecting a new Al workflow from the dynamic Al workflow table to perform the first task. For instance, in example implementations, act 720 involves selecting a new Al workflow from the dynamic Al workflow table to perform the first task, wherein the new Al workflow was not included in the dynamic Al workflow table at a previous operation call from the caller to perform the first task. In one or more implementations, obtaining the set of dynamic Al workflows and executing the first Al workflow occurs on the client device without making any network calls to remote devices or remote resources.

[0108] In various implementations, the first Al workflow includes input parameters associated with a first Al model on the client device. In some implementations, executing the first Al workflow includes providing an input included in the operation call according to the input parameters to the first Al model and receiving the first output from the first Al model.

[0109] As fiirther shown, the series of acts 700 includes act 730 of executing the new Al workflow to generate a first output. For instance, in some implementations, act 730 involves executing the new Al workflow from the set of dynamic Al workflows on the client device to generate a first output.

[0110] Furthermore, the series of acts 700 includes act 740 of providing the first output to the caller. For instance, in example implementations, act 740 involves providing the first output from executing the new Al workflow to the caller in response to the operation call.

[0111] The series of acts 700 can include additional acts. For example, in some implementations, the series of acts 700 includes acts of obtaining a previous version of the dynamic Al workflow table that does not include the new Al workflow upon or when receiving the previous operation call from the caller to perform the first task, executing an Al workflow from the previous version of the dynamic Al workflow table to generate a second output, and providing the second output to the caller in response to the previous operation call. In various implementations, the series of acts 700 also includes acts of receiving a modified version of a second Al workflow, where the second Al workflow is included in the dynamic Al workflow table, and updating the second Al workflow within the dynamic Al workflow table with the modified version of the second Al workflow.

[0112] In various implementations, the series of acts 700 includes acts of determining that a second Al workflow includes a fault and removing the second Al workflow from the dynamic Al workflow table without affecting the operations of other Al workflows in the dynamic Alworkflow table.

[0113] To illustrate, in FIG. 8, the series of acts 800 includes act 810 of obtaining a dynamic Al workflow table upon receiving an operation call from a caller to perform a first task. For instance, in example implementations, act 810 involves obtaining a set of dynamic Al workflows, wherein at least one Al workflow in the set of dynamic Al workflows uses an Al model stored on the client device to perform operations upon receiving an operation call from a caller to perform a first task.

[0114] As farther shown, the series of acts 800 includes act 820 of executing a first Al workflow to generate a first output. For instance, in example implementations, act 820 involves performing the first task by executing a first Al workflow from the set of dynamic Al workflows on the client device to generate a first output.

[0115] As fiirther shown, the series of acts 800 includes act 830 of executing a second Al workflow to generate a second output. For instance, in some implementations, act 830 involves performing the first task by executing a second Al workflow on the client device to generate a second output. In some implementations, the caller is unaware that the second Al workflow has been executed in connection with performing the first task. In one or more implementations, act 830 includes receiving the second Al workflow to perform the first task, updating the set of dynamic Al workflows stored in a database on the client device to include the second Al workflow, and executing the second Al workflow to perform the first task in response to receiving a subsequent operation call to perform the first task.

[0116] Furthermore, the series of acts 800 includes act 840 of providing the first output to the caller. For instance, in example implementations, act 840 involves providing the first output from executing the first Al workflow to the caller in response to the operation call.

[0117] As fiirther shown, the series of acts 800 includes act 850 of storing the second output without providing it to the caller. For instance, in example implementations, act 850 involves storing the second output from executing the second Al workflow without providing the second output to the caller.

[0118] The series of acts 800 can include additional acts. For example, in some implementations, the series of acts 800 includes acts of comparing the first output of the first Al workflow with the second output of the second Al workflow to determine a more favorable Al workflow for performing the first task. In one or more implementations, the second Al workflow is not included in the set of dynamic Al workflows and is selected from a testing set of dynamic Al workflows. In various implementations, the series of acts 800 includes acts of adding the second Al workflow to the set of dynamic Al workflows and removing the first Al workflow from the set of dynamic Al workflows based on determining that the second output is more favorablethan the first output.

[0119] In various implementations, the first Al workflow and the second Al workflow are selected from different instances of the set of dynamic Al workflows. In one or more implementations, the series of acts 800 includes acts of removing the second Al workflow from the set of dynamic Al workflows based on determining that the first output is more favorable to the first output.

[0120] FIG. 9 illustrates certain components that may be included within a computer system 900. The computer system 900 may be used to implement the various computing devices, components, and systems described herein (e.g., by performing computer-implemented instructions). As used herein, a “computing device” refers to electronic components that perform a set of operations based on a set of programmed instructions. Computing devices include groups of electronic components, client devices, server devices, etc.

[0121] In various implementations, the computer system 900 represents one or more of the client devices, server devices, or other computing devices described above. For example, the computer system 900 may refer to various types of network devices capable of accessing data on a network, a cloud computing system, or another system. For instance, a client device may refer to a mobile device such as a mobile telephone, a smartphone, a personal digital assistant (PDA), a tablet, a laptop, or a wearable computing device (e.g., a headset or smartwatch). A client device may also refer to a non-mobile device such as a desktop computer, a server node (e.g., from another cloud computing system), or another non-portable device.

[0122] The computer system 900 includes a processing system including a processor 901. The processor 901 may be a general-purpose single- or multi-chip microprocessor (e.g., an Advanced Reduced Instruction Set Computer (RISC) Machine (ARM)), a special -purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. The processor 901 may be referred to as a central processing unit (CPU) and may cause computer- implemented instructions to be performed. Although the processor 901 shown is just a single processor in the computer system 900 of FIG. 9, in an alternative configuration, a combination of processors (e.g., an ARM and DSP) could be used.

[0123] The computer system 900 also includes memory 903 in electronic communication with the processor 901. The memory 903 may be any electronic component capable of storing electronic information. For example, the memory 903 may be embodied as random-access memory (RAM), read-only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, on-board memory included with the processor, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, and so forth, including combinations thereof.

[0124] The instructions 905 and the data 907 may be stored in the memory 903. The instructions 905 may be executable by the processor 901 to implement some or all of the functionality disclosed herein. Executing the instructions 905 may involve the use of the data 907 that is stored in the memory 903. Any of the various examples of modules and components described herein may be implemented, partially or wholly, as instructions 905 stored in memory 903 and executed by the processor 901. Any of the various examples of data described herein may be among the data 907 that is stored in memory 903 and used during the execution of the instructions 905 by the processor 901.

[0125] A computer system 900 may also include one or more communication interface(s) 909 for communicating with other electronic devices. The one or more communication interface(s) 909 may be based on wired communication technology, wireless communication technology, or both. Some examples of the one or more communication interface(s) 909 include a Universal Serial Bus (USB), an Ethernet adapter, a wireless adapter that operates according to an Institute of Electrical and Electronics Engineers (IEEE) 902.11 wireless communication protocol, a Bluetooth® wireless communication adapter, and an infrared (IR) communication port.

[0126] A computer system 900 may also include one or more input device(s) 911 and one or more output device(s) 913. Some examples of the one or more input device(s) 911 include a keyboard, mouse, microphone, remote control device, button, joystick, trackball, touchpad, and light pen. Some examples of the one or more output device(s) 913 include a speaker and a printer. A specific type of output device that is typically included in a computer system 900 is a display device 915. The display device 915 used with implementations disclosed herein may utilize any suitable image projection technology, such as liquid crystal display (LCD), light-emitting diode (LED), gas plasma, electroluminescence, or the like. A display controller 917 may also be provided, for converting data 907 stored in the memory 903 into text, graphics, and / or moving images (as appropriate) shown on the display device 915.

[0127] The various components of the computer system 900 may be coupled together by one or more buses, which may include a power bus, a control signal bus, a status signal bus, a data bus, etc. For clarity, the various buses are illustrated in FIG. 9 as a bus system 919.

[0128] Furthermore, upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to non-transitory computer-readable storage media (devices), or vice versa. For example, computer-executable instructions or data structures received over a network or data link can be buffered in random-access memory (RAM) within a network interface module (NIC), and then it is eventually transferred to computer system RAM and / or to less volatile computer storage media (devices) at a computer system. Thus, it should be understoodthat computer-readable storage media (devices) can be included in computer system components that also (or even primarily) utilize transmission media.

[0129] Computer-executable instructions include instructions and data that, when executed by a processor, cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions. In some implementations, computer-executable and / or computer-implemented instructions are executed by a general- purpose computer to turn the general-purpose computer into a special-purpose computer implementing elements of the disclosure. The computer-executable instructions may include, for example, binaries, intermediate format instructions such as assembly language, or even source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0130] Those skilled in the art will appreciate that the disclosure may be practiced in network computing environments with many types of computer system configurations, including, personal computers, desktop computers, laptop computers, message processors, hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. The disclosure may also be practiced in distributed system environments where local and remote computer systems, which are linked (either by hardwired data links, wireless data links, or a combination of hardwired and wireless data links) through a network, both perform tasks. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0131] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof unless specifically described as being implemented in a specific manner. Any features described as modules, components, or the like may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory processor-readable storage medium, including instructions that, when executed by at least one processor, perform one or more of the methods described herein (including computer- implemented methods). The instructions may be organized into routines, programs, objects, components, data structures, etc., which may perform particular tasks and / or implement particular data types, and which may be combined or distributed as desired in various implementations.

[0132] Computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that storecomputer-executable instructions are non-transitory computer-readable storage media (devices). Computer-readable media that carry computer-executable instructions are transmission media. Thus, by way of example, implementations of the disclosure can include at least two distinctly different kinds of computer-readable media: non-transitory computer-readable storage media (devices) and transmission media.

[0133] As used herein, computer-readable storage media (devices) may include RAM, ROM, EEPROM, CD-ROM, solid-state drives (SSDs) (e.g., based on RAM), Flash memory, phasechange memory (PCM), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computer.

[0134] The steps and / or actions of the methods described herein may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is required for the proper operation of the method that is being described, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0135] The term “determining” encompasses a wide variety of actions and, therefore, “determining” can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a data repository, or another data structure), ascertaining, and the like. Also, “determining” can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Also, “determining” can include resolving, selecting, choosing, establishing, and the like.

[0136] The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. Additionally, it should be understood that references to “one implementation” or “implementations” of the present disclosure are not intended to be interpreted as excluding the existence of additional implementations that also incorporate the recited features. For example, any element or feature described concerning an implementation herein may be combinable with any element or feature of any other implementation described herein, where compatible.

[0137] The present disclosure may be embodied in other specific forms without departing from its spirit or characteristics. The described implementations are to be considered illustrative and not restrictive. The scope of the disclosure is indicated by the appended claims rather than by the foregoing description. Changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

CLAIMS1. A computer-implemented method for dynamic utilizing one or more artificial intelligence (Al) workflows on a client device (102), comprising: upon receiving an operation call (132) from a caller (130) to perform a first task (134), obtaining a set of Al workflows (138), wherein at least one Al workflow in the set of Al workflows (138) uses an Al model stored on the client device (102) to perform operations; performing the first task (134) by: executing a first Al workflow (142) from the set of Al workflows (138) on the client device (102) to generate a first output (144); and executing a second Al workflow (146) on the client device (102) to generate a second output (148); providing the first output (144) from executing the first Al workflow (142) to the caller (130) in response to the operation call (132); and storing the second output (148) from executing the second Al workflow (146) without providing the second output (148) to the caller (130).

2. The computer-implemented method of claim 1, wherein the caller is unaware that the second Al workflow has been executed in connection with performing the first task.

3. The computer-implemented method of claim 1, further comprising comparing the first output of the first Al workflow with the second output of the second Al workflow to determine a more favorable Al workflow for performing the first task.

4. The computer-implemented method of claim 3, wherein: the second Al workflow is not included in the set of Al workflows; and the second Al workflow is selected from a testing set of Al workflows.

5. The computer-implemented method of claim 4, further comprising, based on determining that the second output is more favorable than the first output: adding the second Al workflow to the set of Al workflows; and removing the first Al workflow from the set of Al workflows.

6. The computer-implemented method of claim 3, wherein the first Al workflow and the second Al workflow are selected from different instances of the set of Al workflows.

7. The computer-implemented method of claim 6, further comprising, based on determining that the first output is more favorable than the second output, removing the second Al workflow from the set of Al workflows.

8. The computer-implemented method of claim 1, wherein: the first Al workflow includes input parameters associated with a first Al model on the client device; andexecuting the first Al workflow includes: providing an input included in the operation call according to the input parameters to the first Al model; and receiving the first output from the first Al model.

9. The computer-implemented method of claim 1 , wherein obtaining the set of Al workflows and executing the first Al workflow occurs on the client device without making any network calls to remote devices or remote resources.

10. The computer-implemented method of claim 1 , further comprising: receiving the second Al workflow to perform the first task; updating the set of Al workflows stored in a data store on the client device to include the second Al workflow; and in response to receiving a subsequent operation call to perform the first task, executing the second Al workflow to perform the first task.

11. The computer-implemented method of claim 1 , wherein obtaining the set of Al workflows includes querying a data store that includes a dynamic Al workflow table that includes the set of Al workflows.

12. A computer-implemented method for dynamic utilizing one or more artificial intelligence (Al) workflows on a client device (102), comprising: upon receiving an operation call (132) from a caller (130) to perform a first task (134), obtaining a dynamic Al workflow table (136) that includes a set of Al workflows (138), wherein at least one Al workflow in the dynamic Al workflow table (136) uses an Al model on the client device (102) to perform operations; selecting a new Al workflow (140) from the dynamic Al workflow table (136) to perform the first task (134), wherein the new Al workflow (140) was not included in the dynamic Al workflow table (136) at a previous operation call (132) from the caller (130) to perform the first task (134); executing the new Al workflow (140) from the set of Al workflows (138) on the client device (102) to generate a first output (144); and providing the first output (144) from executing the new Al workflow (140) to the caller (130) in response to the operation call (132).

13. The computer-implemented method of claim 12, further comprising: upon receiving the previous operation call from the caller to perform the first task, obtaining a previous version of the dynamic Al workflow table that does not include the new Al workflow; executing an Al workflow from the previous version of the dynamic Al workflow table togenerate a second output; and providing the second output to the caller in response to the previous operation call.

14. The computer-implemented method of claim 13, further comprising: receiving a modified version of a second Al workflow, wherein the second Al workflow is included in the dynamic Al workflow table; and updating the second Al workflow within the dynamic Al workflow table with the modified version of the second Al workflow.

15. The computer-implemented method of claim 12, further comprising: determining that a second Al workflow includes a fault; and removing the second Al workflow from the dynamic Al workflow table without affecting the operations of other Al workflows in the dynamic Al workflow table.

16. The computer-implemented method of claim 12, wherein an application programming interface (API) on an operating system of the client device: receives the operation call to perform the first task from the caller, and selects the new Al workflow from the dynamic Al workflow table to perform the first task.

17. The computer-implemented method of claim 12, wherein the caller is an application on the client device.

18. The computer-implemented method of claim 12, wherein obtaining the dynamic Al workflow table and executing the new Al workflow occurs on the client device without making any network calls to remote devices or remote resources.

19. A system comprising: a client device (102) having a dynamic Al workflow table (136) with a set of Al workflows (138), wherein at least one Al workflow in the set of Al workflows (138) uses an Al model stored on the client device (102) to perform operations; a processor (901); and a non-transitory computer memory (903) comprising instructions (905) that, when executed by the processor (901), cause the system to perform operations of: upon receiving an operation call (132) from a caller (130) to perform a first task (134), obtaining the set of Al workflows (138); performing the first task (134) by: executing a first Al workflow (142) from the set of Al workflows (138) on the client device (102) to generate a first output (144); and executing a second Al workflow (146) on the client device (102) to generate a second output (148); providing the first output (144) from executing the first Al workflow (142) to thecaller (130) in response to the operation call (132); and storing the second output (148) from executing the second Al workflow (146) without providing the second output (148) to the caller (130).

20. The system of claim 19, farther comprising additional instructions that, when executed by the processor, cause the system to perform the operations of: comparing the first output of the first Al workflow with the second output of the second Al workflow to determine a more favorable Al workflow for performing the first task; and based on determining that the second output is more favorable than the first output: adding the second Al workflow to the set of Al workflows; and removing the first Al workflow from the set of Al workflows.

Images