Materializing the analytical workspace

Abstract

Techniques for creating a workspace are disclosed. A data processing system receives a request to create a workspace for executing a portion of a model deployed in a production environment. One or more data objects and their associated metadata related to the portion of the model, and an execution location of the workspace are obtained. A set of instructions for executing the one or more data objects and associated metadata in the workspace is generated. The workspace is created in the execution location by instantiating the portion of the model, the one or more data objects, and the associated metadata in the workspace. The portion of the model is processed in the workspace using the one or more data objects and associated metadata according to the set of instructions, and the production environment is updated by the data processing system based on the processing.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to Indian Provisional Application No. 202041033832, filed August 7, 2020, and U.S. Utility Application No. 17 / 333,738, filed May 28, 2021, the entire contents of which are incorporated herein by reference for all purposes.

[0002] Field The present disclosure relates generally to data processing systems, and more particularly to techniques for creating workspaces in a seamless manner for executing at least a portion of a model deployed in a production environment. [Background technology]

[0003] background

[0003] With the popularity of the Internet, enterprises in various industries are constantly generating and processing huge amounts of data. To utilize this data, various enterprises extract many different data indexes from the data according to their business preferences, and these data indexes are used to measure the business development status of the enterprises. For example, in an e-commerce website application, the data indexes for measuring enterprise services may include the number of active user visits to a web page per day, the user click-through rate, the conversion rate per web page, etc.

[0004] Typically, the processes undertaken by an organization are modeled as workflows, or pipelines, and various staff members, such as data scientists, modelers, and analysts, can monitor the performance of the workflow. In such workflows, building a complex set of deterministic and statistical data transformations to support use cases that derive predictive outcomes typically requires the creation of private workspaces. These private workspaces are typically addressed via a host of desktop applications or manual processes to re-correlate the results with the source data and connect the resulting logical / computational transformations with the generative use cases. This approach to creating workspaces is inefficient because it tends to create silos and lose lineage to the overall workflow.

[0005] The embodiments described herein address these and other issues individually and collectively. Summary of the Invention [Means for solving the problem]

[0006] overview Techniques (e.g., methods, systems, non-transitory computer-readable media storing code or instructions executable by one or more processors) are provided for creating a workspace for executing at least a portion of a model deployed in a generative environment. Various embodiments are described herein, including methods, systems, programs executable by one or more processors, non-transitory computer-readable storage media storing code or instructions, etc.

[0007] In various embodiments, a method is provided, the method comprising: a data processing system receiving a request to create a workspace for executing at least a portion of a model deployed in a production environment; the data processing system obtaining one or more data objects and their associated metadata related to the portion of the model; the data processing system obtaining an execution location for the workspace; the data processing system generating a set of instructions for executing the one or more data objects and the associated metadata in the workspace; and the data processing system creating the workspace in the execution location, the creating comprising instantiating the portion of the model, the one or more data objects and the associated metadata in the workspace; the method further comprising processing the portion of the model in the workspace using the one or more data objects and the associated metadata in accordance with the set of instructions; and the data processing system updating the production environment based on the processing.

[0008] In some embodiments, the method further comprises retrieving a plurality of data objects and a plurality of pieces of metadata from one or more databases associated with the production environment, the one or more databases maintaining a mapping of models and associated data objects deployed to the production environment, and filtering the plurality of data objects and the plurality of pieces of metadata to retrieve the one or more data objects and the associated metadata associated with the portion of the model; and iteratively performing a set of lookup operations on the one or more databases associated with the production environment to retrieve at least one data object related to the one or more data objects.

[0009] In some embodiments, the execution location is an area within an application instance within the production environment, or a remote application instance.

[0010] In some embodiments, the method further comprises receiving the request from a user to execute the portion of the model in the workspace; verifying whether the user is authorized to access the one or more data objects related to the portion of the model and their associated metadata; and, in response to successful verification, generating a provisioning executable that creates the workspace within the execution location.

[0011] In some embodiments, the workspace is further configured to ingest data from a data source external to the data processing system and create at least one sandbox application or analytics pipeline based on the portion of the model, and the portion of the model in the workspace is processed using the at least one sandbox application or analytics pipeline.

[0012] In some embodiments, the set of instructions is generated in a format that is convertible into an equivalent set of instructions that can be executed at the execution location that hosts the workspace.

[0013] In some embodiments, updating the production environment further comprises obtaining an initial state of the production environment before creating the workspace within the execution location, the initial state including a first output of the model deployed to the production environment, and updating the production environment further comprises replacing the portion of the model deployed to the production environment with the portion of the model processed in the workspace, and generating an updated state of the production environment based on the replacement, the updated state including a second output of the model deployed to the production environment.

[0014] In some embodiments, the model deployed to the production environment includes a plurality of sub-models, and the portion of the model corresponds to a sub-model of the plurality of sub-models.

[0015] In some embodiments, processing the portion of the model in the workspace further comprises retraining the portion of the model on a training dataset and modifying one or more parameters of the portion of the model based on the retraining.

[0016] In some embodiments, creating the workspace within the execution location further comprises the data processing system generating a provisioning executable that enables the workspace to be created within the execution location, and executing the provisioning executable to create the workspace within the execution location.

[0017] In some embodiments, the method further comprises storing configuration information of the workspace in a database of the data processing system, the configuration information including at least the one or more data objects related to the portion of the model and their associated metadata, and information related to the execution location, and the method further comprises the data processing system deleting the workspace created at the execution location, and the data processing system generating a new workspace at another execution location based on the configuration information.

[0018] In some embodiments, the plurality of sub-models comprises a neural network model, a linear regression model, a support vector machine model, or a clustering model.

[0019] In some embodiments, a computing device is provided, the computing device including a processor and a memory including instructions that, when executed by the processor, cause the computing device to at least receive a request to create a workspace for executing at least a portion of a model deployed to a generative environment; obtain one or more data objects and their associated metadata associated with the portion of the model; obtain an execution location for the workspace; generate a set of instructions for executing the one or more data objects and the associated metadata in the workspace; create the workspace in the execution location by instantiating the portion of the model, the one or more data objects and the associated metadata in the workspace; process the portion of the model in the workspace using the one or more data objects and the associated metadata in accordance with the set of instructions; and update the generative environment based on the processing.

[0020] In some embodiments, a non-transitory computer-readable medium is provided that stores certain computer-executable instructions that, when executed by a processor, cause a computer system to at least receive a request to create a workspace for executing at least a portion of a model deployed to a generative environment; obtain one or more data objects and their associated metadata associated with the portion of the model; obtain an execution location for the workspace; generate a set of instructions for executing the one or more data objects and the associated metadata in the workspace; create the workspace in the execution location by instantiating the portion of the model, the one or more data objects, and the associated metadata in the workspace; process the portion of the model in the workspace using the one or more data objects and the associated metadata in accordance with the set of instructions; and update the generative environment based on the processing.

[0021] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. As described in more detail below, some example implementations and contexts are provided with reference to the following drawings. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is an illustration of a data processing system according to various embodiments. [Figure 2] FIG. 1 is an illustration of provisioning a workspace according to various embodiments. [Figure 3] FIG. 1 illustrates details of provisioning a workspace, according to various embodiments. [Figure 4] FIG. 2 illustrates a data interface unit according to various embodiments. [Figure 5] FIG. 1 illustrates a flowchart illustrating a workspace provisioning process, according to various embodiments. [Figure 6] 1 is a simplified diagram of a distributed system for implementing various embodiments. [Figure 7] FIG. 1 is a simplified block diagram of one or more components of a system environment in which services provided by one or more components of an embodiment system may be provided as cloud services, according to various embodiments. [Figure 8] FIG. 1 illustrates an exemplary computer system that can be used to implement various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0023] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments of the present invention. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, illustration, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0024] Data Processing System Overview Generally, enterprises require constant monitoring of workflows that measure risk to comply with prevailing regulatory and supervisory standards. In such workflows, measurements often involve substantial computation and validation using enterprise data. Data is transformed to support such measurements and calculations. The processes involved in transforming data involve large amounts of time data, and the processing methodology can be thought of as a series of tasks that perform data transformations and actions (including inserting / updating calculated values) that can be achieved by a set of prescribed rules.

[0025] A complex analytical workflow is modeled as an overall model (e.g., a machine learning model) composed of several submodels. Specifically, several submodels are assembled (i.e., combined) in a predetermined manner to generate the overall model. These submodels are executed in a predetermined order to determine one or more output variables of the overall model. Each submodel includes a set of tasks that operate on one or more datasets and perform data transformations to calculate one or more output variables of the overall model. Thus, execution of the overall model includes (i) determining multiple input variables (i.e., input parameters to one or more submodels), (ii) executing one or more submodels according to a predetermined order, and (iii) generating one or more output variables of the overall model. Such an overall model is deployed to an enterprise's production environment. Various staff members (e.g., developers, data scientists, etc.) from various departments of the enterprise can monitor the overall model in the production environment. The submodels may include machine learning models such as neural network models, linear regression models, support vector machine models, or clustering models.

[0026] A requirement in such complex analytical workflows is that various entities may want to extract portions (e.g., sub-models) of the overall model for further processing and development. For example, a corporate division that builds a particular sub-model of the overall model may want to extract the sub-model (and the associated data objects and metadata used to build this model) with the goal of further processing and improving the sub-model, e.g., retraining the sub-model on new training data or fine-tuning one or more parameters of the sub-model to improve performance. Alternatively, it may be desirable to obtain slices of generative environment data, e.g., data objects, data records, etc., for the purpose of building new predictive models or evaluating scenarios for anomaly detection, examining data patterns, etc. Described herein are mechanisms that enable the creation of workspaces (e.g., areas within an existing application instance, a remote application instance, or a hybrid environment) from which desired sub-models, data objects, metadata, etc. can be extracted for further processing without affecting the functioning of the overall model in the generative environment. According to some embodiments, the pipeline creation unit 111 includes a model aggregator. The model aggregator is programmed to take one or more sub-models (and their associated data objects and metadata) and combine the one or more sub-models in a user-defined manner to generate an overall model. The model aggregator may be implemented as a script, e.g., a Python script, that takes one or more sub-models that are combined together to form the overall model.

[0027] The various embodiments described herein may be used in a variety of different domains and contexts. Certain embodiments are particularly applicable to enterprise application software provided in distributed computing technologies. However, the systems and methods described herein can be used to provide data transformation capabilities for any system or application framework in which the creation of a workspace is desired. Techniques (e.g., systems, methods, and machine-readable media) are disclosed for enabling a data processing system to instantiate a workspace and extract relevant portions of an overall model, i.e., sub-models, specific data objects (and their associated metadata) into the workspace (from a generative environment) for further processing.

[0028] 1 illustrates a data processing system 100 including a client device 105, a generative application environment 110, a workspace creation unit 120, a network 130, an operations module 140, and data storage 150 (e.g., a memory store), according to various embodiments. The generative application environment 110 includes a pipeline creation unit 111 and a rules unit 113.

[0029] A user operating a client device 105 may be provided with one or more application interfaces (e.g., graphical user interfaces) that accept input and enable the user to interact with the generative application environment 110 and / or workspace creation unit 120. Examples of client devices 105 include, but are not limited to, workstations, personal computers (PCs), laptop computers, mobile devices such as smartphones, wearable computers, or other network-connected electronic devices. The interfaces displayed on the client device 105 may be accessible using applications executing on the client device 105. The client device 105, the generative application environment 110, the workspace creation unit 120, and the operation module 140 may communicate via one or more communication networks 130. Examples of communication networks 130 may include a mobile network, a wireless network, a cellular network, a local area network (LAN), a wide area network (WAN), other wireless communication networks, or combinations thereof.

[0030] According to some embodiments, the generative application environment 110 includes a pipeline creation unit 111 that allows a user to define an overall model as a pipeline (also referred to herein as a workflow), i.e., a combination of one or more sub-models that are organized / combined to form the overall model. The pipeline creation unit 111 also allows a user to define and execute a set of rules, reporting objects, and processes required to transform data in the data processing system 100. The pipeline creation unit 111 provides an interface that allows a user to define and execute rules, processes, and manage the definition of each sub-model used to form the overall model, for example, via a client device 105. For example, many companies require constant monitoring and measurement of risk to comply with prevailing regulatory and supervisory standards. These measurements often involve substantial computations and validations using the company's data. Data is frequently transformed to support such measurements and calculations. These data transformations may be achieved by a set of prescribed rules.

[0031] The pipeline creation unit 111 provisions for a user to define rules 113 for executing the overall model, e.g., to define the execution order of one or more sub-models that form the overall model. For example, the rules may include a first subset of one or more sub-models that are executed simultaneously, and a second subset of one or more sub-models that are executed sequentially to determine one or more output variables of the overall model. Furthermore, the pipeline creation unit 111 also provisions for a user to define, via an application interface, one or more datasets, e.g., obtained from an external data source, on which the overall model will be evaluated. Such datasets may be obtained from the external data source via the operation module 140.

[0032] The rules unit 113 facilitates users' definition of a set of rules, reporting objects, and processes to be executed (e.g., on submodels) to transform data in a storage device, such as data storage 150. For example, the rules unit 113 provides a framework that facilitates the definition and maintenance of transformations. A metadata abstraction layer may be used in the definition of rules, where users are permitted to reclassify attributes in data store 150, thereby transforming data. Basic metadata objects, such as hierarchies, which are non-large or non-list datasets and processors, drive rule functionality. Rules may include various types. For example, a Type 1 rule may involve creating a subset of records from a given set of records in a data model based on a specific filter. This process may involve transformation, aggregation, or a combination thereof. Type 1 rule definition may be achieved through table-to-table (T2T) extraction. Type 2 rules may involve reclassifying records in a table in a data model based on criteria that include complex group-by clauses and subqueries within the table. A Type 3 rule may involve calculating a new value or metric based on a simple measure and updating a set of identified records in the data model with the calculated value. The pipeline creation unit 111 includes a set of tools to help users work with various rule definitions.

[0033] In various embodiments, a rule is defined to include one or more components. These components may include a dataset, a source, a target, a mapping, and / or a node identifier. A dataset is a set of tables linked by a key. A dataset may have at least one fact table. A fact table stores process measurements, metrics, or facts. A fact table may be at the center of a star or snowflake schema surrounded by dimension tables. Dimension tables store reference data (e.g., a list of products, branches, geographies, currencies, etc., that an entity handles, which are relatively static). When multiple fact tables are used, they may be arranged as a fact constellation schema. Fact tables generally have two types of columns: columns that contain facts and columns that are foreign keys to dimension tables. The primary key of a fact table is typically a composite key composed of all its foreign keys. Fact tables contain the contents of the data store 150 and store various types of measures, such as additive, non-additive, and semi-additive measures. A Type 3 rule definition may be based on a dataset that includes two or more fact tables. A Type 2 rule definition may be based on a dataset that includes one fact table. Values ​​in one or more columns of a fact table in the dataset may be transformed with new values.

[0034] According to some embodiments, a set of rules collectively form a process. A process definition may be represented as a process tree. The rules unit 113 provides a framework that facilitates process definition and maintenance. By defining a process, a user can logically group a collection of rules related to a functional process. For example, a user can define a process with existing metadata objects using a hierarchical structure, which facilitates the construction of a process tree. A process tree may have many levels and one or many nodes within each level. In this manner, the generative application environment 110 can utilize the rules unit 113 to define a process for a submodel of a complex analytical workflow. It is understood that a submodel may include one or more tasks, each of which is executed to complete the execution of the submodel. Subprocesses (e.g., of a task in a submodel) may be defined in level members, and process hierarchy members form leaf members of the tree. Therefore, processes may include two types: (i) end-to-end processes that represent functional completeness and are ready to be executed, and (ii) non-end-to-end processes that are logical collections of rules but cannot be executed by themselves. A non-end-to-end process may be defined as a sub-process in the end-to-end process being executed. The rules unit 113 may be used to build a process tree by adding one or more members called process nodes. If there are predecessor tasks associated with any member, these tasks, defined as predecessor tasks, precede the execution of that member.

[0035] The definitions created by the rules unit 113 result in unique batch groups. These batches can then be scheduled for execution through an interface provided by the production application environment 110. Any request for a process execution (e.g., execution of a submodel) can become a batch in a batch group. The production application environment 110 allows users to have a workflow for executing processes. According to some embodiments, these batches can then be executed from the operations module 140. For example, a user may select a process definition marked for execution through an application programming interface (API), and the name of the selected process definition is displayed in the execution description text box of the API. The API displays other information, such as the requested execution ID, the requested execution description, the request status, the execution status, and options such as edit.

[0036] The API may have an option to select a batch to be defined for execution. A request status of "open" indicates that the selected process ID is ready to be executed. An execution status of "not started" indicates that the particular process definition / batch has not yet been executed. The user can select an edit option to edit the execution request. The user can also use search and filtering options to search for a particular process based on process name, process execution description, process execution ID, process type, request status, etc. Pagination options may also be provided to help the user manage a collaborative view of existing processes within the data processing system.

[0037] According to some embodiments, operations module 140 facilitates users in managing and processing data (e.g., enterprise data) to create the highest level of efficiency within data processing system 100 and derive results based on one or more predetermined rules. Operations module 140 may include units such as batch maintenance, batch execution, batch scheduler, batch monitoring, batch processing reporting, batch cancellation, log viewing, and report generation.

[0038] The batch maintenance framework in the operations module 140 facilitates users' creation and maintenance of batch definitions. Users can process batches scheduled for execution from batch maintenance and from other modules, such as the advanced analytics infrastructure module (not shown) and the rules unit 113. The batch execution unit in the operations module 140 is configured to launch a batch for processing. According to some embodiments, when a batch is submitted for execution, a series of commands are sent to the data store 150 for specified components and parameters. These commands, when successfully executed, return an array of update numbers (e.g., value definitions). The batch scheduler unit in the operations module 140 facilitates users' scheduling of batches for subsequent processing. Users can define new batch schedules or update previously defined batch schedules for processing.

[0039] The batch monitoring unit of the operation module 140 facilitates users to view the status of executed batch definitions along with task details. Users can track issuance, if any, at regular intervals to ensure smooth batch execution. The event log provides users with real-time status of executed batches. The batch processing report in the operation module 140 facilitates users to view the execution status of each task component defined in a batch. The batch cancellation unit of the operation module 140 facilitates users to cancel or abort a batch or a specific task that is scheduled to execute or is currently executing. The log display unit facilitates users to view the execution status of each task component defined in a batch. The report generation unit of the operation module 140 is configured to generate a performance report of the overall model that is provided to the client device 150.

[0040] The workspace creation unit 120 enables, for example, modelers, data and domain analysts, to create one or more workspaces. In such workspaces, data objects, logical abstractions, metadata, and / or slices of portions of the overall model (i.e., sub-models) from the generative environment can be exported, so that the exported information can be processed independently (i.e., separate from the generative environment) for various purposes. For example, the exported data objects, metadata, and logical abstractions can be used to build predictive models or explore scenarios or data patterns for anomaly detection. Exported sub-models can be further trained and enhanced to improve performance (e.g., optimizing the sub-model's parameters). The workspace creation unit 120 enables pulling both data and associated objects (logic / operations) into workspaces, i.e., areas within existing application instances or remote instances or hybrid environments, while applying common / adapted metadata via interactive, offline-generated instruction sets. Such an approach enables replication, virtualization, and deployment of trained logic / operations / models with automatic binding to targets defined by security credentials, self-expansion, and control.

[0041] According to some embodiments, workspace creation unit 120 may be implemented as an executable script, e.g., Java script, SQL script, etc., that extracts a copy of the sub-model, i.e., the code of the sub-model (i.e., takes a snapshot), and simultaneously extracts a copy of the metadata associated with the sub-model. This metadata may include, for example, data objects, external software packages, and data from external data sources associated with the sub-model. The extracted copy of the sub-model and its associated metadata is packaged into an archive, e.g., an executable zip archive. Workspace creation unit 120 implemented as an executable script may also define dependencies (e.g., data dependencies) in the metadata associated with the sub-model to ensure that all relevant data (required to execute the sub-model) is included in the zip archive. Thus, the archive can be exported to a workspace created for a specific user for the purpose of further developing or refining the sub-model without affecting the functioning of the overall model in the generative environment.

[0042] According to some embodiments, one or more users, e.g., modelers, data / domain analysts, etc., may create individual archives, i.e., packaged sub-models, that are exported to the users' respective workspaces. It is understood that multiple entities may create zip archives corresponding to the same sub-model. The users process the sub-models in their respective workspaces independently. For example, different users may retrain a sub-model using different datasets, different parameters, etc. Once processing of the sub-models in the users' various workspaces is complete, according to some embodiments, a particular sub-model from one of the workspaces is selected to replace the original sub-model in the generative environment. For example, the performance of each sub-model executed in the user's workspace is compared to the performance of other similar sub-models. In this manner, one of the sub-models that outperforms the other sub-models (e.g., achieves higher throughput for the sub-model) is selected to replace the original sub-model in the generative environment. Further processing / refinement of this sub-model by various users is then performed on the superseded sub-model. It is understood that older versions of the sub-models may be stored in the data store 150 associated with the generative environment for subsequent use or audit purposes.

[0043] According to some embodiments, the workspace creation unit 120 provides the client device 105 with a user interface wizard (e.g., part of the operation module 115) that enables the client device 105 to create a workspace. Workspace creation may include (i) data entity selection with controlled user credentials, (ii) automatic listing of associated metadata (logic / operations) associated with the selected logical data elements (i.e., datasets or data objects), (iii) enabling selection of related metadata as needed, (iv) scanning of local and remote environments that can host the workspace (i.e., workspace execution locations), and (v) list selection for workspace deployment (e.g., provisioning for hybrid data materialization). As described herein with reference to FIG. 3 , for each of the above steps, an instruction set is generated to create the workspace and export associated data to the workspace. It is understood that the instruction set can be generated in a domain-specific language or a scripting language such as Shell, Perl, or Python that includes associations to the logical model, metadata objects, and data files. Furthermore, the instruction set is generated in a manner that provisions the instruction set to execute at the desired workspace execution location. More details regarding creating a workspace are provided below with reference to FIGS.

[0044] 2 is an illustration of provisioning a workspace, according to various embodiments. As shown in diagram 200 of FIG. 2, a data model 205 (e.g., an overall model) is deployed to a production environment 201. The data model 205 may be built based on data obtained from an external data source 210. A workspace creation unit 120 receives a request from a client device (e.g., client device 105 of FIG. 1) to create a workspace for executing at least a portion of the data model 205 deployed to the production environment 201.

[0045] According to some embodiments, the portion of the data model 205 desired to be executed in the workspace may correspond to a particular sub-model of the data model (and the associated data objects and metadata used to build the sub-model), with the goal of further processing and improving the sub-model, e.g., retraining the sub-model on new training data or fine-tuning one or more parameters of the sub-model to improve performance. Additionally, the portion of the data model 205 desired to be executed in the workspace may correspond to a slice of generative environment data, e.g., data objects, data records, etc., with the goal of building a new predictive model or evaluating scenarios for anomaly detection, examining data patterns, etc. The portion of the data model 205 desired to be executed in the workspace may be included in a request issued by the client device to the workspace creation unit 120.

[0046] It is understood that data objects, metadata, or data templates (e.g., report templates, analysis templates, product hierarchy templates, organizational hierarchy templates, etc.) may be extracted from the generative environment 201 to be included in the workspace for further processing. In other words, the workspace creation unit 120 creates copies of the data objects, metadata, etc. included in the generative environment 201 and instantiates (i.e., physically deploys) these copies in the workspace. The instantiated data objects, metadata, etc. may be modified based on processing performed in the workspace, such as testing a sub-model, updating a sub-model, etc. According to some embodiments, once processing performed in the workspace is completed, the data objects, metadata, parts of the model, etc. that were modified in the workspace may be pushed back to the generative environment via the workspace creation unit 120. For example, in a scenario where a sub-model of a data model is instantiated in a workspace (e.g., for the purpose of fine-tuning or retraining the sub-model), the retrained / fine-tuned sub-model may be pushed back to the generative environment 201 by the workspace creation unit 120 (upon completion).

[0047] In this manner, an older version of a sub-model deployed to the generative environment 201 may be replaced with a retrained / fine-tuned sub-model from the workspace. Additionally, according to some embodiments, the data processing system of FIG. 1 may provide an option to maintain both the older version of a sub-model and the updated version of the sub-model within the generative environment 201. According to some embodiments, prior to creation of the workspace, an initial state of the generative environment 201 may be stored in a data store (associated with the generative environment 201), where the initial state includes the performance parameters (e.g., first output) of the data model 205 deployed to the generative environment 201. For example, upon replacing an older version of a sub-model of the data model with a retrained / fine-tuned version of the sub-model (obtained from the workspace), the data processing system may generate an updated state of the generative environment, where the updated state includes a new output (i.e., a second output different from the first output) of the model deployed to the generative environment. The updated performance of the data model deployed to the generative environment may be displayed on the client device via an application interface.

[0048] The request issued by the client device to the workspace creation unit 120 includes information regarding the execution location of the workspace to be created. Specifically, this execution location corresponds to the destination environment in which it is desired to create the workspace. According to some embodiments, the execution location corresponds to an area within an application instance within the production environment (i.e., an in-memory workspace implementation) or a remote application instance or a hybrid environment. A hybrid environment is defined herein as an environment hosting a workspace that is configured to extract data from other workspaces, external data sources 210, etc. (in addition to data extracted from the production environment 201).

[0049] Various client devices (e.g., from different departments of an organization) can issue workspace creation requests to workspace creation unit 120. Each request includes information about the execution location of the workspace and the portion of data model 205 to be instantiated in the workspace. Workspace creation unit 120 is configured to create a workspace for each request, e.g., workspace A 215, workspace B 220, and workspace C 225. In each workspace, relevant data objects, metadata, and / or portions of the data model are copied and stored locally for further processing. For example, as shown in FIG. 2, workspace A includes databases 216 and 217 that store copies of corresponding data objects and metadata requested by client devices, whereas workspace B includes databases 221 and 222 that store copies of corresponding requested data objects and metadata, and workspace C (deployed in a hybrid environment) includes databases 226 and 227 that store copies of corresponding data objects and metadata requested by client devices. As will be described in more detail with reference to FIG. 3, a filtering mechanism may be utilized by workspace creation unit 120 to retrieve relevant data to be instantiated in the workspace.

[0050] 2 , it is understood that workspace C is deployed in a hybrid environment because it is configured to extract data from external data source 210 while simultaneously extracting data from other workspaces (e.g., workspace B 220). According to some embodiments, each of the created workspaces 215, 220, and 225 may further create at least one sandbox application (241, 242) or analysis pipeline (243) via application programming interface 230 (e.g., REST API) to support the processing / evaluation (e.g., of submodels) executed in the workspace. For example, in the context of software development, a sandbox application is an isolated environment in which code related to a submodel can be evaluated and tested. Similarly, a particular workspace (e.g., workspace A 215) may create an analysis pipeline 243 that can be used to emulate / test various parts (e.g., processes) included in the submodel being processed in workspace A 215.

[0051]

[0031] Figure 3 illustrates details of provisioning a workspace, according to various embodiments. Specifically, the schematic diagram 300 shown in Figure 3 illustrates components of a workspace creation unit responsible for creating a workspace within an execution location. As shown in Figure 3, the workspace creation unit (e.g., workspace creation unit 120 of Figure 1) includes a provisioning application 330 that retrieves data objects and associated metadata from a remote data store 310 and / or a production environment 320 and instantiates (i.e., physically materializes) the retrieved information in one or more workspaces, e.g., workspace A 341, workspace B 343, and workspace C 345.

[0052] According to some embodiments, the provisioning application 330 includes a data interface unit 333, an object interface unit 335, an instruction unit 336, a configuration database 337, and a materializer unit 338. The data interface unit 333 is configured to extract data (e.g., data objects) from the remote data store 310 and from the data store 322 included in the production environment 320. The object interface unit 335 is configured to extract metadata (e.g., metadata associated with data objects) from the metastore 323 included in the production environment 320. The extracted metadata, along with the extracted data objects, are instantiated by the provisioning application in workspaces, e.g., workspace A 341, workspace B 343, and workspace C 345. As shown in FIG. 3 , the provisioning application 330 of the workspace creation unit can materialize (i.e., physically deploy) various types of workspaces. For example, workspace A 341 and workspace B 343 are deployed to an execution location that is a remote application instance or a hybrid environment, while workspace C is deployed to an area within an application instance in production environment 320, i.e., an in-memory workspace implementation.

[0053] According to some embodiments, the data interface unit 333 and the object interface unit 335 may utilize a filtering mechanism to retrieve relevant data (and metadata) from the remote data store 310 and the production environment 320. Details regarding the operation of the data interface unit 333 are described herein with reference to FIG. 4. The data objects retrieved via the data interface unit 333 and the metadata retrieved via the object interface unit 335 are input to the instruction unit 337. The instruction unit 337 generates a set of instructions for executing one or more data objects and associated metadata in a workspace to be created within the execution location. Specifically, the set of instructions is generated by the instruction unit 337 in a format that can be converted into an equivalent instruction set that can be executed at the execution location hosting the workspace. Thus, the instruction unit 337 generates the instruction set as an executable script (e.g., an artifact) that can be executed in various workspace environments. In other words, the instruction set generated by the instruction unit 337 is target-independent (i.e., workspace-independent) and enables the data objects and metadata to be materialized in various workspace environments through the execution of the executable script. For example, the workspace creation unit can extract data from various types of data stores, such as bit data stores, relational data stores, etc., and materialize the extracted data in any environment, such as a big data environment.

[0054] According to some embodiments, in a workspace creation request issued by a client device, information about the generated instruction set (and associated data objects and metadata), the execution location of the workspace, etc., is stored by the instruction unit 337 in the configuration database 336 as workspace configuration information. Storing such configuration information of various workspaces in the configuration database 336 enables the workspace creation unit 120 to seamlessly create workspaces in the future. For example, consider that configuration information for a particular workspace (e.g., workspace A 341) is stored in the configuration database 336. At some point, workspace A is deleted by a request issued by the client device. If the client device subsequently wants to recreate workspace A, the workspace creation unit can simply retrieve the configuration information from the configuration database 336 and materialize workspace A. In doing so, the workspace creation unit does not need to (again) extract all data objects and associated metadata from the remote data store 310 and / or the generation environment 320, thereby saving computational resources of the data processing system. Furthermore, the workspace creation unit can provide the client device (via an API) with a list of data objects and metadata previously associated with the workspace. The client device may utilize this list of data objects and metadata while instructing the workspace creation unit to search for further data objects (or metadata) while recreating a new instance of the workspace.

[0055] According to some embodiments, the materializer unit 339 is configured to process the extracted data objects and metadata to generate a provisioning executable (e.g., a script) that can (when executed) physically create a workspace within the execution location. Once the workspace is created, the workspace creation unit may instantiate the extracted data objects, metadata, and / or portions of the data model (i.e., sub-models) in the workspace for further processing.

[0056] Figure 4 illustrates a data interface unit, according to various embodiments. Specifically, Figure 4 illustrates a schematic diagram 400 illustrating components of a data interface unit 333 included within the provisioning application 330 of Figure 3. The data interface unit 333 includes a request processing unit 410, a filtering unit 411, a data retrieval unit 413, an impact analysis unit 415, a data processing unit 417, an entitlement 419, and a neutral language translator 421.

[0057] According to some embodiments, a request issued by a client device for creation of a workspace (e.g., a user request issued via client device 105 of FIG. 1 ) is directed to a workspace creation unit. Specifically, the request is directed to a request processing unit 410 of data interface unit 333. Request processing unit 410 is configured to process requests, such as to extract information about the execution location of the workspace or to obtain information about parts of models (e.g., data models 205 deployed in production environment 201) that are executed in the workspace. Upon processing the workspace creation request, request processing unit 410 triggers data retrieval unit 413.

[0058] The data retrieval unit 413 communicates with a driver 403 included in the production environment 410. The driver 403 performs lookup operations on a data store 401 (of the production environment 410) to retrieve data objects, functions, etc. Information extracted from the data store 401 is passed to the data retrieval unit 413 of the data interface unit 333. According to some embodiments, if a user request relates to the creation of a sub-model in a workspace, the data retrieval unit 413 sends an identifier associated with this sub-model to the driver 403. The driver 403 uses this identifier to perform a lookup operation on the data store 401, which maintains a mapping of models deployed in the production environment and associated data objects. In this way, the driver can extract the relevant data objects associated with the sub-model and send the extracted data objects to the data retrieval unit 413.

[0059] According to some embodiments, the data retrieval unit 413 communicates with the impact analysis unit 415 to perform analysis of the retrieved data objects. For example, the impact analysis unit 415 checks the dependencies of the extracted data objects, i.e., determines one or more additional data objects that depend on the retrieved data objects, and notifies the data retrieval unit 413 to retrieve these additional data objects. According to some embodiments, the data interface unit 333 utilizes the filtering unit 411 to perform filtering operations on the retrieved data objects to obtain a set of data objects that are relevant for the request. For example, a request to create a workspace may include information about a user's preferences for data objects. Such user preferences can be utilized as a filtering mechanism to obtain a set of the most relevant data objects to be materialized in the workspace.

[0060] Further, the searched and / or filtered data objects are passed to a data processing unit 417, which utilizes entitlements 419 (i.e., a set of one or more access rights) to determine whether a user (e.g., a user who initiated the workspace creation request) is authorized to access each of the searched and / or filtered data objects. In response to the data processing unit 417 determining that the user is not authorized to access one or more of the searched data objects, the data processing unit 417 triggers the request processing unit 410 to send an electronic message to the client device notifying the user of such determination. Further, the electronic message may include a notification to the user that the workspace creation request cannot be completed due to an access issue. However, in response to successfully determining that the user is authorized to access each of the one or more data objects, the data processing unit 417 launches a neutral language translator 421.

[0061] According to some embodiments, the neutral language translator 421 is configured to convert the format of the data objects, metadata, etc. into a format acceptable to the environment of the workspace execution location. In doing so, the data interface unit 333 ensures that the extracted data object or objects are available for further processing of the workspace. According to some embodiments, when the data processing unit 417 successfully determines that the user is authorized to access each of the one or more data objects, it triggers the materializer (unit 338 in FIG. 3 ) to generate a provisioning executable that creates the workspace within the execution location. Furthermore, the data processing unit 417 may trigger the request processing unit 410 to send an electronic message to the client device indicating the user's successful authorization to the one or more data objects. The one or more data objects processed by the neutral language translator 421 are passed to the instruction unit (i.e., unit 336 in FIG. 3 ) to initiate the process of creating a workspace within the execution location.

[0062] FIG. 5 illustrates a flowchart 500 depicting a workspace provisioning process performed by a workspace creation unit of a data processing system, according to various embodiments. The process illustrated in FIG. 5 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or combinations thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method illustrated in FIG. 5 and described below is intended to be exemplary and non-limiting. While FIG. 5 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, these steps may be performed in some different order, or some steps may be performed in parallel.

[0063] The process begins at step 510, where a workspace creation unit receives a request (e.g., from a client device) to create a workspace for executing at least a portion of a model deployed in the generative environment. The portion of the model (e.g., data model 205 of FIG. 5 ) desired to be executed in the workspace may correspond to a particular sub-model of the data model (and the associated data objects and metadata used to build this sub-model) with the aim of further processing and improving the sub-model, e.g., retraining the sub-model on new training data or fine-tuning one or more parameters of the sub-model to improve performance. According to some embodiments, the portion of the model desired to be executed in the workspace may correspond to a slice of generative environment data, e.g., data objects, data records, etc., with the aim of building a new predictive model, or evaluating scenarios for anomaly detection, examining data patterns, etc.

[0064] The process then proceeds to step 520, where the workspace creation unit retrieves data objects and associated metadata related to portions of the model from the production environment (e.g., data stores and metastores included in the production environment). The workspace creation unit retrieves the related data objects and associated metadata by performing lookup operations on the data stores and metastores, which maintain mappings of models deployed in the production environment and the associated data objects and metadata utilized by these models. In step 530, the workspace creation unit retrieves the execution location, i.e., the destination where the workspace will be created. The workspace creation unit extracts the execution location information embedded in the workspace creation request issued by the client device.

[0065] The process then proceeds to step 540, where the workspace creation unit generates a set of instructions for executing one or more data objects and associated metadata in the workspace. It is understood that the set of instructions is generated in a format that can be converted into an equivalent instruction set that can be executed at an execution location that hosts the workspace. In step 550, the workspace creation unit generates a provisioning executable (e.g., a script) that, when executed (step 560), can physically create the workspace at the execution location. It is understood that, according to some embodiments, the workspace creation unit generates the provisioning executable in response to determining that the user (associated with the client device) that issued the request to create the workspace is authorized to access the relevant data objects and associated metadata obtained in step 520.

[0066] In step 570, the workspace creation unit executes the set of instructions in the created workspace. Specifically, the workspace creation unit instantiates the extracted data objects, metadata, and / or portions of the data model (i.e., sub-models) in the workspace for further processing. Once processing of the sub-models in the workspace is complete, the process in step 580 updates the production environment. For example, an older version of the sub-model deployed in the production environment 201 may be replaced with the processed sub-model, i.e., the retrained / fine-tuned sub-model, from the workspace.

[0067] Further, in some embodiments, the data processing system may provide the option to maintain both the old version of the sub-model and the updated version of the sub-model in the generative environment. As noted above, an initial state of the generative environment may be stored in data storage associated with the generation prior to creating the workspace, this initial state including the performance parameters (e.g., first output) of the data model deployed to the generative environment 201. For example, upon replacing the old version of the sub-model of the data model with a retrained / fine-tuned version of the sub-model (obtained from the workspace), the data processing system may generate an updated state of the generative environment, this updated state including a new output (i.e., a second output different from the first output) of the model deployed to the generative environment.

[0068] Exemplary System 6 is a simplified diagram of a distributed system 600. In the illustrated example, the distributed system 600 includes one or more client computing devices 602, 604, 606, and 608, which are coupled to a server 612 via one or more communication networks 610. The client computing devices 602, 604, 606, and 608 may be configured to execute one or more applications.

[0069] In various examples, server 612 may be adapted to execute one or more services or software applications that enable one or more embodiments described in this disclosure. In certain examples, server 612 may also provide other services or software applications, which may include non-virtualized and virtualized environments. In some examples, these services may be provided to users of client computing devices 602, 604, 606, and / or 608 as web-based or cloud services, such as under a Software as a Service (SaaS) model. Users operating client computing devices 602, 604, 606, and / or 608 may then utilize one or more client applications to interact with server 612 to utilize the services provided by these components.

[0070] 6, server 612 may include one or more components 618, 620, and 622 that implement the functions performed by server 612. These components may include software components, hardware components, or a combination thereof that may be executed by one or more processors. It should be understood that a variety of different system configurations are possible that may differ from distributed system 600. The example shown in FIG. 6 is therefore one example of a distributed system for implementing an exemplary system and is not intended to be limiting.

[0071] A user may use client computing devices 602, 604, 606, and / or 608 to execute one or more applications, models, or chatbots, which may generate one or more events or models that may be executed or provided in accordance with the teachings of this disclosure. The client devices may provide an interface that allows a user of the client device to interact with the client device. The client devices may also output information to the user through this interface. While FIG. 6 shows only four client computing devices, any number of client computing devices may be supported.

[0072] Client devices may include various types of computing systems, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Linux® or Linux-like operating systems (such as Google Chrome® OS)), including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android®, BlackBerry®, Palm OS®). Portable handheld devices may include mobile phones, smartphones (e.g., iPhone®), tablets (e.g., iPad®), personal digital assistants (PDAs), etc. Wearable devices may include Google Glass® head-mounted displays and other devices.The gaming systems may include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices, Sony PlayStation® systems, various gaming systems offered by Nintendo®, etc.), etc. The client devices may be capable of running a variety of different applications, such as various Internet-related apps, communication applications (e.g., email applications, Short Message Service (SMS) applications), etc., and may use a variety of communication protocols.

[0073] Network 610 may be any type of network familiar to those skilled in the art that is capable of supporting data communications using any of a variety of available protocols, including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), AppleTalk®, etc. By way of example only, network 610 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., a network operating under the IEEE (Institute of Electrical and Electronics) 1002.11 suite of protocols, Bluetooth® and / or other wireless protocols), and / or any combination of these and / or other networks.

[0074] The servers 612 may be comprised of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or other suitable configurations and / or combinations. The servers 612 may include one or more virtual machines running a virtual operating system or other computing architectures that involve virtualization, such as one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices for the servers. In various examples, the servers 612 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0075] The computing systems in server 612 may run one or more operating systems, including any of those described above, and any commercially available server operating system. Server 612 may also run any of a variety of other server and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, JAVA servers, database servers, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle®, Microsoft®, Sybase®, IBM® (International Business Machines), etc.

[0076] In some implementations, server 612 may include one or more applications for analyzing and consolidating data feeds and / or event updates received from users of client computing devices 602, 604, 606, and 608. By way of example, the data feeds and / or event updates may include, but are not limited to, Twitter® feeds, Facebook® updates, or real-time updates received from one or more third-party sources and continuous data streams, which may include real-time events related to sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc. Server 612 may also include one or more applications for displaying the data feeds and / or real-time events via one or more display devices of client computing devices 602, 604, 606, and 608.

[0077] The distributed system 600 may also include one or more data repositories 614, 616. In certain examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 614, 616 may be used to store information, such as information related to chatbot performance or generated models used by the chatbot that are used by the server 612 in performing various functions according to various embodiments. The data repositories 614, 616 may be in a variety of locations. For example, the data repository used by the server 612 may be local to the server 612 or may be remote from the server 612 and in communication with the server 612 via a network-based or dedicated connection. The data repositories 614, 616 may be of various types. In certain examples, the data repository used by the server 612 may be a database, e.g., a relational database, such as databases provided by Oracle Corporation and other vendors. One or more of these databases may be adapted to allow data to be stored, updated, and retrieved from the database in response to SQL-formatted commands.

[0078] In particular examples, one or more of the data repositories 614, 616 may also be used by an application to store application data. The data repositories used by an application may be of various types, such as, for example, a key-value store repository, an object store repository, or a general storage repository supported by a file system.

[0079] In certain examples, the functionality described in this disclosure may be provided as services via a cloud environment. Figure 7 is a simplified block diagram of a cloud-based system environment in which various services may be provided as cloud services, according to certain examples. In the example shown in Figure 7, cloud infrastructure system 702 may provide one or more cloud services that may be requested by users using one or more client computing devices 704, 706, and 708. Cloud infrastructure system 702 may comprise one or more computers and / or servers, which may include those described above for server 612. The computers in cloud infrastructure system 702 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or other suitable configurations and / or combinations.

[0080] Network 710 may facilitate the communication and exchange of data between clients 704, 706, and 708 and cloud infrastructure system 702. Network 710 may include one or more networks. These networks may be of the same type or different types. Network 710 may support one or more communication protocols, including wired and / or wireless protocols, to facilitate communication.

[0081] The example shown in Figure 7 is merely one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in some other examples, cloud infrastructure system 702 may have more or fewer components than those shown in Figure 7, may combine two or more components, or may have a different configuration or arrangement of components. For example, while Figure 7 shows three client computing devices, any number of client computing devices may be supported in alternative examples.

[0082] The term cloud service is generally used to refer to services made available to users on demand via a communication network such as the Internet by a service provider's system (e.g., cloud infrastructure system 702). Generally, in a public cloud environment, the servers and systems that make up the cloud service provider's system are distinct from a customer's own on-premise servers and systems. The cloud service provider's system is managed by the cloud service provider. Therefore, customers can use these services without having to purchase separate licenses, support, or hardware and software resources for the cloud services offered by the cloud service provider. For example, the cloud service provider's system may host an application, and users can order and use this application on demand via the Internet without having to purchase infrastructure resources to run the application. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, several cloud services, such as middleware services, database services, and Java cloud services, are offered by Oracle Corporation of Redwood Shores, California.

[0083] In particular examples, cloud infrastructure system 702 may provide one or more cloud services using various models, such as under a Software as a Service (SaaS) model, a Platform as a Service (PaaS) model, an Infrastructure as a Service (IaaS) model, etc., including a hybrid service model. Cloud infrastructure system 702 may include a set of applications, middleware, databases, and other resources that enable the delivery of various cloud services.

[0084] The SaaS model allows applications or software to be delivered to customers as a service over a communications network such as the Internet, without the customer having to purchase hardware or software for the underlying application. For example, the SaaS model may be used to provide customers with access to on-demand applications hosted by cloud infrastructure system 702. Examples of SaaS services offered by Oracle include, but are not limited to, various services for human capital / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, etc.

[0085] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage, hardware and networking resources) as cloud services to customers to provide elastic compute and storage capabilities. Various IaaS services are offered by Oracle Corporation.

[0086] The PaaS model is commonly used to provide platform and environment resources as a service, enabling customers to develop, run, and manage applications and services without the need for customers to procure, build, or maintain such resources. Examples of PaaS services offered by Oracle include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), data management cloud services, and various application development solution services.

[0087] Cloud services are generally provided in an on-demand self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 702 via a subscription order. Cloud infrastructure system 702 then executes processing to provide the services requested in the customer's subscription order. For example, a user may use utterances to request the cloud infrastructure system to take a particular action (e.g., intent), as described above, and / or to provide a service to a chatbot system, as described herein. Cloud infrastructure system 702 may be configured to provide one or more cloud services.

[0088] Cloud infrastructure system 702 may provide cloud services through a variety of deployment models. In a public cloud model, cloud infrastructure system 702 may be owned by a third-party cloud service provider, and cloud services are offered to any general public customer, which may be an individual or a business. In certain other examples, under a private cloud model, cloud infrastructure system 702 may be operated within an organization (e.g., within a corporate organization), and services are offered to customers within the organization. For example, customers may be various departments of a company, such as a human resources department, a payroll department, or individuals within the company. In certain other examples, under a community cloud model, cloud infrastructure system 702 and the services offered may be shared by several organizations within an associated community. Various other models, such as hybrids of the above models, may also be used.

[0089] Client computing devices 704, 706, and 708 may be of various types (such as client computing devices 602, 604, 606, and 608 shown in FIG. 6 ) and may be capable of running one or more client applications. Users may use the client devices to interact with cloud infrastructure system 702, such as to request services provided by cloud infrastructure system 702. For example, users may use the client devices to request information or actions from a chatbot as described in this disclosure.

[0090] In some examples, the processing performed by cloud infrastructure system 702 to provide services may include model training and deployment. This analysis may include using, analyzing, and manipulating datasets to train and deploy one or more models. This analysis may be performed by one or more processors, possibly processing the data in parallel or performing simulations using the data. For example, big data analysis may be performed by cloud infrastructure system 702 to generate and train one or more models for a chatbot system. The data used in this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blobs (binary large objects)).

[0091] 7, cloud infrastructure system 702 may include infrastructure resources 730 utilized to facilitate the provision of various cloud services offered by cloud infrastructure system 702. Infrastructure resources 730 may include, for example, processing resources, storage or memory resources, networking resources, etc. In particular examples, storage virtual machines available to provide storage requested by applications may be part of cloud infrastructure system 702. In other examples, the storage virtual machines may be part of a different system.

[0092] In particular examples, to facilitate efficient provisioning of these resources to support various cloud services offered by cloud infrastructure system 702 to various customers, resources may be bundled into sets of resources or resource modules (also referred to as “pods”). Each resource module or pod may comprise a pre-integrated and optimized combination of one or more types of resources. In particular examples, different pods may be pre-provisioned for different types of cloud services. For example, a first set of pods may be provisioned for database services, a second set of pods, which may include a different combination of resources than the pods in the first set of pods, may be provisioned for Java services, etc. In some services, resources allocated for provisioning of a service may be shared between services.

[0093] Cloud infrastructure system 702 itself may internally use services 732 that are shared by various components of cloud infrastructure system 702, which facilitates the provisioning of services by cloud infrastructure system 702. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services to enable cloud support, email services, notification services, file transfer services, etc.

[0094] Cloud infrastructure system 702 may comprise multiple subsystems. These subsystems may be implemented in software, hardware, or a combination thereof. As shown in FIG. 7 , these subsystems may include a user interface subsystem 712 that allows users or customers of cloud infrastructure system 702 to interact with cloud infrastructure system 702. User interface subsystem 712 may include a variety of different interfaces, such as a web interface 714, an online store interface 716 (through which cloud services offered by cloud infrastructure system 702 are advertised and available for purchase by consumers), and other interfaces 718. For example, a customer may use a client device to request one or more services offered by cloud infrastructure system 702 (service request 734) using one or more of interfaces 714, 716, and 718. For example, a customer may access an online store to browse cloud services offered by cloud infrastructure system 702 and place a subscription order for one or more services offered by cloud infrastructure system 702 for which the customer wishes to subscribe. This service request may include information identifying the customer and the one or more services for which the customer wishes to subscribe. For example, a customer may place a subscription order for services provided by cloud infrastructure system 702. As part of the order, the customer may provide information identifying the chatbot system for which the services are to be provided, and optionally, one or more credentials for that chatbot system.

[0095] 7, cloud infrastructure system 702 may include an Order Management Subsystem (OMS) 720 configured to process new orders. As part of this processing, OMS 720 may be configured to create a customer account if not already done, receive billing and / or billing information from the customer that is used to issue a bill to the customer for providing the customer with the requested services, verify the customer information, and once verified, finalize the customer's order, and orchestrate various workflows to prepare the order for provisioning.

[0096] Upon proper authentication, the OMS 720 may then invoke the Order Provisioning Subsystem (OPS) 724. The OPS 724 is configured to provision resources for the order, including processing, memory, and networking resources. Provisioning may include allocating resources for the order and configuring these resources to facilitate the service requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service ordered by the customer. For example, according to one workflow, the OPS 724 may be configured to determine that a specific cloud service is requested and identify the number of pods that would have been pre-configured for that specific cloud service. The number of pods allocated to the order may depend on the size / amount / level / scope of the requested service. For example, the number of pods allocated may be determined based on the number of users supported by the service, the duration for which the service is requested, etc. The allocated pods may then be customized to the particular requesting customer to provide the requested service.

[0097] In certain examples, the above setup phase processing may be performed by cloud infrastructure system 702 as part of a provisioning process. Cloud infrastructure system 702 may generate an application ID and select a storage virtual machine for the application from among storage virtual machines provided by cloud infrastructure system 702 itself or from storage virtual machines provided by other systems other than cloud infrastructure system 702.

[0098] Cloud infrastructure system 702 may send a response or notification 744 to the requesting customer indicating when the requested service will be ready for use. In some cases, information (e.g., a link) may be sent to the customer that enables the customer to begin using and utilizing the benefits of the requested service. In a particular example, when a customer requests a service, the response may include a chatbot system ID generated by cloud infrastructure system 702 and information identifying the chatbot system selected by cloud infrastructure system 702 that corresponds to the chatbot system ID.

[0099] Cloud infrastructure system 702 may provide services to multiple customers. For each customer, cloud infrastructure system 702 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to these orders, and providing the requested services to the customer. Cloud infrastructure system 702 may also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics may be collected on the amount of storage used, the amount of data transferred, the number of users, and system up and down time. This usage information may be used to bill the customer. Billing may occur, for example, on a monthly cycle.

[0100] Cloud infrastructure system 702 may provide services to multiple customers concurrently. Cloud infrastructure system 702 may store information of these customers, including potentially proprietary information. In particular examples, cloud infrastructure system 702 includes an Identity Management Subsystem (IMS) 728 configured to manage customer information and segregate the managed information so that information associated with one customer is not accessible by another customer. IMS 728 may be configured to provide various security-related services, such as identity services (information access management, authentication and authorization services, services for managing customer identities and roles and related functions, etc.).

[0101] 8 is a diagram illustrating an example of a computer system 800. In some examples, the computer system 800 may be used to implement a digital assistant or chatbot system in a distributed environment, as well as any of the various servers and computer systems described above. As shown in FIG. 8, the computer system 800 includes various subsystems, including a processing subsystem 804, which communicates with several other subsystems via a bus subsystem 802. These other subsystems may include a processing acceleration unit 806, an I / O subsystem 808, a storage subsystem 818, and a communication subsystem 824. The storage subsystem 818 may include a non-transitory computer-readable storage medium, including a storage medium 822 and a system memory 810.

[0102] Bus subsystem 802 provides a mechanism for allowing the various components and subsystems of computer system 800 to communicate with each other as intended. While bus subsystem 802 is shown schematically as a single bus, alternative examples of the bus subsystem may utilize multiple buses. Bus subsystem 802 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus, etc., using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0103] The processing subsystem 804 controls the operation of the computer system 800 and may comprise one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). These processors may include single-core or multi-core processors. The processing resources of the computer system 800 may be organized into one or more processing units 832, 834, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 804 may include one or more special-purpose coprocessors, such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem may be implemented using customized circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0104] In some examples, processing units within processing subsystem 804 can execute instructions stored in system memory 810 or on computer-readable storage medium 822. In various examples, the processing units can execute various program or code instructions and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code being executed can be in system memory 810 and / or on computer-readable storage medium 822 (including, in some cases, on one or more storage devices). Through suitable programming, processing subsystem 804 can provide the various functions described above. In instances where computer system 800 is running one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0105] In certain examples, the processing acceleration unit 806 may be optionally provided to perform customized processing to accelerate the overall processing performed by the computer system 800 or to offload portions of the processing performed by the processing subsystem 804.

[0106] I / O subsystem 808 may include devices and mechanisms for inputting information into computer system 800 and / or outputting information from or through computer system 800. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into computer system 800. User interface input devices may include, for example, keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion-sensing and / or gesture-recognition devices such as Microsoft Kinect motion sensors that allow a user to control and interact with the input device, a Microsoft Xbox 360 game controller, and devices that provide an interface for receiving input using gestures and verbal commands. The user interface input devices may also include eye gesture recognition devices, such as a Google Glass blink detector, that detects eye movements from the user (e.g., "blinks" while taking a picture and / or making a menu selection) and translates the eye gestures as input to the input device (e.g., Google Glass). Additionally, the user interface input devices may include a voice recognition sensing device that allows the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0107] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, as well as audio / visual devices (such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, and eye-tracking devices). Additionally, user interface input devices may include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0108] In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 800 to a user or to another computer. User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be flat-panel devices such as those using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touch screens, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / visual information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, audio output devices, and modems.

[0109] Storage subsystem 818 provides a repository or data store for storing information and data used by computer system 800. Storage subsystem 818 provides a tangible, non-transitory, computer-readable storage medium for storing the basic programming and data structures that provide some example functionality. Storage subsystem 818 may store software (e.g., programs, code modules, instructions) that, when executed by processing subsystem 804, provide the functionality described above. This software may be executed by one or more processing units of processing subsystem 804. Storage subsystem 818 may also provide authentication according to the teachings of the present disclosure.

[0110] The storage subsystem 818 may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. As shown in FIG. 8, the storage subsystem 818 includes a system memory 810 and a computer-readable storage medium 822. The system memory 810 may include several memories, including volatile main random access memory (RAM) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 800, such as during start-up, may typically be stored in ROM. The RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 804. In some implementations, system memory 810 may include multiple different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), and the like.

[0111] 8, system memory 810 may load running application programs 812 (which may include various applications such as a web browser, a middle-tier application, a relational database management system (RDBMS), etc.), program data 814, and operating system 816. By way of example, operating system 816 may include various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems, various commercially available UNIX or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome OS, etc.), and / or mobile operating systems such as iOS, Windows Phone, Android OS, BlackBerry OS, Palm OS, etc.

[0112] The computer-readable storage medium 822 may store programming and data structures that provide some example functionality. The computer-readable medium 822 may provide storage of computer-readable instructions, data structures, program modules, and other data for the computer system 800. Software (programs, code modules, instructions) that, when executed by the processing subsystem 804, provide the above-described functionality may be stored in the storage subsystem 818. By way of example, the computer-readable storage medium 822 may include non-volatile memory such as a hard disk drive, a magnetic disk drive, a CD-ROM, a DVD, an optical disk drive such as a Blu-Ray® disk, or other optical media. The computer-readable storage medium 822 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, etc. The computer-readable storage medium 822 may also include flash memory-based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM-based SSDs and flash memory-based SSDs.

[0113] In particular examples, storage subsystem 818 may also include a computer-readable storage medium reader 820 that may be further connected to a computer-readable storage medium 822. Reader 820 may be configured to receive and read data from a memory device such as a disk, flash drive, or the like.

[0114] In certain examples, computer system 800 may support virtualization techniques, including but not limited to virtualization of processing and memory resources. For example, computer system 800 may provide support for running one or more virtual machines. In certain examples, computer system 800 may execute a program such as a hypervisor that facilitates configuration and management of virtual machines. Each virtual machine may be assigned memory resources, computational resources (e.g., processors, cores), I / O resources, and networking resources. Each virtual machine generally operates independently from other virtual machines. A virtual machine generally runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines executed by computer system 800. Thus, in some cases, multiple operating systems may be executed simultaneously by computer system 800.

[0115] The communications subsystem 824 provides an interface to other computer systems and networks. The communications subsystem 824 serves as an interface for sending and receiving data between other systems and the computer system 800. For example, the communications subsystem 824 may enable the computer system 800 to establish a communications channel with one or more client devices over the Internet for sending and receiving information to and from the client devices. For example, if the computer system 800 is used to implement the bot system 120 shown in FIG. 1, the communications subsystem may be used to communicate with a chatbot system selected for the application.

[0116] The communications subsystem 824 may support both wired and / or wireless communications protocols. In particular examples, the communications subsystem 824 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular telephone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.XX family of standards), or other mobile communications technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, the communications subsystem 824 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0117] The communications subsystem 824 can send and receive data in a variety of formats. In some examples, the communications subsystem 824 may receive incoming communications in the form of structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., in addition to other formats. For example, the communications subsystem 824 may be configured to receive (or send) data feeds 826 in real time from users of social media networks and / or other communications services, such as web feeds, such as Twitter feeds, Facebook updates, Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0118] In particular examples, the communications subsystem 824 may be configured to receive data in the form of a continuous data stream, which may include an event stream 828 of real-time events and / or event updates 830 that may be continuous or infinite in nature without a clear end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0119] Communications subsystem 824 may also be configured to communicate data from computer system 800 to other computer systems or networks. This data may be communicated in a variety of different formats, such as structured and / or unstructured data feeds 826, event streams 828, event updates 830, etc., to one or more databases, which may communicate with one or more streaming data source computers coupled to computer system 800.

[0120] Computer system 800 can be of various types, including a handheld portable device (e.g., an iPhone mobile phone, an iPad computing tablet, a PDA), a wearable device (e.g., a Google Glass head-mounted display), a personal computer, a workstation, a mainframe, a kiosk, a server rack, or other data processing system. Because the nature of computers and networks is constantly changing, the description of computer system 800 shown in FIG. 8 is intended only as one illustrative example. Many other configurations are possible, having more or fewer components than the system shown in FIG. 8. It should be understood that other ways and / or methods for implementing various examples exist, based on the disclosure and teachings herein.

[0121] While specific examples have been described, various modifications, variations, alternative configurations, and equivalents are possible. The examples are not limited to operation in any particular data processing environment, but can freely operate in multiple data processing environments. Furthermore, while specific examples have been described using a particular sequence of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be limiting. While some flowcharts describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. Also, the order of operations may be rearranged. A process may have additional steps not included in the figures. Various features and aspects of the above examples may be used individually or together.

[0122] Furthermore, while particular examples have been described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. Particular examples may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be implemented on the same processor in any combination, or on different processors in any combination.

[0123] Although a device, system, component, or module is described as being configured to perform particular operations or functions, such configuration may be achieved, for example, by designing an electronic circuit to perform the operations, by programming a programmable electronic circuit (such as a microprocessor) to perform the operations, such as executing computer instructions or code that is programmed to execute code or instructions stored on a non-transitory memory medium, or a processor or core, or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different process pairs may use different techniques, and the same process pair may use different techniques at different times.

[0124] Specific details are provided in this disclosure to ensure a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the examples. This description provides only illustrative examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the above description of the examples will provide one skilled in the art with an enabling description for implementing various examples. Various changes may be made in the function and arrangement of elements.

[0125] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made therein without departing from the broader spirit and scope of the appended claims. Accordingly, although particular examples have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0126] While the foregoing specification describes aspects of the disclosure with reference to specific examples thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above disclosure may be used individually or together. Moreover, the examples may be utilized in many environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense.

[0127] In the above description, the methods have been described in a particular order for purposes of illustration. It should be understood that, in alternative examples, the methods may be performed in an order different from that described. It should also be understood that the methods described above may be performed by hardware components or embodied in a sequence of machine-executable instructions that can be used to cause a machine, such as a general-purpose or special-purpose processor or logic circuitry programmed with the instructions, to perform the methods. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy diskette, ROM, RAM, EPROM, EEPROM, a magnetic or optical card, flash memory, or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0128] Where a component is described as being configured to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or by any combination thereof.

[0129] Although illustrative examples of the present application have been described in detail herein, it is to be understood that the concepts of the present invention may be variously embodied and utilized in other forms, and that the appended claims are intended to be construed to include such modifications except insofar as limited by the prior art.

Claims

1. A method implemented by a computing device, comprising: receiving, by the computing device, a plurality of requests to create workspaces for executing at least a portion of a model deployed in a production environment; obtaining, by the computing device, one or more data objects and their associated metadata related to the portion of the model; The computing device obtains an execution location of the workspace corresponding to each received request; generating, by the computing device, a set of instructions for executing the one or more data objects and the associated metadata in each of the workspaces; and creating the workspace within the execution location, the creating comprising instantiating the portion of the model, the one or more data objects, and the associated metadata in the workspace, the method further comprising: processing the portion of the model using the one or more data objects and the associated metadata in each workspace according to the set of instructions; and wherein the computing device selects the portion of the model that was processed from one of the workspaces corresponding to each of the requests and updates the portion of the model deployed to the production environment with the selected portion of the model.

2. The updating step comprises: The method further comprises obtaining an initial state of the production environment before creating the workspace within the execution location, the initial state including a first output of the model deployed to the production environment, and the updating step further comprises: replacing the portion of the model deployed in the production environment with the selected portion of the model; and generating an updated state of the production environment based on the replacement, the updated state including a second output of the model deployed to the production environment.

3. A method implemented by a computing device, comprising: receiving, by the computing device, a request to create a workspace for executing at least a portion of a model deployed in a production environment; obtaining, by the computing device, one or more data objects and their associated metadata related to the portion of the model; the computing device obtaining an execution location of the workspace; generating, by the computing device, a set of instructions for executing the one or more data objects and the associated metadata in the workspace; and creating the workspace within the execution location, the creating comprising instantiating the portion of the model, the one or more data objects, and the associated metadata in the workspace, the method further comprising: processing the portion of the model in the workspace using the one or more data objects and the associated metadata in accordance with the set of instructions; updating the generated environment based on the processing; The step of updating the creation environment includes: The method further comprises obtaining an initial state of the production environment before creating the workspace within the execution location, the initial state including a first output of the model deployed to the production environment, and updating the production environment further comprising: replacing the portion of the model deployed in the production environment with the portion of the model processed in the workspace; generating an updated state of the production environment based on the replacement, the updated state including a second output of the model deployed to the production environment.

4. retrieving a plurality of data objects and a plurality of metadata from one or more databases associated with the production environment, the one or more databases maintaining a mapping of models and associated data objects deployed to the production environment; filtering the plurality of data objects and the plurality of metadata to obtain the one or more data objects and the associated metadata related to the portion of the model; and iteratively performing a set of lookup operations on the one or more databases associated with the production environment to retrieve at least one data object related to the one or more data objects.

5. The method of any one of claims 1 to 4, wherein the execution location is an area within an application instance in the production environment or a remote application instance.

6. receiving the request from a user to execute the portion of the model in the workspace; verifying whether the user is authorized to access the one or more data objects and their associated metadata related to the portion of the model; The method of any one of claims 1 to 5, further comprising the step of: in response to successful validation, generating a provisioning executable that creates the workspace within the execution location.

7. The workspace further comprises: ingesting data from a data source external to said computing device; 7. The method of claim 1, further comprising: creating at least one sandbox application or analysis pipeline based on the portion of the model; and wherein the portion of the model in the workspace is processed using the at least one sandbox application or analysis pipeline.

8. The method of any preceding claim, wherein the set of instructions is generated in a format that is convertible into an equivalent set of instructions that can be executed at the execution location that hosts the workspace.

9. 1. A computing device comprising: a processor; and a memory containing instructions that, when executed by the processor, cause the computing device to perform the method of any one of claims 1 to 8.

10. A program which, when executed by a processor, causes a computing device to perform the method of any one of claims 1 to 8.

Images