Robot Scalability Infrastructure

Abstract

The present disclosure relates to a robot scalability infrastructure. The present disclosure generally relates to techniques for extending or customizing a base skill (e.g., a chatbot). According to certain embodiments, a robot extension infrastructure is provided to facilitate customization and / or extension of a base skill, separately track different versions of the base skill and the extension, apply an extension to different versions of the base skill, or apply different versions of an extension to the same base skill. The extension to the base skill includes a JSON extension that indicates modifications to the metadata of the base skill. A base skill (e.g., a base skill downloaded from a skill store) can be extended or customized by applying a JSON extension that describes the changes to be made to the metadata of the base skill.

Description

[0001] Cross - Reference to Related Applications

[0002] This patent application claims the benefit and priority of U.S. Provisional Patent Application Ser. No. 62 / 839,585, filed on Apr. 26, 2019, entitled “Bot Extensibility Infrastructure”, which is assigned to the assignee of the present application and is hereby incorporated by reference in its entirety for all purposes.

[0003] Copyright Notice

[0004] A portion of the disclosure of this patent application contains material that is subject to copyright protection. The copyright owner does not object to the reproduction by anyone of the patent application or patent disclosure as it appears in the Patent and Trademark Office patent file or records, but reserves all copyright rights otherwise. Technical Field

[0005] This disclosure generally relates to chatbots, and more particularly, to an infrastructure for extending a base chatbot that may be developed by another party and available in a chatbot store where different chatbots and / or different versions of chatbots may be stored for download. Background Art

[0006] To obtain instant responses, many users around the world use instant messaging or chat platforms. Organizations often use these instant messaging or chat platforms to conduct real - time sessions with customers (or end - users). However, hiring service personnel to communicate with customers or end - users in real - time can be very expensive for an organization. Thus, chatbots (also referred to hereinafter as “bots” or “skills”) have been developed to communicate with end - users, especially via the Internet. An end - user or customer can communicate with a bot through a messaging application that the end - user has installed and uses. Intelligent bots (usually powered by artificial intelligence (AI)) can communicate more intelligently and contextually in a real - time session and thus can allow for a more natural conversation between the bot and the end - user to improve the conversation experience. Instead of the end - user having to learn a fixed set of keywords or commands that the bot knows how to respond to, an intelligent bot can understand the end - user's intent based on natural - language user utterances and respond accordingly.

[0007] However, chatbots are difficult to build because these automated solutions require specific knowledge in certain domains and the application of certain technologies that may only be within the capabilities of professional developers. As part of building such a chatbot, developers can first understand the needs of the enterprise and end users. The developers can then analyze and make decisions related to, for example: selecting the data set to be used for analysis; preparing the input data set for analysis (e.g., cleaning the data, extracting, formatting, and / or transforming the data, performing data feature engineering, etc. before analysis); identifying the appropriate machine learning (ML) technique(s) or ML model(s) to perform the analysis; and improving the technique or model to improve the results / effectiveness based on feedback. The task of identifying the appropriate model(s) can include developing multiple models (possibly in parallel), iteratively testing and experimenting with these models, and then identifying a specific model (or models) for use. Further, solutions based on supervised learning typically involve a training phase, followed by an application (i.e., inference) phase, and an iterative loop between the training phase and the application phase. The developers can be responsible for carefully implementing and monitoring these phases to achieve an optimal solution.

[0008] Accordingly, building an appropriate chatbot can be very complex and time-consuming, and developers may play a central role in building such a solution. However, the number of developers with the necessary skills to build such a chatbot is very limited. Many times, developers also have to be experts in the specific domain corresponding to the problem being solved, such as business analysts. The number of such developers who are also domain experts is extremely limited. Due to the lack of such expert developers, this results in many organizations or individuals being unable to develop AI- or ML-based chatbots. This makes AI- or ML-based chatbot solutions expensive and / or inaccessible to many individuals and organizations. SUMMARY OF THE INVENTION

[0009] The techniques disclosed herein generally relate to chatbots. More specifically and without limitation, the techniques disclosed herein relate to an infrastructure for extending a base chatbot, which can be developed by another party and is available in a chatbot store, where different chatbots and / or different versions of chatbots can be stored for download. Various inventive embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0010] According to some embodiments, a computer-implemented method may include: obtaining metadata associated with a base application (e.g., a base robot); modifying the metadata associated with the base application to generate an extended application (e.g., an extended robot); determining a difference between the extended application and the base application; and storing the difference between the extended application and the base application as a JavaScript Object Notation (JSON) file in a database. The JSON file in the database may be referred to as a robot extension (or skill extension) and may be downloaded to implement the extended application by modifying the base application according to the JSON file.

[0011] In some embodiments, a computer-implemented method may include: obtaining a base application that includes implementation data and a first JSON file that includes metadata associated with the base application; and obtaining a second JSON file associated with an extended application of the base application. The second JSON file describes the changes to be made to the first JSON file and is referred to as a robot extension or skill extension. The computer-implemented method may further include making the changes described in the second JSON file to the first JSON file to generate metadata associated with the extended application. Then, the extended application may be implemented based on the implementation data of the base application and the metadata associated with the first JSON file modified based on the second JSON file.

[0012] According to some embodiments, a computer-implemented method may include: obtaining a first JSON file from a database that stores metadata for multiple applications as JavaScript Object Notation (JSON) files, the first JSON file including metadata for a first application; receiving a modification to the first application to generate an extended application; determining a difference between the metadata of the extended application and the metadata of the first application; and storing a second JSON file in the database, the second JSON file describing the changes to be made to the metadata of the first application to generate the metadata of the extended application.

[0013] In some embodiments, the plurality of applications can be chatbot applications. The JSON files of the plurality of applications can be stored in the database as compressed files. In some embodiments, the plurality of applications can include two or more versions of the first application. In some embodiments, the second JSON file can be compatible with two or more versions of the first application for extending the two or more versions in the first application. In some embodiments, the second JSON file can be incompatible with at least one of the two or more versions of the first application, and the database can include a third JSON file that is compatible with at least one of the two or more versions of the first application, wherein the second JSON file and the third JSON file can have different version numbers.

[0014] In some embodiments, the metadata of the first application can include metadata for at least one of the following: configuring the first application, training the first application, testing the first application, or executing the first application. In some embodiments, the metadata of the first application can include metadata for at least one of the intents, entities, utterances, custom components, or session flows of the first application. The second JSON file can include metadata for custom components in the extended application. In some embodiments, modifying the first application to generate the extended application can include modifying the first application via a Representational State Transfer Application Programming Interface (REST API).

[0015] According to certain embodiments, a computer-implemented method can include obtaining a first application from a database storing a plurality of applications, wherein the first application can include implementation data of the first application and a first JavaScript Object Notation (JSON) file including metadata associated with the first application. The computer-implemented method can further include obtaining from the database a second JSON file associated with an extended application of the first application, wherein the second JSON file describes changes to the first JSON file. The computer-implemented method can further include: applying the changes to the first JSON file described in the second JSON file to generate metadata associated with the extended application; and implementing the extended application based on the implementation data of the first application and the metadata associated with the extended application.

[0016] In some embodiments, the plurality of applications and the extended application may be chatbot applications. The metadata associated with the first application may include metadata for at least one of the following: configuring the first application, training the first application, testing the first application, or executing the first application. The metadata associated with the first application may include metadata for at least one of the intent, entities, utterances, custom components, or conversation flow of the first application. Applying the changes to the first JSON file described in the second JSON file may include: determining that the second JSON file conflicts with the first JSON file; and resolving one or more conflicts between the second JSON file and the first JSON file.

[0017] In some embodiments, the computer-implemented method may further include: obtaining a third JSON file from the database, the third JSON file describing changes to the metadata associated with the extended application; applying the changes to the metadata associated with the extended application described in the third JSON file to generate metadata associated with a second extended application; and implementing the second extended application based on the implementation data of the first application and the metadata associated with the second extended application. In some embodiments, the computer-implemented method may further include: obtaining a second application from the database, the second application may include implementation data of the second application and a third JSON file including metadata associated with the second application; applying the changes to the first JSON file described in the second JSON file to the third JSON file to generate metadata associated with a second extended application; and implementing the second extended application based on the implementation data of the second application and the metadata associated with the second extended application. In some embodiments, the database may include a first repository storing implementation data of the plurality of applications and a second repository storing the first JSON file and the second JSON file.

[0018] According to some embodiments, a non-transitory computer-readable storage medium may store computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations. The operations may include: obtaining a first JSON file from a database that stores metadata for a plurality of applications in JavaScript Object Notation (JSON) files, the first JSON file including metadata for a first application; receiving a modification to the first application to generate an extended application; determining a difference between the metadata of the extended application and the metadata of the first application; and storing a second JSON file in the database, the second JSON file describing changes to the metadata of the first application for generating the metadata of the extended application.

[0019] According to some embodiments, a computer system may include one or more processors and a non-transitory computer-readable storage medium storing computer-executable instructions. The instructions, when executed by the one or more processors, may cause the one or more processors to perform operations, the operations may include: obtaining a first application from a database storing a plurality of applications, the first application may include implementation data of the first application and a first JavaScript Object Notation (JSON) file including metadata associated with the first application. The operations may further include obtaining a second JSON file associated with an extended application of the first application from the database, wherein the second JSON file describes changes to the first JSON file. The operations may further include: applying the changes to the first JSON file described in the second JSON file to generate metadata associated with the extended application; and implementing the extended application based on the implementation data of the first application and the metadata associated with the extended application.

[0020] The techniques and infrastructure disclosed herein can also allow a client to obtain, on demand, a representation of the differences between an extended application and a corresponding base application. In some embodiments, metadata associated with an extended application and stored in a JSON file of a bot extension can include a definition of a test suite that can be used by the infrastructure to validate the extended application when needed. In some embodiments, separate version control can be used for the bot extension and the base application so that the base application and the bot extension can evolve separately. Metadata of the version of the bot extension stored in the JSON file can indicate the range of versions of the base application that can be compatible with the version of the bot extension. Thus, in some embodiments, a bot extension can be re-based to different versions of the same base application.

[0021] The terms and expressions that have been employed are used in a descriptive rather than a restrictive sense, and in using such terms and expressions, no intention is made to exclude any equivalents of the features shown and described or portions thereof. However, it should be recognized that various modifications are possible within the scope of the claimed systems and methods. Accordingly, it should be understood that although the inventive systems and methods have been specifically disclosed by way of example and optional features, those skilled in the art will recognize modifications and variations of the concepts disclosed herein, and such modifications and variations are considered to be within the scope of the inventive systems and methods as defined by the appended claims.

[0022] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the appropriate portions of the entire specification of this disclosure, any or all of the drawings, and each claim.

[0023] The foregoing and other features and examples will be described in more detail hereinafter in the following specification, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Exemplary examples are described in detail hereinafter with reference to the following drawings.

[0025] Figure 1 is a simplified block diagram of a distributed environment incorporating an exemplary embodiment.

[0026] Figure 2 depicts a distributed system according to certain embodiments, the distributed system implementing a bot system for communicating with end users using one or more messaging applications.

[0027] Figure 3Depicts an integrated system according to certain embodiments, the integrated system including a robotic system and a robotic analytics system for monitoring, analyzing, visualizing, and improving the performance of the robotic system.

[0028] Figure 4 Is a simplified flowchart illustrating an example of a process for developing a robot according to certain embodiments.

[0029] Figure 5 Depicts a system block diagram of an example of a robotic scalability infrastructure according to certain embodiments.

[0030] Figure 6 Illustrates an example of tracking the versions of a base robot and a robot extension and the compatibility between the versions of the base robot and the robot extension in a robotic scalability architecture according to certain embodiments.

[0031] Figure 7 Illustrates an example of creating a new skill extension to a published skill using a graphical user interface (GUI) according to certain embodiments.

[0032] Figure 8 Illustrates an example of creating a new skill extension to a published skill using a context menu in the GUI according to certain embodiments.

[0033] Figure 9 Illustrates an example of a dialog box in a GUI image for creating an extended skill according to certain embodiments.

[0034] Figure 10 Illustrates an example of a GUI image according to certain embodiments, showing certain inherited components of a base robot and new components of an extended robot.

[0035] Figure 11 Illustrates an example of a GUI image according to certain embodiments, showing form fields that can be added, edited, or removed for extending a skill.

[0036] Figure 12 Illustrates an example of a GUI image according to certain embodiments, showing a "Restore" icon for restoring the value of a field of a skill to its original value.

[0037] Figure 13 Illustrates an example of a GUI image according to certain embodiments, showing a dialog box for restoring the value of a field of a skill to its original value.

[0038] Figure 14 Illustrates an example of a GUI image according to certain embodiments, showing the comparison result between a base skill and an extended skill.

[0039] Figure 15 Illustrates an example of a GUI image according to certain embodiments, showing a dialog box for re-basing a skill extension to different versions of a base skill.

[0040] Figure 16 Illustrates an example of a GUI image according to certain embodiments, showing a dialog box for creating a new version of a published extended skill.

[0041] Figure 17 Illustrates an example of a GUI image according to certain embodiments, showing a dialog box for confirming the re-basing of a skill extension to a new version of a base skill.

[0042] Figure 18 Illustrates an example of a GUI image according to certain embodiments, presenting information about re-basing a skill extension to a new version of a base skill.

[0043] Figure 19 Illustrates an example of a GUI image according to certain embodiments, showing a list of changes from a previous base skill to a new base skill.

[0044] Figure 20 Illustrates an example of a GUI for comparing session flow content involved in the re-basing process according to certain embodiments.

[0045] Figure 21 Illustrates an example of a GUI image according to certain embodiments, showing a dialog box for confirming or canceling the re-basing of a skill extension.

[0046] Figure 22 Illustrates an example of a GUI image according to certain embodiments, showing an example of a custom component of a skill.

[0047] Figure 23 Illustrates an example of a GUI image according to certain embodiments for creating a new service in a skill.

[0048] Figure 24A Illustrates an example of a GUI image according to certain embodiments, including an icon indicating that the skill is an extended skill and that a new base skill update is available.

[0049] Figure 24B Illustrates an example of a GUI image according to certain embodiments, indicating an icon that the skill is an extended skill and has a pending re-basing.

[0050] Figure 25 Is a simplified flowchart illustrating an example of a process for generating an extended skill according to certain embodiments.

[0051] Figure 26It is a simplified flowchart illustrating an example of a process for implementing extended skills according to certain embodiments.

[0052] Figure 27 A simplified block diagram depicting an example of a distributed system for implementing some embodiments is shown.

[0053] Figure 28 It is a simplified block diagram of an example of a cloud-based system environment for implementing some embodiments.

[0054] Figure 29 An example of a computer system for implementing some embodiments is illustrated. Detailed Description

[0055] The present disclosure generally relates to chatbots. More specifically but not limited thereto, the techniques disclosed herein relate to infrastructure for extending a base chatbot, which may be developed by another party and available for download in a chatbot store where different chatbots and / or different versions of chatbots may be stored. Various inventive embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, etc.

[0056] Enterprises may wish to create chatbot systems for various purposes. A chatbot system may include one or more user intent classification engines for identifying an end-user's intent based on a user utterance and one or more dialogue engines for intelligently and contextually composing a message in response to the user utterance based on the determined end-user intent. However, building a chatbot system (including both a user intent classification engine that can determine an end-user's intent based on a user utterance and a dialogue engine for intelligently and contextually generating a response) is a challenging task, partly due to the nuances and uncertainties of natural language and the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (number of intents). The number of robot developers who are also domain experts is extremely limited. This makes AI- or ML-based chatbot solutions expensive and / or inaccessible to many individuals and organizations that want to use robots to communicate or otherwise interact with end-users or customers.

[0057] According to some embodiments, chatbots for various domains or applications can be built by developers with the necessary skills and made available to enterprises or individuals. The chatbots can be available in a chatbot store, which may allow developers and partners to publish chatbots and may allow customers to download chatbots to implement customized chatbots. The chatbot store may have certain features such as notifications, automatic updates, version control, etc.

[0058] According to some embodiments, a bot extensibility infrastructure can be provided such that a customer can customize and / or extend a chatbot downloaded from a chatbot store to tailor the chatbot to specific features, processes, terminology, culture, etc. that may be suitable for a particular application of the chatbot. The bot extensibility infrastructure can facilitate the extensibility of chatbots in an integral, general, flexible, secure, and maintainable manner. For example, in some embodiments, a base chatbot downloaded from a chatbot store can be extended or customized by applying extensions (e.g., described in a JavaScript Object Notation (JSON) file) that make changes to the base chatbot. The base chatbot and the extensions can be tracked separately. In some embodiments, extensions can be applied to different versions of the base chatbot. In some embodiments, extensions can be re-based to different versions of the base chatbot.

[0059] As used herein, a “chatbot,” “bot,” “skill,” or “skill bot” can refer to a computer program designed to simulate a conversation with a human end user, especially over the Internet. Separate skills can be designed to interact with the end user and complete specific types of tasks, such as ordering food, making a reservation, changing contact information, technical support, and customer service. Each skill can assist the end user in completing the task through a combination of visual, audio, or text messages and UI elements (such as buttons, tables, lists, etc.).

[0060] As used herein, a “base bot” or “base skill” can refer to a bot developed by a software as a service (SaaS) provider or developer with the necessary skills and made available to an enterprise or individual for implementing a custom bot. The base bot can be downloaded from a bot store (also referred to as a skill store) and can be customized and / or extended. An “extended bot” or “extended skill” can refer to a bot that has been customized and / or extended (e.g., augmented or enhanced) from a base bot to tailor the base bot to specific functions, processes, terminology, culture, etc. A “bot extension,” “skill extension,” or “extension” can refer to the code and / or data that can be used to extend a base bot into an extended bot.

[0061] As used herein, the term "intent" can refer to the category of actions or tasks that the end user expects the skill to perform for them. The term "entity" can refer to a variable that identifies information from the user input that enables the skill to complete the task. The term "component" can refer to the various functions that the skill can use to respond to the end user, such as outputting text, returning information from the backend, and executing custom logic. The term "dialog flow" can refer to the definition of the skill-user interaction and can describe how the skill responds and behaves based on the user input. The term "channel" can refer to the platform-specific configuration used to allow the skill to access the messaging platform or client messaging application. A single skill can be configured with multiple channels so that the skill can run simultaneously on different services or platforms that the end user may prefer to use.

[0062] As used herein, an "utterance" or "message" can refer to one or more sentences exchanged during a conversation, where a conversation can refer to a communication session that can include one or more utterances or messages. A conversation can include one or more phases or states. A conversation flow can be an abstraction of multiple conversations that include the same phases or states and the same transitions from phase (or state) to phase (or state). Each conversation can be a specific instance of the corresponding conversation flow. The state (or phase) of a conversation (or conversation flow) can be associated with the state of a state machine maintained by a robotic system for conversing with other robotic systems or people. In some cases, the state can correspond to the intent or goal of the end user. As used herein, an end user can refer to the end user of a robotic system, such as a person or another entity that converses with the robotic system via a messaging application or platform. For example, the end user can be a customer or client of an enterprise that owns the robotic system. As used herein, a user of a robotic system can refer to the owner, operator, administrator, or developer of the robotic system.

[0063] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of examples of the present disclosure. However, it will be apparent that the various examples may be practiced without these specific details. The following description merely provides examples and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of the examples will provide those skilled in the art with an enabling description for implementing the examples. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present disclosure as set forth in the appended claims. The drawings and description are not intended to be restrictive. Circuits, systems, networks, processes, and other components may be shown in block diagram form as components so as not to obscure the examples with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the examples. The teachings disclosed herein may also be applied to various types of applications such as mobile applications, non-mobile applications, desktop applications, web applications, enterprise applications, etc. Further, the teachings of the present disclosure are not limited to a particular operating environment (e.g., operating system, device, platform, etc.), but rather may be applied to a number of different operating environments.

[0064] In addition, it should be noted that individual examples may be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. Additionally, the order of the operations may be rearranged. When the operations of a process are completed, the process terminates, but there may be additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0065] The term "example" or "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or design described herein as "exemplary" or "example" need not be construed as more preferred or advantageous than other embodiments or designs.

[0066] The terms "machine-readable storage medium" or "computer-readable storage medium" include, but are not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. A machine-readable storage medium or computer-readable storage medium may include a non-transitory medium in which data can be stored and which does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memories, memories, or memory devices. A computer program product may include code and / or machine-executable instructions that may represent a process, a function, a subroutine, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted by any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0067] In addition, examples may be implemented by hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments for performing the necessary tasks (e.g., a computer program product) may be stored in a machine-readable medium. One or more processors may perform the necessary tasks. The systems depicted in some of the figures may be provided in various configurations. In some examples, the system may be configured as a distributed system in which one or more components of the system are distributed across one or more networks in a cloud computing system. In cases where a component is described as "configured to" perform certain operations, such configuration may be accomplished, for example, by designing electronic circuitry or other hardware to perform the operation, by programming or controlling an electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or any combination thereof.

[0068] I. Skills

[0069] Figure 1 is a simplified block diagram of a distributed environment 100 incorporating an exemplary embodiment. The distributed environment 100 includes a Digital Assistant Builder Platform (DABP) 102 that enables an enterprise to create and deploy digital assistants for its end users. The DABP 102 can be used to create one or more digital assistants (DAs). The DABP 102 can be used by multiple enterprises to create digital assistants for their end users. For example, as Figure 1As shown, user 104 representing a specific enterprise can use DABP 102 to create and deploy digital assistant 106 for the end users of the specific enterprise. For example, a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables the restaurant's customers to order food (e.g., order pizza).

[0070] For the purposes of the present disclosure, a "digital assistant" refers to an entity that helps the end user of the digital assistant complete various tasks through natural language conversations. A digital assistant can be implemented using only software (e.g., the digital assistant is a digital entity implemented using a program, code, or instructions executable by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be embodied or implemented in various physical systems or devices such as a computer, mobile phone, watch, appliance, vehicle, etc. A digital assistant is sometimes also referred to as a chatbot system.

[0071] For example, as Figure 1 shown, end user 108 can use digital assistant 106 to perform various tasks through a natural language-based conversation with digital assistant 106. As part of the conversation, the user can provide one or more user inputs 110 and receive a returned response 112 from digital assistant 106. Through these conversations, the user can request that the digital assistant perform one or more tasks, and in response, the digital assistant can perform the tasks requested by the user and respond to the user with an appropriate response.

[0072] User input 110 can be in the form of natural language and can be referred to as an utterance. The user utterance can be in text form (e.g., when the user types something as an input to digital assistant 106) or in audio input or speech form (e.g., when the user speaks something as an input to digital assistant 106). The utterance is typically in the language form spoken by end user 108. When the user input is in speech form, the user input can be converted into a text utterance in the specific language, and then the text utterance is processed by digital assistant 106. Various speech-to-text processing techniques can be used to convert the speech or audio input into a text utterance, which is then processed by digital assistant 106.

[0073] The text utterances generated by user input or by converting voice input into text form can be text fragments, sentences, multiple sentences, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) techniques to the text utterances to understand the meaning of the user input. As part of the NLU processing for the utterance, the digital assistant 106 is configured to perform processing for understanding the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intent.

[0074] For example, the user input can request to order a pizza, e.g., "I want to order a pizza". The digital assistant 106 is configured to understand the meaning of the utterance and take appropriate actions, which can involve responding to the end user with a question asking the user to input the type of pizza the end user desires to order, the size of the pizza, any toppings for the pizza, etc. The response provided by the digital assistant 106 can also be in natural language form, which can involve natural language generation (NLG) processing performed by the digital assistant 106. Once the digital assistant 106 receives the necessary information from the user, then the digital assistant 106 can place the pizza order. The digital assistant 106 can end the session with the end user by outputting information indicating that the pizza has been ordered.

[0075] In some embodiments, the utterances received by the digital assistant 106 as input undergo a series of processing steps or a pipeline of processing steps. These steps can include, for example, parsing the utterance grammatically, understanding the meaning of the utterance, refining and restructuring the utterance to develop a more understandable structure, determining the actions to be performed in response to the utterance, causing the actions to be executed, generating a response to be output to the end user in response to the generation of the user utterance, outputting the response to the end user, etc.

[0076] The NLU processing performed by a digital assistant, such as digital assistant 106, may include various NLP-related processes such as sentence semantic analysis (e.g., tokenizing, lemmatizing, identifying the part-of-speech tags of a sentence, identifying named entities in a sentence, generating a dependency tree to represent the sentence structure, splitting the sentence into clauses, analyzing individual clauses, resolving anaphoras, performing chunking, etc.). In some embodiments, the NLU processing or portions thereof may be performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 may use other resources to perform portions of the NLU processing. For example, the syntax and structure of a sentence can be identified by processing the sentence using a parser, a part-of-speech tagger, and / or a named entity recognizer. In one implementation, for the English language, a parser, a part-of-speech tagger, and a named entity recognizer provided by the Stanford Natural Language Processing (NLP) Group are used to analyze the sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0077] Although the various examples provided in this disclosure illustrate utterances in the English language, this is meant as an example only. In some embodiments, the digital assistant 106 is also capable of processing utterances in languages other than English. In some embodiments, the digital assistant 106 provides subsystems (e.g., components implementing NLU functionality) that are configured to process different languages. These subsystems can be implemented as pluggable units that can be invoked from the NLU core server using service calls. This makes the NLU processing flexible and extensible for each language, including allowing different processing sequences. A language pack can be provided for each corresponding language, where the language pack can register a list of subsystems that can provide services from the NLU core server and, if needed, also utilize the provided common subsystems.

[0078] A digital assistant, such as digital assistant 106, can be made available to its end users through various different channels, such as but not limited to through certain applications (also referred to as apps), through social media platforms, through various messaging services and applications, and other applications or channels. A single digital assistant can be configured with several channels for itself, such that a single digital assistant can run on different services and be accessed through different services simultaneously.

[0079] A digital assistant includes one or more skills or is associated with one or more skills. In some embodiments, these skills are separate chatbots designed to interact with end users and complete specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing widgets, etc. For example, forFigure 1 In the depicted embodiment, the digital assistant 106 includes skill bots 116-1, 116-2, 116-3, etc. As described above, in the present disclosure, the terms "skill" and "skills" are used synonymously with the terms "skill bot" and "skill bots", respectively.

[0080] Each skill bot associated with the digital assistant helps the end user of the digital assistant complete tasks through a session with the end user, where the session can include a combination of text or audio input provided by the end user and responses provided by the skill bot. These responses can take the form of text or audio messages to the end user and / or simple user interface elements (e.g., selection lists) presented to the end user for the end user to select.

[0081] There are various ways to add skills or skill bots to the digital assistant. In some instances, a skill bot can be developed by an enterprise and then added to the digital assistant using the DABP 102. In other instances, a skill bot can be developed and created using the DABP 102 and then added to a digital assistant created using the DABP 102. In yet other instances, the DABP 102 provides an online digital store (referred to as the "skill store") that offers multiple skills related to a wide range of tasks. The skills provided through the skill store can showcase various cloud services. Users of the DABP 102 can access the skill store through the DABP 102, select a desired skill, and add the selected skill to a digital assistant created using the DABP 102. Skills from the skill store can be added to the digital assistant as they are or in a modified form. For example, a user of the DABP 102 can select and clone a specific skill bot provided by the skill store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using the DABP 102.

[0082] In certain embodiments, the digital assistant created and deployed using the DABP 102 is implemented using the master bot / sub (or secondary) bot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master bot that interacts with one or more sub bots that are skill bots. For example, in Figure 1 the depicted embodiment, the digital assistant 106 includes a master bot 114 and skill bots 116-1, 116-2, 116-3, etc. that are sub bots of the master bot 114. In certain embodiments, the digital assistant 106 itself acts as the master bot.

[0083] A digital assistant implemented according to a master-slave robot architecture enables an end user of the digital assistant to interact with multiple skills via a unified user interface. When the end user interacts with the digital assistant, the user input is received by the master robot, which then processes the user input to identify the user request. Based on the processing, the master robot determines whether the user request can be handled by the master robot itself. If it is determined that the user request will not be handled by the host robot, the host robot selects an appropriate skill robot for handling the user request and routes the conversation to the selected skill robot. This enables the end user to talk to and use several skill robots configured to perform specific tasks via a common single interface. For example, for a digital assistant developed for an enterprise, the master robot of the digital assistant can interface with skill robots having specific functions, such as a customer relationship management (CRM) robot for performing functions related to customer relationship management, an enterprise resource planning (ERP) robot for performing functions related to enterprise resource planning, a human capital management (HCM) robot for performing functions related to human capital management, etc. In this way, the end user or customer of the digital assistant only needs to know how to access the digital assistant.

[0084] In a master robot / slave robot infrastructure, the master robot is configured to know a list of skill robots. The master robot can access metadata identifying various available skill robots, and for each skill robot, the capabilities of the skill robot include tasks that can be performed by the skill robot. After receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot from among the multiple available skill robots that can best serve or handle the user request. The master robot then routes the utterance (or a portion of the utterance) to the specific skill robot for further processing. Thus, control flows from the master robot to the skill robot. The master robot can support multiple input and output channels.

[0085] Although Figure 1 embodiments show that the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, and 116-3, this is not intended to be limiting. The digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems can be implemented only in software (e.g., code, instructions stored on a computer-readable medium and executable by one or more processors), only in hardware, or by a combination of software and hardware.

[0086] DABP 102 provides the infrastructure for users of DABP 102 to create digital assistants that include one or more skill bots associated with a digital assistant, as well as various services and features. For example, a skill bot can be created by cloning an existing skill bot, cloning an existing skill bot and then modifying the skill bot, or can be created from scratch using the tools and services provided by DABP 102. In some embodiments, DABP 102 provides a skill store or skill catalog that provides multiple skill bots for performing various tasks. Users of DABP 102 can clone skill bots from the skill store and create new skill bots.

[0087] DABP 102 also enables users (e.g., skill bot designers) to create skill bots from scratch. In some embodiments, at a high level, creating a skill bot involves the following operations:

[0088] (1) Configure settings for the new skill bot;

[0089] (2) Configure one or more intents for the skill bot;

[0090] (3) Configure entities for one or more intents;

[0091] (4) Train the skill bot;

[0092] (5) Create a dialogue flow for the skill bot;

[0093] (6) Add custom components to the skill bot; and

[0094] (7) Test and deploy the skill bot.

[0095] (1) Configure settings for the new skill bot - The skill bot designer can specify one or more invocation names for the skill bot being created. These invocation names can be used in utterances to explicitly identify and invoke the skill bot in the digital assistant. The skill bot designer can also specify example utterances for the skill bot. These example utterances represent the utterances of the skill bot. When a user input is received, the intent analysis engine of the digital assistant compares the user input with these example utterances to determine whether to invoke a specific skill bot.

[0096] (2) Configure one or more intents for the skill robot - The skill robot designer can configure one or more intents (also known as robot intents) for the skill robot being created. These intents identify the tasks that the skill robot can perform for the end user of the digital assistant. Each intent is given a name. For example, for a skill robot configured to help users perform various banking transactions, the intents can be specified by the skill robot designer for the skill robot, such as "Check balance", "Transfer funds", "Deposit inquiry", etc. For each intent, the skill robot designer specifies a set of example utterances that represent and explain the meaning of the intent and are typically associated with the task performed by the intent. For example, for the "Check balance" intent, the example utterances can include "What is the balance of my savings account?", "How much money is in my current deposit account?", "How much money is in my account?", etc. The arrangement of typical user requests and statements can also be specified as example utterances of the intent.

[0097] (3) Configure entities for one or more intents of the skill robot - In some instances, additional context may be required to enable the skill robot to correctly respond to user requests. For example, there may be cases where two or more user input utterances are parsed into the same intent in the skill robot. For example, in the above example, the utterances "What is the balance of my savings account?" and "How much money is in my current deposit account?" are both parsed into the same "Check balance" intent, but these utterances are different requests for different things. To clarify such requests, one or more entities are added to the intent. Using the banking skill example, an entity called "Account type" (which defines values such as "Current deposit" and "Savings") can enable the skill robot to parse the user request and respond appropriately. One or more entities can be specified for certain intents configured for the skill robot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and enable the skill robot to complete the user request. In certain embodiments, there are two types of entities: (a) built-in entities provided by the DABP 102; and (b) custom entities that can be specified by the skill robot designer. Built-in entities are general entities that can be used with various robots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for the banking skill, the "Account type" entity can be defined by the skill robot designer to enable various banking transactions by checking keywords entered by the user (such as "Current deposit", "Savings", "Credit card", etc.).

[0098] (4) Training the Skill Robot - The skill robot is configured to receive user input, parse, or otherwise process the received user input and identify or select an intent related to the received user input. To make this happen, the skill robot can be trained. In some embodiments, the skill robot is trained based on the intents configured for the skill robot and the example utterances associated with the intents (collectively referred to as training data), such that the skill robot can parse the user input into one of its configured intents. In some embodiments, the skill robot is represented by a model that is trained using the training data and that allows the skill robot to discern what the end user is saying (or, in some cases, trying to say). DABP 102 provides various different training techniques that can be used by skill robot designers to train the skill robot, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion of the training data (e.g., 80%) is used to train the skill robot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the skill robot can then be used to process and respond to user utterances. In some cases, the user's utterance can be a question that only requires a single answer and no additional conversation. To handle this, the skill robot can be configured with a Q&A (question and answer) intent. This enables the skill robot to output a response to the user's request without having to update the conversation definition. The Q&A intent is created in a manner similar to a regular intent. However, the conversation flow for the Q&A intent is different from that of a regular intent.

[0099] (5) Creating a Conversation Flow for the Skill Robot - The conversation flow specified for the skill robot describes how the skill robot reacts when the received user input is parsed into different intents of the skill robot. The conversation flow defines the actions or operations that the skill robot will take (e.g., how the skill robot responds to user utterances, how the skill robot prompts the user for input, how the skill robot returns data, etc.). The conversation flow is like the process that the skill robot follows. Figure 1 The skill robot designer specifies the conversation flow using a language such as the markdown language. In some embodiments, a YAML version called OBotML can be used to specify the conversation flow of the skill robot. The conversation flow definition for the skill robot serves as a model of the conversation itself, which is a model that enables the skill robot designer to plan the interaction between the skill robot and the end user that the skill robot serves.

[0100] In some embodiments, the conversation flow definition includes three parts:

[0101] (a) The context part;

[0102] (b) Default transition section; and

[0103] (c) Status section

[0104] Context section - The skill bot designer can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to: variables for error handling, variables for built-in or custom entities, user variables that enable the skill bot to recognize and adhere to user preferences, etc.

[0105] Default transition section - The transitions of the skill bot can be defined in the conversation flow state section or in the default transition section. The transitions defined in the default transition section act as a fallback and are triggered when there is no applicable transition defined within the state or the conditions required to trigger a state transition are not met. The default transition section can be used to define the routing that allows the skill bot to gracefully handle unexpected user actions.

[0106] Status section - The conversation flow and its related operations are defined as a sequence that manages the transient state of the logic within the conversation flow. Each state node within the conversation flow definition names the components that provide the functionality required at the point in the conversation. Thus, the state is built around the components. The state contains component-specific properties and defines the transitions to other states that are triggered after the component execution.

[0107] Special case scenarios can be handled using the status section. For example, sometimes it may be desirable to provide the end user with the option to temporarily leave the first skill with which the end user is interacting and handle things in a second skill within the digital assistant. In one example, if the end user is busy talking to a shopping skill (e.g., the user has made some purchase selections), the end user may want to jump to a banking skill (e.g., the end user may want to ensure they have enough money for the purchase) and then return to the shopping skill to complete the end user's order. To address this, the actions within the first skill can be configured to initiate an interaction with a second different skill within the same digital assistant and then return to the original flow.

[0108] (6) Add custom components to the skill bot — As described above, the states specified in the conversation flow of the skill bot name the components required to provide the state's functionality. Components enable the skill bot to perform functions. In some embodiments, the DABP 102 provides a set of pre-configured components for performing a wide range of functions. The skill bot designer can select one or more of these pre-configured components and associate them with states in the skill bot's conversation flow. The skill bot designer can also use the tools provided by the DABP 102 to create custom components or new components and associate the custom components with one or more states in the skill bot's conversation flow.

[0109] (7) Test and deploy the skill bot — The DABP 102 provides several features that enable the skill bot designer to test the skill bot being developed. The skill bot can then be deployed and included in the digital assistant.

[0110] While the above description describes how to create a skill bot, similar techniques can also be used to create a digital assistant (or master bot). At the master bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master bot) can handle without invoking the skill bots associated with the digital assistant. Examples of system intents defined for the master bot include: (1) Exit: applicable when the end user signals a desire to exit the current session or context in the digital assistant; (2) Help: applicable when the end user requests help or directions; and (3) UnresolvedIntent: applicable to user inputs that do not closely match the exit intent and the help intent. The digital assistant also stores information about one or more skill bots associated with the digital assistant.

[0111] At the master bot or digital assistant level, when the end user inputs a phrase or utterance to the digital assistant, the digital assistant is configured to perform processing for determining how to route the conversation. The digital assistant uses a routing model to determine this, and the routing model can be rule-based, artificial intelligence-based, or a combination thereof. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input is to be routed to a specific skill for processing, to be processed by the digital assistant or the master bot itself according to the built-in system intents, or to be processed into a different state in the current conversation flow.

[0112] In some embodiments, as part of this process, the digital assistant determines whether the user input identifies a skill bot using its invocation name. The presence of an invocation name in the user input can be considered an explicit invocation of the skill bot corresponding to the invocation name. In such a scenario, the digital assistant can route the user input to the explicitly invoked skill bot for further processing. In some embodiments, if there is no specific invocation, the digital assistant evaluates the received user input and calculates confidence scores for system intents and skill bots associated with the digital assistant. The scores calculated for a skill bot or system intent represent how likely the user input represents a task that the skill bot is configured to perform or represents a system intent. Any system intent or skill bot for which the associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill bot from the identified candidates for further processing of the user input. In some embodiments, after one or more skill bots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and confidence scores are applied to each intent. Any intent for which the confidence score exceeds the threshold is generally considered a candidate flow. If a specific skill bot is selected, the user input is routed to the skill bot for further processing. If a system intent is selected, one or more actions are performed according to the selected system intent.

[0113] As described above, a skill (also referred to as a bot, chatbot, conversational bot, skillbot, or talkbot) is a computer program that can perform a conversation with an end user. A bot can typically respond to natural language messages (e.g., questions or comments) via a messaging application that uses natural language messages. An enterprise can use one or more bot systems to communicate with an end user via a messaging application. The messaging application (which may be referred to as a channel) can be a messaging application preferred by the end user that the end user has installed and is familiar with. Thus, the end user can chat with the bot system without having to learn a programming language and download and install a new application. The messaging application can include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), native or hybrid extensions of mobile and web applications / responsive mobile applications or web applications with chat functionality, or voice-based input (such as a device or application having an interface with Siri, Microsoft Cortana, Google Voice, or other voice input for interaction).

[0114] In some examples, a bot system can be associated with a uniform resource identifier (URI). The URI can identify the bot system using a string of characters. The URI can be used as a webhook for one or more messaging application systems. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The bot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. The HTTP post call message can relate to the URI from the messaging application system. In some embodiments, the message can be different from the HTTP post call message. For example, the bot system can receive messages from a short message service (SMS). Although the discussion herein may refer to the communication received by the bot system as a message, those of ordinary skill in the art will recognize that the message can be an HTTP post call message, an SMS message, or any other type of communication between the two systems.

[0115] End users can interact with a robotic system through conversational interactions (sometimes referred to as a conversational user interface (UI)), just as people interact with each other. In some cases, the interaction can include the end user saying "Hello" to the robot and the robot responding with "Hi" and asking the end user how the robot can help. In some cases, the interaction can also be a transactional interaction with, for example, a banking robot, such as transferring money from one account to another. The interaction can also be an information interaction with, for example, a human resources (HR) robot, such as checking vacation balances. The interaction can be with, for example, a retail robot, such as discussing returning a purchased item or seeking technical support.

[0116] In some embodiments, the robotic system can intelligently process end user interactions without interacting with an administrator or developer of the robotic system. For example, an end user can send one or more messages to the robotic system to achieve a desired goal. The messages can include some kind of content, such as text, emojis, audio, images, video, or other means of conveying a message. In some embodiments, the robotic system can convert the content into a standardized form (e.g., a representational state transfer (REST) call for an enterprise service with appropriate parameters) and generate a natural language response. The robotic system can also prompt the end user for additional input parameters or request other additional information. In some embodiments, the robotic system can also initiate a conversation with the end user rather than passively responding to the end user's utterance.

[0117] A conversation with the robot can follow a specific conversation flow that includes multiple states. The flow can define what happens next based on the input. In some embodiments, a state machine that includes user-defined states (e.g., end user intents) and actions to be taken within or between states can be used to implement the robotic system. The conversation can take different paths based on the end user input, which may affect the decisions the robot makes regarding the flow. For example, at each state, based on the end user input, the robot can determine the end user's intent in order to determine the appropriate next action to take.

[0118] An intent can include the goal that the end user wants to achieve. An intent maps the end user input to actions that the backend system can perform for the end user. Thus, based on the phrases spoken by the end user in natural language, the bot can map the end user utterance to a specific use case or task, such as ordering pizza, getting account balance, transferring money, making a purchase, getting a return, etc. Human conversations are typically non-linear in nature. The end user can generally be in different states during a conversation. For example, if the end user wants to transfer funds from account A to the recipient, the end user can start a conversation with the bot system by, for example, asking the bot to pay for the recipient's dinner. The bot can respond with, for example, "Which account to use?" The end user can select a current deposit account but may then realize that they are not sure about the balance in the account. Thus, the end user can switch the context to request the balance and recent transactions, etc. In other words, the end user may trigger a change in the flow and state, for example, from transferring money to querying the balance and then to recent transactions. At a certain point in time, the end user can decide to return to the original intent - paying the recipient. Thus, one task of the bot system is to dynamically determine the end user intent from natural language utterances.

[0119] The bot can use a natural language processing (NLP) engine and / or a machine learning model (e.g., an intent classifier) to map the end user utterance to a specific intent. For example, a machine learning-based NLP engine can learn to understand and classify natural language conversations from the end user and extract the necessary information from the conversation in order to be able to take precise actions, such as performing a transaction or looking up data from the backend record system.

[0120] Figure 2 Depicted is a distributed system 200 according to certain embodiments, which implements a bot system for communicating with an end user using a messaging application. System 200 can include a bot system 220, one or more messaging application systems 215, and one or more end user devices (such as one or more mobile devices 210). In some examples, the messaging application can be installed on an electronic device (e.g., a desktop computer, a laptop computer, mobile device 210, etc.). Although the discussion herein will refer to mobile devices and messaging applications, those of ordinary skill in the art will recognize that any electronic device can be used, and any messaging platform or messaging application can be used, such as Messenger, instant messaging software, mobile text and voice messaging communication services, Messenger, Messenger, SKYPE Messenger, Short Message Service (SMS), or any other messaging application that provides a platform for end - user communication. In other examples, a messaging application can be run through a browser (such as the GOOGLE browser, browser, and INTERNET EXPLORER browser) installed on the mobile device 210. In some embodiments, two or more messaging applications can be installed on the end - user device for communication through two or more messaging platforms, such as two or more messaging application systems 215).

[0121] A messaging application can be facilitated through a messaging platform such as the messaging application system 215. The mobile device 210 can be connected to the messaging application system 215 through a first network (e.g., the Internet). The messaging application system 215 can be a messaging platform provided by a third party, such as Facebook, Tencent, Google, Microsoft, etc. The messaging application system 215 can manage the content sent and received through the messaging application across multiple mobile devices or other end - user devices.

[0122] The robot system 220 (e.g., implemented on one or more servers) can also be communicatively connected to the messaging application system 215 to send and receive messages. The communication between the messaging application system 215 and the robot system 220 can be through a second network (e.g., the Internet). The first network and the second network can be the same network, or they can be similar or completely different networks. The messaging application system 215 can use the Internet to route content (e.g., a message or information from a message) from the mobile device 210 to the robot system 220. In some embodiments, the destination of the content (e.g., the identity of the robot system 220) can be included in the content as a nominal accessed address. In some embodiments, the robot system 220 can also be configured to communicate with two or more messaging application systems 215.

[0123] As discussed above, the content exchanged between end - users or between an end - user and a robot system can include, for example, text, emojis, audio, media (e.g., pictures, videos, links), or any other method of conveying a message. Message examples received by the robot system 220 from, for example Messenger can include:

[0124]

[0125] The robot system 220 can receive content from the messaging application system 215 using a connector 230, which serves as an interface between the messaging application system 215 and the robot system 220. In some embodiments, the connector 230 can standardize the content from the messaging application system 215 so that the robot system 220 can analyze content across different messaging application systems. The content standardization process can include formatting the content from each type of messaging application into a common format for processing. In some embodiments, for each messaging application (e.g., Messenger, instant messaging software, mobile text and voice messaging communication services, Messenger, Messenger, and SKYPE Messenger, Short Message Service (SMS)), the robot system 220 can include one or more connectors. In some implementations, the connector 230 can route the content to the message input queue 240. The message input queue 240 can include a buffer (e.g., a first-in-first-out (FIFO) buffer) that stores the content in the order of receipt. In some embodiments, each connector 230 can be associated with one or more message input queues.

[0126] When the message processor 250 becomes available, the message input queue 240 can send the content to the message processor 250. In some embodiments, the message processor 250 can pull the content from the message input queue 240. As described in detail below, the message processor 250 can parse the message and determine the intent of the parsed message. In some embodiments, the message processor 250 can include a natural language processor 252 and an intent determination subsystem 254. The natural language processor 252 can parse the message and perform some semantic analysis, such as identifying the subject, predicate (e.g., action), and / or object. The intent determination subsystem 254 can determine the end-user intent based on the parsed message. As described above, the intent can include the purpose of the message. For example, the purpose of the message can be to order pizza, order a computer, transfer money, ask a question about delivery, etc. In some embodiments, the parameters associated with the intent of the action to be taken (the parameters can be referred to as entities) can also be extracted from the message by the natural language processor 252 and / or the intent determination subsystem 254.

[0127] After the message processor 250 determines the end-user intent based on the content, the determined intent (along with the parameters associated with the intent) can be sent to the action engine 260. The action engine 260 can be used to determine the action to be performed based on the intent (and the parameters associated with the intent) as described above and the current state (or context) of the state machine. For example, the action engine 260 can send certain outbound content as a response to the message output queue 270 and / or can send commands to or obtain information from some enterprise services (such as enterprise service 225). The message output queue 270 can send the outbound content to the connector 230. The connector 230 can then send the outbound content to the messaging application system indicated by the action engine 260, which can be the same as or different from the messaging application system 215. The messaging application system 215 can then forward the outbound content to the messaging application on the mobile device 210.

[0128] The robotic system 220 can communicate with one or more enterprise services (such as enterprise service 225), one or more storage systems for storing and / or analyzing messages received by the robotic system 220, or a content system for providing content to the robotic system 220. The enterprise service 225 can communicate with one or more connectors 230, the action engine 260, or any combination thereof. The enterprise service 225 can communicate with the connector 230 in a manner similar to the messaging application system 215. The enterprise service 225 can send content to the connector 230 to be associated with one or more end-users. The enterprise service 225 can also send content to the connector 230 to cause the robotic system 220 to perform an action associated with the end-user. The action engine 260 can communicate with the enterprise service 225 to obtain information from the enterprise service 225 and / or to instruct the enterprise service 225 to take an action identified by the action engine 260.

[0129] In some embodiments, the robotic system 220 can include one or more timers. The timer can cause the action engine 260 to send content to the end-user using the connector 230 and the messaging application system 215 after a period of time has passed. In some embodiments, the timer can send content to the robotic system 220 similar to an end-user or the enterprise service 225. For example, the timer can send a message to the robotic system 220 to be analyzed in the same way as a message from an end-user would be analyzed.

[0130] In a particular embodiment, an end user can use a mobile device 210 to send a message to a robot system 220 via a messaging application system 215. The message can include a greeting such as "Hello" or "Hi". The robot system can determine that a new session with the end user has started and initiate a state machine. In some embodiments, the robot system can identify one or more characteristics of the end user. For example, the robot system can use a profile associated with the end user on the messaging application system to identify the name of the end user. Using the one or more characteristics, the robot system can respond to the end user via the messaging application. The response can include a message to the end user that responds to the message received from the end user. For example, the response can include a greeting with the name of the end user, such as "Hi, Tom, what can I do for you?". Depending on the enterprise associated with the robot system, the robot system can proceed to accomplish the goals of the enterprise. For example, if the robot system is associated with a pizza delivery enterprise, the robot system can send a message to the end user asking if the end user wants to order a pizza. The session between the robot system and the end user can continue from there, back and forth, until the robot system completes the session or the end user stops responding to the robot system.

[0131] In some embodiments, the robot system can initiate a session with the end user. The session initiated by the robot system can be in response to a previous session with the end user. For example, the end user can order a pizza in a previous session. The robot system can then initiate a session after the pizza is ready. In some embodiments, when receiving an indication from the enterprise associated with the robot system, the robot system can determine that the pizza is ready (e.g., an employee sends a message to the robot system indicating that the pizza is ready). The session can include a message sent to the end user indicating that the pizza is ready.

[0132] In some embodiments, the robot system can send a message to the end user via a messaging application different from the messaging application that previously received the message. For example, the robot system can determine to use Short Message Service (SMS) instead of Messenger to send the message. In such an implementation, the robot system can integrate multiple messaging applications.

[0133] In some embodiments, the robot system can determine to start a session based on a timer. For example, the robot system can determine to schedule a one-week timer for the end user after a pizza is ordered. The expiration of the one-week timer may cause the robot system to start a new session with the end user for ordering another pizza. The timer can be configured by the enterprise and implemented by the robot system.

[0134] As described above, in some embodiments, the action engine 260 may send commands to or obtain information from some enterprise services 225. For example, when the robot system 220 (more specifically, the message processor 250) determines the intent to check the balance, the robot system 220 may determine which of several accounts (e.g., a checking or savings account) to check the balance of. If the end user enters "What is the balance in my savings account?", the robot system 220 may extract "savings" and send a command to the bank server to check the balance and then send the received balance information to the end user via a message. If the end user initially only says "What is the balance in my account?", the robot system 220 may send a message to prompt the end user to further specify a particular account, or may obtain information about all of the end user's accounts and send the account information to the end user for the end user to select from.

[0135] In some embodiments, the robot system may maintain information between sessions. The information may be used later so that the robot system does not need to ask some questions every time a new session is started between the end user and the robot system. For example, the robot system may store information about a previous pizza order of the end user. In a new session, the robot system may send a message to the end user asking if the end user wants the same order as last time.

[0136] In some embodiments, the robot system 220 may store information associated with the end user in a cache. After sending an outbound message from the connector 230 to the messaging application system, the cache may be written to a database to save the information. In other embodiments, the cache may be written to the database at different times (e.g., after a particular event, after each event, after a certain time, or any other metric used to determine when to write to the database).

[0137] When deceleration is recognized, the robot system 220 may allow scaling of each component. For example, if the robot system 220 recognizes that the number of messages arriving at the connector 230 exceeds a threshold, one or more additional connectors may be added to the connector 230. Additionally, the number of message input queues, message processors, action engine instances, and message output queues may increase depending on where the deceleration occurs. In this implementation, additional components may be added without having to add other additional components. For example, connectors may be added without having to add additional instances of the action engine. In some implementations, one or more components or parts of the components of the robot system 220 may run on a virtual machine. By running on a virtual machine, additional virtual machines may be started as desired.

[0138] As described above, building a robotic system, such as a user intent classification engine that can determine a final user's intent based on the final user's utterance, is a challenging task, in part due to the nuances and ambiguities of natural language and the dimensionality of the input space (e.g., possible final user utterances) and the size of the output space (number of intents). Thus, to improve the performance of the robotic system and the user experience with the robotic system, it may be necessary to monitor, debug, and modify the new robotic system. In many cases, without using analysis or optimization tools, it may be difficult to more specifically identify the root cause of the robotic system's performance falling below the expected performance and determine how to improve the robotic system.

[0139] In some cases, a robot owner, developer, or administrator may want to monitor the operational state of the robot and understand how the robot is being used and where the final user has abandoned the robot, in order to improve the robot. For example, a robot owner, developer, or administrator may want to know which robot sessions are successful and which are not, in order to identify and diagnose elements of poor performance in the robotic system.

[0140] According to some embodiments, an analytics system can be integrated with the robotic system. The analytics system can monitor events that occur during a session between a final user and the robotic system, aggregate and analyze the collected events, and graphically provide information about the session at different levels of generalization (such as all sessions, different categories of sessions, and individual sessions) on a graphical user interface. For example, the graphical user interface can display options for filtering or selecting certain types of sessions or individual sessions and graphically display the selected information, such as by visualizing the path of the session. The analytics system can also provide recommendations, options, or other information for improving the robotic system.

[0141] Figure 3Illustrates an integrated system 300 according to certain embodiments, the integrated system including a robotic system (such as robotic system 220) and a robotic analysis system for monitoring, analyzing, visualizing, and improving the performance of the robotic system. As illustrated, the robotic system may include a connector 330 and multiple robotic engines such as a dialogue engine 312, an intent modeler 314, an entity resolver 316, and a custom component 318. The robotic system may also include a database 340, a management application programming interface (API) 350, a user interface 354, and a UI server 352. The robotic analysis system may include a collector 355, an enrichment engine 360, a database 370, and a REST server 380. The robotic analysis system may also include a user interface 392 and a UI server 390. The collector 355 of the robotic analysis system may collect events 305 occurring at the robotic system. Feedback 394 from the robotic analysis system may be provided to the robotic system via the user interface 392 and the user interface 354.

[0142] The connector 330 may act as an interface between the robotic system and one or more end users via one or more channels (such as channels 320 and 322). Each channel may be a messaging application, such as a messaging channel (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), a virtual personal assistant (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), a native or hybrid extension of a mobile and web application extension / a responsive mobile application or web application with chat functionality, or a voice-based input (such as a device or application with Siri, Microsoft Xiaoice, Google Assistant, or other voice input for interaction). In some embodiments, the connector 330 may standardize the content from different channels such that the robotic system can analyze the content across different messaging application systems. The content standardization process may include formatting the content from each type of messaging application into a common format for processing. In some embodiments, for each channel, the robotic system may include one or more connectors.

[0143] The intent modeler 314 can be used to determine the end - user intent associated with an end - user utterance. In some embodiments, the intent modeler 314, which is used to determine the end - user intent based on one or more messages received from the end - user by the robotic system, can use a natural - language processor to tag word forms (verbs, nouns, adjectives), find lemmas / stems (runs / running / ran -> run) and tag entities (Texas -> LOCATION). In some embodiments, the intent modeler 314 can standardize the messages. For example, "Mary ran to Texas" can be transformed into "PERSON run to LOCATION". The intent modeler can also include logic for detecting words with the same meaning in the end - user message. For example, if the training data set includes: "Mary ran to Texas" and "Bob walked to Detroit", both of which map to the same intent and run / walk appear in the same set of intents, then the intent modeler 314 can learn that, for the purpose of intent parsing, run = walk. In an illustrative example, "Mary ran to Texas" can be transformed into "PERSON run to LOCATION", and "Bob walked to Detroit" can be transformed into "PERSON walk to LOCATION". In the illustrated example, both sentences can be associated with the same intent because, for the purpose of intent parsing, "noun run to noun" is the same as "noun walk to noun". In another example, "I want to order a large cheese pizza" and "I want to order a small pepperoni pizza" can both be standardized to "I want to order a Bots_PizzaSize Bots_Toppings pizza".

[0144] After normalization, the probability that the occurrence of a word can indicate a certain intent can be determined. In some examples, a basic probability algorithm can be used to combine the probabilities as if the probabilities were independent. For example, if the probability that "order" implies ordering pizza is 20% and the probability that "pizza" implies ordering pizza is 10%, then the total probability is 1-(1 - 0.2)(1 - 0.1) = 28%. Some probabilities can be based on the presence of a word or on the presence of certain language elements such as negation or personal pronouns.

[0145] Another level of rules can be template rules, which are combinations of words. In some examples, each sentence in the training dataset can automatically become a rule once it is normalized. In such examples, the training dataset can include a very small number of short sentences. The template rules can return a probability of 1. New rules can be generated from the rules through an induction process. For example, the following sentences can belong to tracking expenses: "How much did I spend on gas last month?" and "How much did I spend on food in May?". The sentences can be used to induce the rule "How much did I spend" because that is the part they share. In other examples, the training dataset can include the phrase "How much did I spend" to achieve the same result.

[0146] The examples described above allow the definition of intents not to include duplicates such as variations of named entities (e.g., "Send money to Sue" and "Send money to Bob"). However, similar sentences with one or two different words can be used for training. Similar sentences can allow the model to learn which words may have the same intent parsing meaning and which words may be common spelling mistakes.

[0147] If a particular word or group of words (such as a verb) is important for an intent, the probability can be manipulated by having more examples use such a word (and its synonyms) and fewer examples with such a word for other intents.

[0148] Examples can also be provided to prevent the model from making incorrect assertions. For example, a particular subphrase or a word that only appears for a certain intent may lead to an incorrect assertion. Similarly, the model can be prevented from synthesizing broad rules using similar sentences belonging to different intents for training.

[0149] The entity resolver 316 can identify entities (e.g., objects) associated with the end-user intent. For example, in addition to the end-user intent identified by the intent modeler 314 (such as "order pizza"), the entity resolver 316 can resolve entities associated with the intent such as pizza type, toppings, etc.

[0150] The dialogue engine 312 can be used to handle conversations between the end user and the robot system. For example, the dialogue engine 312 can respond to an end user utterance based on the end user intent identified by the intent modeler 314 and the entity associated with the end user intent identified by the entity resolver 316. In some embodiments, the dialogue engine 312 can use a state machine including user-defined states (e.g., end user intents) and actions to be taken within or between states to handle the conversation with the end user.

[0151] The custom component 318 can include a customization module for a specific robot system. For example, a finance robot can include a custom component that can be used to, for example, query balances, transfer funds, or pay bills.

[0152] The database 340 can be used to store data of the robot system, such as data of the classification model, session logs, etc. The management API 350 can be used by the administrator or developer of the robot system to manage the robot system, such as retraining the classification model, editing intents, or otherwise modifying the robot system. The administrator or developer can use the user interface 354 and the UI server 352 to manage the robot system.

[0153] Various events can be generated when the robot system is running. The events can be generated based on one or more instructions included in the robot system. For example, when the robot system has entered a specific state, which is defined by the administrator or developer of the robot system, an event can be generated. When an event is generated, the event can be collected, stored, and analyzed by the robot analysis system. When an event is captured, additional information associated with the event can also be collected, where the additional information can indicate the current context in which the event is generated.

[0154] For example, session events can be generated by the dialogue engine 312. Session events can include messages received by the robot system from the end user device (referred to as msg_received). Msg_received can include one or more of the following parameters or variables: the content of the message, the time when the message is received by the robot system, the language of the received message, the nature of the device (e.g., version or name), the nature of the operating system (e.g., version or name), the nature of the geographical location (e.g., Internet protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, robot system ID, tenant ID, etc.), timestamp (e.g., timestamp created by the device, sent by the device, obtained by the collector), channel, etc.

[0155] Session events can also include messages sent by the bot system to the end-user device (referred to as msg_sent). Msg_sent can include one or more of the following: the content of the message (e.g., the text or HTML of the message), the time the message was sent by the bot system, the language of the message, the creator of the message (e.g., the bot system or the end-user device), the device nature, the operating system nature, the browser nature (e.g., version or name), the application nature (e.g., version or name), the geographical location nature (e.g., Internet protocol address, latitude, longitude, etc.), the identification information (e.g., user ID, session ID, bot system ID, tenant ID, etc.), the channel (e.g., Facebook or Webhook), etc.

[0156] The dialogue engine 312 can also generate dialogue state execution events. As described above, the dialogue engine 312 can use a state machine to determine the conversation flow with the end user. The state machine can include a set of states and transition rules between the states. The dialogue engine 312 can execute the state machine for each end-user session, and the dialogue state execution events can be generated for each state that the dialogue engine 312 steps through to process the end-user utterance. The attributes of the dialogue state execution events can include, for example, state name, component name, next action, entity match, intent match, variables, user query statement, response statement, execution time, communication language, device nature, operating system nature, geographical location nature, identification information, timestamp, channel, etc. The state name can be the name of the currently executed state or the "error state". The component name can be the name of the bot component executed for the current state. The next action can be the next action to be executed. The entity match can be the entity parsed in the current message. The intent match can be the intent parsed with a score value. The variables can be the variable values of the current state. The query statement can be the message sent by the end user. The response statement can be the message sent to the end user. The execution time can be the timestamp of the completed state execution. The communication language can be the language of the messages in the conversation. The device nature and / or the operating system nature can be associated with the end user interacting with the bot system. The browser nature and / or the application nature can be associated with the end user interacting with the bot system. The geographical location nature can be the location of the end user interacting with the bot system.

[0157] Due to the execution of the intent modeler 314, intent parsing events may occur. The intent modeler 314 can identify the end-user intent from a set of intents based on the end-user utterance using a trained or otherwise defined classification model. The result of the intent classification can be captured as an intent parsing event attribute, which can include, for example, the final intent classification result (e.g., the identified intent) and the confidence score associated with each corresponding intent in the set of intents.

[0158] The entity resolver 316 can generate entity resolver events. An entity is an object associated with the end-user intent. When the robotic system is created, entity definition rules can be determined. For example, in addition to parsing end-user intents such as "order pizza", the robotic system can also use the entity resolver 316 to parse associated entities such as pizza type, filling, etc. The entity resolver events can be captured during entity resolution. Examples of the attributes associated with the entity resolver events can include entity name, applied rules, search terms, parsed status, query statement, entity type, execution time, communication language, device nature, operating system nature, browser nature, application nature, geographical location nature, identification information, timestamp, channel, etc. The entity name can be the name of the entity currently being parsed. The applied rules can be, for example, the aforementioned, subsequent, or aggregated ones. The search terms can be from, to, destination, origin, etc. The parsed status can be the dialogue status for entity resolution. The query statement can be a message containing the entity value. The entity type can be the system or obtained. The execution time can be the timestamp of entity resolution. The communication language can be the language of the message in the conversation. The device nature and / or the operating system nature can be associated with the end-user interacting with the robotic system. The browser nature and / or the application nature can be associated with the end-user interacting with the robotic system. The geographical location nature can be the location of the end-user interacting with the robotic system.

[0159] Custom components can also generate events such as predefined events or custom events. Predefined events can be properties captured when a custom component is executed. Examples of the properties of predefined events can include: component name, event name, payload, execution time, communication language, device properties, operating system properties, browser properties, application properties, geographical location properties, identification information, timestamp, channel, etc. The component name can be the name of the currently executed custom component. The event name can be invoked, invocation_failed, replied, replied_failed, etc. The payload can be the reason for failure (in case of failure), stack trace, etc. The execution time can be a timestamp indicating when the event occurred. The communication language can be the language of the message in the conversation. The device properties and / or operating system properties can be associated with the end user interacting with the robot system. The browser properties and / or application properties can be associated with the end user interacting with the robot system. The geographical location property can be the location of the end user interacting with the robot system.

[0160] Custom components can also issue custom events during the execution of the custom component. Examples of the properties of custom events can include component name, event name, custom payload, execution time, communication language, device properties, operating system properties, browser properties, application properties, geographical location properties, identification information, timestamp, channel, etc. The component name can be the name of the currently executed custom component. The event name can be a user-defined event name (e.g., Balance_Retrieved). The payload can be, for example, {"amount": "100 US dollars", "account": "current deposit"}. The execution time can be a timestamp indicating when the event occurred. The communication language can be the language of the message in the conversation. The device properties and / or operating system properties can be associated with the end user interacting with the robot system. The browser properties and / or application properties can be associated with the end user interacting with the robot system. The geographical location property can be the location of the end user interacting with the robot system.

[0161] Error events and timeout events can also be generated by the robot system during execution. When an error occurs, an error event can be generated. When the end user session has been inactive for a period of time, a timeout event can be generated, and the timeout event can be configured at the channel.

[0162] When the robot system has a conversation with the end user and generates corresponding events, the robot analysis system can collect the events and additional information. For example, the collector 355 can collect the events and additional information and send the collected information to the queue. In some embodiments, the collector 355 can be configurable and can be programmed to collect different events and / or event attributes described above as desired. For example, the collector 355 can be configured to capture conversation state attributes, intent resolution attributes, entity resolution attributes, as well as error attributes and timeout attributes. In some embodiments, the collector 355 can also be configured to collect information about the events 395 generated by systems other than the robot system.

[0163] The enrichment engine 360 can perform validation and enrichment on the collected events and other information and write the collected events and other information to the database 370. For example, based on the collected IP address, the enrichment engine 360 can determine the location of the end user associated with the IP address. As another example, the enrichment engine 360 can extract certain features from the collected information, such as determining the web browser or channel used by the end user. The REST server 380 can analyze the enriched events and other information and generate various reports based on a certain aggregation metric 372. The reports can be displayed on the user interface 392 by the UI server 390 to the owner, administrator, or developer of the robot system. The owner, administrator, or developer of the robot system can provide feedback 394 to the robot system for improving the robot system.

[0164] Figure 4 FIG. 400 is a simplified flowchart illustrating an example of a process for developing a skill according to certain embodiments. The process may include creating an intent at 410, training a skill at 420, creating an entity at 430, integrating custom components at 440, creating a dialog flow at 450, testing a skill at 460, routing to a channel at 470, and reviewing an insights report for improving the skill at 480.

[0165] At 410, an intent of the skill can be created. The intent describes the various actions that the skill can help its end user complete. For example, if the skill enables the user to perform various banking transactions, the intent of the skill can include, for example, "query balance" or "transfer funds". The intent not only describes what the skill can do, but can also be an intelligent component of the skill. The intent enables the skill to recognize user input because each intent can have a set of typical user statements (i.e., utterances) associated with it. Although these utterances can share the same meaning, they can be different (e.g., "What is my savings account balance?" and "How much money do I have in my checking account?").

[0166] At 420, when the skill parses the user input, the skill can be trained to infer the user's intent. Specifically, the skill can be trained with intents and their utterances (collectively referred to as training data) so that the skill can parse the user input into one of the intents. The trained skill can not only identify sample phrases belonging to each intent but also identify similar phrases corresponding to each intent.

[0167] At 430, entities for the skill can be created. In some embodiments, the skill may require some additional context to complete the user request. Although some user requests can be parsed into the same intent (e.g., "What is my savings account balance?" and "How much money is in my checking account?" will both be parsed into the "query balance" intent), they are requesting different things. To clarify the request, one or more entities can be added to the intent. Using the banking skill example, the entity "account type" (which defines values such as "checking" and "savings") can enable the skill to parse the user request and respond appropriately.

[0168] At 440, custom components can be integrated into the skill. Before integrating the components into the skill, the skill can identify the user input but may not be able to respond to it. The components can enable the skill to perform its functions. The components can perform functions (such as outputting text based on the intent parsed from the end user's message) or perform tasks specific to a particular skill (such as querying the account balance).

[0169] At 450, a dialogue flow can be created. The dialogue flow describes how the skill reacts when different intents are parsed. The dialogue flow defines the actions or operations that the skill bot will take (such as how the skill bot responds to the user's utterance, how the skill bot prompts the end user for input, how the skill bot returns data, etc.). In some embodiments, the dialogue flow can be a flowchart described in a simple markup language such as a version of YAML (e.g., OBotML).

[0170] At 460, the skill can be tested after being developed and deployed in a digital assistant such as described above. The skill can be tested by conducting one or more sessions with the skill using one or more dialogue flows.

[0171] Optionally, at 470, the skill can be routed to one or more channels for user messaging and other capabilities. For example, if the skill is not to be added to the digital assistant, the skill can be added to one or more user messaging channels. The user can chat with the skill through one or more of these user messaging channels, such as various messaging platforms, proprietary messaging applications, and web pages. The skill will run on any of these channels, whether they are plain text or can support scrolling cards, UI elements, images, and other non-text content. In addition to the channels connected to the user interface, the skill can also be routed to other channels, such as a channel that links the skill to a customer support system or a channel that routes notifications from an external application that initiated a session with a prompting skill.

[0172] At 480, the developer can review the insights report to improve the skill. For example, the developer can review the insights report related to the skill to find out whether the customers are using the skill as expected. The insights report can include, for example, high-level usage metrics and session trends, individual views of intents, their execution paths, and conversation transcripts. The insights report can provide different perspectives on how well the skill supports its customers and where the skill is preventing its customers from completing tasks. These insights reports can not only allow the developer to quickly identify problems but also suggest user inputs that can improve the intent parsing of the skill.

[0173] In some embodiments, different versions of the skill can be generated. In some embodiments, the skill can be a composite skill that includes two or more related skills to perform more complex functions as described above. For example, composite skill A can include skill B and skill C, where, for example, the output from skill B can be used as the input to skill C.

[0174] II. Skill Store

[0175] As described above, an enterprise or individual that needs to use a bot to communicate or otherwise interact with an end user may not have the expertise to build a skill. In many cases, the skill can be built by a developer as described above and made available to the enterprise or individual. The developed skills can be available in a skill store. In some embodiments, the skill store can allow developers and partners (or customers) to publish skills. The skill store can allow developers to upload a skill package as part of a product release to the skill store. In some embodiments, pushing a skill to the skill store can be available from the skill editing page. The skill store can have a specific directory structure and can provide a detailed skill description.

[0176] The skill store can provide detailed skill descriptions. In some embodiments, the skill store can include a skill store UI that allows customers to access the skill store. In some embodiments, the skill store UI can use an object repository REST API to collect skill details and facilitate browsing of skill functionality. Skills, versions, skill descriptions, and additional data can be provided via the skill store UI for customers to view the available skills. For example, the skill store UI can allow customers to browse skills, search for specific skills, search for specific versions of skills, search by keyword in the name, summary, description, etc., and sort skills by name, category, date, etc. When a skill is pushed to the skill store, the category can be input by the developer. Skills can be displayed along with the name, version, short description (e.g., summary), long description details, etc. In some embodiments, the skill store can have other functions such as notifications, automatic updates, etc.

[0177] The skill store UI can also allow downloading of specific skill versions. The skill store UI can access the skill store via an administrative API. The administrative API can redirect all requests to the skill store via, for example, a Jersey REST client. The administrative API can use a standard library to access the storage device for obtaining authorization tokens / passwords. Account information can be stored in the configuration of the robot.

[0178] The skill store can store the implementation and metadata of skills. The implementation of a skill can include, for example, a pre-trained model in Java. The implementation of a skill can include implementation data such as the intent, entities, custom components, dialog flow, model, etc. of the skill. In some embodiments, the implementation of a skill can be stored in the skill store as a zip archive. The metadata of a skill can include information about the implementation of the skill (such as the name, category, version, short description (e.g., summary), long description details, etc. of the intent, entities, custom components, dialog flow, model, etc. of the skill). For example, in some embodiments, the skill store can include a backend that can store skills as zip archives. The backend can parse and store binary data, create and store metadata in JSON format, and control the workflow with existing skills in the skill store using overwrite parameters. In some embodiments, the implementation and metadata of a skill can be stored separately.

[0179] In some embodiments, the backend can also list all skills (e.g., show the latest version by date) with pagination, sorting, and filtering support. The skill store can obtain the version, summary, and description information of a skill from the skill metadata. The backend can be configured to obtain all versions of a skill by name. The backend can allow downloading of a skill using the path to the zip archive and automatically install the downloaded skill. The backend can also allow deletion of all skills or deletion of a skill by name.

[0180] In some embodiments, the Skills Store may include a REST API that can be used to transfer content between different storage devices. In some embodiments, the Skills Store may include a REST API that can return all the content of the Skills Store as a zip archive. During import / export, the Skills Store may also support the training model option. For example, the Bot Import API in the Management API can be used to import skills and automatically train the skills. In some embodiments, a trained bot with a model can be exported.

[0181] In some embodiments, the Skills Store may include a REST API that can be used to push skills to the Skills Store. The REST API can accept a zip archive and some details of the skill for uploading the skill to the Skills Store. In some embodiments, the REST API can control the workflow with existing skills in the Skills Store using override parameters. In some embodiments, the REST API may not be available through the Skills Store UI, such that for security reasons, customers may not be able to push skills to the store.

[0182] In some embodiments, the Skills Store REST API may include various REST APIs such as an Object Store REST API for accessing data in an object storage device (e.g., listing all containers or all objects in a container or bucket or downloading an object) and a Skills Store Management REST API for accessing data in the object storage device through the management API of the robot (e.g., with a pre-installed JSON formatter). External APIs that allow calling endpoints (e.g., servers) from the management API container outside the robot's environment may be used to upload and download skills. The user interface may use the Skills Store Management REST API to browse the skills in the Skills Store and install skills in the local environment. For example, the Skills Store Management REST API may list skills, list all robots in the robot's environment, list recent robots in the Skills Store, and identify outdated robots previously imported in the robot's environment from the Skills Store in response to a simple list request. The Skills Store Management REST API may list skills by pagination or sorting. The Skills Store Management REST API may list skills filtered by name, description, summary, category, etc. In some embodiments, the Skills Store Management REST API may list all versions of a skill with the same name. The Skills Store Management REST API may allow downloading skills from the Skills Store using a download link path, downloading and installing skills from the Skills Store, deleting skills in the Skills Store by name and version, etc. In some embodiments, the Skills Store REST API may include a Skills Store Upload (Push) REST API, a Skills Store Delete (deleting) (delete) REST API, a Skills Store Install (Pull) REST API, a Skills Store Install All Digital Assistant Skills (Pull) REST API, etc.

[0183] In some embodiments, the Skills Store can be an object repository where any type of data (regardless of content type) can be stored as an object. An object can include the object itself and the metadata of the object. Each object can be stored in a bucket. A bucket can be a logical container for storing objects. A user or system can create buckets as needed. A bucket can be associated with a single compartment that has a policy for determining what actions a user can perform on the bucket and the objects in the bucket. The Skills Store can include a namespace which can be a logical entity that serves as the top-level container for all buckets and objects, thus allowing a user to control bucket naming within the user's lease. Each lease can be set with a unique but non-editable object storage namespace that is global and spans all compartments and regions. Bucket names can be unique within a lease. In the object storage namespace, buckets and objects can exist in a flat hierarchy or can be arranged in a directory structure to assist in navigating a large set of objects (e.g., robots / skills / billing, robots / skills / crm, etc.). A compartment can be a collection of related resources that can be accessed only by those who have been explicitly granted access rights by an administrator. Compartments can help users organize resources and control access to those resources. When a compartment is provisioned, the Skills Store can include a root compartment. An administrator can create additional compartments within the root compartment and add access rules for those compartments. Buckets can exist in only one compartment.

[0184] In some embodiments, each skill can be placed in a container having a name such as " / bots / {uuid}.zip". After a skill is uploaded as a file to the Skills Store, corresponding metadata can be created for each skill. The metadata file can be in JSON format and can have a name such as " / bots / {name} / {version} / bot.json". An example of the metadata for a skill is shown below:

[0185]

[0186] III. Skill Expansion

[0187] As described above, each skill can be defined by a set of artifacts (e.g., entities) representing the metadata required to configure, train, and execute the skill. Skills for specific functions can be built by software as a service (SaaS) providers to allow SaaS customers to download and use. In many cases, it may be necessary to customize and extend the skills. For example, a customer may want to customize and / or extend a factory-built skill to adapt the skill to specific characteristics, processes, terminology, culture, etc. A robotic extensibility infrastructure that can provide extensibility of skills in a holistic, general, flexible, secure, and maintainable manner may be desirable.

[0188] According to some embodiments, robotic extensions defined in a JSON file can be used to customize or extend a base skill. A robotic extension can include the changes to be made to the base skill and can be applied to the base skill to customize or extend it. The base skill includes the original metadata to be customized and / or extended by the extension. Customization can refer to changing one or more properties of the existing metadata artifacts or resources of the skill. Extension can refer to enhancing the existing metadata or adding new artifacts or resources to the metadata of the skill. Depending on the granularity of the changes applied, customization and extension can overlap. For the sake of clarity, simplicity, and readability, both customization and extension are referred to as "extension" herein.

[0189] All metadata resources of a skill can be extended. These metadata resources can include, for example, (1) top-level skill definitions and their configurations and settings; (2) intents (the intents can be created, modified, or disabled); (3) entities (the entities can be created, modified, or disabled); (4) utterances (the utterances can be created, modified, or disabled); (5) custom components (e.g., the custom components can be created or modified); (6) session flows (the session flows can be modified); and (7) resource bundles (e.g., adding new message keys and default language messages, adding new supported languages, or adding / modifying translated messages). In various embodiments, different combinations of these metadata artifacts may be allowed to be extended. In some embodiments, some metadata artifacts can be disabled. Extensions are not allowed to remove or delete any metadata artifacts associated with the base skill.

[0190] The robot extensibility infrastructure disclosed herein can support the extension of any published version of a skill (i.e., a base skill) by managing a web application and / or REST APIs such as the application composer of Oracle Fusion Applications or the command line interface (CLI) tool of the robot. Any artifacts or resources in the skill metadata can be extensible. The infrastructure can allow clients to obtain, on demand, a representation of the differences between an extension and its corresponding base skill. The metadata associated with an extension can optionally support the definition of a test suite that can be used by the infrastructure to validate the extension when necessary. The solution can follow the version control capabilities of the robot, where each extension can have its own version separate from the version of its base skill, such that the base skill and the extension can evolve separately. The version of an extension can keep track of the range of base skill versions that the version of the extension is compatible with.

[0191] The robot extensibility infrastructure can support the upgrade of a base skill and can determine the changes between different versions of the base skill and the changes to be made to the upgraded metadata of the upgraded version of the base skill. After upgrading a base skill to a newer version, an extension can still be applicable to its base skill as long as (1) metadata compatibility is retained, (2) no metadata change conflicts are detected or are outstanding for resolution, and (3) a validation test (if defined by the extension) is successfully executed on the extended skill. To support upgradability, the infrastructure can store and track the version of the extension metadata separately from the base skill to which the extension can be applied. This can allow a particular version of an extension to remain valid and applicable across multiple upgrades or versions of its base skill as long as those upgrades do not break the backward compatibility described above.

[0192] The infrastructure can execute any validation tests defined by an extension. The infrastructure can provide meaningful feedback to identify any issues that may cause the skill of an extension to become invalid as its corresponding base skill is upgraded. When an upgrade to a base skill breaks compatibility with an extension, the infrastructure can be able to detect the incompatibility early. The infrastructure can make its best effort to automatically resolve metadata conflicts during an upgrade. In some embodiments, the infrastructure can provide meaningful feedback to the client / user about any issues that it cannot automatically resolve. The infrastructure can abort the upgrade, automatically disable incompatible extensions, provide user-friendly mechanisms to assist the user with manual / interactive conflict resolution, and allow the user to publish a new version of the extension when needed.

[0193] According to certain embodiments, robot extensions can be implemented in most cases by storing and managing metadata that belongs only to the extension separately from the original base metadata being extended (such as storing only the differences or changes between the base skill and the extended skill). Then, when the extension is applied, the differences or changes can be used to modify the base metadata. For example, the difference between two versions of the same skill can be calculated and stored in, for example, a JSON file, and later the stored difference can be used to extend the base skill by modifying the metadata according to the JSON file to generate the extended skill from the base skill.

[0194] In some embodiments, multiple extensions can be applied on top of the base skill in an ordered stack (e.g., a layer), where one layer (e.g., one extension) can "extend" on top of another layer (i.e., another extension or the base skill). However, using multiple extension layers can exponentially increase the overall complexity. Therefore, in some embodiments, a single-layer solution for robot extensions can be used for most, if not all, use cases. In some embodiments, the infrastructure can be designed and implemented in a scalable manner to allow new extension layers to be added when necessary without having to redesign or significantly modify or refactor the infrastructure.

[0195] There can be several ways to implement these features. In some embodiments, the metadata of the extension and the skill can be stored in a database in JSON format. For example, the metadata artifacts of the base skill can be described in a JSON file and stored in a skill store, and JSON extensions (e.g., RFC-6902 summarized at https: / / en.wikipedia.org / wiki / JSON_Patch) can be used to capture the differences between the base metadata artifacts and their extensions.

[0196] In a simplified example, some resources or metadata artifacts of the base skill can be described in the original document of a JSON file as:

[0197]

[0198] The extension representing the differences or changes between the base skill and the extended skill can be described in an extension document (referred to as a JSON extension document) in the JSON file as:

[0199]

[0200] Therefore, applying the JSON-formatted extension document to the original document may result in a new document (e.g., a JSON file describing some resources of the extended skill):

[0201]

[0202] Among them, the value of the resource "baz" (e.g., "qux") can be replaced with a new value (e.g., "boo"), the resource "foo" and its value can be removed or disabled, and the new resource "hello" and its value can be added.

[0203] The JSON extension document can be a "semantic" difference document that identifies the differences between the metadata of the recognized extension and the original metadata. The JSON extension document can also be used for conflict resolution. Since the metadata of the base skills in the skills store is stored in JSON documents, there will be no need to introduce a new metadata format and no need for format conversion. Therefore, the JSON extension document for storing skill extensions is compatible with the existing metadata model of the skills, thus minimizing the impact on the existing code base (especially the model abstraction layer). The JSON extension will have a minimal increase in storage requirements and will allow the use of existing open-source Java libraries (e.g., JSON extension and Jackson) to implement the infrastructure of the skill extensions described herein.

[0204] Generally, custom code may not be part of the metadata of a skill. In some embodiments, the metadata of a skill can include one or more custom components. The custom components can include references and parameters that can allow the skill to execute custom code. Therefore, a skill extension can include the addition and / or modification of custom components, while the custom code itself may not be included in the metadata of the skill extension and is not stored together with the metadata of the skill extension. If a skill extension needs to modify the custom code associated with a custom component, the metadata of the custom component can be customized in the metadata of the skill extension so that the extended skill can correctly reference the endpoint that can execute the desired custom code. The custom code itself can be provided through the mechanism used to develop any other regular skill.

[0205] In some embodiments, the robot system can include an internal management service and a REST API. The REST API can expose the metadata of the robot as REST resources and can allow create, read, update, and delete (CRUD) operations on the metadata of the robot. Since it is desired to keep the extension model as close as possible to the metadata model of the skills, it may be desirable to use the management REST API to provide programmatic access to the extensible functionality.

[0206] The technology disclosed herein allows a client to interact with extended and regular robots as seamlessly as possible. Since no other new REST APIs will be required, the technology will also have a minimal impact on existing UI code and can simplify the implementation of extensibility in the UI. The technology disclosed herein can also be used for other application management databases such as MongoDB, where different versions or extensions of an application can be stored by storing only the differences between the application and the version or extension of the original application in a JSON document.

[0207] Figure 5 A system block diagram depicting an example of a robot extensibility infrastructure 500 in accordance with certain embodiments is shown. The robot extensibility infrastructure 500 can include a metadata repository 510 that can store metadata 512 of a base robot and metadata 514 of a skill extension. In some embodiments, the metadata repository 510 can be part of the skill store described above.

[0208] The robot extensibility infrastructure 500 can also include one or more robot systems 520, where each robot system 520 can include a management service application 522 and a runtime application 524. The management service application 522 can be used to create extensions, store extensions, track the versions of skills and extensions, check the compatibility between an extension and a base skill, and rebase an extension to different versions of the base skill. At a robot developer device 542 connected to the management service application 522 via a secure link 550, a GUI 534, and a REST API 532, a robot developer can create an extension to a base skill, store the extension in the metadata repository 510, track the versions of the base skill and the extension to the base skill, check the compatibility between the metadata 514 of the extension and the metadata 512 of the base skill, or rebase the extension to different versions of the base skill. In some embodiments, the management service application 522 and the REST API 532 can be used to perform the creation and management of extensions via, for example, a CLI tool 544, a software development kit (SDK) 548, or other systems 546 that can be connected to the REST API 532 via a secure link 552.

[0209] To execute an extended skill, the robotic system 520 can obtain the metadata 512 of the base skill and the extended metadata 514 from the metadata repository 510, apply the metadata 514 to the metadata 512 to generate new parsed metadata 526. The runtime application 524 can use the parsed metadata 526 and the implementation of the base skill to execute the extended skill. For example, the runtime application 524 can apply the metadata to the implementation of the base skill and communicate with the end user through the REST API 528 and various channels 530.

[0210] As described above, the robotic extensibility infrastructure can store and track the versions of robotic extensions separately from the base skills to which the robotic extensions can be applied. This can allow a version of a robotic extension to apply across multiple upgrades or versions of a base skill, as long as the upgrade does not break the backward compatibility as described above. This can also allow different extensions to be applied to the same base skill.

[0211] Figure 6 An example is illustrated of tracking the version of a base robot and the version of a robotic extension and the compatibility between the version of the base robot and the version of the robotic extension in a robotic extensibility architecture according to certain embodiments. In Figure 6 In the illustrated example, there can be four versions of the base robot, including version 1 (610), version 2 (612), version 3 (614), and version 4 (616). A version 1 (620) of an extension of the base robot can be created based on version 1 (610) of the base robot. A management service application, such as the management service application 522, can check the compatibility between the version of the robotic extension and the version of the base robot. When an upgrade to the base skill breaks the compatibility with the extension, the infrastructure can be able to detect the incompatibility early. In Figure 6In the illustrated example, an extended version 1 (620) of the base robot can be compatible with version 1 (610), version 2 (612), and version 3 (614) of the base robot. However, at 622, when version 4 (616) of the base robot is released in the skill store, the management service application can determine that the extended version 1 (620) of the base robot is not compatible with version 4 (616) of the base robot. In some embodiments, the infrastructure (e.g., the management service application) can perform verification tests defined by the extension to determine compatibility or validity. In some embodiments, when incompatibility or invalidity is detected, the infrastructure can perform a best effort attempt at 624 to automatically resolve the compatibility issue. In some embodiments, if the compatibility issue is not resolved, the infrastructure can fail or abort the upgrade or can automatically disable the incompatible extension. In some embodiments, the infrastructure can provide feedback to the client / user about any issues that the infrastructure was unable to automatically resolve, such that the client or user can manually resolve the outstanding issues and release a new version of the extension if needed. For example, a new version 2 (626) of the extension to the base robot that is compatible with version 4 (616) of the base robot can be generated automatically or manually.

[0212] IV. Skill Extension GUI

[0213] Typically, extending a published skill will be as seamless as creating a new robot version. As described above with respect to Figure 5 Developers can use a GUI (e.g., GUI 534) to create and manage extensions to the base skill. An example of a GUI for robot extensions is described below.

[0214] According to certain embodiments, a new skill extension can be created from a skill tile or from a context menu. Figure 7 An example of creating a new skill extension to a published skill using a graphical user interface (GUI) 700 according to certain embodiments is illustrated. In the illustrated example, the user can select a skill 712 from a list 710 and select an operation to apply to the skill 712 from a drop-down menu 720. Operations can include, for example, creating a new skill, creating a new version of an existing skill, extending an existing skill, exporting a skill, etc.

[0215] Figure 8Illustrates an example of creating a new skill extension for a published skill 810 using the context menu 818 in the GUI 800 according to some embodiments. The GUI 800 shows a skill list with multiple tiles, where each tile can show certain information about a skill, such as a short description 812, a training model 814, and an update time 816 of the skill 810. The context menu 818 can be used to select operations to be applied to the skill 810, such as viewing the skill, creating a new version, extending the skill, exporting the skill, exporting session logs, deleting the skill, etc.

[0216] Figure 9 Illustrates an example of a dialog box in the GUI image 900 for creating an extended skill according to some embodiments. The GUI image 900 shows the name and version of the base skill to be extended (e.g., "Agent Robot 1.0"). As Figure 9 shown, a developer can use the "Create Extended Skill" dialog box to enter the display name, name, version, and short description of the extended skill, where the values of the display name, name, version, and short description of the selected skill can be edited. However, not all properties can be edited in the extended skill. Once the extended skill is created, some properties may not be modifiable.

[0217] Figure 10 Illustrates an example of the GUI image 1000 according to some embodiments, showing some inherited components of the base robot and new components of the extended robot. Inherited items such as the original intent 1010 of the base robot can be modified or customized without being deleted. For example, the name, description, and example utterances of the original intent can be edited. Example utterances can be added, edited, or removed. New components such as a new intent 1020 can be added and edited to the extended skill.

[0218] Figure 11 Illustrates an example of the GUI image 1100 according to some embodiments, showing form fields that can be added, edited, or removed for extending a skill. In Figure 11 the example shown, the original me Figure 1 can be renamed to the intent "check in" (1110). Utterances such as "return", "checking in", "hello", etc. can be added to the example of the utterances of the intent "check in". In some embodiments, properties that have been added or updated in the UI (e.g., the description of the "check in" intent) can be highlighted, and the user can choose to roll back the property value to its original state.

[0219] Figure 12An example of a GUI image 1200 according to some embodiments is illustrated, showing a "Restore" icon 1210 for restoring the values of fields such as the description of the intent, etc. to their original values. By clicking on the "Restore" icon 1210, a dialog box 1220 can be displayed that has the original values of the fields and options for restoring the fields to their original values.

[0220] Figure 13 An example of a GUI image 1300 according to some embodiments is illustrated, showing a dialog box 1310 for restoring the value of a field to its original value. The dialog box 1310 can be displayed after the developer selects "Restore" in the dialog box 1220. The developer can use the dialog box 1310 to cancel the restoration or confirm the restoration.

[0221] After the basic skills are extended, the developer can compare the changes made in the extension. Before generating a bot extension that describes the changes from the basic skills to the extended skills, a graphical user interface can be used to compare the metadata of the basic skills with the metadata of the extended skills side by side.

[0222] Figure 14 An example of a GUI image 1400 according to some embodiments is illustrated, showing the comparison result between the basic skills and the extended skills. In the illustrated example, the left window 1410 shows the metadata of the basic skill "Agent Bot", while the right window 1420 shows the metadata of the extended skill "Extended Agent Bot". The developer can compare the metadata to determine the changes made in the extension. In some embodiments, the comparison can be automatically done using the Code Mirror merge feature. The Code Mirror merge feature can be enabled or disabled by the "Compare with Base" button. As Figure 14 shown, the differences or the changes made can be highlighted in the right window 1420.

[0223] As described above, in some embodiments, an extension to the basic skills can be re-based to a different version of the basic skills. Re-basing is the action of re-applying the changes made in the extended skills on top of a new version of the base robot. For example, the developer may have created an extended robot (Pizza Bot 1.0) by extending the base robot 1.0. After some time, a new base robot 1.1 becomes available with several bug fixes, and the developer may want to take advantage of the new fixes in the base robot 1.1. This can be done by "re-basing" the Pizza Bot 1.0 to the base robot 1.1. When re-basing, if more than one possible version is available, the developer can select the target version of the base robot. By default, the latest version of the base robot can be selected as the target version.

[0224] Figure 15 Illustrates an example of the GUI image 1500 according to certain embodiments, showing a dialog box for re - baselining a skill extension. As shown in the GUI image 1500, a user may want to re - baseline an extension (e.g., "Extended Agent", version 1.0) of a base skill (e.g., "Agent Bot 1.0") to a newer version of the base skill (e.g., "Agent Bot 1.1"). The user can select the desired version from the menu 1510 and click the "Re - baseline" button 1520. In some embodiments, if the extended skill is in a published state, the re - baselining action may force the developer to create a new version of the extended skill. If the extended skill is in a draft state, the extended skill can inherit all new features.

[0225] Figure 16 Illustrates an example of the GUI image 1600 according to certain embodiments, showing a dialog box for creating a new version of a published extended skill (e.g., "Extended Agent Bot 1.0"). The GUI image 1600 shows that the developer is required to create a new skill version "Extended Agent Bot 1.1" for re - baselining "Extended Agent Bot Extension 1.0" to a new version of the base skill ("Agent Bot 1.1"), because the extended skill "Extended Agent Bot 1.0" may have been published. Before the merge re - baselining action, a dialog box can be presented to the developer to confirm re - baselining to the new version of the base skill.

[0226] Figure 17 Illustrates an example of the GUI image 1700 according to certain embodiments, showing a dialog box for confirming re - baselining a skill extension to a new version of the base skill. In Figure 17 the illustrated example, the developer may be required to confirm that the developer wants to re - baseline the extended skill "Extended Agent Bot 1.0" to a new version of the base skill ("Agent Bot 1.1").

[0227] Figure 18 Illustrates an example of the GUI image 1800 according to certain embodiments, presenting information about re - baselining a skill extension to a new version of the base skill. Figure 18 The shown wizard - style dialog box is used to present information about the new base skill such as "Agent Bot 1.1". The information about the new base skill can include, for example, a brief description, a detailed description, and a sample utterance. The developer can determine whether to continue or return based on the information about the new base skill. If the developer decides to continue with the re - baselining, a list of differences between the previous base skill and the new base skill can be presented to the developer.

[0228] Figure 19Illustrates an example of a GUI image 1900 according to certain embodiments, showing a list of changes from a previous base skill (e.g., "Agent Bot 1.0") to a new base skill generated from the previous base skill (e.g., "Agent Bot 1.1"). In the illustrated example, the new base skill (e.g., "Agent Bot 1.1") may include two new intents (e.g., "Int Figure 1 " and "Int Figure 2 ") and an updated intent (e.g., "Intent A"). The new base skill may not include "Entity 1" which may be in the previous base skill. The developer can choose to proceed with re-basing or return. If the developer chooses to proceed with re-basing, a GUI for comparing the differences between all the skills involved in the re-basing process can be presented to the developer.

[0229] Figure 20 Illustrates an example of a GUI image 2000 according to certain embodiments for comparing the session flow content between all the skills involved in the re-basing process. For example, the GUI can be used to perform a comparison between any two of the previous base skill (e.g., "Agent Bot 1.0"), the new base skill (e.g., "Agent Bot 1.1"), the previously extended skill (e.g., "Extended Agent Bot 1.0") and the newly extended skill (e.g., "Extended Agent Bot 1.1"). Figure 20 Shows the comparison result between the metadata of the new base skill (e.g., "Agent Bot 1.1" shown in the right window 2010) and the metadata of the previously extended skill (e.g., "Extended Agent Bot 1.0" shown in the left window 2020). The "Preview" button 2030 can put the newly extended skill into a transitional state, which can allow the developer to check all the changes and run necessary tests before applying the re-basing action. If any changes are made to the skill while it is in the transitional state, the developer may encounter a dialog box asking the developer to create a new version. By clicking the "Cancel" button, the transitional state disappears and all the changes are reverted to the previous skill state.

[0230] Figure 21 Illustrates an example of a GUI image 2100 according to certain embodiments, showing a dialog box for confirming or canceling the re-basing of a skill extension. The developer can review the information related to the re-basing and choose to confirm or cancel the re-basing.

[0231] In some embodiments, a skill may include custom components. A published skill may implement custom components of the "embedded container" type. The user is able to edit metadata such as name and description and / or replace the existing package file of the custom component. An undo operation can also be supported on the above fields.

[0232] Figure 22 An example of a GUI image 2200 according to certain embodiments is illustrated, showing an example of a custom widget of the "embedded container" type. The GUI image 2200 may show, for example, the name, description, status, version, service type, package file, etc. of the "embedded container" widget. A developer can change the package file by selecting the button "Change" 2220. In the GUI image 2200, the "Enable Service" switch 2210 can be disabled and the "Delete" button can be omitted.

[0233] In some embodiments, a developer can also add a new service. Figure 23 An example of a GUI image 2300 for creating a new service according to certain embodiments is illustrated. The service can be an external service, an embedded container service, an Oracle Mobile Cloud Service, etc. A developer can specify a name, description, type, metadata location, verification information, etc. for the new service.

[0234] In some embodiments, an icon can be used to notify that a skill is an extended skill and / or signal whether a new base bot update is available. Figure 24A An example of a GUI image 2400 according to certain embodiments is illustrated, including an icon 2410 indicating that a skill is an extended skill and a new base skill update is available. If a skill has a pending rebase, a different icon must be used. For example, Figure 24B An example of a GUI image 2450 according to certain embodiments is illustrated, where the icon 2410 indicates that a skill is an extended skill and has a pending rebase.

[0235] Figure 25 is a simplified flowchart 2500 according to certain embodiments, illustrating an example of a process for generating an extended skill. At 2510, a computer system can obtain a first JSON file including metadata of a first application from a database storing metadata of multiple applications in a JOSN file. The multiple applications may include a chatbot application. In some embodiments, the multiple applications may be stored in the database as a compressed file. In some embodiments, the multiple applications may include two or more versions of the first application. In some embodiments, the metadata of the first application may include metadata of at least one of the following: configuring the first application, training the first application, testing the first application, or executing the first application. In some embodiments, the metadata of the first application may include metadata of at least one of the intent, entity, utterance, custom widget, or session flow of the first application.

[0236] At 2520, the computer system can receive, e.g., via a REST API, a modification to a first application to generate an extended application. The modification can include customizing and / or extending the first application to adapt the first application to specific features, processes, terms, cultures, etc. Customizing can include changing one or more properties of existing metadata artifacts or resources of the first application. Extending can include enhancing existing metadata or adding new artifacts or resources to the metadata of the first application.

[0237] At 2530, the computer system can determine the differences between the metadata of the extended application and the metadata of the first application. At 2540, the computer system can store a second JSON file in a database, where the second JSON file describes the changes made to the metadata of the first application to generate the metadata of the extended application. In some embodiments, the second JSON file can be compatible with at least two versions of the first application for extending the at least two versions in the first application. In some embodiments, the second JSON file can be incompatible with at least one of the two or more versions of the first application, and the database can include a third JSON file that is compatible with at least one of the two or more versions of the first application, where the second JSON file and the third JSON file can have different version numbers. In some embodiments, the second JSON file can include metadata of custom components in the extended application.

[0238] Figure 26 FIG. 2600 is a simplified flowchart according to certain embodiments, illustrating an example of a process for implementing extended skills. At 2610, the computer system can obtain a first application from a database storing multiple applications. The first application can include implementation data of the first application and a first JSON file that includes metadata associated with the first application. The multiple applications can include a chatbot application. The metadata associated with the first application can include metadata for at least one of the following: configuring the first application, training the first application, testing the first application, or executing the first application. The metadata associated with the first application can also include metadata for at least one of the intents, entities, utterances, custom components, or session flows of the first application.

[0239] At 2620, the computer system can obtain, from a database, a second JSON file associated with an extended application of a first application, where the second JSON file can describe changes to the first JSON file. In some embodiments, the database can include a first repository storing implementation data of the plurality of applications and a second repository storing the first JSON file and the second JSON file.

[0240] At 2630, the computer system can apply the changes to the first JSON file described in the second JSON file to generate metadata associated with the extended application. In some embodiments, applying the changes to the first JSON file described in the second JSON file can include: determining that the second JSON file conflicts with the first JSON file; and resolving one or more conflicts between the second JSON file and the first JSON file.

[0241] At 2640, the computer system can implement the extended application based on the implementation data of the first application and the metadata associated with the extended application. In some embodiments, the computer system can also obtain, from the database, a third JSON file (the third JSON file describes changes to the metadata associated with the extended application), apply the changes to the metadata associated with the extended application described in the third JSON file to generate metadata associated with a second extended application, and implement the second extended application based on the implementation data of the first application and the metadata associated with the second extended application. In some embodiments, the computer system can also obtain a second application from the database (the second application includes implementation data of the second application and a third JSON file including metadata associated with the second application), apply the changes to the first JSON file described in the second JSON file to the third JSON file to generate metadata associated with the second extended application, and implement the second extended application based on the implementation data of the second application and the metadata associated with the second extended application.

[0242] Although Figure 25 and Figure 26 the operations may be described as a sequential process, many of the operations can be performed in parallel or simultaneously. Additionally, the order of the operations can be rearranged. The process can have additional steps not included in the figure. Further, embodiments of the method can be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the associated tasks can be stored in a computer-readable medium such as a storage medium. One or more processors can perform the associated tasks.

[0243] Examples of the system

[0244] Figure 27 FIG. depicts a simplified diagram of a distributed system 2700 for implementing some embodiments. In the illustrated example, the distributed system 2700 includes one or more client computing devices 2702, 2704, 2706, and 2708 coupled to a server 2712 via one or more communication networks 2710. The client computing devices 2702, 2704, 2706, and 2708 can be configured to execute one or more applications.

[0245] In various examples, the server 2712 can be adapted to run one or more services or software applications that implement one or more embodiments described in the present disclosure. In certain examples, the server 2712 can also provide other services or software applications that can include non-virtual environments and virtual environments. In some examples, these services can be provided as web-based services or cloud services (such as under a software as a service (SaaS) model) to users of the client computing devices 2702, 2704, 2706, and / or 2708. Users operating the client computing devices 2702, 2704, 2706, and / or 2708 can then utilize one or more client applications to interact with the server 2712 to utilize the services provided by these components.

[0246] In Figure 27 the depicted configuration, the server 2712 can include one or more components 2718, 2720, and 2722 that implement the functions performed by the server 2712. These components can include software components that can be executed by one or more processors, hardware components, or combinations thereof. It should be understood that various different system configurations that may be different from the distributed system 2700 are possible. Thus, Figure 27 the example shown is one example of a distributed system for implementing an example system and is not intended to be limiting.

[0247] Users can use the client computing devices 2702, 2704, 2706, and / or 2708 to execute one or more applications, and the one or more applications can generate one or more storage requests that can then be serviced in accordance with the teachings of the present disclosure. The client device can provide an interface that enables the user of the client device to interact with the client device. The client device can also output information to the user through this interface. Although Figure 27 only four client computing devices are depicted, any number of client computing devices can be supported.

[0248] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptop computers, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, such as Google Chrome TM OS, etc., Linux or Linux-like operating systems), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM , Palm ). Portable handheld devices can include cellular phones, smartphones (e.g., ), tablet computers (e.g., ), personal digital assistants (PDAs), etc. Wearable devices can include Google head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., Microsoft gaming consoles with or without a gesture input device, Sony systems, various gaming systems provided by and others), etc. Client devices can be capable of executing various different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications) and can use various communication protocols.

[0249] (Multiple) communication networks 2710 can be any type of network familiar to those skilled in the art that supports data communication using any of a variety of available protocols, the available protocols including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (System Network Architecture), IPX (Internetwork Packet Exchange), etc. By way of example only, (multiple) communication networks 2710 can be a local area network (LAN), an Ethernet-based network, token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol family, and / or a network operating on any other wireless protocol) and / or any combination of these networks and / or other networks.

[0250] The server 2712 can be constituted by: one or more general-purpose computers, dedicated server computers (by way of example, including PC (personal computer) servers, servers, midrange servers, mainframes, rack servers, etc.), a server farm, a server cluster, or any other suitable arrangement and / or combination. The server 2712 can include one or more virtual machines running a virtual operating system or other computing architectures involving virtualization (such as one or more flexible pools of logical storage devices that can be virtualized to maintain the virtual storage devices of the server. In various examples, the server 2712 can be adapted to run one or more services or software applications that provide the functions described in the foregoing disclosure.

[0251] The computing system in the server 2712 can run one or more operating systems, and the one or more operating systems include any one of the operating systems discussed above and any commercially available server operating system. The server 2712 can also run any one of various additional server applications and / or middleware applications, including HTTP (HyperText Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, servers, database servers, etc. Exemplary database servers include, but are not limited to, those commercially available from (International Business Machines Corporation), etc.

[0252] In some embodiments, the server 2712 can include one or more applications to analyze and combine data feeds and / or event updates received from users of the client computing devices 2702, 2704, 2706, and 2708. By way of example, the data feeds and / or event updates can include, but are not limited to feeds, updates, or real-time updates received from one or more third-party information sources and continuous data streams, and the real-time updates can include real-time events related to sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc. The server 2712 can also include one or more applications to display data feeds and / or real-time events through one or more display devices of the client computing devices 2702, 2704, 2706, and 2708.

[0253] The distributed system 2700 may also include one or more data repositories 2714, 2716. In some examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 2714, 2716 may be used to store information such as information related to storing virtual machines, information mapping application IDs to applications that select storage virtual machines, and other information used by the server 2712 when performing an authentication function. The data repositories 2714, 2716 may reside in various locations. For example, the data repository used by the server 2712 may be local to the server 2712 or may be remote from the server 2712 and communicate with the server 2712 via a network-based or dedicated connection. The data repositories 2714, 2716 may be of different types. In some examples, the data repository used by the server 2712 may be a database, such as a relational database, such as those provided by Oracle and other vendors. One or more of these databases may be adapted to implement storage, update, and retrieval of data to and from the database in response to commands in SQL format.

[0254] In some examples, one or more of the data repositories 2714, 2716 may also be used by applications to store application data. The data repositories used by applications may be of different types, such as, for example, key-value storage repositories, object storage repositories, or general storage repositories supported by a file system.

[0255] In some examples, the functions described in this disclosure may be provided as services via a cloud environment. Figure 28 is a simplified block diagram of a cloud-based system environment system 2800 for implementing some embodiments. In the cloud-based system environment system 2800, according to some examples, various services may be provided as cloud services. In Figure 28 the depicted example, the cloud infrastructure system 2802 may provide one or more cloud services that may be requested by a user using one or more client computing devices 2804, 2806, and 2808. The cloud infrastructure system 2802 may include one or more computers and / or servers, which may include those described above for the server 1612. The computers in the cloud infrastructure system 2802 may be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0256] (Multiple) networks 2810 can facilitate data communication and exchange between client computing devices 2804, 2806, and 2808 and cloud infrastructure system 2802. (Multiple) networks 2810 can include one or more networks. The networks can be of the same or different types. (Multiple) networks 2810 can support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0257] Figure 28 The example depicted is only one example of a cloud infrastructure system and is not intended to be restrictive. It should be understood that in some other examples, cloud infrastructure system 2802 can have more or fewer components than Figure 28 those depicted, can combine two or more components, or can have a different component configuration or arrangement. For example, although Figure 28 three client computing devices are depicted, in alternative examples, any number of client computing devices can be supported.

[0258] The term cloud service is generally used to refer to services that are made available to a user on demand via a communication network such as the Internet through a service provider's system (e.g., cloud infrastructure system 2802). Generally, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Thus, a customer can enable itself to utilize the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system can host an application, and a user can order and use the application on demand via the Internet without the user having to purchase the infrastructure resources for executing the application. Cloud services are designed to provide easy and scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle of Redwood Shores, California offers several cloud services such as middleware services, database services, Java cloud services, and other services.

[0259] In some examples, cloud infrastructure system 2802 can provide one or more cloud services using different models (such as under a software as a service (SaaS) model, a platform as a service (PaaS) model, an infrastructure as a service (IaaS) model, and other models including hybrid service models). Cloud infrastructure system 2802 can include a set of applications, middleware, databases, and other resources that implement the provisioning of various cloud services.

[0260] The SaaS model enables applications or software to be delivered as a service to customers over a communication network such as the Internet, without the customer having to purchase the hardware or software of the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by the cloud infrastructure system 2802. Oracle Examples of SaaS services provided by Oracle include, but are not limited to, various services for human resources / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytics services, social applications, and others.

[0261] The IaaS model is typically used to provide infrastructure resources (e.g., servers, storage devices, hardware, and networking resources) as cloud services to customers to provide elastic computing and storage capabilities. Provided by Oracle Provides various IaaS services.

[0262] The PaaS model is typically used to provide platform and environmental resources as a service that enables customers to develop, run, and manage applications and services without the customer having to procure, build, or maintain such resources. Provided by Oracle Examples of PaaS services provided by Oracle include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and others.

[0263] Cloud services are typically provided in an on-demand self-service basis, subscription-based, elastically scalable, reliable, highly available, and secure manner. For example, a customer can order one or more services provided by the cloud infrastructure system 2802 through a subscription order. Then, the cloud infrastructure system 2802 performs processing to provide the services requested in the customer's subscription order. For example, a user can request the cloud infrastructure system to register an application as described above and provide services to the application according to the specified requirements of the application. The cloud infrastructure system 2802 can be configured to provide one or even multiple cloud services.

[0264] The cloud infrastructure system 2802 can provide cloud services through different deployment models. In the public cloud model, the cloud infrastructure system 2802 can be owned by a third-party cloud service provider, and cloud services are provided to any general public customers, where the customers can be individuals or enterprises. In some other examples, in the private cloud model, the cloud infrastructure system 2802 can be operated within an organization (e.g., within an enterprise organization), and services are provided to customers within the organization. For example, the customers can be various departments of an enterprise such as the human resources department, payroll department, etc. or even individuals within the enterprise. In some other examples, in the community cloud model, the cloud infrastructure system 2802 and the provided services can be shared by several organizations in the relevant community. Various other models such as a hybrid of the models mentioned above can also be used.

[0265] The client computing devices 2804, 2806, and 2808 can be of different types and can be capable of operating one or more client applications. A user can use the client device to interact with the cloud infrastructure system 2802, such as requesting services provided by the cloud infrastructure system 2802. For example, a user can use the client device to request the authentication-related services described in this disclosure.

[0266] In some examples, the processing for providing services performed by the cloud infrastructure system 2802 can involve big data analysis. This analysis can involve using, analyzing, and manipulating large datasets to detect and visualize various trends, behaviors, relationships, etc. in the data. This analysis can be performed by one or more processors, thus potentially processing data in parallel, performing simulations using the data, etc. For example, big data analysis can be performed by the cloud infrastructure system 2802 to determine which storage virtual machine to select for a specific application based on the application's stated authentication-related requirements. The data used for this analysis can include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data chunks (binary large objects)).

[0267] As Figure 28 depicted in the examples of , the cloud infrastructure system 2802 can include infrastructure resources 2830 that are used to facilitate the provision of various cloud services provided by the cloud infrastructure system 2802. The infrastructure resources 2830 can include, for example, processing resources, storage devices or memory resources, networking resources, etc. In some examples, the storage virtual machines available for serving storage requests from applications can be part of the cloud infrastructure system 2802. In other examples, the storage virtual machines can be part of different systems.

[0268] In some examples, to facilitate the efficient provisioning of these resources for supporting the various cloud services provided by the cloud infrastructure system 2802 for different customers, the resources can be bundled into resource groups or resource modules (also referred to as "pods"). Each resource module or pod can include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods can be pre-provisioned for different types of cloud services. For example, a first set of pods can be provisioned for database services, a second set of pods can be provisioned for Java services (the second set of pods can include a resource combination different from the pods in the first set), and so on. For some services, the resources allocated for provisioning the services can be shared among the services.

[0269] The cloud infrastructure system 2802 itself can internally use services 2832 that are shared by different components of the cloud infrastructure system 2802 and facilitate the cloud infrastructure system 2802 in provisioning services. These internal shared services can include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for enabling cloud support, email services, notification services, file transfer services, etc.

[0270] The cloud infrastructure system 2802 can include multiple subsystems. These subsystems can be implemented in software or hardware or a combination thereof. As Figure 28 depicted, the subsystems can include a user interface subsystem 2812 that enables users or customers of the cloud infrastructure system 2802 to interact with the cloud infrastructure system 2802. The user interface subsystem 2812 can include various different interfaces, such as a web interface 2814, an online store interface 2816 (wherein cloud services provided by the cloud infrastructure system 2802 are advertised and customers can purchase them), and other interfaces 2818. For example, a customer can use a client device to request (service request 2834) one or more services provided by the cloud infrastructure system 2802 using one or more of the interfaces 2814, 2816, and 2818. For example, a customer can access an online store, browse the cloud services provided by the cloud infrastructure system 2802, and place a subscription order for one or more services provided by the cloud infrastructure system 2802 that the customer wishes to subscribe to. The service request can include information identifying the customer and the one or more services the customer expects to subscribe to. For example, a customer can place a subscription order for a service provided by the cloud infrastructure system 2802. As part of the order, the customer can provide information identifying the application for which the service is to be provided and one or more credentials of the application.

[0271] In some examples (such as Figure 28In the example depicted, the cloud infrastructure system 2802 can include an Order Management Subsystem (OMS) 2820 configured to process new orders. As part of this processing, the OMS 2820 can be configured to: create an account for the customer (if not already created); receive from the customer the billing and / or charging information to be used to bill the customer for the requested services; verify the customer information; after verification, book an order for the customer; and orchestrate various workflows to prepare the order for provisioning.

[0272] Once properly verified, the OMS 2820 can then call an Order Provisioning Subsystem (OPS) 2824 configured to provision resources for the order, including processing resources, memory resources, and networking resources. Provisioning can include allocating resources for the order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned can depend on the type of cloud service the customer has ordered. For example, according to one workflow, the OPS 2824 can be configured to determine the specific cloud service being requested and identify the number of clusters that may have been pre-configured for the specific cloud service. The number of clusters allocated for the order can depend on the size / volume / tier / scope of the service requested. For example, the number of clusters to be allocated can be determined based on the number of users the service is to support, the duration of the service being requested, etc. The allocated clusters can then be customized for the specific requesting customer for providing the requested service.

[0273] In some examples, the setup phase processing as described above can be performed by the cloud infrastructure system 2802 as part of the provisioning process. The cloud infrastructure system 2802 can generate an application ID and select a storage virtual machine for the application from a storage virtual machine provided by the cloud infrastructure system 2802 itself or from a storage virtual machine provided by other systems other than the cloud infrastructure system 2802.

[0274] The cloud infrastructure system 2802 can send a response or notification 2844 to the requesting customer to indicate that the requested service is now ready for use. In some instances, information (e.g., a link) enabling the customer to start using and leveraging the benefits of the requested service can be sent to the customer. In some examples, for the customer requesting the service, the response can include the application ID generated by the cloud infrastructure system 2802 and information identifying the virtual machine selected by the cloud infrastructure system 2802 for the application corresponding to the application ID.

[0275] The cloud infrastructure system 2802 can provide services to multiple customers. For each customer, the cloud infrastructure system 2802 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the orders, and providing the requested services to the customer. The cloud infrastructure system 2802 can also collect usage statistics on the customer's use of the subscribed services. For example, statistics such as the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and system downtime can be collected. This usage information can be used to bill the customer. Billing can be completed, for example, on a monthly basis.

[0276] The cloud infrastructure system 2802 can provide services to multiple customers in parallel. The cloud infrastructure system 2802 can store information about these customers (which may include proprietary information). In some examples, the cloud infrastructure system 2802 includes an identity management subsystem (IMS) 2828 that is configured to manage customer information and provide separation of the managed information such that information related to one customer cannot be accessed by another customer. The IMS 2828 can be configured to provide various security-related services such as identity services, which can include, for example, information access management, authentication and authorization services, services for managing customer identities and roles, and related capabilities.

[0277] Figure 29 An example of a computer system 2900 for implementing some embodiments is illustrated. In some examples, the computer system 2900 can be used to implement any of the application systems, access management systems, systems within a data center, and various servers and computer systems described above. As Figure 29 shown, the computer system 2900 includes various subsystems, which include a processing subsystem 2904 that communicates with a plurality of other subsystems via a bus subsystem 2902. These other subsystems can include a processing acceleration unit 2906, an I / O subsystem 2908, a storage subsystem 2918, and a communication subsystem 2924. The storage subsystem 2918 can include a non-transitory computer-readable storage medium, which includes a computer-readable storage medium 2922 and a system memory 2910.

[0278] The bus subsystem 2902 provides the mechanism for enabling the various components and subsystems of the computer system 2900 to communicate with each other as expected. Although the bus subsystem 2902 is schematically shown as a single bus, alternative examples of bus subsystems can utilize multiple buses. The bus subsystem 2902 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus, etc. using any of a variety of bus architectures. For example, such architectures can include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus (the PCI bus can be implemented as a Mezzanine bus manufactured to the IEEE P1386.1 standard), etc.

[0279] The processing subsystem 2904 controls the operation of the computer system 2900 and can include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processors can include single-core processors or multi-core processors. The processing resources of the computer system 2900 can be organized into one or more processing units 2932, 2934, etc. The processing units can include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 2904 can include one or more dedicated coprocessors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units in the processing subsystem 2904 can be implemented using custom circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0280] In some examples, the processing units in the processing subsystem 2904 can execute instructions stored in the system memory 2910 or on the computer-readable storage medium 2922. In various examples, the processing units can execute various programs or code instructions and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code to be executed can reside in the system memory 2910 and / or on the computer-readable storage medium 2922 (potentially including residing on one or more storage devices). Through suitable programming, the processing subsystem 2904 can provide the various functions described above. In instances where the computer system 2900 is executing one or more virtual machines, one or more processing units can be allocated to each virtual machine.

[0281] In some examples, a processing acceleration unit 2906 may optionally be provided to perform customized processing or to offload some of the processing performed by the processing subsystem 2904, thereby accelerating the overall processing performed by the computer system 2900.

[0282] The I / O subsystem 2908 may include devices and mechanisms for inputting information to the computer system 2900 and / or for outputting information from or through the computer system 2900. Generally, the term input device is intended to include all possible types of devices and mechanisms for inputting information to the computer system 2900. User interface input devices may include, for example, a keyboard, pointing devices such as a mouse or trackball, a touchpad or touchscreen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device having a voice command recognition system, a microphone, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices such as Microsoft Motion Sensors, Microsoft Game Controllers, devices that provide an interface for receiving input using gestures and voice commands. User interface input devices may also include eye gesture recognition devices such as Google ) that detect eye activity from a user (e.g., "blinking" when taking a picture and / or making a menu selection) and transform the eye gesture into an input to the input device (e.g., Google ) Blink Detector. Additionally, user interface input devices may include voice recognition sensing devices that enable a user to interact with a voice recognition system (e.g., ) Navigator) via voice commands.

[0283] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphics tablets, as well as audio / video devices (such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye gaze tracking devices). Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, and the like.

[0284] In general, the term output device is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2900 to a user or another computer. User interface output devices can include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems can be cathode ray tubes (CRTs), flat panel devices (such as flat panel devices using liquid crystal displays (LCDs) or plasma displays), projection devices, touchscreens, etc. For example, user interface output devices can include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headsets, automotive navigation systems, plotters, sound output devices, and modems.

[0285] The storage subsystem 2918 provides a repository or data store for storing the information and data used by the computer system 2900. The storage subsystem 2918 provides a tangible non-transitory computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some examples. The storage subsystem 2918 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 2904, provides the functionality described above. The software can be executed by one or more processing units of the processing subsystem 2904. The storage subsystem 2918 can also provide authentication in accordance with the teachings of the present disclosure.

[0286] The storage subsystem 2918 can include one or more non-transitory memory devices, the one or more non-transitory memory devices including volatile memory devices and non-volatile memory devices. As Figure 29 shown, the storage subsystem 2918 includes system memory 2910 and computer-readable storage medium 2922. System memory 2910 can include multiple memories, the multiple memories including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, the basic input / output system (BIOS), which contains basic routines that help transfer information between elements within the computer system 2900 during startup, can typically be stored in the ROM. The RAM typically contains the data and / or program modules currently being operated on and executed by the processing subsystem 2904. In some embodiments, the system memory 2910 can include various different types of memories such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0287] By way of example and not limitation, as Figure 29As depicted, the system memory 2910 can load the executing application 2912 (the application can include various applications such as a web browser, a middleware application, a relational database management system (RDBMS), etc.), program data 2914, and an operating system 2916. By way of example, the operating system 2916 can include various versions of Microsoft Apple and / or Linux operating systems, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or mobile operating systems such as iOS, Phone, OS, OS, OS operating systems, and other operating systems.

[0288] The computer-readable storage medium 2922 can store programming and data constructs that provide some example functionality. The computer-readable storage medium 2922 can provide storage for the computer system 2900 for computer-readable instructions, data structures, program modules, and other data. Software (programs, code modules, instructions) that provides the functionality described above when executed by the processing subsystem 2904 can be stored in the storage subsystem 2918. By way of example, the computer-readable storage medium 2922 can include non-volatile memories such as hard disk drives, disk drives, optical disc drives (such as CD ROM, DVD, discs or other optical media). The computer-readable storage medium 2922 can include but is not limited to drives, flash memory cards, universal serial bus (USB) flash memory drives, secure digital (SD) cards, DVD discs, digital video tapes, etc. The computer-readable storage medium 2922 can also include non-volatile memory-based SSDs such as flash memory-based solid state drives (SSDs), enterprise-class flash memory drives, solid state ROMs, volatile memory-based SSDs such as solid state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs.

[0289] In some examples, the storage subsystem 2918 can also include a computer-readable storage medium reader 2920 that can be further connected to the computer-readable storage medium 2922. The reader 2920 can receive data from memory devices such as discs, flash memory drives, etc. and is configured to read data from the memory devices.

[0290] In some examples, computer system 2900 may support virtualization technologies, including but not limited to virtualization of processing and memory resources. For example, computer system 2900 may provide support for executing one or more virtual machines. In some examples, computer system 2900 may execute programs such as a hypervisor that facilitates the configuration and management of virtual machines. Each virtual machine may be allocated memory, computing (e.g., processors, cores), I / O, and networking resources. Each virtual machine generally runs independently of other virtual machines. A virtual machine typically runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines running on computer system 2900. Thus, multiple operating systems may potentially be run simultaneously by computer system 2900.

[0291] Communication subsystem 2924 provides an interface to other computer systems and networks. Communication subsystem 2924 serves as an interface for receiving data from other systems and transmitting data from computer system 2900 to other systems. For example, communication subsystem 2924 may enable computer system 2900 to establish a communication channel to one or more client devices via the Internet for receiving information from and / or sending information to the client devices. For example, when computer system 2900 is used to implement Figure 1 the depicted robotic system 120, the communication subsystem may be used to communicate with the application system and the system that executes the stored virtual machine selected for the application.

[0292] Communication subsystem 2924 may support both wired communication protocols and / or wireless communication protocols. In some examples, communication subsystem 2924 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular phone technologies, advanced data network technologies such as 3G, 4G, EDGE (Enhanced Data Rates for Global Evolution), etc. or 5G, WiFi (IEEE 802.XX family standards or other mobile communication technologies or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some examples, in addition to or instead of the wireless interface, communication subsystem 2924 may provide wired network connectivity (e.g., Ethernet).

[0293] Communication subsystem 2924 may receive and transmit various forms of data. In some examples, in addition to other forms, communication subsystem 2924 may also receive input communications in the form of structured and / or unstructured data feeds 2926, event streams 2928, event updates 2930, etc. For example, communication subsystem 2924 may be configured to receive (or send) data feeds 2926 from users of social media networks and / or other communication services in real time, such as feeds, updates, web feeds (such as Rich Site Summary (RSS) feeds), and / or real-time updates from one or more third-party information sources.

[0294] In some examples, the communication subsystem 2924 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2928 and / or event updates 2930 of real-time events (essentially continuous or unbounded without an explicit end). Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, etc.

[0295] The communication subsystem 2924 may also be configured to communicate data from the computer system 2900 to other computer systems or networks. The data may be communicated in various different forms, such as structured and / or unstructured data feeds 2926, event streams 2928, event updates 2930, etc., to one or more databases that may communicate with one or more stream data source computers coupled to the computer system 2900.

[0296] The computer system 2900 may be one of various types, including handheld portable devices (e.g., cellular phones, computing tablet computers, PDAs), wearable devices (e.g., Google head-mounted displays), personal computers, workstations, mainframes, self-service terminals, server racks, or any other data processing system. Due to the ever-changing nature of computers and networks, the description of the Figure 29 depicted computer system 2900 is intended to be only a specific example. Many other configurations with more or fewer components than the Figure 29 depicted system are possible. Based on the present disclosure and the teachings provided herein, those of ordinary skill in the art should understand other ways and / or methods for implementing various examples.

[0297] Although specific examples have been described, various modifications, changes, alternative constructions, and equivalents are possible. Examples are not limited to operating in certain specific data processing environments, but freely operate in multiple data processing environments. Additionally, although certain examples have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be restrictive. While some flowcharts depict operations as sequential processes, many operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. The process may have additional steps not included in the figures. The various features and aspects of the examples described above may be used alone or in combination.

[0298] Further, although certain examples have been described using a specific combination of hardware and software, it should be recognized that other combinations of hardware and software are possible. Some examples may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented in any combination on the same processor or different processors.

[0299] In cases where a device, system, component, or module is described as being configured to perform certain operations or functions, such configuration may be accomplished, for example, by designing an electronic circuit to perform the operations, by programming a programmable electronic circuit (such as a microprocessor) to perform the operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0300] Specific details are given in this disclosure to provide a thorough understanding of the examples. However, the examples may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail to avoid obscuring the examples. This description only provides examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the previous description of the examples will provide those skilled in the art with an enabling description for implementing the various examples. Various changes may be made to the function and arrangement of the elements.

[0301] Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and changes may be made without departing from the broader spirit and scope set forth in the claims. Thus, while specific examples have been described, these examples are not intended to be restrictive. Various modifications and equivalents are within the scope of the following claims.

[0302] In the foregoing specification, aspects of the present disclosure have been described with reference to specific examples of the present disclosure, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used alone or in combination. Further, the examples may be utilized in any number of environments and application scenarios other than those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

[0303] In the foregoing description, for purposes of illustration, the method has been described in a particular order. It should be appreciated that in alternative examples, the method may be performed in an order different from that described. It should also be appreciated that the methods described above may be performed by hardware components or may be embodied in a sequence of machine-executable instructions that may be used to cause a machine, such as a general or special-purpose processor or logic circuitry programmed with the instructions, to perform the method. These machine-executable instructions may be stored on one or more machine-readable media, such as a CD-ROM or other type of optical disk, a floppy disk, a ROM, a RAM, an EPROM, an EEPROM, a magnetic or optical card, a flash memory, or other types of machine-readable media suitable for storing electronic instructions. Alternatively, the methods may be performed by a combination of hardware and software.

[0304] In the case where a component is described as being configured to perform certain operations, such configuration may be accomplished, for example, by designing electronic circuitry or other hardware to perform the operations, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operations, or any combination thereof.

[0305] Although illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept may be embodied and employed in other ways that are different, and the appended claims are intended to be construed to include such variations, except as limited by the prior art.

Claims

1. A computer-implemented method, comprising: obtaining a first JSON file from a database storing metadata for a plurality of skills in JavaScript Object Notation (JSON) files, the first JSON file including metadata for a base skill; receiving, via an application programming interface (API), a modification to the base skill to generate an extended skill, where the modification includes changes to existing metadata artifacts or resources of the base skill, and where the metadata artifacts or resources are top-level skill definitions, intents, entities, utterances, custom components, dialog flows, resource bundles, or combinations thereof associated with the base skill; determining a difference between the metadata of the extended skill and the metadata of the base skill based on the changes to the metadata artifacts or resources of the base skill; generating a second JSON file based on the difference between the metadata of the base skill and the metadata of the extended skill, the second JSON file describing the changes to be made to the metadata of the base skill to generate the metadata of the extended skill, where the second JSON file represents only the difference between the metadata of the base skill and the metadata of the extended skill; storing the second JSON file separately from the first JSON file in the database; using the first JSON file and the second JSON file to modify the metadata of the base skill by applying the changes described in the second JSON file to the metadata of the base skill to generate the metadata of the extended skill; and implementing the extended skill based on the implementation data of the base skill and the metadata generated for the extended skill.

2. The computer-implemented method according to claim 1, wherein, the plurality of skills are chatbot applications.

3. The computer-implemented method according to claim 1, wherein, the JSON files of the plurality of skills are stored in the database as compressed files.

4. The computer-implemented method according to claim 1, wherein, the plurality of skills include two or more versions of the base skill.

5. The computer-implemented method according to claim 4, wherein, the second JSON file is compatible with at least two versions of the base skill for extending the at least two versions of the base skill.

6. The computer-implemented method according to claim 4, wherein: the second JSON file is not compatible with at least one of the two or more versions of the base skill; and the database includes a third JSON file compatible with at least one of the two or more versions of the base skill, where the second JSON file and the third JSON file have different version numbers.

7. The computer-implemented method according to claim 1, wherein, The metadata of the base skill includes metadata for at least one of the following: configuring the base skill, training the base skill, testing the base skill, or running the base skill.

8. The computer-implemented method according to claim 1, wherein, the metadata of the base skill includes metadata for at least one of the following: the intent of the base skill, the entity of the base skill, the utterance of the base skill, the custom component of the base skill, or the conversation flow of the base skill.

9. The computer-implemented method according to claim 1, wherein, the second JSON file includes metadata of a custom component in the extended skill.

10. The computer-implemented method according to claim 1, wherein, modifying the base skill to generate the extended skill includes modifying the base skill via a Representational State Transfer Application Programming Interface (REST API).

11. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations including the following: obtaining a first JSON file from a database that stores metadata for a plurality of skills in JavaScript Object Notation (JSON) files, the first JSON file including metadata of a base skill; receiving, via an application programming interface (API), a modification to the base skill to generate an extended skill, wherein the modification includes a change to an existing metadata artifact or resource of the base skill, and wherein, the metadata artifact or resource is a top-level skill definition, intent, entity, utterance, custom component, conversation flow, resource bundle, or a combination thereof associated with the base skill; determining a difference between the metadata of the extended skill and the metadata of the base skill based on the change to the metadata artifact or resource of the base skill; generating a second JSON file based on the difference between the metadata of the base skill and the metadata of the extended skill, the second JSON file describing the changes to the metadata of the base skill for generating the metadata of the extended skill, wherein the second JSON file represents only the difference between the metadata of the base skill and the metadata of the extended skill; storing the second JSON file separately from the first JSON file in the database; using the first JSON file and the second JSON file to modify the metadata of the base skill by applying the changes described in the second JSON file to the metadata of the base skill to generate the metadata of the extended skill; and implementing the extended skill based on the implementation data of the base skill and the metadata generated for the extended skill.

12. The non-transitory computer-readable storage medium according to claim 11, wherein, The plurality of skills includes two or more versions of the base skill.

13. The non-transitory computer-readable storage medium according to claim 12, wherein, the second JSON file is compatible with at least two versions of the base skill for extending the at least two versions of the base skill.

14. The non-transitory computer-readable storage medium according to claim 12, wherein: the second JSON file is not compatible with at least one of the two or more versions of the base skill; and the database includes a third JSON file that is compatible with at least one of the two or more versions of the base skill, wherein the second JSON file and the third JSON file have different version numbers.

15. A computer system, comprising: one or more processors; and a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including the following: Obtain a first JSON file from a database that stores metadata for a plurality of skills in JavaScript Object Notation (JSON) files, the first JSON file including metadata for a base skill; Receive, via an application programming interface (API), a modification to the base skill to generate an extended skill, where the modification includes a change to an existing metadata artifact or resource of the base skill, and where the metadata artifact or resource is a top-level skill definition, intent, entity, utterance, custom component, dialog flow, resource bundle, or a combination thereof associated with the base skill; Determine a difference between the metadata of the extended skill and the metadata of the base skill based on the change to the metadata artifact or resource of the base skill; Generate a second JSON file based on the difference between the metadata of the base skill and the metadata of the extended skill, the second JSON file describing the changes to be made to the metadata of the base skill to generate the metadata of the extended skill, where the second JSON file represents only the difference between the metadata of the base skill and the metadata of the extended skill; Store the second JSON file separately from the first JSON file in the database; Use the first JSON file and the second JSON file to modify the metadata of the base skill by applying the changes described in the second JSON file to the metadata of the base skill to generate the metadata of the extended skill; and Implement the extended skill based on the implementation data of the base skill and the metadata generated for the extended skill.

16. The computer system according to claim 15, wherein, the plurality of skills includes two or more versions of the base skill.

17. The computer system according to claim 16, wherein, The second JSON file is compatible with at least two versions of the base skill for extending the at least two versions of the base skill.

18. The computer system according to claim 16, wherein: the second JSON file is not compatible with at least one of the two or more versions of the base skill; and the database includes a third JSON file that is compatible with at least one of the two or more versions of the base skill, wherein the second JSON file and the third JSON file have different version numbers.

Images