Robotic scalability infrastructure

Abstract

The invention relates to a robotic scalability infrastructure. The present disclosure generally relates to techniques for extending or customizing underlying skills (e.g., chat robots). In accordance with certain embodiments, a robotic extension infrastructure is provided to facilitate customization and / or extension of underlying skills, separately track the underlying skills and different versions of the extension, apply the extension to different versions of the underlying skills, or apply different versions of the extension to the same underlying skills. The extension to the base skill includes a JSON extension that indicates a modification to metadata of the base skill. A base skill (e.g., a base skill downloaded from a skill store) may be extended or customized by applying a JSON extension that describes changes to be made to the metadata of the base skill.

Description

[0001] This application is a divisional application based on the Chinese invention patent application with application date of April 23, 2020, application number 2020103292990, and invention name “Robotic Scalability Infrastructure”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This patent application is a continuation of U.S. patent application No. 16 / 827,191, filed on March 23, 2020, which claims the benefit of and priority to U.S. provisional patent application serial number 62 / 839,585, entitled “Bot Extensibility Infrastructure,” filed on April 26, 2019, which is assigned to the assignee of the present application and is incorporated herein by reference in its entirety for all purposes.

[0004] Copyright Notice

[0005] A portion of the disclosure of this patent application contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent application or patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0006] The present disclosure relates generally to chatbots, and more particularly to an infrastructure for extending a base chatbot that may be developed by another party and available in a chatbot store where different chatbots and / or different versions of chatbots may be stored for download. Background Art

[0007] In order to obtain an immediate response, many users around the world use instant messaging or chat platforms. Organizations often use these instant messaging or chat platforms to conduct real-time conversations with customers (or end users). However, hiring service personnel to communicate with customers or end users in real time would be very expensive for organizations. Therefore, chatbots (hereinafter also referred to as "bots" or "skills") have begun to be developed to conduct conversations with end users, especially over the Internet. End users or customers can communicate with the bots through messaging applications that the end users have installed and are using. Intelligent bots (usually powered by artificial intelligence (AI)) can communicate more intelligently and contextually in real-time conversations and therefore can allow for more natural conversations between the bots and end users to improve the conversation experience. Instead of the end user learning a fixed set of keywords or commands that the bot knows how to respond to, the intelligent bots are able to understand the end user's intentions based on the user's utterances in natural language and respond accordingly.

[0008] However, chatbots are difficult to build because these automated solutions require specific domain knowledge and the application of certain techniques that may only be within the capabilities of professional developers. As part of building such a chatbot, developers can first understand the needs of the business and end users. Developers can then analyze and make decisions related to, for example: selecting a dataset to analyze; preparing the input dataset for analysis (e.g., cleaning, extracting, formatting, and / or transforming the data before analysis, performing data feature engineering, etc.); identifying appropriate machine learning (ML) techniques or ML model(s) for performing the analysis; and refining the techniques or models to improve results / performance based on feedback. The task of identifying an appropriate model can include developing multiple models (possibly in parallel), iteratively testing and experimenting with these models, and then identifying a specific model (or models) to use. Furthermore, supervised learning-based solutions typically involve a training phase, followed by an application (i.e., inference) phase, and an iterative cycle between the training and application phases. Developers can be responsible for carefully implementing and monitoring these phases to achieve an optimal solution.

[0009] Therefore, building a suitable chatbot can be complex and time-consuming, and developers can play a central role in developing such solutions. However, the number of developers with the necessary skills to build such chatbots is very limited. Often, developers must also be experts in the specific domain corresponding to the problem being solved, such as business analysts. The number of such developers who are also domain experts is extremely limited. This lack of expert developers prevents many organizations and individuals from developing AI- or ML-based chatbots. This makes AI- or ML-based chatbot solutions expensive and / or inaccessible to many individuals and organizations. Summary of the Invention

[0010] The technology disclosed herein generally relates to chatbots. More specifically, and without limitation, the technology disclosed herein relates to an infrastructure for extending a base chatbot that can be developed by another party and available in a chatbot store, where different chatbots and / or different versions of chatbots can be stored for download. Various inventive embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, codes, or instructions executable by one or more processors, and the like.

[0011] According to some embodiments, a computer-implemented method may include: obtaining metadata associated with a base application (e.g., a base robot); modifying the metadata associated with the base application to generate an extended application (e.g., an extended robot); determining differences between the extended application and the base application; and storing the differences between the extended application and the base application as a JavaScript Object Notation (JSON) file in a database. The JSON file in the database may be referred to as a robot extension (or skill extension) and may be downloaded to implement the extended application by modifying the base application according to the JSON file.

[0012] In certain embodiments, a computer-implemented method may include obtaining a base application including implementation data and a first JSON file including metadata associated with the base application; and obtaining a second JSON file associated with an application that is an extension of the base application. The second JSON file describes changes to be made to the first JSON file and is referred to as a bot extension or skill extension. The computer-implemented method may further include making the changes described in the second JSON file to the first JSON file to generate metadata associated with the extended application. The extended application may then be implemented based on the implementation data of the base application and the metadata associated with the first JSON file modified based on the second JSON file.

[0013] According to certain embodiments, a computer-implemented method may include: obtaining a first JavaScript Object Notation (JSON) file from a database that stores metadata for multiple applications in JSON files, the first JSON file including metadata for a first application; receiving modifications to the first application to generate an extended application; determining differences between the metadata of the extended application and the metadata of the first application; and storing a second JSON file in the database, the second JSON file describing changes to be made to the metadata of the first application to generate the metadata of the extended application.

[0014] In some embodiments, the multiple applications may be chatbot applications. The JSON files of the multiple applications may be stored in the database as compressed files. In some embodiments, the multiple applications may include two or more versions of the first application. In some embodiments, the second JSON file may be compatible with two or more versions of the first application for extending the two or more versions of the first application. In some embodiments, the second JSON file may be incompatible with at least one of the two or more versions of the first application, and the database may include a third JSON file compatible with at least one of the two or more versions of the first application, wherein the second JSON file and the third JSON file may have different version numbers.

[0015] In some embodiments, the metadata of the first application may include metadata for at least one of: configuring the first application, training the first application, testing the first application, or executing the first application. In some embodiments, the metadata of the first application may include metadata for at least one of an intent, an entity, an utterance, a custom component, or a conversation flow of the first application. The second JSON file may include metadata for custom components in the extended application. In some embodiments, modifying the first application to generate the extended application may include modifying the first application via a Representational State Transfer Application Programming Interface (REST API).

[0016] According to certain embodiments, a computer-implemented method may include obtaining a first application from a database storing a plurality of applications, wherein the first application may include implementation data of the first application and a first JavaScript Object Notation (JSON) file including metadata associated with the first application. The computer-implemented method may also include obtaining a second JSON file associated with an application that is an extension of the first application from the database, wherein the second JSON file describes changes to the first JSON file. The computer-implemented method may further include: applying the changes to the first JSON file described in the second JSON file to generate metadata associated with the extended application; and implementing the extended application based on the implementation data of the first application and the metadata associated with the extended application.

[0017] In some embodiments, the plurality of applications and the extended applications may be chatbot applications. The metadata associated with the first application may include metadata for at least one of: configuring the first application, training the first application, testing the first application, or executing the first application. The metadata associated with the first application may include metadata for at least one of an intent, an entity, an utterance, a custom component, or a conversation flow of the first application. Applying the changes to the first JSON file described in the second JSON file may include: determining that the second JSON file conflicts with the first JSON file; and resolving one or more conflicts between the second JSON file and the first JSON file.

[0018] In some embodiments, the computer-implemented method may further include: obtaining a third JSON file from the database, the third JSON file describing changes to the metadata associated with the extended application; applying the changes to the metadata associated with the extended application described in the third JSON file to generate metadata associated with a second extended application; and implementing the second extended application based on the implementation data of the first application and the metadata associated with the second extended application. In some embodiments, the computer-implemented method may further include: obtaining a second application from the database, the second application including implementation data of the second application and a third JSON file including metadata associated with the second application; applying the changes to the first JSON file described in the second JSON file to the third JSON file to generate metadata associated with the second extended application; and implementing the second extended application based on the implementation data of the second application and the metadata associated with the second extended application. In some embodiments, the database may include a first repository storing the implementation data of the plurality of applications and a second repository storing the first JSON file and the second JSON file.

[0019] According to certain embodiments, a non-transitory computer-readable storage medium may store computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations. The operations may include: obtaining a first JavaScript Object Notation (JSON) file from a database storing metadata for multiple applications in JSON files, the first JSON file including metadata for a first application; receiving a modification to the first application to generate an extended application; determining differences between the metadata of the extended application and the metadata of the first application; and storing a second JSON file in the database, the second JSON file describing changes to be made to the metadata of the first application to generate the metadata of the extended application.

[0020] According to certain embodiments, a computer system may include one or more processors and a non-transitory computer-readable storage medium storing computer-executable instructions. The instructions, when executed by the one or more processors, may cause the one or more processors to perform operations, the operations may include: obtaining a first application from a database storing a plurality of applications, the first application may include implementation data of the first application and a first JavaScript Object Notation (JSON) file including metadata associated with the first application. The operations may also include obtaining a second JSON file associated with an extended application of the first application from the database, wherein the second JSON file describes changes to the first JSON file. The operations may further include: applying the changes to the first JSON file described in the second JSON file to generate metadata associated with the extended application; and implementing the extended application based on the implementation data of the first application and the metadata associated with the extended application.

[0021] The techniques and infrastructure disclosed herein may also allow a client to obtain, on demand, a representation of the differences between an extended application and a corresponding base application. In some embodiments, metadata associated with the extended application and stored in a JSON file for the bot extension may include a definition of a test suite that can be used by the infrastructure to validate the extended application when needed. In some embodiments, separate version control may be used for the bot extension and the base application so that the base application and the bot extension can evolve separately. The metadata for the version of the bot extension stored in the JSON file may indicate a range of versions of the base application that are compatible with a version of the bot extension. Thus, in some embodiments, the bot extension can be rebased to a different version of the same base application.

[0022] The terms and expressions that have been employed are used as descriptive and not limiting terms, and when such terms and expressions are used, it is not intended to exclude any equivalents of the features shown and described, or portions thereof. However, it should be recognized that various modifications are possible within the scope of the claimed systems and methods. Therefore, it should be understood that although the present systems and methods have been specifically disclosed by way of examples and optional features, those skilled in the art will recognize modifications and variations of the concepts disclosed herein, and such modifications and variations are considered to be within the scope of the present systems and methods as defined by the appended claims.

[0023] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification, any or all drawings, and each claim of the disclosure.

[0024] The foregoing and other features and examples are described in more detail below in the following description, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Illustrative examples are described in detail below with reference to the following figures.

[0026] Figure 1 is a simplified block diagram of a distributed environment incorporating an exemplary embodiment.

[0027] Figure 2 Depicted is a distributed system implementing a robotic system for communicating with end users using one or more messaging applications, according to certain embodiments.

[0028] Figure 3 Depicted is an integrated system including a robotic system and a robotic analysis system for monitoring, analyzing, visualizing, and improving performance of the robotic system, according to certain embodiments.

[0029] Figure 4 is a simplified flow chart illustrating an example of a process for developing a bot in accordance with certain embodiments.

[0030] Figure 5 Depicted is a system block diagram of an example of a robotic scalability infrastructure in accordance with certain embodiments.

[0031] Figure 6 Illustrated is an example of tracking the version of a base robot and the versions of robot extensions, and compatibility between the versions of the base robot and the versions of robot extensions, in a robot extensibility structure according to some embodiments.

[0032] Figure 7 Illustrated is an example of creating a new skill extension to a published skill using a graphical user interface (GUI) in accordance with certain embodiments.

[0033] Figure 8 Illustrated is an example of using a context menu in a GUI to create a new skill extension to a published skill, in accordance with certain embodiments.

[0034] Figure 9 Illustrated is an example of a dialog box in a GUI image for creating an extended skill in accordance with some embodiments.

[0035] Figure 10An example of a GUI image showing certain inherited components of a base robot and new components of an extended robot is illustrated in accordance with certain embodiments.

[0036] Figure 11 Illustrated is an example of a GUI image showing form fields that may be added, edited, or removed for expanding a skill, in accordance with some embodiments.

[0037] Figure 12 An example of a GUI image showing a “restore” icon for restoring the value of a skill's field to the original value is illustrated in accordance with some embodiments.

[0038] Figure 13 An example of a GUI image showing a dialog box for restoring the value of a skill's field to the original value is illustrated in accordance with some embodiments.

[0039] Figure 14 An example of a GUI image showing a comparison between a base skill and an expanded skill is illustrated in accordance with some embodiments.

[0040] Figure 15 Illustrated is an example of a GUI image showing a dialog box for rebasing a skill extension into a different version of a base skill in accordance with some embodiments.

[0041] Figure 16 An example of a GUI image showing a dialog box for creating a new version of a skill that has been published and expanded is illustrated in accordance with some embodiments.

[0042] Figure 17 Illustrated is an example of a GUI image showing a dialog box for confirming rebasing a skill extension to a new version of a base skill, in accordance with certain embodiments.

[0043] Figure 18 Illustrated is an example of a GUI image presenting information about rebasing a skill extension to a new version of a base skill, in accordance with certain embodiments.

[0044] Figure 19 An example of a GUI image showing a list of changes from previous base skills to new base skills is illustrated in accordance with some embodiments.

[0045] Figure 20 Illustrated is an example of a GUI for comparing content of conversation streams involved in a rebasing process, in accordance with certain embodiments.

[0046] Figure 21 Illustrated is an example of a GUI image showing a dialog box for confirming or canceling a rebasing of a skill extension, in accordance with some embodiments.

[0047] Figure 22 Illustrated is an example of a GUI image showing examples of custom components for a skill, in accordance with some embodiments.

[0048] Figure 23 Illustrated is an example of a GUI image for creating a new service in a skill, in accordance with some embodiments.

[0049] Figure 24A An example of a GUI image is illustrated in accordance with some embodiments, including icons indicating that a skill is an expanded skill and that new base skill updates are available.

[0050] Figure 24B Illustrated is an example of a GUI image indicating an icon indicating that a skill is an expanded skill and has a pending rebase, in accordance with some embodiments.

[0051] Figure 25 is a simplified flow chart illustrating an example of a process for generating an expanded skill in accordance with certain embodiments.

[0052] Figure 26 is a simplified flow chart illustrating an example of a process for implementing expanded skills in accordance with certain embodiments.

[0053] Figure 27 Depicted is a simplified block diagram of an example of a distributed system for implementing some embodiments.

[0054] Figure 28 is a simplified block diagram of an example of a cloud-based system environment for implementing some embodiments.

[0055] Figure 29 An example of a computer system for implementing some embodiments is illustrated. DETAILED DESCRIPTION

[0056] The present disclosure generally relates to chatbots. More specifically, but not exclusively, the technology disclosed herein relates to an infrastructure for extending a base chatbot that can be developed by another party and made available for download from a chatbot store where different chatbots and / or different versions of chatbots can be stored. Various inventive embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, codes, or instructions executable by one or more processors, and the like.

[0057] An enterprise may wish to create a chatbot system for various purposes. A chatbot system may include one or more user intent classification engines for identifying end-user intent based on user utterances and one or more dialogue engines for intelligently and contextually composing messages to respond to user utterances based on the determined end-user intent. However, building a chatbot system (including both a user intent classification engine that can determine the end-user's intent based on user utterances and a dialogue engine for intelligently and contextually generating responses) is a challenging task, partly due to the subtlety and uncertainty of natural language and the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (the number of intents). The number of robot developers who are also domain experts is extremely limited. This makes chatbot solutions based on AI or ML expensive and / or inaccessible to many individuals and organizations that want to use robots to communicate with end users or customers or otherwise interact.

[0058] According to some embodiments, chatbots for various fields or applications can be built by developers with the necessary skills and made available to businesses or individuals. Chatbots can be available in a chatbot store, which can allow developers and partners to publish chatbots and can allow customers to download chatbots to implement customized chatbots. The chatbot store can have certain features such as notifications, automatic updates, version control, etc.

[0059] According to some embodiments, a robot extension infrastructure can be provided so that customers can customize and / or extend chatbots downloaded from a chatbot store so as to adjust the chatbot to specific features, processes, terminology, culture, etc. that may be suitable for a specific application of the chatbot. The robot extensibility infrastructure can promote the extensibility of the chatbot in a holistic, universal, flexible, secure, and maintainable manner. For example, in some embodiments, a base chatbot downloaded from a chatbot store can be extended or customized by applying an extension (e.g., described in a JavaScript Object Notation (JSON) file) that changes the base chatbot. The base chatbot and the extension can be tracked separately. In some embodiments, the extension can be applied to different versions of the base chatbot. In some embodiments, the extension can be rebased to a different version of the base chatbot.

[0060] As used herein, a "chatbot," "bot," "skill," or "skillbot" may refer to a computer program designed to simulate a conversation with a human end user, particularly over the Internet. Individual skills may be designed to interact with end users and complete specific types of tasks, such as ordering food, making reservations, changing contact information, technical support, and customer service. Each skill may help end users complete tasks through a combination of visual, audio, or text messages and UI elements such as buttons, forms, lists, and the like.

[0061] As used herein, a "base bot" or "base skill" may refer to a bot developed by a Software as a Service (SaaS) provider or developer with the necessary skills and available to businesses or individuals for implementing custom bots. A base bot may be available for download from a bot store (also known as a skill store) and may be customized and / or extended. An "extended bot" or "extended skill" may refer to a bot that is customized and / or extended (e.g., augmented or enhanced) from a base bot to adapt the base bot to specific functions, processes, terminology, culture, etc. A "bot extension," "skill extension," or "extension" may refer to code and / or data that can be used to extend a base bot into an extended bot.

[0062] As used herein, the term "intent" may refer to a category of actions or tasks that an end user desires a skill to perform for them. The term "entity" may refer to a variable that identifies information from user input that enables a skill to complete a task. The term "component" may refer to various functions that a skill can use to respond to an end user, such as outputting text, returning information from a backend, and executing custom logic. The term "dialog flow" may refer to the definition of a skill-user interaction and may describe how a skill responds and behaves based on user input. The term "channel" may refer to a platform-specific configuration that allows a skill to access a messaging platform or client messaging application. A single skill may have multiple channels configured for it, allowing the skill to run simultaneously on different services or platforms that an end user may prefer to use.

[0063] As used herein, an "utterance" or "message" may refer to one or more sentences exchanged during a conversation, where a conversation may refer to a communication session that may include one or more utterances or messages. A conversation may include one or more phases or states. A conversation flow may be an abstraction of multiple conversations that include the same phases or states and the same transitions from phase (or state) to phase (or state). Each conversation may be a specific instance of a corresponding conversation flow. The states (or phases) of a conversation (or conversation flow) may be associated with the states of a state machine maintained by a robot system for conducting conversations with other robot systems or humans. In some cases, a state may correspond to an end-user's intent or goal. As used herein, an end-user may refer to an end-user of a robot system, such as a person or another entity that conducts a conversation with a robot system through a messaging application or platform. For example, an end-user may be a customer or client of a business that owns a robot system. As used herein, a user of a robot system may refer to an owner, operator, administrator, or developer of a robot system.

[0064] In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the examples of the present disclosure. However, it will be apparent that various examples can be practiced without these specific details. The subsequent description provides only examples and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the subsequent description of the examples will provide those skilled in the art with an enabling description for implementing the examples. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the present disclosure as set forth in the appended claims. The drawings and descriptions are not intended to be restrictive. Circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the examples with unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the examples. The teachings disclosed herein may also be applied to various types of applications such as mobile applications, non-mobile applications, desktop applications, web applications, enterprise applications, etc. Further, the teachings of the present disclosure are not limited to a specific operating environment (e.g., operating system, device, platform, etc.), but may be applied to multiple different operating environments.

[0065] Furthermore, it should be noted that individual examples may be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. While a flowchart may describe operations as a sequential process, many operations may be performed in parallel or simultaneously. Additionally, the order of the operations may be rearranged. When the operations of a process are completed, the process terminates, but may have additional steps not included in the diagram. A process may correspond to a method, function, process, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0066] The word “example” or “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” or “example” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0067] The term "machine-readable storage medium" or "computer-readable storage medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying (multiple) instructions and / or data. A machine-readable storage medium or computer-readable storage medium may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memories, memories, or storage devices. A computer program product may include code and / or machine-executable instructions that may represent any combination of a process, function, subroutine, program, routine, subroutine, module, software package, class, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted by any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0068] In addition, examples can be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program products) for performing the necessary tasks can be stored in a machine-readable medium. (Multiple) processors can perform the necessary tasks. The systems depicted in some figures can be provided in various configurations. In some examples, the system can be configured as a distributed system, wherein one or more components of the system are distributed across one or more networks in a cloud computing system. Where a component is described as being “configured to” perform certain operations, such configuration can be accomplished by, for example, designing an electronic circuit or other hardware to perform the operation, by programming or controlling an electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or any combination thereof.

[0069] I. Skills

[0070] Figure 1 is a simplified block diagram of a distributed environment 100 incorporating an exemplary embodiment. The distributed environment 100 includes a Digital Assistant Builder Platform (DABP) 102 that enables an enterprise to create and deploy digital assistants for its end users. The DABP 102 can be used to create one or more digital assistants (DAs). The DABP 102 can be used by multiple enterprises to create digital assistants for their end users. For example, Figure 1 As shown, a user 104 representing a particular enterprise can use DABP 102 to create and deploy a digital assistant 106 for end users of the particular enterprise. For example, a restaurant (e.g., a pizza shop) can use DABP 102 to create and deploy a digital assistant that enables customers of the restaurant to order food (e.g., order pizza).

[0071] For the purposes of this disclosure, a "digital assistant" refers to an entity that assists end users of the digital assistant in completing various tasks through natural language conversation. A digital assistant may be implemented using software alone (e.g., a digital assistant is a digital entity implemented using a program, code, or instructions executable by one or more processors), hardware, or a combination of hardware and software. A digital assistant may be embodied or implemented in various physical systems or devices, such as computers, mobile phones, watches, appliances, vehicles, and the like. A digital assistant is sometimes also referred to as a chatbot system.

[0072] For example, Figure 1As shown, an end user 108 can use the digital assistant 106 to perform various tasks through a natural language-based conversation with the digital assistant 106. As part of the conversation, the user can provide one or more user inputs 110 and obtain responses 112 in return from the digital assistant 106. Through these conversations, the user can request that one or more tasks be performed by the digital assistant, and in response, the digital assistant can perform the user-requested tasks and respond to the user with an appropriate response.

[0073] User input 110 may be in a natural language and may be referred to as an utterance. User utterances may be in the form of text (e.g., when a user types something as input to the digital assistant 106) or in the form of audio input or speech (e.g., when a user speaks something as input to the digital assistant 106). An utterance is typically in the language spoken by the end user 108. When the user input is in the form of speech, the user input may be converted into a text utterance in the particular language and then processed by the digital assistant 106. Various speech-to-text processing techniques may be used to convert the speech or audio input into a text utterance, which is then processed by the digital assistant 106.

[0074] The text utterance generated by the user input or by converting the voice input into text form can be a text fragment, a sentence, multiple sentences, etc. The digital assistant 106 is configured to apply natural language understanding (NLU) technology to the text utterance to understand the meaning of the user input. As part of the NLU processing of the utterance, the digital assistant 106 is configured to perform a process for understanding the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 106 can perform one or more actions or operations in response to the understood meaning or intent.

[0075] For example, the user input may request to order a pizza, e.g., “I want to order a pizza”. The digital assistant 106 is configured to understand the meaning of the utterance and take appropriate action, which may involve responding to the end user with a question requesting the user to input the type of pizza the end user wishes to order, the size of the pizza, any toppings for the pizza, etc. The response provided by the digital assistant 106 may also be in natural language, which may involve natural language generation (NLG) processing performed by the digital assistant 106. Once the digital assistant 106 obtains the necessary information from the user, the digital assistant 106 may proceed with the pizza order. The digital assistant 106 may end the session with the end user by outputting information indicating that the pizza has been ordered.

[0076] In some embodiments, an utterance received as input by the digital assistant 106 is subjected to a series of processing steps or a pipeline of processing steps. These steps may include, for example, parsing the utterance, understanding the meaning of the utterance, refining and reorganizing the utterance to develop a more understandable structure for the utterance, determining an action to be performed in response to the utterance, causing the action to be performed, generating a response to be output to the end user in response to the user utterance being generated, outputting the response to the end user, etc.

[0077] The NLU processing performed by a digital assistant (such as digital assistant 106) may include various NLP-related processing such as sentence semantic analysis (e.g., tokenizing, lemmatizing, identifying part-of-speech tags of sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing separate clauses, resolving anaphoras, executing chunks, etc.). In some embodiments, the NLU processing or part thereof may be performed by the digital assistant 106 itself. In some other embodiments, the digital assistant 106 may use other resources to perform part of the NLU processing. For example, the syntax and structure of a sentence may be identified by processing the sentence using a grammatical analyzer, a part-of-speech tagger, and / or a named entity recognizer. In one embodiment, for the English language, a grammatical analyzer, a part-of-speech tagger, and a named entity recognizer provided by the Stanford Natural Language Processing (NLP) group are used to analyze sentence structure and syntax. These are provided as part of the Stanford CoreNLP toolkit.

[0078] Although the various examples provided in this disclosure show utterances in the English language, this is meant as an example only. In some embodiments, the digital assistant 106 is also capable of processing utterances in languages other than English. In some embodiments, the digital assistant 106 provides subsystems (e.g., components that implement NLU functionality) that are configured to process different languages. These subsystems can be implemented as pluggable units that can be called from the NLU core server using service calls. This makes NLU processing flexible and extensible for each language, including allowing different processing orders. A language pack can be provided for each corresponding language, wherein the language pack can register a list of subsystems that can be served from the NLU core server and, if desired, can also utilize the provided general subsystems.

[0079] A digital assistant, such as digital assistant 106, can be made available to its end users through a variety of different channels, such as, but not limited to, through certain applications (also referred to as apps), through social media platforms, through various messaging services and applications, and other applications or channels. A single digital assistant can have several channels configured for itself, such that a single digital assistant can run on and be accessed through different services simultaneously.

[0080] A digital assistant contains or is associated with one or more skills. In some embodiments, these skills are individual chatbots designed to interact with end users and complete specific types of tasks (such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making reservations, purchasing widgets, etc.). For example, Figure 1 In the depicted embodiment, the digital assistant 106 includes skill bots 116 - 1 , 116 - 2 , 116 - 3 , etc. As described above, in this disclosure, the terms “skill” and “skills” are used synonymously with the terms “skill bot” and “skill bots,” respectively.

[0081] Each skill bot associated with the digital assistant assists an end user of the digital assistant in completing tasks through a conversation with the end user, wherein the conversation may include a combination of text or audio input provided by the end user and responses provided by the skill bot. These responses may be in the form of text or audio messages to the end user and / or using simple user interface elements (e.g., a list of choices) presented to the end user for selection.

[0082] There are various ways to add skills or skill bots to a digital assistant. In some instances, a skill bot can be developed by an enterprise and then added to a digital assistant using DABP 102. In other instances, a skill bot can be developed and created using DABP 102 and then added to a digital assistant created using DABP 102. In yet other instances, DABP 102 provides an online digital store (referred to as a "skill store") that offers a variety of skills covering a wide range of tasks. The skills offered through the skill store can showcase various cloud services. Users of DABP 102 can access the skill store through DABP 102, select a desired skill, and add the selected skill to a digital assistant created using DABP 102. Skills from the skill store can be added to a digital assistant as is or in a modified form. For example, a user of DABP 102 can select and clone a specific skill bot provided by the skill store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 102.

[0083] In some embodiments, the digital assistant created and deployed using DABP 102 is implemented using a master robot / child (or sub) robot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more child robots that are skill robots. For example, in Figure 1 In the depicted embodiment, the digital assistant 106 includes a master robot 114 and skill robots 116-1, 116-2, 116-3, etc., which are child robots of the master robot 114. In some embodiments, the digital assistant 106 itself acts as the master robot.

[0084] Digital assistants implemented according to a master-subbot architecture enable end users of the digital assistant to interact with multiple skills through a unified user interface. When an end user interacts with the digital assistant, user input is received by the master robot, which then processes the user input to identify the user request. Based on the processing, the master robot determines whether the user request can be handled by the master robot itself. If it is determined that the user request cannot be handled by the master robot, the master robot selects an appropriate skill robot to handle the user request and routes the conversation to the selected skill robot. This enables end users to converse with and use several skill robots configured to perform specific tasks through a common, single interface. For example, for a digital assistant developed for an enterprise, the digital assistant's master robot can interface with skill robots with specific functions (such as a customer relationship management (CRM) robot for performing functions related to customer relationship management, an enterprise resource planning (ERP) robot for performing functions related to enterprise resource planning, a human capital management (HCM) robot for performing functions related to human capital management, etc.). In this way, the end user or customer of the digital assistant only needs to know how to access the digital assistant.

[0085] In the master robot / sub robot infrastructure, the master robot is configured to understand the list of skill robots. The master robot has access to metadata that identifies the various available skill robots, and for each skill robot, the skill robot's capabilities include tasks that can be performed by the skill robot. After receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot from multiple available skill robots that can best serve or handle the user request. The master robot then routes the utterance (or part of the utterance) to the specific skill robot for further processing. Thus, control flows from the master robot to the skill robots. The master robot can support multiple input and output channels.

[0086] Although Figure 1 The embodiment shown in FIG. 1 shows the digital assistant 106 including a master robot 114 and skill robots 116 - 1 , 116 - 2 , and 116 - 3 , but this is not intended to be limiting. The digital assistant can include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems can be implemented solely in software (e.g., code or instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or through a combination of software and hardware.

[0087] DABP 102 provides the infrastructure, services, and features that enable users of DABP 102 to create a digital assistant, including one or more skill bots associated with the digital assistant. For example, a skill bot can be created by cloning an existing skill bot, cloning an existing skill bot and then modifying it, or by creating a skill bot from scratch using the tools and services provided by DABP 102. In some embodiments, DABP 102 provides a skill store or skill catalog that offers multiple skill bots for performing various tasks. Users of DABP 102 can clone skill bots from the skill store and create new skill bots.

[0088] DABP 102 also enables users (e.g., skill robot designers) to create skill robots from scratch. In some embodiments, at a high level, creating a skill robot involves the following operations:

[0089] (1) Configure settings for the new skill robot;

[0090] (2) Configure one or more intents for the skill robot;

[0091] (3) configuring entities for one or more intents;

[0092] (4) Training skill robots;

[0093] (5) Create a dialogue flow for the skill robot;

[0094] (6) Adding custom parts to skill robots; and

[0095] (7) Test and deploy skill robots.

[0096] (1) Configuring Settings for a New Skill Bot - The skill bot designer can specify one or more invocation names for the skill bot being created. These invocation names can be used in utterances to explicitly identify and invoke the skill bot in the digital assistant. The skill bot designer can also specify example utterances for the skill bot. These example utterances represent what the skill bot would say. When receiving user input, the digital assistant's intent analysis engine compares the user input to these example utterances to determine whether to invoke a specific skill bot.

[0097] (2) Configuring one or more intents for a skill bot - A skill bot designer can configure one or more intents (also called bot intents) for the skill bot being created. These intents identify tasks that the skill bot can perform for the end user of the digital assistant. Each intent is given a name. For example, for a skill bot that is configured to help a user perform various banking transactions, intents can be specified by the skill bot designer for the skill bot, such as "check balance", "transfer money", "deposit inquiry", etc. For each intent, the skill bot designer specifies a set of example utterances that represent and illustrate the meaning of the intent and are generally associated with the task performed by the intent. For example, for the "check balance" intent, example utterances can include "What is my savings account balance?", "How much money is in my checking account?", "How much money is in my account?", etc. A collection of typical user requests and statements can also be specified as example utterances for an intent.

[0098] (3) Configuring entities for one or more intents of a skill bot - In some instances, additional context may be needed to enable a skill bot to respond correctly to a user request. For example, there may be situations where two or more user input utterances resolve to the same intent in a skill bot. For example, in the above example, the utterances "What is my savings account balance?" and "How much is in my checking account?" both resolve to the same "check balance" intent, but these utterances are different requests for different things. To clarify such requests, one or more entities are added to the intent. Using the banking skill example, an entity called "Account Type" (which defines values called "checking account" and "savings") can enable the skill bot to parse the user request and respond appropriately. One or more entities can be specified for certain intents configured for a skill bot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and enable the skill bot to complete the user request. In some embodiments, there are two types of entities: (a) built-in entities provided by DABP 102; and (b) custom entities that can be specified by the skill bot designer. Built-in entities are general entities that can be used with various bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. Custom entities are used for more customized applications. For example, for a banking skill, a skill bot designer could define an "Account Type" entity to implement various banking transactions by checking user input for keywords such as "checking account," "savings," "credit card," etc.

[0099] (4) Training a Skill Bot—A skill bot is configured to receive user input, parse or otherwise process the received user input and identify or select an intent associated with the received user input. To enable this to occur, the skill bot can be trained. In some embodiments, the skill bot is trained based on the intents configured for the skill bot and example utterances associated with the intents (collectively referred to as training data) so that the skill bot can parse the user input into one of its configured intents. In some embodiments, the skill bot is represented by a model that is trained using the training data and that allows the skill bot to discern what the end user has said (or, in some cases, is attempting to say). The DABP 102 provides a variety of different training techniques that can be used by skill bot designers to train skill bots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof. In some embodiments, a portion (e.g., 80%) of the training data is used to train the skill bot model and another portion (e.g., the remaining 20%) is used to test or validate the model. Once trained, the skill bot can then be used to process and respond to user utterances. In some cases, the user's utterance may be a question that requires only a single answer and does not require additional conversation. To handle this situation, you can configure a Q&A intent for your skill bot. This enables the skill bot to output responses to user requests without having to update the dialog definition. Q&A intents are created in a similar manner to regular intents. However, the dialog flow for Q&A intents differs from that for regular intents.

[0100] (5) Create a dialogue flow for the skill bot - The dialogue flow specified for the skill bot describes how the skill bot will react when the received user input is parsed into different intents of the skill bot. The dialogue flow defines the actions or behaviors that the skill bot will take (e.g., how the skill bot responds to user utterances, how the skill bot prompts the user for input, how the skill bot returns data, etc.). The dialogue flow is like the process followed by the skill bot. Figure 1 Skill bot designers specify conversation flows using a language such as markdown. In some embodiments, a version of YAML called OBotML can be used to specify conversation flows for skill bots. The conversation flow definition for a skill bot serves as a model for the conversation itself, enabling skill bot designers to orchestrate interactions between the skill bot and the end user it serves.

[0101] In some embodiments, a dialog flow definition contains three parts:

[0102] (a) context;

[0103] (b) the default transition section; and

[0104] (c) Status section

[0105] Context section—The skill bot designer can define variables used in the conversation flow in the context section. Other variables that can be named in the context section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that enable the skill bot to recognize and persist user preferences.

[0106] Default Transitions Section—A skill bot's transitions can be defined in either the Dialog Flow State section or the Default Transitions section. Transitions defined in the Default Transitions section act as fallbacks and are triggered when no applicable transition is defined within a state or when the conditions required to trigger a state transition are not met. The Default Transitions section can be used to define routes that allow a skill bot to gracefully handle unexpected user actions.

[0107] State Part—A dialog flow and its associated operations are defined as a sequence of transient states that manage the logic within the dialog flow. Each state node within a dialog flow definition names a component that provides the functionality required at that point in the dialog. Thus, states are structured around components. States contain properties specific to a component and define transitions to other states that are triggered after the component executes.

[0108] Special case scenarios can be handled using the state section. For example, it may be desirable to provide the end user with the option to temporarily leave the first skill with which the end user is interacting and handle things in a second skill within the digital assistant. In one example, if the end user is busy talking to the shopping skill (e.g., the user has made some purchase choices), the end user may want to jump to the banking skill (e.g., the end user may want to make sure they have enough money for the purchase) and then return to the shopping skill to complete the end user's order. To address this, an action in the first skill can be configured to initiate an interaction with a second, different skill in the same digital assistant and then return to the original flow.

[0109] (6) Adding Custom Components to a Skill Bot - As described above, a state specified in a skill bot's conversation flow names a component that provides the functionality required by the state. Components enable a skill bot to perform a function. In some embodiments, DABP 102 provides a set of preconfigured components for performing a wide range of functions. A skill bot designer can select one or more of these preconfigured components and associate them with a state in a skill bot's conversation flow. A skill bot designer can also use the tools provided by DABP 102 to create custom components or new components and associate the custom components with one or more states in a skill bot's conversation flow.

[0110] (7) Testing and Deploying Skill Bots—DABP 102 provides several features that enable skill bot designers to test the skill bots they are developing. The skill bots can then be deployed and included in digital assistants.

[0111] While the above description describes how to create a skill bot, similar techniques can also be used to create a digital assistant (or master bot). At the master bot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master bot) can handle without invoking the skill bot associated with the digital assistant. Examples of system intents defined for the master bot include: (1) Exit: applicable when the end user signals a desire to exit the current session or context in the digital assistant; (2) Help: applicable when the end user requests help or directions; and (3) Unresolved Intent: applicable to user input that does not quite match the Exit Intent and Help Intent. The digital assistant also stores information about one or more skill bots associated with the digital assistant.

[0112] At the master bot or digital assistant level, when an end user enters a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the conversation. The digital assistant uses a routing model, which can be rule-based, artificial intelligence-based, or a combination thereof, to determine this. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input should be routed to a specific skill for processing, handled by the digital assistant or master bot itself according to built-in system intents, or processed into a different state in the current conversation flow.

[0113] In some embodiments, as part of this processing, the digital assistant determines whether the user input identifies a skill robot using its invocation name. The presence of an invocation name in the user input can be considered an explicit invocation of a skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill robot for further processing. In some embodiments, if no specific invocation is present, the digital assistant evaluates the received user input and calculates confidence scores for the system intents and skill robots associated with the digital assistant. The scores calculated for the skill robots or system intents indicate how likely it is that the user input represents the task the skill robot is configured to perform or represents a system intent. Any system intent or skill robot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill robot from the identified candidates for further processing of the user input. In some embodiments, after one or more skill robots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is applied to each intent. Any intent with a confidence score exceeding the threshold is generally considered a candidate flow. If a specific skill robot is selected, the user input is routed to the skill robot for further processing. If a system intent is selected, one or more actions are performed according to the selected system intent.

[0114] As described above, a skill (also known as a bot, chatbot, conversational bot, skillbot, or conversational bot) is a computer program that can conduct a conversation with an end user. Bots typically respond to natural language messages (e.g., questions or comments) via a messaging application that uses natural language messaging. An enterprise can use one or more bot systems to communicate with end users via messaging applications. The messaging application (which may be referred to as a channel) can be a preferred messaging application that the end user already has installed and is familiar with. Thus, end users can chat with a bot system without having to learn a programming language or download and install a new application. Messaging applications may include, for example, over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), mobile and web application extensions that are native or hybrid extensions / responsive mobile applications or web applications with chat capabilities, or voice-based input (such as devices or applications with interfaces that use Siri, Cortana, Google Voice, or other voice input for interaction).

[0115] In some examples, a robot system can be associated with a uniform resource identifier (URI). A URI can identify a robot system using a string of characters. A URI can be used as a webhook for one or more messaging application systems. A URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). A robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post call messages) from a messaging application system. An HTTP post call message can involve a URI from a messaging application system. In some embodiments, a message can be different from an HTTP post call message. For example, a robot system can receive a message from a short message service (SMS). Although the discussion herein may refer to communications received by a robot system as messages, one of ordinary skill in the art will recognize that a message can be an HTTP post call message, an SMS message, or any other type of communication between two systems.

[0116] End users can interact with the robot system through conversational interactions (sometimes called conversational user interfaces (UIs)), just as people interact with each other. In some cases, the interaction can include the end user saying "hello" to the robot, and the robot responding with "hi" and asking the end user how the robot can help. In some cases, the interaction can also be a transactional interaction with, for example, a banking robot, such as transferring money from one account to another. The interaction can also be an informational interaction with, for example, a human resources (HR) robot, such as checking vacation balances. The interaction can be an interaction with, for example, a retail robot, such as discussing returning a purchase or seeking technical support.

[0117] In some embodiments, the robot system can intelligently handle end-user interactions without interacting with the administrator or developer of the robot system. For example, the end user can send one or more messages to the robot system in order to achieve a desired goal. The message can include some content, such as text, emoticons, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., a Representational State Transfer (REST) call to an enterprise service with appropriate parameters) and generate a natural language response. The robot system can also prompt the end user to enter additional input parameters or request other additional information. In some embodiments, the robot system can also initiate communication with the end user, rather than passively responding to the end user's utterances.

[0118] A conversation with a robot can follow a specific conversation flow that includes multiple states. The flow can define what happens next based on input. In some embodiments, a robot system can be implemented using a state machine that includes user-defined states (e.g., end-user intent) and actions to be taken in or between states. The conversation can take different paths based on the end-user input, which may affect the decisions the robot makes regarding the flow. For example, in each state, based on the end-user input, the robot can determine the end-user's intent in order to determine the appropriate next action to take.

[0119] An intent can include a goal that an end user wants to accomplish. Intents map end-user input to actions that the backend system can perform for the end user. Therefore, based on the phrases spoken by the end user in natural language, the bot can map the end-user utterance to a specific use case or task, such as ordering a pizza, getting an account balance, transferring money, making a purchase, receiving a reward, and so on. Human conversations are often nonlinear in nature. End-users can often enter different states during a conversation. For example, if an end user wants to transfer funds from Account A to a recipient, the end user might begin the conversation with the bot system by asking, for example, the bot to pay the recipient for dinner. The bot might respond with, for example, "Which account?" The end user might choose a checking account but then realize they're unsure of the balance in the account. Therefore, the end user might switch context to request a balance, recent transactions, and so on. In other words, the end user can trigger a change in flow and state, for example, from transferring money to checking a balance and then to recent transactions. At some point, the end user might decide to return to the original intent—paying the recipient. Therefore, one task of the bot system is to dynamically determine the end-user intent from natural language utterances.

[0120] Bots can use natural language processing (NLP) engines and / or machine learning models (e.g., intent classifiers) to map end-user utterances to specific intents. For example, a machine learning-based NLP engine can learn to understand and classify natural language conversations from end users and extract the necessary information from the conversations so that precise actions can be taken, such as executing a transaction or looking up data from a backend system of record.

[0121] Figure 2 A distributed system 200 is depicted that implements a bot system for communicating with end users using a messaging application, according to certain embodiments. The system 200 may include a bot system 220, one or more messaging application systems 215, and one or more end-user devices (e.g., one or more mobile devices 210). In some examples, the messaging application may be installed on an electronic device (e.g., a desktop computer, a laptop computer, a mobile device 210, etc.). Although the discussion herein will refer to mobile devices and messaging applications, one of ordinary skill in the art will recognize that any electronic device may be used, and any messaging platform or messaging application may be used, such as Messenger, Instant messaging software, Mobile text and voice messaging communications services, Messenger, Messenger, SKYPE Messenger, Short Message Service (SMS) or any other messaging application that provides a platform for end users to communicate. In other examples, the browser installed on the mobile device 210 (e.g., GOOGLE Browser, BROWSERS and INTERNET EXPLORER BROWSER) to run the messaging application. In some embodiments, two or more messaging applications can be installed on the end-user device to communicate through two or more messaging platforms (such as two or more messaging application systems 215).

[0122] The messaging application can be facilitated by a messaging platform such as a messaging application system 215. Mobile device 210 can connect to the messaging application system 215 via a first network (e.g., the Internet). The messaging application system 215 can be a messaging platform provided by a third party, such as Facebook, Tencent, Google, Microsoft, etc. The messaging application system 215 can manage content sent and received via the messaging application across multiple mobile devices or other end-user devices.

[0123] The bot system 220 (e.g., implemented on one or more servers) can also be communicatively connected to the messaging application system 215 to send and receive messages. Communication between the messaging application system 215 and the bot system 220 can be over a second network (e.g., the Internet). The first network and the second network can be the same network, or they can be similar or completely different networks. The messaging application system 215 can use the Internet to route content (e.g., a message or information from a message) from the mobile device 210 to the bot system 220. In some embodiments, the destination of the content (e.g., an identification of the bot system 220) can be included in the content as a nominal address. In some embodiments, the bot system 220 can also be configured to communicate with two or more messaging application systems 215.

[0124] As discussed above, content exchanged between end users or between end users and a robot system may include, for example, text, emoticons, audio, media (e.g., pictures, videos, links), or any other method of conveying a message. Examples of messages that Messenger may receive include:

[0125]

[0126] The bot system 220 can receive content from the messaging application system 215 using a connector 230 that acts as an interface between the messaging application system 215 and the bot system 220. In some embodiments, the connector 230 can normalize the content from the messaging application system 215 so that the bot system 220 can analyze the content across different messaging application systems. The content normalization process can include formatting the content from each type of messaging application into a common format for processing. In some embodiments, for each messaging application (e.g., Messenger, Instant messaging software, Mobile text and voice messaging communications services, Messenger, Messenger and Skype Messenger, Short Message Service (SMS), the robot system 220 may include one or more connectors. In some implementations, the connector 230 may route content to the message input queue 240. The message input queue 240 may include a buffer (e.g., a first-in, first-out (FIFO) buffer) that stores content in the order it was received. In some embodiments, each connector 230 may be associated with one or more message input queues.

[0127] When the message processor 250 becomes available, the message input queue 240 can send content to the message processor 250. In some embodiments, the message processor 250 can pull content from the message input queue 240. As described in detail below, the message processor 250 can parse the message and determine the intent of the parsed message. In some embodiments, the message processor 250 can include a natural language processor 252 and an intent determination subsystem 254. The natural language processor 252 can parse the message and perform some semantic analysis, such as identifying the subject, predicate (e.g., action), and / or object. The intent determination subsystem 254 can determine the end-user intent based on the parsed message. As described above, the intent can include the purpose of the message. For example, the purpose of the message can be to order a pizza, order a computer, transfer money, ask a question about delivery, etc. In some embodiments, parameters associated with the intent (which can be referred to as entities) that can more specifically define or clarify the action to be taken can also be extracted from the message by the natural language processor 252 and / or the intent determination subsystem 254.

[0128] After the end-user intent is determined based on the content by the message processor 250, the determined intent (and parameters associated with the intent) can be sent to the action engine 260. The action engine 260 can be used to determine the action to be performed based on the intent (and parameters associated with the intent) and the current state (or context) of the state machine as described above. For example, the action engine 260 can send certain outbound content as a response to the message output queue 270 and / or can send commands to some enterprise services (such as enterprise services 225) or obtain information from them. The message output queue 270 can send the outbound content to the connector 230. The connector 230 can then send the outbound content to the messaging application system indicated by the action engine 260, which can be the same as or different from the messaging application system 215. The messaging application system 215 can then forward the outbound content to the messaging application on the mobile device 210.

[0129] The robot system 220 can communicate with one or more enterprise services (e.g., enterprise service 225), one or more storage systems for storing and / or analyzing messages received by the robot system 220, or a content system for providing content to the robot system 220. The enterprise service 225 can communicate with one or more connectors 230, an action engine 260, or any combination thereof. The enterprise service 225 can communicate with the connectors 230 in a manner similar to the messaging application system 215. The enterprise service 225 can send content to the connectors 230 to be associated with one or more end users. The enterprise service 225 can also send content to the connectors 230 to cause the robot system 220 to perform an action associated with the end user. The action engine 260 can communicate with the enterprise service 225 to obtain information from the enterprise service 225 and / or instruct the enterprise service 225 to take an action identified by the action engine 260.

[0130] In some embodiments, the bot system 220 can include one or more timers. The timers can cause the action engine 260 to send content to the end user using the connector 230 and the messaging application system 215 after a period of time has elapsed. In some embodiments, the timer can send content to the bot system 220 similarly to how an end user or enterprise service 225 sends content. For example, the timer can send a message to the bot system 220 for analysis similar to how a message from an end user would be analyzed.

[0131] In one particular embodiment, an end user can use mobile device 210 to send a message to bot system 220 via messaging application system 215. The message can include a greeting such as "Hello" or "Hi." The bot system can determine that a new session with the end user has begun and initiate a state machine. In some embodiments, the bot system can identify one or more characteristics of the end user. For example, the bot system can use a profile associated with the end user on the messaging application system to identify the end user's name. Using these one or more characteristics, the bot system can respond to the end user via the messaging application. The response can include a message to the end user in response to the message received from the end user. For example, the response can include a greeting with the end user's name, such as "Hi, Tom, how can I help you?" Depending on the business associated with the bot system, the bot system can proceed to accomplish the business's goals. For example, if the bot system is associated with a pizza delivery business, the bot system can send a message to the end user asking if the end user would like to order a pizza. The conversation between the bot system and the end user can continue from there, going back and forth until the bot system completes the conversation or the end user stops responding to the bot system.

[0132] In some embodiments, the robot system can initiate a session with the end user. The session initiated by the robot system can be in response to a previous session with the end user. For example, the end user may have ordered a pizza in a previous session. The robot system can then initiate a session after the pizza is ready. In some embodiments, the robot system can determine that the pizza is ready when an indication is received from the business associated with the robot system (e.g., an employee sends a message to the robot system indicating that the pizza is ready). The session can include a message sent to the end user indicating that the pizza is ready.

[0133] In some embodiments, the robot system may send a message to the end user via a messaging application different from the messaging application that the message was previously received by. For example, the robot system may determine to use Short Message Service (SMS) instead of Messenger to send messages. In this embodiment, the robot system can integrate multiple messaging applications.

[0134] In some embodiments, the bot system can determine to start a session based on a timer. For example, the bot system can determine that a one-week timer is scheduled for the end user after a pizza is ordered. Expiration of the one-week timer may cause the bot system to start a new session with the end user to order another pizza. Timers can be configured by the enterprise and implemented by the bot system.

[0135] As described above, in some embodiments, the action engine 260 can send commands to some enterprise services 225 or obtain information from them. For example, when the robot system 220 (more specifically, the message processor 250) determines the intention to check the balance, the robot system 220 can determine which account balance to check among several accounts (e.g., current account or savings account). If the end user enters "What is the balance in my savings account?", the robot system 220 can extract "savings" and send a command to the bank server to check the balance and then send the received balance information to the end user via a message. If the end user initially only says "What is the balance in my account?", the robot system 220 can send a message to the end user prompting the end user to further specify a specific account, or can obtain information about all of the end user's accounts and send the account information to the end user for the end user to select.

[0136] In some embodiments, the bot system can maintain information between sessions. This information can be used later, eliminating the need for the bot system to ask questions each time a new session begins between the end user and the bot system. For example, the bot system can store information about the end user's previous pizza order. In a new session, the bot system can send a message to the end user asking if the end user wants the same order as last time.

[0137] In some embodiments, the robot system 220 can store information associated with the end user in a cache. After an outbound message is sent from the connector 230 to the messaging application system, the cache can be written to the database to save the information. In other embodiments, the cache can be written to the database at different times (e.g., after a specific event, after each event, after a certain amount of time, or any other metric used to determine when to write to the database).

[0138] When a slowdown is identified, the robotic system 220 can allow for scaling of each component. For example, if the robotic system 220 identifies that the number of messages arriving at connector 230 exceeds a threshold, one or more additional connectors can be added to connector 230. Additionally, the number of message input queues, message processors, action engine instances, and message output queues can be increased depending on where the slowdown is occurring. In such an embodiment, additional components can be added without having to add other additional components. For example, connectors can be added without having to add additional instances of the action engine. In some embodiments, one or more components or portions of components of the robotic system 220 can run on a virtual machine. By running on a virtual machine, additional virtual machines can be started as desired.

[0139] As described above, building a robotic system (such as a user intent classification engine that can determine an end user's intent based on an end user utterance) is a challenging task, in part due to the subtlety and ambiguity of natural language and the dimensionality of the input space (e.g., possible end user utterances) and the size of the output space (the number of intents). As such, in order to improve the performance of the robotic system and the user experience with the robotic system, it may be necessary to monitor, debug, and modify the new robotic system. In many cases, without the use of analysis or optimization tools, it may be difficult to more specifically identify the root cause of the robotic system's performance being lower than expected and determine how to improve the robotic system.

[0140] In some cases, a bot owner, developer, or administrator may want to monitor the operational status of a bot and understand how the bot is being used and where end users are abandoning the bot in order to improve the bot. For example, a bot owner, developer, or administrator may want to know which bot sessions are successful and which are unsuccessful in order to identify and diagnose underperforming elements of the bot system.

[0141] According to some embodiments, an analysis system can be integrated with the robot system. The analysis system can monitor events that occur during a session between an end user and the robot system, aggregate and analyze the collected events, and graphically present information about the session on a graphical user interface at different levels of summary (e.g., all sessions, different categories of sessions, and individual sessions). For example, the graphical user interface can display options for filtering or selecting certain types of sessions or individual sessions and graphically present the selected information, for example, by visualizing the session's path. The analysis system can also provide suggestions, options, or other information for improving the robot system.

[0142] Figure 3An integrated system 300 is depicted, according to certain embodiments, comprising a robot system (such as robot system 220) and a robot analytics system for monitoring, analyzing, visualizing, and improving the performance of the robot system. As illustrated, the robot system may include a connector 330 and multiple robot engines, such as a conversation engine 312, an intent modeler 314, an entity resolver 316, and custom components 318. The robot system may also include a database 340, an administrative application programming interface (API) 350, a user interface 354, and a UI server 352. The robot analytics system may include a collector 355, an enrichment engine 360, a database 370, and a REST server 380. The robot analytics system may also include a user interface 392 and a UI server 390. The collector 355 of the robot analytics system may collect events 305 occurring at the robot system. Feedback 394 from the robot analytics system may be provided to the robot system via the user interfaces 392 and 354.

[0143] Connector 330 can act as an interface between the robot system and one or more end users through one or more channels (such as channels 320 and 322). Each channel can be a messaging application, such as a messaging channel (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack or SMS), a virtual personal assistant (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), a mobile and web application extension with local or hybrid extensions / responsive mobile applications or web applications with chat functions, or voice-based input (such as a device or application with Siri, Microsoft Cortana, Google Voice or other voice input for interaction). In some embodiments, connector 330 can standardize content from different channels so that the robot system can analyze content across different messaging application systems. Content standardization processing can include formatting the content from each type of messaging application into a common format for processing. In some embodiments, for each channel, the robot system can include one or more connectors.

[0144] The intent modeler 314 can be used to determine the end-user intent associated with the end-user utterance. In some embodiments, the intent modeler 314 for determining the end-user's intent based on one or more messages received by the robot system from the end-user can use a natural language processor to tag word forms (verbs, nouns, adjectives), look up tokens / stems (runs / running / ran->run) and tag entities (Texas->LOCATION). In some embodiments, the intent modeler 314 can standardize messages. For example, "Mary ran to Texas" can become "PERSON run to LOCATION". The intent modeler can also include logic for detecting words with the same meaning in end-user messages. For example, if the training data set includes: "Mary ran to Texas" and "Bob walked to Detroit", both of which are mapped to the same intent and run / walk appear in the same set of intents, then the intent modeler 314 can learn that for the purpose of intent parsing, run = walk. In one illustrative example, "Mary ran to Texas" can become "PERSON run to LOCATION", and "Bob walked to Detroit" can become "PERSON walk to LOCATION". In the illustrated example, both sentences can be associated with the same intent because, for the purpose of intent parsing, "noun run to noun" is the same as "noun walk to noun". In another example, “I want to order a large cheese pizza” and “I want to order a small pepperoni pizza” can both be normalized to “I want to order a Bots_PizzaSize Bots_Toppings pizza”.

[0145] After normalization, the probability that the occurrence of a word indicates a certain intent can be determined. In some examples, the probabilities can be combined using a basic probabilistic algorithm, as if the probabilities were independent. For example, if there is a 20% probability that "order" implies ordering pizza and a 10% probability that "pizza" implies ordering pizza, the total probability is 1-(1-0.2)(1-0.1)=28%. Some probabilities can be based on the presence of words or on certain linguistic elements (such as negation or the presence of personal pronouns).

[0146] Another level of rules can be template rules, which are combinations of words. In some examples, every sentence in the training dataset can automatically become a rule once it is standardized. In such examples, the training dataset can include a very small number of short sentences. Template rules can return a probability of 1. New rules can be generated from rules through an inductive process. For example, the following sentences can belong to tracking expenses: "How much did I spend on gas last month?" and "How much did I spend on food in May?". The sentences can be used to generalize the rule for "How much did I spend" because this is the shared part between them. In other examples, the training dataset can include the phrase "How much did I spend" to achieve the same result.

[0147] The example described above allows the definition of intents to not include duplicates, such as variations of named entities (e.g., "send money to Sue" and "send money to Bob"). However, it is possible to train with similar sentences where one or two words are different. Similar sentences can allow the model to learn which words are likely to have the same intent parsed meaning and which words are likely to be common misspellings.

[0148] If a particular word or set of words (such as a verb) is important for an intent, the probability can be manipulated by having more examples with that word (and its synonyms) and fewer examples with that word for other intents.

[0149] Examples can also be provided to prevent the model from making incorrect assertions. For example, specific subphrases or words that only appear for a certain intent can lead to incorrect assertions. Similarly, this can prevent the model from synthesizing broad rules that are trained using similar sentences belonging to different intents.

[0150] The entity resolver 316 can identify entities (e.g., objects) associated with the end-user intent. For example, in addition to the end-user intent (e.g., "order a pizza") identified by the intent modeler 314, the entity resolver 316 can also resolve entities associated with the intent, such as pizza type, toppings, etc.

[0151] The dialogue engine 312 can be used to process a conversation between an end user and the robot system. For example, the dialogue engine 312 can respond to an end user utterance based on the end user intent identified by the intent modeler 314 and the entities associated with the end user intent identified by the entity resolver 316. In some embodiments, the dialogue engine 312 can use a state machine that includes user-defined states (e.g., end user intent) and actions to be taken in or between states to process a conversation with the end user.

[0152] Custom components 318 may include custom modules for a particular robot system. For example, a financial robot may include custom components that can be used to, for example, check balances, transfer funds, or pay bills.

[0153] Database 340 can be used to store data about the robot system, such as classification model data, session logs, etc. Management API 350 can be used by administrators or developers of the robot system to manage the robot system, such as retraining classification models, editing intents, or otherwise modifying the robot system. Administrators or developers can use user interface 354 and UI server 352 to manage the robot system.

[0154] Various events can be generated during the operation of a robotic system. Events can be generated based on one or more instructions included in the robotic system. For example, an event can be generated when the robotic system enters a specific state, where the specific state is defined by the robotic system administrator or developer. When an event is generated, it can be collected, stored, and analyzed by the robotics analytics system. When an event is captured, additional information associated with the event can also be collected, where this additional information can indicate the current context in which the event was generated.

[0155] For example, a conversation event may be generated by the dialog engine 312. A conversation event may include a message received by the robot system from an end-user device (referred to as msg_received). msg_received may include one or more of the following parameters or variables: the content of the message, the time the message was received by the robot system, the language of the received message, device properties (e.g., version or name), operating system properties (e.g., version or name), geographic location properties (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, robot system ID, tenant ID, etc.), timestamp (e.g., device created, device sent, collector obtained timestamp), channel, etc.

[0156] A conversation event may also include a message sent by the bot system to the end-user device (referred to as msg_sent). msg_sent may include one or more of the following: the content of the message (e.g., the text or HTML of the message), the time the message was sent by the bot system, the language of the message, the creator of the message (e.g., the bot system or the end-user device), device properties, operating system properties, browser properties (e.g., version or name), application properties (e.g., version or name), geographic location properties (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, bot system ID, tenant ID, etc.), channel (e.g., Facebook or webhook), etc.

[0157] The dialogue engine 312 may also generate dialogue state execution events. As described above, the dialogue engine 312 may use a state machine to determine the flow of a conversation with an end user. The state machine may include a set of states and rules for transitioning between states. The dialogue engine 312 may execute the state machine for each end-user conversation, and a dialogue state execution event may be generated for each state that the dialogue engine 312 steps through to process an end-user utterance. Attributes of a dialogue state execution event may include, for example, state name, component name, next action, entity match, intent match, variables, user query, response, execution time, communication language, device properties, operating system properties, geographic location properties, identification information, timestamp, channel, etc. The state name may be the name of the currently executing state or the "error state." The component name may be the name of the robot component executing for the current state. The next action may be the next action to be executed. The entity match may be the entity resolved in the current message. The intent match may be the intent resolved with a score value. The variable may be the variable value of the current state. The query may be the message sent by the end user. The response may be the message sent to the end user. The execution time may be the timestamp of the completed state execution. The communication language may be the language of the conversational messages. Device properties and / or operating system properties may be associated with the end user interacting with the robotic system. Browser properties and / or application properties may be associated with the end user interacting with the robotic system. Geographic location properties may be the location of the end user interacting with the robotic system.

[0158] As a result of the execution of the intent modeler 314, an intent resolution event may occur. The intent modeler 314 may use a trained or otherwise defined classification model to identify an end-user intent from a set of intents based on the end-user utterance. The results of the intent classification may be captured as intent resolution event attributes, which may include, for example, the final intent classification result (e.g., the identified intent) and a confidence score associated with each corresponding intent in the set of intents.

[0159] Entity resolver 316 can generate entity resolver events. An entity is an object associated with an end-user intent. When a bot system is created, entity definition rules can be determined. For example, in addition to resolving end-user intents such as "order a pizza," the bot system can also use entity resolver 316 to resolve entities associated with pizza type, toppings, and so on. Entity resolver events can be captured during entity resolution. Examples of attributes associated with entity resolver events include entity name, applied rules, search terms, resolved state, query statement, entity type, execution time, communication language, device properties, operating system properties, browser properties, application properties, geolocation properties, identifying information, timestamp, channel, and so on. The entity name can be the name of the entity currently being resolved. The applied rules can be, for example, before, after, or aggregate. The search terms can be from, to, destination, origin, and so on. The resolved state can be the state of the conversation for the entity resolution. The query statement can be a message containing an entity value. The entity type can be system or get. The execution time can be the timestamp of the entity resolution. The communication language can be the language of the message being spoken. Device properties and / or operating system properties may be associated with an end user interacting with the robotic system. Browser properties and / or application properties may be associated with an end user interacting with the robotic system. Geographic location properties may be the location of the end user interacting with the robotic system.

[0160] Custom components can also generate events, such as predefined events or custom events. Predefined events can be properties captured when executing a custom component. Examples of predefined event attributes include: component name, event name, payload, execution time, communication language, device properties, operating system properties, browser properties, application properties, geolocation properties, identification information, timestamp, channel, etc. Component name can be the name of the currently executing custom component. Event name can be invoked, invocation_failed, replied, replied_failed, etc. Payload can be the reason for failure (in the case of failure), a stack trace, etc. Execution time can be a timestamp indicating when the event occurred. Communication language can be the language of the conversation message. Device properties and / or operating system properties can be associated with the end user interacting with the robotic system. Browser properties and / or application properties can be associated with the end user interacting with the robotic system. Geolocation properties can be the location of the end user interacting with the robotic system.

[0161] Custom components can also issue custom events during the execution of the custom component. Examples of custom event attributes may include component name, event name, custom payload, execution time, communication language, device properties, operating system properties, browser properties, application properties, geolocation properties, identification information, timestamp, channel, etc. Component name may be the name of the currently executed custom component. Event name may be a user-defined event name (e.g., Balance_Retrieved). Payload may be, for example, {"amount": "$100", "account": "Current Account"}. Execution time may be a timestamp indicating when the event occurred. Communication language may be the language of the conversational message. Device properties and / or operating system properties may be associated with the end user interacting with the robotic system. Browser properties and / or application properties may be associated with the end user interacting with the robotic system. Geolocation properties may be the location of the end user interacting with the robotic system.

[0162] Error events and timeout events can also be generated by the robot system during execution. When an error occurs, an error event can be generated. When an end-user session has been inactive for a period of time, a timeout event can be generated. The timeout event can be configured at the channel.

[0163] When the robot system conducts a conversation with an end user and generates corresponding events, the robot analysis system can collect events and additional information. For example, the collector 355 can collect events and additional information and send the collected information to a queue. In some embodiments, the collector 355 can be configurable and can be programmed to collect the different events and / or event attributes described above as desired. For example, the collector 355 can be configured to capture conversation state attributes, intent resolution attributes, entity resolution attributes, as well as error attributes and timeout attributes. In some embodiments, the collector 355 can also be configured to collect information about events 395 generated by systems other than the robot system.

[0164] The enrichment engine 360 can perform validation and enrichment on the collected events and other information and write the collected events and other information to the database 370. For example, based on the collected IP address, the enrichment engine 360 can determine the location of the end user associated with the IP address. As another example, the enrichment engine 360 can extract certain features from the collected information, such as determining the web browser or channel used by the end user. The REST server 380 can analyze the enriched events and other information and generate various reports based on certain aggregate metrics 372. The reports can be displayed to the owner, administrator, or developer of the robotic system on the user interface 392 via the UI server 390. The owner, administrator, or developer of the robotic system can provide feedback 394 to the robotic system for improvement.

[0165] Figure 4 4 is a simplified flowchart 400 illustrating an example of a process for developing a skill according to some embodiments. The process may include creating an intent at 410, training a skill at 420, creating entities at 430, integrating custom components at 440, creating a dialog flow at 450, testing the skill at 460, routing to a channel at 470, and reviewing an insights report for improving the skill at 480.

[0166] At 410, intents for the skill can be created. Intents describe various actions that a skill can help its end users complete. For example, if a skill enables a user to perform various banking transactions, the skill's intents may include, for example, "check balance" or "transfer money." Intents not only describe what a skill can do, but can also be integral to the skill's intelligence. Intents enable a skill to recognize user input because each intent can have a set of typical user statements (i.e., utterances) associated with it. While these utterances can share the same meaning, they can be different (e.g., "What's my savings account balance?" and "How much money is in my checking account?").

[0167] At 420, when the skill parses the user input, the skill can be trained to infer the user's intent. Specifically, the skill can be trained with intents and their utterances (collectively referred to as training data) so that the skill can parse the user input into one of the intents. The trained skill can not only identify sample phrases belonging to each intent, but also identify similar phrases corresponding to each intent.

[0168] At 430, entities for the skill can be created. In some embodiments, a skill may require some additional context to complete a user request. While some user requests may be parsed into the same intent (e.g., "What is my savings account balance?" and "How much is in my checking account?" would both be parsed into the "check balance" intent), they are requesting different things. To clarify the request, one or more entities can be added to the intent. Using the banking skill example, an entity "Account Type" (which defines values such as "checking account" and "savings") could enable the skill to parse the user request and respond appropriately.

[0169] At 440, the custom widget can be integrated into the skill. Before integrating the widget into the skill, the skill can recognize user input but may not be able to respond to it. The widget enables the skill to perform its functions. The widget can perform a function (such as outputting text based on the intent parsed from the end user's message) or perform tasks specific to the particular skill (such as querying an account balance).

[0170] At 450, a dialog flow can be created. A dialog flow describes how the skill reacts when parsing different intents. The dialog flow defines the actions or behaviors that the skill bot will take (e.g., how the skill bot responds to user utterances, how the skill bot prompts the end user for input, how the skill bot returns data, etc.). In some embodiments, the dialog flow can be a flowchart described in a simple standard language such as a version of YAML (e.g., OBotML).

[0171] At 460, the skill can be tested after being developed and deployed in a digital assistant, such as described above. The skill can be tested by conducting one or more conversations with the skill using one or more dialog flows.

[0172] Optionally, at 470, the skill can be routed to one or more channels for user messaging and other capabilities. For example, if the skill will not be added to a digital assistant, the skill can be added to one or more user messaging channels. The user can chat with the skill through one or more of these user messaging channels (such as various messaging platforms, proprietary messaging applications, and web pages). The skill will run on any of these channels, whether they are text-only or can support scrolling cards, UI elements, images, and other non-text content. In addition to the channels connected to the user interface, the skill can also be routed to other channels (such as a channel that links the skill to a customer support system or a channel that routes notifications from an external application that prompted the skill to initiate a conversation).

[0173] At 480, developers can review insight reports to improve the skill. For example, developers can review insight reports related to the skill to find out whether customers are using the skill as expected. Insight reports can include, for example, high-level usage metrics and conversation trends, individual views of intents, their execution paths, and conversation transcripts. Insight reports can provide different perspectives on how well the skill supports its customers and where the skill prevents its customers from completing tasks. These insight reports not only allow developers to quickly identify problems, but can also suggest user input that can improve the skill's intent parsing.

[0174] In some embodiments, different versions of a skill can be generated. In some embodiments, a skill can be a composite skill that includes two or more related skills to perform more complex functions as described above. For example, composite skill A can include skill B and skill C, where, for example, the output from skill B can be used as input to skill C.

[0175] II. Skill Shop

[0176] As described above, businesses or individuals who need to use robots to communicate or otherwise interact with end users may not have the expertise to build skills. In many cases, skills can be built by developers as described above and made available to businesses or individuals. The developed skills can be available in a skill store. In some embodiments, a skill store can allow developers and partners (or customers) to publish skills. A skill store can allow developers to upload skill packages to the skill store as part of a product release. In some embodiments, pushing skills to the skill store can be available from the skill edit page. A skill store can have a specific directory structure and can provide detailed skill descriptions.

[0177] The skill store can provide detailed skill descriptions. In some embodiments, the skill store can include a skill store UI that allows customers to access the skill store. In some embodiments, the skill store UI can use the object repository REST API to collect details of skills and facilitate browsing of skill functions. Skills, versions, skill descriptions, and other data can be provided through the skill store UI for customers to view available skills. For example, the skill store UI can allow customers to browse skills, search for specific skills, search for skills of a specific version, search by keyword in name, summary, description, etc., and sort skills by name, category, date, etc. When pushing skills to the skill store, the category can be entered by the developer. Skills can be displayed with name, version, short description (e.g., summary), long description details, etc. In some embodiments, the skill store can have other features such as notifications, automatic updates, etc.

[0178] The Skill Store UI can also allow downloading of specific skill versions. The Skill Store UI can access the Skill Store through the Management API. The Management API can redirect all requests to the Skill Store using, for example, the Jersey REST client. The Management API can use standard libraries to access storage for obtaining authorization tokens / passwords. Account information can be stored in the robot's configuration.

[0179] The skill store can store implementations and metadata of skills. The implementations of skills can include, for example, pre-trained models in Java. The implementations of skills can include implementation data such as the intent of the skill, entities, custom components, dialog flows, models, etc. In some embodiments, the implementations of skills can be stored in the skill store as zip archives. The metadata of a skill can include information about the implementation of the skill (such as the name, category, version, short description (e.g., summary), long description details, etc. of the intent of the skill, entities, custom components, dialog flows, models, etc.). For example, in some embodiments, the skill store can include a backend that can store skills in zip archives. The backend can parse and store binary data, create and store metadata in JSON format, and use overwrite parameters to control workflows with existing skills in the skill store. In some embodiments, the implementations and metadata of skills can be stored separately.

[0180] In some embodiments, the backend can also list all skills with paging, sorting, and filtering support (e.g., showing the latest version by date). The skill store can retrieve the skill version, summary, and description information from the skill metadata. The backend can be configured to retrieve all versions of a skill by name. The backend can allow skills to be downloaded using a path for a zip archive and automatically install the downloaded skills. The backend can also allow deleting all skills or deleting skills by name.

[0181] In some embodiments, the skill store may include a REST API that can be used to transfer content between different storage devices. In some embodiments, the skill store may include a REST API that can return all the content of the skill store as a zip archive. The skill store may also support a trained model option during import / export. For example, a robot import API in the management API can be used to import skills and automatically train them. In some embodiments, a trained robot with a model can be exported.

[0182] In some embodiments, the skill store can include a REST API that can be used to push skills to the skill store. The REST API can accept a zip archive and some skill details for uploading the skill to the skill store. In some embodiments, the REST API can use override parameters to control the workflow with existing skills in the skill store. In some embodiments, the REST API may not be available through the skill store UI, so that customers may not be able to push skills to the store for security reasons.

[0183] In some embodiments, the skill store REST API may include various REST APIs such as an object storage REST API for accessing data in an object store (e.g., listing all containers or all objects in a container or bucket or downloading an object) and a skill store management REST API for accessing data in an object store through the robot's management API (e.g., using a pre-installed JSON formatter). Skills may be uploaded and downloaded using an external API outside the robot's environment that allows calling an endpoint (e.g., a server) from the management API container. The user interface may use the skill store management REST API to browse skills in the skill store and install skills in the local environment. For example, the skill store management REST API may list skills in response to a simple list request, list all robots in the robot's environment, list recent robots in the skill store, and identify outdated robots from the skill store that were previously imported into the robot's environment. The skill store management REST API may list skills by paging or sorting. The skill store management REST API may list skills filtered by name, description, summary, category, etc. In some embodiments, the skill store management REST API may list all versions of a skill with the same name. The skill store management REST API can allow downloading skills from the skill store using a download link path, downloading and installing skills from the skill store, deleting skills in the skill store by name and version, etc. In some embodiments, the skill store REST API can include a skill store upload (push) REST API, a skill store deleting (delete) REST API, a skill store install (pull) REST API, a skill store install all digital assistant skills (pull) REST API, etc.

[0184] In some embodiments, the skill store can be an object repository where any type of data, regardless of content type, can be stored as an object. An object can include the object itself and its metadata. Each object can be stored in a bucket. A bucket is a logical container for storing objects. Users or the system can create buckets as needed. Buckets can be associated with a single compartment, which has policies that determine what actions users can perform on the bucket and the objects within it. The skill store can include a namespace, which is a logical entity that serves as the top-level container for all buckets and objects, allowing users to control bucket naming within their tenancy. Each tenancy can be assigned a unique, but non-editable, object storage namespace that is global, spanning all compartments and regions. Bucket names can be unique within a tenancy. Within the object storage namespace, buckets and objects can exist in a flat hierarchy or arranged into a directory structure to facilitate navigation across a large group of objects (e.g., bots / skills / billing, bots / skills / crm, etc.). A compartment can be a collection of related resources accessible only to those explicitly granted access by an administrator. Compartments help users organize resources and control access to those resources. When compartments are provisioned, the skill store can include a root compartment. Administrators can create additional compartments within the root compartment and add access rules for those compartments. Buckets can only exist in one compartment.

[0185] In some embodiments, each skill can be placed in a container with a name such as " / bots / {uuid}.zip". After uploading the skills as files to the skill store, corresponding metadata can be created for each skill. The metadata file can be in JSON format and can have a name such as " / bots / {name} / {version} / bot.json". The following shows an example of metadata for a skill:

[0186]

[0187] III. Skill Expansion

[0188] As described above, each skill can be defined by a set of artifacts (e.g., entities) that represent the metadata required to configure, train, and execute the skill. Skills for specific functions can be built by a Software as a Service (SaaS) provider to allow SaaS customers to download and use them. In many cases, skills may need to be customized and extended. For example, a customer may want to customize and / or extend a factory-built skill to tune the skill to specific features, processes, terminology, culture, etc. A robotic extensibility infrastructure that can provide extensibility of skills in a holistic, general, flexible, secure, and maintainable manner can be desired.

[0189] In accordance with certain embodiments, a base skill may be customized or extended using bot extensions defined in a JSON file. A bot extension may include changes to be made to a base skill and may be applied to a base skill to customize or extend the base skill. A base skill includes raw metadata to be customized and / or extended via an extension. Customization may refer to changing one or more properties of an existing metadata artifact or resource for a skill. Extension may refer to enhancing existing metadata or adding new artifacts or resources to a skill's metadata. Depending on the granularity of the changes applied, customizations and extensions may overlap. For clarity, simplicity, and readability, both customizations and extensions are referred to herein as "extensions."

[0190] All metadata resources for a skill can be extended. These metadata resources can include, for example, (1) the top-level skill definition and its configuration and settings; (2) intents (which can be created, modified, or disabled); (3) entities (which can be created, modified, or disabled); (4) utterances (which can be created, modified, or disabled); (5) custom components (e.g., which can be created or modified); (6) conversation flows (which can be modified); and (7) resource bundles (e.g., to add new message keys and default language messages, add new supported languages, or add / modify translated messages). In various embodiments, different combinations of these metadata artifacts can be allowed to be extended. In some embodiments, some metadata artifacts can be disabled. Extensions are not allowed to remove or delete any metadata artifacts associated with the base skill.

[0191] The Robot extensibility infrastructure disclosed herein can support extensions of any published version of a skill (i.e., a base skill) through a management web application and / or REST API (such as Application Composer for Oracle Fusion Applications or a Command Line Interface (CLI) tool for Robots). Any artifact or resource in the skill metadata can be extensible. The infrastructure can allow clients to obtain, on demand, a representation of the differences between an extension and its corresponding base skill. The metadata associated with an extension can optionally support the definition of a test suite that can be used by the infrastructure to validate the extension when necessary. The solution can adhere to the versioning capabilities of Robots, where each extension can have its own version separate from the version of its base skill, so that the base skill and the extension can evolve separately. The version of the extension can keep track of the range of base skill versions that the extension version is compatible with.

[0192] The robot extensibility infrastructure can support upgrades to base skills and can determine changes between different versions of the base skills and changes to be made to the upgraded metadata of the upgraded versions of the base skills. After upgrading a base skill to a newer version, the extension can still be applied to its base skills as long as (1) metadata compatibility is preserved, (2) no metadata change conflicts are detected or pending resolution, and (3) validation tests (if defined by the extension) are successfully performed on the extended skills. To support upgradability, the infrastructure can store and track versions of extension metadata separately from the base skills to which the extension can be applied. This can allow a particular version of an extension to be valid and applicable throughout multiple upgrades or versions of its base skills, as long as those upgrades do not break the backward compatibility described above.

[0193] The infrastructure may perform any validation tests defined by the extension. The infrastructure may provide meaningful feedback to identify any issues that may cause the extension's skills to become invalid as its corresponding base skills are upgraded. When an upgrade to a base skill breaks compatibility with an extension, the infrastructure may be able to detect the incompatibility early on. The infrastructure may make a best-effort effort to automatically resolve metadata conflicts during upgrades. In some embodiments, the infrastructure may provide meaningful feedback to the client / user about any issues that the infrastructure cannot automatically resolve. The infrastructure may abort upgrades, automatically disable incompatible extensions, provide user-friendly mechanisms to assist users with manual / interactive conflict resolution, and allow users to publish new versions of extensions when needed.

[0194] According to certain embodiments, robot extensions can, in most cases, be implemented by storing and managing only the metadata pertaining to the extension separately from the original base metadata being extended (e.g., storing only the differences or changes between the base skill and the extended skill). Then, when the extension is applied, the differences or changes can be used to modify the base metadata. For example, the differences between two versions of the same skill can be calculated and stored in, for example, a JSON file, and the base skill can later be extended using the stored differences by modifying the metadata according to the JSON file to generate an extended skill from the base skill.

[0195] In some embodiments, multiple extensions can be applied on top of a base skill in an ordered stack (e.g., layers), where one layer (e.g., one extension) can "extend" on top of another layer (i.e., another extension or base skill). However, using multiple extension layers can exponentially increase the overall complexity. Therefore, in some embodiments, a single-layer solution for robot extensions can be used for most (if not all) use cases. In some embodiments, the design and implementation of the infrastructure can be scalable to allow new extension layers to be added when necessary without having to redesign or significantly modify or restructure the infrastructure.

[0196] There are several ways to implement these features. In some embodiments, metadata for extensions and skills can be stored in a database in JSON format. For example, a metadata artifact for a base skill can be described in a JSON file and stored in the skill store, and JSON extensions (e.g., RFC-6902 summarized at https: / / en.wikipedia.org / wiki / JSON_Patch) can be used to capture the differences between the base metadata artifact and its extensions.

[0197] In a simplified example, some of the resources or metadata artifacts of a base skill may be described in the raw documentation of a JSON file as:

[0198]

[0199] Extensions that represent the differences or changes between a base skill and an extended skill can be described in an extension document in a JSON file (referred to as a JSON extension document) as follows:

[0200]

[0201] Therefore, applying a JSON-formatted extension document to an original document may result in a new document (for example, a JSON file describing some resources of the extended skill):

[0202]

[0203] Among them, the value of resource "baz" (e.g., "qux") may be replaced by a new value (e.g., "boo"), resource "foo" and its value may be removed or disabled, and a new resource "hello" and its value may be added.

[0204] JSON extension documents can be "semantic" difference documents that identify the differences between the extended metadata and the original metadata. JSON extension documents can also be used for conflict resolution. Because the metadata for the base skills in the skill store is stored in JSON documents, there will be no need to introduce a new metadata format and there will be no need for format conversion. Therefore, the JSON extension documents used to store skill extensions are compatible with the existing metadata model for skills, thereby minimizing the impact on the existing code base (especially the model abstraction layer). JSON extensions will have minimal storage requirement growth and will allow the use of existing open source Java libraries (e.g., JSON extensions and Jackson) to implement the skill extension infrastructure described in this article.

[0205] Typically, custom code may not be part of a skill's metadata. In some embodiments, a skill's metadata may include one or more custom components. A custom component may include references and parameters that may allow the skill to execute the custom code. Thus, a skill extension may include the addition and / or modification of a custom component, while the custom code itself may not be included in and stored with the skill extension's metadata. If a skill extension requires modification of custom code associated with a custom component, the metadata for the custom component may be customized in the skill extension's metadata so that the extended skill may correctly reference the endpoint that may execute the desired custom code. The custom code itself may be provided through the mechanisms used to develop any other conventional skill.

[0206] In some embodiments, the robot system can include an internal management service and a REST API. The REST API can expose the robot's metadata as REST resources and can allow create, read, update, and delete (CRUD) operations on the robot's metadata. Because it is desirable to keep the extension model as close as possible to the skill's metadata model, it can be desirable to use the management REST API to provide programmatic access to extensibility features.

[0207] The techniques disclosed herein allow clients to interact with extensions and regular bots as seamlessly as possible. They also minimize the impact on existing UI code and simplify the implementation of extensibility in the UI, as no new REST APIs need to be adopted. The techniques disclosed herein can also be used with other application management databases such as MongoDB, where different versions of an application or extension can be stored by storing only the differences between the application and the original application version or extension in a JSON document.

[0208] Figure 5 A system block diagram depicts an example of a robot extensibility infrastructure 500 according to certain embodiments. The robot extensibility infrastructure 500 can include a metadata repository 510 that can store metadata 512 for base robots and metadata 514 for skill extensions. In some embodiments, the metadata repository 510 can be part of the skill store described above.

[0209] The robot extensibility infrastructure 500 may also include one or more robot systems 520, each of which may include a management service application 522 and a runtime application 524. The management service application 522 may be used to create extensions, store extensions, track versions of skills and extensions, check compatibility between extensions and base skills, and rebase extensions to different versions of the base skill. At a robot developer device 542 connected to the management service application 522 via a secure link 550, a GUI 534, and a REST API 532, a robot developer may create extensions to base skills, store extensions in a metadata repository 510, track versions of base skills and extensions to base skills, check compatibility between extension metadata 514 and base skill metadata 512, or rebase extensions to different versions of the base skill. In some embodiments, the creation and management of extensions may be performed using the management service application 522 and the REST API 532, for example, via a CLI tool 544, a software development kit (SDK) 548, or other system 546 that may connect to the REST API 532 via a secure link 552.

[0210] To execute the extended skill, the robotic system 520 can obtain metadata 512 of the base skill and metadata 514 of the extension from the metadata repository 510, and apply metadata 514 to metadata 512 to generate new parsed metadata 526. The runtime application 524 can use the parsed metadata 526 and the implementation of the base skill to execute the extended skill. For example, the runtime application 524 can apply the metadata to the implementation of the base skill and communicate with the end user via the REST API 528 and various channels 530.

[0211] As described above, the bot extensibility infrastructure can store and track versions of bot extensions separately from the base skills to which they can be applied. This can allow a single version of a bot extension to be applicable across multiple upgrades or versions of a base skill, as long as the upgrades do not break backward compatibility as described above. This can also allow different extensions to be applied to the same base skill.

[0212] Figure 6 An example of tracking the version of a base robot and the version of a robot extension, and the compatibility between the version of the base robot and the version of the robot extension, in a robot extensibility structure according to some embodiments is illustrated. Figure 6 In the illustrated example, there may be four versions of the base robot, including version 1 (610), version 2 (612), version 3 (614), and version 4 (616). Version 1 (620) of the extension of the base robot may be created based on version 1 (610) of the base robot. A management service application, such as the management service application 522, may check the compatibility between the version of the robot extension and the version of the base robot. When an upgrade to a base skill breaks compatibility with the extension, the infrastructure may be able to detect the incompatibility early. Figure 6In the illustrated example shown, version 1 (620) of the extension to the base robot may be compatible with version 1 (610), version 2 (612), and version 3 (614) of the base robot. However, at 622, when version 4 (616) of the base robot is published in the skill store, the management service application may determine that version 1 (620) of the extension to the base robot is incompatible with version 4 (616) of the base robot. In some embodiments, the infrastructure (e.g., the management service application) may perform validation tests defined by the extension to determine compatibility or validity. In some embodiments, when incompatibility or invalidity is detected, the infrastructure may perform a best effort attempt at 624 to automatically resolve the compatibility issue. In some embodiments, if the compatibility issue is not resolved, the infrastructure may fail or abort the upgrade or may automatically disable the incompatible extension. In some embodiments, the infrastructure may provide feedback to the client / user about any issues that the infrastructure cannot automatically resolve, so that the client or user can manually resolve outstanding issues and publish a new version of the extension if necessary. For example, a new version 2 (626) of an extension to a base robot may be automatically or manually generated that may be compatible with version 4 (616) of the base robot.

[0213] IV. Skill Expansion GUI

[0214] Typically, extending a published skill will be as seamless as creating a new bot version. Figure 5 As described, developers can use a GUI (e.g., GUI 534) to create and manage extensions to base skills. An example of a GUI for robot extensions is described below.

[0215] According to some embodiments, a new skill extension may be created from a skill tile or from a context menu. Figure 7 An example of creating a new skill extension to a published skill using a graphical user interface (GUI) 700 according to some embodiments is illustrated. In the illustrated example, a user can select a skill 712 from a list 710 and select an action to apply to the skill 712 from a drop-down menu 720. The action may include, for example, creating a new skill, creating a new version of an existing skill, extending an existing skill, exporting a skill, etc.

[0216] Figure 8An example of creating a new skill extension to a published skill 810 using a context menu 818 in a GUI 800 according to some embodiments is illustrated. The GUI 800 shows a list of skills using multiple tiles, where each tile can show certain information about a skill, such as a brief description 812 of the skill 810, a training model 814, and an update time 816. The context menu 818 can be used to select an action to be applied to the skill 810, such as viewing the skill, creating a new version, extending the skill, exporting the skill, exporting the session log, deleting the skill, and the like.

[0217] Figure 9 An example of a dialog box in a GUI image 900 for creating an extended skill according to some embodiments is illustrated. The GUI image 900 shows the name and version of the base skill to be extended (e.g., "Agent Robot 1.0"). Figure 9 As shown, developers can use the Create Extended Skill dialog box to enter the extended skill's display name, name, version, and short description. The display name, name, version, and short description values for the selected skill can be edited. However, not all properties are editable in an extended skill. Once an extended skill is created, some properties may not be editable.

[0218] Figure 10 An example of a GUI image 1000 according to certain embodiments is illustrated, showing certain inherited components of a base robot and new components of an extended robot. Inherited items, such as the base robot's original intent 1010, may not be deleted but may be modified or customized. For example, the original intent's name, description, and example utterances may be edited. Example utterances may be added, edited, or removed. New components, such as new intent 1020, may be added to and edited in an extended skill.

[0219] Figure 11 An example of a GUI image 1100 is illustrated showing form fields that may be added, edited, or removed for expanding a skill, in accordance with some embodiments. Figure 11 In the example shown, the original Figure 1 The intent "check in" (1110) may be renamed. Utterances such as "back," "checking in," and "hello" may be added to the examples of utterances for the intent "check in." In some embodiments, the added or updated property in the UI (e.g., the description of the "check in" intent) may be highlighted, and the user may have the option to roll back the property value to its original state.

[0220] Figure 12An example of a GUI image 1200 is illustrated showing a "restore" icon 1210 for restoring the value of a field, such as an intent description, to its original value, according to some embodiments. By clicking the "restore" icon 1210, a dialog box 1220 can be displayed with the original value of the field and an option to restore the field to the original value.

[0221] Figure 13 An example of a GUI image 1300 is illustrated, according to some embodiments, showing a dialog box 1310 for reverting a field's value to its original value. Dialog box 1310 can be displayed after the developer selects "Revert" in dialog box 1220. The developer can use dialog box 1310 to cancel the revert or confirm the revert.

[0222] After the base skill is extended, developers can compare the changes made in the extension. Before generating a bot extension that describes the changes from the base skill to the extended skill, a graphical user interface can be used to compare the metadata of the base skill side by side with the metadata of the extended skill.

[0223] Figure 14 An example of a GUI image 1400 is illustrated showing the results of a comparison between a base skill and an extended skill in accordance with certain embodiments. In the illustrated example, the left window 1410 shows metadata for the base skill "Agent Robot" and the right window 1420 shows metadata for the extended skill "Extended Agent Robot." Developers can compare the metadata to determine the changes made in the extension. In some embodiments, the comparison can be done automatically using the Code Mirror merge feature. The Code Mirror merge feature can be enabled or disabled via the "Compare to Base" button. Figure 14 As shown, the differences or changes made may be highlighted in the right window 1420 .

[0224] As described above, in some embodiments, an extension to a base skill can be rebased onto a different version of the base skill. Rebasing is the act of reapplying the changes made in the extended skill on top of a new version of the base robot. For example, a developer may have created an extended robot (Pizza Robot 1.0) by extending base robot 1.0. Some time later, a new base robot 1.1 may be available with several bug fixes, and the developer may want to take advantage of the new fixes in base robot 1.1. This can be done by "rebasing" Pizza Robot 1.0 onto base robot 1.1. When rebasing, if there is more than one possible version available, the developer can select a base robot target version. By default, the latest base robot version can be selected as the target version.

[0225] Figure 15 An example of a GUI image 1500 is illustrated showing a dialog box for rebasing a skill extension in accordance with certain embodiments. As shown in GUI image 1500, a user may want to rebase an extension (e.g., "Extension Agent," version 1.0) to a base skill (e.g., "Agent Bot 1.0") to a newer version of the base skill (e.g., "Agent Bot 1.1"). The user may select the desired version from menu 1510 and click a "Rebase" button 1520. In some embodiments, if the extended skill is in a published state, the rebase action may force the developer to create a new extended skill version. If the extended skill is in a draft state, the extended skill may inherit all new features.

[0226] Figure 16 An example of a GUI image 1600 is illustrated, according to certain embodiments, showing a dialog box for creating a new version of a published extension skill (e.g., "Extension Agent Bot 1.0"). GUI image 1600 illustrates a requirement for a developer to create a new skill version, "Extension Agent Bot 1.1," for rebasing "Extension Agent Bot 1.0" onto a new version of a base skill ("Agent Bot 1.1"), since the extended skill "Extension Agent Bot 1.0" may already be published. Before incorporating the rebasing action, the developer may be presented with a dialog box to confirm the rebasing onto the new version of the base skill.

[0227] Figure 17 An example of a GUI image 1700 is illustrated showing a dialog box for confirming rebasing a skill extension to a new version of a base skill, in accordance with some embodiments. Figure 17 In the illustrated example, the developer may be asked to confirm that the developer wants to rebase the extended skill "Extended Agent Bot 1.0" to a new version of the base skill ("Agent Bot 1.1").

[0228] Figure 18 Illustrated is an example of a GUI image 1800 presenting information regarding rebasing a skill extension to a new version of a base skill, in accordance with some embodiments. Figure 18 The wizard-style dialog box shown is used to present information about a new base skill, such as "Agent Bot 1.1." The new base skill information may include, for example, a brief description, a detailed description, and a sample utterance. Based on the new base skill information, the developer can determine whether to proceed or revert. If the developer decides to proceed with rebasing, a list of differences between the previous base skill and the new base skill may be presented to the developer.

[0229] Figure 19An example of a GUI image 1900 according to some embodiments is illustrated, showing a list of changes from a previous base skill (e.g., "agent bot 1.0") to a new base skill (e.g., "agent bot 1.1") generated from the previous base skill. In the illustrated example, the new base skill (e.g., "agent bot 1.1") may include two new intents (e.g., "intents" and "intents"). Figure 1 ” and “ Figure 2 ”) and an updated intent (e.g., “Intent A”). The new base skill may not include “Entity 1” that may have been in the previous base skill. The developer may choose to proceed with the rebasing or go back. If the developer chooses to proceed with the rebasing, the developer may be presented with a GUI for comparing the differences between all skills involved in the rebasing process.

[0230] Figure 20 An example of a GUI image 2000 for comparing conversation flow content between all skills involved in a rebasing process is illustrated in accordance with certain embodiments. For example, the GUI can be used to perform a comparison between any two of a previous base skill (e.g., "agent bot 1.0"), a new base skill (e.g., "agent bot 1.1"), a previous extended skill (e.g., "extended agent bot 1.0"), and a new extended skill (e.g., "extended agent bot 1.1"). Figure 20 2010) and the metadata of the previously extended skill (e.g., "Extended Agent Bot 1.0" shown in the left window 2020). A "Preview" button 2030 may cause the newly extended skill to enter a transitional state, which may allow the developer to review all changes and run necessary tests before applying the rebase action. If any changes are made to the skill while it is in the transitional state, the developer may encounter a dialog box requiring the developer to create a new version. By clicking the "Cancel" button, the transitional state disappears and all changes are reverted to the previous skill state.

[0231] Figure 21 An example of a GUI image 2100 is illustrated showing a dialog box for confirming or canceling a rebase of a skill extension according to some embodiments. The developer can review the information related to the rebase and choose to confirm or cancel the rebase.

[0232] In some embodiments, skills can include custom widgets. Published skills can implement custom widgets of the "Embedded Container" type. Users can edit metadata such as the name and description and / or replace the existing package file for the custom widget. Undo operations can also be supported on these fields.

[0233] Figure 22 An example of a GUI image 2200 is shown, according to certain embodiments, illustrating an example of a custom component of the "Embedded Container" type. GUI image 2200 may display, for example, the name, description, status, version, service type, package file, and the like of the "Embedded Container" component. Developers can change the package file by selecting "Change" button 2220. In GUI image 2200, the "Enable Service" switch 2210 may be disabled, and the "Delete" button may be omitted.

[0234] In some embodiments, developers can also add new services. Figure 23 An example of a GUI image 2300 for creating a new service according to some embodiments is illustrated. The service can be an external service, an embedded container service, an Oracle Mobile Cloud service, etc. The developer can specify a name, description, type, metadata location, verification information, etc. for the new service.

[0235] In some embodiments, an icon may be used to notify that the skill is an expanded skill and / or to signal whether a new base bot update is available. Figure 24A An example of a GUI image 2400 according to some embodiments is illustrated, including an icon 2410 indicating that the skill is an expanded skill and that a new base skill update is available. If the skill has a pending rebase, a different icon must be used. For example, Figure 24B An example of a GUI image 2450 is illustrated in accordance with some embodiments, wherein the icon 2410 indicates that the skill is an expanded skill and has a pending rebase.

[0236] Figure 25 25 is a simplified flowchart 2500 according to certain embodiments, illustrating an example of a process for generating extended skills. At 2510, a computer system may obtain a first JSON file including metadata of a first application from a database that stores metadata of multiple applications in JSON files. The multiple applications may include a chatbot application. In some embodiments, the multiple applications may be stored in a database as compressed files. In some embodiments, the multiple applications may include two or more versions of the first application. In some embodiments, the metadata of the first application may include metadata for at least one of: configuring the first application, training the first application, testing the first application, or executing the first application. In some embodiments, the metadata of the first application may include metadata for at least one of the intents, entities, utterances, custom components, or conversation flows of the first application.

[0237] At 2520, the computer system may receive modifications to the first application, such as via a REST API, to generate an extended application. The modifications may include customizing and / or extending the first application to adapt the first application to specific characteristics, processes, terminology, culture, etc. Customization may include changing one or more properties of existing metadata artifacts or resources of the first application. Extensions may include enhancing existing metadata or adding new artifacts or resources to the metadata of the first application.

[0238] At 2530, the computer system may determine a difference between the metadata of the extended application and the metadata of the first application. At 2540, the computer system may store a second JSON file in a database, the second JSON file describing the changes made to the metadata of the first application to generate the metadata of the extended application. In some embodiments, the second JSON file may be compatible with at least two versions of the first application for use in extending the at least two versions of the first application. In some embodiments, the second JSON file may be incompatible with at least one of the two or more versions of the first application, and the database may include a third JSON file that is compatible with the at least one of the two or more versions of the first application, wherein the second JSON file and the third JSON file may have different version numbers. In some embodiments, the second JSON file may include metadata for custom components in the extended application.

[0239] Figure 26 is a simplified flowchart 2600 according to certain embodiments, illustrating an example of a process for implementing an expanded skill. At 2610, a computer system may obtain a first application from a database storing a plurality of applications. The first application may include implementation data for the first application and a first JSON file including metadata associated with the first application. The plurality of applications may include a chatbot application. The metadata associated with the first application may include metadata for at least one of: configuring the first application, training the first application, testing the first application, or executing the first application. The metadata associated with the first application may also include metadata for at least one of an intent, an entity, an utterance, a custom component, or a conversation flow of the first application.

[0240] At 2620, the computer system may obtain a second JSON file associated with an application that is an extension of the first application from the database, wherein the second JSON file may describe changes to the first JSON file. In some embodiments, the database may include a first repository storing implementation data of the plurality of applications and a second repository storing the first JSON file and the second JSON file.

[0241] At 2630, the computer system may apply the changes to the first JSON file described in the second JSON file to generate metadata associated with the extended application. In some embodiments, applying the changes to the first JSON file described in the second JSON file may include: determining that the second JSON file conflicts with the first JSON file; and resolving one or more conflicts between the second JSON file and the first JSON file.

[0242] At 2640, the computer system may implement the extended application based on the implementation data of the first application and the metadata associated with the extended application. In some embodiments, the computer system may also obtain a third JSON file from a database (the third JSON file describing changes to the metadata associated with the extended application), apply the changes to the metadata associated with the extended application described in the third JSON file to generate metadata associated with the second extended application, and implement the second extended application based on the implementation data of the first application and the metadata associated with the second extended application. In some embodiments, the computer system may also obtain a second application from a database (the second application includes the implementation data of the second application and the third JSON file including metadata associated with the second application), apply the changes to the first JSON file described in the second JSON file to the third JSON file to generate metadata associated with the second extended application, and implement the second extended application based on the implementation data of the second application and the metadata associated with the second extended application.

[0243] although Figure 25 and Figure 26 The operations may be described as a sequential process, but many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. The process may have additional steps not included in the figure. In addition, embodiments of the method may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the associated tasks may be stored in a computer-readable medium such as a storage medium. One or more processors may perform the associated tasks.

[0244] System Example

[0245] Figure 27 A simplified diagram of a distributed system 2700 for implementing some embodiments is depicted. In the illustrated example, the distributed system 2700 includes one or more client computing devices 2702, 2704, 2706, and 2708 coupled to a server 2712 via one or more communication networks 2710. The client computing devices 2702, 2704, 2706, and 2708 can be configured to execute one or more applications.

[0246] In various examples, server 2712 can be adapted to run one or more services or software applications that implement one or more embodiments described in the present disclosure. In some examples, server 2712 can also provide other services or software applications that can include non-virtual environments and virtual environments. In some examples, these services can be provided to users of client computing devices 2702, 2704, 2706, and / or 2708 as web-based services or cloud services (such as under a software-as-a-service (SaaS) model). Users operating client computing devices 2702, 2704, 2706, and / or 2708 can then utilize one or more client applications to interact with server 2712 to utilize the services provided by these components.

[0247] exist Figure 27 In the depicted configuration, the server 2712 may include one or more components 2718, 2720, and 2722 that implement the functions performed by the server 2712. These components may include software components that may be executed by one or more processors, hardware components, or a combination thereof. It should be understood that a variety of different system configurations are possible that may differ from the distributed system 2700. Thus, Figure 27 The illustrated example is one example of a distributed system for implementing the example system and is not intended to be limiting.

[0248] The user may use client computing devices 2702, 2704, 2706, and / or 2708 to execute one or more applications that may generate one or more storage requests that may then be served content according to the present disclosure. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user through this interface. Although Figure 27 Only four client computing devices are depicted, but any number of client computing devices can be supported.

[0249] Client devices can include various types of computing systems, such as portable handheld devices, general-purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computing devices can run various types and versions of software applications and operating systems (e.g., Microsoft Apple or UNIX-like operating systems, such as Google Chrome TM OS, Linux or Linux-like operating systems), including various mobile operating systems (e.g., Microsoft Windows Windows Android TM 、 Palm Portable handheld devices may include cellular phones, smart phones (e.g., ), tablet computers (e.g. ), personal digital assistants (PDAs), etc. Wearable devices may include Google Head-mounted displays and other devices. Gaming systems can include various handheld gaming devices, Internet-enabled gaming devices (e.g., with or without Microsoft Gesture Input Devices Game consoles, Sony System, by Various game systems provided, among others), etc. The client device may be capable of executing various different applications, such as various Internet-related applications, communication applications (e.g., email applications, short message service (SMS) applications) and may use various communication protocols.

[0250] The communication network(s) 2710 may be any type of network familiar to those skilled in the art that supports data communications using any of a variety of available protocols including, but not limited to, TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internet Packet Exchange), By way of example only, the communication network(s) 2710 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suite, and / or any other wireless protocol operating network) and / or any combination of these networks and / or other networks.

[0251] The server 2712 may be composed of one or more general-purpose computers, dedicated server computers (including, by way of example, PC (personal computer) servers, Servers, mid-range servers, mainframe computers, rack servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. Server 2712 may include one or more virtual machines running virtual operating systems or other computing architectures involving virtualization (e.g., logical storage devices may be virtualized to maintain one or more flexible pools of virtual storage devices for the server). In various examples, server 2712 may be adapted to run one or more services or software applications that provide the functionality described in the foregoing disclosure.

[0252] The computing system in the server 2712 can run one or more operating systems, including any of those discussed above and any commercially available server operating system. The server 2712 can also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transport Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include but are not limited to (International Business Machines Corporation) and others.

[0253] In some implementations, server 2712 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client computing devices 2702, 2704, 2706, and 2708. By way of example, data feeds and / or event updates may include, but are not limited to: feed, The server 2712 may also include one or more applications to display the data feeds and / or real-time events via one or more display devices of the client computing devices 2702 , 2704 , 2706 , and 2708 .

[0254] The distributed system 2700 may also include one or more data repositories 2714, 2716. In some examples, these data repositories may be used to store data and other information. For example, one or more of the data repositories 2714, 2716 may be used to store information such as information related to storage virtual machines, information that maps application IDs to applications of selected storage virtual machines, and other information used by the server 2712 when performing authentication functions. The data repositories 2714, 2716 may reside in various locations. For example, the data repository used by the server 2712 may be local to the server 2712 or may be remote from the server 2712 and communicate with the server 2712 via a network-based or dedicated connection. The data repositories 2714, 2716 may be of different types. In some examples, the data repository used by the server 2712 may be a database, such as a relational database, such as that provided by Oracle and databases provided by other suppliers. One or more of these databases can be adapted to respond to SQL format commands to implement storage, update and retrieval of data into and from the database.

[0255] In some examples, one or more of data repositories 2714, 2716 may also be used by applications to store application data. The data repositories used by applications may be of different types, such as, for example, a key-value store, an object store, or a general-purpose storage store backed by a file system.

[0256] In some examples, the functionality described in this disclosure may be provided as a service via a cloud environment. Figure 28 2800 is a simplified block diagram of a cloud-based system environment system 2800 for implementing some embodiments. In the cloud-based system environment system 2800, various services can be provided as cloud services according to certain examples. Figure 28 In the depicted example, cloud infrastructure system 2802 can provide one or more cloud services that can be requested by users using one or more client computing devices 2804, 2806, and 2808. Cloud infrastructure system 2802 can include one or more computers and / or servers, which can include those described above with respect to server 1612. The computers in cloud infrastructure system 2802 can be organized as general-purpose computers, dedicated server computers, server farms, server clusters, or any other suitable arrangement and / or combination.

[0257] Network(s) 2810 can facilitate data communication and exchange between client computing devices 2804, 2806, and 2808 and cloud infrastructure system 2802. Network(s) 2810 can include one or more networks. The networks can be of the same or different types. Network(s) 2810 can support one or more communication protocols (including wired and / or wireless protocols) to facilitate communication.

[0258] Figure 28 The depicted example is only one example of a cloud infrastructure system and is not intended to be limiting. It should be understood that in other examples, the cloud infrastructure system 2802 may have more than Figure 28 Those components depicted may have more or fewer components, may combine two or more components, or may have a different configuration or arrangement of components. For example, although Figure 28 Three client computing devices are depicted, but in alternative examples any number of client computing devices may be supported.

[0259] The term cloud service is generally used to refer to services that are made available to users on demand and via a communication network such as the Internet through a service provider's system (e.g., cloud infrastructure system 2802). Typically, in a public cloud environment, the servers and systems that make up the cloud service provider's system are different from the customer's own on-premises servers and systems. The cloud service provider's system is managed by the cloud service provider. Thus, customers can make use of the cloud services provided by the cloud service provider without having to purchase separate licenses, support, or hardware and software resources for the services. For example, the cloud service provider's system can host applications, and users can order and use the applications on demand over the Internet without having to purchase the infrastructure resources for executing the applications. Cloud services are designed to provide easy, scalable access to applications, resources, and services. Several providers offer cloud services. For example, Oracle of Redwood Shores, California It provides several cloud services such as middleware services, database services, Java cloud services and other services.

[0260] In some examples, cloud infrastructure system 2802 can provide one or more cloud services using different models, such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), and other models (including hybrid service models). Cloud infrastructure system 2802 can include a set of applications, middleware, databases, and other resources that enable the provision of various cloud services.

[0261] The SaaS model enables applications or software to be delivered to customers as a service over a communication network such as the Internet, without the customer having to purchase the hardware or software for the underlying application. For example, the SaaS model can be used to provide customers with access to on-demand applications hosted by a cloud infrastructure system 2802. Oracle Examples of SaaS services provided include, but are not limited to, various services for human resource / capital management, customer relationship management (CRM), enterprise resource planning (ERP), supply chain management (SCM), enterprise performance management (EPM), analytical services, social applications, and others.

[0262] The IaaS model is commonly used to provide infrastructure resources (e.g., servers, storage devices, hardware, and networking resources) to customers as cloud services to provide elastic computing and storage capabilities. Provides various IaaS services.

[0263] The PaaS model is typically used to provide platform and environment resources as a service that enable customers to develop, run, and manage applications and services without having to purchase, build, or maintain such resources. Examples of PaaS services provided include, but are not limited to, Oracle Java Cloud Service (JCS), Oracle Database Cloud Service (DBCS), Data Management Cloud Service, various application development solution services, and other services.

[0264] Cloud services are typically provided on an on-demand, self-service basis, based on subscriptions, in an elastically scalable, reliable, highly available, and secure manner. For example, a customer may order one or more services provided by cloud infrastructure system 2802 through a subscription order. Cloud infrastructure system 2802 then performs processing to provide the services requested in the customer's subscription order. For example, a user may request the cloud infrastructure system to register an application as described above and provide services to the application according to the application's specified requirements. Cloud infrastructure system 2802 may be configured to provide one or more cloud services.

[0265] The cloud infrastructure system 2802 can provide cloud services through different deployment models. In a public cloud model, the cloud infrastructure system 2802 can be owned by a third-party cloud service provider, and the cloud services are provided to any general public customer, wherein the customer can be an individual or a business. In some other examples, under a private cloud model, the cloud infrastructure system 2802 can be operated within an organization (e.g., within an enterprise organization) and provide services to customers within the organization. For example, customers can be various departments of an enterprise such as the human resources department, the payroll department, or even individuals within the enterprise. In some other examples, under a community cloud model, the cloud infrastructure system 2802 and the services provided can be shared by several organizations in a related community. Various other models, such as a hybrid of the models mentioned above, can also be used.

[0266] Client computing devices 2804, 2806, and 2808 can be of different types and can be capable of operating one or more client applications. A user can use a client device to interact with the cloud infrastructure system 2802, such as requesting services provided by the cloud infrastructure system 2802. For example, a user can use a client device to request authentication-related services as described herein.

[0267] In some examples, the processing performed by cloud infrastructure system 2802 to provide services may involve big data analysis. This analysis may involve using, analyzing, and manipulating large data sets to detect and visualize various trends, behaviors, relationships, etc. in the data. This analysis may be performed by one or more processors, thereby potentially processing data in parallel, performing simulations using the data, and the like. For example, big data analysis may be performed by cloud infrastructure system 2802 to determine which storage virtual machine to select for a particular application based on the application's declared certification-related requirements. The data used for this analysis may include structured data (e.g., data stored in a database or structured according to a structured model) and / or unstructured data (e.g., data blocks (binary large objects)).

[0268] like Figure 28 As depicted in the example of , cloud infrastructure system 2802 may include infrastructure resources 2830 used to facilitate the provision of various cloud services provided by cloud infrastructure system 2802. Infrastructure resources 2830 may include, for example, processing resources, storage or memory resources, networking resources, etc. In some examples, a storage virtual machine that can be used to service storage requests from applications may be part of cloud infrastructure system 2802. In other examples, the storage virtual machine may be part of a different system.

[0269] In some examples, to facilitate efficient provisioning of these resources for use in supporting the various cloud services provided by the cloud infrastructure system 2802 for different customers, the resources may be bundled into resource groups or resource modules (also referred to as "pods"). Each resource module or pod may include a pre-integrated and optimized combination of one or more types of resources. In some examples, different pods may be pre-provisioned for different types of cloud services. For example, a first group of pods may be provisioned for a database service, a second group of pods may be provisioned for a Java service (the second group of pods may include a different combination of resources than the pods in the first group), and so on. For some services, the resources allocated for provisioning a service may be shared between services.

[0270] The cloud infrastructure system 2802 itself may internally use services 2832 that are shared by different components of the cloud infrastructure system 2802 and that facilitate the provision of services by the cloud infrastructure system 2802. These internal shared services may include, but are not limited to, security and identity services, integration services, enterprise repository services, enterprise manager services, virus scanning and whitelisting services, high availability, backup and recovery services, services for implementing cloud support, email services, notification services, file transfer services, and the like.

[0271] The cloud infrastructure system 2802 may include multiple subsystems. These subsystems may be implemented in software or hardware or a combination thereof. Figure 28 As depicted, the subsystem may include a user interface subsystem 2812 that enables users or customers of cloud infrastructure system 2802 to interact with cloud infrastructure system 2802. User interface subsystem 2812 may include various interfaces, such as a web interface 2814, an online store interface 2816 (where cloud services provided by cloud infrastructure system 2802 are advertised and available for purchase by customers), and other interfaces 2818. For example, a customer may use a client device to request (service request 2834) one or more services provided by cloud infrastructure system 2802 using one or more of interfaces 2814, 2816, and 2818. For example, a customer may access the online store, browse cloud services provided by cloud infrastructure system 2802, and place a subscription order for one or more services provided by cloud infrastructure system 2802 to which the customer wishes to subscribe. The service request may include information identifying the customer and the one or more services to which the customer wishes to subscribe. For example, a customer may place a subscription order for services provided by cloud infrastructure system 2802. As part of the order, the customer may provide information identifying the application for which the service is to be provided and one or more credentials for the application.

[0272] In some examples (such as Figure 28In the example depicted, cloud infrastructure system 2802 may include an order management subsystem (OMS) 2820 configured to process new orders. As part of this processing, OMS 2820 may be configured to: create an account for the customer (if one has not already been created); receive billing and / or billing information from the customer to be used to bill the customer for providing the requested service to the customer; verify the customer information; after verification, book the order for the customer; and orchestrate various workflows to prepare the order for fulfillment.

[0273] Once properly authenticated, the OMS 2820 may then call an order provisioning subsystem (OPS) 2824 that is configured to provision resources (including processing resources, memory resources, and networking resources) for the order. Provisioning may include allocating resources to the order and configuring the resources to facilitate the services requested by the customer order. The manner in which resources are provisioned for the order and the type of resources provisioned may depend on the type of cloud service that the customer has ordered. For example, according to a workflow, the OPS 2824 may be configured to determine the specific cloud service being requested and identify the number of groups that may have been pre-configured for the specific cloud service. The number of groups allocated to the order may depend on the size / volume / tier / scope of the requested service. For example, the number of groups to be allocated may be determined based on the number of users to be supported by the service, the duration of the service being requested, etc. The allocated groups may then be customized for the specific requesting customer to provide the requested service.

[0274] In some examples, the setup phase processing described above can be performed as part of the provisioning process by cloud infrastructure system 2802. Cloud infrastructure system 2802 can generate an application ID and select a storage virtual machine for the application from storage virtual machines provided by cloud infrastructure system 2802 itself or from storage virtual machines provided by systems other than cloud infrastructure system 2802.

[0275] Cloud infrastructure system 2802 may send a response or notification 2844 to the requesting customer indicating that the requested service is now ready for use. In some instances, information (e.g., a link) may be sent to the customer enabling the customer to begin using and utilizing the benefits of the requested service. In some examples, the response may include, for the customer requesting the service, an application ID generated by cloud infrastructure system 2802 and information identifying a virtual machine selected by cloud infrastructure system 2802 for the application corresponding to the application ID.

[0276] Cloud infrastructure system 2802 can provide services to multiple customers. For each customer, cloud infrastructure system 2802 is responsible for managing information related to one or more subscription orders received from the customer, maintaining customer data related to the order, and providing the requested services to the customer. Cloud infrastructure system 2802 can also collect usage statistics regarding the customer's use of the subscribed services. For example, statistics can be collected on the amount of storage used, the amount of data transferred, the number of users, and the amount of system uptime and downtime. This usage information can be used to bill the customer. Billing can be performed, for example, on a monthly basis.

[0277] Cloud infrastructure system 2802 can provide services to multiple customers in parallel. Cloud infrastructure system 2802 can store information about these customers (possibly including proprietary information). In some examples, cloud infrastructure system 2802 includes an identity management subsystem (IMS) 2828 configured to manage customer information and provide separation of managed information so that information related to one customer cannot be accessed by another customer. IMS 2828 can be configured to provide various security-related services such as identity services, which may include, for example, information access management, authentication and authorization services, services for managing customer identities and roles, and related capabilities.

[0278] Figure 29 An example of a computer system 2900 for implementing some embodiments is shown. In some examples, the computer system 2900 can be used to implement any of the application systems, access management systems, systems within a data center, and various servers and computer systems described above. Figure 29 As shown, computer system 2900 includes various subsystems, including a processing subsystem 2904 that communicates with multiple other subsystems via a bus subsystem 2902. These other subsystems may include a processing acceleration unit 2906, an I / O subsystem 2908, a storage subsystem 2918, and a communication subsystem 2924. The storage subsystem 2918 may include non-transitory computer-readable storage media, including computer-readable storage media 2922 and system memory 2910.

[0279] The bus subsystem 2902 provides a mechanism for making the various components and subsystems of the computer system 2900 communicate with each other as expected. Although the bus subsystem 2902 is schematically shown as a single bus, the alternative example of the bus subsystem can utilize multiple buses. The bus subsystem 2902 can be any of several types of bus structures including a memory bus or a memory controller, a peripheral bus, a local bus, etc. using any of a variety of bus architectures. For example, this architecture can include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standards association (VESA) local bus, and a peripheral component interconnect (PCI) bus (the PCI bus can be implemented as a mezzanine bus manufactured into an IEEE P1386.1 standard).

[0280] The processing subsystem 2904 controls the operation of the computer system 2900 and may include one or more processors, application specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs). The processor may include a single-core processor or a multi-core processor. The processing resources of the computer system 2900 may be organized into one or more processing units 2932, 2934, etc. A processing unit may include one or more processors, one or more cores from the same or different processors, a combination of cores and processors, or other combinations of cores and processors. In some examples, the processing subsystem 2904 may include one or more dedicated coprocessors such as a graphics processor, a digital signal processor (DSP), etc. In some examples, some or all of the processing units of the processing subsystem 2904 may be implemented using custom circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs).

[0281] In some examples, the processing unit in the processing subsystem 2904 can execute instructions stored in the system memory 2910 or on the computer-readable storage medium 2922. In various examples, the processing unit can execute various programs or code instructions and can maintain multiple programs or processes that are executed simultaneously. At any given time, some or all of the program code to be executed can reside in the system memory 2910 and / or on the computer-readable storage medium 2922 (potentially including residing on one or more storage devices). Through suitable programming, the processing subsystem 2904 can provide the various functions described above. In an instance where the computer system 2900 executes one or more virtual machines, one or more processing units can be assigned to each virtual machine.

[0282] In some examples, a processing acceleration unit 2906 may optionally be provided for performing custom processing or for offloading some of the processing performed by the processing subsystem 2904 , thereby speeding up the overall processing performed by the computer system 2900 .

[0283] The I / O subsystem 2908 may include devices and mechanisms for inputting information to the computer system 2900 and / or for outputting information from or through the computer system 2900. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information to the computer system 2900. User interface input devices may include, for example, a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may also include motion sensing and / or gesture recognition devices, such as Microsoft Windows, that enable a user to control and interact with the input device. Motion sensor, Microsoft 240 Game controller, a device that provides an interface for receiving input using gestures and verbal commands. The user interface input device may also include an eye gesture recognition device, such as detecting eye activity from the user (e.g., "blinking" when taking a picture and / or making a menu selection) and converting eye gestures into input devices (e.g., Google ) In addition, the user interface input device may include an input device that enables the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0284] Other examples of user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing rods, game controllers and graphics tablets and audio / video devices (such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye tracking devices). In addition, user interface input devices can include, for example, medical imaging input devices, such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound equipment. User interface input devices can also include, for example, audio input devices, such as MIDI keyboards, digital musical instruments, etc.

[0285] In general, the term output device is used to include all possible types of devices and mechanisms for outputting information from the computer system 2900 to a user or another computer. User interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices. A display subsystem may be a cathode ray tube (CRT), a flat panel device (such as one using a liquid crystal display (LCD) or plasma display), a projection device, a touch screen, etc. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, sound output devices, and modems.

[0286] The storage subsystem 2918 provides a repository or data store for storing information and data used by the computer system 2900. The storage subsystem 2918 provides a tangible, non-transitory, computer-readable storage medium for storing basic programming and data constructs that provide some example functionality. The storage subsystem 2918 can store software (e.g., programs, code modules, instructions) that, when executed by the processing subsystem 2904, provide the functionality described above. The software can be executed by one or more processing units of the processing subsystem 2904. The storage subsystem 2918 can also provide authentication according to the teachings of the present disclosure.

[0287] The storage subsystem 2918 may include one or more non-transitory memory devices, including volatile memory devices and non-volatile memory devices. Figure 29 As shown, the storage subsystem 2918 includes system memory 2910 and computer-readable storage media 2922. The system memory 2910 may include multiple memories, including volatile main random access memory (RAM) for storing instructions and data during program execution and non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some embodiments, the basic input / output system (BIOS), which contains basic routines that help transfer information between elements within the computer system 2900 during startup, may typically be stored in ROM. RAM typically contains data and / or program modules currently being operated on and executed by the processing subsystem 2904. In some embodiments, the system memory 2910 may include a variety of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc.

[0288] By way of example and not limitation, Figure 29As depicted, system memory 2910 may load executing applications 2912 (which may include various applications such as web browsers, middle-tier applications, relational database management systems (RDBMS), etc.), program data 2914, and an operating system 2916. By way of example, operating system 2916 may include various versions of Microsoft Apple and / or Linux operating systems, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google OS, etc.) and / or such as iOS, Phone, OS, OS, OS operating system and other mobile operating systems, and other operating systems.

[0289] Computer-readable storage media 2922 can store programming and data structures that provide some example functionality. Computer-readable storage media 2922 can provide storage for computer-readable instructions, data structures, program modules, and other data for computer system 2900. Software (programs, code modules, instructions) that provide the functionality described above when executed by processing subsystem 2904 can be stored in storage subsystem 2918. By way of example, computer-readable storage media 2922 can include, for example, a hard drive, a magnetic disk drive, an optical disk drive (e.g., a CD ROM, DVD, Computer readable storage media 2922 may include but is not limited to non-volatile memory such as disks or other optical media. Drives, flash memory cards, universal serial bus (USB) flash memory drives, secure digital (SD) cards, DVD disks, digital video tapes, etc. The computer-readable storage medium 2922 may also include non-volatile memory-based SSDs such as flash memory-based solid-state drives (SSDs), enterprise-class flash memory drives, solid-state ROMs, etc., volatile memory-based SSDs such as solid-state RAM, dynamic RAM, static RAM, etc., DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs.

[0290] In some examples, storage subsystem 2918 may also include a computer-readable storage medium reader 2920 that may be further connected to a computer-readable storage medium 2922. Reader 2920 may receive data from a storage device such as a disk, a flash memory drive, or the like and may be configured to read data from the storage device.

[0291] In some examples, computer system 2900 can support virtualization technology, including but not limited to virtualization of processing and memory resources. For example, computer system 2900 can provide support for executing one or more virtual machines. In some examples, computer system 2900 can execute programs such as a hypervisor that facilitates the configuration and management of virtual machines. Each virtual machine can be allocated memory, computing (e.g., processor, core), I / O, and networking resources. Each virtual machine typically runs independently of other virtual machines. A virtual machine typically runs its own operating system, which can be the same as or different from the operating system run by other virtual machines run by computer system 2900. Therefore, multiple operating systems can potentially be run simultaneously by computer system 2900.

[0292] The communication subsystem 2924 provides an interface to other computer systems and networks. The communication subsystem 2924 is used as an interface for receiving data from other systems and transmitting data from the computer system 2900 to other systems. For example, the communication subsystem 2924 can enable the computer system 2900 to establish a communication channel to one or more client devices via the Internet for receiving information from the client devices and / or sending information to the client devices. For example, when the computer system 2900 is used to implement Figure 1 As depicted in the robotic system 120 , the communication subsystem may be used to communicate with the application system and the system executing the storage virtual machine selected for the application.

[0293] The communication subsystem 2924 can support both wired communication protocols and / or wireless communication protocols. In some examples, the communication subsystem 2924 can include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular phone technology, advanced data network technologies such as 3G, 4G, EDGE (Enhanced Data Rates for Global Evolution) or 5G, WiFi (IEEE 802.XX family of standards or other mobile communication technologies or any combination thereof), a global positioning system (GPS) receiver component, and / or other components. In some examples, in addition to or instead of a wireless interface, the communication subsystem 2924 can provide wired network connectivity (e.g., Ethernet).

[0294] The communication subsystem 2924 can receive and transmit various forms of data. In some examples, the communication subsystem 2924 can receive input communications in the form of structured and / or unstructured data feeds 2926, event streams 2928, event updates 2930, etc., among other forms. For example, the communication subsystem 2924 can be configured to receive (or send) data feeds 2926 in real time from users of social media networks and / or other communication services, such as feed, Updates, web feeds (such as Rich Site Summary (RSS) feeds), and / or real-time updates from one or more third-party information sources.

[0295] In some examples, the communication subsystem 2924 can be configured to receive data in the form of a continuous data stream, which can include an event stream 2928 of real-time events (which may be inherently continuous or unbounded and may not have an explicit end) and / or event updates 2930. Examples of applications that generate continuous data can include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0296] The communication subsystem 2924 can also be configured to communicate data from the computer system 2900 to other computer systems or networks. Data can be transmitted in various forms such as structured and / or unstructured data feeds 2926, event streams 2928, event updates 2930, etc. to one or more databases that can communicate with one or more streaming data sources coupled to the computer system 2900.

[0297] Computer system 2900 can be of various types, including a handheld portable device (e.g., Cellular phones, Computing tablets, PDAs), wearable devices (e.g., Google head-mounted display), personal computers, workstations, mainframes, kiosks, server racks, or any other data processing system. Due to the ever-changing nature of computers and networks, the Figure 29 The description of the depicted computer system 2900 is intended to be a specific example only. Figure 29 Many other configurations of more or fewer components of the depicted systems are possible.Based on the present disclosure and the teachings provided herein, a person of ordinary skill in the art will appreciate other ways and / or methods for implementing the various examples.

[0298] Although specific examples have been described, various modifications, changes, alternative constructions, and equivalents are possible. The examples are not limited to operation in certain specific data processing environments, but are free to operate in multiple data processing environments. In addition, although certain examples have been described using a specific series of transactions and steps, it should be apparent to those skilled in the art that this is not intended to be restrictive. Although some flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations can be rearranged. The process may have additional steps not included in the figure. The various features and aspects of the examples described above can be used individually or in combination.

[0299] Furthermore, although certain examples have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are possible. Certain examples may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors.

[0300] Where a device, system, component or module is described as being configured to perform certain operations or functions, such configuration may be accomplished, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (such as microprocessors) to perform the operations (such as by executing computer instructions or code), or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0301] Specific details are provided in this disclosure to provide a thorough understanding of the examples. However, the examples can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid obscuring the examples. This description provides only examples and is not intended to limit the scope, applicability, or configuration of other examples. Rather, the previous description of the examples will provide those skilled in the art with an enabling description for implementing the various examples. Various changes can be made to the functions and arrangements of the elements.

[0302] The specification and drawings should, therefore, be viewed in an illustrative rather than a restrictive sense. However, it will be apparent that additions, subtractions, deletions, and other modifications and changes may be made thereto without departing from the broader spirit and scope set forth in the claims. Therefore, while specific examples have been described, these examples are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0303] In the foregoing description, various aspects of the present disclosure have been described with reference to specific examples of the present disclosure, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used individually or in combination. Further, without departing from the broader spirit and scope of the specification, the examples may be utilized in any number of environments and application environments other than those described herein. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

[0304] In the foregoing description, for the purpose of illustration, method is described in a particular order. It should be understood that in an alternative example, method can be performed in an order different from the described order. It should also be understood that the method described above can be performed by hardware components or can be embodied in a sequence of machine-executable instructions, which can be used to make a machine (such as a general or special processor or logic circuit programmed with instructions) perform the method. These machine-executable instructions can be stored on one or more machine-readable media, such as CD-ROM or other types of optical disks, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic cards or optical cards, flash memories or other types of machine-readable media suitable for storing electronic instructions. Alternatively, method can be performed by a combination of hardware and software.

[0305] Where a component is described as being configured to perform certain operations, such configuration may be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0306] While illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be interpreted to encompass such variations, except to the extent limited by the prior art.

Claims

1. A computer-implemented method comprising: Basic skills are obtained from a database storing a plurality of skills, where the basic skills include: Implementation data for the basic skills; and Obtaining a first JavaScript Object Notation (JSON) file, the first JSON file including basic metadata associated with the basic skill; Obtaining from the database a second JSON file including the extended skills of the basic skills; Comparing the first JSON file with the second JSON file to determine differences between the basic skills and the extended skills; In response to determining the difference between the basic skill and the extended skill, determining that the second JSON file conflicts with the first JSON file; Resolving the conflict between the second JSON file and the first JSON file; applying changes to the first JSON file using the second JSON file to generate extended metadata associated with the extended skill, wherein the second JSON file indicates changes between the base metadata and the extended metadata; implementing the extended skill based on the implementation data of the base skill and the extension metadata associated with the extended skill; and Determining that the extended skill is backward compatible with the basic skill; and In response to determining that the extended skill is backward compatible, the extended skill and a version identifier are stored in the database storing the plurality of skills.

2. The computer-implemented method of claim 1 , wherein: The plurality of skills and the extended skill are chatbot applications.

3. The computer-implemented method of claim 1 , wherein: The basic metadata includes at least one of: configuring the basic skill, training the basic skill, testing the basic skill, or executing the basic skill.

4. The computer-implemented method of claim 1 , wherein: The basic metadata includes at least one of the following: intent, entity, utterance, custom component, or conversation flow of the basic skill.

5. The computer-implemented method of claim 1 , further comprising: obtaining a third JSON file from the database, the third JSON file describing changes to the metadata associated with the extended skill; Applying the change to the metadata associated with the extended skill described in the third JSON file to generate metadata associated with a second extended skill; as well as The second extended skill is implemented based on the implementation data of the basic skill and the metadata associated with the second extended skill.

6. The computer-implemented method of claim 1 , further comprising: Obtain a second skill from the database, where the second skill includes: Implementation data for the second skill; and a third JSON file including metadata associated with the second skill; applying the changes to the first JSON file described in the second JSON file to the third JSON file to generate metadata associated with a second extended skill; and The second extended skill is implemented based on the implementation data for the second skill and the metadata associated with the second extended skill.

7. The computer-implemented method of claim 1 , wherein: The database includes: a first repository storing implementation data of the plurality of skills; and A second repository stores the first JSON file and the second JSON file.

8. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by one or more processors of a computing system, cause the one or more processors to perform operations comprising: Basic skills are obtained from a database storing a plurality of skills, where the basic skills include: Implementation data for the basic skills; as well as Obtaining a first JavaScript Object Notation (JSON) file, the first JSON file including basic metadata associated with the basic skill; Obtaining a second JSON file associated with the extended skill of the basic skill from the database; Comparing the first JSON file with the second JSON file to determine differences between the basic skills and the extended skills; In response to the difference, determining that a conflict exists between the second JSON file and the first JSON file; Resolving the conflict between the second JSON file and the first JSON file; applying changes to the first JSON file using the second JSON file to generate metadata associated with the extended skill, wherein the second JSON file indicates changes between the base metadata and the metadata; implementing the extended skill based on the implementation data of the base skill and the metadata associated with the extended skill; and Determining that the extended skill is backward compatible with the basic skill; and In response to determining that the extended skill is backward compatible, the extended skill and a version identifier are stored in the database storing the plurality of skills.

9. The non-transitory computer-readable storage medium of claim 8, wherein: The plurality of skills and the extended skill are chatbot applications.

10. The non-transitory computer-readable storage medium of claim 8, wherein: The basic metadata includes at least one of: configuring the basic skill, training the basic skill, testing the basic skill, or executing the basic skill.

11. The non-transitory computer-readable storage medium of claim 8, wherein: The basic metadata includes at least one of the following: intent, entity, utterance, custom component, or conversation flow of the basic skill.

12. The non-transitory computer-readable storage medium of claim 8, further comprising: obtaining a third JSON file from the database, the third JSON file describing changes to the metadata associated with the extended skill; Applying the change to the metadata associated with the extended skill described in the third JSON file to generate metadata associated with a second extended skill; as well as The second extended skill is implemented based on the implementation data of the basic skill and the metadata associated with the second extended skill.

13. The non-transitory computer-readable storage medium of claim 8, further comprising: Obtain a second skill from the database, where the second skill includes: Implementation data for the second skill; and a third JSON file including metadata associated with the second skill; applying the changes to the first JSON file described in the second JSON file to the third JSON file to generate metadata associated with a second extended skill; and The second extended skill is implemented based on the implementation data for the second skill and the metadata associated with the second extended skill.

14. The non-transitory computer-readable storage medium of claim 8, wherein: The database includes: a first repository storing implementation data of the plurality of skills; and A second repository stores the first JSON file and the second JSON file.

15. A computer system comprising: one or more processors; as well as A non-transitory computer-readable storage medium storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations including: Basic skills are obtained from a database storing a plurality of skills, where the basic skills include: Implementation data for the basic skills; and Obtaining a first JavaScript Object Notation (JSON) file, the first JSON file including basic metadata associated with the basic skill; Obtaining a second JSON file associated with the extended skill of the basic skill from the database; Comparing the first JSON file with the second JSON file to determine differences between the basic skills and the extended skills; In response to the difference, determining that a conflict exists between the second JSON file and the first JSON file; Resolving the conflict between the second JSON file and the first JSON file; applying changes to the first JSON file using the second JSON file to generate metadata associated with the extended skill, wherein the second JSON file indicates changes between the base metadata and the metadata; implementing the extended skill based on the implementation data of the base skill and the metadata associated with the extended skill; and Determining that the extended skill is backward compatible with the basic skill; and In response to determining that the extended skill is backward compatible, the extended skill and a version identifier are stored in the database storing the plurality of skills.

16. The computer system of claim 15, wherein: The plurality of skills and the extended skill are chatbot applications.

17. The computer system of claim 15, wherein: The basic metadata includes at least one of: configuring the basic skill, training the basic skill, testing the basic skill, or executing the basic skill.

18. The computer system of claim 15, wherein: The basic metadata includes at least one of the following: intent, entity, utterance, custom component, or conversation flow of the basic skill.

19. The computer system of claim 15, further comprising: obtaining a third JSON file from the database, the third JSON file describing changes to the metadata associated with the extended skill; Applying the change to the metadata associated with the extended skill described in the third JSON file to generate metadata associated with a second extended skill; as well as The second extended skill is implemented based on the implementation data of the basic skill and the metadata associated with the second extended skill.

20. The computer system of claim 15, further comprising: Obtain a second skill from the database, where the second skill includes: Implementation data for the second skill; and a third JSON file including metadata associated with the second skill; applying the changes to the first JSON file described in the second JSON file to the third JSON file to generate metadata associated with a second extended skill; and The second extended skill is implemented based on the implementation data for the second skill and the metadata associated with the second extended skill.

21. The computer system of claim 15, wherein: The database includes: a first repository storing implementation data of the plurality of skills; and A second repository stores the first JSON file and the second JSON file.

Images