Automatically generate conversational services from computing applications

Abstract

Automatic generation of one or more task-oriented conversational robot programs is disclosed. Illustratively, systems and methods are provided that allow tracking of interactions of one or more computing applications, including: interactions with one or more programmatic elements of the one or more computing applications, interactions with (multiple) graphical user interfaces of the one or more computing applications, and / or operations of one or more computing environments on which the one or more computing applications are executing, for collecting various state data. The state data can be illustratively represented graphically to show the overall execution path of one or more functions / operations of the one or more computing applications for use in generating one or more instructions representing a desired task-oriented conversational robot program, which instructions can be operatively executed through one or more application programming interfaces of the one or more computing applications.

Description

Background Art

[0001] In recent years, the prospects and excitement about conversational services have grown rapidly. In addition to the popularity of intelligent assistants, there is also a rise in the use and execution of specialized robot programs (bots) (e.g., also called "skills" in the ALEXA virtual assistant from AMAZON.COM, INC. and the CORTANA virtual assistant from MICROSOFT CORP., and "actions" in the GOOGLE ASSISTANT virtual assistant from GOOGLE LLC.). Particularly useful are task-oriented chatbot programs, which act as agents on behalf of users, interact with external services using natural language dialogues to achieve specific tasks (booking a taxi, making a restaurant reservation, or finding a recipe) through the execution of one or more operations of a corresponding computer application.

[0002] Currently, most conversational services can be built using a slot-filling approach. With such an approach, a user's phrase (e.g., "I want a cup of coffee") refers to a schematic, system-supported action, such as "order coffee" (along with additional parameters, such as type, size). The input parameters necessary for the execution of the intent (such as coffee type and size) can be described as slots. Using the slot-filling approach, the control structure can operatively define the operations for a multi-turn conversation between the system and the user to collect all the slots necessary to satisfy the intent.

[0003] Slot filling has proven to be reliable, but requires significant developer effort. First, control structures, usually in the form of finite state automata, need to be hand-designed for each task. Such control structures can be complex because they need to consider many possible execution paths. Second, to support user interactions using natural language, models for understanding user questions and answers need to be trained. Training usually requires many utterances, i.e., example phrases that users can use during interactions. For example, even for simple questions like "What is the size of your party?", users can answer in many different ways, such as "3", "all of us", "my wife and I", or "Not sure, I'll tell you later". In order to train a robust model, developers must consider all such possible phrase variations and provide many utterances for each slot and for each intent. Finally, developers must enumerate possible values ​​for each slot to facilitate slot recognition in language understanding. As a result, this entire process requires significant manual coding, thereby hindering scalability to new tasks and domains.

[0004] An alternative to slot filling is a corpus-based approach, in which the bot is automatically trained from a dataset of past conversations. This approach has shown promise for non-task-oriented "small talk" bots, but it is unclear whether it can model task-oriented bots on its own. Systems that are fully machine-learned cannot guarantee that key task constraints are met (e.g., a user cannot make a restaurant reservation without specifying a time), and they lack models to ensure that the actual goal of the task is achieved. These systems are also difficult to train due to the paucity of domain-specific conversation logs.

[0005] Hybrid approaches also exist. For example, hybrid code networks (HCNs) are an approach to making machine learning systems practical by combining rules that are hand-coded by developers with recurrent neural networks. HCNs reduce the amount of training data at the expense of developer effort. Another hybrid approach is a “knowledge-based” conversational model that combines knowledge from text data (e.g., ) into conversational data However, these models only work for single-turn responses and rely on knowledge sources in the domain.

[0006] It is with respect to these and other considerations that the disclosure made herein is presented. Summary of the invention

[0007] The techniques described herein allow for the automatic creation of task-oriented conversational bot programs that are used to operate in cooperation with one or more computing applications. Operationally, the methods of the systems described herein provide for tracing of interactions with computing applications (including interactions with the GUI of a computing application) in the context of performing one or more computing application tasks / operations to achieve the results of a specific task, to retrieve state data about the computing application, the computing environment on which the computing application is executed, and the state of the GUI, for automatic generation of task-oriented bot programs that operate to perform one or more desired tasks. With such an approach, an immediate technical benefit may be achieved in the form of improving the overall processing efficiency associated with the generation and execution of one or more task-oriented conversational bot programs.

[0008] To achieve the technical benefits briefly mentioned above, illustratively, the systems and methods described herein provide support for automatic generation of conversational services, which results in enhanced processing efficiency and scalability of computing processes that generate conversational services. In illustrative operations, data can be collected that represents one or more interactions with one or more computing applications executable on one or more computing environments, and / or, one or more operations / features / commands of one or more computing environments and / or GUIs of one or more computing environments (if available). The collected data can include, but is not limited to, data representing the state of one or more computing applications, the state of one or more computing environments, and the state of one or more GUIs (if available).

[0009] In one illustrative implementation, the collected data may be used to define one or more service application programming interfaces (APIs) that are operable to cooperate with one or more computing applications, one or more computing environments, and / or one or more GUIs (if available) of one or more computing applications. The collected data may also be used to generate one or more task models usable by an exemplary bot or personal assistant system for the creation and execution of one or more conversational services.

[0010] Additionally, the collected data may be used as input data to an exemplary question-answer interface generation system that includes a neural network model that has been trained using open domain conversation logs. In an illustrative implementation, one or more generated conversational services are executable by one or more mobile applications so that created service API calls can be executed based on each user's interaction. Illustratively, the collected data specific to interactions with one or more GUIs of one or more computing applications may be operatively used to create task-oriented "skills" and / or "actions" for one or more personal digital assistants (such as the CORRANA virtual assistant from MICROSOFT CORP).

[0011] In one illustrative implementation, an authoring tool for a task-oriented conversational robot program may be provided, which may include a graphical user interface to allow a participating user (e.g., developer / author) to interact with data that is collected for display in an illustrative authoring tool GUI as part of the automatic creation of a task-oriented conversational robot program. In this illustrative implementation, the collected data may include a state diagram having one or more execution paths that show the execution of various operations / features / commands of one or more computing applications, one or more GUIs, and / or one or more computing environments. Illustratively, a task-oriented conversational robot program may be generated synchronously or asynchronously with the operation of the selected one or more execution paths to achieve a series of desired functions / operations to be performed by one or more computing applications.

[0012] It should be understood that, although described with respect to a system, the technical solutions described above may also be implemented as computer-controlled devices, computer processes, computing systems, or products such as computer-readable media. These and various other features will become apparent upon reading the following detailed description and reviewing the accompanying drawings. The present invention content introduces some concepts in a simplified form, which will be further described in the following detailed description.

[0013] This content of the present invention is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The detailed description is described with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference number identifies the drawing in which the reference number first appears. The same reference number in different drawings indicates similar or identical items. References to individual items in a set of plural items may use reference numbers with letters in the alphabetic sequence to refer to each individual item. Generalized references to the set of plural items may use specific reference numbers without the alphabetic sequence.

[0015] Figure 1 An exemplary slot-based conversational bot program building process is illustrated.

[0016] Figure 2 An exemplary user interface for use in a robot program building process in accordance with the systems and methods described herein is illustrated.

[0017] Figure 3 A block diagram of an exemplary architecture for use in building a robot program in accordance with the systems and approaches described herein is illustrated.

[0018] Figure 4An exemplary interaction trajectory of a user with an illustrative computing application for creating an automated task-oriented conversational bot program in accordance with the systems and methods described herein is illustrated.

[0019] Figure 5 An exemplary user interface trajectory for an illustrative computing application for creating an automated task-oriented conversational bot program in accordance with the systems and methods described herein is illustrated.

[0020] Figure 6 An exemplary bot visual user interface is illustrated that is operable to allow participating users to graphically illustrate operations used in the creation of a task-oriented conversational bot.

[0021] Figure 7 is a flow chart of an illustrative process for automatically creating a task-oriented conversational bot program in accordance with the systems and methods described herein.

[0022] Figure 8 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing device according to one embodiment.

[0023] Fig. 9 is a network diagram illustrating a distributed computing environment in which aspects of the disclosed technology may be implemented in accordance with the various embodiments presented herein. DETAILED DESCRIPTION

[0024] The following detailed description describes techniques that allow for the automatic creation of task-oriented conversational robot programs for execution on one or more computing applications. In an illustrative implementation, one or more computing applications may support a graphical user interface having one or more programmatic elements that may operatively provide support for the execution of one or more operations on the one or more computing applications. These exemplary one or more programmatic elements may include, but are not limited to, buttons, menus, dialog boxes, etc., which, when interacted with, initiate the execution of one or more instructions in the one or more computing applications.

[0025] Operationally, the methods of the systems described herein support tracking interactions with computing applications (including interactions with the GUI of a computing application) in the context of performing one or more computing application tasks / operations to achieve a result of a specific task to retrieve state data about the computing application, the computing environment on which the computing application is executed, and the state of the GUI for automatically generating a task-oriented robot program operable to perform one or more desired tasks. By automatically bootstrapping a conversational service from one or more computing applications, the number of processing steps for generating instructions required to perform one or more operations on one or more computing applications is reduced, thereby improving the processing efficiency of the computing environment on which the one or more computing applications can be executed.

[0026] To achieve the technical benefits briefly mentioned above, illustratively, the systems and methods described herein provide support for the automatic generation of conversational services. In illustrative operations, data may be collected that represents one or more interactions with one or more computing applications executable on one or more computing environments, and / or, one or more operations / features / commands of one or more computing environments and / or one or more computing environments and / or GUIs of one or more computing environments (if available). The collected data may include, but is not limited to, data representing the state of one or more computing applications, the state of one or more computing environments, and the state of one or more GUIs (if available).

[0027] In one illustrative implementation, the collected data can be used to define one or more service application programming interfaces (APIs) that can operate to cooperate with one or more computing applications, one or more computing environments, and / or one or more GUIs of one or more computing applications (if available). The collected data can also be used to generate one or more task models that can be used by an exemplary robot program or personal assistant system for the creation and execution of one or more tasks. In addition, the collected data can be used as input data by an exemplary question-answering interface generation system, which can be operated to use one or more neural network models that have been trained based on open domain conversational logs. In an illustrative implementation, one or more generated conversational services are executable by one or more mobile applications so that the created service API calls can be executed based on each user's interaction. Illustratively, the collected data specific to the interaction with one or more GUIs of one or more computing applications can be operatively used to create task-oriented "skills" and / or "actions" for one or more personal digital assistants (such as the CORRANA virtual assistant from MICROSOFT CORP).

[0028] In one illustrative implementation, an authoring tool for a task-oriented conversational robot program may be provided, which may include a graphical user interface to allow a participating user (e.g., developer / author) to interact with data that is collected for display in an illustrative authoring tool GUI as part of the automatic creation of a task-oriented conversational robot program. In this illustrative implementation, the collected data may include a state diagram having one or more execution paths showing the execution of various operations / features / commands of one or more computing applications, one or more GUIs, and / or one or more computing environments. Illustratively, a task-oriented conversational robot program may be generated synchronously or asynchronously along selected (multiple) operations of one or more execution paths to achieve a desired series of functions / operations performed by one or more computing applications.

[0029] In an illustrative implementation, the systems and methods described herein can use an exemplary user interface (UI) tree model to represent exemplary computer application interaction data in a screen of the UI, including but not limited to UI elements and the relationships between UI elements, and one or more interactions that can be performed on them. In an illustrative implementation, by operationally abstracting UI screens and events into intents and slots, a task model from the interaction with an exemplary computing application can be derived through one or more trajectories. In an illustrative operation, the trajectory data can be further segmented to identify one or more operational states of a computing application that executes an exemplary user interface. Such states can include dynamic and static computing application states. Computing application state data can be used when generating a robot program to guide how the computing application operates during the execution of application operations and / or functions.

[0030] Illustratively, the task model can represent the logical backbone of the robot program. In an illustrative implementation, a question-answer interface for a conversation can be automatically established, which operatively utilizes a hybrid rule-based and neural network approach. In an illustrative operation, using the GUI of an exemplary computing application and the content on the screen, the properties and semantics of the slot can be derived, so that prompt questions can be generated by means of some semantic rules. In order to generate questions for slots that cannot be semantically classified, and to generate discourse for user answers, one or more neural network transduction models can be trained (for example, by utilizing millions of dialogue pairs provided by one or more large social media networks (such as the TWITTER communication service from TWITTER INC.)).

[0031] In one illustrative implementation, a task-oriented robot can understand user requests and perform related tasks. Figure 1As shown, the exemplary task-oriented robot program creation process 100 may include, but is not limited to, three operations: 110; 130 and 150. As shown, operation 110 may include defining a control structure. Operationally, a software developer may define a control structure that represents the kinds of actions (referred to as "intents" 115) that the exemplary computing environment may support. Illustratively, an intent 115 may be associated with one or more slots 120 and may have one or more constraints or dependencies 125 on other intents. Constraints may be viewed as one or more conditions specific to the execution of an intent.

[0032] For example, in a restaurant reservation bot, an intent named reserve a restaurant may be defined and may have slots that may include: 1) restaurant name; 2) party size; and 3) time. A corresponding intent named confirm reservation may have a dependency on the intent reserve a restaurant. In exemplary operation, based on such an exemplary structure, the bot may perform slot filling, i.e., collect slots (e.g., required data elements) for each intent and then perform the associated action(s).

[0033] At operation 130, a developer may define (e.g., by training a machine learning model capable of language processing) a conversational interface for a control structure (e.g., a set of intents 135 that the exemplary computing environment may support), operative to perform one or more functions, including, but not limited to: 1) identifying user intent 135 and slots 120; 2) querying the user 140 for the requested slots 120; and 3) understanding the user response 145. For example, from the phrase "I want a table for two," the exemplary computing environment executing the creation process 100 may recognize an intent named "book a restaurant," and a slot party size equal to two, and then query the user for the missing slot time. At operation 150, the resulting intent may then be communicated to a collaborative service backend operation of the exemplary computing environment, which is capable of invoking one or more service APIs and returning results 160 from such API calls to participating users (not shown) (e.g., confirming the reservation).

[0034] like Figure 2 As shown, an exemplary robot program creation tool 200 with various user interface controls and dialog boxes is shown. Operationally, in using the exemplary robot program creation tool 200, a developer can supply one or more examples of user utterances 210, which can be operatively mapped to intents 220 in a task. In this illustrative operation, the developer can define one or more slots 230 for the intent 220, mark one or more slot values ​​230 that appear in the provided utterances 210 (the marking of the slot value can illustratively provide the data elements required for the execution of the intent), and provide prompt questions for each slot and specify the order of the prompts 240.

[0035] Operationally, the developer can repeat these operations to parse the user answer, i.e., specify phrases for possible user answers, and mark slot values ​​in the example answers. Operationally, to facilitate identifying slots from user phrases, possible values ​​for each slot 240 can be provided. In one illustrative implementation, after specifying many utterance samples for intents 220 and slots 240, a machine learning model can be trained that can operationally provide reasonable accuracy on inputs similar to the examples.

[0036] Figure 3 The various cooperating components and functions of an exemplary task-oriented conversational robot program generation environment 300 are illustrated. As shown, the exemplary task-oriented conversational robot program generation environment 300 may include a neural sequence transduction module 310, a task model extraction module 330, and a question / answer generation module 370. In addition, as shown in FIG. Figure 3 As shown, the neural sequence transduction module can be operable to perform various functions, including but not limited to data collection 315 (e.g., data set 317), data annotation 320, and model training 325. Operationally, data collection can be considered as the process of collecting data required for the generation of a conversational service, data annotation can be considered as the process of associating one or more selected features to the collected data, and model training can be considered as one or more instructions that can be inferred from the collected data.

[0037] The task model extraction module 330 can be operable to perform various functions including, but not limited to, trace collection 335 (e.g., collecting data representing a user's interaction with an exemplary computing application), trace aggregation 340 (e.g., integrating the collected traces according to one or more selected rules), and intent / slot extraction 345 (e.g., defining intents and slots associated with the intents based on the collected trace data).

[0038] The question / answer generation module 370 may include, but is not limited to, generating slot answers 375, which may be operatively generated from the question-to-answer transformation 355 (e.g., operations of associating answers to questions and associating questions to answers based on one or more selected rules), the question-to-answer transformation 355 operating on data provided by the neural sequence transduction module 310 and an exemplary data template from the robot program template 365. Similarly, the question and answer generation module 370 may include questions 380, which may be operatively generated using data provided by the entity extraction module 350, which may operatively receive data from the data annotation function 320 of the neural sequence transduction module 310. In addition, the slot questions 380 may also be generated by the answer-to-question transformation 360, which operates on data provided by the neural sequence transduction module 310.

[0039] Figure 4 Three exemplary screens (a, b, and c) from a user's interaction with an exemplary mobile computing application 400 (e.g., an application for utilizing an OPENTABLE restaurant reservation service from OPENTABLE) are shown, showing the user searching for a restaurant (screen a-410) and selecting a restaurant from a list (not shown), viewing information about the selected restaurant (screen b-420), and making a reservation (screen c-430). Operationally, various intents and slots can be inferred: start a restaurant search (screen a-410), view restaurant information, view restaurant menus, and view restaurant reviews (screen b-420) (which can be defined as a slot - restaurant name (selected in the restaurant list screen)), and make a reservation at a restaurant (screen c-430) (taking the reservation time as the slot (selected in screen b-420)).

[0040] In one illustrative operation, to programmatically extract intents and slots, an "application language" may be operatively translated into a "bot language." In the application language, a user may perform tasks via interacting with UI elements and transitioning from one page to another. In the bot language, a user may perform tasks via filling slots and navigating a graph of intents. Operationally, to extract information from a mobile application, (1) static analysis may be applied, which examines the source code of the mobile application without executing it, and / or (2) dynamic analysis, where the application is analyzed by executing it.

[0041] In an illustrative implementation, an interaction trajectory can be represented as a sequence of UI events. Each UI event can be associated with a UI element, on which an action is performed (e.g., clicking on a button or typing text in a text field), and a UI tree, which includes the UI elements on the screen at the moment of interaction and their hierarchical relationships, types, and contents. In this illustrative implementation, one or more application trajectories provided by the developer can be processed to extract a task model. A task model can consist of a set of intents, which can also include a set of slots and dependencies from other intents. A slot can have a name and a set of possible values. In an illustrative implementation, task model extraction can be achieved using trajectory aggregation, intent extraction, and slot extraction.

[0042] Figure 5 It is shown that in the use of Figure 4 The OPENTABLE application is briefly described as all UI events reported during an exemplary user interaction with an exemplary intent search criteria activity. Operationally, interactions 510, 525, and 530 may be used in such Figure 3, described extraction process. In one illustrative operation, user interactions 505, 515, 550, and 555 can be exempt from processing because such interactions are associated with immutable content (e.g., a button of an exemplary mobile application with a static label that, when interacted with, causes a transition to a new page or dialog box) that does not provide informative data specific to the generation of a conversational bot. Additionally, interactions 535, 540, and 545 can be exempt from extraction operations because such interactions are associated with invisible UI elements, which can occur when an application overlaps multiple layouts on top of each other. Similarly, interaction 520 can be exempt from processing because such an interaction is associated with empty content. Operationally, slots can be extracted from the remaining UI elements, and appropriate intents can be assigned to such slots.

[0043] Figure 6 An exemplary restaurant reservation modeling tool 600 and the Figure 4 and Figure 5 Briefly describe the example model extracted from the OPENTABLE mobile application. As shown, Figure 6 The nodes in may represent intent actions supported by the system, with parameters specific to the intent. Illustratively, the input parameters may be the necessary data elements for the intent to execute. The flow begins with an intent named StartOfConversation and ends with an intent named EndOfConversation.

[0044] In one illustrative operation, restaurants can be searched by viewing restaurant suggestions such as "nearby restaurants" or "outdoor seating" (via search result intent), or by conducting a customized search (via search criteria and search result_2). For each restaurant identified in the results, the user can retrieve an overview, reservation options, menu, and reviews (using the corresponding intent), and then proceed to make a reservation (confirm reservation) and submit it (confirm reservation_done). The slot can be represented as Figure 6 A customized search (search result_2) may require several slots: search query, city, party size, and time. In turn, a slot may maintain several fields, name, identifier, value, prompt question, utterance for possible answers; additional fields may be generated by additional operations.

[0045] In this illustrative implementation, the previous operations may produce a logic flow for the robot. Operationally, in order to navigate the logic flow using natural language, the robot needs to be able to ask questions and understand answers. In this illustrative implementation, question generation may be performed (i.e., identifying an appropriate prompt question for each slot), and answer generation may be performed (i.e., generating a large set of possible answers for each prompt so that an answer understanding model can be trained).

[0046] Figure 7 700 is a flow chart of an illustrative process 700 for automatic creation of a task-oriented conversational bot by an exemplary computing environment that results in enhanced processing efficiency of the exemplary computing environment. As shown, processing begins at block 705, where data representing one or more interactions with the computing environment may be received. Processing then proceeds to block 710, where the received data is segmented according to one or more states of the computing environment.

[0047] One or more dependencies between states may be identified at block 715, and, as described above, such dependencies are used to identify one or more service APIs used by the exemplary computing application. Processing then proceeds to block 720, where one or more questions and one or more answers may be generated from the determined dependencies. The dependency data, along with the generated question / answer data, and the state data, may be used to generate input data for the service API.

[0048] Processing then proceeds to block 730 where a check is performed to determine if there is additional data requiring processing. If there is no additional interactive data, processing terminates at block 735. If additional interactive data requires processing, processing withdraws to block 710 and continues from there.

[0049] Figure 8 is a computer architecture diagram showing an illustrative computer hardware and software architecture for a computing device that can be operatively executed by the methods of the system described herein as described. In particular, Fig. 9 The architecture illustrated in the figure can be utilized to implement a server computer, a mobile phone, an e-reader, a smart phone, a desktop computer, an augmented or virtual reality (AR / VR) device, a tablet computer, a laptop computer, or other types of computing devices.

[0050] Figure 8The computer 800 shown in FIG. 8 includes a central processing unit 802 (“CPU”), a system memory 804 including a random access memory (“RAM”) 806 and a read-only memory (“ROM”) 808, and a system bus 810 that couples the system memory 804 to the CPU 802. A basic input / output system (“BIOS” or “firmware”) containing basic routines that help transfer information between elements within the computer 800 (such as during startup) may be stored in the ROM 808. The computer 800 also includes a mass storage device 812 for storing an operating system 822, application programs, and other types of programs. The mass storage device 812 may also be configured to store other types of programs and data.

[0051] The mass storage device 812 is connected to the CPU 802 through a mass storage controller (not shown) connected to the bus 810. The mass storage device 812 and its associated computer readable media provide non-volatile storage for the computer 800. Although the description of computer readable media contained herein refers to a mass storage device, such as a hard drive, a CD-ROM drive, or a USB memory key, it should be understood by those skilled in the art that the computer readable media can be any available computer storage media or communication media that can be accessed by the computer 800.

[0052] Communication media includes computer readable instructions, data structures, program modules or other data in a modulated data signal (such as a carrier wave or other transport mechanism) and includes any transmission media. The term "modulated data signal" means a signal: one or more of its characteristics are changed or set so as to encode information in the signal. By way of example, and not limitation, communication media includes wired media (such as a wired network or a direct-wired connection) and wireless media (such as acoustic waves, radio frequency, infrared and other wireless media). Any combination of the above should also be included within the scope of computer readable media.

[0053] By way of example, and not limitation, computer storage media may include volatile and nonvolatile, removable and non-removable media implemented with any method and technology for storage of information, such as computer readable instructions, data structures, program modules, and other data. For example, computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state storage technology, CD-ROM, digital versatile disk ("DVD"), HD-DVD, BLU-RAY, or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and accessed by the computer 800. For the purposes of the claims, the phrase "computer storage media" and its variants do not include waves or signals themselves or communications media.

[0054] According to various configurations, the computer 800 can operate in a networked environment using logical connections to remote computers through a network, such as network 820. The computer 800 can be connected to the network 820 via a network interface unit 816 connected to the bus 810. It should be understood that the network interface unit 816 can also be utilized to connect to other types of networks and remote computer systems. The computer 800 can also include an input / output controller 818 for receiving and receiving data from a number of other devices 827, including a keyboard, a mouse, a touch input, and an electronic pen ( Figure 8 )) or physical sensors (such as video cameras) receive and process input. Similarly, the input / output controller 818 can provide output to a display screen or other type of output device 825.

[0055] It should be understood that the software components described herein, when loaded into CPU 802 and executed, can convert CPU 802 and overall computer 800 from a general-purpose computing device to a dedicated computing device customized to facilitate the functionality presented herein. CPU 802 can be constructed from any number of transistors or other discrete circuit elements, which can assume any number of states individually or collectively. More specifically, CPU 802 can operate as a finite state machine in response to executable instructions contained in the software modules disclosed herein. These computer executable instructions can be converted to CPU 802 via specifying how CPU 802 transitions between states, thereby converting the transistors or other discrete hardware elements that constitute CPU 802.

[0056] Encoding the software modules presented herein may also transform the physical structure of the computer-readable medium presented herein. In different implementations of this description, the specific physical transformation depends on various factors. Examples of such factors include, but are not limited to, the technology used to implement the computer-readable medium, whether the computer-readable medium has the characteristics of a primary or secondary storage device, etc. For example, if the computer-readable medium is implemented as a semiconductor-based memory, the software disclosed herein may be encoded on the computer-readable medium by transforming the physical state of the semiconductor memory. For example, the software may transform transistors, capacitors, or other discrete circuit elements that constitute the semiconductor memory. The software may also transform the physical state of such components to store data thereon.

[0057] As another example, the computer readable medium disclosed herein can be implemented using magnetic or optical technology. In such an implementation, the software presented herein can transform the physical state of the magnetic or optical medium when the software is encoded therein. These transformations can also include changing the characteristics or properties of specific locations in a given optical medium to change the optical properties of those locations. Without departing from the scope and spirit of this description, transformations of other physical media are possible, and the aforementioned examples are only provided to facilitate this discussion.

[0058] In view of the above, it should be understood that many types of physical transformations occur in the computer 800 to store and execute the software components presented herein. It should also be understood that Figure 8 The architecture shown for computer 800, or a similar architecture, may be utilized to implement other types of computing devices, including handheld computers, video game devices, embedded computer systems (such as smart phones, tablets, and AR / VR devices), and other types of computing devices known to those skilled in the art. It is also contemplated that computer 800 may not include Figure 8 All components shown in the assembly may include Figure 8 Other components not explicitly shown in the figure or may be used in conjunction with Figure 8 The architecture shown in is completely different from the architecture shown in.

[0059] Fig. 9 900 is a network diagram illustrating a distributed network computing environment in which various aspects of the disclosed technology may be implemented according to various embodiments presented herein. Fig. 9As shown, one or more server computers 900 can be interconnected with a number of client computing devices via a communication network 820 (which can be either a fixed-line or wireless LAN, WAN, intranet, extranet, peer-to-peer network, virtual private network, Internet, Bluetooth communication network, dedicated low-voltage communication network or other communication network, or a combination of the above), and the client computing devices include, but are not limited to, tablet computers 900B, game consoles 900C, smart watches 900D, phones 900E (such as smart phones), personal computers 900F, and AR / VR devices 900G.

[0060] In a network environment in which the communication network is the Internet, for example, the server 900A can be a dedicated server computer that is operable to process and communicate data to and from the client computing devices 900B-900G via any number of known protocols, such as the Hypertext Transfer Protocol (“HTTP”), the File Transfer Protocol (“FTP”), or the Simple Object Access Protocol (“SOAP”). Additionally, the networked computing environment 900 can utilize various data security protocols, such as Secure Sockets Layer (“SSL”) or Pretty Good Privacy (“PGP”). Each of the client computing devices 900B-900G can be equipped with an operating system that is operable to support one or more computing applications or terminal sessions, such as a web browser ( Fig. 9 ), or other graphical user interface ( Fig. 9 ), or a mobile desktop environment ( Fig. 9 ) obtains access to server computer 900A.

[0061] The server computer 900A can be communicatively coupled to other computing environments ( Fig. 9 ), and receives data about the interaction / resource network of participating users. In one illustrative operation, a user ( Fig. 9 ) can interact with computing applications running on client computing devices 900B-900G to obtain desired data and / or execute other computing applications.

[0062] Data and / or computing applications may be stored on one or more servers 900A and communicated to cooperating users via client computing devices over exemplary communication network 820. Participating Users ( Fig. 9 900A) can request access to specific data and applications that are hosted in whole or in part on the server computer 900A. For processing and storage, this data can be communicated between the client computing devices 900B-900G and the server computer 900A.

[0063] The server computer 900A can host computing applications, processes, and applets for data and application generation, authentication, encryption, and communication, and can communicate with other server computing environments ( Fig. 9 Not shown), third-party service providers ( Fig. 9 ), network attached storage ("NAS"), and storage area network ("SAN") to enable application / data transactions.

[0064] It should be understood that for the sake of discussion, Figure 8 The computational architecture shown in Fig. 9 The distributed network computing environment shown in has been simplified. It should also be understood that the computing architecture and distributed computing network may include and utilize many more computing components, devices, software programs, networking devices, and other components not specifically described herein.

[0065] Sample Clauses

[0066] The disclosure presented herein includes the technical solutions stated in the following clauses.

[0067] Example clause A, a system comprising at least one processor and at least one memory having computer-readable instructions stored thereon, which, when executed by the at least one processor, cause the at least one processor to: receive first input data, the first input data comprising data representing one or more user interactions with one or more programmatic elements, data representing the one or more programmatic elements, and data representing a state of an application in which the one or more programmatic elements are being executed; process the first input data to generate intermediate processed data by: segmenting the input data according to one or more selected states of the application; identifying one or more dependencies between the one or more selected application states; and extracting one or more service APIs from the application using the identified one or more dependencies; and generate a natural user interface using the intermediate processed input data by: generating one or more questions for the identified one or more dependencies; generating samples of user answers for the generated one or more questions; and generating samples of user speech for triggering the extracted one or more service APIs to be executed by the application.

[0068] Example clause B, a system as claimed in Example clause A, wherein one or more user interactions include one or more traces of behavior of the application.

[0069] Example clause C. The system of example clause A or B, wherein the data representing the one or more programmatic elements comprises: data representing a user interface of an application executable on the computing environment.

[0070] Example clause D, a system of example clauses A to C, wherein the computer-readable instructions further cause at least one processor to segment the first input data according to one or more selected application states, the one or more selected application states comprising: one or more states of a user interface of an application executable on the computing environment, and one or more internal processing states representing the application executable on the computing environment.

[0071] Example clause E, the system of example clauses A to D, wherein the data representing the state of an application in which the programmatic element is executing includes: a dynamic application state, a static application state, and a state of an underlying operating system of a computer environment on which the application is executed.

[0072] Example clause F, a system of example clauses A to E, wherein the computer readable instructions further cause at least one processor to order one or more selected application states as one or more elements of a state diagram, the state diagram including one or more dependencies representing the one or more selected application states.

[0073] Example clause G, the system of example clauses A to F, wherein the computer readable instructions further cause the at least one processor to extract the one or more service APIs as one or more paths on a state diagram, the one or more paths comprising one or more selected states representing the application.

[0074] Example Clause H, a computer-implemented method, comprising: receiving first input data by a computing environment, the first input data including data representing one or more user interactions with one or more programmatic elements, data representing the one or more programmatic elements, and data representing a state of an application in which the one or more programmatic elements are being executed; processing the first input data to generate intermediate processed input data by: segmenting the input data according to one or more selected application states; sorting the one or more selected application states as one or more elements of a state diagram; and extracting one or more service application programming interfaces (APIs) as one or more paths on the state diagram, the one or more paths including data representing one or more parameters for the extracted one or more service APIs; and generating a natural user interface using the intermediate processed input data by: generating one or more questions for the identified parameters; generating one or more samples of user answers for the generated one or more questions; and generating one or more user utterances for triggering the extracted one or more service APIs to be executed by the application.

[0075] Example Clause I, the computer-implemented method of Example Clause H further includes performing tracing of an application operating on the computing environment, the tracing generating data representing data processed by one or more functions of the application.

[0076] A computer-implemented method of Example Clause J, Example Clauses H and I, wherein segmenting the first input data according to one or more selected application states includes segmenting the first input data according to one or more states of the user interface of the application and one or more internal processing states of the application.

[0077] Example clause K, the computer-implemented method of example clauses H to K, wherein the state diagram further comprises data representing one or more dependencies between one or more selected application states.

[0078] Example clause L, the computer-implemented method of example clauses H to K, further comprising generating a skill from the generated input data, providing the skill to one or more service APIs for execution.

[0079] Example clause M, the computer-implemented method of example clauses H to K, further comprising: generating a skill and providing the skill to one or more user channels for execution, the one or more user channels comprising a collaborative computing application.

[0080] Example clause N, the computer-implemented method of example clauses H to M, further comprising identifying one or more service APIs from the received input data.

[0081] Example clause O, the computer-implemented method of example clauses H to N, further comprising training the computing environment to: generate one or more questions for the identified parameters, and generate samples of one or more user responses to the generated one or more questions.

[0082] Example clause P, a computer-readable storage medium having computer-executable instructions stored thereon, which, when the instructions are executed by one or more processors of a computing device, causes the one or more processors of the computing device to: receive first input data, the first input data comprising data representing one or more user interactions with a programmatic element, data representing the programmatic element, and data representing a state of an application in which the programmatic element is being executed; process the first input data to generate intermediate processed input data by: segmenting the first input data according to one or more selected states; identifying one or more dependencies between the one or more selected application states; and extracting one or more service APIs from the application using the identified one or more dependencies; and generate a natural user interface using the intermediate processed input data by: generating one or more questions for the identified one or more dependencies; generating samples of user answers for the generated one or more questions; and generating one or more samples of user utterances for the extracted one or more service APIs to be executed by the application.

[0083] Example clause Q, the computer-readable storage medium of example clause P, wherein the instructions further cause one or more processors of the computing device to: track generated data representing data processed by one or more functions of the application.

[0084] Computer-readable storage medium of example clause R, example clause P and Q, wherein the instructions further cause one or more processors of the computing device to: order one or more selected application states as one or more elements of a state diagram, the state diagram including data representing one or more dependencies between the one or more selected application states.

[0085] Example clause S, computer-readable storage medium of example clauses P to R, wherein the instructions further cause one or more processors of the computing device to: extract one or more service APIs as one or more paths on a state diagram, the one or more paths comprising data representing one or more states of an application.

[0086] Example clause T, the computer-readable storage medium of example clauses P to S, wherein the instructions further cause one or more processors of the computing device to: define one or more service APIs from the received input data.

[0087] in conclusion

[0088] Finally, although the technical solution has been described in language specific to structural features and / or method actions, it should be understood that the technical solution defined in the claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as example forms of implementing the claims.

Claims

1. A system for automatically generating a conversational service from a computing application, comprising: at least one processor; as well as at least one memory having computer-readable instructions stored thereon, which, when executed by the at least one processor, cause the at least one processor to: receiving first input data, the first input data comprising data representing one or more user interactions with one or more programmatic elements, data representing the one or more programmatic elements, and data representing a state of an application in which one or more of the programmatic elements are being executed, wherein the one or more user interactions are exempted from processing when at least one user interaction of the one or more user interactions is associated with a particular type of content and / or a particular attribute of a user interface element; The first input data is processed by the following operations to generate intermediate processed input data: segmenting the first input data according to one or more selected application states; identifying one or more dependencies between the one or more selected application states; as well as extracting one or more service application programming interfaces (APIs) from the application using the identified one or more states and one or more dependencies; as well as The user interface is generated using the intermediate processed input data by the following operations: generating one or more questions for the one or more identified dependencies; Generating a sample of user answers for the one or more questions generated; as well as A sample of user speech is generated to trigger the extracted one or more service APIs to be executed by the application. 2 . The system of claim 1 , wherein the one or more user interactions include one or more traces of behavior of the application.

3. The system of claim 1 , wherein the data representing the one or more programmatic elements comprises: Data representing the user interface of the application.

4. The system of claim 1 , wherein the instructions further cause the at least one processor to segment the first input data according to one or more selected application states, the one or more selected application states comprising: One or more states of a user interface of an application executable on a computing environment, and one or more states representing one or more internal processing states of the application executable on the computing environment.

5. The system of claim 1 , wherein the data representing the state of the application comprises: A dynamic application state, a static application state, and a state of an underlying operating system for a computer environment on which the application is executing in which the programmatic element is executing.

6. The system of claim 1, wherein the computer-readable instructions further cause the at least one processor to: order the one or more selected application states as one or more elements of a state diagram, the state diagram comprising: Data representing one or more dependencies between the one or more selected application states.

7. The system of claim 6, wherein the computer-readable instructions further cause the at least one processor to: extract the one or more service APIs as one or more paths on the state diagram, the one or more paths comprising: Data representing one or more selected states of the application.

8. A computer-implemented method comprising: receiving, by a computing environment, first input data, the first input data comprising data representing one or more user interactions with one or more programmatic elements, data representing the one or more programmatic elements, and data representing a state of an application in which the one or more programmatic elements are being executed, wherein the one or more user interactions are exempted from processing when at least one user interaction of the one or more user interactions is associated with a particular type of content and / or a particular attribute of a user interface element; The first input data is processed by the following operations to generate intermediate processed input data: segmenting the first input data according to one or more selected application states; ordering the one or more selected application states as one or more elements of a state diagram, wherein the state diagram includes: data representing one or more dependencies between the one or more selected application states; and Extracting one or more service application programming interfaces API as one or more paths of the state diagram, the one or more paths comprising: data representing one or more parameters for the one or more extracted service APIs; and A natural user interface is generated using the intermediate processed input data by the following operations: generating one or more questions for the identified parameters; generating a sample of one or more user responses for the one or more questions generated; and One or more samples of user utterances are generated for the extracted one or more service APIs to be executed by the application.

9. The computer-implemented method of claim 8, further comprising: Tracing of the application operating on the computing environment is performed, the tracing generating data representing data processed by one or more functions of the application.

10. The computer-implemented method of claim 8, wherein segmenting the first input data according to one or more selected application states comprises: The first input data is segmented according to one or more states of a user interface of the application and one or more internal processing states of the application.

11. The computer-implemented method of claim 8, wherein one or more paths include data representing one or more states of the application.

12. The computer-implemented method of claim 8, further comprising: A skill is generated and provided to one or more user channels for execution, the one or more user channels comprising a collaborative computing application.

13. The computer-implemented method of claim 8, further comprising: One or more service APIs are identified from the received input data or user request.

14. The computer-implemented method of claim 8, further comprising: The computing environment is trained to generate the one or more questions for the identified parameters, and to generate samples of the one or more user answers to the generated one or more questions, and to generate samples of the one or more user utterances for the identified actions or service APIs.

15. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by one or more processors of a computing device, cause the one or more processors of the computing device to: Receive first input data, the first input data comprising: data representing one or more user interactions with a programmatic element, data representing the programmatic element, and data representing a state of an application in which the programmatic element is being executed, wherein the one or more user interactions are exempted from processing when at least one user interaction of the one or more user interactions is associated with a particular type of content and / or a particular attribute of a user interface element; The first input data is processed by the following operations to generate intermediate processed input data: segmenting the first input data according to one or more selected application states; identifying one or more dependencies between the one or more selected application states; as well as extracting one or more service application programming interfaces (APIs) from the application using the identified one or more dependencies; as well as A natural user interface is generated using the intermediate processed input data by the following operations: generating one or more questions for the one or more identified dependencies; Generating a sample of user answers for the one or more questions generated; as well as A sample of one or more utterances is generated for the extracted one or more service APIs to be executed by the application.

16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions further cause the one or more processors of the computing device to: Tracing generates data representing data processed by one or more functions of the application.

17. The non-transitory computer-readable storage medium of claim 16, wherein the instructions further cause the one or more processors of the computing device to: ordering the one or more selected application states as one or more elements of a state diagram, the state diagram comprising: Data representing one or more dependencies between the one or more selected application states.

18. The non-transitory computer-readable storage medium of claim 15, wherein the instructions further cause the one or more processors of the computing device to: Extracting the one or more service APIs as one or more paths on the state diagram, the one or more paths comprising: Data representing one or more states of the application.

19. The non-transitory computer-readable storage medium of claim 15, wherein the instructions further cause the one or more processors of the computing device to: One or more service APIs are defined from the received input data.

Images