Generating a natural language interface from a graphic user interface

Abstract

An approach is disclosed for automatically generating Natural Language Interfaces (NLI) directly from Graphical User Interface (GUI) code. The approach leverages mappings between GUI components and predefined NLI components to generate the required NLI components (e.g., intent examples, entities, etc.) from the GUI code representation. The approach leverages predefined patterns to generate these intent examples for each type of NLI component. The created NLI dialogs can be used in conjunction with the GUI or as standalone features.

Description

[Technical Field]

[0001] The present invention relates generally to GUIs (graphical user interfaces), and more particularly to generating natural language interfaces from GUIs. [Background technology]

[0002] Currently, the primary way users interact with applications is through a GUI (Graphical User Interface). However, attempts to build applications that also work with NLI (Natural Language Interface) are becoming increasingly common. There are some applications and web pages in use today that would be much easier to use if they had NLI, such as navigating a shopping checklist at the grocery store.

[0003] Building an NLI from scratch is a significant undertaking, even if a GUI has already been built, and requires knowledge of conversation design that is not common among application developers. This means that companies and developers who want to add an NLI to their applications must spend a lot of time researching and implementing this interface. Furthermore, the two interfaces must be kept synchronized, meaning that changes to one interface will likely require changes to the other interface as well. Summary of the Invention

[0004] Aspects of the present invention disclose a computer-implemented method, a computer system, and a computer program product for creating a natural language (NL) dialog from graphical user interface (GUI) code. The computer-implemented method may be performed by one or more computer processors. The computer-implemented method may include receiving a GUI code representation of an application, mapping the GUI code representation to one or more natural language interface (NLI) components, determining a predefined intent pattern based on the one or more NLI components, creating one or more intent examples based on the predefined intent pattern, creating one or more entities based on the one or more intent examples, mapping action functions to back-end operations of the one or more NLI components, and generating a NL dialog based on the one or more intent examples, the one or more entities, the mapped action functions, and a predefined conversation component.

[0005] According to another embodiment of the present invention, there is provided a computer system including a processing unit and a memory, coupled to the processing unit, that stores instructions that, when executed by the processing unit, perform operations of a method according to an embodiment of the present invention.

[0006] According to yet another embodiment of the present invention, there is provided a computer program product tangibly stored on a non-transitory machine-readable medium and including machine-executable instructions that, when executed on a device, cause the device to perform operations of a method according to embodiments of the present invention.

[0007] Preferred embodiments of the present invention will now be described, by way of example only, with reference to the following drawings: [Brief explanation of the drawings]

[0008] [Figure 1]FIG. 1 is a functional block diagram illustrating a natural language environment 100, according to one embodiment of the present invention. Note that FIG. 1 includes an expanded view of a natural language component 111, featuring several internal subcomponents, according to one embodiment of the present invention. [Figure 2A] 1 is a screenshot of a main screen of a sample application according to one embodiment of the present invention. [Figure 2B] 2B is a screenshot of a settings screen for a sample application (of FIG. 2A) according to one embodiment of the present invention. [Figure 2C] FIG. 2B illustrates an example of a GUI to NLI mapping for a sample application (of FIG. 2A) according to one embodiment of the present invention. [Figure 2D] FIG. 2 illustrates a sample conversation and corresponding logic according to one embodiment of the present invention. [Figure 2E] FIG. 10 illustrates another sample conversation and corresponding logic according to one embodiment of the present invention. [Figure 3] 3 is a schematic flow chart illustrating the operation 300 of the natural language component 111 according to an embodiment of the present invention. [Figure 4] 4 is a block diagram 400 of components of a server computer capable of executing the natural language component 111 within the natural language environment of FIG. 1 according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Recognizing shortcomings in the current state of the art, embodiments of the present invention provide an approach for automatically generating natural language interfaces (NLIs) directly from graphical user interface (GUI) code. In embodiments, mappings between GUI components and predefined NLI components are utilized to generate examples of the required intents and entities (e.g., single choice, multiple choice, free text, etc.). This approach can leverage predefined patterns to generate intent examples for each predefined intent of an NLI component. NLIs use the same operations and examples. For example, predefined intents (i.e., "report" intents) exist as part of a predefined NLI component, and predefined patterns are used to generate training examples for these intents (i.e., "I would like to report that {X} is {Y}," where X and Y are replaced with actual values ​​extracted from the GUI).

[0010] Briefly summarized, this approach consists of the following steps: i) mapping GUI components to corresponding NLI components (i.e., drop-down selections are mapped to single choice input NLI components); ii) creating intent and entity examples for each type of NLI component using predefined patterns; iii) mapping action components; and iv) generating a natural language dialog representation of the application.

[0011] We further discuss intents, entities, and actions. Every NLI component has a corresponding "intent" and a predefined intent pattern. Intents are based on the names and IDs (identifiers) of GUI components in the code and can be used as the basis for creating intent examples from predefined patterns. Entities are based on components parsed from the GUI code representation, and parts of the GUI component's name and ID can be used as entities and synonyms. As part of the mapping described above, actions are mapped from action calls in the GUI code representation (such as a button's "onClick" callback function) to backend operations in the NL (natural language) component. Furthermore, the mapping can be performed while searching for restrictions on actions that need to be taken into account in the NLI (Natural Language Interface). Note that the created natural language dialog can be used in conjunction with the GUI or as a standalone feature.

[0012] Other embodiments of the present invention address shortcomings in the current technology and can provide the following advantages: i) completely different interfaces can be created without any changes to existing GUIs; ii) natural language interfaces can be automatically created from graphical user interfaces; iii) different types of GUI and NLI components can be utilized; iv) different intent and entity training can be utilized; and v) NLI can automatically discover GUIs with known conversational components to create new interfaces.

[0013] References herein to "one embodiment," "an embodiment," "an example embodiment," or the like indicate that, while a described embodiment may include a particular feature, structure, or characteristic, not all embodiments necessarily include that particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with one embodiment, it is believed to be within the knowledge of one skilled in the art to relate such feature, structure, or characteristic to other embodiments, whether or not explicitly stated.

[0014] It should be noted that the drawings are merely schematic representations and are not to scale, and the same reference numerals may be used throughout the drawings to refer to the same or similar parts.

[0015] Figure 1 is a functional block diagram of a natural language environment 100 in accordance with one embodiment of the present invention. Figure 1 is intended to be illustrative of one implementation and is not intended to imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by one of ordinary skill in the art without departing from the scope of the present invention as defined by the claims.

[0016] The natural language environment 100 includes a product network 101 , a computing device 102 , and a server 110 .

[0017] Network 101 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the three, and may include wired, wireless, or fiber optic connections. Network 101 may include one or more wired and / or wireless networks capable of transmitting and receiving data, voice, and / or video signals, including multimedia signals including voice, data, and video information. In general, network 101 may be any combination of connections and protocols capable of supporting communication between server 110, computing device 102, and other computing devices (not shown) within natural language environment 100. Note that the other computing devices may include, but are not limited to, computing device 102 and any electromechanical device capable of executing a sequence of computing instructions.

[0018] The computing device 102 may include a cluster of end-user machines and software programming platforms. The computing device 102 provides functionality for users (i.e., programmers) to create NLI dialogs from GUI components.

[0019] Server 110 and computing device 102 may be standalone computing devices, management servers, web servers, mobile computing devices, or any other electronic devices or computing systems capable of receiving, transmitting, and processing data. In other embodiments, server 110 and computing device 102 may represent a server computing system that utilizes multiple computers as a server system, such as a cloud computing environment. In another embodiment, server 110 and computing device 102 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smartphone, or any other programmable electronic device capable of communicating with other computing devices (not shown) in natural language environment 100 via network 101. In another embodiment, server 110 and computing device 102 represent a computing system that utilizes clustered computers and components (e.g., database server computers, application server computers, etc.) that operate as a single pool of seamless resources when accessed within natural language environment 100.

[0020] An embodiment of the present invention may reside on a server 110. The server 110 includes a natural language component 111 and a database 116. Other embodiments of the present invention may reside on a computing device 102.

[0021] The natural language component 111 provides functionality for generating the necessary NLI components (e.g., intents, entities, etc.) from the GUI code representation. The natural language component 111 can parse the GUI code representation (using machine learning) to extract all information relevant to the NLI. After extraction, the natural language component 111 can generate intent examples (and even entities based on the intent examples) for each type of NLI component using predefined patterns (using machine learning). Based on the generated intents, entities, and other components, the natural language component 111 can create NLI dialog boxes and / or other NLI components (using machine learning).

[0022] The database 116 is a repository of data used by the natural language component 111. The database 116 can be implemented by any type of storage device capable of storing data and configuration files that can be accessed and utilized by the server 110, such as a database server, a hard disk drive, or flash memory. The database 116 uses one or more of several techniques known in the art to store information. In the illustrated embodiment, the database 116 resides on the server 110. In other embodiments, the database 116 can reside elsewhere within the natural language environment 100, provided that the natural language component 111 can access the database 116. The database 116 can store information related to, but not limited to, predetermined intent examples, predetermined specific conversational components, previously stored mapping information between NLI components and GUI components, information for determining GUI code from software source code, and general mapping information for action calls and other GUI components.

[0023] In the illustrated embodiment, the natural language component 111 includes a component mapping component 121, an intent and entity component 122, an action mapping component 123, and a generation component .

[0024] As described in more detail herein below, the component mapping component 121 of the present invention provides functionality for mapping GUI components to NLI components. For example (see FIG. 2C), upon receiving a GUI code representation of the application / website to be parsed, all GUI components (e.g., buttons, dropdowns, radio buttons, charts, etc.) are recognized by their declarations in the code and mapped to their corresponding NLI components (e.g., a dropdown selection is mapped to a single-select input NLI component). Note that the GUI code representation can be derived (i.e., manually by a user or automatically determined) from the main source code of the application / website software.

[0025] Parsing the GUI code representation includes extracting all information related to the NLI, which may include, but is not limited to, i) entities, ii) synonyms for entities, iii) mappings between GUI components and NLI components, iv) actions (i.e., the onClick method of a button), and v) any other information that may be related to the NLI.

[0026] As described in more detail herein below, the intents and entities component 122 of the present invention provides functionality for creating intents / intent examples and entities. Intents / intent examples are goals or objectives expressed from input such as "report field" or "query value," and the system uses predefined patterns and examples extracted from the GUI to train its NLU (Natural Language Understanding) to better identify these intents when receiving input from a user.

[0027] Additionally, the intent and entity component 122 of the present invention has the ability to create new predefined patterns and / or edit existing predefined patterns. Predefined patterns (i.e., like templates) are intent instances with placeholders for entities that can be used to generate multiple intent instances. For example, when the system looks at the pattern for the "report" intent, it has the following: "The value of {field-name} is {field-value}." The system creates many example intents from this pattern by replacing {field-name} and {field-value} with values ​​automatically extracted from the code. In Figure 2A, "site id" and "work order id" are example {field-name} values, while example {field-value} values ​​need to be defined or inferred from the code. For example, for "site id," the corresponding {field-value} can be "any number." Users may edit existing predefined intent patterns based on their application needs.

[0028] There are two options for determining these predetermined patterns. The first option is to use a "package" of patterns already defined in the system (i.e., patterns for the "report" intent that the system already has), and the developer can add / remove patterns from this list. The package of patterns can be obtained from a database (i.e., database 116) or from another server. The second option is for the developer to create their own list of patterns and intents. For example, in one embodiment, an intent base set of patterns for a particular intent (i.e., the "report" intent) can be determined. In one embodiment, the base set of patterns can be used, or the developer can edit / modify existing base patterns to suit the current extraction.

[0029] For every intent (i.e., the "report" intent), there are predefined entities (for "report," the entities could be "field" and "value"), and only training examples for these entities are extracted (e.g., for "value," examples could be "low," "high," etc.).

[0030] Entities represent information from user input that is relevant to the user's intent. For example, if a user wants to report a field, the system needs to identify the intent ("report field") and the field (entity) the user wants to report. For example, in the "set oil level to low" example, the intent is "report value", the "field name" entity is "oil level", and the "field value" entity is "low". In Figure 2C, "field" and "fixed data" are all entities. As another example, in the "inspection form - single choice" example, the "field name" entity is "inspection form" and the "field value" entity has the values ​​generated from the code (all possible answers from the original dropdown). The "single choice" part refers to the type of NLU component the GUI component is mapped to; in this case, "dropdown" is mapped to "single choice".

[0031] In an alternative embodiment, the intent and entity component 122 can utilize NLI components to be used as entities (i.e., a single-selection field NLI component requires a value entity). In an embodiment, for these NLI components, the required entities can be populated. The population process can be done in a similar manner to extracting intent instances, but with the component type in mind (i.e., a closed single-selection field NLI component requires a fixed list of possible values).

[0032] As described in more detail herein below, the action mapping component 123 of the present invention provides a mapping function for actions (e.g., extracts, callbacks, etc.). As part of the action mapping, the action mapping includes a mapping between an action call in the GUI code representation (e.g., a button's "onClick" callback function) and the backend operation of the NL component. During the action mapping, the action mapping component 123 can search for and identify restrictions on the action (e.g., a drop-down component may have predetermined allowed values) that need to be considered in the NLI.

[0033] Action mapping means that all the extracted callbacks or code (that the GUI component had) are referenced in the corresponding NLI component and used in the return value of the NLI component.

[0034] Further functionality of the action mapping component 123 includes building predetermined specific conversation components. Predetermined specific conversation components are conversation building blocks (not visible to the user) that are used to build a conversation. These can be considered equivalent to component classes in a GUI (e.g., a drop-down menu class). To build the blueprint for an entire conversation, embodiments use these generic blocks (e.g., "report single field" and "query field value").

[0035] As described in more detail below, the generation component 124 of the present invention provides the functionality to generate natural dialogue based on the results from the component mapping component 121, the intent and entity component 122, and the action mapping component 123. The generation component 124 can use the NLI components, intents, entities, and actions, along with predefined domain-specific conversation components, to generate a natural language dialogue file.

[0036] Figure 2A is a screenshot of a main screen of a sample application, according to one embodiment of the present invention. It shows a fictitious application of a work order inspection system with voice input and / or voice commands (i.e., the IBM® Maximo Inspection screen is used for illustrative purposes only and is not an actual screenshot of the embodiment). Figure 2B is a screenshot of a settings screen of the fictitious application (from Figure 2A), according to one embodiment of the present invention.

[0037] Figure 2C illustrates an example GUI to NLI mapping for the sample application (of Figure 2A) according to one embodiment of the present invention. In Figure 2C, all "fields" and "fixed data" are shown as example entities. Thus, "dropdown" is mapped to "single select."

[0038] Figure 2D is a diagram illustrating a sample conversation and corresponding logic according to one embodiment of the present invention. Figure 2E is a diagram illustrating another sample conversation and corresponding logic according to one embodiment of the present invention. Both figures (2D and 2E) provide possible NLU / NLI dialog representations of a fictitious application.

[0039] FIG. 3 is a flow chart illustrating the operation 300 of the natural language component 111 according to one embodiment of the present invention.

[0040] The natural language component 111 receives a GUI code representation (step 302). In one embodiment, the natural language component 111 receives a GUI code representation of the application / website to be parsed. As an example (see FIG. 2A), each step will be explained using a sample work order inspection application. The natural language component 111 retrieves the main source code of the work order inspection application. The natural language component 111 can identify the GUI code from the main source code of the application. Furthermore, the natural language component 111 can retrieve all relevant and necessary information (i.e., information not related to the main source code) from the application to be analyzed.

[0041] The natural language component 111 maps the GUI code representation (step 304). In one embodiment, the natural language component 111 maps the GUI code representation to one or more NLI components via the component mapping component 121. For example, all GUI components (e.g., buttons, dropdowns, radio buttons, charts, etc.) are recognized by declarations in the code and mapped to corresponding NLI components (i.e., a dropdown selection is mapped to an NLI component with a single select input).

[0042] The natural language component 111 determines a predetermined intent pattern (step 306). In one embodiment, the natural language component 111, via the intent and entity component 122, determines the predetermined intent pattern based on one or more NLI components.

[0043] The natural language component 111 creates intent examples (step 308). In one embodiment, the natural language component 111 creates intent examples based on predefined intent patterns via the intent and entity component 122. For example, every NLI component has a corresponding predefined intent and a predefined intent pattern. For each matching component, the natural language component 111 extracts the corresponding label, ID, and name from the GUI code. The natural language component 111 creates intent examples for the required intent via the intent and entity component 122 (i.e., the intent pattern). If the user wants to report a field, the natural language component 111 can identify the intent as "Report Field" (from FIG. 2C).

[0044] The natural language component 111 creates entities (step 310). In one embodiment, the natural language component 111 creates entities based on intent examples via the intent and entity component 122. Some NLI components also use entities (i.e., a single-select field NLI component requires a value entity). In an embodiment, for these NLI components, the required entities must be entered in a similar manner to extracting intent examples, but with the component type in mind (i.e., a closed single-select field NLI component requires a fixed list of possible values). For example, based on the intent "report a field" (see step 308 above), the entities can be "field" and "fixed data" (see FIG. 2C). As another example, for the same "report" intent, there can be predefined entities such as "field" and "value." However, only training examples of these entities are extracted (i.e., for "value," examples could be "low," "high," etc.).

[0045] Furthermore, the natural language component 111 can map the actions of the GUI components via the action mapping component 123 (step 312). Any callbacks or code extractions that the GUI components had are referenced in the corresponding NLI components and used in the return values ​​of the NLI components. Finally, the natural language component 111 can create predetermined specific conversation components via the action mapping component 123. The predetermined specific conversation components are conversation building blocks (not visible to the user) used to build a conversation. They can be considered equivalent to component classes in the GUI (i.e., classes of drop-down menus). To build the blueprint of the entire conversation, embodiments use these generic blocks (e.g., "report a single field" or "query a field value").

[0046] The natural language component 111 generates a natural language dialog (step 314). In one embodiment, the natural language component 111, via the generation component 124, generates the natural language dialog based on the intent, the entities, the mapped actions, and the specific predetermined conversational components. For example, there are two possible dialog boxes (see FIGS. 2D and 2E) that are generated based on the intent, the examples, the mapped actions, and the specific predetermined conversational components:

[0047] 4 illustrates a block diagram 400 of components in a natural language component 111 application in accordance with an exemplary embodiment of the present invention. Note that FIG. 4 is merely an illustration of one implementation and is not intended to imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made.

[0048] 4 includes processor(s) 401, cache 403, memory 402, persistent storage 405, communication unit 407, input / output (I / O) interface(s) 406, and communications fabric 404. Communications fabric 404 enables communication between cache 403, memory 402, persistent storage 405, communication unit 407, and input / output (I / O) interface 406. Communications fabric 404 can be implemented with any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripherals, and other hardware components in a system. For example, communications fabric 404 can be implemented with one or more buses or crossbar switches.

[0049] Memory 402 and persistent storage 405 are computer-readable storage media. In this embodiment, memory 402 includes RAM. Generally, memory 402 may include any suitable volatile or non-volatile computer-readable storage medium. Cache 403 is fast memory that improves the performance of processor 401 by retaining recently accessed data and data near recently accessed data from memory 402.

[0050] Program instructions and data (e.g., software and data) used to implement embodiments of the present invention may be stored in persistent storage 405 and memory 402 for execution by each of the one or more processors 401 via cache 403. In one embodiment, persistent storage 405 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk, persistent storage 405 may include a solid-state hard drive, a semiconductor storage device, a ROM, an erasable programmable ROM (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0051] The media used by persistent storage 405 may be removable. For example, a removable hard drive may be used for persistent storage 405. Other examples include optical disks, magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 405. The natural language components 111 may be stored in persistent storage 405 for access and / or execution by one or more of the processors 401 via cache 403.

[0052] In these examples, communications unit 407 enables communication with other data processing systems or devices. In these examples, communications unit 407 includes one or more network interface cards. Communications unit 407 may enable communication using either or both physical and wireless communications links. Program instructions and data (e.g., natural language component 111) used to implement embodiments of the present invention may be downloaded to persistent storage 405 via communications unit 407.

[0053] The I / O interface 406 allows data to be input and output from other devices that may be connected to each computer system. For example, the I / O interface 406 allows connection to one or more external devices 408, such as a keyboard, keypad, touchscreen, or other suitable input device or combination thereof. The external device 408 may also include portable computer-readable storage media, such as thumb drives, portable optical disks, portable magnetic disks, and memory cards. Program instructions and data used to implement embodiments of the present invention (e.g., the natural language component 111) may be stored on such portable computer-readable storage media and loaded into persistent storage 405 via the I / O interface 406. The I / O interface 406 also connects to a display 409.

[0054] Display 409 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0055] The programs described herein are identified based on the application in which they are implemented in particular embodiments of the invention, although specific program names herein are used for convenience only and, therefore, the invention is not limited to use with the particular application identified and / or suggested by such names.

[0056] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0057] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. The computer-readable storage medium may be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, punch cards, or mechanically encoded devices having instructions recorded on ridge-in-groove structures, and the like, and suitable combinations thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over a wire.

[0058] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing device / processing device. Alternatively, they can be downloaded to an external computer or external storage device via a network (e.g., the Internet, a LAN, a WAN, or a wireless network, or a combination thereof). The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium in the respective computing device / processing device for storage.

[0059] The computer-readable program instructions for carrying out the operations of the present invention can be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions can execute entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a LAN or WAN, or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to customize the electronic circuitry for carrying out aspects of the present invention.

[0060] Aspects of the present invention are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. Each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer-readable program instructions.

[0061] These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, whereby the instructions, executed by the processor of such computer or other programmable data processing apparatus, create means for performing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored on a computer-readable storage medium that can instruct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner. The computer-readable storage medium having instructions stored thereon thereby constitutes an article of manufacture including instructions for performing aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0062] Computer-readable program instructions may also be loaded into a computer, other programmable apparatus, or other device and a series of operational steps executed on the computer, other programmable apparatus, or other device to create a computer-implemented process, whereby the instructions executing on the computer, other programmable apparatus, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.

[0063] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for performing specific logical functions. In some other implementations, the functions shown in the blocks may be executed in an order different from that shown in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may be executed in reverse order, depending on the functionality involved. Note that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs specific functions or operations or executes a combination of dedicated hardware and computer instructions.

[0064] While various embodiments of the present invention have been described by way of example, they are not intended to be exhaustive or limited to these embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope of the present invention. The terms used herein have been selected to best explain the principles of the embodiments, their practical applications, or technical improvements over commercially available technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for creating natural language (NL) dialog from graphical user interface (GUI) code, comprising: receiving a GUI code representation of an application; mapping the GUI code representation to one or more natural language interface (NLI) components; determining a predetermined intent pattern based on the one or more NLI components; creating one or more example intents based on the predetermined intent pattern; creating one or more entities based on the one or more example intents; Mapping action functions to backend operations of said one or more NLI components; generating a NL dialog based on the one or more intent examples, the one or more entities, mapped action functions, and predefined conversation components; 11. A computer-implemented method comprising:

2. receiving the GUI code representation of the application; Identifying the source code of the application; obtaining relevant information from the application; obtaining the GUI code from the source code of the application; The computer-implemented method of claim 1 , further comprising:

3. Mapping the GUI code representation to the one or more NLI components comprises: determining one or more declarations of said GUI code; Associating the one or more declarations of the GUI code with the one or more NLI components; The computer-implemented method of claim 1 , further comprising:

4. Determining the predetermined intent pattern based on the one or more NLI components includes: Retrieving a package of patterns from the database; Modifying the pattern package based on user requirements; The computer-implemented method of claim 1 , further comprising:

5. generating the one or more intent examples based on the predetermined intent pattern, extracting, for each matching NLI component, said GUI code representation, corresponding label, ID, and name; generating the one or more intent examples based on the GUI code representation, the corresponding label, the ID, and the name; The computer-implemented method of claim 1 , further comprising:

6. creating the one or more entities based on the one or more intent examples, The computer-implemented method of claim 1 , further comprising generating the one or more entities based on the one or more intent examples and component types of the one or more intent examples.

7. Mapping action functions to the backend operations of the one or more NLI components includes: Identifying restrictions on one or more invocations of the action; Associating the invocation of the GUI code action with a backend NL component operation along with the specified constraints; The computer-implemented method of claim 1 , further comprising:

8. 1. A computer program for creating natural language (NL) dialog from graphical user interface (GUI) code, comprising: program instructions for receiving a GUI code representation of an application; program instructions for mapping the GUI code representation to one or more natural language interface (NLI) components; program instructions for determining a predetermined intent pattern based on the one or more NLI components; program instructions for generating one or more intent examples based on the predetermined intent pattern; program instructions for creating one or more entities based on the one or more example intents; program instructions for mapping action functions to back-end operations of said one or more NLI components; program instructions for generating a NL dialog based on the one or more intent examples, the one or more entities, mapped action functions, and predefined conversation components; a computer program comprising:

9. receiving the GUI code representation of the application; program instructions for identifying source code of the application; program instructions for obtaining relevant information from the application; program instructions for obtaining the GUI code from the source code of the application; The computer program of claim 8 further comprising:

10. Mapping the GUI code representation to the one or more NLI components comprises: program instructions for determining one or more declarations of said GUI code; program instructions for associating the one or more declarations of the GUI code with the one or more NLI components; The computer program of claim 8 further comprising:

11. Determining the predetermined intent pattern based on the one or more NLI components includes: program instructions for retrieving a package of patterns from a database; program instructions for modifying said pattern package based on a user request; The computer program of claim 8 further comprising:

12. generating the one or more intent examples based on the predetermined intent pattern, program instructions for extracting, for each matching NLI component, said GUI code representation, corresponding label, ID, and name; program instructions for generating the one or more intent examples based on the GUI code representation, the corresponding label, the ID, and the name; The computer program of claim 8 further comprising:

13. creating the one or more entities based on the one or more intent examples, The computer program of claim 8 , further comprising program instructions for generating the one or more entities based on the one or more intent examples and component types of the one or more intent examples.

14. Mapping action functions to the backend operations of the one or more NLI components includes: program instructions for specifying restrictions on one or more invocations of the action; program instructions for associating the invocation of the GUI code action with a backend NL component operation with specified constraints; The computer program of claim 8 further comprising:

15. 1. A computer system for creating natural language (NL) dialog from graphical user interface (GUI) code, comprising: one or more computer processors; one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising: program instructions for receiving a GUI code representation of an application; program instructions for mapping the GUI code representation to one or more natural language interface (NLI) components; program instructions for determining a predetermined intent pattern based on the one or more NLI components; program instructions for generating one or more intent examples based on the predetermined intent pattern; program instructions for creating one or more entities based on the one or more example intents; program instructions for mapping action functions to back-end operations of said one or more NLI components; program instructions for generating a NL dialog based on the one or more intent examples, the one or more entities, mapped action functions, and predefined conversation components; 2. A computer system comprising:

16. receiving the GUI code representation of the application; program instructions for identifying source code of the application; program instructions for obtaining relevant information from the application; program instructions for obtaining the GUI code from the source code of the application; 16. The computer system of claim 15, further comprising:

17. Mapping the GUI code representation to the one or more NLI components comprises: program instructions for determining one or more declarations of said GUI code; program instructions for associating the one or more declarations of the GUI code with the one or more NLI components; 16. The computer system of claim 15, further comprising:

18. Determining the predetermined intent pattern based on the one or more NLI components includes: program instructions for retrieving a package of patterns from a database; program instructions for modifying said pattern package based on a user request; 16. The computer system of claim 15, further comprising:

19. generating the one or more intent examples based on the predetermined intent pattern, program instructions for extracting, for each matching NLI component, said GUI code representation, corresponding label, ID, and name; program instructions for generating the one or more intent examples based on the GUI code representation, the corresponding label, the ID, and the name; 16. The computer system of claim 15, further comprising:

20. creating the one or more entities based on the one or more intent examples, 16. The computer system of claim 15, further comprising program instructions for generating the one or more entities based on the one or more intent examples and component types of the one or more intent examples.

Images