Automatic domain-specific constrained decoding from speech input to structured resources

Abstract

Methods, systems, and computer program products for automatic domain-specific constrained decoding from speech input to structured resources are provided herein. A computer-implemented method includes converting at least a portion of at least a user-provided speech utterance to text by processing the at least user-provided speech utterance using an artificial intelligence-based automatic speech recognition model; automatically training an artificial intelligence-based decoding engine, wherein automatically training the artificial intelligence-based decoding engine includes constraining the artificial intelligence-based decoding engine based at least in part on a domain-specific model and the artificial intelligence-based automatic speech recognition model; and generating at least one of one or more domain-specific text outputs related to one or more structured resources associated with a domain and one or more domain-specific action outputs related to the one or more structured resources associated with the domain by processing at least a portion of the text using the artificial intelligence-based decoding engine.

Description

Technical Field

[0001] This application relates generally to information technology, and more specifically to data processing technology. Background Technology

[0002] More specifically, speech recognition technology attempts to convert audio signals into corresponding text output. However, conventional speech recognition methods often produce inaccurate and / or incomplete output. Summary of the Invention

[0003] In at least one embodiment, techniques are provided for automatic domain-specific constrained decoding from speech input to structured resources. One illustrative computer-implemented method includes converting at least a portion of a user-provided speech utterance into text by processing it using an AI-based automatic speech recognition model. The method also includes automatically training an AI-based decoding engine, wherein automatically training the AI-based decoding engine includes constraining the AI-based decoding engine at least partially based on a domain-specific model and the AI-based automatic speech recognition model. Additionally, the method includes generating at least one of one domain-specific text outputs associated with one or more structured resources associated with a domain and one domain-specific action outputs associated with one or more structured resources associated with a domain by processing at least a portion of the text using the AI-based decoding engine.

[0004] Another embodiment of the invention or elements thereof can be implemented in the form of a computer program product tangibly embodying computer-readable instructions that, when implemented, cause a computer to perform a plurality of method steps as described herein. Furthermore, another embodiment of the invention or elements thereof can be implemented in the form of a system including a memory and at least one processor coupled to the memory and configured to perform the method steps. Even further, another embodiment of the invention or elements thereof can be implemented in the form of an apparatus or elements thereof for performing the method steps described herein; the apparatus may include one or more hardware modules or a combination of hardware and software modules, wherein the software modules are stored in a tangible computer-readable storage medium (or a plurality of such media).

[0005] These and other objects, features, and advantages of the present invention will become clear from the following detailed description of illustrative embodiments thereof, which is taken in conjunction with the accompanying drawings. Attached Figure Description

[0006] Figure 1 This is a diagram illustrating a system architecture according to an exemplary embodiment of the present invention;

[0007] Figure 2This is a simplified diagram illustrating the system architecture according to an exemplary embodiment of the present invention;

[0008] Figure 3 This is a diagram illustrating a system architecture according to an exemplary embodiment of the present invention;

[0009] Figure 4 A diagram illustrating the system architecture according to an exemplary embodiment of the present invention;

[0010] Figure 5 A diagram illustrating the system architecture according to an exemplary embodiment of the present invention;

[0011] Figure 6 This is a flowchart illustrating a technique according to an exemplary embodiment of the present invention;

[0012] Figure 7 This is a system diagram of an exemplary computer system on which at least one embodiment of the present invention can be implemented;

[0013] Figure 8 A cloud computing environment according to an example embodiment of the present invention is described; and

[0014] Figure 9 An abstract model layer according to an example embodiment of the present invention is described. Detailed Implementation

[0015] As described herein, at least one embodiment includes automated domain-specific constrained decoding from speech input to a structured resource. Such embodiments include using constrained decoding to learn and / or process contextual information relating to the spoken and / or being analyzed utterance. As those skilled in the art will understand, speech recognition typically involves converting speech signals into corresponding text. Thus, one or more embodiments may include converting speech signals into arbitrary structured resources, such as, for example, forms. As used herein, constrained decoding refers to the concept that not only can text be transcribed, but the process is constrained by form elements. Accordingly, one or more embodiments include generating output that includes actions on form elements constrained by the form (e.g., if the form includes radio buttons, such embodiments may include selecting a relevant form element when audio corresponding to a relevant word is processed).

[0016] Furthermore, in one or more embodiments, speech variations dominated by frequently occurring phonemes are used in conjunction with domain information (in conjunction with the decoding task). Additionally, as detailed herein, at least one embodiment includes training and implementing an Automatic Speech Recognition (ASR) model to decode resources of arbitrary structure (e.g., selecting radio buttons, filling out forms, etc.).

[0017] Figure 1 This is a diagram illustrating a system architecture according to an exemplary embodiment of the present invention. By way of example, Figure 1 The description depicts a scenario where user 102 provides spoken utterances to ASR encoder 104, which processes the spoken utterances and generates output (which may include, for example, one or more phonemes and / or one or more graphemes). The output is provided to ASR constrained decoding engine 110, which also receives input from domain model 108, trained using domain context information 106 (e.g., tables, web pages, forms, etc.). Based at least partially on processing the output from ASR encoder 104 in conjunction with at least a portion of the input provided by domain model 108, ASR constrained decoding engine 110 generates one or more domain-specific text outputs 112 and / or one or more domain-specific action outputs 114. The following is a continuation of... Figures 2 to 5 Further details Figure 1 Illustrative example implementations of the system described herein.

[0018] Figure 2 This is a simplified diagram illustrating a system architecture according to an exemplary embodiment of the present invention. More specifically, Figure 2 An example implementation involving a web form and its corresponding radio buttons is depicted. Through this example, Figure 2 The scenario depicts a user 202 providing the spoken phrase "I am right-handed" to an ASR encoder 204, which processes the spoken phrase and generates output (which may include, for example, one or more phonemes and / or one or more glyphs associated with the "I am right-handed" input). The output is provided to an ASR constrained decoding engine 210, which also receives input from a domain model 208 trained using a language model tuned on a web form 206, which includes radio buttons for "right-handed," "left-handed," and "other." Based at least in part on processing the output from the ASR encoder 204 in conjunction with at least a portion of the input provided by the domain model 208, the ASR constrained decoding engine 210 generates a domain-specific action output 214, which includes selecting the "right-handed" radio button on the web form.

[0019] Figure 3 This is a diagram illustrating a system architecture according to an exemplary embodiment of the present invention. More specifically, Figure 3 This describes a sample implementation involving a web form and one or more corresponding text selections. The example demonstrates... Figure 3 The diagram depicts a user 302 providing spoken utterances to an ASR encoder 304, which processes the spoken utterances and generates outputs including versions and / or variations of the input spoken utterances (here, "The Moon," "The Moin," and "The Men"). The outputs are provided to an ASR constrained decoding engine 310, which also receives input from a domain model 308 trained using a language model tuned on a web form 306, which includes text input fields for "City" and "Phone." Based at least in part on processing the output from the ASR encoder 304 in conjunction with at least a portion of the input provided by the domain model 308, the ASR constrained decoding engine 310 generates a domain-specific text output 312 and a domain-specific action output 314. The domain-specific text output 312 results in the selection of a "DesMoines" radio button on the web form, and the domain-specific action output 314 includes the automatic input of "DesMoines" in the appropriate text input field on the web form.

[0020] As illustrated in the example figures and described in further detail herein in conjunction with one or more embodiments, constrained decoding facilitates understanding the context in which a given utterance is made. In the case of ASR, speech variations can be dominated by frequently occurring phonemes. Using such input, incorporating domain-specific information (e.g., a domain model), further enhances decoding.

[0021] Furthermore, one or more embodiments may include semantic parsing in conjunction with a constrained decoder (also referred to herein as a constrained decoding engine). Constrained decoding techniques may be used, for example, in conjunction with semantic parsing to generate a program from natural language constrained by a syntactic and functional catalogue of a given execution engine. In at least one embodiment, a hierarchical long short-term memory (LSTM) decoder that decodes in a syntactically constrained manner may be used. In such embodiments, the LSTM decoder first generates the grammar for the function y = f(x), [y, f, x]. Subsequently, the LSTM decoder recursively fills slots with further predictions of [y, f, x]. As used in this context, a slot refers to an entity within a domain model of a given sentence. For example, in a dialogue, for each utterance, a slot includes a mapping from entity name to entity type (e.g., if a user wants to fly from Boston to San Jose, the slots for the relevant utterance might include "source_destination: Boston" and "target_destination: San Jose"). Such slots can be useful in a conversation because they are generally not repeated in the dialogue. For example, if the user's next utterance is "I want a morning flight," all slots can be collected and used to generate a query in the database with one or more appropriate filters. Hypertext Markup Language (HTML) generation and / or parsing methods can also use similar recursive approaches.

[0022] Figure 4 This is a diagram illustrating a system architecture according to an exemplary embodiment of the present invention. More specifically, Figure 4 An example implementation involving ASR-constrained decoding for HTML is depicted. Through examples, Figure 4The description depicts a user 402 providing spoken utterances to an ASR encoder 404, which processes the spoken utterances and generates output embeddings (e.g., a high-level representation of the speech input, e.g., in the form of floating-point vectors). The output embeddings are provided to an ASR constrained decoding engine 410, which also receives input from a language model 408 trained using text 406 observed in or around one or more web forms. The ASR constrained decoding engine 410 additionally receives input from an HTML-specific feature directory 444 (which may store details, for example, related to the semantics and execution profiles of each associated HTML), common synonyms for different fields, and HTML form-related input. Specifically, an empty HTML form 440 is processed by an HTML parsing component 442, which parses the empty HTML form 440 into a string encoding, which is then provided to the ASR constrained decoding engine 410. In at least one embodiment, parsing the HTML form 440 into a string encoding may include using the HTML form 440 and converting the HTML form 440 into a machine representation via parsing means.

[0023] Based at least in part on processing at least a portion of the input provided by the language model 408, the HTML-specific feature catalog 444, and the HTML parsing component 442, the ASR constrained decoding engine 410 generates domain-specific action output 414, which includes syntactically constrained filling of missing and / or unchecked slots in an HTML form (e.g., HTML form 440). As used herein, with respect to syntactically constrained manner, for example, if the identified field is a radio button, the field will be clicked, and if the identified field is a text field, the corresponding text will be filled. Depending on the form element type (i.e., constrained by the form element type), this embodiment includes taking and / or initiating appropriate execution steps (filling, checking, etc.).

[0024] In one or more embodiments, such as Figure 4 In the example embodiments depicted herein, the ASR encoder may include, for example, at least one convolutional neural network (CNN) encoder, at least one quantizer, at least one transformer encoder, and / or at least one contextual representation. Additionally, in such embodiments, the ASR constrained decoder (also referred to herein as an ASR constrained decoding engine) may include, for example, one or more combined attention mechanisms and LSTM model (ATT-LSTM) components.

[0025] Figure 5 A diagram illustrating a system architecture according to an exemplary embodiment of the present invention. More specifically, Figure 5An example implementation of a running example involving ASR-constrained decoding for HTML is described. By way of example, Figure 5 The description depicts a user 502 providing the spoken phrase "I am right-handed" to an ASR encoder 504, which processes the spoken phrase and generates an output embedding. The output embedding is provided to an ASR constrained decoding engine 510, which also receives input from a language model 508, an HTML-specific feature directory 544, and HTML form-related input. Specifically, an empty HTML form 540 is processed by an HTML parsing component 542, which parses the empty HTML form 540 into a string encoding, which is then provided to the ASR constrained decoding engine 510.

[0026] At least in part, based on processing at least a portion of the input provided by the combined language model 508, the HTML-specific feature catalog 544, and the output from the ASR encoder 504, and the HTML parsing component 542, the ASR constrained decoding engine 510 generates constrained decoded text output 512 (e.g., in...). Figure 5 In one embodiment, text output 512 includes an instruction that the execution of a given HTML is prepared, which is then applied to a form, and is then used to execute domain-specific action output 514, which includes automatically selecting a “right-handed” radio button on a given web form (i.e., an empty HTML form 540).

[0027] Therefore, one or more embodiments may include implementing a model in which an ASR encoder-decoder is trained to decode an HTML program using examples. In such embodiments using a new language (e.g., a new Structured Query Language (SQL), etc.), the model is retrained accordingly. Additionally, the constrained decoder ensures that the output is syntactically correct. For example, consider... Figure 4 and Figure 5 In the embodiments described herein, the constrained decoder uses an empty HTML form and an HTML function directory to facilitate the insertion of one or more speech-to-text decoding tokens into valid slots on the HTML form.

[0028] Similarly, in one or more embodiments, ASR encodes speech utterances and decodes at least a portion of the utterance guided by a language model (LM) (e.g., “boy”). The LM can also help the ASR decoder determine one or more similarities between variations of utterances in the text space (e.g., the similarity between “boy” and “male”).

[0029] Alternatively or concurrently, at least one embodiment includes a framework for self-supervised learning using speech representations in conjunction with LSTM (e.g., a wav2vec2.0 encoder). More specifically, such embodiments may include a fully supervised approach using at least one pre-trained speech encoder (e.g., a wav2vec2.0 encoder) and a hierarchical LSTM decoder. In conjunction with training such an encoder and decoder, the training data (X, y) may include speech signals from a user providing information for a given form (e.g., a web form, an HTML form, etc.) in a natural setting (i.e., without requiring specific prompts or waiting for a system to identify the correct portion of the form). Furthermore, in one or more embodiments, the ground-based populated form may include text containing HTML tags in correct grammar and response tokens.

[0030] In at least one embodiment, such training subroutines may include speech encoding, wherein a given speech signal is encoded using, for example, wav2vec 2.0 pre-trained on speech data of the same language. The model may produce a fixed-length vector codebase. Furthermore, ground-based HTML files are parsed and tokenized, and valid HTML tags are preprocessed. For example, consider the following: <label for＝"fname”> First name:< / label> <input type＝"text”id＝"fname”name＝"fname”value＝"John”> is converted to<input_fn> ="Firstname:"<input_val> =“John”. In one or more embodiments, for LSTM to predict only missing slots, LSTM predicts missing token names with the assistance of LM.

[0031] One or more embodiments may also include one or more extensions to implement a given decoder. Stacks of LSTMs may also be used to predict hierarchical trees, such that the output prediction resembles a tree structure slot-filled LSTM. Additionally or alternatively, in at least one embodiment, a language conversion decoder model with SQL functions replaced by HTML functions (e.g., a text2SQL decoder model) may be utilized and / or implemented. In such embodiments, this model can produce additional HTML tags as permitted by the scores of a given function catalog. In combination with one or more variations, at least one additional language model may be used to prune search results to ensure that voice tokens are converted into known language words, but only for missing slots in a given form.

[0032] In deep learning settings, as detailed herein, one or more embodiments include the use of constrained LSTM. For example, in a speech-to-text context, due to the large amount of data involved in speech, noise, etc., the model may use a long-beam search technique (e.g., with a beam size of 100 or greater) to search for a text copy of a given speech. Furthermore, the example speech-to-text model uses at least one language model to assist such beam search to produce decoded text with lower and / or reduced uncertainty than the language model.

[0033] In addition, at least one embodiment may include a form that is filled directly from spoken speech by utilizing the structure of HTML tags, and the content is filled in a slot-filled manner (in combination with one or more given structured resources).

[0034] Figure 6 This is a flowchart illustrating a technique according to an embodiment of the present invention. Step 602 includes converting at least a portion of at least one user-provided speech utterance into text by processing at least one user-provided speech utterance using an AI-based automatic speech recognition model. In one or more embodiments, converting at least a portion of at least one user-provided speech utterance into text includes processing at least one user-provided speech utterance using an AI-based automatic speech recognition model, determining multiple speech variations of the user-provided speech utterance at least partially based on one or more phonemes. Furthermore, in at least one embodiment, the AI-based automatic speech recognition model includes a supervised encoder regarding speech representation.

[0035] Step 604 includes automatically training an AI-based decoding engine, wherein automatically training the AI-based decoding engine includes constraining the AI-based decoding engine based at least in part on a domain-specific model and an AI-based automatic speech recognition model. In at least one embodiment, the AI-based decoding engine may include a set of one or more hierarchical LSTM decoders and / or one or more combined ATT-LSTM components. Furthermore, in one or more embodiments, constraining the AI-based decoding engine is also based at least in part on one or more structured resources associated with the domain.

[0036] Furthermore, one or more embodiments may include using domain context information to automatically train a domain-specific model. Further, in such embodiments, the domain-specific model may include a language-specific model and / or a topic-specific model.

[0037] Step 606 includes processing at least a portion of the text using an AI-based decoding engine to generate at least one of one or more domain-specific text outputs associated with one or more structured resources associated with the domain and one or more domain-specific action outputs associated with one or more structured resources associated with the domain. Also as detailed herein, in one or more embodiments, the one or more structured resources may include one or more web forms (e.g., one or more HTML-based web forms).

[0038] Figure 6 The techniques described may also include automatically performing one or more domain-specific action outputs. In one or more embodiments, automatically performing one or more domain-specific action outputs may include automatically selecting one or more radio buttons that are associated with one or more structured resources associated with the domain and / or automatically inserting text into one or more web forms associated with the domain.

[0039] Furthermore, at least one embodiment may also include retraining the AI-based decoding engine at least in part based on at least one of the one or more domain-specific text outputs and the one or more domain-specific action outputs. Further, in one or more embodiments, the software implementing the method may be provided as a service in a cloud environment.

[0040] It should be recognized that, as used herein, a “model” refers to a set of executable instructions and data values ​​stored in an electronic digital store that are interconnected and capable of receiving and responding to programming or other digital calls, invocations, or requests for parsing based on specified input values ​​to produce one or more output values ​​that can be used as the basis for computer-implemented recommendations, output data displays, machine control, etc. Those skilled in the art will find it convenient to express models using mathematical equations; however, the form of such expressions does not limit the models disclosed herein to abstract concepts; rather, each model herein has a practical application in a computer in the form of stored executable instructions and data used to implement the model.

[0041] As described here, Figure 6The techniques described herein may also include providing a system comprising different software modules, each of which is embodied on a tangible computer-readable and recordable storage medium. For example, all modules (or any subset thereof) may be on the same medium, or each may be on a different medium. Modules may include any or all of the components shown in the figures and / or described herein. In embodiments of the invention, these modules may, for example, run on a hardware processor. The different software modules of the system described above, executed on a hardware processor, can then be used to perform method steps. Further, a computer program product may include a tangible computer-readable and recordable storage medium having code adapted to perform at least one method step described herein, the method comprising providing the different software modules to the system.

[0042] also, Figure 6 The techniques described herein can be implemented via a computer program product that may include computer-usable program code stored in a computer-readable storage medium within a data processing system, wherein the computer-usable program code is downloaded from a remote data processing system via a network. Furthermore, in embodiments of the invention, the computer program product may include computer-usable program code stored in a computer-readable storage medium within a server data processing system, wherein the computer-usable program code is downloaded via a network to a remote data processing system for use with the remote system.

[0043] Embodiments of the present invention or elements thereof may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and configured to perform exemplary method steps.

[0044] Furthermore, embodiments of the present invention can utilize software running on a computer or workstation. (See reference) Figure 7This implementation may, for example, employ a processor 702, a memory 704, and an input / output interface, such as a display 706 and a keyboard 708. The term "processor" as used herein is intended to include any processing device, for example, a processing device including a CPU (Central Processing Unit) and / or other forms of processing circuitry. Furthermore, the term "processor" may refer to more than one single processor. The term "memory" is intended to include memory associated with a processor or CPU, such as RAM (Random Access Memory), ROM (Read-Only Memory), fixed memory devices (e.g., hard disk drives), removable memory devices (e.g., disk drives), flash memory, etc. Furthermore, the phrase "input / output interface" as used herein is intended to include, for example, mechanisms for inputting data to the processing unit (e.g., a mouse), and mechanisms for providing results associated with the processing unit (e.g., a printer). The processor 702, the memory 704, and the input / output interface such as the display 706 and the keyboard 708 may be interconnected, for example, via a bus 710 that is part of a data processing unit 712. Suitable interconnects (e.g. via bus 710) may also be provided to network interface 714 (such as a network card) and media interface 716 (such as a floppy disk or CD-ROM drive). Network interface 714 may be provided to interface with a computer network, and media interface 716 may be provided to interface with media 718.

[0045] Therefore, as described herein, computer software including instructions or code for performing the methods of the present invention can be stored in an associated memory device (e.g., ROM, fixed or removable memory) and, when ready for use, can be partially or fully loaded (e.g., loaded into RAM) and implemented by a CPU. Such software may include, but is not limited to, firmware, resident software, microcode, etc.

[0046] A data processing system suitable for storing and / or executing program code will include at least one processor 702, which is directly or indirectly connected to a memory element 704 via a system bus 710. The memory element may include local memory used during the actual implementation of the program code, mass storage, and a cache memory that provides temporary storage for at least some of the program code to reduce the number of times code must be retrieved from the mass storage during implementation.

[0047] Input / output or I / O devices (including but not limited to keyboard 708, display 706, point devices, etc.) can be coupled to the system directly (e.g. via bus 710) or through an intermediate I / O controller (omitted for clarity).

[0048] Network adapters (such as network interface 714) can also be attached to the system to enable the data processing system to connect to other data processing systems or remote printers or storage devices via accessed private or public networks. Modems, cable modems, and Ethernet cards are just some of the types of network adapters currently available.

[0049] As used herein, including the claims, "server" includes a physical data processing system that runs server programs (e.g., such as...). Figure 7 (System 712 shown). It will be understood that such a physical server may or may not include a monitor and keyboard.

[0050] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.

[0051] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punched cards, or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0052] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device or to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the corresponding computing / processing device.

[0053] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.

[0054] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0055] These computer-readable program instructions may be provided to a computer processor or other programmable data processing apparatus to produce a machine, such that these instructions, which execute via the computer processor or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more boxes of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, thereby comprising an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0056] These computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, thereby causing these instructions to perform on the computer, other programmable apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0057] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the figures. For example, two blocks shown consecutively may actually be completed as a single step, executed simultaneously, substantially simultaneously, or with partial or complete temporal overlap, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0058] It should be noted that any method described herein may include additional steps of providing a system comprising various software modules embodied on a computer-readable storage medium; these modules may include any or all of the components detailed herein. The method steps may then be performed using the various software modules and / or submodules of the system as described above, which execute on hardware processor 702. Furthermore, the computer program product may include a computer-readable storage medium having code adapted for implementing at least one method step described herein, the method comprising providing these various software modules to the system.

[0059] In any case, it should be understood that the components shown herein can be implemented in different forms of hardware, software, or combinations thereof, such as application-specific integrated circuits (ASICs), functional circuits, suitably programmed digital computers with associated memory, etc. Given the teachings of the invention provided herein, those skilled in the art will be able to conceive of other implementations of the components of the invention.

[0060] Furthermore, it should be understood in advance that the implementation of the teachings listed herein is not limited to a specific computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any type of computing environment now known or developed in the future.

[0061] For example, cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0062] The features are as follows:

[0063] On-demand self-service: Cloud consumers can automatically and unilaterally configure computing power, such as server time and network storage, as needed, without human interaction with the service provider.

[0064] Extensive network access: Capabilities are available through the network and accessed via standard mechanisms that facilitate the use of heterogeneous thin client platforms or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0065] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0066] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.

[0067] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.

[0068] The service model is as follows:

[0069] Software as a Service (SaaS): This provides consumers with the ability to use a provider's applications running on cloud infrastructure. These applications can be accessed from different client devices via a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.

[0070] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created by the consumer or acquired using programming languages ​​and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of possible application hosting environments.

[0071] Infrastructure as a Service (IaaS): This provides consumers with the capability to offer processing, storage, networking, and other basic computing resources, where consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).

[0072] The deployment model is as follows:

[0073] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.

[0074] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0075] Public cloud: Cloud infrastructure available to the general public or large industry groups and owned by organizations that sell cloud services.

[0076] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).

[0077] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.

[0078] See now Figure 8The diagram illustrates an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 to which local computing devices used by cloud consumers can communicate. These local computing devices include, for example, personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platforms, and / or software as services that cloud consumers do not need to maintain on their local computing devices. It should be understood that... Figure 8 The types of computing devices 54A-N shown are intended to be illustrative only, and computing node 10 and cloud computing environment 50 can communicate with any type of computerized device via any type of network and / or network-addressable connectivity (e.g., using a web browser).

[0079] See now Figure 9 This demonstrates a cloud computing environment of 50 ( Figure 8 This provides a set of functional abstractions. It should be understood beforehand that... Figure 9 The components, layers, and functions shown are merely illustrative, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:

[0080] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and network and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0081] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75. In one example, the management layer 80 can provide the functionality described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources.

[0082] In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides the allocation and management of cloud computing resources to meet the required service levels. Service level agreement (SLA) planning and fulfillment 85 provides the pre-scheduling and procurement of cloud computing resources based on anticipated future needs according to the SLA.

[0083] Workload layer 90 provides examples of functionalities that can leverage a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis and processing 94; transaction processing 95 according to one or more embodiments of the invention; and automatic speech recognition 96.

[0084] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” are intended to also include the plural forms. It should also be understood that when the terms “comprises” and / or “comprising” are used in this specification, they specify the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of another feature, step, operation, element, component, and / or combination thereof.

[0085] At least one embodiment of the present invention can provide beneficial effects, such as automatic domain-specific constrained decoding from voice input to structured resources.

[0086] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method, comprising: At least a portion of the at least one user-provided speech utterance is converted into text by using an AI-based automatic speech recognition model to process the speech utterance provided by at least one user. Automatically train an AI-based decoding engine, wherein automatically training the AI-based decoding engine includes constraining the AI-based decoding engine at least in part based on a domain-specific model, one or more structured resources, and the AI-based automatic speech recognition model, wherein the one or more structured resources include one or more web forms associated with the domain. By using the AI-based decoding engine to process at least a portion of the text, one or more domain-specific text outputs are generated, and at least a portion of the one or more domain-specific text outputs is automatically inserted into at least one of the one or more web forms associated with the domain; as well as By using the AI-based decoding engine to process at least a portion of the text, one or more domain-specific actions are automatically performed within the one or more web forms associated with the domain; The method is performed by at least one computing device.

2. The computer-implemented method according to claim 1 further includes: Automatically execute one or more domain-specific action outputs.

3. The computer-implemented method according to claim 2, wherein, Automatically executing one or more domain-specific action outputs includes automatically selecting one or more radio buttons that combine one or more structured resources associated with the domain.

4. The computer-implemented method according to claim 1, wherein, The AI-based decoding engine includes a set of one or more hierarchical long short-term memory decoders.

5. The computer-implemented method according to claim 1, wherein, The AI-based decoding engine includes one or more combined attention mechanisms and Long Short-Term Memory (ATT-LSTM) components.

6. The computer-implemented method according to claim 1, wherein, Converting at least a portion of the at least one user-provided speech utterance into text includes: processing the at least one user-provided speech utterance using the AI-based automatic speech recognition model to determine, at least in part, multiple speech variations of the user-provided speech utterance based on one or more phonemes.

7. The computer-implemented method according to claim 1, wherein, The AI-based automatic speech recognition model includes a supervised encoder for speech representation.

8. The computer-implemented method according to claim 1, further comprising: The domain-specific model is automatically trained using domain environment information.

9. The computer-implemented method according to claim 1, wherein, The domain-specific model includes at least one of a model about a specific language and a model about a specific topic.

10. The computer-implemented method according to claim 1, wherein, The one or more web forms include one or more web forms based on Hypertext Markup Language.

11. The computer-implemented method according to claim 1, further comprising: The AI-based decoding engine is retrained at least in part based on at least one of the one or more domain-specific text outputs and the one or more domain-specific action outputs.

12. The computer-implemented method according to claim 1, wherein, The software that implements the method is provided as a service in a cloud environment.

13. A computer program product comprising program instructions executable by a computing device to cause the computing device to perform the steps of the method according to any one of claims 1-12.

14. A system comprising: The memory is configured to store program instructions; as well as A processor, operatively coupled to the memory, executes the program instructions to implement the steps of the method according to any one of claims 1-12.

Images