Conversion from natural language to application programming interfaces

Abstract

The system and method provide obtaining a corpus of training data based on multiple natural language processing sessions, wherein the corpus of training data comprises multiple training data input vectors and multiple reference data output vectors, where one of the reference data output vectors represents a natural language query about travel bookings, and one of the multiple training data input vectors represents a travel-based search result as a desired output generated from one of the multiple training data input vectors; and training a natural language to API model using the corpus of training data, wherein the natural language to API model is trained to generate predicted travel-based search results using a natural language query input.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority based on U.S. Patent Application No. 18 / 497,304, filed on October 30, 2023, and U.S. Provisional Patent Application No. 63 / 494,597, filed on April 6, 2023, the contents of each of which are incorporated herein by reference and made a part of this specification.

Background Art

[0002] Natural language processing systems include various components for receiving input from a user and processing that input to determine what the user means. In some embodiments, a natural language processing system receives text input such as text entered by a user or a transcription of the user's utterance. A natural language processing system can determine the meaning of the text input in a way that can be actuated by a computer application.

[0003] Embodiments of various progressive features will now be described with reference to the following drawings. Throughout the drawings, reference numbers may be reused to indicate corresponding relationships between elements mentioned. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of the disclosure.

Brief Description of the Drawings

[0004] [Figure 1] A diagram of an exemplary natural language processing system with various components for processing natural language queries from a user device, according to some embodiments. [Figure 2] A flowchart of an exemplary routine for generating training data and training a machine learning model, according to some embodiments. [Figure 3]This diagram illustrates the development of parameters for a natural language processing session and an application programming interface request based thereon, according to several embodiments. [Figure 4] This diagram illustrates the development of parameters for a separate natural language processing session and an application programming interface request based thereon, according to several embodiments. [Figure 5] This is a flowchart of exemplary routines for orchestrating a natural language processing session using an interactive query parameter manager or machine learning model, according to several embodiments. [Figure 6] This diagram illustrates a natural language processing session that generates application programming interface requests using a machine learning model, according to several embodiments. [Figure 7] This is a block diagram of an exemplary computing system configured to implement an aspect of the present disclosure, according to several embodiments. [Modes for carrying out the invention]

[0005] This disclosure covers generating training data for training a machine learning model using a natural language query processing pipeline. The machine learning model may be trained using the training data to translate natural language queries into computer-executable requests. Furthermore, this disclosure covers orchestrating natural language query processing across different pipelines to evaluate and improve the performance of a machine learning model trained to translate natural language queries into computer-executable requests.

[0006] Some traditional computing systems provide natural language user interfaces that allow users to access system functions and, otherwise, interact with the system using spoken or typed input that approximates a conversational language. These inputs may be called natural language queries or natural language requests (when expressed as commands, or otherwise not in the form of questions). Some examples of systems that provide such interfaces include travel booking systems, search engines, and customer service systems. Systems may use interactive multi-turn dialogue interfaces to obtain the information necessary to generate and execute application programming interface (API) requests in response to natural language queries from users. For example, some systems use natural language processing (NLP) to determine the intent behind a user's query and determine the corresponding executable request that the system will perform to provide information or take action in response to the user's intent. Some systems may use a database of questions or prompts to obtain parameters or other information for executing a request, while others may use natural language generation (NLG) processing (e.g., generative artificial intelligence) to generate natural language prompts and, otherwise, manage the dialogue. Each system prompt and corresponding user response may be called a single "turn" in a multi-turn dialogue. These systems typically require multiple turns to retrieve the parameters necessary to respond to a request. In some cases, the system may fail to retrieve some parameters even after multiple interaction turns. As a result, the system's performance may be insufficient, as defined in terms of computing resource efficiency, processing accuracy, latency, or user experience.

[0007] Several aspects of this disclosure address some or all of the above problems, among other things, by using machine learning models that translate natural language queries into API requests. Advantageously, such models can be trained to generate API requests from natural language queries with fewer dialogue turns after the user's initial natural language query, or to directly generate API requests with no additional dialogue turns at all. To obtain training data for training such models, a multi-turn dialogue pipeline may be used to manage a natural language processing session, process natural language queries, retrieve parameters, and generate API requests in response to natural language queries. The initial natural language query, the resulting API request, and optionally other information about the natural language processing session may be stored and used to train the model to generate API requests from the initial natural language query.

[0008] Additional aspects of this disclosure relate to distributing natural language queries across different processing pipelines, evaluating the performance of each pipeline, and adjusting the aspects of individual pipelines based on their performance. In some embodiments, one pipeline may use a machine learning model trained to generate API requests in response to natural language queries with few (or no) additional interaction turns, and another pipeline may be, or include, a multi-turn chat-based interaction pipeline used to acquire data for training the machine learning model. Advantageously, when the performance of one of the pipelines, such as the machine learning model pipeline, falls below a performance threshold, the machine learning model may be retrained, or the pipeline may be otherwise modified to improve its performance. In some embodiments, there may be multiple machine learning model pipelines, multiple chat-based interaction pipelines, or any combination thereof (e.g., using different types of models, models trained on different data, etc.).

[0009] An orchestration system, also referred to herein as an orchestrator, can determine which of several available request processing pipelines will process and respond to each incoming natural language query. The orchestrator or component of each pipeline may store data related to natural language processing, such as the natural language query input, the number of interaction turns required to generate the API request, and the user actions taken based on the API request's outcome. Based on this data, the performance of each pipeline (defined in terms of computing resource efficiency, processing accuracy, latency, or user experience) can be determined and corrected.

[0010] Referring to an exemplary embodiment, a multi-turn chat-based conversational pipeline (also simply called a “conversational pipeline” for brevity) may be implemented to process and respond to natural language queries. Queries handled by the conversational pipeline may relate to travel activities and bookings, such as hotels, flights, rental cars, or packages of such activities. The conversational pipeline may be configured to generate API requests, such as computer-executable search queries, to retrieve data in response to the user’s travel queries. As used herein, the terms “natural language query” and “natural language request” are not necessarily limited to questions and may also include other statements made by the user to prompt a response or other action. For example, a user may initiate a natural language processing session by speaking or typing a statement such as “Show me family travel packages for Thanksgiving.” Although not expressed as a question, it is clear that the user wants to explore possible travel packages related to the stated conditions.

[0011] An interactive pipeline may include a natural language understanding (NLU) component that evaluates either the text of a query entered by the user or the text of a query generated by the automatic speech recognition (ASR) component of the interactive pipeline. The NLU processing component may identify specific words or phrases in the query that reveal the user's intent when creating the query and provide details about that intent. These words or phrases may be called “named entities,” and the NLU processing component that identifies them may be called a named entity recognition (NER) component. In this embodiment, the NER component may identify the intent as “search for travel packages,” and may identify additional entities such as the “departure date” entity (based on the phrase “around Thanksgiving”) and the “number of travelers” entity (based on the phrase “family”).

[0012] To respond to the intent "search for travel packages," and therefore to the user's natural language query, the interactive pipeline may determine whether additional parameters are needed or whether identified parameters should be checked in order to generate an API request. Parameters for the API request may correspond to specific named entities. For example, the intent "search for travel packages" may require "departure date" and "number of travelers" parameters, which the user can specify via the corresponding named entities. Furthermore, the intent "search for travel packages" may require "destination" and "return date" parameters. The interactive pipeline may generate text or graphical user interface (GUI) components that are presented to the user in one or more interactive turns. In this embodiment, the interactive pipeline may generate prompts for the "destination" parameter (e.g., a text-based natural language prompt, a GUI map component, or a dropdown list) and another prompt for the "return date" parameter (e.g., a second text-based prompt, a GUI calendar component). The interactive pipeline may also generate one or more prompts to confirm parameters determined based on recognized named entities (e.g., a text-based summary and prompt for confirmation, one or more interactive GUI components with pre-selected determined parameters, etc.). Once a sufficient set of parameters has been obtained and an API request has been generated, the interactive pipeline may send the API request to an application or system for execution. In this embodiment, the interactive pipeline may send the API request to a travel search application, which may generate search results to present to the user. The natural language processing session may then continue or terminate.

[0013] An interactive pipeline can process many such natural language queries (e.g., thousands, hundreds of thousands, millions, or more) and generate and store telemetry data related to that processing. For example, for each natural language processing session, the interactive pipeline may store the initial natural language query that triggered the session, and the API request (and corresponding parameters) generated in response to that query. In some embodiments, additional telemetry data may be generated and stored, such as data reflecting the interaction turns between the initial query and the last API request, and contextual data about the natural language processing session (e.g., time, date, user identifier, etc.).

[0014] Telemetry data about natural language processing sessions managed by an interactive pipeline can be used to generate training data for training a machine learning model that translates natural language queries into API requests. For example, a corpus of training data pairs may be generated from telemetry data, so that a single training data pair may include the session's first natural language query as a training data input item and the last API request generated during the natural language processing session as a reference data output item, which is the desired or "correct" output generated by the machine learning model from the training data input item. In some embodiments, additional data may be included in the training data input items, such as data reflecting additional interaction turns between the first query and the last API request, contextual data about the natural language processing session (e.g., time, date, user identifier, etc.) or contextual data about the user (e.g., demographics, interaction history, etc.). For example, interaction history may provide indications such as what travel bookings the user has made and whether the user is currently on a trip to which the current query may be associated. Such history may be used when determining parameter values ​​for API requests and therefore when translating the user's natural language processing query into an API request.

[0015] A machine learning model may be trained to generate API requests from natural language queries or any other natural language input using training data generated from telemetry data. The trained model may be called a natural language to API model, or for brevity, an "NL2API model." In some embodiments, a natural language to API model may be a transformation model implemented using an artificial neural network (NN) or a variation thereof, such as a deep neural network (DNN), a recurrent neural network (RNN), a convolutional neural network (CNN), a transformer, an encoder-decoder, an ensemble of multiple models, or other models suitable for transformation. The exemplary machine learning model types and architectures described herein are illustrative only and are not intended to be limiting, mandatory, or exhaustive. In some embodiments, other types of models may be used, including those not based on NNs.

[0016] Model-based pipelines can be implemented to process and respond to natural language queries using a trained natural language-to-API model. The queries handled by the model-based pipeline can be of the same type as those handled by the interactive pipeline described above. For example, natural language text, either entered by a user or generated by an ASR component, can be used as input to a trained natural language-to-API model. The model's output may be an API request, with a partial or complete set of parameters, to be executed in response to the natural language input. The API request can then be executed directly, or a representation of the request or its parameters may be presented to the user for pre-execution confirmation. Thus, a natural language-to-API model can be used to generate a starting point that eliminates the need for multi-turn dialogues that would normally be used to obtain all the necessary parameters for a given API request.

[0017] Interactive pipelines can continue to be used to process natural language queries after model-based pipelines have been implemented. For example, an orchestrator may assign new natural language processing sessions to either interactive or model-based pipelines based on randomized algorithms (e.g., using pseudorandom number generators), load balancing selection algorithms, test execution logic such as A / B testing logic, etc.

[0018] The performance of a model-based pipeline can be evaluated in isolation or in comparison to an interactive pipeline or another model-based pipeline. For example, metrics related to computing resource efficiency, processing accuracy, latency, user experience, or other aspects of natural language query processing can be recorded for each natural language query (or subset thereof) processed. If the performance of a particular natural language-to-API model falls below a threshold, or if it is significantly below the threshold and underperforming the performance of an interactive pipeline or another natural language-to-API model, that particular natural language-to-API model may be subject to retraining, replacement, or other corrective action.

[0019] In some embodiments, the orchestrator may allocate new natural language processing sessions based on the historical performance of a particular natural language-to-API model or pipeline that processes similar natural language queries, users, etc. For example, the orchestrator may implement a rule-based selection algorithm, or it may use a selection model trained on the previous performance of available pipelines that process similar natural language queries. In this way, processing of natural language queries can be further optimized when a single model-based path does not provide optimal performance.

[0020] Various aspects of this disclosure are described herein with respect to specific examples and embodiments, which are intended to illustrate, not limit, the disclosure. While the examples and embodiments described herein focus on specific machine learning models, API requests, natural language processing pipelines, and algorithms for illustrative purposes, those skilled in the art will understand that the examples are illustrative and not limiting. Furthermore, any feature, process, device, or component of any embodiment described and / or illustrated herein may be used by itself, in conjunction with, or in place of, any other feature, process, device, or component of any other embodiment described and / or illustrated herein.

[0021] Exemplary natural language processing system Figure 1 shows an exemplary natural language processing system 100 configured to process and respond to natural language queries, such as natural language queries, received from various user devices 102 via a network 104. The network 104 may be a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a global area network (GAN), or any combination thereof, and any or all of these may or may not be accessible from and to the Internet.

[0022] The user device 102 can be, or can include, any of a variety of computing devices configured to receive user input, communicate with the natural language processing system 100, and present the output from the natural language processing system 100. For example, the user device 102 can include a mobile phone with program execution and network communication capabilities (e.g., a "smartphone"), a wearable device with program execution and network communication capabilities (e.g., a "smartwatch", "smart eyewear"), a tablet computing device, an e-reader device, a handheld video game device, a media player, a television with program execution and network communication capabilities (e.g., a "smart TV"), a television set-top box, a video game console, a speaker with program execution and network communication capabilities (e.g., a "smart speaker"), a laptop computer, a desktop computing device, a terminal computing device, and the like.

[0023] The natural language processing system 100 can be configured to receive natural language input from various user devices 102 and generate a response or, alternatively, perform an action based on the natural language input. As shown, the natural language processing system 100 can include a plurality of components. In some embodiments, the natural language processing system 100 can include an orchestrator 110 for orchestrating the processing of natural language queries, a dialog pipeline 120 for processing natural language queries according to a chat-based multi-turn dialogue paradigm, a model-based pipeline 130 for processing natural language queries according to a machine learning model paradigm, and one or more applications 140 for executing API requests determined in response to natural language queries.

[0024] The conversational pipeline 120 may include various components for managing chat-based multi-turn conversations. For example, the conversational pipeline 120 includes a query parameter manager 122 that determines which parameters are necessary to execute an API request and facilitates the acquisition of those parameters, a conversational question data store 124 that functions as a repository of questions or other prompts for obtaining API request parameters or otherwise advancing a chat-based multi-turn conversation, an entity recognizer 126 that evaluates natural language input and identifies named entities such as intent and entities related to API request parameters, a conversation manager 128 that dynamically generates responses or prompts (e.g., as an alternative or supplement to the conversational question data store 124), and / or various other chat-based multi-turn conversation management components.

[0025] The model-based pipeline 130 may include various components for managing the generation of API requests using a model from natural language to API trained in response to a natural language query. For example, the model-based pipeline 130 includes an inference system 132 that generates or predicts an API request in response to a natural language query using a natural language to API model 150, a telemetry data store 134 that stores telemetry data regarding previous natural language processing sessions, including telemetry data obtained using the conversational pipeline 120, a natural language to API model 150, other models 150, and an evaluation and training system 136 that evaluates the performance of the conversational pipeline, a user data store 138 that stores and provides data that the inference system 132 and the natural language to API model 150 may use in addition to natural language query input, and / or various other model-based conversation management components.

[0026] Individual components of the natural language processing system 100 may be implemented on one or more computing devices. For example, each component may be implemented on a separate computing device or a set of separate computing devices. As another example, a single computing device or set of computing devices may be shared among multiple components. In some embodiments, the features and services provided by the natural language processing system 100 may be provided by one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more computing resources that are rapidly provisioned and released, such as computing devices, networking devices, and / or storage devices. The hosted computing environment may also be called a cloud computing environment. In some embodiments, each component of the natural language processing system 100, or a subset of components, may be implemented in different computing environments, and any or all of the computing environments may include or not include a cloud computing environment.

[0027] Exemplary training data generation routine Figure 2 shows an exemplary routine 200 that uses an interactive pipeline 120 to generate training data and uses that training data to train a model from natural language to an API. Routine 200 begins at block 202. In some embodiments, routine 200 may be started in response to an event, such as a natural language processing system 100 that starts operation or a natural language processing system 100 that is instructed to generate training data. Once routine 200 has started, executable instructions are loaded into computer-readable memory or otherwise accessed and can be executed by one or more computing devices of the natural language processing system 100, such as the computing system 700 shown in Figure 7.

[0028] In block 204, the orchestrator 110, or any other component of the natural language processing system 100, may initiate a natural language processing session using the user device 102. The natural language processing session may be initiated in response to the receipt of a natural language query from the user device 102, in response to the establishment of a connection between the user device 102 and the natural language processing system 100, or in response to any other event.

[0029] In block 206, the interactive pipeline 120 may manage a natural language processing session using one or more components of the interactive pipeline 120. For example, the query parameter manager 122 may use the entity recognizer 126 to identify information about parameters for an API request, or other information associated with the API request, or other information from which the API request may be otherwise determined, in the initial natural language query and other natural language user input. The query parameter manager 122 may use the interactive query data store 124 and / or the interactive manager 128 to generate prompts to the user to generate an API request to be executed in order to retrieve API request parameters. The orchestrator 110 may then send the API request to the application 140 for execution. An exemplary natural language processing session managed using the interactive pipeline 120 is shown in Figures 3 and 4.

[0030] Figure 3 shows an exemplary natural language processing session 300 processed using an interactive pipeline 120 to generate a set of processing results 320 as parameters for an API request. The user starts the session by typing or speaking an initial query 302 to have the ASR system generate a text version of the initial query 302. In this example, the initial query 302 is "I want to book a family trip to a sunny place around Thanksgiving." The desired response is not directly stated in the initial query 302, but it is clear that the user wants to explore possible travel packages related to the stated conditions ("around Thanksgiving", "family", "sunny place").

[0031] Based on the initial query 302, the interactive pipeline 120 may determine a set of parameters that need to be determined in order to execute the API request. In this example, the interactive pipeline 120 determines a set of parameters 322 associated with searching for a travel package. In some embodiments, as will be described in more detail below with respect to Figure 4, the interactive pipeline 120 may also make initial determinations of one or more parameters based on the initial query. In the example shown in Figure 3, the interactive pipeline does not make any initial determinations of parameters based on the initial query 302. Rather, the interactive pipeline 120 generates an initial response 304 prompting the user for a parameter ("number of nights"). For example, the query parameter manager 122 may determine the parameter set 322 and request the interactive question data store 124 or the interactive manager 128 to provide a prompt for the initial response 304. The user responds with information for two parameters ("November 20th" and "7 nights").

[0032] The entity recognizer 126, or any other component of the interactive pipeline 120, may identify the values ​​of the “start date” and “number of nights” parameters in the user’s response, and the query parameter manager 122 may generate a parameter set 324 based on this. The query parameter manager 122 may then request the interactive question data store 124 or the interaction manager 128 to provide a prompt for another parameter (“How many people are you traveling with?”), and the user responds with the information for the parameter (“Three children, two grandparents, my wife, and myself”).

[0033] The entity recognizer 126, or any other component of the interactive pipeline 120, may identify the value of the “number of travelers” parameter in the user’s response, and the query parameter manager 122 may generate a parameter set 326 based on it. The query parameter manager 122 may then request the interactive question data store 124 or the interaction manager 128 to provide a prompt for another parameter (“Are you flying from Seattle?”). In this case, user data such as profile data or historical interaction data may be used to determine this user’s usual departure location (Seattle) that should be included in the prompt. The user responds with the information for the parameter (“yes”).

[0034] The entity recognizer 126, or any other component of the interactive pipeline 120, may determine that "yes" confirms the value of the "origin" parameter as "Seattle," and the query parameter manager 122 may generate a parameter set 328 based on this. The query parameter manager 122 may then request the interactive question data store 124 or the interaction manager 128 to provide a prompt for another parameter ("Do you prefer the Caribbean or Polynesia?"). In this case, user data, such as profile data or historical interaction data, may be used to determine this user's usual destination (Caribbean or Polynesia), or a potential destination associated with the named entity of the initial query 302 ("To a sunny place") may be selected to be included in the prompt. The user responds with information for the parameter ("Polynesia"). In some embodiments, usage data associated with the current user and / or a larger user set may be evaluated to determine what the phrase "a sunny place" could mean. The results may be used for automatic parameter determination, interactive prompts, or both.

[0035] The entity recognizer 126, or any other component of the interactive pipeline 120, may identify the value of the “destination” parameter, and the query parameter manager 122 may generate a parameter set 330 based on it. The query parameter manager 122 may then determine that the parameter set 330 is a complete parameter set and may return the parameter set to the orchestrator 110 for execution. The orchestrator 110 may then trigger the execution of an API request. For example, the orchestrator 110 may send an API request to an application 140, such as a travel package search application, for execution.

[0036] Figure 4 shows an exemplary natural language processing session 400 processed using the interactive pipeline 120 to generate a set of processing results 420 as parameters for an API request. In contrast to the natural language processing session 300 shown in Figure 3, the natural language processing session 400 shown in Figure 4 may involve the presentation of graphical user interface controls instead of, or in addition to, text chat responses. Alternatively or additionally, the management of the natural language processing session 400 may involve the initial identification of one or more named entities in the initial natural language query and the use of the initially identified named entities as default values ​​in the chat prompts.

[0037] In the illustrated example, the user starts a session by typing the first query 402 or by speaking it to the ASR system to generate a text version of the first query 402. This example uses the same first query 402 as described in the example above ("I would like to book a family trip to a sunny place during Thanksgiving").

[0038] Based on the initial query 402, the interactive pipeline 120 may determine a set of parameters that need to be determined in order to execute the API request. In this example, the interactive pipeline 120 determines a set of parameters 422 associated with searching for travel packages. The interactive pipeline 120 may also make initial determinations on one or more parameters based solely on the initial query 402, or in combination with user data from the user data store 138. For example, the entity recognizer 126, or any other component of the interactive pipeline 120, may identify the value of the “start date” parameter in the user’s response. As another example, the entity recognizer 126, or any other component of the interactive pipeline 120, may identify the term “family” and look up in the user data store 138 the number of family travelers the user has traveled with in the past. As yet another example, the query parameter manager 122 may look up in the user data store 138 the most frequently used departure points for the user, even if the departure point was not mentioned or pointed to in the initial query 302.

[0039] The query parameter manager 122 may add the determined parameters to the initial parameter set 422. Next, the query parameter manager 122 may generate one or more prompts for other parameters. In the illustrated example, the query parameter manager 122 may generate GUI controls 404 for each parameter in the initial parameter set 422, or any combination thereof. Parameters whose values ​​are predetermined may have values ​​reflected in the presentation.

[0040] The interactive pipeline 120 may add or update parameters based on user interaction with the presented GUI controls. For example, the value of the "start date" parameter is updated in parameter set 424 based on the changes selected by the user. The value of "destination" is added to parameter set 426, and the value of "number of nights" is added to the final parameter set 428.

[0041] The query parameter manager 122 may determine that parameter set 428 is a complete parameter set and return the parameter set to the orchestrator 110 for execution. The orchestrator 110 may then trigger the execution of an API request. For example, the orchestrator 110 may send an API request to an application 140, such as a travel package search application, for execution.

[0042] The exemplary natural language processing sessions shown in Figures 3 and 4 are illustrative only and are not intended to be restrictive, mandatory, or exhaustive. In some embodiments, different interfaces, dialogue flows, and query parameter determination processes may be used. For example, a portion of the results may be presented during a session (such as session 300) based on a set of query parameters determined at different points in the session. As the session continues through subsequent dialogue turns and additional parameters are determined, the results may be filtered or updated accordingly.

[0043] Returning to Figure 2, in block 208, the orchestrator 110 may store data about the natural language processing session. This data about the natural language processing session may be called telemetry data. As will be explained in more detail below, the telemetry data from this session and any number of other sessions (e.g., thousands, hundreds of thousands, millions, or more) will later be used to train the Natural Language to API model 150.

[0044] For example, during or after the completion of the natural language processing session 300 shown in Figure 3, the orchestrator 110 may store telemetry data about the session in the telemetry data store 134. The orchestrator 110 may store telemetry data such as a unique session identifier, the initial query 302, and the final parameter set 330 or the complete API call made to the application 140. As another example, during or after the completion of the natural language processing session 400 shown in Figure 4, the orchestrator 110 may store telemetry data about the session in the telemetry data store 134. The orchestrator 110 may store telemetry data such as a unique session identifier, the initial query 402, and the final parameter set 428 or the complete API call made to the application 140. In some embodiments, the telemetry data may include additional data such as one or more natural language queries or responses made by the user after the initial query 302 or 402, one or more parameter sets determined by the interactive pipeline 120 before the final parameter set 330 or 428, and contextual data about the natural language processing session (e.g., time, date, user identifier, etc.).

[0045] In decision block 210, the orchestrator, or any other component of the natural language processing system 100, may determine whether the conditions for generating training data are met. In some embodiments, the conditions may be a threshold number of performance for the natural language processing session, or a threshold number of storage for the first natural language query and the corresponding API requests generated from the first natural language query. If the training data conditions are met, routine 200 may proceed to block 212. Otherwise, routine 200 may return to block 204 for the management of additional natural language processing sessions. In some embodiments, the portion of routine 200 starting from block 212 may be performed in parallel or asynchronously with respect to additional iterations of the portion of routine 200 from block 204 to block 208.

[0046] In block 212, the evaluation and training system 136, or any other component of the natural language processing system 100, may generate training data for training the natural language to API model 150. The training data may be generated from telemetry data associated with a natural language processing session managed using the interactive pipeline 120.

[0047] In some embodiments, the corpus of training data pairs may be generated from telemetry data. A training data pair may include a training data input item and a corresponding reference data output item, which is the desired output of the model for the training data input item. The training data input item may be a string containing the first natural language query of the session, or a representation thereof suitable for input to the model. For example, the training data input item may be a vector containing the first query string converted to numeric values ​​for each character. As another example, the training data input item may be a vector containing tokenized versions of the first query, such as different numeric token representations of each word or phrase in the first query. As yet another example, the training data input item may be a vector containing word embeddings for each word of the input query (e.g., BERT word embeddings), or sentence embeddings for the entire first query (e.g., BERT sentence embeddings).

[0048] In some embodiments, additional input data, such as query context data, may be included in the training data input field. For example, a context vector may be generated using data about a session (e.g., time, date, user identifier, additional interaction turns, etc.). As another example, a context vector may include data about the user, such as data from the user data store 138 (e.g., profile data, demographic data, historical interaction data, etc.). The context vector may be appended to the vector representing the initial query to generate an extended input vector used as a training data input field.

[0049] The reference data output field can be a string containing an API request, which is the desired output from the training data input field. For example, the reference data output field could be a complete API request with parameters. As another example, the reference data output field could be a set of values ​​or tokens from which a string representation of the desired API request can be derived.

[0050] In block 214, the evaluation and training system 136 may train a natural language to API model 150 using a corpus of training data generated from telemetry data. In some embodiments, the natural language to API model 150 may be a transformation model implemented using an artificial neural network or a variation thereof, such as an RNN, CNN, transformer, encoder-decoder, ensemble of multiple models, or other models suitable for transformation. The exemplary machine learning model types and architectures described herein are illustrative and are not intended to be limiting, mandatory, or exhaustive.

[0051] Generally speaking, neural networks (NNs), including DNNs, CNNs, RNNs, and other NN-based models such as transformer-encoder-decoder models, have multiple layers of nodes, also called neurons. Exemplary, an NN may include an input layer, an output layer, and any number of hidden, inner, or "hidden" layers between the input and output layers. Each layer may contain any number of distinct nodes. Nodes in adjacent layers may be logically connected to one another, and each logical connection between different nodes in adjacent layers may be associated with their respective weights. Conceptually, a node can be thought of as a computational unit that computes output values ​​as a function of multiple different input values. Nodes can be considered connected when they include the output of a function associated with a node in a previous layer, where the input values ​​to the function associated with the current node are multiplied by the weights associated with the individual connections between the current node and the nodes in the previous layer. When processing input data in the form of input vectors or matrices of input vectors (e.g., batches of input vectors for training data) using an NN, each NN may perform "forward propagation" to generate output vectors or matrices of output vectors. Each input vector may contain n distinct data elements or dimensions corresponding to the n nodes in the NN input layer (where n is some positive integer). Each data element can be a floating-point or integer value. Forward propagation typically involves multiplying the matrix of input vectors by a matrix representing the weights associated with the connections between the nodes of the input layer and the nodes of the next layer, and then applying an activation function to the results. This process is then repeated for each subsequent NN layer. Some NNs have hundreds of thousands or millions of nodes and millions of weights for the connections between all neighboring layer nodes.

[0052] Each connection between individual nodes in adjacent layers is associated with trainable parameters, such as weights and / or bias terms, which are applied to the values ​​passed from the nodes of the previous layer to the activation function of the nodes of the subsequent layer. For example, the weights associated with the connection from the input layer to the inner layer to which the input layer is connected may be located in a weight matrix W of size m × n, where m represents the number of nodes in the inner layer and n represents the dimension of the input layer. Each row of the weight matrix W may correspond to an individual node in the input layer, and each column of the weight matrix W may correspond to an individual node in the inner layer. The weights w associated with the connection from any node in the input layer to any node in the inner layer may be located at the corresponding intersections in the weight matrix W.

[0053] Exemplary, training data input items, constructed as input vectors (or extended input vectors as described above) (or converted to input vectors), may be provided to a computer processor that stores or otherwise has access to a weight matrix W. The processor then multiplies the input vectors by the weight matrix W to produce intermediate vectors. The processor may adjust the individual values ​​of the intermediate vectors using offsets or biases associated with the inner layers (e.g., by adding or subtracting values ​​other than the weights applied). Furthermore, the processor may apply activation functions to the individual values ​​of the intermediate vectors (e.g., by using the individual values ​​as inputs to a sigmoid function or a normalized linear unit (ReLU) function). In some embodiments, there may be multiple inner layers, each inner layer may or may not have the same number of nodes as each other inner layer. Weights associated with the connection from one inner layer to the next may be placed in a weight matrix similar to the weight matrix W. The process of multiplying the intermediate vectors by the weight matrix and applying activation functions to the individual values ​​of the resulting intermediate vectors may be performed for each inner layer following the first inner layer. The output layer of the neural network (NN) makes output decisions based solely on information from the last inner layer, or by combining it with information from previous inner layers.

[0054] The reference data output items, which are constructed as output vectors (or converted to output vectors), may contain the "correct" output that Model 150 should produce for the corresponding training data input items, or otherwise the desired output, as described above. The goal of training may be to minimize the difference between the output vector (generated by the model from the training data input items) and the corresponding reference data output vector. The evaluation of the training data output vector relative to the reference data output vector may be done using a loss function (also called an objective function), such as a binary cross-entropy loss function, a weighted cross-entropy loss function, a squared error loss function, a softmax loss function, some other loss function, or a composite function of loss functions. The gradient of the loss function with respect to the model parameters (e.g., weights) may be calculated. The gradient can be used to determine the direction in which the individual parameters of Model 150 should be adjusted to minimize the loss function, and thus minimize the extent to which future outputs (e.g., the training data output vector) differ from the predicted or desired output (the reference data output vector). The degree to which individual parameters are adjusted can be predetermined or dynamically determined (for example, based on gradients and / or hyperparameters). For example, hyperparameters such as the learning rate can be specified or used to determine the magnitude of the adjustments applied to the individual parameters of model 150.

[0055] In some embodiments, a model training system can compute gradients on a subset of the training data rather than on the entire set of training data. Therefore, the gradients may be called "partial gradients" because they are not based on the entire corpus of training data. Instead, they are based on the difference between the training data output vector and the reference data output vector when processing only a specific subset of the training data.

[0056] In some embodiments, the model training system can update some or all of the parameters of Model 150 using gradient descent in backpropagation. In backpropagation, the training error is determined using a loss function (e.g., as described above). The training error can be used to update the individual parameters of the model in order to reduce the training error. For example, the gradient of the loss function can be calculated to determine how the weights of the weight matrix should be adjusted to reduce the error. The adjustments can be backpropagated layer by layer through Model 150.

[0057] The evaluation and training system 136 can manage the training process by evaluating one or more stopping conditions. For example, the stopping condition may be based on the accuracy of the model 150, which is determined using a loss function, a test set, or both. If the accuracy meets a threshold or other condition, the model 150 may be considered to have "converged" to a desired or satisfactory result, and the training process may be stopped. As another example, the stopping condition may be based on the number of training iterations performed (e.g., "epochs"), the elapsed training time, etc.

[0058] The exemplary training processes described herein are illustrative and not intended to be limiting, mandatory, or exhaustive. In some embodiments, the training process may include additional steps, fewer steps, and / or alternative steps, depending on the type of model being trained, the application to which the model is being trained, and so on.

[0059] In block 216, the trained natural language to API model 150 can be deployed for use in processing natural language queries. For example, the evaluation and training system 136 may send the model (e.g., a file of model parameters, executable code to implement the model algorithm, etc.) to the inference system 132 for use.

[0060] In block 218, the orchestrator may distribute subsequent natural language processing sessions between the interactive pipeline 120 and the model-based pipeline 130 for processing. An exemplary routine for distributing sessions in this manner is shown in Figure 5 and will be described in more detail below.

[0061] In some embodiments, multiple natural language-to-API models 150 may be trained. For example, different structures may be used, such as one RNN-based model and one encoder-decoder model. Another example is the use of different sets of training data, different training algorithms, different training hyperparameters, or different combinations thereof. Multiple models may be used to effectively implement a multi-model-based pipeline 130.

[0062] Exemplary multi-pipeline session management routine Figure 5 shows an exemplary routine 500 for distributing natural language processing sessions across multiple natural language processing pipelines, including a model-based pipeline 130 that implements the natural language to API model 150. Routine 500 begins at block 502. In some embodiments, routine 500 may be started in response to an event, such as a natural language processing system 100 that starts operation, or a natural language processing system 100 after the deployment of the trained natural language to API model 150. Once routine 500 is started, executable instructions are loaded into computer-readable memory or otherwise accessed and can be executed by one or more computing devices of the natural language processing system 100, such as the computing system 700 shown in Figure 7.

[0063] In block 504, the orchestrator 110, or any other component of the natural language processing system 100, may receive a natural language query from the user device 102 and initiate a natural language processing session.

[0064] In decision block 506, the orchestrator 110 may decide which of the multiple available processing pipelines should be assigned to manage the natural language processing session. Figure 5 illustrates the orchestrator 110's decision between the interactive pipeline 120 and the model-based pipeline 130. This figure is provided for illustrative purposes only and is not intended to be limiting, mandatory, or exhaustive. In some embodiments, the orchestrator 110 may decide between multiple model-based pipelines 130 (e.g., different pipelines each having a different natural language to API model 150, or a single pipeline configured to use any of several different natural language to API models 150). In some embodiments, the orchestrator 110 may decide between multiple interactive pipelines 120 (e.g., one interactive pipeline 120 configured to use the interactive question data store 124, and another interactive pipeline 120 configured to use the generative AI-based dialogue manager 128) instead of, or in addition to, one or more model-based pipelines 130.

[0065] In one example, the orchestrator 110 may select pipelines based on a randomized algorithm (e.g., using a pseudo-random number generator). In another example, the orchestrator 110 may select pipelines based on a load balancing algorithm (e.g., based on which pipeline has the fewest open connections, based on a metric of available computing resources, based on a round-robin algorithm, etc.). In yet another example, the orchestrator 110 may select pipelines using a selection model that considers natural language input and / or information associated with it (e.g., user, time, initial classification of natural language query subject, etc.) to select the pipeline that is most likely to provide satisfactory performance (e.g., measured in terms of user satisfaction, resource usage, etc.). The exemplary pipeline selection algorithms described herein are illustrative only and are not intended to be limiting, mandatory, or exhaustive. Additional or alternative selection algorithms may be used in some embodiments.

[0066] In block 508, when the orchestrator 110 assigns a natural language processing session to the interactive pipeline 120, the interactive pipeline 120 conducts the session and determines the parameters for the API request to be generated. In block 510, the interactive pipeline 120 uses the determined parameters to generate the API request to be executed. An exemplary natural language processing session managed using the interactive pipeline 120 is shown in Figures 3 and 4 and described in more detail above.

[0067] In block 512, when the orchestrator 110 assigns a natural language processing session to the model-based pipeline 130, the model-based pipeline 130 may begin by retrieving context data associated with the natural language query and / or user. For example, the context data may include data about the session (e.g., time, date, user identifier, etc.). Alternatively, the context data may include data about the user or user device 102 that received the natural language query, such as data from the user data store 138 (e.g., profile data, demographic data, historical interaction data, etc.). In some embodiments, no context data is used, and block 512 is not executed.

[0068] In some embodiments, one or more interaction turns may occur to obtain parameters for a generated API request, or to obtain data that can determine the parameters for an API request. For example, an interaction management and / or interaction question data store may be used to generate prompts for additional information.

[0069] In block 514, the inference system 132, or any other component of the model-based pipeline 130, may use the natural language to API model 150 to process natural language queries and (optionally) contextual data to generate API requests to be executed.

[0070] Figure 6 shows an exemplary natural language processing session 600 processed using a model-based pipeline 130 to predict, or otherwise generate, an API request 624 containing a set of parameters. In contrast to sessions managed using an interactive pipeline 120 (e.g., natural language processing sessions 300 and 400 shown in Figures 3 and 4), the natural language processing session 600 does not necessarily have to proceed with a multi-turn dialogue and may involve the presentation of search results 604 (or other results of the executed API request) after the initial query 602.

[0071] In the illustrated example, the user initiates a session by typing the first query 602 or by speaking it to the ASR system to generate a text version of the first query 602. This example uses the same first query 602 as described in the example above ("I would like to book a family trip to a sunny place during Thanksgiving").

[0072] Based on the initial query 602, the inference system 132, or any other component of the model-based pipeline 130, can generate input for analysis by the Natural Language to API model 150. In this example, the inference system 132 generates a natural language query representation 620, such as a vector. The natural language query representation 620 may contain the text of the initial query 602 in numerical form. For example, the natural language query representation 620 may contain the initial query string converted to a numeric value for each character. As another example, the natural language query representation 620 may contain a tokenized version of the initial query 602, such as a different numeric token-based representation of each word or phrase in the initial query 602. As yet another example, the natural language query representation 620 may contain word embeddings for each word (e.g., BERT word embeddings) or sentence embeddings of the initial query 402 (e.g., BERT sentence embeddings).

[0073] In some embodiments, additional data, such as query context data, may be included as input to the model 150 from natural language to API. For example, the inference system 132 may use data about the initial query 602 or session 600 (e.g., time, date, user identifier, etc.) to generate a context data representation 622, such as a vector. As another example, the inference system 132 may include data about the user, such as data from the user data store 138 (e.g., profile data, demographic data, historical interaction data, etc.).

[0074] A natural language query expression 620 (extended by adding a context data expression 622, if used) is processed by a natural language to API model 150, which may generate an API request 624. In some embodiments, as illustrated, the API request 624 may be generated as a complete request with parameters. In some embodiments, the output of the natural language to API model 150 may be a set of values ​​or tokens. For example, the natural language to API model 150 may generate an output representing the API request to be made (e.g., a call to the "search.foo" uniform resource locator in this example) and a set of parameter values ​​for the parameters associated with the API request (e.g., in this example, "SEA", "French Polynesia", "11 / 25", "7", and "7"). The inference system 132 may assemble the final API request 624 from the model output, or the model output may be provided to an orchestrator 110 or some other component for assembly into the final API request 624 to be executed. The orchestrator 110 may then trigger the execution of the API request 624. For example, the orchestrator 110 may send an API request 624 to an application 140, such as a travel package search application, for execution.

[0075] Returning to routine 500, in block 516, the orchestrator 110 may execute an API request or send an API request to application 140 for execution and continue managing the session. In some embodiments, session management may involve additional interaction turns and additional natural language queries that may be processed using one of the natural language processing pipelines. The natural language processing pipeline may be the same pipeline that processed the session's first natural language query, or routine 500 may return to block 504 for each new natural language query.

[0076] In some embodiments, the execution of an API request may return one or more errors instead of, or in addition to, a response. If a particular error is received, one or more interaction turns may be made to obtain parameters for the API request to be generated, or to obtain data that can determine the parameters and / or the API request. For example, an interaction management and / or interaction question data store may be used to generate prompts for additional information.

[0077] In block 518, the orchestrator 110, or any other component of the natural language processing system 100, may store session data relating to the session being managed in the current iteration of routine 500. For example, during or after a natural language processing session, the orchestrator 110 may store telemetry data such as a unique session identifier, the first query, the API request(s) generated during the session, performance data (e.g., elapsed time to generate the API request(s), the number of interaction turns before generating the API request(s), the computing resources used to generate the API request(s)), and user experience data (indicators of whether the session was interrupted, indicators of user engagement during or as a result of the session).

[0078] In block 520, the evaluation and training system 136 may evaluate the performance of a model (and / or one or more pipelines). The performance of a given model or pipeline may be evaluated over a set of sessions, such as tens, hundreds, thousands, or more. In some embodiments, the evaluation and training system 136 may determine one or more metrics from telemetry data stored for the sessions used in the evaluation. For example, the evaluation and training system 136 may determine statistics such as mean, median, and standard deviation with respect to one or more of the following metrics: elapsed time to generate an API request(s), the number of interaction turns before generating an API request(s), the computing resources used to generate an API request(s), an engagement metric representing user engagement during or as a result of a session, or any other metric(s).

[0079] In decision block 522, the evaluation and training system 136 may determine whether the model retraining condition is met. In some embodiments, the model retraining condition may relate to thresholds of one or more metrics determined in block 520. For example, if a metric falls below a threshold (e.g., user engagement) or exceeds a threshold (e.g., elapsed time, number of interaction turns), the model training condition may be met and routine 500 may proceed to block 524. As another example, if a metric is below a threshold by a certain amount and is inferior to a similar metric in another pipeline (for example, the engagement of sessions managed using model-based pipeline 130 is below a threshold by a certain amount and is inferior to the engagement of sessions managed using another model-based pipeline 130 or interactive pipeline 120), or if a metric is above a threshold by a certain amount and is superior to a similar metric in another pipeline (for example, the elapsed time of sessions managed using model-based pipeline 130 exceeds the elapsed time of sessions managed using another model-based pipeline 130 or interactive pipeline 120), the model training condition may be met and routine 500 may proceed to block 524.

[0080] In some embodiments, model retraining conditions do not necessarily have to be related to performance metrics. Rather, model retraining conditions can be time-based or data volume-based. For example, model retraining may be triggered regularly, such as daily, weekly, monthly, or quarterly. As another example, model retraining may be triggered when a threshold amount of telemetry data is stored, from which new training data can be generated.

[0081] In block 524, the evaluation and training system 136 may retrain the natural language to API model 150. In some embodiments, the evaluation and training system 136 may re-execute the training process in which the natural language to API model 150 was first trained, as described in more detail above. For example, the evaluation and training system 136 may generate a corpus of training data using telemetry data from a more recent natural language processing session and retrain the model 150. After retraining the natural language to API model 150, it may be redeployed and used in subsequent iterations of routine 500.

[0082] Execution environment Figure 7 shows various components of an exemplary computing system 700 configured to perform the various functions described herein. The computing system 700 may be one or more physical host computing devices, or may include them.

[0083] In some embodiments, as shown in the figures, the computing system 700 may include one or more computer processors 702 such as a physical central processing unit (CPU), one or more network interfaces 704 such as a network interface card (NIC), one or more computer-readable media drives 706 such as a high-density disk (HDD), a solid-state drive (SSD), a flash drive, and / or other persistent non-temporary computer-readable media, and one or more computer-readable memories 710 such as random-access memory (RAM) and / or other volatile non-temporary computer-readable media.

[0084] Computer-readable memory 710 may contain computer program instructions executed by one or more computer processors 702 to implement one or more embodiments, and / or data used by one or more computer processors 702. For example, computer-readable memory 710 may store an operating system 712 to provide general management of the model training system 700. As another example, computer-readable memory 710 may store interactive pipeline instructions 714 for implementing the systems, components, and features of the interactive pipeline 120. As yet another example, computer-readable memory 710 may store model-based pipeline instructions 716 for implementing the systems, components, and features of the model-based pipeline 130. As yet another example, computer-readable memory 710 may store orchestrator instructions 718 for implementing the features of the orchestrator 110. As yet another example, computer-readable memory 710 may store application instructions 720 for implementing the features of an application 140 that executes API requests generated by the interactive pipeline 120 or the model-based pipeline 130. As another example, computer-readable memory 710 can store evaluation and training instructions 722 for implementing features of the evaluation and training system 136.

[0085] term Depending on the embodiment, any particular action, event, or function of any of the processes or algorithms described herein may be performed in a different order, added, combined, or omitted entirely (for example, not all described actions or events are essential for the implementation of this algorithm). Furthermore, in certain embodiments, the actions or events may be performed simultaneously rather than sequentially, for example, through multithreading, interrupt handling, or via multiple processors or processor cores, or in other parallel architectures.

[0086] The various exemplary logic blocks, modules, routines, and algorithmic steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware or as a combination of electronic hardware and computer software. To clearly illustrate such interoperability, the various exemplary components, blocks, modules, and steps are generally described above in terms of their functionality. Whether such functionality is realized as hardware or implemented as software running on hardware depends on the specific application and the design constraints imposed on the overall system. The described functionality can be implemented in a way that varies with respect to each specific application, but such a decision on implementation form should not be construed as resulting in a departure from the scope of this disclosure.

[0087] Furthermore, various exemplary logic blocks and modules described in relation to the embodiments disclosed herein can be implemented or performed by machines such as, for example, processor devices, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. A processor device may be a microprocessor; alternatively, a processor device may be a controller, microcontroller, or state machine, or a combination thereof. A processor device may include electrical circuits configured to process computer executable instructions. In another embodiment, a processor device includes an FPGA or other programmable device that performs logical operations without processing computer executable instructions. A processor device can also be implemented as a combination of computing devices, for example, a DSP and a combination of a microprocessor, multiple microprocessors, one or more microprocessors working in conjunction with a DSP core, or any other such configuration. Although this specification primarily describes digital technologies, processor devices may also primarily include analog components. For example, some or all of the algorithms described herein may be implemented in analog circuits or in mixed analog and digital circuits. The computing environment may include, but is not limited to, any type of computer system, including, a few examples, microprocessors, mainframe computers, digital signal processors, portable computing devices, device controllers, and computer systems based on the computing engine inside a device.

[0088] Elements of methods, processes, routines, or algorithms described in connection with embodiments disclosed herein can be directly embodied in hardware, in software modules executed by a processor device, or in a combination of both. Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of non-temporary computer-readable storage medium. An exemplary storage medium may be coupled to a processor device so that the processor device can read information from and write data to the storage medium. Alternatively, the storage medium may be integrated with the processor device. The processor device and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor device and storage medium may reside as discrete components in a user terminal.

[0089] In particular, conditional language used herein, such as “can,” “could,” “might,” “may,” and “e.g.,” is generally intended to convey that certain embodiments include certain features, elements, and / or steps, and other embodiments do not, unless otherwise explicitly stated or understood within the context in which they are used. Therefore, such conditional language is generally not intended to imply that features, elements, and / or steps are required in any way with respect to one or more embodiments, or that one or more embodiments necessarily include logic for determining, with or without other inputs or prompts, whether these features, elements, and / or steps are included in any particular embodiment or should be performed in any particular embodiment. Terms such as “comprising,” “including,” and “having” are synonymous and are used inclusively in an open-ended manner, without excluding additional elements, features, actions, behaviors, etc. Furthermore, the term "or" is used in an inclusive sense (not an exclusive sense), and when used to connect a list of elements, for example, the term "or" means one of the elements in the list, some of them, or all of them.

[0090] Disjunctive language, such as the phrase "at least one of X, Y, and Z," is generally understood in context to indicate that an item, term, etc., may be X, Y, Z, or any combination thereof (e.g., X, Y, and / or Z), unless it is specifically stated otherwise. Therefore, such disjunctive language is generally not intended, and should not, imply that a particular embodiment requires the presence of at least one X, at least one Y, or at least one Z, respectively.

[0091] Unless otherwise specified, articles such as "a" or "an" should generally be interpreted as including one or more listed items. Therefore, phrases such as "devices configured to..." are intended to include one or more enumerated devices. Such one or more enumerated devices may also be collectively configured to perform the listed enumerations. For example, "processors configured to perform enumerations A, B, and C" may include a first processor configured to perform enumeration A, which works in conjunction with a second processor configured to perform enumerations B and C.

[0092] While the above detailed description illustrates, explains, and points out novel features applicable to various embodiments, it can be understood that various omissions, substitutions, and modifications to the form and details of the exemplary devices or algorithms may be made without departing from the spirit of this disclosure. As can be recognized, certain embodiments described herein may be embodied in a form that does not provide all of the features and benefits described herein, since some features may be used or implemented separately from others. The scope of the specific embodiments disclosed herein is indicated not by the above description but by the appended claims. All modifications that fall within the meaning and equivalents of the claims shall be encompassed within those scopes.

Claims

1. Computer-readable memory that stores executable instructions, The aforementioned executable instructions ensure that at least, It manages multiple natural language processing sessions, and one of the multiple natural language processing sessions is Receiving natural language queries, Generating one or more natural language prompts for parameter data associated with an application programming interface (API) request to be executed, Receiving one or more corresponding natural language responses to one or more natural language prompts, This includes the execution of the aforementioned API request, A corpus of training data is generated based on the aforementioned multiple natural language processing sessions, the corpus of training data includes a plurality of training data input vectors and a plurality of reference data output vectors, one of the plurality of reference data output vectors representing the API request as a desired output generated from one of the plurality of training data input vectors, which represents the natural language query. A system comprising one or more programmed processors to train a natural language to API model using the aforementioned training data corpus, the natural language to API model being trained to generate API requests using natural language query inputs.

2. The system according to claim 1, wherein the API request includes a search query relating to a travel reservation.

3. The system according to claim 1, wherein a particular API request generated by the natural language-to-API model includes a string representing a plurality of parameters, and the natural language query input used by the natural language-to-API model to generate the particular API request represents fewer than all of the plurality of parameters.

4. The system according to claim 1, wherein the training data input vector represents both the natural language query and contextual data relating to the user account associated with the natural language query.

5. The system according to claim 4, wherein the natural language to API model is trained to generate a specific API request using both a natural language query input and context data input associated with the natural language query input.

6. The system according to claim 1, wherein the one or more processors are further programmed by the executable instructions to transmit the natural language to API model to the inference system.

7. The one or more processors, by means of the executable instructions, Using the aforementioned natural language-to-API model, manage a second set of natural language processing sessions. Based on the management of the second set of natural language processing sessions, the performance of the natural language-to-API model is evaluated. The system according to claim 1, further programmed to decide to retrain the natural language to API model based on an evaluation of the performance of the natural language to API model.

8. The one or more processors, by means of the executable instructions, For a first subset of natural language processing sessions of a second set of multiple natural language processing sessions, it is decided to manage each natural language processing session of the first subset using the natural language to API model. The system according to claim 1, further programmed to decide to manage each of the natural language processing sessions in a second subset of a second set of natural language processing sessions using an interactive query parameter manager.

9. The system according to claim 8, wherein the one or more processors are further programmed by the executable instructions to determine, using a selection model, whether one of the second plurality of natural language processing sessions should be managed using the natural language to API model or the interactive query parameter manager.

10. The system according to claim 8, wherein one or more processors are further programmed by the executable instructions to determine, for each of the natural language processing sessions of a third subset of the second plurality of natural language processing sessions, to use a second natural language to API model different from the natural language to API model.

11. The one or more processors, by means of the executable instructions, Based on the management of the first subset of natural language processing sessions, the performance of the natural language-to-API model is evaluated. Based on the management of the third subset of natural language processing sessions, the performance of the second natural language to API model is evaluated. The system according to claim 10, further programmed to decide to retrain the second natural language to API model based on the performance of the natural language to API model, where the performance of the second natural language to API model exceeds a threshold amount.

12. The system according to claim 1, wherein the natural language to API model includes one of a transformer-based artificial neural network, a recurrent neural network, a convolutional neural network, or an encoder-decoder machine learning model.

13. Under the control of a computing system including one or more processors configured to execute specific instructions, Obtaining a corpus of training data based on multiple natural language processing sessions, wherein the corpus of training data includes multiple training data input vectors and multiple reference data output vectors, and one of the reference data output vectors represents a natural language query relating to travel bookings, and the desired output generated from one of the multiple training data input vectors represents travel-based search results. A computer implementation method comprising training a natural language to API model using the aforementioned training data corpus, wherein the natural language to API model is trained to generate predicted travel-based search results using natural language query inputs.

14. The computer implementation method according to claim 13, wherein training the natural language to API model includes generating a specific API request containing a string representing a plurality of parameters, and the natural language query input used by the natural language to API model to generate the specific API request represents fewer than all of the plurality of parameters.

15. The computer implementation method according to claim 13, wherein the training data input vector represents both the natural language query and context data relating to a user account associated with the natural language query, and training the natural language to API model includes training the natural language to API model to generate a specific API request using both the natural language query input and the context data input associated with the natural language query input.

16. The computer implementation method according to claim 13, further comprising transmitting the model from natural language to API to an inference system.

17. Under the control of a computing system including one or more processors configured to execute specific instructions, Receiving natural language queries and, To generate a model input to the API from natural language representing at least a portion of the aforementioned natural language query, The process involves generating a model output using a natural language-to-API model and a natural language-to-API model input, wherein the model output represents an API request executed in response to the natural language query. A computer implementation method, which includes executing the aforementioned API request.

18. To decide to manage each of the natural language processing sessions in a first subset of multiple natural language processing sessions using the natural language-to-API model, A computer implementation method according to claim 17, further comprising deciding to manage each of the natural language processing sessions in a second subset of multiple natural language processing sessions using an interactive query parameter manager.

19. The computer implementation method according to claim 18, further comprising using a selection model to determine whether one of the plurality of natural language processing sessions should be managed using the natural language to API model or the interactive query parameter manager.

20. To manage each of the natural language processing sessions in a third subset of the aforementioned multiple natural language processing sessions using a second natural language to API model different from the aforementioned natural language to API model, Evaluating the performance of the natural language-to-API model based on the management of the first subset of natural language processing sessions, Evaluating the performance of the second natural language to API model based on the management of the third subset of natural language processing sessions, The computer implementation method according to claim 18, further comprising deciding to retrain the second natural language to API model based on the performance of the natural language to API model, such that the performance of the second natural language to API model exceeds a threshold amount.

Images