Natural Language for Application Programming Interface Translation

Description

1 / 41 “NATURAL LANGUAGE FOR APPLICATION PROGRAMMING INTERFACE TRANSLATION” CROSS-REFERENCE ON RELATED REQUEST

[001] This application claims priority to U.S. Patent Application 18 / 497,304, filed October 30, 2023, and to U.S. Provisional Patent Application 63 / 494,597, filed April 6, 2023, the contents of each of which are incorporated by reference herein and form part of this descriptive report. FUNDAMENTALS

[002] Natural language processing systems include several components to receive input from a user and process the input to determine what the user means. In some implementations, a natural language processing system receives textual input, such as text entered by a user or a transcript of a user's utterance. The natural language processing system can determine the meaning of the textual input in a way that can be triggered by a computer application. BRIEF DESCRIPTION OF THE DRAWINGS

[003] Embodiments of various inventive features will now be described with reference to the following drawings. Throughout the drawings, reference numbers may be reused to indicate the correspondence between referenced elements. The drawings are provided to illustrate exemplary embodiments described in this document and are not intended to limit the scope of disclosure.

[004] FIG. 1 is an illustrative diagram of a natural language processing system with various components for processing natural language queries from user devices according to some modalities.

[005] FIG. 2 is a flow diagram of an illustrative routine for generating training data and training a machine learning model according to Petition 870250107768, dated 11 / 25 / 2025, page 6 / 53 2 / 41 some modalities.

[006] FIG. 3 is a diagram of a natural language processing session and parameter development for an application programming interface request based on the same according to some modalities.

[007] FIG. 4 is a diagram of another natural language processing session and parameter development for an application programming interface request based on the same according to some modalities.

[008] FIG. 5 is a flow diagram of an illustrative routine for orchestrating natural language processing sessions using a dialog-based query parameter manager or a machine learning model according to some modalities.

[009] FIG. 6 is a diagram of a natural language processing session with generation of an application programming interface request using a machine learning model according to some modalities.

[010] FIG. 7 is a block diagram of an illustrative computing system configured to implement aspects of the present disclosure according to some embodiments. DETAILED DESCRIPTION

[011] This disclosure is directed to the use of a natural language query processing pipeline to generate training data for training a machine learning model. The machine learning model can be trained using the training data to translate natural language queries into executable requests by computer. Furthermore, this disclosure is directed to orchestrating natural language query processing across different pipelines to evaluate and improve the performance of the machine learning model trained to translate natural language queries. Petition 870250107768, dated 11 / 25 / 2025, page 7 / 53 3 / 41 natural in computer-executable requests.

[012] Some conventional computing systems offer natural language user interfaces that allow users to access system functionality and interact with systems using spoken or typed inputs that approximate conversational language. These inputs may be referred to as natural language queries or natural language requests (whether expressed as commands or not in the form of questions). Some examples of systems that provide such interfaces include travel booking systems, search engines, customer service systems, and the like. Systems may use multi-turn interactive dialog interfaces to obtain information needed to generate and execute application programming interface (API) requests in response to users' natural language queries.For example, some systems use natural language processing (NLP) to determine the intent behind a user's query and to determine a corresponding executable request that the system should execute to provide information or take action responsive to the user's intent. Some systems may use a database of questions or requests to obtain parameters or other information to execute the request, while other systems may use natural language generation (NLG) processing (e.g., generative artificial intelligence) to generate natural language requests and manage a dialog box. Each system prompt and corresponding user response can be referred to as a single turn of a multi-turn dialog box. These systems typically take multiple turns to obtain the parameters needed to respond to requests.In some cases, systems may fail to obtain certain parameters even after engaging in multiple rounds of dialogue. As a result, system performance—defined in terms of computing resource efficiency, processing accuracy, etc.—is affected. Petition 870250107768, dated 11 / 25 / 2025, page 8 / 53 4 / 41 Latency or user experience - may be unsatisfactory.

[013] Some aspects of this disclosure address some or all of the problems noted above, among others, through the use of a machine learning model that translates natural language queries into API requests. Advantageously, this model can be trained to generate API requests from natural language queries with fewer dialog turns after the initial natural language queries from users, or directly on API requests without additional dialog turns. To obtain training data to train such a model, a multi-turn dialog pipeline can be used to manage natural language processing sessions, process natural language queries, obtain parameters, and generate API requests in response to natural language queries.Initial natural language queries, resulting API requests, and optionally other information about natural language processing sessions can be stored and used to train a model to generate API requests from initial natural language queries.

[014] Additional aspects of this disclosure relate to the distribution of natural language queries across different processing pipelines, evaluating the performance of each pipeline, and adjusting aspects of individual pipelines based on their performance. In some embodiments, one pipeline may use a machine learning model that has been trained to generate API requests in response to natural language queries with few (or no) additional dialogue turns, and another pipeline may be or include the multi-turn chat-based dialogue pipeline that was used to obtain data to train the machine learning model. Advantageously, when the performance of one of the pipelines, such as the machine learning model pipeline, falls below a performance threshold, the machine learning model can be retrained or the pipeline can be otherwise modified to improve its performance. Petition 870250107768, dated 11 / 25 / 2025, page 9 / 53 5 / 41 performance. In some modalities, there may be multiple machine learning model pipelines (e.g., using different model types, models trained using different data, etc.), multiple chat-based dialogue pipelines, or any combination of each.

[015] An orchestration system, also referred to here as an orchestrator, can determine which pipelines, from the various available request processing pipelines, should process and respond to each of the received natural language queries. The orchestrator or components of the individual pipelines can store data about natural language processing, such as the natural language query input, the number of dialog turns required to generate an API request, the user actions taken based on the API request results, and so on. Based on this data, the performance of individual pipelines—defined in terms of computing resource efficiency, processing accuracy, latency, or user experience—can be determined, and corrections can be implemented.

[016] With reference to an illustrative embodiment, a multi-turn chat-based dialogue pipeline (also referred to simply as a dialogue-based pipeline for brevity) can be implemented to process and respond to natural language queries. The queries to be handled by the dialogue-based pipeline may relate to travel activities and bookings, such as hotels, airline tickets, car rentals, or packages of such activities. The dialogue-based pipeline can be configured to generate API requests, such as computer-executable search queries, to obtain responsive data to user travel queries. As used in this document, the terms natural language query and natural language request are not necessarily limited to questions, but may also include other statements made by a user to request a response or other action. Petition 870250107768, dated 11 / 25 / 2025, page 10 / 53 6 / 41 For example, a user might initiate a natural language processing session by speaking or typing a statement such as "Show me travel packages for my family during Thanksgiving." While not phrased as a question, it is understood that the user would like to search for potential travel packages related to the stated criteria.

[017] The dialog-based pipeline can include natural language understanding (NLU) components that evaluate the query text as typed by the user or as generated by an automatic speech recognition (ASR) component of the dialog-based pipeline. An NLU processing component can identify specific words or phrases in the query that reveal the user's intent in making the query and provide details about the intent. These words or phrases can be referred to as named entities, and the NLU processing component that identifies them can be referred to as a named entity recognition (NER) component. In the present example, an NER component can identify the intent as searching for travel packages and can identify additional entities such as a departure date entity (based on the expression during Thanksgiving) and a number of travelers entity (based on the expression my family).

[018] To respond to the intent to search for travel packages and thus to respond to the user's natural language query, the dialog-based pipeline can determine whether additional parameters are needed or whether the identified parameters must be confirmed to generate an API request. The parameters for the API request can correspond to specific named entities. For example, an intent to search for travel packages might require a departure date parameter and a number of travelers parameter that a user can specify through corresponding named entities. Additionally, the intent to search for travel packages might require a location parameter. Petition 870250107768, dated 11 / 25 / 2025, page 11 / 53 7 / 41 destination and a return date parameter. The dialog-based pipeline can generate text or graphical user interface (GUI) components to be presented to the user in one or more dialog loops. In the present example, the dialog-based pipeline can generate a prompt for the destination location parameter (e.g., a text-based natural language prompt, a GUI map component or dropdown list, etc.) and another prompt for the return date parameter (e.g., a second text-based prompt, a GUI calendar component, etc.). The dialog-based pipeline can also generate one or more prompts to confirm parameters that have been determined based on recognized named entities (e.g., a text-based summary and confirmation request, one or more interactive GUI components with certain pre-selected parameters, etc.).Once a sufficient set of parameters has been obtained and an API request has been generated, the dialog-based pipeline can send the API request to an application or system for execution. In this example, the dialog-based pipeline can send the API request to a scroll search application, and the scroll search application can generate search results for presentation to the user. The natural language processing session can then continue or end.

[019] The dialog-based pipeline can process many of these natural language queries (e.g., thousands, hundreds of thousands, millions, or more) and can generate and store telemetry data about the processing. For example, the dialog-based pipeline can store, for each natural language processing session, the initial natural language query that triggered the natural language processing session and the API request (and corresponding parameters) that was generated in response to the query. In some embodiments, additional telemetry data can be generated and stored, such as data that Petition 870250107768, dated 11 / 25 / 2025, page 12 / 53 8 / 41 reflects the dialogue loops between the initial query and the final API request, context data about the natural language processing session (e.g., time, date, user identifier, etc.) and the like.

[020] Telemetry data relating to natural language processing sessions managed by the dialogue-based pipeline can be used to generate training data to train a machine learning model that translates natural language queries into API requests. For example, a corpus of training data pairs can be generated from the telemetry data, whereby a single training data pair might include an initial natural language query from a session as a training data input item and the final API request that was generated during the natural language processing session as a reference data output item which is the desired or correct output to be generated by the machine learning model from the training data input item.In some modalities, additional data may be included in the training data input items, such as data reflecting additional dialogue turns between the initial query and the final API request, context data about the natural language processing session (e.g., time, date, user identifier, etc.) or user (e.g., demographic data, interaction history, etc.). For example, interaction history might provide an indication of what travel bookings a user has made, whether a user is currently on a trip with which the current query might be associated, or similar. This history can be used in determining parameter values ​​for API requests and thus used in translating the user's natural language processing queries into API requests.

[021] A machine learning model can be trained using training data generated from telemetry data to generate a Petition 870250107768, dated 11 / 25 / 2025, page 13 / 53 9 / 41 API request from a natural language query or some other natural language input. The trained model may be referred to as a natural language model for API or as an NL2API model for brevity. In some embodiments, the natural language model for API may be a translation model implemented using an artificial neural network (NN) or a variant thereof, such as a deep neural network (DNN), a recurrent neural network (RNN), a convolutional neural network (CNN), a transformer, an encoder / decoder, a set of several models, or some other model suitable for translation. The example machine learning model types and architectures described in this document are for illustrative purposes only and are not intended to be limiting, necessary, or exhaustive. In some embodiments, other model types may be used, including those not based on NNs.

[022] A model-driven pipeline can be implemented to process and respond to natural language queries using the trained natural language model for API. The queries to be handled by the model-driven pipeline can be of the same type as queries handled by the dialog-based pipeline described above. For example, natural language text—entered by a user or generated by an ASR component—can be used as input to the trained natural language model for API. The model's output can be an API request, with a partial or complete set of parameters, to be executed in response to the natural language input. The API request can then be executed directly, or a representation of the request or its parameters can be presented to the user for confirmation before execution.In this way, the natural language model for APIs can be used to generate a starting point that can short-circuit the multi-loop dialog box that would otherwise be used to obtain all the parameters needed for a given API request. Petition 870250107768, dated 11 / 25 / 2025, page 14 / 53 10 / 41

[023] The dialog-based pipeline can continue to be used to process some natural language queries after the model-based pipeline has been implemented. For example, the orchestrator can assign new natural language processing sessions to the dialog-based pipeline or the model-based pipeline based on a randomized selection algorithm (e.g., using a pseudo-random number generator), based on a load-balancing selection algorithm, based on test implementation logic, such as A / B testing logic or similar.

[024] The performance of a model-based pipeline can be evaluated alone or in comparison to a dialog-based pipeline or another model-based pipeline. For example, metrics relating to the efficiency of computing resources, processing accuracy, latency, user experience, or other aspects of natural language query processing can be logged for each processed natural language query (or a subset thereof). If the performance of a specific API natural language model falls below a threshold or lags more than the threshold degree behind the dialog-based pipeline or other API natural language models, the specific API natural language model may be retrained, replaced, or subjected to some other corrective action.

[025] In some embodiments, the orchestrator can assign new natural language processing sessions based on the past performance of specific natural language models for APIs or pipelines that process natural language queries, users, or similar. For example, the orchestrator can implement a rule-based selection algorithm, or the orchestrator can use a selection model trained on the past performance of available pipelines that process similar natural language queries. In this way, natural language query processing can be even more Petition 870250107768, dated 11 / 25 / 2025, page 15 / 53 11 / 41 optimized for cases where a single model-based path does not provide optimal performance.

[026] Several aspects of the disclosure will now be described in relation to certain examples and embodiments intended to illustrate, but not limit, the disclosure. Although the examples and embodiments described in this document focus, for illustrative purposes, on specific machine learning models, API requests, natural language processing pipelines, and algorithms, one skilled in the art will appreciate that the examples are merely illustrative and are not intended to be limiting. Furthermore, any feature, process, device, or component of any embodiment described and / or illustrated in this descriptive report may be used on its own, or with or instead of any other feature, process, device, or component of any other embodiment described and / or illustrated in this descriptive report. Example of a Natural Language Processing System

[027] FIG. 1 illustrates an example of a natural language processing system 100 configured to process and respond to natural language queries, such as those received from various user devices 102 through a network 104. The network 104 may be or include a personal area network (PAN), local area network (LAN), wide area network (WAN), global area network (GAN), or some combination thereof, any or all of which may or may not have access to and / or the Internet.

[028] User devices 102 may be or include any of a variety of computing devices configured to receive user input, communicate with the natural language processing system 100, and present output from the natural language processing system 100. For example, user devices 102 may include mobile phones with program execution and network communication capabilities (e.g., phones). Petition 870250107768, dated 11 / 25 / 2025, page 16 / 53 12 / 41 smart devices), portable devices with program execution and network communication capabilities (e.g., smartwatches, smart glasses), tablet computing devices, e-readers, portable video game devices, media players, televisions with program execution and network communication capabilities (e.g., smart TVs), television set-top boxes, video game consoles, speakers with program execution and network communication capabilities (e.g., smart speakers), laptops, desktop computing devices, terminal computing devices, and the like.

[029] The natural language processing system 100 can be configured to receive natural language input from various user devices 102 and generate responses or otherwise perform actions based on the natural language input. As shown, the natural language processing system 100 can include a plurality of components. In some embodiments, the natural language processing system 100 may include: an orchestrator 110 to orchestrate the processing of natural language queries, a dialogue-based pipeline 120 to process natural language queries according to a chat-based multi-turn dialogue paradigm, a model-based pipeline 130 to process natural language queries according to a machine learning model paradigm, and one or more applications 140 to execute the determined API requests in response to the natural language queries.

[030] The 120 dialogue-based pipeline can include several components to manage multi-turn chat-based dialogues. For example, the 120 dialogue-based pipeline can include: a 122 query parameter manager to determine which parameters are needed to execute an API request and to facilitate obtaining those parameters; a store Petition 870250107768, dated 11 / 25 / 2025, page 17 / 53 13 / 41 of dialog question data 124 that serves as a repository for questions or other prompts to obtain API request parameters or otherwise move a chat-based multi-turn dialog forward; an entity recognizer 126 to evaluate natural language input and identify named entities, such as those related to API request intents and parameters; a dialog manager 128 to dynamically generate responses or prompts (e.g., as an alternative to or complement to dialog question data storage 124); and / or various other chat-based multi-turn dialog management components.

[031] The 130 model-based pipeline can include several components to manage the generation of API requests using a natural language model for APIs trained in response to natural language queries.For example, the model-based pipeline 130 might include: an inference system 132 that uses a natural language model for API 150 to generate or predict API requests in response to natural language queries; a telemetry data store 134 to store telemetry data relating to previous natural language processing sessions, including those conducted using the dialogue-based pipeline 120; an evaluation and training system 136 to evaluate the performance of the natural language model for API 150, other models 150, and the dialogue-based pipeline; a user data store 138 to store and provide data that can be used by the inference system 132 and natural language model for API 150 in addition to natural language query input; and / or various other model-based dialogue management components.

[032] The individual components of the natural language processing system 100 can be implemented in one or more devices of Petition 870250107768, dated 11 / 25 / 2025, page 18 / 53 14 / 41 computing. For example, each component can be implemented on a separate computing device or on a separate set of computing devices. As another example, a single computing device or set of computing devices can be shared among multiple components. In some embodiments, the resources and services provided by the natural language processing system 100 can be provided by one or more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly provisioned and released computing resources, such as computing devices, network devices, and / or storage devices. A hosted computing environment may also be referred to as a cloud computing environment.In some embodiments, each component of the natural language processing system, or subsets of the components, can be implemented in different computing environments, any or all of which may or may not include a cloud computing environment. Example of a Training Data Generation Routine

[033] FIG. 2 illustrates an example of routine 200 for generating training data using a dialogue-based pipeline 120 and using the training data to train a natural language model for API. Routine 200 starts in block 202. In some embodiments, routine 200 may start in response to an event, such as the natural language processing system 100 initiating operation or being instructed to generate training data. When routine 200 starts, executable instructions may be loaded or otherwise accessed in computer-readable memory and executed by one or more computing devices of the natural language processing system 100, such as the computing system 700 shown in FIG. 7.

[034] In block 204, orchestrator 110 or some other component of Petition 870250107768, dated 11 / 25 / 2025, page 19 / 53 15 / 41 natural language processing system 100 can initiate a natural language processing session with a user device 102. The natural language processing session can begin in response to receiving a natural language query from the user device 102, in response to establishing a connection between the user device 102 and the natural language processing system 100, or in response to some other event.

[035] In block 206, the dialog-based pipeline 120 can manage the natural language processing session using one or more components of the dialog-based pipeline 120. For example, a query parameter manager 122 can employ an entity recognizer 126 to identify, in an initial natural language query and other natural language user inputs, information about parameters for an API request or other information associated with the API request or from which the API request can otherwise be determined. The query parameter manager 122 can use a dialog query data store 124 and / or a dialog manager 128 to generate prompts to the user to obtain API request parameters and to otherwise generate an API request to be executed. The orchestrator 110 can then send the API request to an application 140 for execution.Examples of natural language processing sessions managed using the dialog-based pipeline 120 are shown in FIGS. 3 and 4.

[036] FIG. 3 illustrates an example of a natural language processing session 300 processed using a dialogue-based pipeline 120 to generate a processing result set 320 as parameters for an API request. A user initiates the session by typing an initial query 302 or by speaking and having an ASR system generate a textual version of the initial query 302. In this example, the initial query 302 is I would like to book a trip during Thanksgiving for my family to a sunny place. Although the desired response is not Petition 870250107768, dated 11 / 25 / 2025, page 20 / 53 16 / 41 being stated directly in the initial query 302, it is understood that the user would like to search for possible travel packages related to the stated criteria (during Thanksgiving, my family, sunny place).

[037] Based on the initial query 302, the dialog-based pipeline 120 can determine a set of parameters that must be determined in order to execute an API request. In this example, the dialog-based pipeline 120 determines a set of parameters 322 associated with the search for travel packages. In some embodiments, as described in more detail below in relation to FIG. 4, the dialog-based pipeline 120 can also make an initial determination of one or more parameters based on the initial query. In the example shown in FIG. 3, the dialog-based pipeline did not make an initial determination of any parameters based on the initial query 302. Instead, the dialog-based pipeline 120 generates an initial response 304 to prompt the user for a parameter (how many nights).For example, query parameter manager 122 can determine parameter set 322 and request that dialog query data store 124 or dialog manager 128 provide the prompt for the initial response 304. The user responds with information for two parameters (November 20 and 7 nights).

[038] Entity recognizer 126 or some other dialog-based pipeline component 120 can identify values ​​for the StartDate and NightNumber parameters in the user's response, and query parameter manager 122 can generate the parameter set 324 based on it. Query parameter manager 122 can then request that dialog question data store 124 or dialog manager 128 provide a prompt for another parameter (How many people will be traveling with you?), to which the user responds with information for the parameter (3 children, 2 grandparents, my wife and I).

[039] Entity recognizer 126 or some other component of the pipeline Petition 870250107768, dated 11 / 25 / 2025, page 21 / 53 17 / 41 based on dialog 120 can identify a value for the NrViajantes parameter in the user's response, and the query parameter manager 122 can generate the parameter set 326 based on it. The query parameter manager 122 can then request that the dialog question data store 124 or the dialog manager 128 provide a prompt for another parameter (You will be flying from Seattle). In this case, user data, such as profile data or historical interaction data, can be used to determine a typical origin for this user (Seattle) to be included in the prompt. The user responds with information for the parameter (Yes).

[040] Entity recognizer 126 or some other dialog-based pipeline component 120 can determine that Yes confirms the value of Seattle for the Origin parameter and query parameter manager 122 can generate parameter set 328 based on it. Query parameter manager 122 can then request that dialog question data store 124 or dialog manager 128 provide a prompt for another parameter (Would you prefer the Caribbean or Polynesia). In this case, user data, such as profile data or historical interaction data, can be used to determine a typical destination for this user (Caribbean or Polynesia) or potential destinations associated with an entity named in the initial query 302 (in a sunny place) can be selected to be included in the prompt. The user responds with information for the parameter (Polynesia).In some modes, usage data associated with the current user and / or a larger set of users can be evaluated to determine what the phrase "sunny place" might mean. The results can be used for automatic parameter determination, dialog prompts, or both.

[041] The entity recognizer 126 or some other dialog-based pipeline component 120 can identify a value for the Destination parameter and the query parameter manager 122 can generate the parameter set 330 Petition 870250107768, dated 11 / 25 / 2025, page 22 / 53 18 / 41 based on it. The query parameter manager 122 can then determine that the parameter set 330 is a complete parameter set and can return the parameter set to the orchestrator 110 for execution. The orchestrator 110 can then cause the API request to be executed. For example, the orchestrator 110 can send the API request to an application 140, such as a travel package search application, for execution.

[042] FIG. 4 illustrates an example of a 400 natural language processing session processed using a 120 dialogue-based pipeline to generate a 420 processing result set as parameters for an API request. In contrast to the 300 natural language processing session shown in FIG. 3, the 400 natural language processing session shown in FIG. 4 may involve the presentation of graphical user interface controls instead of—or in addition to—textual chat responses. Alternatively, or in addition, the management of the 400 natural language processing session may involve an initial identification of one or more named entities in the initial natural language query and the use of the initially identified named entities as default values ​​in chat requests.

[043] In the illustrated example, a user starts the session by typing an initial 402 query or speaking and having an ASR system generate a text version of the initial 402 query. In this example, the same initial 402 query is used as in the example discussed above (I would like to book a trip during Thanksgiving for my family to a sunny place).

[044] Based on the initial 402 query, the 120 dialog-based pipeline can determine a set of parameters that must be determined in order to execute an API request. In this example, the 120 dialog-based pipeline determines a 422 parameter set associated with the travel package lookup. The 120 dialog-based pipeline can also make an initial determination of one or more Petition 870250107768, dated 11 / 25 / 2025, page 23 / 53 19 / 41 parameters based on the initial query 402 alone or in combination with user data from user data store 138. For example, entity recognizer 126 or some other dialog-based pipeline component 120 can identify the value for the StartDate parameter in the user's response. As another example, entity recognizer 126 or some other dialog-based pipeline component 120 can identify the term "my family" and look up, in user data store 138, the number of travelers in the user's family with whom the user has traveled in the past. As a further example, query parameter handler 122 can look up, in user data store 138, the most frequently used origin location for the user, even if an origin location was not mentioned or referenced in the initial query 302.

[045] The query parameter manager 122 can add the determined parameters to the initial parameter set 422. The query parameter manager 122 can then generate one or more prompts for other parameters. In the illustrated example, the query parameter manager 122 can generate GUI controls 404 for each parameter in the initial parameter set 422 or some combination thereof. The parameters for which values ​​have been preliminarily determined can have their values ​​reflected in the presentation.

[046] The dialog-based pipeline 120 can add or update parameters based on user interactions with the presented GUI controls. For example, the value for the StartDate parameter was updated in parameter set 424 based on a change selected by the user. The value for Destination was added in parameter set 426, and the value for NrNights was added in the final parameter set 428.

[047] The query parameter manager 122 can determine that the parameter set 428 is a complete parameter set and can return the parameter set to the orchestrator 110 for execution. The orchestrator 110 can Petition 870250107768, dated 11 / 25 / 2025, page 24 / 53 20 / 41 then cause the API request to be executed. For example, orchestrator 110 might send the API request to an application 140, such as a travel package search application, for execution.

[048] The example natural language processing sessions shown in FIGS. 3 and 4 are for illustrative purposes only and are not intended to be limiting, necessary, or exhaustive. In some embodiments, different interfaces, dialog flows, and query parameter determination processes may be used. For example, partial results may be presented during a session (such as session 300) based on the set of query parameters determined at different points in the session. As the session continues through subsequent dialog turns and additional parameters are determined, the results may be filtered or updated accordingly.

[049] Returning to FIG. 2, in block 208 orchestrator 110 can store data relating to the natural language processing session. The data about the natural language processing session can be referred to as telemetry data. The telemetry data for this session and any number of other sessions (e.g., thousands, hundreds of thousands, millions, or more) can be used later to train a natural language model for API 150, as described in more detail below.

[050] For example, during or after the completion of the natural language processing session 300 shown in FIG. 3, the orchestrator 110 can store telemetry data relating to the session in the telemetry data store 134. The orchestrator 110 can store telemetry data such as: a unique session identifier, the initial query 302 and the final parameter set 330, or the complete API call made to the application 140. As another example, during or after the completion of the natural language processing session 400 shown in FIG. 4, the orchestrator 110 can store telemetry data relating to Petition 870250107768, dated 11 / 25 / 2025, page 25 / 53 21 / 41 session in telemetry data storage 134. The orchestrator 110 can store telemetry data, such as: a unique session identifier, the initial query 402 and the final parameter set 428, or the complete API call made to the application 140. In some embodiments, the telemetry data may include additional data, such as one or more of the natural language queries or responses made by the user after the initial query 302 or 402, one or more of the parameter sets determined by the dialog-based pipeline 120 before the final parameter set 330 or 428, context data about the natural language processing session (e.g., time, date, user identifier, etc.) or similar.

[051] In decision block 210, the orchestrator or some other component of the natural language processing system 100 may determine whether a criterion for generating training data is met. In some embodiments, the criterion may be the performance of a limit number of natural language processing sessions or the storage of a limit number of initial natural language queries and corresponding API requests generated from the initial natural language queries. If the training data criterion has been met, routine 200 may proceed to block 212. Otherwise, routine 200 may return to block 204 for handling additional natural language processing sessions. In some embodiments, the portion of routine 200 beginning in block 212 may be performed in parallel or asynchronously with respect to additional iterations of the portion of routine 200 from block 204 to block 208.

[052] In block 212, the evaluation and training system 136 or some other component of the natural language processing system 100 may generate training data to train a natural language model for API 150. The training data may be generated from associated telemetry data. Petition 870250107768, dated 11 / 25 / 2025, page 26 / 53 22 / 41 natural language processing sessions managed using the 120 dialogue-based pipeline.

[053] In some embodiments, a corpus of training data pairs can be generated from the telemetry data. A training data pair can include a training data input item and a corresponding reference data output item that is the desired model output for the training data input item. A training data input item can be a column comprising an initial natural language query from a session, or a representation thereof suitable for input into a model. For example, a training data input item could be a vector including the initial query sequence converted into numeric values ​​for each character. As another example, the training data input item could be a vector including a tokenized version of the initial query, such as a different numeric token representation of each word or phrase in the initial query.As a further example, the training data input item could be a vector including word embeddings for each word (e.g., BERT word embeddings) from the input query, or a phrase embedding for the initial query as a whole (e.g., a BERT phrase embedding).

[054] In some modalities, additional input data, such as query context data, may be included in the training data input item. For example, a context vector may be generated using data about the session (e.g., time, date, user identifier, additional dialog turns, etc.). As another example, a context vector may include data about the user, such as data from the user data store 138 (e.g., profile data, demographic data, historical interaction data, etc.). The context vector may be appended to the vector representing the initial query to generate an augmented input vector to be used as a training data input item. Petition 870250107768, dated 11 / 25 / 2025, page 27 / 53 23 / 41 training.

[055] A reference data output item can be a column comprising an API request that is the desired output of the training data input item. For example, the reference data output item can be a fully formed API request with parameters. As another example, the reference data output item can be a set of values ​​or tokens from which a column representation of the desired API request can be derived.

[056] In block 214, the evaluation and training system 136 can train a natural language model for API 150 using the training data corpus generated from the telemetry data. In some embodiments, the natural language model for API 150 may be a translation model implemented using an artificial neural network or a variant thereof, such as an RNN, a CNN, a transformer, an encoder-decoder, a set of several models, or some other model suitable for translation. The example machine learning model types and architectures described in this document are for illustrative purposes only and are not intended to be limiting, necessary, or exhaustive.

[057] Generally described, NNs – including DNNs, CNNs, RNNs, and other NN-based models such as transformers and encoder-decoder models – have multiple layers of nodes, also known as neurons. Illustratively, an NN may include an input layer, an output layer, and any number of intermediate, internal, or hidden layers between the input and output layers. Individual layers may include any number of separate nodes. Nodes of adjacent layers may be logically connected to each other, and each logical connection between the various nodes of adjacent layers may be associated with a respective weight. Conceptually, a node can be thought of as a computational unit that computes an output value in Petition 870250107768, dated 11 / 25 / 2025, page 28 / 53 24 / 41 function of a plurality of different input values. Nodes can be considered connected when the input values ​​for the function associated with a current node include the output of functions associated with nodes in a previous layer, multiplied by weights associated with the individual connections between the current node and the nodes in the previous layer. When a NN is used to process input data in the form of an input vector or an array of input vectors (for example, a batch of input vectors of training data), the NN can perform a direct pass-through to generate an output vector or an array of output vectors, respectively. The input vectors can each include n separate data elements or dimensions, corresponding to the n nodes of the NN input layer (where n is some positive integer). Each data element can be a value, such as a number or floating-point integer.A direct pass typically involves multiplying the input vector matrix by a matrix representing the weights associated with the connections between the nodes of the input layer and the nodes of the next layer, and applying an activation function to the results. The process is then repeated for each subsequent NN layer. Some NNs have hundreds of thousands or millions of nodes and millions of weights for connections between the nodes of all adjacent layers.

[058] The connections between individual nodes of adjacent layers are each associated with a trainable parameter, such as a weight and / or bias term, which is applied to the value passed from the node of the previous layer to the activation function of the node of the subsequent layer. For example, the weights associated with the connections from an input layer to an internal layer to which it is connected can be arranged in a weight matrix W with a size of m x n, where m denotes the number of nodes in the internal layer and n denotes the dimensionality of the input layer. The individual rows in the weight matrix W can correspond to the individual nodes in the input layer, and the individual columns in the weight matrix W can correspond Petition 870250107768, dated 11 / 25 / 2025, p. 29 / 53 25 / 41 to the individual nodes in the inner layer. The weight w associated with a connection from any node in the input layer to any node in the inner layer can be located at the corresponding intersection site in the weight matrix W.

[059] Illustratively, a structured training data input item—or converted into—an input vector (or augmented input vector, as described above)—can be provided to a computer processor that stores or otherwise has access to the weight matrix W. The processor then multiplies the input vector by the weight matrix W to produce an intermediate vector. The processor can adjust individual values ​​in the intermediate vector using an offset or bias that is associated with the inner layer (e.g., by adding or subtracting a separate value from the weight that is applied). Additionally, the processor can apply an activation function to the individual values ​​in the intermediate vector (e.g., by using the individual values ​​as input to a sigmoid function or a rectified linear unit function (ReLU)).In some embodiments, there may be multiple inner layers, and each inner layer may or may not have the same number of nodes as each other inner layer. The weights associated with the connections from one inner layer to the next inner layer can be arranged in a weight matrix similar to the W weight matrix. The process of multiplying intermediate vectors by weight matrices and applying activation functions to the individual values ​​in the resulting intermediate vectors can be performed for each inner layer subsequent to the initial inner layer. The output layer of an NN makes output determinations of the last inner layer alone or in combination with information from previous inner layers, etc.

[060] A structured reference data output item—or converted into—an output vector may include the correct or otherwise desired output that a model 150 should produce for the data input item of Petition 870250107768, dated 11 / 25 / 2025, p. 30 / 53 26 / 41 corresponding training, as described above. The goal of the training may be to minimize the difference between output vectors (generated by the model from training input data items) and corresponding reference data output vectors. The evaluation of training data output vectors relative to reference data output vectors can be performed using a loss function (also referred to as an objective function), such as a binary cross-entropy loss function, a weighted cross-entropy loss function, a squared error loss function, a softmax loss function, some other loss function, or a composite of loss functions. A gradient of the loss function with respect to the model parameters (e.g., weights) can be calculated.The gradient can be used to determine the direction in which the individual parameters of the 150 model should be adjusted in order to minimize the loss function and thus minimize the degree to which the future output (e.g., training data output vectors) differs from the expected or desired output (reference data output vectors). The degree to which the individual parameters are adjusted can be predetermined or determined dynamically (e.g., based on the gradient and / or a hyperparameter). For example, a hyperparameter, such as a learning rate, can specify or be used to determine the magnitude of the adjustment to be applied to individual parameters of the 150 model.

[061] In some embodiments, the model training system may calculate the gradient for a subset of the training data, rather than the entire training data set. Therefore, the gradient may be referred to as a partial gradient because it is not based on the entire training data corpus. Instead, it is based on the differences between the training data output vectors and the reference data output vectors when processing only a specific subset of the training data. Petition 870250107768, dated 11 / 25 / 2025, p. 31 / 53 27 / 41

[062] In some embodiments, the model training system can update some or all of the model 150 parameters using a gradient descent method with backpropagation. In backpropagation, a training error is determined using a loss function (for example, as described above). The training error can be used to update individual model parameters in order to reduce the training error. For example, a gradient can be calculated for the loss function to determine how the weights in the weight matrices should be adjusted to reduce the error. The adjustments can be propagated back through the model 150 layer by layer.

[063] The evaluation and training system 136 can manage the training process by evaluating one or more stopping criteria. For example, a stopping criterion might be based on the accuracy of the model 150, as determined using the loss function, a test set, or both. If the accuracy satisfies a threshold or other criterion, the model 150 can be considered to have converged to a desired or adequate result and the training process can be stopped. As another example, a stopping criterion might be based on the number of training iterations (e.g., epochs) that have been performed, the elapsed training time, or similar.

[064] The example training process described in this document is for illustrative purposes only and is not intended to be limiting, necessary, or exhaustive. In some modalities, the training process may include additional, fewer, and / or alternative steps, depending on the type of model being trained, the application for which the model is being trained, etc.

[065] In block 216, the trained natural language model for API 150 can be deployed for use in processing natural language queries. For example, the evaluation and training system 136 can send the model (e.g., a file of the model parameters, executable code to implement it). Petition 870250107768, dated 11 / 25 / 2025, page 32 / 53 28 / 41 the model algorithm, etc.) for the inference system 132 for use.

[066] In block 218, the orchestrator can distribute subsequent natural language processing sessions between the dialogue-based pipeline 120 and the model-based pipeline 130 for processing. An example routine for distributing sessions in this way is shown in FIG. 5 and described in more detail below.

[067] In some embodiments, several natural language models for API 150 can be trained. For example, different frameworks can be used, such as an RNN-based model, an encoder-decoder model, etc. As another example, different training datasets, different training algorithms, different training hyperparameters, or different combinations thereof can be used. Multiple models can be used to effectively implement multiple model-based pipelines 130. Example of a Multi-Pipeline Session Management Routine

[068] FIG. 5 illustrates an example of routine 500 for distributing natural language processing sessions among multiple natural language processing pipelines, including a model-based pipeline 130 implementing a natural language model for API 150. Routine 500 starts in block 502. In some embodiments, routine 500 may start in response to an event, such as the natural language processing system 100 starting operation or after the deployment of a trained natural language model for API 150. When routine 500 starts, executable instructions may be loaded or otherwise accessed in computer-readable memory and executed by one or more computing devices of the natural language processing system 100, such as computing system 700 shown in FIG. 7.

[069] In block 504, orchestrator 110 or some other component of Petition 870250107768, dated 11 / 25 / 2025, page 33 / 53 29 / 41 natural language processing system 100 can receive a natural language query from a user device 102 and can start a natural language processing session.

[070] In decision block 506, orchestrator 110 can determine which processing pipeline from the multiple available processing pipelines should be assigned to manage the natural language processing session. Although FIG. 5 shows orchestrator 110 determining between a dialogue-based pipeline 120 and a model-based pipeline 130, the illustration is provided for example purposes only and is not intended to be limiting, necessary, or exhaustive. In some embodiments, orchestrator 110 can determine between multiple model-based pipelines 130 (e.g., different pipelines, each with a different natural language model for API 150, or a single pipeline configured to use any of several different natural language models for API 150).In some embodiments, the orchestrator 110 may determine between several dialogue-based pipelines 120 (for example, one dialogue-based pipeline 120 configured to use a dialogue question data store 124 and another dialogue-based pipeline 120 configured to use a generative AI-based dialogue manager 128) instead of, or in addition to, one or more model-based pipelines 130.

[071] In one example, orchestrator 110 can select a pipeline based on a randomized selection algorithm (e.g., using a pseudo-random number generator). As another example, orchestrator 110 can select a pipeline based on a load balancing algorithm (e.g., based on which pipeline has the fewest open connections, based on available computing resource metrics, based on a round-robin algorithm, etc.). As a further example, orchestrator 110 can select a pipeline using a selection model that considers language input. Petition 870250107768, dated 11 / 25 / 2025, page 34 / 53 30 / 41 natural and / or associated information (e.g., user, time of day, initial classification of the natural language query subject, etc.) and selects the pipeline most likely to provide satisfactory performance (e.g., measured in terms of user satisfaction, resource usage, etc.). The example pipeline selection algorithms described in this document are for illustrative purposes only and are not intended to be limiting, necessary, or exhaustive. In some embodiments, additional or alternative selection algorithms may be used.

[072] In block 508, if orchestrator 110 has assigned the natural language processing session to dialog-based pipeline 120, dialog-based pipeline 120 conducts the session to determine the parameters for the API request to be generated. In block 510, dialog-based pipeline 120 uses the determined parameters to generate the API request to be executed. Examples of natural language processing sessions managed using a dialog-based pipeline 120 are shown in FIGS. 3 and 4 and are described in more detail above.

[073] In block 512, if orchestrator 110 has assigned the natural language processing session to model-driven pipeline 130, model-driven pipeline 130 may begin by obtaining context data associated with the natural language query and / or the user. For example, context data may include session data (e.g., time, date, user identifier, etc.). As another example, context data may include data about the user or user device 102 from which the natural language query was received, such as data from the user's data store 138 (e.g., profile data, demographic data, historical interaction data, etc.). In some embodiments, no context data is used and block 512 is not executed.

[074] In some modes, one or more dialogue loops may be performed to obtain parameters for the API request to be generated or to obtain Petition 870250107768, dated 11 / 25 / 2025, page 35 / 53 31 / 41 data from which the parameters for the API request can be determined. For example, a dialogue management and / or a dialog question data store can be used to generate a prompt for additional information.

[075] In block 514, the inference system 132 or some other model-based pipeline component 130 may process the natural language query and (optionally) context data using a natural language model for API 150 to generate an API request to be executed.

[076] FIG. 6 illustrates an example of a natural language processing session 600 processed using a model-based pipeline 130 to predict or generate an API request 624 including a set of parameters. In contrast to sessions managed using the dialog-based pipeline 120 (e.g., natural language processing sessions 300 and 400 shown in FIGS. 3 and 4), the natural language processing session 600 may involve the presentation of search results 604 (or other results of an executed API request) after an initial query 602, without necessarily proceeding with a multi-round dialog box.

[077] In the illustrated example, a user starts the session by typing an initial 602 query or speaking and having an ASR system generate a text version of the initial 602 query. In this example, the same initial 602 query is used as in the examples discussed above (I would like to book a trip during Thanksgiving for my family to a sunny place).

[078] Based on the initial query 602, the inference system 132 or some other model-based pipeline component 130 can generate input for analysis by a natural language model for API 150. In this example, the inference system 132 generates a natural language query representation 620, as a vector. The natural language query representation 620 can include the text Petition 870250107768, dated 11 / 25 / 2025, page 36 / 53 32 / 41 of the initial query 602 in numeric form. For example, the natural language query representation 620 might include the initial query string converted into numeric values ​​for each character. As another example, the natural language query representation 620 might include a tokenized version of the initial query 602, such as a different numeric token-based representation of each word or phrase in the initial query 602. As a further example, the natural language query representation 620 might include word embeddings for each word (e.g., BERT word embeddings), or a phrase embedding for the initial query 402 (e.g., a BERT phrase embedding).

[079] In some embodiments, additional data, such as query context data, may be included as input to the natural language model for API 150. For example, the inference system 132 may generate a context data representation 622, such as a vector, using data about the initial query 602 or the session 600 (e.g., time, date, user identifier, etc.). As another example, the inference system 132 may include data about the user, such as data from the user data store 138 (e.g., profile data, demographic data, historical interaction data, etc.).

[080] The natural language query representation 620 – augmented by appending a context data representation 622, if used – can be processed by the natural language model for API 150 to generate an API request 624. In some embodiments, the API request 624 can be generated as a fully formed request with parameters, as illustrated. In some embodiments, the output of the natural language model for API 150 can be a set of values ​​or tokens. For example, the natural language model for API 150 can generate output representing the API request to be made (e.g., a call to the uniform resource locator search.foo in this example) and a set of parameter values ​​for the parameters associated with the Petition 870250107768, dated 11 / 25 / 2025, page 37 / 53 33 / 41 API request (e.g., SEA, FrenchPolynesia, 11 / 25, 7 and 7 in this example). The inference system 132 can assemble the final API request 624 from the model output, or the model output can be provided to the orchestrator 110 or some other component for assembly into the final API request 624 to be executed. The orchestrator 110 can then cause the API request 624 to execute. For example, the orchestrator 110 can send the API request 624 to an application 140, such as a travel package search application, for execution.

[081] Returning to routine 500, in block 516 orchestrator 110 can execute the API request or send the API request to an application 140 for execution and continue with session management. In some modes, session management may involve additional dialog loops and potentially additional natural language queries that are processed using one of the natural language processing pipelines. The natural language processing pipeline may be the same pipeline that processed the session's initial natural language query, or routine 500 may return to block 504 with each new natural language query.

[082] In some modes, the execution of the API request may return one or more errors instead of - or in addition to - a response. If certain errors are received, one or more dialog loops may be performed to obtain parameters for the API request to be generated or to obtain data from which the parameters and / or the API request can be determined. For example, a dialog management and / or dialog question data store may be used to generate a prompt for additional information.

[083] In block 518, orchestrator 110 or some other component of the natural language processing system 100 may store session data relating to the session that was managed in the current iteration of routine 500. For example, during or after the completion of the language processing session. Petition 870250107768, dated 11 / 25 / 2025, page 38 / 53 34 / 41 naturally, the 110 orchestrator can store telemetry data, such as: a unique session identifier, the initial query, the API requests generated during the session, performance data (e.g., time elapsed to generate API requests, number of dialog turns performed before generating API requests, computing resources used to generate API requests), user experience data (an indicator of whether the session was abandoned, an indicator of user engagement during or as a result of the session), etc.

[084] In block 520, the 136 evaluation and training system can evaluate the performance of the model (and / or one or more pipelines). The performance of a given model or pipeline can be evaluated across a set of multiple sessions, such as dozens, hundreds, thousands, or more. In some modes, the 136 evaluation and training system can determine one or more metrics from stored telemetry data for the sessions being used in the evaluation. For example, the 136 evaluation and training system can determine statistics such as mean, median, standard deviation, etc., for one or more of: elapsed time to generate API requests; number of dialog turns performed before generating API requests; computing resources used to generate API requests; an engagement metric representing user engagement during or as a result of the session; or some other metrics.

[085] In decision block 522, the evaluation and training system 136 can determine whether a model retraining criterion has been met. In some modalities, the model retention criterion may refer to a threshold for one or more metrics determined in block 520. For example, if a metric falls below a threshold value (e.g., user engagement) or exceeds a threshold value (e.g., elapsed time, number of dialogue turns), the model training criterion may be met and routine 500 may proceed to block 524. As another example, if a metric falls more than a threshold amount Petition 870250107768, dated 11 / 25 / 2025, page 39 / 53 If a metric falls 35 / 41 behind a similar metric for another pipeline (for example, the engagement for sessions managed using model-based pipeline 130 falls more than a threshold amount behind the engagement for session management using another model-based pipeline 130 or a dialogue-based pipeline 120), or if a metric exceeds a similar metric for another pipeline by more than a threshold amount (for example, the elapsed time for sessions managed using model-based pipeline 130 exceeds the elapsed time for sessions managed using another model-based pipeline 130 or a dialogue-based pipeline 120), then the model training criterion can be satisfied and routine 500 can proceed to block 524.

[086] In some modalities, the model retraining criterion may not necessarily refer to performance metrics. Instead, the model retraining criterion may be based on time or amount of data. For example, model retraining may be triggered periodically, such as daily, weekly, monthly, quarterly, annually, etc. As another example, model retraining may be triggered once a limit amount of telemetry data has been stored and from which new training data can be generated.

[087] In block 524, evaluation and training system 136 can retrain a natural language model for API 150. In some embodiments, evaluation and training system 136 can re-execute the training process by which the natural language model for API 150 was originally trained, as described in more detail above. For example, evaluation and training system 136 can use telemetry data for more recently conducted natural language processing sessions to generate a training data corpus and retrain model 150. After retraining the natural language model for API 150, it can be redeployed and used in subsequent iterations of routine 500. Petition 870250107768, dated 11 / 25 / 2025, pp. 40 / 53 36 / 41 Execution Environment

[088] FIG. 7 illustrates several components of an example 700 computing system configured to implement various functionalities described herein. The 700 computing system may be or include one or more physical host computing devices.

[089] In some embodiments, as shown, a computing system 700 may include: one or more computer processors 702, such as physical central processing units (CPUs); one or more network interfaces 704, such as network interface cards (NICs); one or more computer-readable media units 706, such as high-density disk drives (HDDs), solid-state drives (SSDs), flash drives and / or other persistent non-transient computer-readable media; and one or more computer-readable memories 710, such as random-access memory (RAM) and / or other volatile non-transient computer-readable media.

[090] Computer-readable memory 710 may include computer program instructions that one or more computer processors 702 execute and / or data that one or more computer processors 702 use in order to implement one or more modes. For example, computer-readable memory 710 may store an operating system 712 to provide general administration of the model training system 700. As another example, computer-readable memory 710 may store dialogue-based pipeline instructions 714 to implement the systems, components, and features of a dialogue-based pipeline 120. As another example, computer-readable memory 710 may store model-based pipeline instructions 716 to implement the systems, components, and features of a model-based pipeline 130. As another example, computer-readable memory 710 may store orchestrator instructions 718 to implement the features of a Petition 870250107768, dated 11 / 25 / 2025, pp. 41 / 53 37 / 41 orchestrator 110. As another example, computer-readable memory 710 can store application instructions 720 to implement the features of an application 140 that executes API requests generated by a dialog-based pipeline 120 or model-based pipeline 130. As another example, computer-readable memory 710 can store evaluation and training instructions 722 to implement the features of an evaluation and training system 136. Terminology

[091] Depending on the mode, certain acts, events, or functions of any of the processes or algorithms described in this document may be performed in a different sequence, may be added, merged, or left out (for example, not all operations or events described are necessary for the practice of the algorithm). Furthermore, in certain modes, operations or events may be performed simultaneously, for example, through multithreaded processing, interrupt processing, or multiple processors or processor cores, or in other parallel architectures, rather than sequentially.

[092] The various illustrative logic blocks, modules, routines, and algorithm steps described in connection with the embodiments disclosed in this document may be implemented as electronic hardware or combinations of electronic hardware and computer software. To clearly illustrate this interchangeability, various illustrative components, blocks, modules, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or as software that runs on hardware depends on the specific application and design constraints imposed on the overall system. The described functionality may be implemented in various ways for each particular application, but such implementation decisions should not be interpreted as causing a deviation from the scope of the disclosure. Petition 870250107768, dated 11 / 25 / 2025, pp. 42 / 53 38 / 41

[093] In addition, the various illustrative logic blocks and modules described in connection with the embodiments disclosed in this document may be implemented or executed by a machine, such as a processing device, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described in this document. A processing device may be a microprocessor, but alternatively, the processing device may be a controller, microcontroller, or state machine, combinations thereof, or similar. A processing device may include electrical circuits configured to process computer-executable instructions.In another embodiment, a processing device includes an FPGA or other programmable device that performs logical operations without processing computer-executable instructions. A processing device may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Although described in this document primarily in relation to digital technology, a processing device may also include primarily analog components. For example, some or all of the algorithms described in this document may be implemented in analog circuits or mixed analog-digital circuits.A computing environment can include any type of computer system, including, but not limited to, a microprocessor-based computer system, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computing engine within an appliance, to name a few. Petition 870250107768, dated 11 / 25 / 2025, pages 43 / 53 39 / 41

[094] The elements of a method, process, routine, or algorithm described in connection with the embodiments disclosed in this document may be incorporated directly into hardware, in a software module executed by a processing device, or in a combination of both. A software module may reside in RAM, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of non-transient computer-readable storage medium. An exemplary storage medium may be coupled to the processing device so that the processing device can read information from and write information to the storage medium. Alternatively, the storage medium may be integral to the processing device. The processing device and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal.Alternatively, the processing device and storage medium can reside as discrete components in a user terminal.

[095] The conditional language used in this document, such as, but not limited to, may, could, might, can, for example, and the like, unless specifically indicated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain modalities include, while other modalities do not include, certain features, elements, and / or steps. Thus, such conditional language is not generally intended to imply that features, elements, and / or steps are in any way necessary for one or more modalities or that one or more modalities necessarily include logic to decide, with or without further input or request, whether those features, elements, and / or steps are included or must be performed in any particular modality.The terms comprising, including, having, and similar terms are synonymous and are used inclusively, in an open-ended way, and do not exclude additional elements, characteristics, acts, operations, and so forth. Furthermore, Petition 870250107768, dated 11 / 25 / 2025, pages 44 / 53 40 / 41 the term or is used in its inclusive sense (and not in its exclusive sense), so that when used, for example, to connect a list of elements, the term or means one, some, or all of the elements in the list.

[096] Disjunctive language, such as the phrase at least one of X, Y, Z, unless specifically indicated otherwise, is understood differently with the context used in general to present that an item, term, etc., can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain modalities require at least one of X, at least one of Y, or at least one of Z for each to be present.

[097] Unless explicitly stated otherwise, articles such as "a" or "an" should generally be interpreted to include one or more described items. Therefore, phrases such as "a device configured to" are intended to include one or more recited devices. Such one or more recited devices may also be collectively configured to perform the stated recitations. For example, a processor configured to perform recitations A, B, and C may include a first processor configured to perform recitation A working in conjunction with a second processor configured to perform recitations B and C.

[098] Although the detailed description above has shown, described, and pointed out new features applied to various embodiments, it may be understood that various omissions, substitutions, and alterations in the form and details of the illustrated devices or algorithms may be made without departing from the spirit of the disclosure. As may be recognized, certain embodiments described in this document may be incorporated in a form that does not provide all the features and benefits set forth in this document, since some features may be used or practiced separately from others. The scope of certain Petition 870250107768, dated 11 / 25 / 2025, pages 45 / 53 41 / 41 of the modalities disclosed in this document are indicated by the attached claims instead of the previous description. All changes that fall within the meaning and range of equivalence of the claims shall be included within their scope. Petition 870250107768, dated 11 / 25 / 2025, pp. 46 / 53

Claims

1 / 7 CLAIMS 1. A system CHARACTERIZED in that it comprises: computer-readable memory storing executable instructions; and one or more processors programmed by the executable instructions to at least: manage a plurality of natural language processing sessions, wherein a natural language processing session of the plurality of natural language processing sessions comprises: receiving a query in natural language; generating one or more natural language prompts for parameter data associated with an application programming interface (API) request to be executed; receiving one or more natural language responses corresponding to one or more natural language requests; and executing the API request;Generate a training data corpus based on a plurality of natural language processing sessions, wherein the training data corpus comprises a plurality of training data input vectors and a plurality of reference data output vectors, wherein a reference data output vector from the plurality of reference data output vectors represents the API request as a desired output to be generated from a training data input vector, from the plurality of training data input vectors, representing the natural language query; and train a natural language model for API using the training data corpus, wherein the natural language model for API is trained to generate API requests using natural language query input.

2. System, according to claim 1, CHARACTERIZED by the fact that the API request comprises a search query regarding a travel reservation.

3. System according to claim 1, CHARACTERIZED in that a specific API request generated by the natural language model for API comprises a sequence of characters representing a plurality of parameters and in which a natural language query input, used by the natural language model for API to generate the specific API request, represents less than the entire plurality of parameters.

4. System, according to claim 1, CHARACTERIZED in that the training data input vector represents both the natural language query and the context data with respect to a user account associated with the natural language query.

5. System, according to claim 4, CHARACTERIZED in that the natural language model for the API is trained to generate a specific API request using both the natural language query input and the context data input associated with the natural language query input.

6. System, according to claim 1, CHARACTERIZED in that one or more processors are further programmed by executable instructions to send, to an inference system, the natural language model for API.

7. System, according to claim 1, CHARACTERIZED in that one or more processors are further programmed by executable instructions to: manage a second plurality of natural language processing sessions using the natural language model for API; evaluate the performance of the natural language model for API based on the management of the second plurality of natural language processing sessions; and Petition 870250090552, dated 10 / 03 / 2025, page 9 / 15 3 / 7 determine, based on the evaluation of the performance of the natural language model for API, to retrain the natural language model for API.

8. System according to claim 1, CHARACTERIZED in that one or more processors are further programmed by executable instructions to: determine, for a first subset of natural language processing sessions from a second plurality of natural language processing sessions, manage each natural language processing session of the first subset using the natural language model for API; and determine, for a second subset of natural language processing sessions from the second plurality of natural language processing sessions, manage each natural language processing session of the second subset using a dialog-based query parameter manager.

9. System, according to claim 8, CHARACTERIZED in that one or more processors are further programmed by executable instructions to use a selection model to determine whether a natural language processing session from the second plurality of natural language processing sessions should be managed using the natural language model for API or the dialog-based query parameter manager.

10. System, according to claim 8, CHARACTERIZED in that one or more processors are further programmed by executable instructions to determine, for a third subset of natural language processing sessions of the second plurality of natural language processing sessions, to manage each natural language processing session of the third subset of natural language processing sessions using a second natural language model for API different from the natural language model for API.

11. System, according to claim 10, CHARACTERIZED in that one or more processors are further programmed by executable instructions to: evaluate the performance of the natural language model for API based on the management of the first subset of natural language processing sessions; evaluate the performance of the second natural language model for API based on the management of the third subset of natural language processing sessions; and determine, based on the performance of the natural language model for API exceeding the performance of the second natural language model for API by a limiting amount, to retrain the second natural language model for API.

12. System, according to claim 1, CHARACTERIZED in that the natural language model for the API comprises one of: a transformer-based artificial neural network, a recurrent neural network, a convolutional neural network, or an encoder-decoder machine learning model.

13. Computer-implemented method CHARACTERIZED by the fact that it comprises: under the control of a computing system comprising one or more processors configured to execute specific instructions, obtaining a training data corpus based on a plurality of natural language processing sessions, wherein the training data corpus comprises a plurality of training data input vectors and a plurality of reference data output vectors, wherein a reference data output vector of the plurality of reference data output vectors. Petition 870250090552, dated 03 / 10 / 2025, p.11 / 15 5 / 7 cia represents a travel-based search request as a desired output to be generated from a training data input vector, from the plurality of training data input vectors, representing a natural language query regarding a travel booking; and train a natural language model for API using the training data corpus, wherein the natural language model for API is trained to generate predicted travel-based search requests using natural language query input.

14. A computer-implemented method according to claim 13, CHARACTERIZED in that the training of the natural language model for the API comprises generating a specific API request comprising a sequence of characters representing a plurality of parameters, wherein a natural language query input, used by the natural language model for the API to generate the specific API request, represents less than the entire plurality of parameters.

15. A computer-implemented method according to claim 13, CHARACTERIZED in that the training data input vector represents both the natural language query and the context data with respect to a user account associated with the natural language query, and in that training the natural language model for API comprises training the natural language model for API to generate a specific API request using both the natural language query input and the context data input associated with the natural language query input.

16. A computer-implemented method according to claim 13, CHARACTERIZED in that it further comprises sending the natural language model to an API for an inference system.

17. Computer-implemented method CHARACTERIZED by the fact that Petition 870250090552, dated 10 / 03 / 2025, page 12 / 15 6 / 7 comprises: under the control of a computing system comprising one or more processors configured to execute specific instructions, receiving a query in natural language; generating natural language model input for API representing at least a portion of the natural language query; generating model output using a natural language model for API and the natural language model input for API, wherein the model output represents an API request to be executed in response to the natural language query; and executing the API request.

18. A computer-implemented method according to claim 17, CHARACTERIZED in that it further comprises: determining, for a first subset of natural language processing sessions from a plurality of natural language processing sessions, managing each natural language processing session of the first subset using the natural language model for API; and determining, for a second subset of natural language processing sessions from the plurality of natural language processing sessions, managing each natural language processing session of the second subset using a dialog-based query parameter manager.

19. A computer-implemented method according to claim 18, CHARACTERIZED in that it further comprises the use of a selection model to determine whether a natural language processing session from a plurality of natural language processing sessions should be managed using the natural language model for API or the dialog-based query parameter manager. Petition 870250090552, dated 10 / 03 / 2025, p. 13 / 15 7 / 7 20. A computer-implemented method according to claim 18, CHARACTERIZED in that it further comprises: determining, for a third subset of natural language processing sessions from the plurality of natural language processing sessions, managing each natural language processing session of the third subset of natural language processing sessions using a second natural language model for API different from the natural language model for API; evaluating the performance of the natural language model for API based on the management of the first subset of natural language processing sessions; evaluating the performance of the second natural language model for API based on the management of the third subset of natural language processing sessions;and determine, based on the performance of the natural language model for API exceeding the performance of the second natural language model for API by a limiting amount, to retrain the second natural language model for API. Petition 870250090552, dated 10 / 03 / 2025, pp. 14 / 15;

Images