Application programming interface response compression

Abstract

Systems and methods are provided for an application programming interface (API) response compression system for use in conjunction with API requests made by a large language model (LLM) agent in response to a prompt made to the LLM. The API response compression (ARC) system can receive an API response, generate an attribute manifest identifying a set of fields in the API response, generate a filtered attribute manifest identifying fields in the API response that are relevant to the prompt, generate a reduced API response, and process the prompt and the reduced API response at the LLM to generate an LLM output.

Description

Background Technology

[0001] Generally, computing devices and communication networks can be used to exchange data or information. In common applications, a computing device can request content from another computing device via a communication network. For example, a client with access to a computing device can use a software application to request content from a server computing device via a network (e.g., the Internet). In such implementations, the client's computing device may be referred to as a client computing device, and the server computing device may be referred to as a content provider.

[0002] In some applications, network service providers may instantiate various web-based services that can handle client requests for data. For example, a web service associated with query processing or question-answering assistants (e.g., chatbots) may correspond to a web-based service that involves human interaction to provide information (e.g., information about the web-based service, how to use the web-based service, etc.). Attached Figure Description

[0003] Embodiments of various inventive features will now be described with reference to the following accompanying drawings. Throughout the drawings, reference numerals may be reused to indicate correspondences between referred elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of this disclosure. For ease of identification of any particular element or action, the most significant digit in the reference numerals typically refers to the drawing number in which that element was first introduced.

[0004] Figure 1 A schematic diagram depicting a cloud provider network implementing an application programming interface (API) response compression system according to various aspects of this disclosure.

[0005] Figure 2 Based on all aspects of this application Figure 1 A visualization depicting the environment in which the LLM agent used for query / prompt processing, the API endpoints, and the API response compression system used to generate compressed API responses interact illustratively.

[0006] Figure 3 This is a flowchart illustrating a routine for compressing API responses according to various aspects of this application.

[0007] Figure 4 This is a flowchart illustrating a routine for generating compressed API responses using an API response compression system, comprising a manifest builder, attribute selector, and response refiner, according to various aspects of this application.

[0008] Figure 5 This is a block diagram illustrating the architecture of an API response compression system according to various aspects of this application. Detailed Implementation

[0009] Generally speaking, aspects of this disclosure relate to systems and methods for compressing API responses using Application Programming Interface (API) response compression middleware to generate “compressed” API responses suitable for input into a Large Language Model (LLM). An LLM can be understood as a machine learning model that uses artificial intelligence (AI) to generate human-like text in response to various types of input. An LLM can be a generative AI model trained on large amounts of text data to generate new content based on the training data. An LLM can be instantiated and executed on a computer or any number of computing devices. In some examples, an LLM can use an LLM agent to interact with endpoints (e.g., network endpoints). An LLM agent can be understood as an LLM instance with additional interface code (e.g., an HTTP interface) that allows the LLM instance to output in a given format (e.g., HTTP GET / POST) to result in a corresponding network call. This LLM agent can then receive a response to the network call and input this response back into the LLM instance. For example, an LLM can receive a prompt (e.g., from a human end-user or computing system) instructing the relevant LLM agent to make a request to an API. In response, the API returns an API response to the LLM agent, which then provides the API response as input to the LLM for use in generating output in response to the received prompts. However, APIs often return verbose responses, and while these long API responses work well in traditional software applications, API responses with more information than required may not be well-suited for LLMs.

[0010] More specifically, feeding verbose API responses into an LLM can lead to inaccuracies. By design, API responses often contain more information than is needed for a given task. Traditional software applications can deterministically locate relevant information from API responses and discard the remaining irrelevant information. In contrast, LLMs are stochastic, driven by random probability distributions rather than deterministic logic. For this reason, LLMs output inaccurate information with a certain probability, and the probability of inaccurate LLM outputs increases when the LLM receives irrelevant input. In other words, feeding irrelevant information into an LLM can reduce the accuracy of the corresponding LLM output. For this reason, feeding “long” API responses into an LLM—or any API response containing irrelevant information beyond the required relevant information that is not needed for the task at hand—can reduce the accuracy of the corresponding LLM output.

[0011] Besides reducing the accuracy of LLM output, inputting API responses that are longer than necessary into the LLM can also increase its latency. LLM resource usage is typically proportional to the number of tokens involved in responding to prompts (e.g., the basic text units that act as building blocks for the LLM to understand and generate text). Resource-intensive tasks in LLM can be understood as tasks that require a large number of tokens, and tasks that require more tokens require more computing power. Generally, the more computing power required to complete a task, the longer that task takes. For the purposes of this discussion, this additional time required can be understood as latency. In the context of LLM, increased latency means that the LLM's response time to received prompts may become slower. Slower LLM response times can lead to several negative consequences, including increased operating costs and reduced end-user adoption. Because inputting API responses that are longer than necessary into the LLM typically requires the use of thousands of LLM tokens, passing such API responses to the LLM generally results in increased latency for LLM performance. For example, due to increased wait times, an LLM might receive an API response that is longer than necessary and take a long time to respond to the user's prompt, causing the user to give up before receiving a response. In some cases, the API response might be too long to be entered into the LLM at all.

[0012] For example, a user might prompt the LLM to include the email addresses of all attendees scheduled for a meeting with the user on a given calendar day. In response, the LLM agent might make a request to a calendar API, which might return a longer API response than necessary, listing various types of irrelevant calendar meeting data (in addition to relevant data including attendee email addresses). In this example, what the LLM needs from the calendar API response is not all meeting data (e.g., meeting time, location, attachments, etc.), but rather relevant meeting data (e.g., email addresses of each meeting attendee). In some instances, such API responses might be designed to be verbose: the calendar API might be designed to provide additional information in a format beneficial to traditional software applications. In other instances, the specific task at hand might not correspond well to the available API response format, and therefore the LLM agent must make the most appropriate call possible (even if such a call results in an API response that also includes irrelevant information). In this case, the LLM agent in this example calls the calendar API, which returns an API response filled with irrelevant calendar meeting data, because that particular calendar API call is the best way to retrieve meeting attendee email addresses from the calendar API. Even so, if the LLM agent then directly feeds this longer-than-necessary API response along with all irrelevant calendar meeting data into the LLM for use in generating the output for the user prompt, the LLM may suffer from latency and accuracy issues. Therefore, as illustrated in this example, a method is needed to compress the API response in context before returning it to the LLM in compressed form for use in generating output. More specifically, the LLM generates a need for the ability to compress API responses such that the compressed API response retains the original structure of the API response while excluding most or all information irrelevant to the prompt provided to the LLM. It is worth noting that, for the purposes of this disclosure, the term "compression" refers to the removal of irrelevant information (rather than to the ability to efficiently store the same information with fewer bits). For this reason, the related concept of "compression" as removing irrelevant information from API responses is also referred to throughout this disclosure as "reducing" or "filtering" irrelevant information from API responses.

[0013] The API Response Compression (ARC) system disclosed herein addresses the challenges mentioned above, as well as others. Various aspects of this disclosure relate to using an ARC system as middleware that identifies and reduces API responses before returning newly filtered (“compressed”) responses to an LLM for use in further query processing. In some embodiments, the ARC system may incorporate one or more machine learning algorithms configured according to the LLM. Illustratively, various aspects of this application correspond to identifying API responses received in response to API calls generated by an LLM agent, supplying the API responses to the ARC system to generate filtered API responses, and returning the resulting filtered API responses to the LLM for use in answering prompts. In some embodiments, the ARC system may include three components for generating filtered API responses: a manifest builder, an attribute selector, and a response refiner. Using these three components, the ARC system can illustratively generate an attribute manifest, customize the attribute manifest to select relevant attributes, and refine the API response (e.g., recursively, iteratively, etc.) to generate context-filtered API responses for use by the LLM.

[0014] Previous attempts to address the accuracy and latency challenges faced by LLMs when dealing with API responses that are longer than necessary have, at best, required developers and engineers to manually customize and update the API response format. However, such manual customization attempts (besides being costly, prone to human error, inconsistent in large systems, and time-consuming) typically still fail to solve the challenges posed by API responses that are longer than necessary. That is, LLMs frequently encounter queries that developers cannot anticipate while customizing the API response format. Furthermore, manually attempting to customize the API response format often introduces more problems than it solves—making it possible for the system to unintentionally change or completely remove necessary parts of the API response format during customization. Therefore, even previous methods designed to manually shorten API responses, which could be well-intentioned by developers, are ultimately ineffective. Figure 1 Even with meticulous execution, such manual customization inevitably leads to isolated, inconsistent, and error-prone systems that still suffer from challenges in accuracy and latency.

[0015] If developers don't attempt to manually customize and address the challenges posed by API responses that are longer than necessary, the LLM will face another type of negative consequence: it might fail to produce any output at all. That is, an LLM encountering an API response that is too long to be used might generate an error message, thus creating a negative experience for the end user or system providing the prompts. This trade-off in reliability negatively impacts user adoption and trust levels, as well as the efficiency of systems that rely on the LLM to resolve complex issues involving API calls. For this reason, simply ignoring the problems caused by API response lengths that are longer than necessary is not a viable solution for function call brokers that use APIs.

[0016] Therefore, this disclosure represents improvements to many generative AI systems (and thus computing systems in general) that use function call agents and APIs, thereby increasing the output accuracy of LLM agents while reducing the latency caused by traditionally verbose API responses in such agents. The implementation of the ARC system disclosed herein improves the ability of computing systems (such as cloud computing systems providing generative AI services) to implement such services without sacrificing the accuracy of the generated output or incurring additional latency from irrelevant API response content. By providing orchestrators such as LLM agents with hints of the relevant information in the API responses while maintaining the original structure of the API responses, the ARC system leverages the capabilities of LLM to improve the LLM technology itself. Furthermore, the ARC system eliminates the need for developers or engineers to reconstruct the API to support the implementation of LLM agents, thus instead providing a scalable and consistent solution that can be implemented even on the largest distributed systems.

[0017] Various aspects of this application will be discussed sequentially and in combination. However, each of the individual aspects may be implemented individually or in combination with other implementations. Although the aspects of this disclosure will be described relative to illustrative network components, interactions, and routines, one or more aspects of this disclosure may be implemented according to various environments, system architectures, client computing device architectures, etc. Similarly, references to specific devices (such as user computing devices) should be considered general references and are not intended to provide additional meaning or configuration for individual user computing devices. Therefore, the disclosed examples are illustrative in nature and should not be construed as limiting unless specifically indicated otherwise.

[0018] Now turn to the attached diagram. Figure 1A block diagram depicts an exemplary environment 100 implementing an API response compression system 120 (hereinafter referred to as "ARC system 120") within the context of a cloud provider network 110. Illustratively, ARC system 120 may act as middleware between an instance of LLM 172 and various application programming interface endpoints 160 (hereinafter referred to as "API endpoints 160"). In some implementations, ARC system 120 compresses API responses for LLM agent 170 to make requests to API endpoint 160 on behalf of LLM 172. Following API response compression, ARC system 120 can immediately output the resulting compressed API response to LLM agent 170, and LLM agent 170 can, in turn, sequentially pass the compressed API response as input to LLM 172 when generating output in response to prompts.

[0019] In some implementations, cloud provider network 110 can provide generative AI capabilities to user computing device 102 via LLM 172. Illustratively, LLM 172 can be any trained machine learning model (e.g., a sequence-to-sequence model, also known as a "Seq2Seq" model) that utilizes deep learning algorithms to process and understand natural language queries or prompts and generate outputs (e.g., text, images, audio, video, etc.). LLM 172 can be trained on large corpora. Furthermore, LLM 172 can be a transformer-based network or other self-focused networks (e.g., an encoder-decoder transformer architecture or a decoder-only transformer architecture). Additionally, LLM 172 can process or compute various language tasks, such as translating languages, analyzing the properties of API responses, chatbot dialogues, etc. LLM 172 can process or compute conversational text data, identify one or more entities and the relationships between them, and generate coherent and grammatically accurate new text.

[0020] As described herein, LLM 172 can process transcription based on prompts and generate output to perform the identified function indicated in the prompt. The prompt may also include additional input information, such as recordings, historical information, profile information, geographic identifiers, etc. Additionally, the prompt may include information that identifies the type or format of the generated output. The aspects associated with ARC system 120 can be implemented as one or more components associated with one or more functions, services, or machine learning models, as well as other components.

[0021] Figure 1User computing device 102 may be connected to LLM 172 via network 104, or LLM 172 may reside on user computing device 102. User computing device 102 may send natural language questions or prompts to LLM 172 (e.g., input from the user via the user interface of user computing device 102) and receive output generated based on natural language questions or prompts from LLM 172. User computing device 102 may be configured to have at least one processor. That processor may communicate with memory to maintain computer-executable instructions. User computing device 102 may be physical or virtual. User computing device 102 may be a mobile device, personal computer, server, or other type of device. User computing device 102 may have a display, speakers, and other output and input devices through which the user interacts with user interface components.

[0022] like Figure 1 As depicted, network 104 connects the devices and modules of environment 100. The network can connect any number of devices. Network 104 can be a personal area network (PAN), local area network (LAN), wide area network (WAN), over-the-air broadcast network (e.g., for radio or television), cable network, satellite network, cellular telephone network, or a combination thereof. As another example, network 104 can be a publicly accessible network of linked networks that can be operated by various different parties, such as the Internet. In some implementations, network 104 can be a private or semi-private network, such as a corporate or university intranet. Network 104 may include one or more wireless networks, such as a Global System for Mobile Communications (GSM) network, a Code Division Multiple Access (CDMA) network, a Long Term Evolution (LTE) network, or any other type of wireless network. Network 104 can use protocols and components for communication via the Internet or any of the aforementioned types of networks. For example, protocols used by network 104 may include Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), Message Queuing Telemetry Transfer (MQTT), Constrained Application Protocol (CoAP), and so on. The protocols and components used for communication via the Internet or any of the aforementioned types of communication networks are well known to those skilled in the art and therefore are not described in more detail herein.

[0023] Cloud provider network 110 can provide an on-demand, scalable computing platform to user computing devices 102 via network 104. For example, cloud provider network 110 allows users to own scalable “virtual computing devices” at their disposal via computing servers (which provide computing instances using one or both a central processing unit (“CPU”) and a graphics processing unit (“GPU”, optionally with local storage) and block storage servers (which provide virtualized persistent block storage for the specified computing instances). These virtual computing devices have the attributes of a personal computing device, including hardware (various types of processors, local memory, random access memory (“RAM”), hard disk and / or solid-state drive (“SSD”) storage devices), operating system selection, networking capabilities, and pre-loaded application software. Each virtual computing device can also virtualize its console input and output (e.g., keyboard, monitor, and mouse). This virtualization allows users to connect to their virtual computing device using computer applications (such as browsers, application programming interfaces, software development kits, etc.) to configure and use their virtual computing device as they would do with a personal computing device. Unlike personal computing devices that have a fixed amount of hardware resources available to the user, the hardware associated with a virtual computing device can be scaled up or down depending on the resources the user needs.

[0024] An API can be understood as an interface and / or communication protocol between a client and a server, enabling the client to receive a response in a specific format or initiate a defined action if the client issues a request in a predefined format. An API can have a specific location within the API that allows the client to interact with the API resource; this specific location can be referred to as the API endpoint. (See diagram). Figure 1 As depicted, API endpoint 160 may be a URL that acts as a point of contact between the API client and the API server. In some implementations, API endpoint 160 may reside outside the cloud provider network 110.

[0025] In an alternative implementation (not shown), API endpoint 160 may reside within cloud provider network 110, thereby providing clients with a gateway to cloud infrastructure by allowing clients to obtain data from or initiate actions within the cloud provider network, enabling the development of applications that interact with resources and services hosted in cloud provider network 110. Such API endpoint 160 may also enable different services within the cloud provider network to exchange data with each other. Users may choose to deploy their virtual computing systems to provide network-based services for their own use and / or for their clients.

[0026] Users can connect to the LLM 172 via user computing device 102 through network 104. More specifically, users can speak or type prompts for the LLM 172 to user computing device 102, which delivers the prompts to the LLM 172 via network 104. In an alternative embodiment, a computing system or device may provide automatic prompts to the LLM 172. The LLM 172 may be part of a larger generative AI service that includes multiple LLM instances provided by a cloud provider network 110 for query / prompt processing and other AI-based tasks. The LLM 172 can dynamically predict and generate the most accurate and appropriate output in the context of a given prompt using probability distributions derived from training data.

[0027] LLM 172 may include additional interface code that enables LLM 172 to interact with API endpoint 160: this additional interface code may be referred to as LLM agent 170. In some implementations, LLM agent 170 may be a coordinator or function call agent for LLM 172. Illustratively, LLM agent 170 may include LLM 172 and an HTTP interface that allows the output of LLM 172 matching a given format (e.g., HTTP GET / POST) to result in a corresponding network call to API endpoint 160. LLM agent 170 may receive an API response returned from API endpoint 160 and then provide this API response as input to LLM 172.

[0028] More specifically, upon receiving a prompt (e.g., a query) involving a network call at LLM 172, LLM agent 170 may immediately send a request to the relevant API endpoint (e.g., via network 104) as part of a process to collect relevant information needed to generate output in response to the prompt. In response to the request from LLM agent 170, API endpoint 160 may return an API response to LLM agent 170, providing the requested information. In some implementations, this API response may be too long to be directly passed to LLM 172 by LLM agent 170 (or may be determined to contain irrelevant information), and therefore the API response will be sent by LLM agent 170 to ARC system 120 for compression, and then input back to LLM 172 (in a reduced / filtered form). In this way, LLM agent 170 uses ARC system 120 as a compression middleware for API responses before providing the reduced API responses to LLM 172, thus mitigating the latency and accuracy issues that LLM 172 might encounter if the API responses were not reduced by ARC system 120 before being input into LLM 172.

[0029] In some implementations, ARC system 120 is used to find elements of the API response relevant to a given query presented to LLM 172 in order to generate a reduced API response. To infer which elements of the API response are relevant to generating the reduced API response, ARC system 120 may implement a three-phase process. In the first phase, ARC system 120 may identify elements present in the API response. In the second phase, ARC system 120 may filter the elements identified in the first phase to those relevant to the query at hand. In the third phase, ARC system 120 may generate the reduced response using the relevant elements from the second phase. In this way, the three-phase process results in a reduced API response, which can be passed as input to LLM 172 by LLM agent 170. Illustratively, ARC system 120 may include three sub-components, and each of the sub-components of ARC system 120 may correspond to one of the three phases of the compression process of ARC system 120. More specifically, exemplary ARC system 120 may include the following three sub-components: a manifest builder 130, an attribute selector 140, and a response refiner 150.

[0030] In the first phase, ARC system 120 may use a subcomponent called manifest builder 130 to identify elements present in the API response provided to ARC system 120 by LLM agent 170. Illustratively, manifest builder 130 may generate an attribute manifest (e.g., a list of elements present in the API response). In some embodiments, manifest builder 130 generates an element list from a response from a given API endpoint 160, thereby optionally enabling the generation of a dynamic attribute manifest based on the actual API response provided to ARC system 120 by LLM agent 170 for compression. In this way, in such embodiments, manifest builder 130 may generate an attribute manifest without using a specific API response.

[0031] In the second stage, the ARC system 120 may include a subcomponent called an attribute selector 140 to filter the elements identified in the attribute list to those relevant to the query at hand. Illustratively, the attribute selector 140 may filter the elements of the attribute list based on salience and relevance to the hints provided to the LLM 172, thus generating a filtered attribute list with relevant elements for use in the third stage.

[0032] In the third stage, the ARC system 120 may include a subcomponent called a response refiner 150 to generate a reduced response using a filtered list of attributes from the relevant elements of the second stage. Illustratively, the response refiner 150 may reduce (e.g., recursively, iteratively, etc.) the API response based on elements selected by the attribute selector 140 from the filtered list of attributes. After this reduction of the API response is complete, the response refiner 150 can then immediately output the resulting reduced API response (illustratively maintaining the same structure as the original API response) to the LLM agent 170. In turn, the LLM agent 170 can pass the reduced API response as input to the LLM 172 for use in generating output in response to a given prompt.

[0033] Figure 2 Based on all aspects of this disclosure Figure 1 A visualization 200 of environment 100 depicts the illustrative interactions between LLM agent 170, API endpoint 160, and ARC system 120 to generate a compressed API response 226 for use by LLM 172. In one implementation, when LLM 172 receives a prompt 210, it enables... Figure 2 The interaction. Prompt 210 may consist of user input from user computing device 102, or, in an alternative embodiment, prompt 210 may be automatically generated from computing devices within the cloud provider network 110. In another alternative embodiment, prompt 210 may be generated by an LLM: for example, LLM 172 may generate prompt 210 for a second LLM (not shown). In another embodiment, an LLM agent may generate prompt 210 for other LLM agents or even itself (e.g., LLM agent 170 may generate prompt 210 for itself). In some embodiments, prompt 210 represents multiple interactions or queries in a conversation with LLM 172, sometimes referred to as a “prompt chain.” In this way, when prompt 210 represents an entire chain of prompts, LLM 172 (in some embodiments, a dialogue model) may receive nuanced context of multi-step or complex prompts from the user before acting on prompt 210.

[0034] In some implementations, LLM agent 170 (e.g., a coordinator) can be a function call agent powered by LLM 172, which generates output with instructions to perform tasks by invoking (e.g., requesting) specific functions of API endpoint 160. In this way, LLM agent 170 can dynamically specify a given API request 220 as a prompt 210 appropriate for the task at hand. For example, a chat-based generative AI service may support a list of function call APIs, and when a user interacts with LLM 172, the supported function call APIs may generate actions or output messages for LLM agent 170, which then passes such actions / output messages to LLM 172 for output to the user.

[0035] Once LLM 172 receives prompt 210, LLM agent 170 can predict whether a function call is needed to generate output in response to prompt 210. For example, prompt 210, which asks LLM 172 to “summarize the outstanding issues in the code repository,” might cause LLM agent 170 to predict that a function call is needed to answer prompt 210 (i.e., an API request 220 for the code repository, requesting a list of repositories and a list of outstanding issues in those repositories). When LLM agent 170 determines that a function call is needed, it sends API request 220 related to prompt 210 to API endpoint 160 of API 262.

[0036] Next, API endpoint 160 generates an API response in response to API request 220 issued by LLM agent 170 and returns it to LLM agent 170. In some embodiments, at this point, LLM agent 170 reaches an optional decision box, whereby LLM agent 170 classifies the API response received from API endpoint 160 as a "long" API response 222 or a "short" API response 224. In some embodiments, factors such as (but not limited to) the calculated token length, token count, number of attributes in the API response, or number of entries in the API response may be included in the determination of the "long" or "short" length of the API response (e.g., calculated by LLM agent 170). In another embodiment, after receiving the API response returned from API endpoint 160, there is no optional decision box for LLM agent 170 because LLM agent 170 is configured to pass all API responses to ARC system 120 for compression / reduction, regardless of their length (e.g., "long" or "short"). In yet another alternative implementation, LLM agent 170 passes the API response to ARC system 120 for compression in response to detecting an error message generated by LLM 172 (e.g., an error message warning that the attempted API response input is too long for the input to LLM 172).

[0037] In some alternative implementations, if API endpoint 160 returns a short API response 224 to LLM agent 170, LLM agent 170 may not pass the short API response 224 to ARC system 120 for compression, because the API response length is already within the acceptable range for optimal operation of LLM 172. However, in this implementation, if API endpoint 160 returns a long API response 222 to LLM agent 170, LLM agent 170 may pass the long API response 222 to ARC system 120 for compression. More specifically, LLM agent 170 passes the long API response 222 as input to manifest builder 130 and response refiner 150, as will be discussed in more detail herein.

[0038] In some implementations, the compression / reduction of the long API response 222 by the ARC system 120 begins at the manifest builder 130. The task of the manifest builder 130 is to generate an attribute manifest 232 (which, in turn, can be used as input to the attribute selector 140 during the compression process). Illustratively, the attribute manifest 232 may list the fields present in the long API response 222. Notably, in some examples, the attribute manifest 232 may describe the fields (e.g., email address, date, time) without including the values ​​within the fields (e.g., jane.smith@email.net, November 30, 11:00 AM). In such examples, the attribute manifest 232 may be designed to describe the fields (rather than the values) because, among other reasons, the API response 222 typically consists primarily of value data (as opposed to field data, which typically appears at a much lower proportion relative to the overall length of the API response). Therefore, by using the fields in the descriptive attribute list 232, the list builder 130 avoids the need to process large amounts of long API responses 222 (e.g., value data), thus saving computational resources, time, and costs associated with operating the LLM 172 while reducing the overall latency of the LLM 172. In some examples, if a given field has multiple values, the attribute list 232 may contain a unique path to each field in the API response (as opposed to a unique path to each value within a given field) (e.g., in a nested JSON tree structure). However, in alternative implementations, the list builder 130 may construct the attribute list 232 from both the fields and values ​​found in the long API response 222.

[0039] In some implementations, attribute list 232 may not describe all fields from the long API response 222. Instead, in such implementations, attribute list 232 may contain a subset (also referred to as "attributes" or "elements") of fields that appear in the actual API response 222. In this way, some irrelevant fields are avoided from being included in attribute list 232. However, in alternative implementations, attribute list 232 may describe all fields contained in the API response 222.

[0040] In an alternative implementation, the manifest builder 130 may not receive a specific API response (e.g., long API response 222) as input, but instead generate a list of attributes for relevant elements / fields based on information provided to the manifest builder 130 by the API endpoint 160. For this reason, in some implementations, an API response (long or short) may not be required as input to the manifest builder 130.

[0041] In some implementations, in addition to the long API response 222, the manifest builder 130 optionally receives API specification 264 as input. While in some implementations the ARC system 120 does not require API specification 264, the API specification provides additional context for the fields in the API response when the attribute manifest 232 is generated by the manifest builder 130. API specification 264 may contain, for example, a description of each field, the possible data types of the fields, and any other such metadata associated with the fields that allows the manifest builder 130 to generate the attribute manifest 232 more accurately. In some implementations, the manifest builder 130 may be instantiated as a machine learning model (e.g., LLM or other sequence-to-sequence model) instructed to collect fields (and values, if applicable) from the API endpoint 160 used to generate the attribute manifest 232, the long API response 222, the short API response 224, and / or the API specification 264. In another implementation, the manifest builder 130 may be instantiated as a regular expression or other parsing software instructed to collect fields (and values, if applicable) from the API endpoint 160, long API response 222, short API response 224 and / or API specification 264 used to generate the attribute manifest 232.

[0042] Once the manifest builder 130 completes the generation of the attribute manifest 232, it outputs the attribute manifest 232. The attribute selector 140 then receives the attribute manifest 232 as input to the next step in the compression process of the ARC system 120. Additionally, at this step, the attribute selector 140 receives the hint 210 initially provided to the LLM agent 170 as input. The attribute selector 140 can use a machine learning model (e.g., LLM or other sequence-to-sequence model) by instructing the machine learning model to select the most relevant attributes (e.g., fields) from the attribute manifest 232. Notably, in some embodiments, this machine learning model may be separate from the LLM 172. Meanwhile, in other embodiments, the machine learning model of the attribute selector 140 may be the same as the LLM 172. In some embodiments, the hint from the attribute selector 140 to the machine learning model can guide the machine learning model to select attributes based on the significance and relevance estimates of each attribute in the attribute manifest 232. For this purpose, the attribute selector 140 can use the hint 210, thereby providing it to the machine learning model as further context for significance and relevance determination.

[0043] Illustratively, attribute selector 140 has completed its task when the resulting filtered attribute list 232 has been narrowed down to the attributes required to answer prompt 210. Because API responses (e.g., long API response 222) can be nested code (e.g., also known as "JSON" in JavaScript object notation), individual attribute lists can actually each contain lists of further nested attributes. For this reason, in some implementations, the output of attribute selector 140 takes the form of a JSON path, thereby allowing the deterministic selection of desired attributes and values ​​from long API response 222. In some alternative implementations, attribute selector 140 may make additional calls to a machine learning model (e.g., LLM or other sequence-to-sequence models) to perform quality checks on the resulting filtered attribute list 232.

[0044] The resulting filtered attribute list 232 is output from the attribute selector and provided as input to the response refiner 150. As previously mentioned, at this step, the response refiner 150 also receives the long API response 222 as input. Using the filtered attribute list 232 output from the attribute selector 140 and the long API response 222, the response refiner 150 prunes the long API response 222 so that it includes fields and values ​​related to the prompt 210 (e.g., selected by the attribute selector 140 and specified in the filtered attribute list 232). In this way, the resulting compressed API response 226 generated by the response refiner 150 at this step contains attributes related to the prompt 210 while maintaining the original structure of the long API response 222 (e.g., a nested JSON structure). In some implementations, to maintain the original structural integrity of the nested JSON code during this step, a path tree is constructed based on the JSON paths of the selected attributes in the filtered attribute list 232. Then, irrelevant content outside such paths can be removed from unnecessary parts of the long API response 222 (e.g., recursively, iteratively, etc.) to produce the final compressed API response 226.

[0045] In an alternative implementation, response refiner 150 may use masking techniques to refine long API responses 222 into reduced lengths. For example, some long API responses have an attribute containing a Uniform Resource Locator (“URL”) that is thousands of characters long. In this example, response refiner 150 may replace this “long” URL with a shorter placeholder of that URL, thus generating a compressed API response 226 for input at LLM 172. In this example, the shorter placeholder may again be replaced with the full URL before LLM 172 generates the final output for prompt 210.

[0046] In another implementation, the response refiner 150 can use a machine learning model (e.g., LLM or other sequence-to-sequence model) to instruct the machine learning model to generate a compressed API response 226 from inputs such as a filtered attribute list 232 and a long API response 222.

[0047] Once the response refiner 150 generates a compressed API response 226, the compressed API response 226 is provided back to the LLM agent 170. At this point, the LLM agent 170 can use the compressed API response 226 to answer prompt 210, or, in an alternative implementation, the LLM agent 170 can invoke another function based on the compressed API response 226, thus repeating... Figure 2The compression process is described. In this way, the ARC system 120 reduces the resource usage and time of the LLM agent 170 in processing API responses, or alternatively, enables the LLM agent to process API responses when the unprocessed portion is too large for the context window of the LLM agent 170.

[0048] Figure 3 This is a flowchart illustrating a routine 300 for compressing long API responses according to various aspects of this application. Routine 300 may start automatically immediately upon receiving prompt 210 from a computing device at LLM 172, or the routine may be enabled on a specific basis by a client or end user. The client or end user may use an interactive system to enable routine 300 or pre-schedule the routine. Routine 300 may be embodied in a set of executable program instructions stored on a computer-readable medium, such as one or more disk drives of a computing system such as a node or server. When routine 300 is enabled, the executable program instructions may be loaded into memory such as random access memory (“RAM”) and processed by the computing system (such as...) Figure 5 One or more processors of the ARC system 120 shown are used for execution.

[0049] Routine 300 begins at box 302, where LLM 172 receives natural language prompt 210. At box 304, LLM agent 170 determines (e.g., predicts) that responding to prompt 210 involves sending API request 220 to API 262 (specifically, API endpoint 160 of API 262). Therefore, at box 306, LLM agent 170 sends API request 220 to API endpoint 160. In turn, at box 308, a response to API request 220 is generated by API 262 and sent from API endpoint 160 to LLM agent 170. Thus, decision box 310 is reached, where LLM agent 170 generates a determination regarding whether the API response generated by API endpoint 160 can be marked as a “long” API response 222 for the purposes of the compression process of ARC system 120. When the API response is short, the routine can proceed to box 318, where the short API response 224 is fed into the LLM agent 170 for use in generating the response output to prompt 210, thus ending the routine. However, conversely, if at decision box 310 the API response is a long API response 222, the routine proceeds to box 312.

[0050] At box 312, the long API response 222 is the input to the ARC system 120 for compression (see [link]). Figure 4The detailed description of routine 400 (the compression technique of ARC system 120 that occurs within box 312 of routine 300) follows. After routine 400 completes within box 312, the resulting compressed API response 226 is immediately received at box 314. It is worth noting that, according to some implementations, the compressed API response 226 is based on the long API response 222 and retains its original path structure (e.g., JSON structure). The routine then ends at box 316, where the compressed API response 226 is fed to LLM agent 170 for use in generating the response output to prompt 210. It is worth noting that, in alternative implementations, routine 300 may be repeated instead of generating the response output at box 316.

[0051] Figure 4 This is a flowchart illustrating a routine 400 for generating a compressed API response 226 using an ARC system 120 with a manifest builder 130, attribute selector 140, and response refiner 150, according to various aspects of this disclosure. The calendar API example previously presented in this disclosure (i.e., an example where a user prompts an LLM for the email addresses of all attendees scheduled to have a meeting with the user on a given calendar day) will be discussed as an illustrative example of routine 400 throughout the description of routine 400. Thus, suppose user Jane Smith provides the following prompt to LLM 172: “Provide the email addresses of all attendees scheduled to have a meeting with Jane Smith on November 30th.” LLM agent 170 may accordingly make an API request 220 to the calendar API for Jane's calendar meeting data, and this calendar API may return a longer API response 222 than necessary, listing (in addition to relevant data including attendee email addresses) various types of irrelevant calendar meeting data (e.g., meeting time, location, attachments, etc.).

[0052] Routine 400 thus begins at box 402, where the manifest builder 130 receives a long API response 222 related to the prompt 210 made to the LLM agent 170. In the calendar API example, the manifest builder 130 receives a calendar API response for Jane that lists meeting times, locations, meeting attachments, meeting attendee names, and meeting attendee emails. It is worth noting that in some implementations, the long API response 222 may be generated from another user or system trigger, rather than from the prompt 210 to the LLM. For example, the calendar API response might be returned due to an internal automation system trigger as part of a larger automation process, rather than from a direct prompt from Jane.

[0053] At box 404, manifest builder 130 generates property manifest 232 from long API response 222 (and additionally API specification 264). In some implementations, API specification 264 is an optional input to box 404. For example, a calendar API provides manifest builder 130 with a calendar API specification that lists all possible fields that the calendar API might return, along with descriptions of those fields and their data types. Note that in this example, if the calendar API does not provide a calendar API specification, manifest builder 130 may alternatively generate property manifest 232 simply from the long calendar API response.

[0054] At box 406, attribute selector 140 receives attribute list 232 and prompt 210 as input. For example, attribute selector 140 receives Jane's prompt and attribute list 232, which lists the following fields from the calendar API: meeting time, location, meeting attachments, meeting attendee names, and meeting attendee emails.

[0055] Next, at box 408, attribute selector 140 narrows down attribute list 232 to those attributes considered relevant to answer prompt 210 (e.g., the LLM prompted by attribute selector 140). For example, attribute selector 140 may prompt the LLM (not shown) with attributes relevant to Jane's prompt, and the LLM may return the following fields: meeting attendee name and meeting attendee email.

[0056] At box 410, attribute selector 140 descriptively outputs the path (e.g., JSON path) for each attribute deemed relevant from box 408. For example, attribute selector 140 may generate a filtered list 232 of attributes that simply lists the JSON paths to the fields “Meeting Attendees Name” and “Meeting Attendees Email” within the Long Calendar API response 222.

[0057] In this way, at box 412, response refiner 150 can obtain the output from box 410 and construct a compressed API response 226 based on the relevant attribute paths. In some implementations, this construction of the compressed API response 226 may require a recursive or iterative algorithm. For example, response refiner 150 may recursively iterate over the long calendar API response 222 to remove JSON code associated with the following fields / values ​​not included in the filtered attribute list 232: meeting time, meeting location, and meeting attachments.

[0058] Finally, at box 414, the routine ends when the compressed API response 226 is returned to LLM 172 for use in generating the response output to prompt 210. It is worth noting that in some implementations, there may be multiple additional steps between providing the compressed API response 226 to LLM 172 and (by LLM 172) generating the final response output to prompt 210. For example, prompt 210 in response to Jane may involve multiple separate calendar API requests 220 (and therefore multiple long API responses 222). In this example, the routine 300, which lists the email addresses of attendees at Jane's meeting on November 30th, may be executed multiple times (e.g., iteratively, in parallel, etc.) before LLM 172 generates the final response output to prompt 210.

[0059] Figure 5 An exemplary architecture of a computing system (referred to as computing system 500) is depicted, which can be used to perform the operations described herein or Figures 1 to 4 One or more techniques illustrated. Figure 5 The overall architecture of the described computing system 500 includes an arrangement of computer hardware and software modules that can be used to implement one or more aspects of this disclosure. The computing system 500 may include more than Figure 5 The diagram shows more (or fewer) of those components. However, it is not necessary to show all of these components to provide a disclosure that can be implemented. As illustrated, computing system 500 includes processor 510, network interface 520, computer-readable medium 530, and input / output device interface 540, all of which can communicate with each other via a communication bus. Network interface 520 provides connectivity to one or more networks or computing systems. Processor 510 can therefore receive information and instructions from other computing systems or services via a network (e.g., connecting computing system 500 to environment 100).

[0060] Processor 510 may also communicate with memory 560. Memory 560 may contain computer program instructions (grouped into modules or units in some embodiments) that are executed by processor 510 to implement one or more aspects of this disclosure. Memory 560 may include random access memory (RAM), read-only memory (ROM), and / or other persistent, auxiliary, or non-transitory computer-readable media. Memory 560 may store an operating system 570 that provides computer program instructions used by processor 510 in the general management and operation of computing system 500. Memory 560 may further contain computer program instructions and other information for implementing one or more aspects of this disclosure. For example, in one embodiment, memory 560 includes a user interface module that generates a user interface (and / or instructions for said user interface) for display on the user computing device, for example, via a navigation and / or browsing interface such as a browser or application mounted on the user computing device.

[0061] In addition to and / or in conjunction with the operating system 570, the memory 560 includes an API response compression system 120 that enables the functionality of this disclosure.

[0062] Although ARC system 120 is Figure 5 While illustrated as part of computing system 500, in other embodiments, all or part of ARC system 120 may be implemented by another computing device. For example, in some embodiments of this disclosure, another computing device communicating with computing system 500 may include several modules or components that operate similarly to those illustrated as part of computing system 500. In some instances, ARC system 120 may be implemented as one or more virtualized computing devices. Furthermore, ARC system 120 may be implemented wholly or partially as a distributed computing system, including a set of devices that collectively implement the functions discussed herein.

[0063] Various additional exemplary embodiments of this disclosure may be described by the following additional terms: Clause 1. A computing device for reducing the response of an application programming interface (API) used by a large language model (LLM), said computing device comprising: A computer-readable storage medium storing executable instructions; and A processor, which communicates with the computer-readable storage memory and is programmed by the executable instructions to: Receive API response data from the API, wherein the API response data is generated by the API in response to a call to the API; A list of attributes is generated for the API response data, and the list of attributes identifies a set of fields in the API response data; Generate a filtered list of attributes that identify the fields in the API response data that are determined to be related to the prompts made to the LLM; At least from the API response data and the filtered attribute list, reduced API response data is generated, the reduced API response data containing one or more values ​​from each field in the filtered attribute list identified as related to the prompt made to the LLM, and excluding at least one value from the API response data that does not correspond to a field in the filtered attribute list not identified as related to the prompt made to the LLM; and The prompt and the reduced API response data are processed at the LLM to generate LLM output.

[0064] Clause 2. The computing device according to Clause 1, wherein the processor is further programmed by the executable instructions to determine that the API response data meets a criterion reduced according to at least one of the following: The token length of the API response data; The attribute count of the API response data; or The number of entries in the API response data.

[0065] Clause 3. The computing device according to Clause 1, wherein the processor is further programmed by the executable instructions to: It was determined that the second API response data did not meet the reduction criteria; Send the second API response data to the LLM; and The prompt and the second API response data are processed at the LLM to generate a second LLM output.

[0066] Clause 4. The computing device according to Clause 1, wherein the processor is further programmed by the executable instructions to generate the property list using an API specification, wherein the API specification provides a description of the set of fields in the API response data.

[0067] Clause 5. A computer-implemented method comprising: Receive API response data from the application programming interface (API); A list of attributes is generated for the API response data, and the list of attributes identifies a set of fields in the API response data; Generate a filtered list of attributes that identify fields in the API response data that are determined to be relevant to prompts made to a large language model (LLM); At least reduced API response data is generated from the API response data and the filtered attribute list, the reduced API response data excluding at least one value of the API response data that is not identified in the filtered attribute list as corresponding to the prompt made to the LLM; and The prompt and the reduced API response data are processed at the LLM to generate LLM output.

[0068] Clause 6. The computer implementation method according to Clause 5 further includes: Determine that the API response data meets a reduction criterion based on at least one of the following: The token length of the API response data; The attribute count of the API response data; or The number of entries in the API response data.

[0069] Clause 7. The computer implementation method according to Clause 5 further includes: It was determined that the second API response data did not meet the reduction criteria; Send the second API response data to the LLM; and The prompt and the second API response data are processed at the LLM to generate a second LLM output.

[0070] Clause 8. The computer implementation method according to Clause 5, wherein generating the attribute list includes using an API specification, wherein the API specification provides a description of the set of fields in the API response data.

[0071] Clause 9. The computer implementation method according to Clause 5, wherein generating the filtered attribute list includes excluding a set of values ​​from the API response data.

[0072] Clause 10. The computer implementation method according to Clause 5, wherein generating the filtered attribute list includes using a second LLM to filter the attribute list of the API response data.

[0073] Clause 11. The computer implementation method according to Clause 5, wherein generating the property list includes a list of APIs using the API.

[0074] Clause 12. The computer implementation method according to Clause 11, wherein generating the reduced API response data includes masking the Uniform Resource Locator (URL) with placeholder variables.

[0075] Clause 13. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a processor, cause the processor to: Receive API response data from the API, wherein the API response data is generated by the API in response to a call to the API; A list of attributes is generated for the API response data, and the list of attributes identifies a set of fields in the API response data; Generate a filtered list of attributes that identify the fields in the API response data that are determined to be related to the prompts made to the LLM; At least from the API response data and the filtered attribute list, reduced API response data is generated, the reduced API response data containing one or more values ​​from each field in the filtered attribute list identified as related to the prompt made to the LLM, and excluding at least one value from the API response data that does not correspond to a field in the filtered attribute list not identified as related to the prompt made to the LLM; and The prompt and the reduced API response data are processed at the LLM to generate LLM output.

[0076] Clause 14. One or more non-transitory computer-readable media as described in Clause 13, further comprising instructions that, when executed by the processor, cause the processor to: Determine that the API response data meets a reduction criterion based on at least one of the following: The token length of the API response data; The attribute count of the API response data; or The number of entries in the API response data.

[0077] Clause 15. One or more non-transitory computer-readable media as described in Clause 13, further comprising instructions that, when executed by the processor, cause the processor to: It was determined that the second API response data did not meet the reduction criteria; Send the second API response data to the LLM; and The prompt and the second API response data are processed at the LLM to generate a second LLM output.

[0078] Clause 16. One or more non-transitory computer-readable media as described in Clause 13, wherein, when executed by the processor, the computer-executable instructions further cause the processor to use an API specification to generate the attribute list, wherein the API specification provides a description of the set of fields in the API response data.

[0079] Clause 17. One or more non-transitory computer-readable media as described in Clause 13, wherein, when executed by the processor, the computer-executable instructions further cause the processor to generate the filtered list of attributes that excludes a set of values ​​from the API response data.

[0080] Clause 18. One or more non-transitory computer-readable media as described in Clause 13, wherein, when executed by the processor, the computer-executable instructions further cause the processor to generate the filtered attribute list by using a second LLM to filter the attribute list of the API response data.

[0081] Clause 19. One or more non-transitory computer-readable media as described in Clause 13, wherein, when executed by the processor, the computer-executable instructions further cause the processor to use an API manifest of the API to generate the attribute manifest.

[0082] Clause 20. One or more non-transitory computer-readable media as described in Clause 19, wherein, when executed by the processor, the computer-executable instructions further cause the processor to generate the attribute list independently of the API response data.

[0083] Depending on the implementation, certain actions, events, or functions of any process or algorithm described herein may be performed in a different order, or may be added, combined, or omitted entirely (e.g., not all described operations or events are necessary for the practical algorithm). Furthermore, in some implementations, operations or events may be performed simultaneously rather than sequentially, for example, through multithreaded processing, interrupt handling, or on multiple processors or processor cores or other parallel architectures.

[0084] The various illustrative logic blocks, modules, routines, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware or a combination of electronic hardware and computer software. To clearly illustrate this interchangeability, the various illustrative components, blocks, modules, and steps have been generally described above in accordance with their functionality. Whether such functionality is implemented as hardware or as software running on hardware depends on the specific application and design constraints imposed on the system as a whole. For each specific application, the described functionality may be implemented in different ways, but such implementation decisions should not be construed as deviating from the scope of this disclosure.

[0085] Furthermore, the various illustrative logic blocks and modules described in conjunction with the embodiments disclosed herein can be implemented or executed by machines designed to perform the functions described herein, such as processor devices, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or any combination thereof. The processor device may be a microprocessor, but alternatively, it may be a controller, microcontroller, or state machine, a combination thereof, etc. The processor device may include an electronic circuit system configured to process computer-executable instructions. In another embodiment, the processor device includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. The processor device may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, one or more microprocessors together with a DSP core, or any other such configuration. Although this document is primarily described with respect to digital technologies, the processor device may also primarily comprise analog components. For example, some or all of the algorithms described herein may be implemented in analog circuit systems or hybrid analog and digital circuit systems. A computing environment can include any type of computer system, including, but not limited to, microprocessor-based computer systems, mainframe computers, digital signal processors, portable computing devices, device controllers, or computing engines within appliances.

[0086] Elements of the methods, processes, routines, or algorithms described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, software modules executed by a processor device, or a combination of both. Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of non-transitory computer-readable storage medium. An exemplary storage medium may be coupled to a processor device, enabling the processor device to read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor device. The processor device and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor device and storage medium may reside as discrete components in the user terminal.

[0087] Unless otherwise specified or understood in the context in which they are used, the conditional language used herein (such as “can,” “may,” “possibly,” “may,” “for example,” etc.) is generally intended to convey that certain embodiments include certain features, elements, or steps that are not included in other embodiments. Therefore, such conditional language is not generally intended to imply that a feature, element, or step is necessary for one or more embodiments in any way, or that one or more embodiments necessarily include logic for determining whether such features, elements, or steps are included in or will be performed in any particular embodiment, with or without additional input or prompts. The terms “comprising,” “including,” “having,” etc., are synonymous and used in an open-ended manner as inclusive, without excluding additional elements, features, actions, operations, etc. Furthermore, the term “or” is used in its inclusive (not exclusive) sense, such that when used, for example, to connect lists of elements, the term “or” means one, some, or all of the elements in the list.

[0088] Unless otherwise specified, disjunctive language (such as the phrase "at least one of X, Y, or Z") should be understood in the context as commonly used to indicate that an item, term, etc., can be X, Y, or Z or any combination thereof (e.g., X, Y, or Z). Therefore, such disjunctive language is generally not intended and should not imply that certain implementations require at least one of X, at least one of Y, and at least one of Z to be present individually.

[0089] Unless otherwise expressly stated, throughout this application, articles such as “a / an” should generally be understood to include one or more of the described items. Therefore, phrases such as “a device configured to…” are intended to include one or more of the described devices. Such one or more described devices may also be configured collectively to perform the described descriptions. For example, “a processor configured to perform descriptions A, B, and C” may include a first processor configured to perform description A and co-operating with a second processor configured to perform descriptions B and C. Unless otherwise expressly stated, the terms “group” and “set” should generally be interpreted to include one or more of the described items throughout this application. Therefore, phrases such as “a set of devices configured to…” or “a collection of devices configured to…” are intended to include one or more of the described devices. Such one or more described devices may also be configured collectively to perform the described descriptions. For example, “a set of servers configured to perform descriptions A, B, and C” may include a first server configured to perform description A and co-operating with a second server configured to perform descriptions B and C.

[0090] While the above detailed description has shown, described, and pointed out novel features applicable to various embodiments, it will be understood that various omissions, substitutions, and changes in the form and details of the illustrated apparatus or algorithm may be made without departing from the spirit of this disclosure. As will be appreciated, certain embodiments described herein may be embodied in forms that do not provide all the features and benefits set forth herein, because some features may be used or practiced separately from other features. All changes within the meaning and scope of the equivalents of the claims will be included within the scope of the claims.

Claims

1. A computing device for reducing the response of an application programming interface (API) used by a large language model (LLM), the computing device comprising: A computer-readable storage device that stores executable instructions; as well as A processor, which communicates with the computer-readable storage memory and is programmed by the executable instructions to: Receive API response data from the API, wherein the API response data is generated by the API in response to a call to the API; A list of attributes is generated for the API response data, and the list of attributes identifies a set of fields in the API response data; Generate a filtered list of attributes that identify the fields in the API response data that are determined to be related to the prompts made to the LLM; At least reduced API response data is generated from the API response data and the filtered attribute list, the reduced API response data containing one or more values ​​of each field in the filtered attribute list that are identified as being related to the prompt made to the LLM, and excluding at least one value of the API response data that does not correspond to a field in the filtered attribute list that is not identified as being related to the prompt made to the LLM; and The prompt and the reduced API response data are processed at the LLM to generate LLM output.

2. The computing device of claim 1, wherein the processor is further programmed by the executable instructions to determine that the API response data satisfies a reduction criterion according to at least one of the following: The token length of the API response data; The attribute count of the API response data; or The number of entries in the API response data.

3. The computing device of claim 1, wherein the processor is further programmed by the executable instructions to: It was determined that the second API response data did not meet the reduction criteria; Send the second API response data to the LLM; and The prompt and the second API response data are processed at the LLM to generate a second LLM output.

4. The computing device of claim 1, wherein the processor is further programmed by the executable instructions to exclude a set of values ​​from the API response data to generate the filtered list of attributes.

5. The computing device of claim 1, wherein the processor is further programmed by the executable instructions to generate the filtered attribute list by using a second LLM to filter the attribute list of the API response data.

6. The computing device of claim 1, wherein the processor is further programmed by the executable instructions to generate the property list using an API specification, wherein the API specification provides a description of the set of fields in the API response data.

7. A computer-implemented method, comprising: Receive API response data from the application programming interface (API); A list of attributes is generated for the API response data, and the list of attributes identifies a set of fields in the API response data; Generate a filtered list of attributes that identify fields in the API response data that are determined to be relevant to prompts made to a large language model (LLM); At least reduced API response data is generated from the API response data and the filtered attribute list, the reduced API response data excluding at least one value of the API response data that is not identified in the filtered attribute list as being related to the prompt made to the LLM; and The prompt and the reduced API response data are processed at the LLM to generate LLM output.

8. The computer implementation method according to claim 7, further comprising: Determine that the API response data meets a reduction criterion based on at least one of the following: The token length of the API response data; The attribute count of the API response data; or The number of entries in the API response data.

9. The computer implementation method according to claim 7, further comprising: It was determined that the second API response data did not meet the reduction criteria; Send the second API response data to the LLM; and The prompt and the second API response data are processed at the LLM to generate a second LLM output.

10. The computer implementation method of claim 7, wherein generating the attribute list includes using an API specification, wherein the API specification provides a description of the set of fields in the API response data.

11. The computer implementation method of claim 7, wherein generating the filtered attribute list includes excluding a set of values ​​from the API response data.

12. The computer implementation method of claim 7, wherein generating the filtered attribute list includes using a second LLM to filter the attribute list of the API response data.

13. The computer implementation method of claim 7, wherein generating the attribute list includes an API list using the API.

14. The computer implementation method of claim 13, wherein the attribute list is generated independently of the API response data.

15. The computer implementation method of claim 13, wherein generating the reduced API response data includes masking the Uniform Resource Locator (URL) using placeholder variables.

Images